10  Module 10: A/B Testing

Learning objectives

• Discuss why the methods learned in past courses are not sufficient to answer the more complex research problems being posed in this course (in particular stopping an A/B test early).

• Explain sequential testing and principled peeking and how it can be used for early stopping of an experiment (e.g., A/B testing).

• Write a computer script to perform A/B testing optimization with and without using principled peeking.

• Discuss the tradeoff between stopping earlier and certainty of significance, and the real world implications (e.g., what does the FDA require for early stopping of clinical trials versus Facebook ads optimization?).

• List other questions related to A/B testing optimization that may be relevant in a real data application (e.g., what features cause a Facebook ad to perform best?)