Experimentation Resources

 Building a program from the ground up

• Experimentation at Yelp: Harvard Business School case on the work my team and I did to build the Yelp experimentation program

• Building a Culture of Experimentation by Stefan Thomke

• The Yelp Story: Building an Experimentation Program From the Ground Up by Rohan Katyal

Terminology

• Power analysis: [Primer](https://machinelearningmastery.com/statistical-power-and-power-analysis-in-python/#:~:text=Statistical power is the probability,effect size%2C and statistical power.), Calculator

• Multiple comparison: Primer, Bonferroni correction

• Sample ratio mismatch: Primer & Diagnosis

• Confidence intervals

• Simpson's Paradox

• Hypothesis

Avoiding the common gotchas

• Ron Kohavi's talk on pitfalls in online controlled experiments

• Good Experiment, Bad Experiment by Fareed Mosavat

• Avoid the Pitfalls of A/B testing

• Diagnosing Sample Ratio Mismatch

Writing a great hypothesis

• A strong hypothesis

Prioritization

• Darius Contractor's experimentation prioritization framework called EVELYN

• The Growth Experiment Management System that Tripled Our Testing Velocity

Building your own platform

• Under the Hood of Uber's Experimentation Platform

• Scaling Airbnb's Experimentation Platform

• Facebook' machine learning experimentation tool Ax

Building safeguards

• Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned

Statistics

• Bayesian A/B Testing vs Frequentist A/B Testing – What’s the Difference?

• The Power of Bayesian A/B Testing

Experimentation at different companies

• Airbnb

• Netflix

• Twitter

• Microsoft