Traditionally online experimentation has used statistical significance as the gold standard for shipping changes. But it has turned out to be fool's gold. Using statistical significance to decide to ship changes costs businesses a substantial amount in lost value. Moreover, sometimes teams refrain from running experiments because they don't think they'll be able to reach statistical significance or that the experiment will slow down velocity. This results in poorer measurement and less understanding for the business. In this session we'll talk about a few alternatives to statistical significance and then use concrete examples to show how they allow you to make decisions faster with less data, and add more value to the business. Concepts covered will include Test & Roll, Multi-Armed Bandits, and Bayesian Statistics.

Takeaway

Learn how to use your existing experimentation framework to make better decisions
Learn how to think about risk when making decisions under uncertainty
Be able to communicate the value in making data-driven decisions outside of the confines of statistical significance