For the Practical Issues in A/B Testing Session I gave a talk on some issues that come up when integrating 3rd party Online Controlled Experiments. The full abstract and slides can be found below.

Data Science Practitioners have access to an array of different solutions when deciding how to implement an AB-testing platform for a product. At the top level of decision making is the traditional build vs. buy decision and, assuming the “buy” option there are a multitude of solution providers (Google Analytics, Split, Amplitude, etc.) which compete on both costs and features. A frequently overlooked aspect of this decision is that it is made against the backdrop of the analytics or data tracking system currently in place at an organization. Surprisingly, the success or failure of an AB-testing initiative is often determined by the integration between whatever AB-testing solution is chosen and the current analytics / tracking system. In this talk we will present a framework for analyzing this integration while will allow us to understand and predict the issues that may arise and, preemptively, avoid them.

The slides to the talk can be found here