For an Invited Session entitled Online Experiements and A/B Testing I gave a talk on some issues that come up when integrating 3rd party Online Controlled Experiments. The full abstract and slides can be found below.

Data Science Practitioners have an array of options when implementing an AB-testing or Online Controlled Experiment system. At the first level is the “build” vs. “buy” decision with all its traditional trade-offs. If “buy” is chosen there are a multitude of solution providers (Google Analytics, Split, Amplitude, etc.) who compete on both costs and features. A relatively overlooked aspect of this decision is how the success or failure of an AB-testing initiative is intertwined with an organization’s current data infrastructure. Organizations frequently underestimate the total costs to integrate with a third-party system and, in doing so, can end up choosing a suboptimal option. These costs can include engineering time, a loss of product features, time spent importing and exporting data as well as time lost to pursuing down data rabbit. In this talk we will present a framework for analyzing integrations which will allow us to understand and predict issues before they arise and preemptively avoid them.

The slides to the talk can be found here