Advertising Incrementality Measurement Using Controlled Geo-Experiments: The Universal App Campaign Case Study

Measuring the incrementality (effectiveness) of advertising is a critical task for advertisers for financial planning and optimal budget allocation among different online channels. Recent literature has consistently recommended the use of experiments to estimate advertising effectiveness reliably. Closed Ad networks often prevent advertisers from having direct access to user-level traffic for business privacy reasons. As a result, running experiments by randomizing users is rarely feasible for advertisers. We present a controlled experiment design and an effect estimation framework focused on advertisers’ side by leveraging geo-targeted spend interventions at market-level. Our method is based on the selection of the best pair of markets for testing, conditional on a pre-determined effect estimation method, preventing any model tuning bias. We use a Bayesian structural time series to predict the treatment conversions counterfactual based on the observed control market conversions. We present the results of a field experiment of a Universal AppCampaign (UAC), a recent mobile ad campaign format. We find evidence that this advertising format causes incremental conversions, despite the limited campaign customization options. We measure a 6.57% decrease of conversions (statistically significant) when UAC spend is suspended. To our knowledge, our work is one of the earliest studies that successfully measures the incremental value ofUAC with controlled experiments.