Abstract

The measurement of ad effectiveness is one of the central problems of online advertising. Typically the performance is measured by investigating the proportion of people who converted or performed other success actions after they saw the ads. These metrics commonly overestimate campaign effectiveness since they do not account for users who would have performed actions even if the campaign did not happen. Conventional metrics also fail to answer the following questions that are important to advertisers: 1) Which users convert because they see the ad and which users would have converted even if they do not see the ad? 2) What is the cumulative effect of multiple advertising strategies on performance? 3) How does a campaign affect the size of the potential audience pool? In this paper we propose a general methodology for assessing campaign performance that addresses all of these questions. Our method does not require randomized experiments or additional ads to be shown. We develop a unified causal modeling framework that establishes a causal relationship between seeing an ad and performing an action, which is based on propensity methodology embedded in a parallel computation algorithm. We derive a novel robust rank test for model validation. We also provide innovative interpretations of the estimation results by the causal inference, addressing `smart cheating' of online ads (i.e. targeting the users who are likely to convert even without any ad exposure, which does not add value to the advertisers). The three components (model, validation, and interpretation) complete a unified solution to ad effectiveness measurement. The framework is applied to three online campaigns involving millions of unique users. Results from real online campaigns show that this methodology is robust to online data sparseness, high dimensionality and biases from user features. This paper focuses on measuring the effectiveness of online ads, but the framework is readily applicable to measure the effectiveness of other kinds of treatments on various user metrics, for example the impact of different strategies on user engagement metrics.