Advertising Works

Yahoo! Research is measuring the effects of advertising on sales through a controlled experiment on Yahoo! display advertising. Researcher David Reiley has recently collaborated with Yahoo!’s Marketing Insights team in their ongoing efforts to help advertisers evaluate the effectiveness of their advertising campaigns. Reiley, who joined Yahoo! Research because of his longstanding interest in field experiments, approached this project with tremendous enthusiasm because of the unique opportunities to generate experimental data on a large scale with Yahoo! and its advertisers. “We want to provide a service to advertisers to measure whether their advertising campaigns are producing the desired effects on actual sales,” says Reiley.

The question of advertising’s effects on sales has long been debated by both academics and practitioners. When an advertiser runs a TV commercial, a newspaper ad, or an online ad campaign, how much does this advertising increase their sales? Previous work has usually struggled to measure the causal impact of advertising because of data limitations. The question is not just the quantity of data, but also its quality, according to Reiley. Usually, sales data cannot be linked directly to advertising data, but even when it can, subtle problems of interpretation can arise.

For example, Reiley points out a potential weakness of a study reported in a recent Harvard Business Review article, which measured large increases in sales due to online advertising. The study used large quantities of data from comScore, a key online information provider that logs the Internet browsing behavior of two million users worldwide. By comparing the purchases of those who saw a given online ad with the purchases of those who do did not see it, the study concluded that there are large positive effects of online advertising.

However, this straightforward approach raises some issues. “The population of people who sees a particular ad may be very different from the population who does not see the same ad,” says Reiley. For example, consider measuring the effects of a search advertisement placed by eTrade. An example of this is an eTrade ad that shows up on someone’s search results. Those who saw the ad were people who searched for related terms, such as “online brokerage.” By contrast, those who did not see the ad did not search for those same terms. When we compare sales of new eTrade accounts across the two groups, then, the differences may not be due merely to the presence of ads. Even in the absence of ads, people who searched for “online brokerage” were more likely to sign up for an eTrade account than people who did not. Instead of being a causal effect of the ads, the difference in sales might have reflected a difference between two different populations of users.

At Yahoo!, researchers chose to run a controlled experiment in order to eliminate such potential biases in measuring the causal impact of advertising. Reiley -- together with Taylor Schreiner from the Marketing Insights group -- made use of a database match between Yahoo! and a nationwide retailer by identifying users who registered the same email address with both companies. After finding over one million matched users, the researchers randomly assigned them to treatment and control groups for one of the retailer’s online advertising campaigns. The treatment group saw three ad campaigns in the fall of 2007 and the winter of 2008. The control group did not see those same campaigns.

The project then tracked sales each week at the retailer, both online and in stores. Customer privacy was respected by making sure data shared between Yahoo! and the retailer could not later be traced to any individual user email address or other identifying information. Comparing purchases by the treatment group to those by the control group allowed the researchers to measure the impact of advertising on sales while holding all other factors constant.

The researchers produced a number of interesting statistical findings over the summer. In a paper co-authored with summer intern and MIT PhD student Randall Lewis, Reiley found that the online display advertising increased total revenues by approximately 5% for those users exposed to the ads, with 93% of the total effect happening in offline sales. They also observed online ads to have a large impact on sales even when the ads are not clicked: 78% of the increase in sales came from those who viewed, but did not click, the ads. The positive effects of advertising on sales persisted for a period of weeks after the campaign ended. The ads had a statistically significant effect both on the probability of purchase and on the average purchase amount, with about ¾ of the treatment effect coming through increases in the average purchase amount. Lastly, the team found evidence suggesting that the effects of advertising may be countercyclical, in the sense that advertising’s impact was greatest in weeks when total sales were smallest.

Reiley hopes to continue his experiments with other retailers to further investigate new measurements about the economic effects of advertising. He notes that historically, firms have not made much use of experimental methods, but believes that this is beginning to change. “It’s risky to try something different than the norm in your field,” says Reiley. “But if advertisers can be provided with a solid way of measuring the effect of their ads and how to best direct their ad campaigns, I think firms will start to experiment more.”