Media Testing Proves Impact

Summary

We tested the impact of two versions of a TV spot for Goodwill to assess the difference in impact on people’s intentions to donate items. The study used experimental design with an unexposed control group serving as the baseline. Both TV spots had a significant impact on people’s intention to donate, with one TV spot driving an 8% lift.

Our Role

  • Digital Media
  • Traditional Media
  • Video Production
  • Campaign Development
  • Customer Experience

Summary

We tested the impact of two versions of a TV spot for Goodwill to assess the difference in impact on people’s intentions to donate items. The study used experimental design with an unexposed control group serving as the baseline. Both TV spots had a significant impact on people’s intention to donate, with one TV spot driving an 8% lift.

Our Role

  • Digital Media
  • Traditional Media
  • Video Production
  • Campaign Development
  • Customer Experience

The Challenge

Goodwill of Central and Northern Arizona is the largest Goodwill organization in the country. The client operates 84 retail stores, 20 donation centers, 3 clearance outlets, 3 furniture stores, and 20 career centers and has revenues nearing $150 million.

Zion & Zion has served as the client’s Agency of Record for the past several years.

The challenge was twofold:

  1. Confirm that both of the two TV commercials we developed for Goodwill will have an impact on people’s intention to donate items to Goodwill
  2. Determine which of the two TV commercials will have the greatest impact on people’s intent to donate goods

The Methodology

The obvious intent of developing the TV commercials for Goodwill in the first place was to drive audience action. Specifically, these spots were aimed at driving people’s donation of items to Goodwill.

And while we know that donations increased during the period that these TV spots ran, we wanted to answer questions related to attribution and the exact effects these TV spots had on donation intention.

To answer these questions, we developed the following hypotheses:

H1: Family Affair will increase donation intentions as compared with the control group

H2: Playing House will increase donation intentions as compared with the control group

H3: Playing House will increase donation intentions as compared with Family Affair

To test these hypotheses our strategy and research team used an experimental design with three groups as follows:

Group 1: 2364 people in the control group were unexposed to either TV spot

Group 2: 2460 people were exposed to the TV commercial nicknamed “Family Affair”

Group 3: 2268 people were exposed to the TV commercial nicknamed “Playing House”

The Results

All of our hypotheses were tested using one-tailed tests. The results were as follows:

Results

 

Social desirability bias (Maccoby and Maccoby 1954) is likely to be present in many of the respondents’ answers in this study, as it has been found to occur in nearly all types of self-report measures (Levy 1981; Peltier and Walsh 1990; Robinette 1991; Simon and Simon 1975). However, while social desirability bias may be partially responsible for the results in H1 (Family Affair outperforming the control) and H2 (Playing House outperforming the control), this is likely not a factor in H3 (Playing House outperforming Family Affair) as there is no reason to believe that one of these TV spots drive a social desirability response bias any more or less than the other TV spot.

As part of our ongoing partnership with Goodwill, we set out to further understand attribution and relative impact of different creative on our audience’s formation of intentions, and indeed this case study shows that both creative treatments that we produced increased intention to donate and that one creative treatment decisively outperforms the other.

All too often in the advertising agency industry, too much emphasis is placed on the “cool factor” of creative and not enough attention is paid to understanding the quantitative impact of creative and the mechanisms by which it operations. It is this kind of ongoing, data-driven, creative development and analysis that continues to be a hallmark of our approach here at Zion & Zion.

Resources

Levy, Sidney J. (1981), “Interpreting Consumer Mythology: A Structural Approach to Consumer Behavior,” Journal of Marketing, 45 (Summer), 49-61.

Maccoby, Eleanor E. and Nathan Maccoby (1954), “The Interview: A Tool of Social Science,” in Handbook of Social Psychology, Vol 1, ed. Gardiner Lindzey, Cambridge, MA: Addison-Wesley, 449-487.

Peltier, B. David and James A. Walsh (1990), “An Investigation of Response Bias in the Chapman Scales,” Educational and Psychological Measurement, 50 (Winter), 803-815.

Robinette, Randy L. (1991), “The Relationship between the Marlowe-Crowne Form C and the Validity Scales of the MMPI,” Jounral of Clinical Psychology, 47 (May), 396-399.

Simon, Julian and Rita Simon (1975), “The Effect of Money Incentives on Family Size:  A Hypothetical-Question Study,” Public Opinion Quarterly, 38 (Winter), 585-595.

Top