Designing marketing experiments

Business Case #2 DESIGNING MARKETING EXPERIMENTS Introduction In his quarterly budget presentation, Larry Culp, brand manager for BigHoney cereal, requests funds for an advertising campaign highlighting new packaging that retains freshness better and longer. In response, Mark Weinberg, BigHoney CFO, asks Culp, “Can you convince me that sales of BigHoney will be hurt if you do not advertise?” As a follow -up, he asks, “You have requested $500,000 for a national campaign? Is that the right amount? Can you get the same result for $250,000?” How can Culp convince Weinberg? Culp’s challenge is typical for marketing managers who need to invest mone y in the marketing mix with the expectation that sales will increase in the future. Attributing an increase in sales to a specific marketing action is a major challenge, because the effect on a brand of any single marketing activity is difficult to isolate ; it consists of several levers being pulled at the same time, including price promotion, new product introductions, competitive actions, television advertising, PR events, and seasonality. Also, sales resulting from such inputs take time; a television adv ertisement is unlikely to compel a viewer to immediately jump off the couch, run to the store, and buy a soda. Isolating the influence of a specific event on consumer behavior can be a daunting task. One way to isolate such influences is through experimen ts. In our example, Culp could, on a smaller scale, measure the effect of his proposed campaign on the brand’s sales. Return on investment (ROI) — and a prospective budget — could then be estimated by projecting any identified lift to a national scale. But how does one design an experiment that would provide accurate results? Establishing Causality Four key rules determine a causal relationship between two variables or factors. Let us consider Culp’s challenge. The marketing campaign may be considered effecti ve if:  Launching the marketing campaign increases BigHoney sales.  Not launching the marketing campaign causes no change to the sales figures.  Launching the marketing campaign today affects sales in subsequent time periods.  There were no other establish ed external factors (e.g., competitive action) affecting the sales of BigHoney. To clearly ascertain whether the targeted sales increase could be achieved without spending marketing dollars or whether any marketing spend is warranted, it becomes very impo rtant for Culp to establish causality. After -Only Experiment An experiment provides a mechanism to manipulate one or more input factors while controlling all other factors and observe changes in an output of interest, such as sales or brand awareness. Business Case #2 A very basic experiment that can be designed by Culp is illustrated in Figure 1. Culp recruits 1,000 participating customers, half of whom — the test group — are exposed to the new advertisement highlighting the new packaging technology; the other half — the cont rol group — are exposed to the old advertisements. If cereal purchases by all 1,000 customers are tracked, the difference in sales between the test and control groups will indicate the magnitude of the potential sales lift provided by the new advertising ca mpaign. Such an experimental design is called after -only because we measure the sales of BigHoney among the participants of the experiment only after they are exposed to the advertisement. Figure 1. After -only experiment design. The after -only design s atisfies two of the four conditions for causality: sales increase in the short term and in subsequent periods. It cannot indicate whether the increase might have occurred without the new advertisement, nor whether preference differences existed prior to th e experiment. The underlying issue here is the extent to which the participants in the test and control groups are similar in terms of the factors relevant to the experiment. The more similar the factors are, the more the two pools of subjects are exposed to the same external environment — store promotions, competitive reactions, even the same weather — and then the causal inferences are more reliable, because the only difference would be the advertisement campaign to which each was exposed.

Therefore, the exp erimenter could, with confidence, attribute a causal relationship between the marketing input and product sales. Test and Control Group Participants For the experimenter, deciding how to distribute customers between the test group and the control group is critical. There are two primary ways to select control groups: randomization and attribute matching. Randomization involves allocating participants r andomly between the test group and the control group. With a big enough sample size, randomization will help improve the similarity between the test and control groups. Consider that Culp is using an e -mail advertisement campaign Business Case #2 and has at his disposal a list of 1,000 BigHoney customers. In a random assignment, he would assign every other customer to the test group and the rest of the customers to the control group.

Randomization creates fairly homogeneous test and control groups because it removes all sou rces of extraneous variation, which are not controllable by the experimenter. The chance that the test and control groups end up being different even with random assignment decreases as the sample size increases. For most practical marketing applications, a sample size needs to exceed at least 100 participants for a reliable random assignment process. Attribute matching is used when the available sample size is not large enough to permit random assignment. Participants are assigned based on certain known a ttributes such as demography, geography, or annual income. If Culp were testing the effect of a TV advertisement campaign, he would be better off choosing cities that are similar in key demographic or psychographic attributes critical to BigHoney’s sales. Before -After Experiment A before -after design (Figure 2) requires an experimenter to measure the output of interest both before and after the participants have been exposed to the inputs. In this before -after design for an e -mail advertising campaign, C ulp would randomly divide the 1,000 participants into test and control groups, as with the after -only design. Both groups would be exposed to the old advertising campaign, and sales in the respective campaigns would be recorded. Let Sales before be the dif ference in sales between the test and control groups when they are exposed to the old campaign. The test group is then exposed to the new advertising campaign, whereas the control group is still exposed to the old campaign. The difference in sales between the test and control groups now is termed Sales after . The lift in sales due to the new advertising campaign is then calculated as Sales = Sales after − Sales before . Subtracting Sales before from Sales after allows the experimenter to control for preexis ting differences between the test and control groups. This before -after design, along with random or matched assignment of participants, provides a belt -and -suspenders approach for controlling for all external differences between test and control groups. Figure 2. Before -after experiment design. Business Case #2 If BigHoney cereal is sold to retailers for $1.59, and the cost of goods is 99 cents, the unit contribution equals $0.60. The lift of 100 units in the experiment translates to $60. If the cost of a single e -mail sent to a customer is $0.10, the cost of e -mails to 500 test -group customers is $50. The experiment suggests that the e -mail campaign provides an ROI of 20%. This ROI estimate can be used by Culp to plan national campaigns. Alternatively, if 500 e -mails p rovide $10 in contribution, the contribution from a single e -mail is $0.02. If the target lift from a national campaign is $100,000, the experiment suggests that Culp would require 5 million e -mails to attain the target lift. Field Experiments When exper iments are conducted within a natural setting, they are termed field experiments. In some industries, field experiments are a part of everyday business. Retail outfits regularly use catalogs or e -mails to conduct massive field experiments that assess consu mer price sensitivity and optimal catalog design. One advantage of field experiments is that subjects are seldom aware that they are part of an experiment, so the collected data is more likely to represent the realities prevalent in the marketplace. The di sadvantage is that it is very difficult to control extraneous variables or to manipulate inputs precisely. Business Case #2 Field experiments are also very transparent to the competition, and competitive reaction could cloud the results. If, during Culp’s experiment, his competition, BigSugar, launches a promotion, the results could be overly pessimistic. But field experiments are still preferred because they allow the marketer to test a campaign with customers in a natural setting, increasing the accuracy of any predictio n. In general, it is easier to experiment with pricing, product, or promotion decisions than with place or channel management decisions. If Culp also wants to determine the best price for his new packaging, he can create different test conditions where ad vertising, promotions, and coupons are all the same, and the only difference is price. As part of the experiment, Culp could introduce BigHoney with its new packaging in three cities (selected because they are similar in factors that affect BigHoney’s sale s). The only difference is that the products are priced differently in each city: $1.59, $1.89, and $2.15. The sales figures are then tracked in the three cities over time. The city where the product is priced at $1.59 could be expected to have higher sale s volume — the question will be how much higher? The experiment results, based as before on the original cost of goods sold being 99 cents, are given in Table 1: Table 1. BigHoney sales data. Given this data, Culp is better off introducing the product at the $1.59 price point, because it provides the highest profit. Web Experiments Because they can be executed quickly and cheaply, web experiments have gained a significant edge over traditiona l offline field experiments. Consider the difference between TV and e -mail advertising campaigns for Culp. With TV advertising, Culp has to buy spots in different channels in the test markets with significant lead time. Once the video is shot, it is diffic ult, time -consuming, and very expensive to change it. Furthermore, the cost of the experiment increases rapidly with each new version that Culp would like to test. The e -mail advertisement, on the other hand, can be created much more quickly and at a much lower cost, making it easier and less expensive for Culp to test different versions. The faster execution and lower cost of web experiments allows marketers to easily test the simultaneous influence of multiple inputs. When Culp wants to test three differ ent campaigns, “Lasts Longer,” “Tastes Better,” and the current campaign, “Good for You,” each at three different price points ($1.59, $1.89, and $2.15), he can create the full factorial design shown in Table 2. Table 2. Full factorial design. Business Case #2 Each cel l in Table 2 represents a combination of advertisement copy and price point that is tested in the experiment. Since we are testing three types of copy and three price points, the total possibilities that need to be tested are 3 × 3 = 9. The profit from eac h combination is provided within each cell. A T V advertisement campaign would have required Culp to recruit nine different cities for the experiment (one for each cell), as well as retailers in each city willing to manipulate the prices, a very expensive and time -consuming process. If BigHoney is sold direct to consumers through a website, then Culp can randomize the e -mails sent to consumers to match one of the nine cells in Table 2. The e -mail open rates, click -throughs to the website, and the subsequent purchases when consumers visit the website can be tracked for each consumer. This provides Culp with a much stronger sense of the effectiveness of the campaign because the same consumer is tracked from exposure to purchase. The full factorial design also allows Culp to test combinations of the advertisement message and price point. In Table 2, we see that the “Tastes Better” message with a price point of $2.15 provides the highest profit, followed by the “Lasts Longer” campaign at the $1.59 price point. W e also see that when the price is maintained at the current level of $1.89, the “Lasts Longer” campaign provides the highest profit. At least in this case, had Culp tested only the advertisement campaigns and not the different prices, he would have wrongly concluded that the “Lasts Longer” copy provided the highest profits. Natural Experiments In a natural experiment, a marketer observes the effect of certain naturally occurring incidents on customer behavior and other factors, such as sales volume. Recog nizing such occurrences allows companies to learn about their customers at no or little additional expense. A classic example is Amazon collecting sales tax data from California residents. Analyzing the effect of a newly levied tax on sales volume will giv e Amazon an opportunity to discover how a sales tax affects online retailing. Amazon could compare sales before and after the sales tax introduction for customers who lived on either side of the state’s border. The only change would be the newly introduced taxation of online purchases, which affects consumers only on one side of the border. The most important part of identifying and analyzing natural experiments is to find test and control groups created by some external factor. Many marketers resort to ge ographic segmentation for natural experiments, but it will not always be a distinguishing characteristic. For example, when the Ford Motor Company introduced an employee pricing promotion, there was no natural geographic separation; all customers were offe red the same deal. Instead, marketers compared sales in the weeks immediately before and after the program was introduced. Business Case #2 Ford Motor Company discovered that the jump in sales levels was accompanied by a sharp increase in prices. Customers presumed that t hey were getting a good deal, but the prices on many models were actually lower before the promotion than at the time of the employee discount prices. But customers responded to the promotion despite the prices, not because of the prices. T he program led t o many happy customers, even though they were paying higher prices. Challenges The accuracy of data obtained from a marketing experiment increases with the experiment’s duration. For example, experiments that have shorter durations might not adequately a ccount for the carry -over effects of marketing interventions; however, marketing decisions are mostly sensitive to time, highlighting the tension between quick versus accurate decisions. The longer the gap between the field experiment and the full campaign , the less accurate the prediction from the field experiment. Yet the time required to obtain buy -in for the field experiment results could delay the timing of the full campaign and thereby the relevance of the field experiment. If an experiment involves s alespeople, the mere knowledge of being in an experiment could change their behavior (the demand effect), leading to biased conclusions. Experiments provide a bridge between new ideas and management decisions. Digital marketing has popularized experimenta tion in the marketing community. Organizations can succeed if they develop a system to learn from experiments and strive toward continuous improvement. Business Case #2 Questionnaire Q1. Mark Weinberg is quite correct in asking if the budget of $500,000 for an advertising campaign could be streamlined, what systems are in place that would prove beyond doubt that a cheaper campaign would have less impact on sales? Q2. As stated in the case that not launching the campaign does not affect sales then how much increase in sales can be projected if a budgeted campaign of half a million dollars is given the go ahead and would it be cost effective? Q3. If the test group of 1000 people are in full k nowledge of the product design changes would this give a true account of the alterations in buying behavior or would it merely suggest that because of the trial behaviors would revert back to the original state when it was completed? Q4. When considering t he description of figure 2 and the cost of e -mails against the uplift i n sales can it truly be said that such a test of a pool of 1000 customers is going to make such an impact on overall sales? Q5. (True/False) The inception of web advertising proves to be a much more cost effective than that of television ads, campaigns can be altered with minimal cost and can be directly targeted to existing customers by the use of emails, and with the use of a "read" receipt the data can be collected more precisely. Q6. Advertising is such a vital tool in the promotion of any product, especially in such a market as food sales, that constant updating of campaigns are required. So easily can one product have sales reduced by a mor e intensive campaign by a rival? Q7. When producing an a dvertising campaign is the duration of the campaign an exacting factor, as in, will the length of the run be exponentially more beneficial is extra sales or will the growth plateau before the end, thus increasing the overall cost without additional sales? Business Case #2 Q8. In a normal situation would a reduction in the cost of the unit be more effective in additional sales, for instance by $0.10, than the cost of running an expensive ad campaign? Q9. (True/False) It has been shown, table 1 that a reduced number of sales due to addit ional price per unit makes no difference to overall profit, in fact it reduces in comparison to larger sales by the lower priced item.

Therefore the sensible approach would be to create a recommended retail price across all cities. Q10. There is to be a ca mpaign which promotes a new design on the cereal packaging which projects an increase in sales, however would it not be more effective if this was coupled with a slightly improved product?