QUESTION

cuisine, the alternative algorithm shows advertisements when a user searches for a specific restaurant

and selects two restaurants with similar ratings and hours. While they are reasonably satisfied with

how the current algorithm shows ads, they are open to the possibility that the alternative design may

be significantly better (or significantly worse) in providing benefit for their advertisers.

Experiment

For the experiment, RG randomly selected 30,000 restaurants that were active on their platform but

were not currently advertising, yielding a sample that is representative of their population of

restaurants in the US.

For the one-month duration of the experiment, 10,000 restaurants were randomly selected to receive

free ads using the current advertising approach, and another 10,000 restaurants were randomly

selected to receive the alternatively designed ads The main difference between these two treatment

groups is that the alternative design used a very different algorithm to decide when to deliver ads and

which ads to pair with each search, as described above. The rest of the 10,000 restaurants received no

The spreadsheet supplement for this exercise (HBS No. 916-702) contains a variety of outcomes

observed for these 30,000 restaurants during the one month of the experiment.

The spreadsheet supplement contains data for each restaurant in the experiment observed in the

month during the experiment. The unit of observation is a restaurant-month, so data of each restaurant

are in a single row. For example, in a row, pageviews means the number of unique visits on restaurant's

RG page during that month. The variables included are as follows:

Variables

Variable Name Definition

treatment =0: in the control group

=1: in the first treatment group (ads of current design)

=2: in the second treatment group (ads of alternative design)

restaurant type =chain: chain restaurants

=independent: independent restaurants

pageviews # of visits to the restaurant's RG page per month

calls # of phone calls made from the restaurant's mobile RG page per month

reservations # of reservations made from the restaurant's RG page per month

Questions

1.      Given that there is already a set of Restaurants on the platform that have bought Ads, do you need an experiment? Or can you simply compare the outcomes of Restaurants that advertise and those that do not advertise?

2.      In the exercise, which outcome is most useful to consider? Are there other outcome variables that you think would be useful to measure?

3.      Should Restaurant Grades stick with their current design or switch to the alternative? How confident you are with your decision, and what are possible concerns you may have?