Week 4 - Discussion 1 Quantitative Research Designs A research design is like a blueprint for conducting research. It guides the researcher in determining when and how often the data will be collecte

Malec, T. & Newman, M. (2013). Research methods: Building a knowledge base. San Diego, CA: Bridgepoint Education, Inc. ISBN-13: 9781621785743, ISBN-10: 1621785742.
Chapter 5: Experimental Designs – Determining Cause-and-Effect Relationships

hapter 5

Experimental Designs—Determining Cause-and-EffectRelationships

Week 4 - Discussion 1 Quantitative Research Designs A research design is like a blueprint for conducting research. It guides the researcher in determining when and how often the data will be collecte 1

Cosmo Condina/Stone/Getty Images

Chapter Contents

Experiment Terminology
Key Features of Experiments
Experimental Validity
Experimental Designs
Analyzing Experiments
Wrap-Up: Avoiding Error
Critiquing a Quantitative Study
Mixed Methods Research Designs

One of the oldest debates within psychology concerns the relative contributions that biology and the environment make in shaping ourthoughts, feelings, and behaviors. Do we become who we are because it is hard-wired into our DNA or in response to early experiences? Dopeople take on their parents’ personality quirks because they carry their parents’ genes or because they grew up in their parents’ homes? Thereare, in fact, several ways to address these types of questions. In fact, a consortium of researchers at the University of Minnesota has spent thepast 2 decades comparing pairs of identical and fraternal twins to tease apart the contributions of genes and environment. You can read moreat the research group’s website, Minnesota Center for Twin and Family Research, http://mctfr.psych.umn.edu/.

Creatas Images/Thinkstock

Researchers at the University ofMinnesota work with twins in order tostudy the impact of genetics versusupbringing on personality traits.

An alternative to using twin pairs to separate genetic and environmental influence is through the use of experimental designs, which have the primary goal of explaining the causes of behavior. Recall fromChapter 2 (Section 2.1, Overview of Research Designs) that experiments can speak to cause and effectbecause the experimenter has control over the environment and is able to manipulate variables. Oneparticularly ingenious example comes from the laboratory of Michael Meaney, a professor of psychiatryand neurology at McGill University, using female rats as experimental subjects (Francis, Dioro, Liu, &Meaney, 1999). Meaney’s research revealed that the parenting ability of female rats can be reliablyclassified based on how attentive they are to their rat pups, as well as how much time they spendgrooming the pups. The question tackled in this study was whether these behaviors were learned fromthe rats’ own mothers or transmitted genetically. To answer this question experimentally, Meaney andcolleagues had to think very carefully about the comparisons they wanted to make. It would have beeninsufficient to simply compare the offspring of good and bad mothers—this approach could notdistinguish between genetic and environmental pathways.

Instead, Meaney decided to use a technique called cross-fostering, or switching rat pups from one mother to another as soon as they wereborn. This resulted in four combinations of rats: (1) those born to inattentive mothers but raised by attentive ones, (2) those born to attentivemothers but raised by inattentive ones, (3) those born and raised by attentive mothers, and (4) those born and raised by inattentive mothers.Meaney then tested the rat pups several months later and observed the way they behaved with their own offspring. The setup of thisexperiment allowed Meaney to make clear comparisons between the influence of birth mothers and the rearing process. At the end of thestudy, the conclusion was crystal clear: Maternal behavior is all about the environment. Those rat pups that ultimately grew up to be inattentivemothers were those who had been raised by inattentive mothers.

This final chapter is dedicated to experimental designs, in which the primary goal is to explain behavior. Experimental designs rank highest onthe continuum of control (see Figure 5.1) because the experimenter can manipulate variables, minimize extraneous variables, and assignparticipants to conditions. The chapter begins with an overview of the key features of experiments and then covers the importance of bothinternal and external validity of experiments. From there, the discussion moves to the process of designing and analyzing experiments and asummary of strategies for minimizing error in experiments. It concludes with guidelines for critiquing a quantitative study.

Figure 5.1: Experimental designs on the continuum of control

5.1 Experiment Terminology

Flying Colours Ltd/Photodisc/Thinkstock

A variable is any factor that has morethan one value, such as height.

Before we dive into the details, it is important to cover the terminology that we will use to describedifferent aspects of experimental designs. Much of this will be familiar from previous chapters, with afew new additions. First, let’s review the basics.

Recall that a variable is any factor that has more than one value. For example, height is a variablebecause people can be short, tall, or anywhere in between. Depression is a variable because people canexperience a wide range of symptoms, from mild to severe. The independent variable (IV) is the variablethat is manipulated by the experimenter in order to test hypotheses about cause. The dependentvariable (DV) is the variable that is measured by the experimenter in order to assess the effects of theindependent variable. For example, in an experiment testing the hypothesis that fear causes prejudice,fear would be the independent variable and prejudice would be the dependent variable. To keep theseterms straight, it is helpful to think of the main goal of experimental designs. That is, we test hypothesesabout cause by manipulating an independent variable and then looking for changes in a dependentvariable. Thus, our independent variable causes changes in the dependent variable; for example, fear is hypothesized to cause changes inprejudice.

Any manipulation of independent variables results in two or more versions of the variable. One common way to describe the versions of theindependent variable is in terms of different groups, or conditions. The most basic experiments have two conditions: The experimentalcondition receives treatment designed to test the hypothesis, while the control condition does not receive this treatment. In our fear andprejudice example, the participants who make up the experimental condition would be made to feel afraid, while the participants who make upthe control condition would not. This setup allows us to test whether introducing fear to one group of participants leads them to express moreprejudice than the other group of participants, who are not made fearful.

Another common way to describe these versions is in terms of levels of the independent variable. Levels describe the specific set ofcircumstances created by manipulating a variable. For example, in the fear and prejudice experiment, the variable of fear would have two levels—afraid and not afraid. There are countless ways to introduce fear into the experiment. One option would be to adopt the technique used bythe Stanford social psychologist Stanley Schachter (1959), who led participants to believe they would be exposed to a series of painful electricshocks. In Schachter’s study, the painful shocks never happened, but they did induce a fearful state as people anticipated them. So those at the“afraid” level of the independent variable might be told to expect these shocks, while those at the “not afraid” level of the independent variablewould not be given this expectation.

At this stage, it may seem odd to have two sets of vocabulary terms—“levels” and “conditions”—for the same concept. However, there is asubtle difference in how these terms are used once we get into advanced experimental designs. As the designs become more complex, it isoften necessary to expand independent variables to include several groups and multiple variables. Once this happens, we will need differentterminology to distinguish between the versions of one variable and the combinations of multiple variables. We will return to this complexitylater in the chapter, in Section 5.4, Experimental Designs.

5.2 Key Features of Experiments

The overview of research designs (Chapter 2, Section 2.1) described the overall process of experiments in the following way: A researchercontrols the environment as much as possible so that all participants have the same experience. He or she then manipulates, or changes, onekey variable, and then measures the outcomes in another key variable. In this section, we will examine this process of control in more detail.Experiments can be distinguished from all other designs by three key features: manipulating variables, controlling the environment, andassigning people to groups comprising experimental and control conditions.

Manipulating Variables

The most crucial element of an experiment is that the researcher must manipulate, or change, some key variable. To study the effects of hunger,for example, a researcher could manipulate the amount of food given to the participants. Or, to study the effects of temperature, theexperimenter could raise and lower the temperature of the thermostat in the laboratory. Because these factors are under the researchers’ directcontrol, they can feel more confident that changing them contributes to changes in the dependent variables.

In Chapter 2 we discussed the main shortcoming of correlational research: These designs do not allow us to make causal statements. As you’llrecall from that chapter (as well as from Chapter 4), correlational research is designed to predict one variable from another.

One of the examples in Chapter 2 concerned the correlation between income levels and happiness, with the goal of trying to predict happinesslevels based on knowing people’s income level. If we measure these as they occur in the real world, we cannot say for sure which variablecauses the other. However, we could settle this question relatively quickly with the right experiment. Let’s say we bring two groups into thelaboratory, give one group $100 and a second group nothing. If the first group were happier at the end of the study, this would support theidea that money really does buy happiness. Of course, this is a rather simplistic look at the connection between money and happiness, butbecause it manipulated levels of money, this study would bring us closer to making causal statements about the effects of money.

To manipulate variables, it is necessary to have at least two versions of the variable. That is, to study the effects of money, we need acomparison group that does not receive money. To study the effects of hunger, we would need both a hungry and a not-hungry group. Havingtwo versions of the variable distinguishes experimental designs from the structured observations discussed in Chapter 3 (ObservationalResearch), in which all participants received the same set of conditions in the laboratory. Even the most basic experiment must have two sets ofconditions, which are often an experimental group and a control group. But, as we will see later in this chapter, experiments can become muchmore complex. You might have one experimental group and two control groups, or five degrees of food deprivation, ranging from 0 to 12 hourswithout food. Your decisions about the number and nature of these groups will depend on consideration of both your hypotheses and previousliterature.

When it comes to the manipulation of variables, there are three options available. First, environmental manipulations involve changing someaspect of the setting. Environmental manipulations are perhaps the most common in psychology studies, and they include everything fromvarying the temperature to varying the amount of money people receive. The key is to change the way different groups of people experiencetheir time in the laboratory—it is either hot or cold, and they either receive or don’t receive $100. Second, instructional manipulations involvechanging the way a task is described in order to change participants’ mind-sets. For example, you could give all participants the same math testbut describe it as an intelligence test for one group and a problem-solving task for another. Because an intelligence test is thought to haveimplications for life success, you might expect participants in this group to be more nervous about their scores. Finally, an invasive manipulationinvolves taking measures to change internal, physiological processes and is usually conducted in medical settings. For example, studies of newdrugs involve administering the drug to volunteers to determine whether it has an effect on some physical or psychological symptom. Or, forexample, studies of cardiovascular health often involve having participants run on a treadmill to measure how the heart functions under stress.

iStockphoto/Thinkstock

Medical studies often use invasivemanipulation to change internal,physiological processes.

Finally, there is one qualification to the rule that we must manipulate a variable. In many experiments,researchers divide up participants based on an inherent difference (e.g., gender) or personality measures(e.g., self-esteem or neuroticism) that capture stable individual characteristics among people. The ideabehind these personality measures is that someone scoring high on a measure of neuroticism (forexample) would be expected to be more neurotic across situations than someone scoring lower on themeasure. Using this technique allows us to compare how, for example, men and women, or people withhigh and low self-esteem, respond to manipulations. When existent differences are used in anexperimental context, they are referred to as quasi-independent variables—“quasi,” or “nearly,” becausethey are being measured, not manipulated, by the experimenter, and thus do not meet the criteria for aregular independent variable. Because these variables are not manipulated, an experimenter cannotmake causal statements about them. In order for a study to count as an experiment, these quasi-independent variables would have to be combined with a true independent variable. This could be assimple as comparing how men and women respond to a new antidepressant drug—gender would bequasi-independent while drug type would be a true independent variable.

Controlling the Environment

The second important element of experimental designs is that the researcher has a high degree ofcontrol over the environment. In addition to manipulating variables, a researcher conducting anexperiment ensures that the other aspects of the environment are the same for all participants. Forinstance, if you were interested in the effects of temperature on people’s mood, you could manipulatetemperature levels in the laboratory so that some people experienced warmer temperatures and otherpeople cooler temperatures. But it would be equally important to make sure that other potential influences on mood were the same for bothgroups. That is, you would want to make sure that the “warm” and “cool” groups were tested in the same room, around the same time of day,and by the same experimenter.

The overall goal, then, is to control extraneous variables, or variables that add noise to your hypothesis test. In essence, the more you are ableto control extraneous variables, the more confidence you can have in the results of your hypothesis test. As we will discuss in the section“Experimental Validity,” the impact of extraneous variables can vary in a study. Let’s say we conduct the study on temperature and mood, andall of our participants are in a windowless room with a flickering fluorescent light. This would likely have an influence on mood—makingeveryone a little bit grumpy—but cause few problems for our hypothesis test because it would affect everyone equally. Table 5.1 showshypothetical data from two variations of this study, using a 10-point scale to measure mood ratings. In the top row, participants were in a well-lit room; we can see that participants in the cooler room reported being in a better mood (i.e., an 8 versus a 5). In the bottom row, allparticipants were in the windowless room with flickering lights. These numbers suggest that people were still in a better mood in the coolerroom (5) than a warm room (2), but the flickering fluorescent light had a constant dampening effect on everyone’s mood.

Table 5.1: Influence of an extraneous variable
	Cool Room	Warm Room
Variation 1: Well-Lit
Variation 2: Flickering Fluorescent

Assigning People to Conditions

The third key feature of experimental designs is that the researcher can assign people to receive different conditions, or versions, of theindependent variable. This is an important piece of the experimental process: The experimenter not only controls the options—warm versuscool room; $100 versus no money, etc.—he or she also gets to control which participants get each option. Whereas a correlational design mightassess the relationship between current mood and choosing the warm room, an experimental design will have some participants assigned to thewarm room and then measure the effects on their mood. In other words, an experimenter is able to make causal statements because thatperson causes things to happen.

The most common, and most preferable, way to assign people to conditions is through a process called random assignment. An experimenterwho uses random assignment makes a separate decision for each participant as to which group he or she will be assigned to before theparticipant arrives. As the term implies, this decision is made randomly—by flipping a coin, using a random number table (for an example, click here), drawing numbers out of an envelope, or some other random process. The overall goal is to try to balance out existent differences amongpeople, as illustrated in Figure 5.2. So, for example, some people might generally be more comfortable in warm rooms, while others might bemore comfortable in cold rooms. If each person who shows up for the study has an equal chance of being in either group, then the groups inthe sample should reflect the same distribution of differences as the population.

Figure 5.2: Random assignment

The 25 participants in our sample consist of a mix of happy and sad people. The goal of random assignment is to havethese differences distributed equally across the experimental conditions. Thus, the two groups on the right each consistof six happy and six sad people, and our random assignment was successful.

Another significant advantage of forming groups through random assignment is that it helps to avoid bias in the selection and assignment ofsubjects. For example, it would be a bad idea to assign people to groups based on a first impression of them because participants might beplaced in the cold room if they arrived at the laboratory dressed in warm clothing. Experimenters who make decisions about conditionassignments ahead of time can be more confident that the independent variable is responsible for changes in the dependent variable.

It is worth highlighting the difference here between random selection and random assignment (discussed in Chapter 4, Section 4.3, SamplingFrom the Population). Random selection means that the sample of participants is chosen at random from the population, as with the probabilitysampling methods discussed in Chapter 4. However, most psychology experiments use a convenience sample of individuals who volunteer tocomplete the study. This means that the sample is often far from fully random. However, a researcher can still make sure that the groupassignments are random so that each condition contains an equal representation of the sample.

In some cases—most notably, when samples are small—random assignment may not be sufficient to balance an important characteristic thatmight affect the results of a particular study. Imagine conducting a study that compared two strategies for teaching students complex mathskills. In this example, it would be especially important to make sure that both groups contained a mix of individuals with, say, average andabove-average intelligence. For this reason, it would be necessary to take extra steps to ensure that intelligence was equally distributed amongthe groups, which can be accomplished with a variation on random assignment called matched random assignment. This requires theexperimenter to obtain scores on an important matching variable (in this case, intelligence), rank participants based on the matching variable,and then randomly assign people to conditions. Figure 5.3 shows how this process would unfold in our math skills study. First, participants aregiven an IQ test to measure existing differences in intelligence. Second, the experimenter ranks participants based on these scores, from highestto lowest. Third, the experimenter would move down this list in order and randomly assign each participant to one of the conditions. Thisprocess still contains an element of random assignment, but adding the extra step of rank ordering ensures a more balanced distribution ofintelligence test scores across the conditions.

Figure 5.3

The 20 participants in our sample represent a mix of very high, average, and very low intelligence test scores (measured1–100). The goal of matched random assignment is to ensure that this variation is distributed equally across the twoconditions. The experimenter would first rank participants by intelligence test scores (top box) and then distribute theseparticipants alternately between the conditions. The result is that both groups (lower boxes) contain a good mix of high,average, and low scores.

Research: Making an Impact The Stanford Prison Experiment

The landmark 1971 Stanford Prison Experiment had an extremely widespread impact on multiple fieldsand real-life settings such as prison reform, the ethics of human research, and terrorism and torturetactics (Clements, 1999; Haney & Zimbardo, 1998). The study, conducted by Phillip Zimbardo and hiscolleagues at Stanford University, placed volunteer participants in a simulated prison environment andrandomly assigned them to play the roles of “guards” and “prisoners.” These 24 participants had beenselected based on their personality traits that marked them as “good apples” and who had no previousevidence of antisocial behavior (Haney, Banks, & Zimbardo, 1973). Nevertheless, the simulated prisonquickly took on the characteristics of a real prison, with simulated situations of dominance,dehumanization, severe psychological distress, and the unexpected phenomena of social control,obedience, and effects of power (Haney, 2002; Banuazizi & Movahedi, 1975). The prisoners weresubjected to humiliation similar to what has been seen in prison scandals such as Abu Ghraib:nakedness, sexual humiliation, verbal torment, chains, and bags over prisoners’ heads. Although plannedto run for 2 weeks, the experiment was stopped after only 6 days due to the severe psychological harmit was causing the prisoners and the unexpected behavior of the prison guards.

The study led to major reform in the ethical guidelines for psychological research and humanetreatment, and in fact has never been replicated in a scientific setting. Following publication of thisexperiment, the Supreme Court saw an influx of cases regarding prisoner treatment and the structure ofthe prison system. In an interesting twist, one of the original researchers, Craig Haney, was inspired topursue a career in prison reform based on what he learned from this study. Zimbardo himself went onto testify on behalf of the soldiers accused of abuse at Abu Ghraib, highlighting the powerful influenceof social roles (e.g., prison guard) on our behavior. An overwhelming majority of social and psychologicalresearch has found the punitive punishment system to not only be ineffective but actually deleterious toprisoner behavior and recidivism. The suggestions from current research include calls to restructureprison “power” dynamics in an effort to increase prisoner safety and reduce guard brutality.

5.3 Experimental Validity

Chapter 2 (Section 2.2) discussed the concept of validity, or the degree to which measures capture the constructs that they were designed tocapture. For example, a measure of happiness needs to actually capture differences in people’s levels of happiness. In this section, we return tothe subject of validity in an experimental context. Similar to our earlier discussion, validity refers here to whether the experimental results aredemonstrating what we think they are demonstrating. We will cover two types of validity that are relevant to experimental designs. The first is internal validity, which assesses the degree to which results can be attributed to independent variables. The second is external validity, whichassesses how well the results generalize to situations beyond the specific conditions laid out in the experiment. Taken together, internal andexternal validity provide a way to assess the merits of an experiment. However, each of these has its own threats and remedies, as discussed inthe following sections.

Internal Validity

In order to have a high degree of internal validity, experimenters strive for maximum control over extraneous variables. That is, they try todesign experiments so that the independent variable is the only cause of differences between groups. But, of course, no study is ever perfect,and there will always be some degree of error. In many cases, errors are the result of unavoidable causes, such as the health or mood of theparticipants on the day of the experiment. In other cases, errors are caused by factors that are, in fact, under the experimenter’s control. In thissection, we will focus on several of these more manageable threats to internal validity and discuss strategies for reducing their influence.

Experimental Confounds

To avoid threats to the internal validity of an experiment, it is important to control and minimize the influence of extraneous variables thatmight add noise to a hypothesis test. In many cases, extraneous variables can be considered relatively minor nuisances, as when our moodexperiment was accidentally run in a depressing room. But now, let’s say we ran our study on temperature and mood, and owing to a lack ofcareful planning, we accidentally placed all of the warm-room participants in a sunny room, and the cool-room participants in a windowlessroom. We might very well find that the warm-room participants were in a much better mood. But would this be the result of warmtemperatures or the result of exposure to sunshine? Unfortunately, we would be unable to tell the difference because of a confoundingvariable, or confound (in the case of correlation studies, third variable). The confounding variable changes systematically with the independentvariable. In this example, room lighting would be confounded with room temperature because all of the warm-room participants were alsoexposed to sunshine, and all of the cool-room participants to artificial lighting. This combination of variables would leave us unable todetermine which variable actually had the effect on mood. The result would be that our groups differed in more than one way, which wouldseriously hinder our ability to say that the independent variable (the room) caused the dependent variable (mood) to change.

Digital Vision/Thinkstock

The demeanor of the person running astudy may be a confounding variable.

It may sound like an oversimplification, but the way to avoid confounds is to be very careful in designingexperiments. By ensuring that groups are alike in every way but the experimental condition, one cangenerally prevent confounds. This is somewhat easier said than done because confounds can come fromunexpected places. For example, most studies involve the use of multiple research assistants whomanage data collection and interact with participants. Some of these assistants might be more or lessfriendly than others, so it is important to make sure each of them interacts with participants in allconditions. If your friendliest assistant works with everyone in the warm-room group, for example, itwould result in a confounding variable (friendly versus unfriendly assistants) between room and researchassistant. Consequently, you would be unable to separate the influence of your independent variable(the room) from that of the confound (your research assistant).

Selection Bias

Internal validity can also be threatened when groups are different before the manipulation, which isknown as selection bias. Selection bias causes problems because these inherent differences might be the driving factor behind the results.Imagine you are testing a new program that will help people stop smoking. You might decide to ask for volunteers who are ready to quitsmoking and put them through a 6-week program. But by asking for volunteers—a remarkably common error—you gather a group of peoplewho are already somewhat motivated to stop smoking. Thus, it is difficult to separate the effects of your new program from the effects of this apriori motivation.

One easy way to avoid this problem is through either random or matched-random assignment. In the stop-smoking example, you could still askfor volunteers, but then randomly assign these volunteers to one of the two programs. Because both groups would consist of people motivatedto quit smoking, this would help to cancel out the effects of motivation. Another way to minimize selection bias is to use the same people inboth conditions so that they serve as their own control. In the stop-smoking example, you could assign volunteers first to one program and thento the other. However, you might run into a problem with this approach—participants who successfully quit smoking in the first program wouldnot benefit from the second program. This technique is known as a within-subject design, and we will discuss its advantages and disadvantagesin the subsection “Within-Subject Designs” in Section 5.4, Experimental Designs.

Differential Attrition

Despite your best efforts at random assignment, you could still have a biased sample at the end of a study as a result of differential attrition.The problem of differential attrition (sometimes called the mortality threat) occurs when subjects drop out of experimental groups for differentreasons. Let’s say you’re conducting a study of the effects of exercise on depression levels. You manage to randomly assign people to either 1week of regular exercise or 1 week of regular therapy. At first glance, it appears that the exercise group shows a dramatic drop in depressionsymptoms. But then you notice that approximately one third of the people in this group dropped out before completing the study. Chances areyou are left with those who are most motivated to exercise, to overcome their depression, or both. Thus, you are unable to isolate the effectsof your independent variable on depression symptoms. While you cannot prevent people from dropping out of your study, you can lookcarefully at those who do. In many cases, you can spot a pattern and use it to guide future research. For example, it may be possible todiscover a profile of people who dropped out of the exercise study and use this knowledge to increase retention for the next attempt.

Outside Events

As much as we strive to control the laboratory environment, participants are often influenced by events in the outside world. These events—sometimes called history effects—are often large-scale events such as political upheavals and natural disasters. The threat to research is that itbecomes difficult to tell whether participants’ responses are the result of the independent variable or the historical event(s). One great exampleof this comes from a paper published by social psychologist Ryan Brown, now a professor at the University of Oklahoma, on the effects ofreceiving different types of affirmative action as people were selected for a leadership position. The goal was to determine the best way toframe affirmative action in order to avoid undermining the recipient’s confidence (Brown, Charnsangavej, Keough, Newman, & Rentfrow, 2000).For about a week during the data collection process, students at the University of Texas, where the study was being conducted, were protestingon the main lawn about a controversial lawsuit regarding affirmative action policies. The result was that participants arriving for this laboratorystudy had to pass through a swarm of people holding signs that either denounced or supported affirmative action. These types of outsideevents are difficult, if not impossible, to control. But, because these researchers were aware of the protests, they made a decision to excludefrom the study data gathered from participants during the week of the protests, thus minimizing the effects of these outside events.

Expectancy Effects

One final set of threats to internal validity results from the influence of expectancies on people’s behavior. This can cause trouble forexperimental designs in three related ways. First, experimenter expectancies can cause researchers to see what they expect to see, leading tosubtle bias in favor of their hypotheses. In a clever demonstration of this phenomenon, the psychologist Robert Rosenthal asked his graduatestudents at Harvard University to train groups of rats to run a maze (Rosenthal & Fode, 1963). He also told them that based on a pretest, therats had been classified as either bright or dull. As you might have guessed, these labels were pure fiction, but they still influenced the way thatthe students treated the rats. Rats labeled “bright” were given more encouragement and learned the maze much more quickly than rats labeled“dull.” Rosenthal later extended this line of work to teachers’ expectations of their students (Rosenthal & Jacobson, 1968) and found support forthe same conclusion: People often bring about the results they expect by behaving in a particular way.

One common way to avoid experimenter expectancies is to have participants interact with a researcher who is “blind” (i.e., unaware) to thecondition that each participant is in. The researcher may be fully aware of the research hypothesis, but his or her behavior is unlikely to affectthe results. In the Rosenthal and Fode (1963) study, the graduate students’ behavior influenced the rats’ learning speed only because they wereaware of the labels “bright” and “dull.” If these had not been assigned, the rats would have been treated fairly equally across the conditions.

Second, participants in a research study often behave differently based on their own expectancies about the goals of the study. Theseexpectancies often develop in response to demand characteristics, or cues in the study that lead participants to guess the hypothesis. In a well-known study conducted at the University of Wisconsin, psychologists Leonard Berkowitz and Anthony LePage found that participants wouldbehave more aggressively—by delivering electric shocks to another participant—if a gun was in the room than if there were no gun present(Berkowitz & LePage, 1967). This finding has some clear implications for gun control policies, suggesting that the mere presence of gunsincreases the likelihood of violence. However, a common critique of this study is that participants may have quickly clued in to its purpose andfigured out how they were “supposed” to behave. That is, the gun served as a demand characteristic, possibly making participants act moreaggressively because they thought it was expected of them.

To minimize demand characteristics, researchers use a variety of techniques, all of which attempt to hide the true purpose of the study fromparticipants. One common strategy is to use a cover story, or a misleading statement about what is being studied. In Chapter 1 (Section 1.4,Hypotheses and Theories, and Section 1.7, Ethics in Research), we discussed Milgram’s famous obedience studies, which discovered that peoplewere willing to obey orders to deliver dangerous levels of electric shocks to other people. In order to disguise the purpose of the study, Milgramdescribed it to people as a study of punishment and learning. And the affirmative action study by Ryan Brown and colleagues (Brown et al.,2000) was presented as a study of leadership styles. The goal in using these cover stories is to give participants a compelling explanation forwhat they experience during the study and to direct their attention away from the research hypothesis.

Another strategy is to use the unrelated-experiments technique, which leads participants to believe that they are completing two differentexperiments during one laboratory session. The experimenter can use this bit of deception to present the independent variable during the firstexperiment and then measure the dependent variable during the second experiment. For example, a study by Harvard psychologist MargaretShih and colleagues (Shih, Pittinsky, & Ambady, 1999) recruited Asian American females and asked them to complete two supposedly unrelatedstudies. In the first, they were asked to read and form impressions of one of two magazine articles; these articles were designed to make themfocus on either their Asian American identity or their female identity. In the second experiment, they were asked to complete a math test asquickly as possible. The goal of this study was to examine the effects on math performance of priming different aspects of identity. Based onprevious research, these authors predicted that priming an Asian American identity would remind participants of positive stereotypes regardingAsians and math performance, whereas priming a female identity would remind participants of negative stereotypes regarding women and mathperformance. As expected, priming an Asian American identity led this group of participants to do better on a math test than did priming afemale identity. The unrelated-experiments technique was especially useful for this study because it kept participants from connecting theindependent variable (magazine article prime) with the dependent variable (math test).

A final way in which expectancies shape behavior is the placebo effect, meaning that change can result from the mere expectation that changewill occur. Imagine you wanted to test the hypothesis that alcohol causes people to become aggressive. One relatively easy way to do this wouldbe to give alcohol to a group of volunteers (aged 21 and older) and then measure how aggressive they became in response to being provoked.The problem with this approach is that people also expect alcohol to change their behavior, so you might see changes in aggression simplybecause of these expectations. Fortunately, there is an easy solution: Add a placebo control group to your study that mimics the experimentalcondition in every way but one. In this case, you might tell all participants that they will be drinking a mix of vodka and orange juice but onlyadd vodka to half of the participants’ drinks. The orange-juice-only group serves as the placebo control, so any differences between this groupand the alcohol group can be attributed to the alcohol itself.

External Validity

In order to attain a high degree of external validity in their experiments, researchers strive for maximum realism in the laboratory environment. External validity means that the results extend beyond the particular set of circumstances created in a single study. Recall that science is acumulative discipline and that knowledge grows one study at a time. Thus, each study is more meaningful to the extent that it sheds light on areal phenomenon and to the extent that the results generalize to other studies. Let’s examine each of these criteria separately.

Mundane Realism

The first component of external validity is the extent to which an experiment captures the real-world phenomenon under study. One popularquestion in the area of aggression research is whether rejection by a peer group leads to aggression. That is, when people are rejected from agroup, do they lash out and behave aggressively toward the members of that group? Researchers must find realistic ways to manipulaterejection and measure aggression without infringing on participants’ welfare. Given the need to strike this balance, how real can things get inthe laboratory? How do we study real-world phenomena without sacrificing internal validity?

The answer is to strive for mundane realism, meaning that the research replicates the psychological conditions of the real-world phenomenon(sometimes referred to as ecological validity). In other words, we need not re-create the phenomenon down to the last detail; instead, we aimto make the laboratory setting feel like the real world. Researchers studying aggressive behavior and rejection have developed some ratherclever ways of doing this, including allowing participants to administer loud noise blasts or serve large quantities of hot sauce to those whorejected them. Psychologically, these acts feel like aggressive revenge because participants are able to lash out against those who rejected them,with the intent of causing harm, even though the behaviors themselves may differ from the ways people exact revenge in the real world.

In a 1996 study, Tara MacDonald and her colleagues at Queen’s University in Ontario, Canada, examined the relationship between alcohol andcondom use (MacDonald, Zanna, & Fong, 1996). The authors pointed out a puzzling set of real-world data: Most people reported that theywould use condoms when engaging in casual sex, but the rates of unprotected sex (i.e., having sexual intercourse without a condom) were alsoremarkably high. In this study, the authors found that alcohol was a key factor in causing “common sense to go out the window” (p. 763),resulting in a decreased likelihood of condom use. But how on earth might they study this phenomenon in the laboratory? In the authors’words, “even the most ambitious of scientists would have to conclude that it is impossible to observe the effects of intoxication on actualcondom use in a controlled laboratory setting” (p. 765).

To solve this dilemma, MacDonald and colleagues developed a clever technique for studying people’s intentions to use condoms. Participantswere randomly assigned to either an alcohol or placebo condition, and then they viewed a video depicting a young couple that was faced withthe dilemma of whether to have unprotected sex. At the key decision point in the video, the tape was stopped and participants were askedwhat they would do in the situation. As predicted, participants who were randomly assigned to consume alcohol said they would be morewilling to proceed with unprotected sex. While this laboratory study does not capture the full experience of making decisions about casual sex,it does a nice job of capturing the psychological conditions involved.

Generalizing Results

The second component of external validity is the extent to which research findings generalize to other studies. Generalizability refers to theextent to which the results extend to other studies, using a wide variety of populations and a wide variety of operational definitions (sometimesreferred to as population validity). If we conclude that rejection causes people to become more aggressive, for example, this conclusion shouldideally carry over to other studies of the same phenomenon, using different ways of manipulating rejection and different ways of measuringaggression. If we want to conclude that alcohol reduces intentions to use condoms, we would need to test this relationship in a variety ofsettings—from laboratories to nightclubs—using different measures of intentions.

Thus, each study that we conduct is limited in its conclusions. In order for your particular idea to take hold in the scientific literature, it must be replicated, or repeated in different contexts. These replications can take one of four forms. First, exact replication involves trying to re-createthe original experiment as closely as possible in order to verify the findings. This type of replication is often the first step following a surprisingresult, and it helps researchers to gain more confidence in the patterns. The second and much more common method, conceptual replication,involves testing the relationship between conceptual variables using new operational definitions. Conceptual replications would include testingour aggression hypotheses using new measures or examining the link between alcohol and condom use in different settings. For example,rejection might be operationalized in one study by having participants be chosen last for a group project. A conceptual replication might take adifferent approach: operationalizing rejection by having participants be ignored during a group conversation or voted out of the group. Likewise,a conceptual replication might change the operationalization of aggression by having one study measure the delivery of loud blasts of noise andanother measure the amount of hot sauce that people give to their rejecters. Each variation studies the same concept (aggression or rejection)but uses slightly different operationalizations. If all of these variations yielded similar results, this would provide further evidence of theunderlying ideas—in this case, that rejection causes people to be more aggressive.

The third method, participant replication, involves repeating the study with a new population of participants. These types of replication areusually driven by a compelling theory as to why the two populations differ. For example, you might reasonably hypothesize that the decision touse condoms is guided by a different set of considerations among college students than among older, single adults. Finally, constructivereplication re- creates the original experiment but adds elements to the design. These additions are typically designed to either rule outalternative explanations or extend knowledge about the variables under study. In our rejection and aggression example, you might test whethermales and females respond the same way or perhaps compare the impact of being rejected by a group versus an individual.

Internal Versus External Validity

We have focused on two ways to assess validity in the context of experimental designs. Internal validity assesses the degree to which results canbe attributed to independent variables; external validity assesses how well results generalize beyond the specific conditions of the experiment.In an ideal world, studies would have a high degree of both of these. That is, we would feel completely confident that our independent variablewas the only cause of differences in our dependent variable, and our experimental paradigm would perfectly capture the real-worldphenomenon under study.

In reality, though, there is often a trade-off between internal and external validity. In MacDonald and colleagues’ study on condom use(MacDonald et al., 1996), the researchers sacrificed some realism in order to conduct a tightly controlled study of participants’ intentions. InBerkowitz and LePage’s (1967) study on the effect of weapons, the researchers risked the presence of a demand characteristic in order to studyreactions to actual weapons. These types of trade-offs are always made based on the goals of the experiment. To give you a better sense ofhow researchers make these compromises, let’s evaluate three fictional examples.

Scenario 1: Time Pressure and Stereotyping

Dr. Bob is interested in whether people are more likely to rely on stereotypes when they are in a hurry. In a well-controlled laboratoryexperiment, participants are asked to categorize ambiguous shapes as either squares or circles, and half of these participants are given a shorttime limit to accomplish the task. The independent variable is the presence or absence of time pressure, and the dependent variable is theextent to which people use stereotypes in their classification of ambiguous shapes. Dr. Bob hypothesizes that people will be more likely to usestereotypes when they are in a hurry because they will have fewer cognitive resources to carefully consider all aspects of the situation. Dr. Bobtakes great care to have all participants meet in the same room. He uses the same research assistant every time, and the study is alwaysconducted in the morning. Consistent with his hypothesis, Dr. Bob finds that people seem to use shape stereotypes more under time pressure.

The internal validity of this study appears high—Dr. Bob has controlled for other influences on participants’ attention span by collecting all of hisdata in the morning. He has also minimized error variance by using the same room and the same research assistant. In addition, Dr. Bob hascreated a tightly controlled study of stereotyping through the use of circles and squares. Had he used photographs of people (rather thanshapes), the attractiveness of these people might have influenced participants’ judgments. But here’s the trade-off: By studying the socialphenomenon of stereotyping using geometric shapes, Bob has removed the social element of the study, thereby posing a threat to mundanerealism. The psychological meaning of stereotyping shapes is rather different from the meaning of stereotyping people, which makes this studyrelatively low in external validity.

Scenario 2: Hunger and Mood

Dr. Jen is interested in the effects of hunger on mood; not surprisingly, she predicts that people will be happier when they are well fed. Shetests this hypothesis with a lengthy laboratory experiment, requiring participants to be confined to a laboratory room for 12 hours with very fewdistractions. Participants have access to a small pile of magazines to help pass the time. Half of the participants are allowed to eat during thistime, and the other half is deprived of food for the full 12 hours. Dr. Jen—a naturally friendly person—collects data from the food-deprivationgroups on a Saturday afternoon, while her grumpy research assistant, Mike, collects data from the well-fed group on a Monday morning. Herindependent variable is food deprivation, with participants either not deprived of food or deprived for 12 hours. Her dependent variable consistsof participants’ self-reported mood ratings. When Dr. Jen analyzes the data, she is shocked to discover that participants in the food-deprivationgroup were much happier than those in the well-fed group.

Compared with our first scenario, this study seems high on external validity. To test her predictions about food deprivation, Dr. Jen actuallydeprives her participants of food. One possible problem with external validity is that participants are confined to a laboratory setting during thedeprivation period with only a small pile of magazines to read. That is, participants may be more affected by hunger when they do not haveother things to distract them. In the real world, people are often hungry but distracted by paying attention to work, family, or leisure activities.But Dr. Jen has sacrificed some external validity for the sake of controlling how participants spend their time during the deprivation period. Thelarger problem with her study has to do with internal validity. Dr. Jen has accidentally confounded two additional variables with her independentvariable: Participants in the deprivation group have a different experimenter and data are collected at a different time of day. Thus, Dr. Jen’ssurprising results most likely reflect the fact that everyone is in a better mood on Saturday than on Monday and that Dr. Jen is more pleasant tospend 12 hours with than Mike is.

Scenario 3: Math Tutoring and Graduation Rates

Dr. Liz is interested in whether specialized math tutoring can help increase graduation rates among female math majors. To test her hypothesis,she solicits female volunteers for a math skills workshop by placing flyers around campus, as well as by sending email announcements to allmath majors. The independent variable is whether participants are in the math skills workshop, and the dependent variable is whetherparticipants graduate with a math degree. Those who volunteer for the workshop are given weekly skills tutoring, along with informal discussiongroups designed to provide encouragement and increase motivation. At the end of the study, Dr. Liz is pleased to see that participants in theworkshops are twice as likely as nonparticipants to stick with the major and graduate.

The obvious strength of this study is its external validity. Dr. Liz has provided math tutoring to math majors, and she has observed a differencein graduation rates. Thus, this study is very much embedded in the real world. But, as you might expect, this external validity comes at a cost tointernal validity. The biggest flaw is that Dr. Liz has recruited volunteers for her workshops, resulting in selection bias for her sample. Peoplewho volunteer for extra math tutoring are likely to be more invested in completing their degree and might also have more time available todedicate to their education. Dr. Liz would also need to be mindful of how many people drop out of her study. If significant numbers ofparticipants withdrew, she could have a problem with differential attrition, so that the most motivated people stayed with the workshops. Onerelatively easy fix for this study would have been to ask for volunteers more generally, and then randomly assign these volunteers to take part ineither the math tutoring workshops or a different type of workshop. While the sample might still have been less than random, Dr. Liz would atleast have had the power to assign participants to different groups.

A Note on Qualitative Research Validity and Reliability

As discussed in Chapter 3, the validity of a quantitative study hinges on whether the experimental results demonstrate what we think they aredemonstrating; reliability refers to whether the experimental results will yield the same or similar results in other experiments. The concepts ofvalidity and reliability in qualitative research do not carry the same meanings as they do in quantitative research, nor is the goal to generalizethe results to individuals, sites, or places outside of those under study. As Creswell (2009) notes, qualitative validity “means the researcherchecks for the accuracy of the findings by employing certain procedures, while qualitative reliability indicates that the researcher’s approach isconsistent across different researchers and different projects” (p. 190). Because qualitative research does not include experimental results,numerical output, and data analyses, many qualitative researchers argue that they must evaluate the quality of their results differently and focusmore on the trustworthiness or overall worth of their data. Thus, they ask, are the findings worthy of attention? And, how do you evaluatethem?

To evaluate the trustworthiness or the validity and reliability of qualitative studies, Guba and Lincoln (1994) proposed the following alternativecriteria outlined in Table 5.2 that are utilized by many qualitative researchers.

Table 5.2: Criteria for evaluating quantitative research and qualitative research

Criteria for evaluating quantitative research

Alternative criteria for evaluating qualitative research

Internal validity

Assesses whether the independentvariable is the only possible explanationfor the dependent variable

Credibility

Used to assess “the accuracy of the identification and description of thesubject of the study” (Smith & Davis, 2010, p. 51)
Examines whether the research is credible or believable from the perspectiveof the participant

External validity

Evaluates whether the results can beapplied to different populations andsettings

Transferability

Focuses on the transferability of findings to other settings and groups
Transferability is enhanced by providing thorough and clear reports so thatthe results can be transferred to a different context

Objectivity

Empirical findings that can be confirmedby others and corrected throughsubsequent research

Confirmability

The extent to which the qualitative report “is accurate, unbiased, and can beconfirmed by others” (Smith & Davis, 2010, p. 51)
Confirmability is enhanced by having other researchers review drafts andpoint out inconsistencies, contradictions, and biases
Confirmability is also enhanced by providing thorough reports regarding theprocedures that were used to check and recheck the data

Reliability

Assesses whether the methods will yieldsimilar results in other studies

Dependability

“The extent to which the researcher believes the same results would beproduced if the study were replicated” (Smith & Davis, 2010, p. 51)
Emphasizes the need for the researcher to account for and describe thechanges that occur in the setting and how these changes affected the waythe researcher approached the study

There have been lengthy debates about the value of including alternative sets of criteria for judging qualitative research. Although the criteriafor evaluating validity and reliability in quantitative and qualitative research may seem similar and appear to be mere relabeling of concepts, itshould be noted that the procedures utilized to assess them are not. For example, to ensure validity in qualitative studies, researchers employone or more of the following strategies:

Triangulation: Using multiple sources of data collection to build justification for themes. If multiple sources of data confirm themes, thisprocess can be considered to add to the validity of the study.
Member checking: Asking participants to review the final report and confirm whether the descriptions or themes are accurate.
Providing rich, thick descriptions: Describing in detail the setting, participants, and procedures. This process can add to the validity of thefindings.
Clarifying researcher bias: Self-reflections on any bias the researcher brings to the study helps create an open and honest report.
Presenting negative or discrepant information: The researcher discussing any information that runs counter to the themes.
Peer debriefing: Utilizing peer debriefers that review and ask questions about the study.
Spending prolonged time in the field: Spending long periods of time in the field allows the researcher to develop in-depth understandings ofthe phenomenon of interest. The more experience a researcher has with participants in their natural setting, the more valid the findings willbe.
External auditors: Employing an independent reviewer who is not familiar with the research or project who can provide an objectiveassessment of the study.

To determine whether qualitative research approaches are consistent or reliable, researchers must first ensure that all steps of the proceduresare documented thoroughly. Gibbs (2007) suggests the following strategies:

Checking transcripts: The researcher checks written transcripts against tape-recorded information to ensure that mistakes were not madeduring transcription.
Ensuring codes are stable: Verifying that a shift in the meaning of codes did not occur during the process of coding. This is accomplished byconstantly comparing data with codes and providing detailed descriptions of the codes.
Coordinating communication: The researcher communicates the analyses to coders through regular documented meetings.
Cross-checking codes: The researcher cross-checks codes developed by other researchers and compares the results with his or her own.

5.3 Experimental Validity

Internal Validity

Experimental Confounds

Digital Vision/Thinkstock

The demeanor of the person running astudy may be a confounding variable.

Selection Bias

Differential Attrition

Outside Events

Expectancy Effects

External Validity

Mundane Realism

Generalizing Results

Internal Versus External Validity

Scenario 1: Time Pressure and Stereotyping

Scenario 2: Hunger and Mood

Scenario 3: Math Tutoring and Graduation Rates

A Note on Qualitative Research Validity and Reliability

Table 5.2: Criteria for evaluating quantitative research and qualitative research

Criteria for evaluating quantitative research

Alternative criteria for evaluating qualitative research

Internal validity

Assesses whether the independentvariable is the only possible explanationfor the dependent variable

Credibility

Used to assess “the accuracy of the identification and description of thesubject of the study” (Smith & Davis, 2010, p. 51)
Examines whether the research is credible or believable from the perspectiveof the participant

External validity

Evaluates whether the results can beapplied to different populations andsettings

Transferability

Focuses on the transferability of findings to other settings and groups
Transferability is enhanced by providing thorough and clear reports so thatthe results can be transferred to a different context

Objectivity

Empirical findings that can be confirmedby others and corrected throughsubsequent research

Confirmability

The extent to which the qualitative report “is accurate, unbiased, and can beconfirmed by others” (Smith & Davis, 2010, p. 51)
Confirmability is enhanced by having other researchers review drafts andpoint out inconsistencies, contradictions, and biases
Confirmability is also enhanced by providing thorough reports regarding theprocedures that were used to check and recheck the data

Reliability

Assesses whether the methods will yieldsimilar results in other studies

Dependability

“The extent to which the researcher believes the same results would beproduced if the study were replicated” (Smith & Davis, 2010, p. 51)
Emphasizes the need for the researcher to account for and describe thechanges that occur in the setting and how these changes affected the waythe researcher approached the study

Triangulation: Using multiple sources of data collection to build justification for themes. If multiple sources of data confirm themes, thisprocess can be considered to add to the validity of the study.
Member checking: Asking participants to review the final report and confirm whether the descriptions or themes are accurate.
Providing rich, thick descriptions: Describing in detail the setting, participants, and procedures. This process can add to the validity of thefindings.
Clarifying researcher bias: Self-reflections on any bias the researcher brings to the study helps create an open and honest report.
Presenting negative or discrepant information: The researcher discussing any information that runs counter to the themes.
Peer debriefing: Utilizing peer debriefers that review and ask questions about the study.
Spending prolonged time in the field: Spending long periods of time in the field allows the researcher to develop in-depth understandings ofthe phenomenon of interest. The more experience a researcher has with participants in their natural setting, the more valid the findings willbe.
External auditors: Employing an independent reviewer who is not familiar with the research or project who can provide an objectiveassessment of the study.

Checking transcripts: The researcher checks written transcripts against tape-recorded information to ensure that mistakes were not madeduring transcription.
Ensuring codes are stable: Verifying that a shift in the meaning of codes did not occur during the process of coding. This is accomplished byconstantly comparing data with codes and providing detailed descriptions of the codes.
Coordinating communication: The researcher communicates the analyses to coders through regular documented meetings.
Cross-checking codes: The researcher cross-checks codes developed by other researchers and compares the results with his or her own.

.4 Experimental Designs

There are three types of experimental designs that can be used in research studies: pre-experimental designs, quasi-experimental designs, andtrue experimental designs. Each design differs in the degree to which it controls for confounding or hidden variables; in turn, the degree ofcontrol affects the internal validity of the study. Pre-experimental designs have little to almost no control; instead, they involve studying a singlegroup or unbalanced groups that are not randomly assigned and then introducing an intervention or treatment during the study. Quasi-experimental designs offer some control and may or may not include a control group, but the participants are not randomly assigned to groups. True experimental designs give the researcher maximum control and involve randomly assigning participants and manipulating the independentvariable. As we will see later in this discussion, true experiments include randomized group selection and assignments, a control group, and alarge degree of control over confounding variables.

The best types of designs are those that offer control over confounding variables and include random assignment of the participants. Althoughtrue experiments provide a much stronger and more valid design, sometimes they cannot be used for a study for particular reasons. Thefollowing sections will discuss examples of the various types of designs. This is not an exhaustive list, and researchers can modify designs orcombine them in various ways.

Pre-Experimental Designs

Pre-experimental designs follow basic experimental procedures but cannot show cause-and-effect relationships. Some researchers argue thatpre-experimental designs confer so little control that they have minimal scientific value. As a result, these designs should be used only whenmaking tentative hypotheses and should be followed up with more controlled research (Leedy & Ormrod, 2010). As we will see in the followingthree sections, pre-experimental designs generally enroll a single group, although some designs include experimental and control groups that arenot randomly selected.

The three pre-experimental designs we describe next are very limited in terms of the conclusions that can be drawn from them. In comparison,quasi-experimental designs (described later in this section) confer more control over confounding variables.

One-Shot Case Study

One-shot case studies are the most basic type of experimental designs. Because they do not include a control or comparison group, it isimpossible to know whether the outcome results would have been better if the intervention had not been provided. This type of pre-experimental design involves using one group (Group A), introducing an intervention (X), and then administering a posttest observation (O) todetermine the effects of the intervention, as shown below:

Group A: X_____________O

Additionally, since this type of design does not use a pretest, it is impossible to determine whether changes within the group have actuallytaken place. As a result, one-shot case studies have low internal validity.

In addition, because one-shot case studies confer little to no control over confounding variables, variables such as maturation or otherenvironmental conditions could have influenced the participant’s performance. For instance, let’s say a researcher wants to measure whetherpraising first-grade students makes a difference in their reading scores. The researcher selects a group of students and increasingly praises thosestudents. At the end of the study, the researcher finds that reading scores have improved. However, the researcher cannot be sure that thescores improved because of the praise, as it is possible that the students selected were already good in reading or that their parents werehelping them to learn reading strategies at home. While this type of design is easy to carry out, its results are nearly meaningless.

One-Group Pretest–Posttest Design

The one-group pretest–posttest design involves one group but includes both a pretest (O1) before the intervention (X) is given and a posttest(O2) after the intervention is provided, as shown below:

Group A: O1_____________X_____________O2

This design gives the researcher a little more control over the study than a one-shot case study, as it includes a pretest to measure performancebefore the intervention is provided. Thus, after the posttest is administered, one can know that some sort of change has taken place. However,even though it includes a pretest and a posttest, the researcher still cannot be sure whether some sort of confounding variable contributed tothe results.

Suppose, for example, that a researcher wanted to know whether having a group of students take an 8-week course on GRE preparation wouldimprove their scores. The researcher would give the students a form of the GRE test, have all of them participate in the 8-week preparationcourse, and then give the students an alternate form of the GRE. If the scores improved after the 8-week course, the researcher might concludethat the course produced an increase in GRE scores. However, this conclusion may or may not be accurate, and here is why. Let’s say some ofthe students studied GRE material at home or participated in another GRE study group at the same time. How could we say that the resultswere the result of only the 8-week GRE preparation course? The answer is, we cannot.

Static Group Comparison or Posttest-Only With Nonequivalent Groups

Static group comparison (or posttest only, with nonequivalent groups’ designs) involves both an experimental group and a control group,although participants are not randomly assigned to either group. This type of design is similar to the one-shot case study design, in that anintervention (X) is provided to one group (Group A) and then a posttest is used to measure the effects of the intervention (O). However, thisdesign includes a second control group (Group B) that does not receive the intervention. The idea here is to include the second group to seewhether receiving the intervention really affects the participants’ performance (O). Using our reading score example from the one-shot casestudy section, the researcher would include a group of students (Group A) who are praised and a group of students (Group B) who are notpraised. Since the participants were not randomly assigned to the groups, there would be no way to determine whether the groups wereequivalent or whether the results could be attributed to praising or some other factor, such as motivation or IQ. Thus, this design still offerslimited control over confounding variables and shares similar problems with one-shot case studies and one-group pretest–posttest designs.

The posttest-only with nonequivalent groups’ design takes the following form:

Group A: X_____________O

Group B: ______________O

Quasi-Experimental Designs

Quasi-experimental designs fare better than pre-experimental designs in terms of both level of control and internal validity. They employ bothexperimental and control groups with which to make comparisons. However, similar to pre-experimental designs, quasi-experimental designs donot include randomization, either in the selection of group participants or in the presentation of different interventions within the experimentalgroup (Leedy & Ormrod, 2010). Although quasi-experimental designs control more for confounding variables than do pre-experimental designs,they do not control for them completely. These variables must be considered and acknowledged when analyzing and interpreting the data, asalternative explanations for the results are possible. The following sections discuss the four most common types of quasi-experiments.

Nonequivalent Pretest–Posttest Control-Group Design

The nonequivalent pretest–posttest control-group design is probably the most popular quasi-experimental design. In this design, both theexperimental group (Group A) and the control group (Group B) receive a pretest observation (O) and a posttest observation (O), although onlythe experimental group receives the intervention (X). Neither the experimental group nor the control group is randomly assigned. This designtakes the following form:

Group A: O_____________X_____________O

Group B: O___________________________O

The advantage of this design is that its pretest and posttest allow for comparisons. For example, researchers are able to see whether receivingor not receiving the intervention had an effect on the participants. The disadvantage of this design is that it does not include randomassignment to groups. Random assignment allows us to confirm that the two groups are similar in terms of the dependent variable. However, inthis design, we cannot be sure that the experimental group and control group are equivalent with respect to age, gender, and otherdemographic variables. While this design can rule out some alternative explanations for the study results, it cannot control for all confoundingvariables. As a consequence, the results may not be valid.

Single-Group Interrupted Time Series Design

Time series designs consist of making a series of observations over time, introducing an intervention at some point during the study, and thenmaking additional observations (Leedy & Armrod, 2010). Single-group time series designs contain only one experimental group, which isobserved over time to evaluate predictable patterns of events. For example, suppose you wanted to examine the effects that a 3-weekRepublican presidential campaign (X) would have on the pro-choice beliefs (favoring legalized abortion) of a group of independent voters in aparticular area (Group A). The campaign might include commercial ads, phone calls, or presentations and speeches. Assuming that you had areliable measure of pro-choice beliefs in these voters prior to the campaign, you could start observations (O) a few months before the campaignand then continue observations for a few months after the campaign had ended. The design would take the following form:

Group A: O_______O_______O_______O_______O_______X_______O_______O_______O_______O_______O

So did the campaign affect Group A’s pro-choice beliefs? Researchers could determine whether an increase in beliefs actually occurred. However,they could not determine whether the change was solely due to the campaign without including an equivalent control group, a pretest, and aposttest. Thus, the major disadvantage of this design is that it cannot allow researchers to conclude that the results were caused by theintervention and not by some other real-world event or confounding variable.

Control-Group Interrupted Time Series Design

Another variation of the time series design is the control-group interrupted time series design. This design is similar to the single-group timeseries design; however, it includes a control group that does not receive the intervention. Thus, the design takes the following form:

Group A: O_______O_______O_______O_______O_______X_______O_______O_______O_______O_______O

Group B: O_______O_______O_______O_______O________________O_______O_______O_______O_______O

Using the campaign example discussed in the previous design, in this design, the control group would not be exposed to the campaign. Instead,they would simply receive observations throughout the study at the same time as the experimental group. Providing a control group ensuresgreater internal validity because it allows researchers to examine whether the campaign had a direct effect on participants’ beliefs. Assumingthat the groups were fairly equivalent demographically and that there were no other confounding variables that could have influenced theirbeliefs, we should see a change only in Group A.

Reversal Time Series Design

Reversal time series designs use a variable intervention approach in order to minimize the possibility that outside influences might affect theresults. This type of design includes one experimental group (Group A), presence or absence of the intervention over time (X and No X,respectively) and observations of the dependent variable at regular intervals throughout the study (O). Thus, each participant is either exposedor not exposed to the intervention at various points during the study. After each exposure or nonexposure, observations are made. The designtakes the following form:

Group A: X________O________No X________O________X________O________No X________O

For example, let’s say you were interested in studying whether including audiovisual presentations in an online courseroom would help studentslearn statistics. On some weeks, you might want to include video presentations in addition to PowerPoint slide shows, while on other weeks,you might omit the audiovisual material altogether and include just the PowerPoint slides. This approach would let you examine how effectivelystudents learn under both conditions and how effective adding audiovisual materials can be for student learning. If no confounding variablesinfluenced student performance, and audiovisual materials did, in fact, promote student learning, you should see positive changes in studentperformance on the weeks that included audiovisual materials. If confounding variables were present, you might see very different results.

True Experimental Designs

In contrast to pre-experimental and quasi-experimental designs, true experimental designs offer a greater degree of control, randomly selectparticipants and assign them to groups, and randomly assign treatments or interventions to these groups. These characteristics all increase theinternal validity of a study, which, as we learned earlier in this chapter, reflects the degree to which results can be attributed to theindependent variables. Thus, a higher degree of internal validity ensures that the results are related to the independent variable rather than toother, extraneous variables. And, unlike pre-experimental and quasi-experimental designs, true experiments can determine a clear cause-and-effect relationship between independent and dependent variable. With true experiments, there is a limited chance that the results are notdirectly related to the independent variable.

The following three designs are the most highly recommended true experimental designs in the literature because they have been found to bestcontrol for the threats to internal validity (Campbell & Stanley, 1966). These designs also demonstrate the impact that random selection andassignment have on the internal validity of a study.

Pretest–Posttest Control Group Design

The pretest–posttest control group is a classic true experimental design. It begins with randomly assigning (R) participants into both anexperimental and a control group. Both groups are administered a pretest or observation (O) at the beginning of the study and a posttest orobservation (O) at the end of the study; only the experimental group is exposed to the intervention (X). Thus, the control group is isolated fromthe intervention and is subject only to observations. The design takes the following form:

Group A: R_____________O_____________X_____________O

Group B: R______________O________________________O

Such a design controls for two major threats to internal validity. First, randomly assigning participants to groups allows researchers to assumethat the two groups were equal in terms of demographic characteristics before the study began. Second, including a pretest and a posttest forboth groups allows researchers to control for confounding factors such as history, maturation, and testing, since both groups should be affectedequally by those. Thus, in this design, one should see a change in the experimental group only, which would indicate a cause-and-effectrelationship. If a change occurs in the control group, then the researcher would need to consider whether some other confounding variable,such as the presence of a human observer, may have influenced the study.

Solomon Four-Group Design

One weakness of the pretest–posttest control group design is that the process of observing participants at the beginning of the study mayinfluence how the experimental group responds to the intervention (Leedy & Omrod, 2010). Thus, the pretest observation may motivateparticipants to respond to the intervention in a more positive and beneficial manner. To address this weakness, Solomon (1949) proposed anextension of the pretest–posttest control group design.

As we can see in the following depiction, the Solomon four-group design enrolls the experimental and control groups, included in the pretest–posttest control group design, along with two additional groups. The two additional groups are also randomly assigned (R); however, they do notreceive a pretest observation. Instead, one group receives the intervention (X) and posttest observation (O), and the other group receives only aposttest observation (O). This design allows the researcher to control for the influences of a pretest observation.

Group A: R_____________O_____________X_____________O

Group B: R______________O__________________________O

Group C: R___________________________X______________O

Group D: R_________________________________________O

By adding two groups, the Solomon four-group design also increases the external validity of the study. For instance, if Groups C and D differedin the same way that Groups A and B did, one could assume that a pretest is not required and could generalize the findings to other situationswhere a pretest had not been provided.

Although this design was developed to eliminate the possibility of pretest influences, it is not without limitations. One disadvantage is the largesample size needed to ensure randomly assigned groups. Another is the time demanded for carrying out a more complex design. Finally, Smithand Davis (2007) explained that no available statistical test can treat all four data sets at the same time; therefore, conducting statisticalanalyses may be challenging.

Posttest-Only Control Group Design

The posttest-only control group design resembles the pretest–posttest control group design, except that the pretest is removed from both theexperimental and control groups. It is also identical to the additional two groups’ protocol in the Solomon four-group design.

Posttest-only group designs randomly assigns participants to an experimental group and a control group. Only the experimental group receivesthe intervention (X), whereas both the experimental and control groups receive a posttest observation (O). There are no pretest observations.The design takes the following form:

Group A: R_____________X_____________O

Group B: R___________________________O

Does the lack of a pretest make this design less desirable than the other two true experimental designs? The answer is no. Rather, randomlyassigning participants to groups is critical in this design because it ensures that the groups are demographically equivalent at the beginning ofthe study. Thus, any changes during the posttest observation in the experimental group are likely the result of the intervention and notdifferences between the two groups. So random assignment and including a control group both control for threats to internal validity, makingthis a very powerful design.

As we have observed, true experiments exhibit more control over a study and provide for greater internal validity. They are often preferred overpre-experimental and quasi-experimental designs because their results can be generalized to a larger population. Table 5.3 summarizes thecharacteristics of each type of experimental design.

Table 5.3: Characteristics of pre-experimental, quasi-experimental, and true experimental designs
Design Characteristics	Pre-Experimental Design	Quasi-ExperimentalDesign	True ExperimentalDesign
Inclusion of a control group	In some cases, but notalways	Often	Yes, always
Random selection of participants from a population	No	No	Yes, always
Random assignment of participants to groups	No	No	Yes, always
Random assignment of treatments or interventionsto groups	No	No	Yes, always
Control over extraneous or confounding variables	None	Some	Yes

Designing True Experiments

The process of designing true experiments boils down to deciding what variables to manipulate and how to do so. In this section, we will covertwo broad issues related to experimental design: deciding how to structure the levels (different versions) of an independent variable anddeciding on the number of independent variables necessary to test the hypotheses. While these decisions may seem tedious, they are at thecrux of designing successful experiments, and, therefore, are the key to performing successful tests of research hypotheses.

Levels of the Independent Variable

In designing internally valid experiments, the primary goal is to ensure that the levels of independent variables are equivalent in every way butone. This allows researchers to make causal statements about the effects of that single change. There are two primary ways in whichresearchers can control what levels of the independent variables the groups will receive: using between-subject designs and within-subjectdesigns.

Between-Subject Designs

In some of the research designs discussed so far (primarily quasi-experimental and true experimental designs), the levels of our independentvariables have represented two distinct groups—participants are in either the control group or the experimental group. This type of design isreferred to as a between-subject design because the levels differ between one subject and the next. Each participant who enrolls in theexperiment is exposed to only one level of the independent variable. That is, an individual participant might be in either the experimental orthe control group.

Most of the examples so far illustrated between-subject designs: Participants receive either alcohol or placebo; students read an article designedto prime either their Asian or their female identity; and graduate students train rats that are falsely labeled either bright or dull. This approachis very common and has the advantage of using distinct groups to represent each level of the independent variable. In other words, participantswho are asked to consume alcohol are completely distinct from those asked to consume the placebo drink. However, this is only one option forstructuring the levels of the independent variable. In this section, we will examine two additional ways to structure these levels.

Within-Subject Designs

In some cases, the levels of the independent variable represent the same participants but at different time periods. This type of design isreferred to as a within-subject design because the levels differ within people. Each participant who enrolls in the experiment would be exposedto both or all levels of the independent variable. That is, every participant would be in both the experimental and the control group. Within-subject designs are often used to compare changes over time in response to various stimuli. For example, you might measure anxiety symptomsbefore and after people are locked in a room with a spider or measure depression symptoms before and after people undergo drug treatment.

Within-subject designs have two main advantages over between-subject designs. First, because the same people determine both levels of theindependent variable, these designs require fewer participants. Let’s say you decided to collect data from 20 participants at each level of yourindependent variable. In a between-subject design with three levels, you would need 60 people. However, if you ran the same experiment as awithin-subject design—exposing the same group of people to three different sets of circumstances—you would need only 20 people. Thus,within-subject designs are often a good way to conserve resources.

Second, participants serve as their own control group, allowing the researcher to minimize a big source of error variance. Remember that onekey feature of experimental design is the researcher’s power to assign people to groups; this is done to randomly distribute subject differencesacross the levels of the independent variable. Using a within-subject design solves the problem of subject differences in another way—byexamining changes within people. For instance, in the example regarding spiders and anxiety, some participants would likely have higherbaseline anxiety than others. By measuring changes in anxiety in the same group of people before and after spider exposure, the researcherwould be able to minimize the effects of individual differences.

Problems With Within-Subject Designs

Within-subject designs also have two clear disadvantages compared with between-subject designs. First, there is the risk of carryover effects, inwhich the effects of one level are still present when another level is introduced. Because the same people are exposed to all levels of theindependent variable, it can be difficult to separate the effects of one level from the effects of the others. One common paradigm in researchon emotions is to show participants several film clips that elicit different types of emotion. People might view one clip showing a puppy playingwith a blanket, another showing a child crying, and another showing a surgical amputation. Even without seeing these in full color, you canimagine that it would be hard to shake off the disgust triggered by the amputation in order to experience the joy triggered by the puppy.

When researchers use a within-subject design, they take steps to minimize carryover effects. In studies of emotion, for example, researcherstypically show a brief, neutral clip, such as waves rolling onto a beach, between emotional clips so that participants experience each emotionafter viewing a benign image. Another simple technique is to collect data from the control condition first whenever possible. In the examplestudy of spiders and anxiety, it would be important to measure baseline anxiety at the start of the experiment before exposing people tospiders. Once people had been surprised by a spider, it would be hard to get them to relax enough to collect control ratings of anxiety.

Ryan McVay/Photodisc/Thinkstock

Carryover effects can be understoodthrough the example of monitoringpeople's reactions to different film clips.How they feel about one image mayinfluence how they react to the nextimage.

Second, there is a risk of order effects, meaning that the order in which levels are presented canmoderate their effects. Order effects fall into two categories. The practice effect happens whenparticipants’ performance improves over time simply due to making repeated attempts. This is aparticular problem in studies that examine learning. Let’s say you use a within-subject design to comparetwo techniques for teaching people to solve logic problems. Participants would learn technique A, thentake a logic test, then learn technique B, and then take a second logic test. The possible problem thatarises is that participants will have had more opportunities to practice logic problems by the time theytook the second test. This would make it difficult to separate the effects of practice from the effects ofdifferent teaching techniques.

The flipside of practice effects is the phenomenon of the fatigue effect, which happens whenparticipants’ performance decreases over time owing to repeated testing. Let’s say you ran a variation ofthe experiment just described, attempting to teach people to improve their reaction time. Participantsmight learn each technique and have their reaction time tested several times after each one. Theproblem would be that people gradually started to become tired, and their reaction times slowed downdue to fatigue. Thus, it would be difficult to separate the effects of fatigue from the effects of thedifferent teaching techniques.

The result of both of these problems is that the order of presentation becomes confounded with thelevel of the independent variable. Fortunately, there is a relatively easy way to avoid both carryover and fatigue effects by using a process called counterbalancing. Counterbalancing involves varying the order of presentation to groups of participants. The simplest approach is to divideparticipants into as many groups as there are combinations of levels in the experiment. That is, the researchers create a group for each possibleorder, allowing them to identify the effects of encountering the conditions in different orders. In the preceding examples, the learningexperiments involved two techniques: A and B. To counterbalance these techniques across the study, it would be necessary to divide theparticipants into two groups. One group would be exposed to A and then B; the other group would be exposed to B and then A. When it cametime to analyze the data, researchers would be able to examine the effects of both presentation order and teaching technique. If the order ofpresentation made a difference, then one could conclude that the A/B group differed from the B/A group in some way.

Mixed Designs

The third common way to structure the levels of an independent variable is in a mixed design (discussed in more detail in Section 5.8), whichcontains at least one between-subject variable and at least one within-subject variable. So, in the example just discussed, participants would beexposed to both teaching techniques (A and B), but in only one of two possible orders of presentation. In this case, teaching technique is awithin-subject variable because participants experience both levels. And presentation order is a between-subject variable because participantsexperience only one level. When there is one of each in the overall experiment, this is a mixed design.

One common use of mixed designs is in studies that compare the effects of different drugs. Imagine you wanted to compare three new drugs—Drug X, Drug Y, and a placebo control—to determine which had the strongest effects on reducing depression symptoms. To do this study, youwould want to measure depression symptoms on at least three occasions: before starting drug treatment, after a few months of taking the drug(or placebo), and then again after a few months of stopping the drug (to assess relapse rates). So the participants would be given one of threepossible drugs and then measured at each of three time periods. In this mixed design, measurement time is a within-subject variable becauseparticipants experience all three levels, whereas the drug is a between-subject variable because participants experience only one of threepossible levels.

The hypothetical results of this study are shown in Figure 5.4. You can see that the placebo pill has no effect on depression symptoms;depression scores in this group are the same at all three measurements. Drug X appears to significantly improve depression symptoms;depression scores drop steadily across measurements in this group. And, strangely, Drug Y seems to make depression worse; depression scoresincrease steadily across measurements in this group. The mixed design allows researchers both to track people over time and to comparedifferent drugs in one study.

Figure 5.4: Example of a mixed-subjects design

Research: Thinking Critically Outwalking Depression

By Michael Otto

There is a great deal of evidence that exercise not only improves mood and enhances well-being butalso is an effective intervention for depression. That is, well-conducted clinical trials have repeatedlyshown mood benefit from exercise in adults with clinical depression. Indeed, there is evidence thatexercise provides benefits at levels similar to that found for antidepressant medication.

Now, there is new evidence for the power of exercise when other treatments for depression have notprovided adequate help. A recent study, published in the August 2011 issue of the Journal of PsychiatricResearch, provides data that exercise can provide benefits when medication alone does not (Moto-Pereira et al, 2011). Specifically, researchers in Portugal examined the effects of exercise—in this caseprogrammed episodes of walking for 30–45 minutes five times per week—on depressed adults whofailed to respond to two previous trials of antidepressant medication. In the study, all patients remainedon their antidepressant medication, and two thirds of the sample received the program of regular(walking) exercise.

The results were dramatic. Of those patients who received only medication, no average changes indepression mood ratings were seen over the next 12 weeks. In contrast, clear improvement was seen inthose who exercised, with 10 of the 19 patients who exercised showing a response or full remission insymptoms during this time.

Although this was a small study, the results were consistent with previous exercise studies. These studiesunderscore the importance of considering combined treatment strategies for patients who do notrespond to one type of intervention—if one treatment does not work, consider using other resourceswith evidence for success. In the case of treatment of depression, these resources prominently includepsychotherapy (with a wealth of studies indicating that specific forms of psychotherapy like CognitiveBehavior Therapy or Interpersonal Psychotherapy can bring about timely relief of depression), a range ofmedication treatments, or, with increasing evidence, regular exercise. Within these choices there is not aclear cure-all option, but, importantly, there is a range of options to be pursued to try to find the rightfit for any particular person suffering from depression.

A number of important features of the Portuguese study deserve additional note. First, the studyprovided additional evidence that high intensities of exercise are not required to bring about moodchanges. Moderate exercise is adequate for mood benefit. Second, the study used an exercise programthat relied on the individual efforts of the patient. Exercise was performed on a treadmill at thetreatment center for one out of the five weekly sessions; the rest of the sessions were up to theindividual. This individual program of exercise helped show that the mood effects of exercise were notsimply due to the social contact from a group of patients exercising together. It also showed thatmoderate exercise is an accessible option even for chronically depressed individuals. Third, the exerciseprescription included the use of regular reminders for exercise: keeping walking shoes in a visiblelocation, having support for the walking program from family, or arranging for written or cell phonereminders for walking sessions. Simple reminders like these can have a powerful effect on helpingpeople keep up with any new habit.

Otto, M., & Smits, J. (2011, July 26). Outwalking depression: Consider exercise when the antidepressants have not helped. Psychology Today.Retrieved from http://www.psychologytoday.com/blog/exercise-and-mood/201107/outwalking-depression

Think about it:

Identify the following essential aspects of this experimental design:
1. What are the independent and dependent variables in this study?
2. How many levels does the independent variable have?
3. Is this a between-subject, within-subject, or mixed design?
4. Draw a simple table labeling each condition.

What existent differences between groups should the researchers be sure to take into account? Nameas many as you can.

How should the researchers assign participants to the conditions in order to ensure that existentdifferences cannot account for the results?

How might expectancy effects influence the results of this study? Can you think of any ways to controlfor this?

Briefly state how you would replicate this study in each of the following ways:
1. exact replication
2. conceptual replication
3. participant replication
4. constructive replication

One-Way Versus Factorial Designs

The second big issue in creating experimental designs is to decide on the number of independent variables to manipulate. In some cases, wecan test hypotheses by manipulating a single independent variable and measuring the outcome, for example, giving people either alcohol or aplacebo drink and measuring the intention to use condoms. In other cases, hypotheses involve more complex combinations of variables. Earlierin the chapter, we discussed research findings that people tend to act more aggressively after a peer group has rejected them. But thishypothesis could be extended by asking what happens when people are rejected by members of the same sex versus members of the oppositesex. And one could go a step further and test whether the attractiveness of the rejecters matters, for a total of three independent variables.These examples illustrate two broad categories of experimental design, known as one-way and factorial designs.

One-Way Designs

If a study involves giving people either alcohol or a placebo and then measuring outcomes, it has a one-way design, which has only oneindependent variable with two or more levels. These tend to be the simplest experiments and have the advantage of testing manipulations inisolation. One-way designs are used in the majority of drug studies. These types of study compare the effects on medical outcomes for peoplerandomly assigned, for instance, to take the antidepressant drug Prozac or a placebo. Note that a one-way design can still have multiple levels—in many cases, it is preferable to test several different doses of a drug. So, for example, researchers might test the effects of Prozac by assigningpeople to take doses of either 5 mg, 10 mg, 20 mg once daily, or a placebo control. If so, the independent variable would be the drug dose,and the dependent variable would be a change in depression symptoms. This design would allow researchers to compare all three drug doses toa placebo control, as well as to compare the effects of each dose. Figure 5.5 shows hypothetical results from this study. Notice that even thosereceiving the placebo showed a drop in depression symptoms, with the maximum benefit caused by the 10-mg dose of Prozac.

Figure 5.5: Comparing drug doses in a one-way design

Factorial Designs

Despite the appealing simplicity of one-way designs, experiments conducted in the field of psychology with only one independent variable arerelatively rare. The real world is much more complicated, which means that studies that focus on people’s thoughts, feelings, and behaviorsmust try to capture the associated complexity. Thus, the rejection and aggression example discussed previously is not that far-fetched. If aresearcher wanted to manipulate the presence of rejection, the sex of the rejecters, and the attractiveness of the rejecters in a single study, theexperiment would need a factorial design. Factorial designs are those that have two or more independent variables, each of which has two ormore levels. When using a factorial design, the purpose is to observe both the effects of individual variables and the combined effects ofmultiple variables.

Factorial designs have their own terminology to reflect the fact that they comprise both individual variables and combinations of variables. Atthe beginning of this chapter, we learned that the versions of an independent variable are referred to as both levels and conditions, and thatthere is a subtle difference between the two. This difference comes into play when discussing factorial designs. Specifically, levels refer to theversions of each independent variable, while conditions refer to the groups formed by combinations of independent variables. Let’s walkthrough one variation of our rejection and aggression example from this perspective: The first independent variable has two levels becauseparticipants are either rejected or not rejected. And the second independent variable has two levels because members of the same sex or theopposite sex do the rejecting. To determine the number of conditions in this study, calculate the number of different experiences thatparticipants can have in the study. This is a simple matter of multiplying the levels of separate variables, so 2 is multiplied by 2, for a total offour conditions.

Researchers also have a way to quickly describe the number of variables in their design: A two-way design has two independent variables; athree-way design has three independent variables; an eight-way design has eight independent variables, and so on. Even more useful, thesystem of factorial notation offers a simple way to describe both the number of variables and the number of levels in experimental designs. Forinstance, you might describe your design as a 2 × 2 (pronounced “two by two”), which instantly communicates two things: (1) you have twoindependent variables, indicated by the presence of two separate numbers, and (2) each independent variable has two levels, indicated by thenumber 2 listed for each one.

The 2 × 2 Design

One of the most common factorial designs also happens to be the simplest one: the 2 × 2 design. These designs have two independentvariables, with two levels each, for a total of four experimental conditions. The simplicity of these designs makes them a useful way to becomemore comfortable with some of the basic concepts of experiments. This section will walk you through an example of a 2 × 2 design and analyzeit in detail.

Beginning in the late 1960s, social psychologists developed a keen interest in understanding the predictors of helping behavior. This wasinspired, in large part, by the tragedy of Kitty Genovese, who was killed outside her apartment building while none of her neighbors called thepolice (Gansberg, 1964). (See Research: Making an Impact in Chapter 2, Section 2.1.) In one representative study, Princeton psychologists JohnDarley and Bibb Latané examined people’s likelihood of responding to a staged emergency. Participants were led to believe that they weretaking part in a group discussion over an intercom system, but in reality, all the other participants’ voices were prerecorded. The keyindependent variable was the number of other people supposedly present, ranging from two to six. A few minutes into the conversation, oneparticipant appeared to have a seizure. The recording went like this (actual transcript; Darley & Latané, 1968):

I could really-er-use some help so if someone would-er-give me a little h-hel-puh-er-er-er c-could somebody er-er-hel-er-uh-uh-uh[choking sounds] . . . I’m gonna die-er-er-I’m . . . gonna die-er-hel-er-er-seizure-er [chokes, then quiet].

What do people do in this situation? Do they help? How long does it take? Darley and Latané discovered that two things happen as the groupbecame larger: People were less likely to help at all, and those who did help took considerably longer to do so. One of the primary conclusionsto come out of this and other studies is that people are less likely to help when other people are present because the responsibility for helpingis diffused among the members of the crowd (Darley & Latané, 1968).

Building on this conclusion, the sociologist Jane Piliavin and her colleagues (Piliavin, Rodin, & Piliavin, 1969; Piliavin, Piliavin, & Rodin, 1975)explored the influence of two additional variables on helping behaviors by staging an emergency on a New York City subway train in which aperson who was in on the study appeared to collapse in pain. Specifically, the researchers manipulated two variables in their staged emergency.The first independent variable was whether there was a medical intern nearby, who could be easily identified by wearing blue scrubs. Thesecond independent variable was whether the victim had a large disfiguring scar on his face. The combination of these variables resulted in fourconditions, as shown in Table 5.4. The dependent variable in this study was the percentage of people taking action to help the person appearingto collapse in pain.

Table 5.4: 2 × 2 Design of the Piliavin et al. study
	No intern	Intern
No scar
Scar

The authors predicted that bystanders would be less likely to help if there was a perceived medical professional nearby since he or she wasconsidered more qualified to help the victim. They also predicted that people would be less likely to help when the confederate had a large scarbecause previous work had demonstrated convincingly that people avoid contact with those who are disfigured or have other stigmatizingconditions (e.g., Goffman, 1963). As seen in Figure 5.6, these hypotheses were supported by the results. Both the presence of a scar and thepresence of a perceived medical professional reduced the percentage of people who came to help. But there’s something else going on in theseresults: When the confederate is not scarred, having an intern nearby leads to a small decrease in helping (from 88% to 84%). However, whenthe confederate had a large facial scar, having an intern nearby decreased helping from 72% to 48%! In other words, it seems these variableshad a combined effect on helping behavior. We will examine these combined effects more closely in the next section.

Figure 5.6: Sample 2 × 2 Design: Results from Piliavin et al.(1975)

Other Factorial Designs

Experimental designs can often be more complex than a simple 2 × 2 format. In another variation of the rejection and aggression study,participants would be categorized in one of eight cells—they are either rejected or not, by a person of either the same sex or the opposite sex,who happens to be either attractive or unattractive. This design would have eight conditions (2 × 2 × 2), as shown in Table 5.5a, with theseparate conditions numbered for illustration purposes.

Table 5.5a: Levels versus conditions in a 2 × 2 × 2 design
	Rejected			Not Rejected
Rejecter	Attractive	Unattractive	Attractive		Unattractive
Same sex
Opposite sex
This is a 2 × 2 × 2 design for a total of eight conditions (numbered 1–8). “Rejection” has two levels: rejected and not rejected. “Attractiveness” has two levels: attractive and unattractive. “Sex” has two levels: the same sex and the opposite sex rejecter.

A different study might describe a design as 2 × 3, which would tell people two things: There are two independent variables, and one variablehas two levels (the 2) while the other variable has three levels (the 3). One possibility for this 2 × 3 design is illustrated in Table 5.5b. For thishypothetical study, the researcher was interested in the effects of discomfort on mood. She manipulated both hunger, with three levels, andpain, with two levels. Participants were asked to abstain from eating for 4 hours (“mild hunger”) or 24 hours (“extreme hunger”), or they wereallowed to eat before the study (“no hunger”). During the laboratory session, participants were asked either to hold their hand in a bucket ofice water (“pain”) or not (“no pain”). As you can see in Table 5.5b, this resulted in six possible conditions for her participants to experience.

Table 5.5b: Levels versus conditions in a 2 × 3 design
	No Hunger	Mild Hunger	Extreme Hunger
Pain
No Pain
This is a 2 × 3 design for a total of six conditions (numbered 1–6). “Pain” has two levels: presence and absence of pain. “Hunger” has three levels: no hunger, mild hunger, and extreme hunger.

Main Effects and Interactions

When experiments involve only one independent variable (IV), the analyses can be as simple as comparing two group means—as with theexample in Chapter 2 (Section 2.4, Hypothesis Testing) comparing the happiness levels of couples with and without children. But what aboutcases where the design has more than one independent variable?

In a factorial design, there are two types of effect: A main effect refers to the effect of each independent variable on the dependent variable,averaging values across the levels of other variables. A 2 × 2 design has two main effects; a 2 × 2 × 2 design has three main effects becausethere are three independent variables. An interaction occurs when the variables have a combined effect; that is, the effects of one independentvariable are different depending on the levels of the other independent variable. So applying this new terminology to the Piliavin et al. (1969,1975) “subway emergency” study yields three possible results (possible because researchers eventually use statistical analyses to verify them):

The main effect of scar: Does the presence of a scar affect helping behavior?
Yes. More people help in absence of a facial scar. In Figure 5.6, you can see that the bars on the left (no scar) are, on average, higher than thoseon the right (scar).
The main effect of intern: Does the presence of an intern affect helping behavior?
Yes. More people help when there is no medical intern on hand. In Figure 5.6, notice that the red bars (no intern) are, on average, higher thanthe orange bars (intern).
The interaction between scar and intern: Does the effect of one variable depend on the effect of another variable?
Yes. If you refer to Figure 5.6, you can see that the presence of a medical intern matters more when the victim has a facial scar. In visual terms,the gap between red and orange bars is much larger in the bars on the right. This indicates an interaction between scar and intern.

Let’s walk through a fictional example. Imagine you were interested in people’s perceptions of actors in different types of movies. You mightpredict that some actors are better suited to comedy and others are better suited to action movies. One simple way to test this hypothesiswould be an experiment that showed four movies in a 2 × 2 design, using the same two actors in two movies (for a total of four conditions).Our first independent variable would be the movie type, with two levels: action and comedy. Our second independent variable would be theactor, with two levels: Will Smith and Arnold Schwarzenegger. Our dependent variable would be the ratings of each movie on a 10-point scale.This gives us three possible results:

The main effect of actor: Do people generally prefer Will Smith or Arnold Schwarzenegger, regardless of the movie?
The main effect of movie type: Do people generally prefer action or comedy movies, regardless of the actor?
The interaction between actor and movie type: Do people prefer each actor in a different kind of movie? (i.e., are ratings affected by thecombination of actor and movie type?)

After collecting data from a sample of participants, we end up with the following average ratings for each movie, shown in Table 5.6.

Table 5.6: Main effects and marginal means: The actor study
	Arnold	Will	Marginal Mean
Action		1.5	3.75
Comedy			6.5
Marginal Mean	5.5	4.75
These two means let us compare movie ratings, ignoring the actor. People have a slight preference for Comedy (6.5) over Action (3.75). These two means let us compare actor ratings, ignoring the movie type. People have a slight preference for Arnold (5.5) over Will (4.75).

Remember that main effects represent the effects of one independent variable, averaging across the levels of the other independent variable. Toaverage across levels, we calculate the marginal mean, or the combined mean across levels of another factor. In other words, the marginalmean for action movies is calculated by averaging together the ratings of both Arnold Schwarzenegger and Will Smith in action movies. Themarginal mean for Arnold Schwarzenegger is calculated by averaging ratings of Arnold Schwarzenegger in both action and comedy movies. Doingthis for our 2 × 2 design results in four marginal means, which are presented alongside the participant ratings in Table 5.6. We would need toverify these patterns with statistical analyses, but it appears we have two main effects, based on these marginal means: People have a slightpreference for comedy over action movies, as well as a slight preference for Arnold Schwarzenegger’s acting over Will Smith’s acting, regardlessof the movie.

What about the interaction? Our main hypothesis here is that some actors perform best in some genres of movies (e.g., action or comedy) thanthey do in other genres. This suggests that the actor and the movie type combine to influence people’s ratings of the movies. We can get asense of this from examining the means in Table 5.6, but it is even easier to appreciate in a graph. Figure 5.7 shows the mean of participants’ratings across the four conditions. Focusing first on the ratings of Arnold Schwarzenegger, notice that participants did have a slight preferencefor him in action (6) versus comedy (5) roles. Then, examining ratings of Will Smith, you can see that participants had a strong preference forhim in comedy (8) versus action (1.5) roles. Together, this set of means indicates an interaction between actor and movie type because theeffects of one variable depend on another. Or, in plain English: People’s perceptions of the actor depend on the type of movie he is in. Thispattern of results is a nice fit for our hypothesis that certain actors are better suited to certain types of movie: Arnold should probably stick toaction movies, and Will should definitely stick to comedies.

Figure 5.7: Interaction in the actor study

Before we move on to analyses, let’s look at one more example from a published experiment. A large body of research in social psychologysuggests that stereotypes can negatively impact performance on cognitive tasks (e.g., tests of math and verbal skills). According to Stanfordsocial psychologist Claude Steele and his colleagues, the fear of confirming negative stereotypes about one’s group acts as a distraction. Thisdistraction—which they term stereotype threat—makes it hard to concentrate and perform well, and thus leads to lower scores on a cognitivetest (Steele, 1997). One of the primary implications of this research is that ethnic differences in standardized test scores can be viewed as asituational phenomenon, so if we change the situation, the differences should go away. In the first published study of stereotype threat, ClaudeSteele and Josh Aronson (1995) found that when African American students at Stanford were asked to indicate their race before taking astandardized test, this was enough to remind them of negative stereotypes, and they performed poorly. But when the testing situation waschanged, and participants were no longer asked their race, these students performed at the same level as Caucasian students. It is worthemphasizing that these were Stanford students and had therefore met admissions standards for one of the best universities in the nation. Eventhis group of elite students was susceptible to situational pressure but performed at their best when the pressure was eliminated.

In a great application of stereotype threat, social psychologist Jeff Stone at the University of Arizona asked both African American and Caucasiancollege students to try their hand at putting on a golf course (Stone, Lynch, Sjomeling, & Darley, 1999). Putting was described as a test ofnatural athletic ability to half of the participants and as a test of sports intelligence to the other half. Thus, there were two independentvariables: the race of the participants (African American or Caucasian) and the description of the task (“athletic ability” or “sports intelligence”).Note that “race” in this study was technically a quasi-independent variable because it was not manipulated. This resulted in a total of fourconditions, and the dependent variable was the number of putts that participants managed to make. Stone and colleagues hypothesized thatdescribing the task as a test of athletic ability would lead Caucasian participants to worry about the stereotypes regarding their poor athleticability. In contrast, describing the task as a test of sports intelligence would lead African American participants to worry about the stereotypesregarding their lower intelligence.

Consistent with their hypotheses, Stone and colleagues found an interaction between race and task description but no main effects. That is,neither race was better at the putting task overall, and neither task description had an overall effect on putting performance. But thecombination of these variables proved fascinating. When the task was described as measuring sports intelligence, the African Americanparticipants did poorly for fear of confirming negative stereotypes about their overall intelligence. But when the task was described asmeasuring natural athletic ability, the Caucasian participants did poorly for fear of confirming negative stereotypes about their athleticism. Thisis a beautiful illustration of an interaction; the effects of one variable (task description) depend on the effects of another (race of participants).And this is further confirmation of the power of the situation: Neither group did better or worse overall, but both were responsive to asituationally induced fear of confirming negative stereotypes.

.4 Experimental Designs

Pre-Experimental Designs

One-Shot Case Study

Group A: X_____________O

One-Group Pretest–Posttest Design

The one-group pretest–posttest design involves one group but includes both a pretest (O1) before the intervention (X) is given and a posttest(O2) after the intervention is provided, as shown below:

Group A: O1_____________X_____________O2

Static Group Comparison or Posttest-Only With Nonequivalent Groups

The posttest-only with nonequivalent groups’ design takes the following form:

Group A: X_____________O

Group B: ______________O

Quasi-Experimental Designs

Nonequivalent Pretest–Posttest Control-Group Design

Group A: O_____________X_____________O

Group B: O___________________________O

Single-Group Interrupted Time Series Design

Group A: O_______O_______O_______O_______O_______X_______O_______O_______O_______O_______O

Control-Group Interrupted Time Series Design

Group A: O_______O_______O_______O_______O_______X_______O_______O_______O_______O_______O

Group B: O_______O_______O_______O_______O________________O_______O_______O_______O_______O

Reversal Time Series Design

Group A: X________O________No X________O________X________O________No X________O

True Experimental Designs

Pretest–Posttest Control Group Design

Group A: R_____________O_____________X_____________O

Group B: R______________O________________________O

Solomon Four-Group Design

Group A: R_____________O_____________X_____________O

Group B: R______________O__________________________O

Group C: R___________________________X______________O

Group D: R_________________________________________O

Posttest-Only Control Group Design

Group A: R_____________X_____________O

Group B: R___________________________O

Table 5.3: Characteristics of pre-experimental, quasi-experimental, and true experimental designs
Design Characteristics	Pre-Experimental Design	Quasi-ExperimentalDesign	True ExperimentalDesign
Inclusion of a control group	In some cases, but notalways	Often	Yes, always
Random selection of participants from a population	No	No	Yes, always
Random assignment of participants to groups	No	No	Yes, always
Random assignment of treatments or interventionsto groups	No	No	Yes, always
Control over extraneous or confounding variables	None	Some	Yes

Designing True Experiments

Levels of the Independent Variable

Between-Subject Designs

Within-Subject Designs

Problems With Within-Subject Designs

Ryan McVay/Photodisc/Thinkstock

Carryover effects can be understoodthrough the example of monitoringpeople's reactions to different film clips.How they feel about one image mayinfluence how they react to the nextimage.

Mixed Designs

Figure 5.4: Example of a mixed-subjects design

Research: Thinking Critically Outwalking Depression

By Michael Otto

Think about it:

Identify the following essential aspects of this experimental design:
1. What are the independent and dependent variables in this study?
2. How many levels does the independent variable have?
3. Is this a between-subject, within-subject, or mixed design?
4. Draw a simple table labeling each condition.

What existent differences between groups should the researchers be sure to take into account? Nameas many as you can.

How should the researchers assign participants to the conditions in order to ensure that existentdifferences cannot account for the results?

How might expectancy effects influence the results of this study? Can you think of any ways to controlfor this?

Briefly state how you would replicate this study in each of the following ways:
1. exact replication
2. conceptual replication
3. participant replication
4. constructive replication

One-Way Versus Factorial Designs

One-Way Designs

Figure 5.5: Comparing drug doses in a one-way design

Factorial Designs

The 2 × 2 Design

Table 5.4: 2 × 2 Design of the Piliavin et al. study
	No intern	Intern
No scar
Scar

Figure 5.6: Sample 2 × 2 Design: Results from Piliavin et al.(1975)

Other Factorial Designs

Table 5.5a: Levels versus conditions in a 2 × 2 × 2 design
	Rejected	Not Rejected
Rejecter	Attractive	Unattractive	Attractive	Unattractive
Same sex
Opposite sex
This is a 2 × 2 × 2 design for a total of eight conditions (numbered 1–8). “Rejection” has two levels: rejected and not rejected. “Attractiveness” has two levels: attractive and unattractive. “Sex” has two levels: the same sex and the opposite sex rejecter.

Table 5.5b: Levels versus conditions in a 2 × 3 design
	No Hunger	Mild Hunger	Extreme Hunger
Pain
No Pain
This is a 2 × 3 design for a total of six conditions (numbered 1–6). “Pain” has two levels: presence and absence of pain. “Hunger” has three levels: no hunger, mild hunger, and extreme hunger.

Main Effects and Interactions

The main effect of scar: Does the presence of a scar affect helping behavior?
Yes. More people help in absence of a facial scar. In Figure 5.6, you can see that the bars on the left (no scar) are, on average, higher than thoseon the right (scar).
The main effect of intern: Does the presence of an intern affect helping behavior?
Yes. More people help when there is no medical intern on hand. In Figure 5.6, notice that the red bars (no intern) are, on average, higher thanthe orange bars (intern).
The interaction between scar and intern: Does the effect of one variable depend on the effect of another variable?
Yes. If you refer to Figure 5.6, you can see that the presence of a medical intern matters more when the victim has a facial scar. In visual terms,the gap between red and orange bars is much larger in the bars on the right. This indicates an interaction between scar and intern.

The main effect of actor: Do people generally prefer Will Smith or Arnold Schwarzenegger, regardless of the movie?
The main effect of movie type: Do people generally prefer action or comedy movies, regardless of the actor?
The interaction between actor and movie type: Do people prefer each actor in a different kind of movie? (i.e., are ratings affected by thecombination of actor and movie type?)

After collecting data from a sample of participants, we end up with the following average ratings for each movie, shown in Table 5.6.

Table 5.6: Main effects and marginal means: The actor study
	Arnold	Will	Marginal Mean
Action		1.5	3.75
Comedy			6.5
Marginal Mean	5.5	4.75
These two means let us compare movie ratings, ignoring the actor. People have a slight preference for Comedy (6.5) over Action (3.75). These two means let us compare actor ratings, ignoring the movie type. People have a slight preference for Arnold (5.5) over Will (4.75).

Figure 5.7: Interaction in the actor study

5.5 Analyzing Experiments

So far, we have been drawing conclusions about our experimental findings in conceptual terms. But naturally, before we actually make adecision about the status of our hypotheses, we have to conduct statistical analyses. In this section, we will cover the most common statisticaltechniques for analyzing experimental data.

Dealing With Multiple Groups

You may find yourself wondering why we need a special technique for experimental designs. After all, we learned in Chapter 2 (Section 2.4,Hypothesis Testing) that we can compare two pairs of means using a t-test. So why not just use several t-tests to analyze our experimentaldesigns? In our movie ratings study, we could analyze the data using a total of six t-tests to capture every possible pair of means: ArnoldSchwarzenegger in a comedy versus Will Smith in a comedy; Arnold Schwarzenegger in an action movie versus Will Smith in an action movie;Arnold Schwarzenegger in a comedy versus an action movie; Will Smith in a comedy versus an action movie; Will Smith in a comedy versusArnold Schwarzenegger in an action movie; and finally Will Smith in an action movie versus Arnold Schwarzenegger in a comedy.

But there is a problem with this approach: The odds of developing a Type I error (a false positive) are increased with every statistical test. Wetypically set our alpha level at .05 for a t-test, meaning that we’re comfortable with a 5% chance of a Type I error. Unfortunately, if we conductsix t-tests, each one has a 5% chance of a Type I error, meaning that we’re now rather likely to draw conclusions about a false positive. Themoral of this story is that we need a statistical approach that reduces the number of comparisons we perform. Fortunately, this can beaccomplished using a statistical technique called the analysis of variance (ANOVA), which tests for differences by comparing the amount ofvariance explained by the independent variables to the amount of variance explained by error. The following sections explore ANOVA in moredetail.

The Logic of ANOVA

The logic behind the analysis of variance is rather straightforward. As we have discussed throughout this course, the variability in data can bedivided into systematic and error variance. That is, we can attribute some of the variability to the factors we are studying, but there will alwaysbe a degree of random error. In our movie ratings study, some of the variability in these ratings can be attributed to our independent variables(differences in actors and movie types), while some of the variability is the result of other factors— perhaps some people simply like moviesmore than other people.

The ANOVA works by comparing the influence of these different sources of variance. We always want to explain as much of the variance aspossible through the independent variables. If the independent variables have more influence than random error does, this is good news. But iferror variance has more influence than the independent variables, this is bad news for the hypotheses. You can get a sense of this by comparingthe three pie charts in Figure 5.8. The proportion of variance explained by our independent variables is shaded in yellow, while the proportionexplained by error is shaded in red. In the top graph, approximately 80% of the variance can be explained by our independent variables, whichcan be viewed as a good result. But in the middle graphs, variance is explained equally by the independent variables and by error, and in thebottom graph, only 30% of the variance is explained by the independent variables. Thus, in the latter two graphs, our independent variables dono better than random error at explaining our results.

Figure 5.8: Comparing sources of variance

One more analogy may be helpful. In the field of engineering, theterm signal-to-noise ratio is used to describe the amount of light,sound, energy, and so forth that is detectable above and beyondbackground noise. This ratio is high when the signal comes throughclearly and low when it is mixed in with static or other interference.Likewise, when you try to tune in your favorite radio station, the goalis to find a clear signal that is not covered up by static. Believe it ornot, the ANOVA statistic is doing the same thing. That is, the analysistells us whether differences in experimental conditions (signal) aredetectable above and beyond error variance (noise). In the nextsection, we’ll take a closer look at how ANOVA makes thesedistinctions.

Calculating ANOVA

The ANOVA statistic—abbreviated with the capital letter F—works bycomparing the variance that is explained by group differences with thevariance explained by error. This is really just a minor adaptation, withnew terminology, of the formula we have encountered before,describing the components of total variance in the dataset:

Total Variance = Systematic Variance + Error Variance

In conducting an ANOVA, these two components will have specialnames, but the idea is the same. Specifically, we refer to systematicvariance as between-group variance, or the variance explained bydifferences in our independent variables. In the movie ratings study,between-groups variance represents the variability in ratings that canbe attributed to different actors, different movies, and thecombinations of these independent variables. In addition, we refer toerror variance as within-group variance, or the total variance thatoccurs across each condition. In the movie ratings study, within-groupsvariance would represent the sum total of all other influences onmovie ratings (i.e., the error).

In this day and age, ANOVA (F) is calculated using one of the numerous statistical software programs; these programs take columns of data andchurn out significance tests to test hypotheses. But in order to understand what these significance tests mean, it is worth walking through aconceptual overview of the elements in the ANOVA formula. The starting point for the F statistic is the calculation of several sum of squares, orthe sum of the squared deviations between individual scores and the overall sample mean. We encountered these sums of squares (abbreviated SS) in Chapter 3 (Section 3.5), in our discussion of calculating the standard deviation.

Using this notation, our components of the variance formula can be rewritten as follows:

SST =	SSBG + SSWG
SST =	sum of squares total
SSBG =	sum of squares between groups
SSWG =	sum of squares within groups

In conceptual terms, SSBG represents the deviation between each condition’s mean and the overall mean of the sample. If you refer to the datafrom our movie ratings study in Table 5.6, you’ll see the four condition means of 6, 5, 1.5, and 8. The overall mean of movie ratings—acrossconditions—is 5.13 (i.e., [6 + 5 + 1.5 + 8] ÷ 4). To compute the SSBG, we subtract each of the four condition means from 5.13, square thedifference, and add up these squared deviation scores.

In conceptual terms, SSWG represents the deviation between each individual score and the condition mean for that individual. To compute SSWGin our movie ratings study, we subtract each individual score from the relevant condition mean, square the difference, and add up the squareddeviation scores.

That’s a lot of math, so let’s take a step back. In essence, these terms are calculating and comparing two sources of variation around the overallmean. When we combine all the participants’ movie ratings, and ignore everything about what they were rating, we get an overall mean of5.13. But, naturally, individual participants deviate from this mean—some people gave higher ratings and some gave lower ratings. The point ofcalculating SS terms is to understand these deviations. Our between-group SS gives us an index of how much each condition deviates from theoverall mean, while our within-group SS gives us an index of how much individuals differ from the mean of their condition. If the former ishigher, it means that the groups’ results matter; if the latter is higher, it means that individual quirks mean more than our independentvariables.

These sums of squares are the first step toward calculating our F statistic, but they have a flaw that has to be addressed. Namely, the sum ofsquares is an imprecise measure because it does not take sample size into account. You’ll notice that the SS terms are calculated by adding upthe squared deviation scores for each participant (or each condition). If our experiment happened to have a large sample, the SSWG wouldautomatically be higher because there would be a greater number of squared deviation scores. Likewise, if our experiment happened to haveseveral conditions, the SSBG would be inflated. Fortunately, this flaw is easily corrected by calculating a value called the mean square, whichcorrects the sum of squares for sample size by dividing by the appropriate degrees of freedom. The resulting values represent the averagedeviation from the sample mean, corrected for the number of scores that went into the calculation—much like the final step in our calculationof standard deviations in Chapter 3 (Section 3.5). So, our components of variance formula is altered one more time:

MST =	MSBG + MSWG
MST =	mean square total
MSBG =	mean square between groups
MSWG =	mean square within groups

We now have all the information we need to calculate the F statistic (ANOVA), using the following formula:

F=between−groupsvariabilitywithin−groupsvariability=MSBGMSWG

This is the formula statistical software programs use to calculate an F value. F is expressed as a ratio of variances, expressing group differencesrelative to error. Thus, the bigger the F value, the higher the ratio of group differences to error, and the bigger the influence of independentvariables. In other words, we want F values to be as large as possible because this indicates that our experimental manipulations make adifference on the outcome variable.

We can also frame the components of this ratio in terms of the null hypothesis versus the experimental hypothesis. If our experimentalhypothesis is true, MSWG represents random variation and MSBG represents meaningful differences between groups. Thus, MSBG > MSWG, and Fwill be a large value. In contrast, if the null hypothesis is true, both MSWG' and MSBG represent random variation because our experimentalmanipulations do not have a meaningful effect. Thus, MSBG = MSWG' and the value of F is 1. One final possibility here is that our value of F canoccasionally be less than 1. This occurs when results are more influenced by random variation than they are by the independent variables; inother words, MSWG > MSBG.

One-Way Versus Factorial ANOVA

Before we move on from the steps in our ANOVA, there is one more level of complexity to explore. The formula we have been discussing so fardescribes an ANOVA only for a one-way design. When there is one independent variable, MST = MSBG + MSWG. However, a factorial design hasmultiple independent variables, and the between-group variance term (MSBG) is divided into several elements. As we discussed, a 2 × 2 designhas three possible results: a main effect of variable A; the main effect of B; and the interaction of A and B. To account for these effects, we splitour between-group variance into three new components:

MSBG =	MSA + MSB + MSAB
MSA =	MSB + MSAB
MSA =	mean square for the main effect of A
MSB =	mean square for the main effect of B
MSAB =	mean square for the interaction of A and B

This results in another update to our components of variance formula, replacing the MSBG term with these three components:

MST =

(MSA + MSB + MSAB) + MSWG

This also results in three F values, expressing each mean square relative to within-group variance. These are illustrated next using our movieratings study. Our 2 × 2 design in that study yields a main effect of actor, a main effect of movie type, and the actor * movie interaction:

FACTOR=MSACTORMSWGFMOVIE=MSMOVIEMSWGFACTOR×MOVIE=MSACTOR×MOVIEMSWG

For illustration purposes, consider the following formula, representing the components of variance in a three-way design (which, incidentally,would yield seven F values!). As our designs become more complex, it becomes easier to appreciate the benefits of statistical softwarepackages.

MST =	(MSA + MSB + MSC + MSAB + MSBC + MSAC + MSABC) + MSWG
MSA =	mean square for the main effect of A
MSB =	mean square for the main effect of B
MSAB =	mean square for the interaction of A and B
MSBC =	mean square for the interaction of B and C
MSAC =	mean square for the interaction of A and C
MSABC =	mean square for the three-way interaction of A, B, and C

Research: Thinking Critically Love Ballad Leaves Women More Open to a Date

Medical News Today

If you’re having trouble getting a date, French researchers suggest that picking the right soundtrackcould improve the odds. Women were more prepared to give their number to an “average” young manafter listening to romantic background music, according to research that appears in the journal Psychology of Music, published by SAGE.

There’s plenty of research indicating that the media affects our behaviour. Violent video games or musicwith aggressive lyrics increases the likelihood of aggressive behaviour, thoughts, and feelings—but doromantic songs have any effect? This question prompted researchers Nicolas Guéguen and Céline Jacobfrom the Université de Bretagne-Sud along with Lubomir Lamy (2009) from Université de Paris-Sud totest the power of romantic lyrics on 18–20 year old single females. And it turns out that at least oneromantic love song did make a difference.

Guéguen and Jacob were part of a research team that had already shown how romantic music played ina flower shop led to male customers spending more money. This time the researchers usedquestionnaires to pinpoint agreed-upon neutral and romantic songs. They chose ‘Je l’aime à mourir,’ awell-known love song by French songwriter Francis Cabrel, and the neutral song ‘L’heure du thé,’ byVincent Delerm. A group of young women separate from the main study rated 12 young male volunteersfor attractiveness, and the researchers picked the one rated closest to “average” to help with theexperiment.

The researchers then set up a scenario where the 87 females each spent time in a waiting room withbackground music playing, before moving to a different room where the experimenter instructed her todiscuss the difference between two food products with the young man. Once the experimenterreturned, she asked them to wait for a few moments alone, and this gave the “average” male a chanceto use his standard chat-up line: “My name is Antoine, as you know, I think you are very nice and I waswondering if you would give me your phone number. I’ll phone you later and we can have a drinktogether somewhere next week.”

The love song in the waiting room almost doubled Antoine’s chances of getting a woman’s number—52% of participants responded to his advances under the influence of Francis Cabrel, whereas only 28%of those who had heard the “neutral” song by Vincent Delerm offered their details.

“Our results confirm that the effect of exposure to media content is not limited to violence and couldhave the potential to influence a high spectrum of behaviour,” says Guéguen. “The results are interestingfor scientists who work on the effect of background music on individuals’ behaviour.”

The results also add weight to a general learning model proposed by Buckley and Anderson (2006) toexplain the effect of media exposure. Their model states that media exposure in general, and not onlyaggressive or violent media, affects individuals’ internal states, which explains why prosocial mediafosters prosocial outcomes.

Why did the music have this effect? It may be that, as shown in earlier research, the music inducedpositive affect (in psychological terms, affect is the experience of feeling or emotion). Positive affect isassociated with being more receptive to courtship requests. Alternatively, the romantic content of thesong may have acted as a prime that then led to displays of behaviour associated with that prime. Ineither case, further research is needed before the researchers will commit to wider generalisations onthe targeted use of love songs. But if you’re a hopeful single, awareness of the background musiccertainly won’t do any harm.

Epping, J. (2010, June 21). Love ballad leaves women more open to a date. Medical News Today. Retrieved from http://www.medicalnewstoday.com/articles/192324.php

Think about it:

In this experiment, the type of song (love song or neutral song) is confounded with at least one othervariable. Try to identify one. Do you think that this confounded variable would make a difference? Howwould you design a study that would overcome this?
Describe how demand characteristics might compromise the internal validity of this study. Can youthink of any ways around this?
Toward the end of the article, the authors suggest that one explanation for these results is that theromantic music put the women into a more positive mood, and that this in turn made them morereceptive to the man. How could you design a study that tested this hypothesis?
Given the nature of the dependent variable in this study, would an ANOVA test be appropriate? Whatwould be the more appropriate statistical test, and why?

Exploring Your Data

Now that we have F values, what do we do with them? At the end of conducting an ANOVA, we have a yes-or-no answer to the followingquestion: Do our experimental groups have a systematic effect on the dependent variable? The answer lets us decide whether to reject the nullhypothesis, but it does not tell us everything we want to know about the data. In essence, a significant F value tells us that there is a significantdifference between the groups, but it does not tell us what the difference is. If we conducted an ANOVA on our movie ratings study, we wouldsee a significant interaction between actor and movie, but we would need to take additional steps to determine the meaning of this interaction.

In this section, we will walk through the process of exploring and interpreting ANOVA results in order to make sense of the data. Our example isdrawn from a published study by Newman, Sellers, and Josephs (2005), which was designed to explore the effects of testosterone on cognitiveperformance. Previous research had suggested that testosterone is involved in two types of complex human behavior. On one hand, people withhigher testosterone tend to perform better on tests of spatial skills, such as having to rotate objects mentally, and perform worse on tests ofverbal skills, such as listing all the synonyms for a particular word. These patterns are thought to reflect the influence of testosterone ondeveloping brain structures. On the other hand, people with higher testosterone are also more concerned with gaining and maintaining highstatus relative to other people. Testosterone is correlated with a person’s position in the hierarchy and tends to rise and fall when people winand lose competitions, respectively. In one study by the sociologist Alan Mazur and his colleagues, testosterone levels were measured before,during, and after a series of professional chess matches. They found that testosterone rose in both players in anticipation of the competition,then rose even further in the winners, but plummeted in the losers (Mazur, Booth, & Dabbs, 1992).

Newman and colleagues (2005) set out to test the combination of these variables. Based on previous research, they hypothesized that peoplewith higher testosterone would be uncomfortable when they were placed in a low-status position, leading them to perform worse on cognitivetasks. The researchers tested this hypothesis by randomly assigning people to a high-status, low-status, or control condition and thenadministering a spatial and a verbal test. This resulted in a 2 (testosterone: high or low) × 3 (condition: high status, low status, control)between-subject design, for a total of six groups. Note that “testosterone” in this study is a quasi-independent variable because it is measuredrather than manipulated by the experimenters.

Once the results were in, the ANOVA revealed an interaction between testosterone and status but no main effects. The results of the study areshown in Figure 5.9. These bars represent z-scores that combine the spatial and verbal tests into one number. So what do these numbersmean? How do we make sense out of the patterns? This involves a combination of comparing means and calculating effect sizes, as discussednext.

Figure 5.9: Exploring your data: Results from Newman et al. (2005)

Comparing Means

The first step in interpreting our results is to compare the various pairs of means within the design. This might seem counterintuitive, since thewhole point of the ANOVA is to test for effects without comparing individual means. Our goal, therefore, is to somehow explore differences inconditions without inflating Type I error rates. There are two strategies for achieving this balance.

Planned comparisons (also called a priori comparisons) involve comparing only the means for which differences were predicted by thehypothesis. In the experiment by Newman et al. (2005), the hypothesis explicitly stated that high-testosterone people should perform better in ahigh-status position than a low-status position. So a planned comparison for this prediction would involve comparing two means with a t-test:high T, high status (the highest red bar); and high T, low status (the lowest yellow bar). Consistent with the researchers’ hypothesis, high-testosterone people did perform higher on both tests, t(27) = 2.35, p = 0.01, but only in a high-status position. Type I errors are of less concernwith planned comparisons because only a small number of theoretically driven comparisons are being conducted.

If you refer to the graph of these results in Figure 5.9 and compare high- with low-testosterone people, you’ll notice another interesting pattern:In a high-status position, high-testosterone people do better than low-testosterone people. But in a low-status position, this pattern is reversed,and high-testosterone people do worse. However, the researchers did not predict these mean comparisons, so it would be cheating to doplanned contrasts. Instead, they would use a second strategy called a post hoc comparison, which controls the overall alpha by taking intoaccount the fact that multiple comparisons are being performed. In most cases, post hoc tests are allowed only if the overall F test result issignificant.

One popular way to conduct post hoc tests while minimizing the error rate is to use a technique called a Bonferroni correction. This technique,named after the Italian mathematician who developed it (Carlo Emilio Bonferroni, 1892–1960), involves simply adjusting the alpha level by thenumber of comparisons that are performed. For example, imagine you want to conduct 10 follow-up post hoc tests to explore your data. TheBonferroni correction would involve dividing the alpha level (.05) by the number of comparisons (10), for a corrected alpha level of .005. Then,rather than using a cutoff of .05 for each test, you use this more conservative Bonferroni-corrected value. Translation: Rather than accepting aType I error rate of 5%, we are moving to a more conservative 0.5% cutoff to correct for the number of comparisons that we are performing.

Another popular alternative to the Bonferroni correction is called Tukey’s HSD (for Honestly Significant Difference). This test works by calculatinga critical value for mean comparisons (the HSD) and then using this critical value to evaluate whether mean comparisons are significantlydifferent. This test manages to avoid inflating Type I error because the HSD is calculated based on the sample size, the number of experimentalconditions, and the MSWG, which essentially tests all the comparisons at once. (Note that ANOVA will tell us only that the means are differentbut not which of them, so for the purpose of identifying the means that are different from others, one can apply Tukey’s test.) In the study byNewman et al. (2005), both of these post hoc tests were significant: Compared with those low in testosterone, high-testosterone people didbetter in a high-status position but worse in a low-status position. This suggests that high testosterone magnifies the effect of testing situationson cognitive performance.

Calculating Effect Size

As you’ll remember from our discussion in Chapter 2 (Section 2.4), statistical significance is only part of the story; researchers also want to knowhow big the effects of their independent variables are. There are several ways to calculate effect size, but in general, bigger values mean astronger effect. One of these statistics, Cohen’s d (named for Jacob Cohen, 1923–1998), is calculated as the difference between two meansdivided by their pooled standard deviation. The resulting values can therefore be expressed in terms of standard deviations; a d of 1 means thatthe means are one standard deviation apart. How big should we expect our effects to be? Based on his analyses of typical effect sizes in thesocial sciences, Cohen suggested the following benchmarks: d = .20 is a small effect; d = .40 is a moderate effect; and d = .60 is a large effect. Inaddition to these qualitative categories, effect size values can be interpreted in terms of standard deviation units. So, a d of 1 is equivalent to astandard deviation of 1. In other words, a large effect in social and behavioral sciences accounts for a little more than half of a standarddeviation.

In interpreting the results of their testosterone experiment, Newman and colleagues (2005) computed effect size measurements for two of thekey mean comparisons. First, they compared high-testosterone people in the high- and low-status conditions; the size of this effect was d = .78.Second, they compared the high- and low-testosterone people in the low-status condition; the size of this effect was d = .61. Both of theseeffects fall into the “large” range based on Cohen’s benchmarks. More important, taken together with the mean comparisons, they help us tounderstand the way testosterone affects behavior. The authors concluded that cognitive performance stems from an interaction between biology(testosterone) and environment (assigned status) such that high-testosterone people are more responsive to their status in a given situation.When they are placed in a high-status position, they relax and perform well. But when placed in a low-status position, they become distractedand perform poorly. This nuanced conclusion is possible only through an exploration of the data, using mean comparisons and effect sizemeasures.

5.6 Wrap-Up: Avoiding Error

As we wrap up our final chapter, it is worth thinking back to one of the key concepts in Chapter 2: Type I and Type II errors. Regardless of theresearch question, the hypothesis, or the particulars of the research design, all studies have the goal of making accurate decisions about thehypotheses. That is, we need to be able to correctly reject the null hypothesis when it is false and to fail to reject the null when it is true. Butfrom time to time, and despite our best efforts, we make mistakes when we draw conclusions about our hypotheses, as summarized in Table5.7. A Type I error, or “false positive,” involves falsely rejecting a null hypothesis and getting excited about an effect that is the result of chance.A Type II error, or “false negative,” involves failing to reject the null hypothesis and missing an effect that is real and interesting. (If you needmore of a refresher on these terms, refer back to Section 2.4, Hypothesis Testing.)

Table 5.7: Review of Type I and Type II errors
	Researcher’s Decision
	Reject Null	Fail to Reject Null
Null is FALSE	Correct Decision	Type II Error
Null is TRUE	Type I Error	Correct Decision

In this section, we take a problem-solving approach to minimizing both of these errors in an experimental context. It turns out that each error isprimarily under the researcher’s control at different stages in the research process, which means there are different strategies for reducing eacherror.

Avoiding Type I Error

Type I errors occur when results are caused by chance but are mistakenly interpreted as significant. We can generally reduce the odds of thishappening by setting our alpha level at p < .05, meaning that we will get excited only about results that have less than a 5% chance of Type Ierror. However, Type I errors can still occur as a result of using either extremely large samples or large numbers of statistical comparisons. Largesamples can make small effects seem highly significant, so it is important to set a more conservative alpha level in large-scale studies. And, aswe have been discussing in this chapter, the odds of Type I error are compounded with each statistical test we conduct.

This means that Type I error is primarily under our control during statistical analysis—the smarter our statistics, the lower our odds of Type Ierror. We have covered several examples of “smart” statistics in this chapter: Instead of conducting lots of t-tests, we use an ANOVA tosimultaneously test for differences across the entire design. Instead of conducting t-tests to compare means after an ANOVA, we use a mix ofplanned contrasts (for comparisons that we predicted) and post hoc tests (for other comparisons we want to explore). There are also moreadvanced statistical techniques that take this a step further. For example, the multivariate analysis of variance (MANOVA) statistic analyzes setsof dependent variables to further reduce the number of individual tests. This approach is used when dependent variables represent differentmeasures of a related concept, such as using heart rate, blood pressure, and muscle tension to capture the stress response. The MANOVAworks, broadly speaking, by computing a weighted sum of these separate dependent variables (called a canonical variable) and using this newvariable as the dependent variable. If you are interested in learning more about this and other advanced statistical techniques, check out theexcellent volume by James Stevens (2002), Applied Multivariate Statistics for the Social Sciences.

Avoiding Type II Error

Type II errors occur when the results are significant but are mistakenly credited to chance. The primary sources of this error are small samplesand bad design. Small samples may fail to capture enough variability and may therefore lead to nonsignificant p values in testing an otherwisesignificant effect. Both large and small mistakes in experimental designs can add noise to the dataset, making it difficult to detect the realeffects of independent variables.

This means that Type II error is primarily under the researcher’s control during the design process—the smarter the research designs, the lowerthe odds of Type II error. First, as we discussed in Chapter 2, it is relatively simple to estimate the sample size needed for our research using apower calculator. These tools take basic information about the number of conditions in the research design and the estimated size of the effectand then estimate the number of people needed to detect this effect.

Second, as we have discussed in every chapter, it is the experimenter’s responsibility to take steps to minimize extraneous variables that mightinterfere with the hypothesis test. Whether conducting an observation, a survey study, or an experiment, the overall goal is to ensure that thevariables of interest are the main cause of changes in the dependent variable. This is perhaps easiest in an experimental context because thesedesigns are usually conducted in a controlled setting where the experimenter has control over the independent variables. Nonetheless, asdiscussed earlier in this chapter, many factors can threaten the internal validity of an experiment—from confounds to sample bias to expectancyeffects. In essence, the more the researcher can control the influence of these extraneous variables, the more confidence he or she can have inthe results of the hypothesis test.

Table 5.8 presents a summary of the information in this section, listing the primary sources of Type I and Type II errors, as well as the timeperiod when these are under experimenter control.

Table 5.8: Summary—Avoiding error
Error	Definition	Main Source	When Researcher Can Control
Type I	False positive	Lots of tests; lots of people	Conducting statistics
Type II	False negative	Bad measures; not enough people	Designing experiments

5.7 Critiquing a Quantitative Study

As discussed in Chapter 1 (Section 1.5, Searching the Literature) and Chapter 3 (Section 3.3, Critiquing a Qualitative Study), the methods andguidelines for evaluating research studies are fairly detailed for both quantitative and qualitative studies. For any type of study, it is important toread the entire article so that you can determine which parts of the study are valid, which parts can be considered acceptable but with caution,and which parts have significant limitations. As mentioned in Chapter 3, a study does not necessarily have valid and reliable results even if itclaims that it does. Not all studies are worthy of being taken seriously, and it is important that you, as a consumer of psychological and othertypes of research, are able to identify potential problems.

There are several factors to consider when evaluating the sections of a quantitative research study or proposal. The following sections discusshow to critically evaluate the literature review, the purpose statement, the sampling methods, the procedures, the instruments, the results, andthe discussion.

Evaluating the Literature Review Section

Similar to a qualitative study, the literature review section of a quantitative study is meant to provide an adequate rationale for conducting thestudy and to show what difference the study hopes to make in the field. Literature reviews should be thorough and comprehensive and containrelevant research on the topic being studied. They should also be objective, unbiased in the selection of articles being reviewed, and includeprevious research that relates to the current study. Considering these daunting requirements, the following questions, adapted from Houser(2009), can help you to evaluate the literature review:

Do the researchers present an adequate rationale for conducting the study?
What is the significance of the study? What difference will it make to the field?
Is the literature review thorough and comprehensive?
Do the researchers demonstrate any potential biases in the literature review?
Are all important concepts clearly defined by the researchers?
Do the researchers clearly describe previous methods that are relevant to understanding the purpose for conducting this study?

Evaluating the Purpose Statement and Hypotheses

As discussed in Chapter 3, the purpose statement does the following: provides the aim or intent of the study; the focus, methodology,population, and geographical location of the study; the specific area of the research and the central concepts or ideas of the study; justificationfor the method of inquiry (qualitative, quantitative, or mixed); and a description of possible conclusions and their implications. It is generallyfound in the Introduction, as the last paragraph before the Literature Review section. Purpose statements can be written in declarative form orin the form of a question or questions. They should include the type of research methodology and design used, as well as a description of thevariables and population studied. When evaluating the purpose statement, it is important to examine whether the purpose and researchproblem are researchable. Purpose statements and research problems are researchable when the variables of interest can be operationalized, ordefined in a measurable and objective way.

It is important to note that not all researchers use purpose statements in quantitative studies, but instead generally provide only hypotheses. Itis not typical for a researcher to include the null hypothesis, as it can be inferred that every researcher has started out with one. Rather,alternative types of research hypotheses are given. For example, some research will include alternative nondirectional hypotheses. Theseindicate that there is enough evidence from the literature to suggest a difference, but not enough to predict the direction of that difference.Another hypothesis is the alternative directional hypothesis, which suggests a difference and the direction of that difference.

Hypotheses guide researchers in selecting appropriate statistical analyses, so there should be consistency between the hypotheses and statisticaltests chosen. The following questions, adapted from Houser (2009), are provided to help you to evaluate the purpose statement:

Does the article clearly state the purpose statement?
Is the purpose statement clearly based on the argument developed in the literature review?
Are the variables of interest (i.e., independent and dependent) clearly identified in the problem statement?
What type of hypothesis is presented?
Is there a clear rationale for the direction of the hypothesis, based on the literature review?
Is the hypothesis testable and falsifiable?
Is the hypothesis consistent with current theories or research?

Evaluating the Methods Section: Sampling

The Sampling section includes thorough, detailed information about the sample that was chosen and the sampling techniques or methods thatwere used to select it. Descriptions of the sample should include all relevant demographic characteristics (e.g., age, ethnicity, gender) as well asthe size. Unlike qualitative approaches, which do not have hard-and-fast rules regarding sample size, quantitative approaches often require muchlarger samples. As noted in Chapter 4 (Section 4.3), Gay, Mills, and Airasian (2009) provided the following guidelines for determining a sufficientsample size based on the size of the population:

For populations that include 100 or fewer individuals, the entire population should be sampled.
For populations that include 400–600 individuals, 50% of the population should be sampled.
For populations that include 1,500 individuals, 20% of the population should be sampled.
For populations that are 5,000 or larger, about 8% of the population should be sampled. (Note, however, that it is possible to have a samplethat is so large that it does not justify the increased costs, and may produce significant differences that are too small to be meaningful).

The sampling techniques employed should be discussed, including detailed information about how the sample was selected and what methodswere used (e.g., simple random sampling, stratified sampling, cluster sampling). The following are a few of the most common samplingprocedures used in quantitative studies. These are discussed in more detail in Chapter 4, Section 4.3, Sampling From the Population.

Simple Random Sampling: The researcher randomly picks study participants from a list of everyone in the population.
Stratified Random Sampling: The researcher breaks the sampling frame into subgroups and then samples a random number from eachsubgroup.
Cluster Sampling: The researcher randomly selects intact groups.
Systematic Random Sampling: The researcher selects every nth participant from a finite list of those in the population.

Sampling errors and sampling biases should also be considered. Sampling errors can occur when the sample selected is not representative ofthe population. This generally occurs when smaller sample sizes are used. Sampling bias occurs when the researcher purposely selectsparticipants in a particular way that introduces bias into the sample. Convenience samples typically contain sampling bias because the samplesare not usually representative of the entire population.

The following questions, adapted from Houser (2009), can help you evaluate the sampling methods section:

What type of sampling method is used?
Are the sampling procedures consistent with the purpose and hypotheses?
Are relevant demographic characteristics of the sample clearly identified?
Do the methods of sample selection provide a good representative sample, based on the population?
Are there any apparent biases in the selection of the sample?
Is the sample large enough for the study proposed? How has size been justified?
To whom can study results be generalized?

Evaluating the Methods Section: Procedures

The Procedures section describes in detail everything that was conducted in the study. For quantitative studies, it discusses the study variables,how informed consent was obtained, how the participants were assigned to groups, and how the researcher controlled for extraneous orconfounding variables. This section should clearly describe the independent variable as well as give details about whether a control groupexisted, and if so, how it was treated. Additionally, it should cover how the participants were randomly assigned to the experimental group andcontrol group. When evaluating the procedures section, one should assess whether there are any threats to the internal or external validity ofthe study. Furthermore, it is important to examine whether the research design was appropriate for the study and whether it was consistentwith the purpose and research questions. The following questions, adapted from Houser (2009), can assist in the evaluation of the Proceduressection:

What type of research design was used?
Was the research design consistent with the purpose and hypotheses?
Did the researcher provide a detailed description of what was conducted?
Did the researcher introduce any bias in the procedures used?
Was the independent variable clearly defined?
Were there any threats to the internal validity of the study?
Were there any threats to the external validity of the study?

Evaluating the Methods Section: Instruments

The Instruments section describes in detail the types of instruments and measures that were used to collect the data. In quantitative research, avariety of common instruments are used: direct observation and behavioral measures, self-report inventories, ratings of peoples’ behaviors,questionnaires, physiological approaches, interviews, and standardized assessments.

When evaluating the Instruments section, consider whether the instruments were appropriate for the study and the sample and whether therewere any limitations in the types of instruments selected. As noted by Houser (2009), “An important consideration in assessing the objectivityand quality of a dependent measure is the extent to which the procedures and scoring methods are standardized” (p. 167). A test is not highlyreliable or valid simply because it is standardized. Whenever evaluating instruments, it is important to assess the test’s reliability and validity aswell as its use in various populations. It is also important to assess whether the tests selected were consistent with the sample selected, theresearch problem, and the hypotheses. Studies that employ standardized assessments or standardized administration and scoring proceduresgenerally obtain more reliable and valid data collection and results. The following questions, adapted from Houser (2009), can help guide theevaluation of the Instruments section:

Is there a clear and adequate description of the instrument (data collection measures) used?
What types of instruments were used in the study (self-report inventories, ratings, etc.)?
Are the instruments standardized?
What are some of the potential problems or limitations with the types of instruments used?
Does the instrument appear to be appropriate for the sample?
Is the instrument consistent with the hypotheses?
Is there a discussion about the reliability and validity of the instruments used?

Evaluating the Results Section

The Results section describes the findings of the study. Unlike qualitative studies, which focus more on descriptions about the findings,quantitative studies are usually presented mathematically through statistical analyses.

Two forms of statistics are presented in the Results section: descriptive and inferential. Descriptive statistics summarize the data collected fromthe sample, such as demographic information. Inferential statistics use statistical methods or analyses to test hypotheses. When evaluating theResults section, consider whether appropriate statistics were used and whether those statistics were consistent with the purpose of the studyand hypotheses. The following questions, adapted from Houser (2009), will help:

What descriptive statistics were used?
Do the descriptive statistics adequately describe the sample?
What types of inferential statistics were used?
Are the inferential statistics used adequate to answer the hypotheses?
Is the research design consistent with the statistical methods used?

Evaluating the Discussion Section

Whether evaluating a quantitative or qualitative study, the Discussion section summarizes the purpose of the research, as well as the findings’implications for future research and actual practice. Additionally, the Discussion section includes alternative explanations and potentiallimitations of the findings. The following questions, adapted from Houser (2009), can shed light on the Discussion section:

Did the researchers clearly restate the purpose and hypotheses?
Did the researchers clearly discuss the implications of the findings and how they relate to theories, other findings, and actual practice?
Did the researchers provide alternative explanations of the findings?
Has there been appropriate generalization of significant findings beyond the sample to the population?
Did the researchers identify potential limitations of the study and the results?
Did the researchers identify possible directions for future research?

5.8 Mixed-Methods Research Designs

This chapter and the previous three have discussed the differences between qualitative and quantitative research methods. From your readings,it may appear that these methods are totally exclusive and are never mixed or combined in a study. However, this is not actually the case. Inrecent years, researchers have started to utilize both approaches in their designs to provide a more comprehensive understanding of researchproblems. Mixed methods research is fairly new, and the integration and mixing of various qualitative and quantitative approaches can be tricky.As a result, books and articles have been published that provide comprehensive overviews of this type of inquiry.

Mixed-methods research uses the process of triangulation, or the combination of multiple research methods to study a research problem. Inorder to successfully achieve this, several approaches can be employed, including scaling, reliability, and convergent validation. Scaling involvesconverting qualitative data into quantitative or numeric form. This involves creating codes and themes in the qualitative data collected and thencounting the number of times those codes or themes occur (Creswell, 2009). This approach allows researchers to compare the qualitative resultswith the quantitative ones. Reliability involves the use of similar methods to measure the same thing. For example, researchers might utilizeunstructured (qualitative) and structured (quantitative) interviews to examine the experiences of dementia caregivers. Finally, convergentvalidation involves utilizing different but complementary methods (Houser, 2009). For example, a researcher might use a survey to collect dataabout AIDS prevention knowledge and then follow up with an unstructured or semistructured interview. In each of these triangulation methods,the researcher is combining qualitative and quantitative approaches to gain a more thorough picture of the research problem.

Mixed-Methods Strategies

Mixed-methods designs generally utilize one of six mixed strategies or models: sequential explanatory, sequential exploratory, sequentialtransformative, concurrent triangulation, concurrent embedded, or concurrent transformative.

The sequential explanatory strategy is one of the most popular mixed methods used, especially among quantitative researchers. This type ofstrategy includes two phases: The first phase involves the quantitative collection and analysis of data, and the second phase involves thequalitative collection and analysis of data that build on the results of the quantitative data. For example, a researcher might administer astandardized instrument to measure depression symptoms and then follow up with a semistructured interview to explore the participants’personal experiences with depression. In this way, the interview would supply additional information about why the participants respondedcertain ways on the standardized instrument. This type of strategy is generally used to further explain and interpret the quantitative results.

The sequential exploratory strategy is similar to the sequential explanatory strategy except that the phases are switched. In this type ofstrategy, the first phase is the qualitative collection and analysis of data, whereas the second phase is the quantitative collection and analysis ofdata that build upon the qualitative results. Thus, the quantitative results are used to further explain and interpret the qualitative results.

The sequential transformative strategy is similar to the other two sequential strategies in that it includes two phases; however, the sequence ofthe phases is decided by the needs of the researcher. Thus, the researcher selects methods that will best support his or her theoreticalperspective.

The concurrent triangulation strategy involves the researcher collecting both qualitative and quantitative data concurrently and then comparingthem for similarities and differences (Creswell, 2009). Concurrent triangulation is often used to support or disconfirm quantitative results. Similarto concurrent triangulation, the concurrent embedded strategy includes collecting qualitative and quantitative data concurrently; however, aprimary method guides the research, and a secondary method provides a supportive role. The secondary method is given less priority and isembedded or “nested” within the primary method. For example, a primary method might include assessing treatment outcomes (a quantitativeapproach) while a secondary method explores the processes people experience during the study (a qualitative approach) (Creswell, 2009).

The last model is concurrent transformative strategy. It resembles the sequential transformative strategy in that the methods it employs arebased on the theoretical perspective of the researcher, but the data are collected concurrently. And, similar to the concurrent embeddedstrategy, this strategy also utilizes a primary method and a secondary method that is embedded within the primary method.

The following are examples of mixed-methods designs.

Example 1: Sequential Explanatory Design

Celestine Pea (2012) conducted research to examine whether environmental factors (e.g., human, sociocultural, design) affect inquiry-basedteaching. She employed a mixed methods, sequential explanatory design with a multicase research approach to conduct the study. A survey ofmiddle-school science teachers was followed by case studies of three teachers from the survey pool (two females and one male). Here is anexcerpt describing the study’s methods:

Data Collection and Analysis

Quantitative methods: Data collection and analysis. An open-ended Likert-style survey was used to collect quantitative data. Thesurvey included a section for demographics information and individual variables that included gender, years of teaching science,degree, certification in subject taught, and semester credits in science. The survey data were prepared for computation. Then,descriptive statistics, a paired i-test, and analysis of variance (ANOVA) were used to compute the survey data. The outcomes wereanalyzed and interpreted to build a cross-case report.

Qualitative methods: Data collection and analysis. Interviews and observations were used to collect qualitative data. Protocolswere developed for each of these methods. Each teacher was interviewed before, during, and after being observed using a semi-structured approach. An exit interview was also conducted at the end of the observation sessions. The interviews lasted 30–60minutes. (Pea, 2012, pp. 36–37)

Notice how the data collection section is organized into quantitative methods and qualitative methods subsections. This is common practice insequential studies, whose aim is to conduct separate data collections, with the latter elaborating on either the original qualitative or quantitativefindings.

Example 2: Concurrent Embedded Design

Brunetto and Farr-Wharton (2008) conducted a study to . . .

compare the impact of the quality of employee enablers (in this case measured by the quality of the communication relationshipbetween local government employees and supervisors) with the ideal “high quality social capital network” on the quality ofoutputs (such as employees’ level of ambiguity in relation to customers and/or their level of productivity). (p. 37)

The following is an excerpt describing the study’s methods.

Methodology

This study uses mixed methods—hence, both qualitative and quantitative research tools are used (Rocco et al., 2003). However, aconcurrent nested strategy is used in that the primary source of evidence used for the findings is quantitative—in both datacollection and analysis (Cresswell, 2003). Therefore data obtained from qualitative research methods is used only to supporttrends identified using quantitative analysis and aid triangulation, which in turn, increases the external validity and achievesgreater understanding of the phenomenon (Rocco et al., 2003). Quantitative data was obtained using a cross-sectional, survey-based, self-report strategy. While there is some criticism of this method (because of the problems of common method varianceand same source bias), it is in this case, the best way of identifying the intangible values of the organisation by asking publicsector local government employees about their perceptions of the quality of information-sharing. On the other hand, the role ofqualitative research methods (such as interviews) is to generate data that can explain the reasons for the trends evident fromanalyzing the quantitative results (Yin, 1994; Rocco et al., 2003). Once the data was collected, the findings were compared withthe findings of previous research analysed in the literature review, that is, using a pattern-matching logic (Yin, 1994). (Brunetto &Farr-Wharton, 2008, p. 44 [footnotes omitted])

Notice how the data in this concurrent embedded study were not described in separate quantitative and qualitative sections as in the sequentialstudy in Example 1. This is because concurrent studies seek to converge or find similarities among the data and results.

Regardless of the type of strategy, it is important to remember that utilizing both quantitative and qualitative methods can be time-consumingand expensive. However, it also can provide more rigor, validity, and utility to a study. For further information on mixed methods designs, reviewthe Handbook of Mixed Methods in the Social and Behavioral Sciences by Tashakkori and Teddlie (2003) or the Journal of Mixed MethodsResearch.

Summary

In this chapter, we have focused on experimental designs, the last group of specific research designs, and the highest point on the continuum ofcontrol. The primary goal of experimental designs is to explain behavior in terms of cause and effect. We began with an overview ofexperimental terminology and the key features of experiments. Experiments can be distinguished from other research designs by three keyfeatures. First, the researcher manipulates a variable, giving him or her a fair amount of confidence that the independent variable causeschanges in the dependent variable. Second, the researcher controls the environment, ensuring that everything about the experimental context isthe same for different groups of participants—except for the level of the independent variable. Third, the researcher has the power to assignparticipants to conditions, using random assignment. This process helps to ensure that standing differences among participants (e.g., in mood,motivation, intelligence) are balanced out across the experimental conditions.

Next, we covered the concept of experimental validity. In evaluating our experiments, we have to take into account both internal validity (theextent to which the independent variable is the cause of changes in the dependent variable) and external validity (the extent to which theresults can be generalized beyond the specific laboratory setting). Several factors can threaten internal validity, including experimentalconfounds, selection bias, and expectancy effects. The common thread among these threats is that they add noise to the hypothesis test andcast doubt on the direct connection between independent and dependent variable. External validity involves two components: the realism ofthe study and the generalizability of the findings. Psychology experiments are designed to study real-world phenomena, but sometimescompromises have to be made to study these phenomena in the laboratory. The balance is often achieved via mundane realism, or replicatingthe psychological conditions of the real phenomenon. Finally, we have more confidence in the findings of our study when they can be replicated, or repeated in different settings with different measures.

We also discussed the three types of experimental designs: pre-experiments, quasi-experiments, and true experiments. In true experiments,researchers have the most control over extraneous or confounding variables and therefore are more likely to uncover cause-and-effectrelationships. In addition, true experiments always include random assignment and manipulation of the independent variable, whereas quasi-experiments and pre-experiments do not include random assignment.

In designing the nuts and bolts of true experiments, we have to make decisions about both the nature and number of independent variables.First, our designs can be described as between-subject, within-subject, or mixed. In a between-subject design, participants are subjected to onlyone experimental condition and receive only one combination of the independent variables. In a within-subject design, participants findthemselves in all experimental conditions and receive all combinations of the independent variables. A mixed design combines between- andwithin-subject variables. Second, our designs can be described as either one-way or factorial. One-way designs consist of only one independentvariable with at least two levels; factorial designs consist of at least two independent variables, each having at least two levels. In a factorialdesign, there are several results to examine: the main effect of each independent variable plus the interaction, or combination, of theindependent variables.

We also discussed the process of analyzing experimental data, using the analysis of variance (ANOVA) statistic. This test works by simultaneouslycomparing sources of variance and, therefore, avoids the risk of inflated Type I error. The ANOVA (or F) is calculated as a ratio of systematicvariance to error variance, or, more specifically, of between-group variance to within-group variance. The bigger this ratio, the moreexperimental manipulations contribute to overall variability in scores. However, the F statistic only tells us that differences exist in the design,and further analyses are necessary to explore these differences. We walked through an example from a published study, discussing the processof comparing means and calculating effect sizes. In comparing means, we use a mix of planned contrasts (for comparisons that we predicted)and post hoc tests (for other comparisons we want to explore).

We then covered two recurring concepts: Type I error (false positive) and Type II error (false negative) and reviewed ways to minimize sucherrors. These errors interfere with our broad goal of making correct decisions about the status of our hypothesis. Type I errors are inflatedprimarily by large samples and performing lots of statistical analyses. Consequently, these errors are under the experimenter’s control at thedata analysis stage. Type II errors are inflated primarily by small samples and flaws in the experimental design. Consequently, these errors areunder the experimenter’s control at the design and planning stage.

An explanation of how to critique a quantitative study was also provided. Finally, we concluded the chapter with a review on mixed methodsdesigns and the six strategies used to incorporate both qualitative and quantitative techniques. Although utilizing mixed methods approachescan be time-consuming and costly, they can also increase the validity and utility of a study.