Research methods unit III Scholarly Activity

Integrating Multiple Qualitative Research Methods (or Avoiding the Precariousness of a One- Legged Stool)* Amy L. Hall The George Washington University Ray C. Rist The World Bank ABSTRACT This article suggests that market research is enhanced when multiple qualitative methods are combined in a triangulated approach to examining marketing questions. The article begins with a case study that illustrates how a qualitative study can, by itself, be used as a basis for making marketing decisions, when methodological triangulation is employed to guard against unreliable or invalid results. Following the case study is a presentation of the three legs of the qualitative stool—interviews (including focus groups), observations, and document analysis. Each method is described, with a discussion of inherent strengths and weaknesses. The article ends with an argument for increasing the use of integrated, triangulated qualitative studies in the field of market research.

© 1999 John Wiley & Sons, Inc.

*The views expressed here are those of the authors, and no endorsement by the World Bank Group is intended or should be inferred.

Psychology & Marketing Vol. 16(4):291-304 (July 1999) ' © 1999 John Wiley & Sons, Inc. CCC 0742-6046/99/040291-14 291 A CASE STUDYi In the early 1990s, a leading bicycle helmet manufacturer, BikeSafe, found itself losing market share in the bicycle helmet industry. BikeSafe had long prided itself on a no-nonsense approach to bicycle safety. The company produced a nigged helmet, shaped for maximum protection and formed of white plastic for greater visibility. BikeSafe sold helmets in sizes suitable for adults and older children. BikeSafe enjoyed a solid reputation in the industry for proAdding a safe product at an affordable price, but now their competitors were doing better while their business was stagnant. What had changed? The CEO of BikeSafe called on his marketing department to provide some answers.

The market researchers at BikeSafe did what many market research- ers do: They organized several focus groups. One focus group consisted of customers who had previously purchased BikeSafe helmets. The sec- ond group was made up of a dozen local high school students and a third group included members of a biking club.

The first group, previous cus- tomers, reported that they were quite satisfied with their bicycle hel- mets and would consider buying another BikeSafe product in the future.

The group of students yielded little information, reporting that bicycle helmets were not "cool" and that they preferred not to wear helmets at all.

The final group, the biking club members, generally focused on safety factors and seemed impressed by the safety rating of BikeSafe's helmets. Their discussion, however, quickly deteriorated into a conver- sation about the group's upcoming ride. In the end, the reseai'chers knew only a little more than when they started, and clearly not enough.

They did learn that the members of the focus group that had previously purchased BikeSafe helmets were generally satisfied, but that did not address the question of why they were not attracting as many new buy- ers.

What went wrong? Why did the focus groups not jdeld better re- sults?

The focus groups conducted by the BikeSafe researchers were subject to some common weaknesses of this qualitative method. One researcher observed that the group consisting of previous purchasers included mostly retired people.

The researchers realized too late that because the focus group was held during a workday morning, they excluded most working-age people. The high school group probably faced the common problem of peer pressure. Once a single, loud student voiced the opinion that it was not acceptable to wear a bicycle helmet, no other student was willing to go against the tide. Finally, the researcher conducting the biking club's focus group was inexperienced with the technique and was not able to keep the discussion focused. Although the focus groups failed to provide answers to BikeSafe's problems, they did yield the im- 'The case study has been created by the authors. It does not refer to actual persons or an existing company.

292 HALL AND RIST portant information that the purchasers participating in the first focus group were generally satisfied. The researchers decided to focus next on how to reach more purchasers.

The researchers decided to reach into the qualitative tool kit for an- other technique. They decided to do some document analysis. BikeSafe included a survey on their warranty cards that approximately 50% of purchasers returned. They began by examining these surveys. What became immediately apparent was that for the last few years, most of their helmets had been purchased by older consumers. They were sell- ing very few helmets to young adults and teenagers. Additionally, the factor most commonly cited for deciding to buy their product as opposed to another brand was the safety reputation of BikeSafe's helmets. The researchers concluded that the company should continue to emphasize the safety features of their helmets, but needed to find a way to recap- ture younger consumers.

Reviewing advertisements in several biking and sports magazines from the previous 6 months revealed some additional answers. The BikeSafe researchers noticed that most helmet ads of their competitors were aimed at the young consumer. The ads often showed young adults engaged in mountain biking and other adventure sports. Previously, BikeSafe had exclusively targeted the biking community and casual bi- cycle riders. Examination of their competitors' ads revealed that they were missing those consumers who purchased a helmet to wear while in-line skating, skateboarding, and playing other dangerous sports.

They also noticed the snazzy colors and sleek designs of their competi- tors' helmets. The BikeSafe researchers had a new idea about the au- dience their own ads should be targeting. But before investing in a new advertising campaign, they wanted to gather additional information.

An important thing the ads could not reveal was how popular these other sports had become and how many new potential consumers this repre- sented. The researchers pulled out that last qualitative tool—observa- tion.

The BikeSafe researchers decided to get a first-hand look at how pop- ular sports such as in-line skating and skateboarding had become.

They chose several parks in the area and on three consecutive weekends made observations about helmet use. They paid particular attention to how many people rode bikes and what types of bikes were ridden, and how many people were skating or skateboarding. Of all these they noted the approximate ages of the people, whether or not they were wearing a helmet, and if a helmet was worn, what it looked like.

At the end of the three weeks, they concluded that their initial impressions from studjdng the ads had been correct; more people were engaging in in-line skating and skateboarding, and the age range of helmet wearers seen most fre- quently was approximately 20-40 years old. Something they concluded from their observations that had not been revealed by either of the other two techniques was that more small children were wearing bicycle hel- INTEGRATING METHODS 293 mets.

BikeSafe had never targeted small children as consumers; they did not even make hicycle helmets to fit small children, hecause children rarely rode hicycles that were hig or fast enough to really cause injury.

Yet it was ohvious from their ohservations that many small chil- dren riding bicycles with their parents were indeed wearing safety helmets.

The researchers finished up their project hy going hack to interviews.

They returned to their focus group lists and invited some of the people from the focus groups in for individual interviews. They made sure to schedule some interviews on weekends so that recent purchasers who worked during the week would he included. The interviewees supported their conclusions that they should expand their marketing to include all age groups and should design a marketing campaign selling their hel- mets as sports helmets, suitable for a variety of sports, rather than simply bicycle helmets. Interviewees also supported the conclusion that consumers want safety with style—colorful helmets with sleek designs.

BikeSafe learned from their research. Within 2 years, BikeSafe had regained their market share of the helmet industry. They had intro- duced a new line of helmets that were designed with safety and style in mind. The helmets were produced in a greater variety of sizes to fit ages "one to ninety-nine," as their new ads stated. An advertising campaign had combined an emphasis on style with the company's already strong reputation for safety. The ads showed people in several age ranges wear- ing BikeSafe helmets while cycling, skating, and skateboarding. There were special ads focused on teenagers.

The company profited from their research. The researchers learned as well. They realized that focus groups alone cannot provide a full, accurate picture of a market situation. Like a one-legged stool, infor- mation gleaned from the initial focus groups would have led to shaky conclusions; in their case, there simply was not enough information for strong conclusions. By adding other research techniques, like adding legs to a stool, they firmed up their conclusions and were on much surer ground when they presented their results to BikeSafe's CEO. The com- pany benefited greatly as a result.

Market researchers have long relied on focus groups as the mainstay of their qualitative techniques. Occasionally other techniques might be used as well, but it is still rare that market researchers rely on a qual- itative study that combines all three of the legs of qualitative research into a triangulated approach.

As the researchers from BikeSafe learned, when the three techniques are combined, they can provide valid, reliable conclusions upon which to base marketing decisions.

The balance of this article aims to (a) demonstrate that there is an array of qualitative methods that can be applied to marketing questions and problems, and (b) outline strengths and weaknesses of each method.

294 HALL AND RIST THE QUALITATIVE TOOL KIT Every good craftsman likes having good tools with which to work. This article presumes that marketing researchers are no different, and sug- gests another look at the many tools available in the qualitative meth- ods tool kit, ones that may not be currently used to tbeir fullest. Mar- keting researchers have long been using a few select qualitative tools, namely, focus groups and some observation methods. But these are largely used in an exploratory fashion and are not often integrated into a multimethod approach to discovery.

For anyone using a new tool, or one with which they are only slightly familiar, it is important to become acquainted with any guidelines per- taining to that particular tool. True, rudimentary tools need only rudi- mentary instructions; but more sophisticated tools need more careful discussion of their use and abuse. No less the case with the careful application of qualitative metbods to marketing research. Furtber, it is important to understand a central reality of qualitative researcb—its strength lies in tbe concurrent use of multiple tools. Tbus, it is not ap- propriate to presume that if one knows how to use one or more tools, one knows bow to conduct qualitative researcb. It is the interactive and simultaneous use of metbods (and an analysis that takes account of tbis interaction) that is unique to qualitative researcb. Stated differently, the key to good qualitative research is metbodological triangulation.

Triangulation is like using a three-legged stool. Remove one leg, and tbe stool is mucb less reliable. It is likely to wobble and collapse under one's weight. Remove a second leg and tbe stool becomes very precarious indeed. So it is witb qualitative research. Tbe three legs of qualitative research methods are interviewing, observation, and document anal- ysis.

With tbe use of a multimetbod approach tbat combines these three major data-collection strategies employed in qualitative research, one bas a basis for drawing conclusions witb strong validity.

Relying on only one or two of tbe methods leaves one subject to questionable validity.

LeCompte and Preissle (1993) note tbat tbe researcher can "pinpoint the accuracy of conclusions drawn by triangulating with several sources of data" (p. 48). Tbe underlying assumption here is straightforward— using multiple data sources can deepen our understanding and bence is advantageous in comparison to using a single method.

The conventional use of the term qualitative methods in marketing has come to imply tbe use of small-group interviews, generally focus groups. Tbe emphasis is on only one leg of the stool. Limiting oneself to focus groups misses the array of other metbods tbat are contained in the qualitative metbods tool kit. Marketing researcb will benefit from an expanded use of qualitative methods in a multimethod approach to learning about market demand, brand loyalty, consumers' preferences INTEGRATING METHODS 295 and needs, consumer behavior, and why it is that what consumers say is often not what they do.

A study that exemplifies the use of these three techniques in con- junction with one another was recently published in the Harvard Busi- ness Review.

In "Making Differences Matter:

A New Paradigm for Man- aging Diversity," Thomas and Ely (1996) examined diversity in the workplace and based their analyses on "interviews, surveys, archival data, and observation" (p.

85). Their use of multiple methods facilitated the exploration of why diversity is so difficult to manage, why it is sel- dom honestly discussed, why the disconnects between values and be- haviors caused dysfunctional performances in the organizations, and why official pronouncements within the organizations were ignored if not outright mocked.

The basis for their development of a new approach to managing diversity was grounded in the multiple qualitative data sources they employed. It is hard to imagine that a sensitive topic like managing diversity could be fully understood with just interviews, or just the review of official documents in the organizations, or just watch- ing behaviors in the cafeteria or in the boardrooms.

THE LOGIC AND RATIONALE OF TRIANGULATION The term triangulation is taken from land surveying, where the sur- veyor uses the sightings of two landmarks to locate his own, the third, position. The logic of triangulation is that no single method ever adequately solves the problem of rival causal factors . . . Because each method reveals different aspects of empir- ical reality, multiple methods of observations must be employed. This is termed triangulation. I now offer as a final methodological rule the principle that multiple methods should be used in every investigation.

(Patton, 1990, p. 187) Denzin (1978) identifies four types of triangulation: data triangula- tion, investigator triangulation, theory triangulation, and methodolog- ical triangulation. Data triangulation refers to the gathering of data at different points in time and from different sources. Investigator trian- gulation is the use of multiple researchers to study the same research question or the same setting, presuming that different researchers will bring different perspectives, thinking, and analysis to the table, thus strengthening the final assessment. Triangulating theory stresses that the research should examine the phenomenon from different theoretical vantage points to see which would be the most robust in helping to clarify and explain what has been studied. Methodological triangulation refers to the use of multiple methods to gain the most complete and detailed data possible on the phenomenon.

296 HALL AND RIST Any strategy of triangulation that adds a system of theoretical/meth- odological checks and halances to a study lends strength to that study.

As Patton (1990) observed, "studies that use only one method are more vulnerable to errors linked to that particular method . . . than studies that use multiple methods in which different types of data provide cross- data validity checks" (p. 188). It is our view that a key means to strengthen market research is to expand the range of methodologies used for the gathering of both factual and perceptual information, and our focus here is on methodological triangulation.

BALANCING STRENGTHS AND WEAKNESSES Each of the three major types of data collection (interviews, observation, and document analysis) has unique strengths and weaknesses. It is in the combination of these strengths and in the compensation for the weaknesses that the intellectual and methodological power of qualita- tive research becomes apparent. In returning to the roots of triangula- tion, the use of multiple methods gives a more precise and accurate fix on the research question than does any one method by itself.

A brief summary of the strengths and weaknesses of the three key methodological approaches suggests why triangulation is so central.

First, the use of focus groups, perhaps the central interviewing tech- nique of the market research community, "allows people to discuss their true feelings, anxieties, and frustrations as well as the depth of their convictions in their own words" (Zikmund, 1982, p. 123). Accurate and truthful responses in these sensitive areas are hard to gather in any research effort, and trjdng to gain them from paper and pencil tests is woefully inadequate. With regard to observation, Aaker, Kumar, and Day (1995) suggested that "observation may be the least expensive and most accurate method of collecting purely behavioral data such as in- store traffic patterns or traffic passing a certain point on a highway system" (p. 189). Observation also allows the researcher to personally see and verify that a particular behavior or interchange did or did not actually happen—independent of the respondent's perception of that event. The third tool, document analysis, can be used to "analyze written material into meaningful units, using carefully applied rules" (Aaker et al., p.

190).

What individuals say is not always what they do.

Thus there is a need to turn to this additional source of information—the written account—which may come in the form of memos, letters, personal bi- ographies, diaries, legal briefs, annual corporate reports, or other doc- uments.

With the strengths of these three methodological approaches, how- ever, come weaknesses. For example, a shortcoming of focus groups is that "without a sensitive and effective moderator, a self-appointed par- ticipant may dominate the session" (Zikmund, 1982, p. 127). Relying INTEGRATING METHODS 297 only on interviewing leaves the researcher vulnerable to selective recall, self-delusion, perceptual distortions, memory loss from the respondent, and subjectivity in the researcher's recording and interpreting of the data. Relying only on the second tool, observation, means that the ac- tor's understanding and motivation for behavior may not be understood by the observer. Inference in this circumstance as to why someone did as they did is risky and often just plain wrong. Document analysis is also subject to error, as it relies solely on the researcher's interpretation of what is in the document being analyzed. Conducting research when one's only source of information is historical documents (pity the poor historian) involves all the problems of selective deposit of the material (not all that is know is written down, let alone then saved), selective survival (remember the 18-minute gap in the Nixon White House tapes), and selective retrieval (not all that survives is found.) INTERVIEWS Interviews can be divided into three basic categories: the individual in- terview, the small-group or focus-group interview, and the large-group interview. Each of these techniques has strengths and liabilities, ben- efits and costs. The emphasis here will be on the first and second of the three types, those of individual and small-group or focus-group inter- viewing, because these are the most applicable to consumer research.

The Individual In-Depth Interview Aaker et al. (1995) described individual in-depth interviews as "inter- views that are conducted face to face with the respondent in which the subject matter of the interview is explored in detail" (p. 176). Calder (1994) noted several advantages of the individual interview, including those related to the amount of in-depth information obtained and the moderator's control of interview timing and related variables. Another advantage of the individual interview is that there is not the potential problem of group conformity. People may be more spontaneous in an individual interview. There is also the benefit that comes from a face- to-face interaction; up-close observation of body language, tone of voice, reaction to distractions, reaction to the interview setting, and personal appearance are all contextual factors that can assist the interviewer in analyzing the interview data. One-on-one interaction may also sensitize the interviewer to the existence of resistance to questions, and efforts to be evasive. There is also the ability to follow up and probe incomplete, vague, or ambiguous responses.

The primary weaknesses of the individual interview technique are logistical in nature. Individual interviews are both time consuming and expensive. A group interview of 20 consumers may require one moder- 298 HALL AND RIST ator and perhaps 2 hours of time.

Individual interviews of the same 20 people would require considerably more resources. In some cases, the information obtained from tbe individual interview may be worth the expense, but often the same information can be obtained through the less expensive foeus-group method.

A second shortcoming of the individual interview is that often the consumer is more comfortable expressing his or her opinions in a sup- portive group environment than in an individual interview. Individual interviews can create a feeling of being isolated or singled out by the interviewer. Another possible shortcoming stems from the possibility of the interviewer and interviewee being in conflict or disliking one an- other. In such cases, it is not likely that the person being interviewed will participate in an open and cooperative fashion.

The Focus Group The focus group is probably the most widely used qualitative technique in marketing research. In fact, "qualitative research is epitomized in marketing practice by the focus group interview" (Calder, 1994, p. 50).

The focus or small-group interview is essentially a data-gathering tech- nique where the interviewer/moderator focuses the attention of a group of 7-12 persons on a deflned set of topics with the intent being for the members of the group to discuss the issues among themselves.

The goal is to learn the views and values of those involved on the topics pre- sented. In general, a group is selected and a moderator begins a discus- sion, allowing the group participants to pick up the flow and, hopefully, express themselves as freely as possible. The focus-group process can alter slightly depending on the speciflc nature of the group.

Focus groups share a number of strengths, including ones related to cost and time. Generally speaking, focus groups are clearly lower on both counts in comparison to individual interviews. Additionally, being in a group compels the individuals to focus on the topic at hand if they are to communicate appropriately in the situation. Another strength of focus-group analysis is that the voice of a number of persons on a topic may lend more credibility to a point of view than that which flows from an equal number of individual interviews. On a related note, the inter- actions in the group can 3deld interesting insights as well.

A report based on a focus group of the phenomenological approach is also likely to de- scribe the group discussion without engaging in extensive technical or scientiflc jargon. One listing of the advantages of focus groups included:

sjTiergism, snowballing, stimulation, security, spontaneity, serendipity, specialization, scientiflc scrutiny, structure, and speed (Zikmund, 1982, pp.

124-126). And although they do not start with the letter s, the mat- ters of cost, time, logistics, and coverage could well be added to Zik- mund's list.

There are, however, three important disadvantages to the focus-group INTEGRATING METHODS 299 methodology. First, the quality of the effort rests heavily on the shoul- ders of the moderator. Second, focus groups present a sampling problem.

How to select 7-12 people, according to what criteria, with what avail- ability, with what previous experience in focus groups, and with what relations to others in tbe group are all matters that can cause tbe dy- namics of tbe group to become dysfunctional and tbe data less tban useful. Tbird, tbere is no ability to generalize from tbe material, nor is it easy to quickly summarize tbe discussions. A 90-minute focus group can easily generate 30 pages of typed notes. Working tbrougb all tbis material to reacb tbe core conclusions is difficult and time consuming.

In addition, using tbe focus group as tbe single source of data can lead to erroneous conclusions.

Clearly, botb focus-group interviews and individual interviews bave strengtbs and weaknesses tbat tbe researcber must weigb against tbe expected results. Wbetber one or botb interview tecbniques are used in a study, tbe researcber sbould consider bow to combine interviews witb metbods from tbe otber two categories presented bere:

observation and document analysis.

OBSERVATION A second major type of qualitative researcb metbodology is observation.

Zikmund (1982) describes two types of observation tecbniques—unob- trusive observation and visible observation. Wben one is performing unobtrusive observation, tbe intent is to collect data on bebavior or in- teractions witbout tbe subject's knowledge. Tbis approacb migbt be most appropriate wben tbe bebavior being observed would make tbe subject self-conscious, tbus leading tbe subject to cbange bis or ber be- bavior. It is also appropriate in tbose situations wbere tbere is no viable alternative. In tbe visible-observation case, tbe subject is aware of tbe observation taking place. Anotber way to cbaracterize observations is participant or nonparticipant. Participant observation requires tbe ob- server to get close enougb to tbose being observed to acbieve a level of comfort witb tbem, tbus allowing unimpeded observations and tbe re- cording of information. For example, a researcber migbt join a consumer rigbts organization to get an insider's look at bow participants feel and interact. In nonparticipant observation, tbe researcber remains apart from tbe subject(s) being studied, in a manner similar to unobtrusive observation. Witb all of tbese metbods, tbe researcber observes ongoing bebavior of tbe subject(s), interpreting meaning from wbat is observed.

Zikmund (1982) suggests six types of bebavior can be observed. First, tbe researcber is able to witness pbysical actions. Do sboppers stop to look at sale ads if tbey are posted at tbe entrance of a store? Verbal and expressive bebavior can also be noted. How do sboppers interact witb store personnel wben a sale item is out of stock? Spatial relations and 300 HALL AND RIST locations can provide important information about consumers, as can temporal patterns. Do posted sale ads get more attention if posted out- side the store, or if more signs are posted in aisles? How much time do shoppers spend examining ads if they are posted outdoors, as opposed to inside the store? The sixth area of observation includes verbal re- cords, or the contents of advertisements. In this article, the latter ob- servational technique is addressed under document analysis.

As with interviewing, observation has both strengths and weaknesses associated with it. Observation can be a relatively inexpensive yet ac- curate means of gathering purely behavioral data. A major strength of the observation method is that because the data from the observations are recorded as they occur, there is less distortion than in interviews, where an important strategy is to encourage recall in the person being interviewed. In an interview, the researcher generally has to rely on the memory and honesty of the respondent. In a group interview, this can be even more critical if a respondent fears the reaction of others in the group. Observation, especially unobtrusive observation, negates this concern. The technique is not without weaknesses, however.

There are several disadvantages associated with observation tech- niques in general. First, behavior may take place out of sight.

A dissat- isfied customer, for example, may not express much emotion while in the store, a public area, but may become very angry upon arriving home.

Because observers can seldom consistently be in the right place at the right time, the observational picture on any phenomenon is always in- complete. Third, the researcher must accurately record the data, with- out injecting subjectivity into the recording. This can be a difficult task.

One strategy for guarding against this in an emotionally charged area is for researchers to put all their own reactions, emotions, and feelings into a separate section at the end of the observational record. Fourth, the researcher must accurately interpret the meaning of the observa- tions.

There are multiple interpretations often possible from even a sim- ple gesture or sigh. Getting the interpretation right is not easy. Partic- ipant observation carries with it the possible disadvantage that subjects may change their behavior as a result of the observation. An ethical consideration that itself could be the subject of another article concerns consent in observational research. Observations and interviews are nat- urally complementary as they allow the researcher to explore the inter- relation of words and deeds. Adding a third technique, document anal- ysis, allows for triangulation and thus still stronger grounds for valid interpretation.

DOCUMENT ANALYSIS "Human behavior often leaves traces, and the study of those traces can tell us a lot" (Bernard, 1995, p. 332). Key among these are the records.

INTEGRATING METHODS 301 documents, letters, newspapers, television programming, books, diaries, and e-mail messages that trace peoples' lives. This study of the traces we leave behind is termed document analysis.

Document analysis aims to take information stored in such sources as noted above and abstract from it key themes, strategies, values, messages, and the like. Coding schemes are frequently used to systematically record and then analyze the collected information.

Document analysis can provide a wealth of information to marketing researchers. Individuals keep records, businesses keep records, orga- nizations keep records, and governments keep records.

Maxine Margolis used advertisements in the Ladies Home Journal to trace "the trans- formation of the middle-class homemaker from an employer of servants to a direct user of household products" (Bernard, p. 341), surely a topic of interest to marketers. Document analysis, like observation, can be inexpensive. It is also unobtrusive, so it is not subject to selective mem- ory or social bias. But not all documents are equal. Some have more credibility and trustworthiness than do others.

Four categories of contemporary (as opposed to historical) documents can be suggested, moving from higher to lower levels of reliability. They are (a) contemporary records, (b) contemporary individual accounts, (c) reconstructed materials, and (d) reports contained in secondary docu- ments. For market researchers, the difficulty comes in that the two cat- egories most often used [(b) and (d)] are vulnerable to distortion and misunderstanding. But as noted, in conjunction with other forms of qualitative data discussed earlier, the use of documents is an integral part of a more comprehensive and thorough understanding of the mo- tivations and beliefs of individuals.

In the first category are contemporary records, such as government legislation and regulations, business and legal papers, office memoran- dums, minutes of meetings, and notes to the file. Materials in this cat- egory are almost always dated, and will state who wrote the document.

They are often written very close to the actual happening of an event.

Contemporary records are often written by one or more participants, for the purpose of keeping track of important issues, ideas, or agreements, so that they may be referred to and used by those same participants in later actions. Many materials in this class are reviewed for their accu- racy by all concerned parties. This process of verification may be quite formal, as in the reading and signing of legal documents, the approval of minutes of meetings, or in the review of important office memoranda before they are issued.

The second class of materials are contemporary individual accounts, such as what one finds in letters, diaries, personal notes from meetings or discussions, or notes in one's personal files. Materials here are gen- erally as contemporary as those in class one, but they are prepared by only one individual without the review, consultation, or approval of oth- 302 HALL AND RIST ers.

Because of the personal, even potentially biased nature of this ma- terial, the credibility is less than class one.

The third class of materials are those that are constructed after the event and separated from that event by some period of time. Recon- structing a complete and sequenced description of the actions, conver- sations, asides, proposals, and so on of even two persons in a conver- sation is difficult; trying to capture a large group discussion or the activities of a board meeting is even more so. There is also the matter that with time, memory recedes and becomes less precise. The old say- ing that "We remember 10 percent of what we read, 20 percent of what we hear, and 30 percent of what we see" may not be precise in the per- centages, but the implications of reduced recollection are real for the trustworthiness of any document prepared some time after the fact.

The fourth class of documents are those that contain elements of con- temporary reports (and not contemporary records as found in class one).

Reports might be those from a state or the national census bureau.

Newspaper accounts, surveys and opinion polls, and investigative ac- counts also fall into this category. Material in this category may be gen- erated on the one hand by statistical sampling (surveys or polls) and on the other by highly individualistic requirements (investigation or news- paper story). In either case, it is hard to know what meaning to make of the information, for it is aggregated, summarized, and sometimes reduced to statistical figures, leaving interpretation to a probability analysis.

It is evident from this discussion that document analysis does have pitfalls, based on the quality of the document being analyzed. But there is another domain of problems that also needs to be considered.

Specif- ically, document analysis invariably involves the coding of the material being examined. Who makes up the code, and how closely it is applied to the material, can be problematic. Coding and analysis are also subject to cultural variability, which can invalidate findings if not considered (Bernard, 1995).

Thus, use of documents in qualitative market research requires careful attention to the source of the document, the intent in using the document, and the coding scheme designed to classify the information within it.

BRINGING TOGETHER INTERVIEWS, OBSERVATIONS, AND DOCUMENT ANALYSIS The market researcher who employs multiple qualitative research methods is the researcher who is taking out an insurance policy on the accuracy of the answer to the research question. The basic premise of this article is not that more research is inherently good. We are not discussing moral precepts.

Rather, the proposition is that a coordinated.

INTEGRATING METHODS 303 triangulated strategy emplo3dng different approaches to the study of a research question can heighten the certainty with which that question is answered. The design and execution of research studies inevitably involve trade-offs among different constraints, for example, scope, cost, time, specificity, level of desired accuracy, and so on.

The emphasis here is that when researchers tend to put all their qualitative methodological eggs in one basket (e.g., focus groups), there are risks. But accuracy and reliability can be enhanced by expanding the range of qualitative re- search methods used in any study. These methods exist and are widely employed in other sectors of qualitative research. Market researchers also have the choice to use these methods or not.

REFERENCES Aaker, D. A., Kumar, V., & Day, G. S. (1995). Marketing research (5th ed.).

New York: John Wiley & Sons.

Bernard, H. R. (1995). Research methods in anthropology: Qualitative and quantitative approaches (2nd ed.). Walnut Creek, CA: AltaMira Press.

Calder, B. J. (1994). Qualitative marketing research. In R. P. Bagozzi (Ed.), Principles of marketing research (pp. 50-72). Cambridge, MA: Basil Black- well Ltd.

Denzin, N. K. (1978). The research act:

A theoretical introduction to sociological methods (2nd ed.). New York: McGraw-Hill Book Company.

Denzin, N. K., & Lincoln, Y.

S. (Eds.) (1994). Handbook of qualitative research.

Thousand Oaks, CA: Sage.

Eisner, E. W. (1991). The enlightened eye: Qualitative inquiry and the en- hancement of educational practice. New York: Macmillan.

Fontana, A., & Frey, J. H. (1994). Interviewing: The art of science. In N. K.

Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 361- 376).

Thousand Oaks, CA: Sage.

LeCompte, M. D., & Preissle, J. (1993). Ethnography and qualitative design in educational research (2nd ed.). New York: Academic Press.

Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.).

Newbury Park, CA: Sage Publications.

Thomas, D. A., & Ely, R. J. (1996, September-October) Making differences matter: A new paradigm for managing diversity. Harvard Business Review, 74, 79-90.

Zikmund, W. G. (1982). Exploring marketing research (4th ed.). Orlando, FL:

Dryden Press.

Correspondence regarding this article should be sent to:

Ray C.

Rist, The World Bank, 1818 H. St. NW, Washington, DC 20433 ([email protected]).

304 HALL AND RIST