Module 3 Assignment 1
Understanding Quantitative Research in Counseling Psychology: An Elusive Endeavor in an Increasingly Diverse Discipline Cheryl Gore-Felton Medical College of Wisconsin This reaction to the three core contributions in the Scientific Forum of the May 2005 issue ofThe Counseling Psychologistdiscusses the strengths of each article, the significance of the findings to the field of counseling psychology, the methodologicallimitations, and the future direction and implications for teaching, research, and practice.
Since the field of counseling psychology was founded in 1946, its focus has expanded. Indeed, the Society of Counseling Psychology contains sec - tions titled Advancement of Women; College and University Counseling Center; Counseling and Psychotherapy Process and Outcome Research; Counseling Health Psychology; Racial and Ethnic Diversity; Independent Practice; Lesbian, Gay, and Bisexual Awareness; Prevention; Positive Psy- chology; and Society for Vocational Psychology (American Psychological Association, 2004). Expanding the knowledge in each section requires the pursuit of scientific inquiry and the need for rigorous scientific methodology to design, implement, and analyze research. Many counseling psychology programs have endorsed this philosophy and train their graduate studentsin the scientist-practitioner model (Chwalisz, 2003). This reaction will concen- trate on the scientist portion of the model, with emphasis on methodological design and statistical analysis using the three core articles in the Scientific Forum as discussion points.
BALANCING METHODOLOGICAL STRENGTHS AND WEAKNESSES Karr and Larson (2005 [this issue]) provide an excellent foundation that supports the need for more theory-driven research in counseling psychology. 367 10.1177/0011000004274129 THE COUNSELING PSYCHOLOGIST / May 2005 Gore-Felton / QUANTITATIVE RESEARCH IN COUNSELING PSYCHOLOGY Correspondence concerning this article should be addressed to Cheryl Gore-Felton, Medical College of Wisconsin, Department of Psychiatry and Behavioral Medicine,2071 North Summit Avenue, Milwaukee, Wisconsin 53202; phone: (414) 456-7700; fax: (414) 287-4209; e-mail:
THE COUNSELING PSYCHOLOGIST, Vol. 33 No. 3, May 2005 367-374 DOI: 10.1177/0011000004274129 © 2005 by the Society of Counseling Psychology Counseling psychologists are in a unique position because they are trained across diverse interest areas to posit and test theories that will move theentire field of psychology forward. A prime example of this unique position is in the area of diversity. During the past decade, counseling psychology has increased its focus toward multiculturalism. TheJournal of Multicultural Counseling & Developmentdemonstrates the need for infusing multicultural aspects into psychology.
Given this focus, it is odd that counseling psychology has not been more proactive in developing models of behavior that not only extend existing models as argued by Karr and Larson (2005 [this issue]) but create new mod - els of psychological functioning and behavior that examine the mediating and moderating effects of culture on behavior and psychological functioning across an individual’s lifespan. An example of models that can be used to inform the dialogue and development of new models within counseling psy - chology can be found in the ecological systems model (Bronfenbrenner, 1989, 1993), transactional model of development (Sameroff & Chandler, 1975), and cultural-ecological theory (Ogbu, 1981). Collectively, these mod- els assert that the environment (e.g., social, cultural, and political) affects thought and behavior across an individual’s lifespan.
Understanding how environmental factors influence psychological func- tioning throughout the life has not been adequately studied. For instance, lit- tle is known about the relationship between cultural factors and psychologi- cal factors such as self-esteem, stress responses, motivation, and resilience.
In the 21st century, we must develop models that explain the rapidly chang- ing global economy and technological advances that are affecting the envi- ronment of diverse populations. Theoretical models that can explain impor- tant contextual factors and their association to psychological well-being and functioning across multicultural, multiethnic groups will increase our knowledge about human behavior and assist in the development of programs and policies that enhance the quality of life among diverse communities.
Be cautious about an all-or-nothing approach to scientific inquiry.
Although theory-driven research is important and necessary to move the field forward, as Karr and Larson (2005 [this issue]) state, other types of scientific inquiry that are applied and atheoretical are also necessary. For instance, some questions can only be answered by examining effects across different studies, which is typically conducted within the context of a meta-analysis or by examining trends (Karr & Larson, 2005 [this issue]). Karr and Larson make the point that an imbalance exists in the type of scientific inquiry being reported in journals that have traditionally published empirical research in counseling psychology. In their study, most of the research they cited from three journals was not based on theory. If these journals represent the field of counseling psychology, this finding is problematic, particularly for those 368 THE COUNSELING PSYCHOLOGIST / May 2005 counseling psychologists who seek to intervene on behaviors to promote health and well-being. Without empirically based studies that are pertinent to the field, counseling psychologists are limited in their ability to use effective methods to improve human functioning among diverse populations across the developmental lifespan—the mission of a counseling psychologist.
Diegelman, Uffelman, Wagner, and Diegelman (2005 [this issue]) nota - bly understand the importance of research productivity among counseling psychology programs. Research productivity is clearly the scholastic bread and butter of any university or college program. How the faculty introduces students to empirical study contributes to new knowledge and improves the ability of society to adopt efficacious programs and policies that enhance the lives of its citizens. However, the Diegelman et al. study raises methodologi - cal concerns that influence their findings and interpretations.
The response rate (34%) to the authors’ e-mail survey of training directors was low. While it is true that the science regarding Internet or e-mail survey methodology is in its infancy, methods can be adapted from mail surveys to enhance return rates (see Dillman, 1978, 1983, 2000; Dodd & Markwiese, 1987; Fox, Crask, & Kim, 1988; Gore-Felton, Koopman, Bridges, Thoresen, & Spiegel, 2002). Researchers have found that follow-up memos reminding respondents to complete the survey and using plain Web surveys rather than surveys that have advanced features and require more memory will increase response rates (Dillman, 2000; Solomon, 2001).
Much research has been conducted on developing paper questionnaires that are respondent friendly and thus reduce measurement error and increase response rates. Similarly, Dillman (2000) has suggested the following princi- ples that can be used when designing Web-based questionnaires: (a) use an introductory screen that is motivational, emphasizes the ease of responding to the questions, and provides instructions on how to proceed to the next page; (b) begin with a question that can be easily comprehended and answered by all respondents; (c) present each question in a conventional for- mat similar to that usually adopted in paper questionnaires; (d) limit theline length of prose to prevent a long line from extending across the respondent’s browser or screen; (e) provide specific instructions on computer actions(e.g., hit enter key, click on the next button, etc.) for responding to the questions; (f) provide computer operation instructions as part of each question where the action must be taken rather than in a separate section (e.g., the beginning of the questionnaire); (g) design the questionnaire so that a question mustbe answered before the respondent can proceed to subsequent questions; (h) construct the questionnaire so that questions scroll from one questionto the other; (i) try to avoid answer choices that exceed more than one screen; (j) include graphical symbols or words that convey a sense of where the respondent is in the completion progress; and (k) attend to question struc- Gore-Felton / QUANTITATIVE RESEARCH IN COUNSELING PSYCHOLOGY 369 tures to prevent known measurement problems (e.g., open-ended or check- all-that-apply questions).
A critical methodological issue that remains in conducting paper and Internet surveys is maximizing the percentage of potential participantswho actually respond by completing the survey. The response rate is critical in survey research for several reasons. First, nonrespondents can introducea source of bias if they differ from respondents in ways that influence their answers (Krathwohl, 1993). Moreover, by increasing the response rate, the sample size increases, which bolsters statistical power, reduces sampling error, and increases generalizability to the larger population. For hypothesis- driven research, increasing the response rate of a study directly bolsters sta - tistical power, thereby reducing the chances of committing a Type II error (e.g., accepting the null hypothesis when the alternative is true; Larsen& Marx, 1990).
The principle of generalizability pertains to external validity of the research design (Krathwohl, 1993). Moreover, generalizability is the extent to which the obtained results can be used to describe the population from which the sample was derived (Krathwohl, 1993). When analyzing survey data from a sample of participants drawn from a specific population, the sam- ple population must represent the larger population for which the researcher wishes to generalize his or her results. Therefore, increasing the response rate increases the researcher’s chance of obtaining a sample that represents the larger population. In the study of Diegelman et al. (2005 [this issue]), the 34% who responded may not represent the pool of counseling psychology training directors, which undermines the validity of responses obtained regarding publication outlets.
The argument of Diegelman et al. (2005 [this issue]) that the journals identified by the respondents is valid because they were identified in a study 10 years earlier (Delgado & Howard, 1994) is misguided because the field of counseling psychology has not remained stagnant during the past 10 years, which could indicate that the outlets in which counseling psychologists pub- lish have changed. This is a good example of how replicating previous stud- ies may not be appropriate because the context in which the phenomenon is being studied has significantly changed.
Martens (2005 [this issue]) underscores the need to use appropriate meth- ods in counseling psychology research. Human behavior is complex in that it is influenced by internal and external factors. The dynamic and complex interaction between these factors requires analytic methods that can explain these relationships in a manner that informs our current theoretical models.
Martens effectively describes the utility of an analytic method that can do just that, namely, structural equation modeling (SEM).
370 THE COUNSELING PSYCHOLOGIST / May 2005 Structural equation modeling is a statistical method that can assist a researcher in understanding latent constructs that encompass theoretical variables. This tool is excellent for counseling psychologists who need to test complex models of behavior in a single analysis. Structural equation model - ing also allows researchers to concentrate on variables or patterns of behav - iors that are contributing the most to a particular phenomenon. Moreover,in increasingly budget-tight environments, SEM can assist program developers in targeting those areas that may benefit most from an intervention.
Although Martens’s (2005 [this issue]) description of SEM and its appro - priateness to counseling psychologists is outstanding, he generalizes his findings to the entire field of counseling psychology based on one journal.
This generalization is similar to the methodological error made by Karr and Larson (2005 [this issue]) and Diegelman et al. (2005 [this issue]) in thata small sample, which may represent the larger population, was used to gener - alize to the entire population (i.e., the field of counseling psychology).
Given the change in counseling psychology and the emergence of more relevant peer-review journals, using traditional outlets may no longer bean appropriate method to accurately depict the scholastic contributions ofcoun- seling psychology. For example, the following journals publish studies using SEM and are appropriate outlets for counseling psychologists:Journal of Youth and Adolescence,Assessment,Journal of Marriage and Family,Psy- chology of Women Quarterly,Health Education Research,American Journal of Family Therapy,Journal of Adolescent Research, andResearch in Higher Education. Note that these are not the only journals in which counseling psy- chologists might publish studies using SEM. However, they well represent peer-reviewed outlets that have recently published studies relevant to the field and that have used SEM to analyze the data.
IMPLICATIONS OF METHODOLOGICAL LIMITATIONS Unless researchers can empirically validate the outlets in which counsel- ing psychologists are publishing, we will be limited in our ability to under- stand quantitative practices in counseling psychology. Karr and Larson (2005 [this issue]), Diegelman et al. (2005 [this issue]), and Martens (2005 [this issue]) illustrate the need to expand our expectations of where counsel- ing psychologists publish and to empirically investigate where the field stands with regard to quantitative research methods and practices. Not to embark on these research questions with a broader view may have deleteri- ous effects on the field. For instance, by following Diegelman et al., a particu- lar program could be considered not viable owing to a lack of research pro- Gore-Felton / QUANTITATIVE RESEARCH IN COUNSELING PSYCHOLOGY 371 ductivity, when in fact the program faculty may be productive and simply publishing in journals that are different from those that were used to deter - mine productivity in the study. This Type II error could result in administra - tive decisions that reduce funding lines or, even worse, discontinue pro - grams. We must realize that our methods matter and affect decisions that are based on our results.
Clearly, counseling psychologists publish in more than four journals.
Moreover, many journals in which counseling psychologists routinely pub - lish may not have been represented in the core articles in the Scientific Forum, yet none of the authors discussed this limitation. One way that future research can avoid this methodological criticism is for researchers to utilize other methods of inquiry to validate the outlets in which counseling psychol - ogists are publishing. For example, qualitative methods such as interviews and focus groups among a cross section of counseling psychologists who conduct research would have bolstered the quantitative methods by either validating the journals that were used in the studies or expanding the poolof journals.
Understanding the strengths and limitations of any scientific method, whether it is quantitative or qualitative, is a necessary first step in generating new knowledge that is valid. Without valid data the interpretations and gener- alizations that are drawn can harm. For example, after reading Karr and Larson (2005 [this issue]) and Martens (2005 [this issue]), one could reason- ably conclude that counseling psychology, as a field, tends to not be theory driven or to use SEM appropriately.
The field of scientific inquiry is rapidly changing with desktop computers and software that can analyze large amounts of data fairly rapidly. Complex questions are being raised that require complex models and subsequent ana- lytic procedures. Many professors and journal reviewers graduated from their doctoral programs long before SEM was routinely taught in advanced statistics courses. However, steps can be taken to address this gap in knowledge.
First, professional organizations must provide continuing education to members at national conferences. Second, universities and colleges must ensure that professors and students have access to the latest scientific models and analytical practices. Third, researchers must develop mechanisms that can easily disseminate methods to those on the front lines of teaching and practice. All of these efforts will increase exposure to complex statistical data analysis methods by teachers, students, and practitioners. This exposure will enhance the likelihood that the number of theoretically complex models will increase and lead to the development of theory-driven research designed to test these models. However, understanding complex techniques and models is insufficient if the methodological design is flawed. How researchers target 372 THE COUNSELING PSYCHOLOGIST / May 2005 and select their population for research needs to be conducted in a manner that reduces selection bias because it dictates the conclusions that can be drawn.
In summary, limitations are inherent in social science research. Resource constraints often dictate the type of research that can be pursued. Therefore, researchers must delineate the limitations of their findings and be cautious when making inferences and generalizations. Quantitative research is an important endeavor for counseling psychologists. However, it may not ade - quately address a specific research question in some instances. Theory, design, statistical analyses, and subsequent conclusions are interrelated in such a manner that a flaw in one summarily affects the others. Teachers, researchers, practitioners, and society benefit from science that effectively integrates all of these components.
REFERENCES American Psychological Association. (2004).Division 17—Society of Counseling Psychology.
Retrieved November 21, 2004, from http://www.apa.org/about/division/div17.html Bronfenbrenner, U. (1989). Ecological systems theory.Annals of Child Development,6, 195- 246.
Bronfenbrenner, U. (1993). The ecology of cognitive development: Research models and fugi- tive findings. In R. H. Wozniak & K. W. Fischer (Eds.),Development in context: Acting and thinking in scientific environments(pp. 3-44). Hillsdale, NJ: Lawrence Erlbaum.
Chwalisz, K. (2003). Evidence-based practice: A framework for twenty-first-century scientist- practitioner training.The Counseling Psychologist,31, 497-528.
Delgado, E. A., & Howard, G. S. (1994). Changes in research productivity incounseling psy- chology: Revisiting Howard (1983) a decade later.Journal of Counseling Psychology,41, 69-73.
Diegelman, N. M., Uffelman, R. A., Wagner, K. S., & Diegelman, S. A. (2005).Current institu- tional trends in research productivity in counseling psychology journals.The Counseling Psychologist,33, 327-339.
Dillman, D. (1978).Mail and telephone surveys: The total design method. New York: John Wiley.
Dillman, D. (1983). Mail and other self-administered questionnaires. InP. H. Rossi, J. D. Wright, & A. B. Anderson (Eds.),Handbook of survey research(pp. 359-377). New York: Academic Press.
Dillman, D. (2000).Mail and Internet surveys: The tailored design method. New York: John Wiley.
Dodd, D. K., & Markwiese, B. J. (1987). Survey response rates as a function of personalized sig- nature on cover letter.Journal of Social Psychology,127, 97-98.
Fox, R. J., Crask, M. R., & Kim, J. (1988). Mail survey response rate: A meta-analysis of selected techniques for inducing response.Public Opinion Quarterly,52, 467-491.
Gore-Felton, C., Koopman, C., Bridges, E., Thoresen, C., & Spiegel, D. (2002). An example of maximizing survey return rates: Methodological issues for health professionals.Evaluation & the Health Professions,25, 152-168. Gore-Felton / QUANTITATIVE RESEARCH IN COUNSELING PSYCHOLOGY 373 Karr, C. A., & Larson, L. M. (2005). Use of theory-driven research in counseling: Investigating three counseling psychology journals from 1990 to 1999.The Counseling Psychologist,33, 299-326.
Krathwohl, D. R. (1993).Methods of educational and social science research: An integrated approach. White Plains, NY: Longman.
Larsen, R. J., & Marx, M. L. (1990).Statistics. Englewood Cliffs, NJ: Prentice Hall.
Martens, M. P. (2005). The use of structural equation modeling in counseling psychology research.The Counseling Psychologist,33, 269-298.
Ogbu, J. U. (1981). Origins of human competence: A cultural-ecological perspective.Child Development,52, 413-429.
Sameroff, A. J., & Chandler, M. J. (1975). Reproductive risk and the continuum of caretaking casualty. In F. D. Horowitz, M. Hetherington, S. Scarr-Salapatek, & G. Siegel (Eds.),Review of child development research: Vol. 4(pp. 187-244). Chicago: University of Chicago Press.
Solomon, D. J. (2001). Conducting Web-based surveys.Practical Assessment, Research & Evalu - ation,7(19). Retrieved December 1, 2004, from http://PAREonline.net/getvn.asp?v=7&n=19 374 THE COUNSELING PSYCHOLOGIST / May 2005
