EDUC 850 - 02/13 Individual Scholarly Works: Second Two Scholarly Works Assignment Individual Scholarly Works Assignment Instructions Overview The literature review is a process where you synthesize c

3-15 Published by: American Educational Research Association Stable URL: https://www.jstor.org/stable/3699805 JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at https://about.jstor.org/terms American Educational Research Association is collaborating with JSTOR to digitize, preserve and extend access to Educational Researcher This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms Featuresi Scholars Before Researchers: On the Centrality of the Dissertation Literature Review in Research Preparation by David N. Boote and Penny Beile A thorough, sophisticated literature review is the foundation and in- spiration for substantial, useful research. The complex nature of edu- cation research demands such thorough, sophisticated reviews. Although doctoral education is a key means for improving education research, the literature has given short shrift to the dissertation liter- ature review. This article suggests criteria to evaluate the quality of dissertation literature reviews and reports a study that examined dis- sertations at three universities. Acquiring the skills and knowledge required to be education scholars, able to analyze and synthesize the research in a field of specialization, should be the focal, integrative activity of predissertation doctoral education. Such scholarship is a prerequisite for increased methodological sophistication and for im- proving the usefulness of education research. e have all heard the joke before-as we move through graduate school, we learn more and more about less and less until we know everything about nothing. It is expected that someone earning a doctorate has a thorough and sophisticated understanding of an area of research and scholarship. Unfortunately, many doctoral dissertations in education belie the joke, their authors failing to master the liter- ature that is supposed to be the foundation of their research. If their dissertation literature reviews are any indication, many of these now-doctors know bits and pieces of a disorganized topic. Yet we cannot blame them for their failure to demonstrate what we, the education research community, have not clearly articu- lated or valued. Acquiring the skills and knowledge required to be education scholars should be the focal, integrative activity of predissertation doctoral education. Preparing students to analyze and synthesize research in a field of specialization is crucial to understanding ed- ucational ideas. Such preparation is prerequisite to choosing a productive dissertation topic and appropriating fruitful methods of data collection and analysis. In this article, we first argue that a thorough, sophisticated re- view of literature is even more important in education research, with its messy, complex problems, than in most other fields and disciplines. We then argue that current initiatives and faculty fo- cuses have ignored the centrality of the literature review in re- search preparation, in turn weakening the quality of education research. This oversight has its roots, we believe, in a too-narrow conception of the literature review-as merely an exhaustive summary of prior research-and a misunderstanding of its role in research. By building on the extant literature that supports the centrality of the literature review, we offer a practical framework from which to analyze the quality of doctoral dissertation reviews of the literature. We end by further developing our understand- ing of the literature review and indicating some means of im- proving the situation. The Role and Purpose of the Literature Review in Education Research A substantive, thorough, sophisticated literature review is a pre- condition for doing substantive, thorough, sophisticated research. "Good" research is good because it advances our collective under- standing. To advance our collective understanding, a researcher or scholar needs to understand what has been done before, the strengths and weaknesses of existing studies, and what they might mean. A researcher cannot perform significant research without first understanding the literature in the field. Not under- standing the prior research clearly puts a researcher at a disadvan- tage. Shulman argues that generativity-along with discipline, publication, and peer review-is one of the hallmarks of schol- arship (1999, p. 162-163yf + H G H I L Q H V J H Q H U D W L Y L W \ D V W K H D E L O L W y to build on the scholarship and research of those who have come before us. Generativity grants our work integrity and sophistica- tion. To be useful and meaningful, education research must be cumulative; it must build on and learn from prior research and scholarship on the topic. Yet the messy, complicated nature of problems in education makes generativity in education research more difficult than in most other fields and disciplines (Berliner, 2002yf D Q G G H P D Q G s that we develop more sophisticated literature reviews. In tradi- tional disciplinary research, where a researcher is communicating with a well-defined audience about commonly accepted problems and where disciplinary research often is based on a canon of shared knowledge, the researcher's literature review is somewhat easier to construct. However, in education research we are often faced with the challenge of communicating with a diverse audi- ence, and it is very difficult for us to assume shared knowledge, Educational Researcher, Vol. 34, No. 6, pp. 3-15 AUGUST/SEPTEMBER 2005 I This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms methodologies, or even commonly agreed-upon problems (Boote & Gaudelli, 2002yf ) H Z R I X V Z R U N Z L W K L Q D V X E I L H O G R I H G X F D W L R n research that approaches "normal science" (T. S. Kuhn, 1970yf - there are very few clear, cumulative research programs in educa- tion. Because such well-formed research communities are the exception rather than the rule, it is all the more important that novice education researchers learn the craft of constructing a foundation on which their research can be built. As the foundation of any research project, the literature review should accomplish several important objectives. It sets the broad context of the study, clearly demarcates what is and what is not within the scope of the investigation, and justifies those deci- sions. It also situates an existing literature in a broader scholarly and historical context. It should not only report the claims made in the existing literature but also examine critically the research methods used to better understand whether the claims are war- ranted. Such an examination of the literature enables the author to distinguish what has been learned and accomplished in the area of study and what still needs to be learned and accomplished. More- over, this type of review allows the author not only to summarize the existing literature but also to synthesize it in a way that permits a new perspective. Thus a good literature review is the basis of both theoretical and methodological sophistication, thereby improving the quality and usefulness of subsequent research. It is a broadly held assumption that successful doctoral candi- dates need to be "comprehensive and up to date in reviewing the literature" (Barry, 1997yf D Q G W K D W W K H L U G L V V H U W D W L R Q V G H P R Q V W U D W e this prowess. For most education researchers, the doctoral dis- sertation is the capstone to formal academic training and, as such, should be high quality and comprehensive and should re- flect emerging research. The academic community ought to be able to assume that a dissertation literature review indicates a doctoral candidate's ability to locate and evaluate scholarly in- formation and to synthesize research in his or her field. Despite the assumption that dissertation literature reviews are comprehensive and up-to-date, the dirty secret known by those who sit on dissertation committees is that most literature reviews are poorly conceptualized and written. Our secret is made pub- lic by editors and reviewers who openly lament the inadequacy of literature reviews in manuscripts submitted for journal publi- cation (Alton-Lee, 1998; Grant & Graue, 1999; LeCompte, Klingner, Campbell, & Menk, 2003yf ) U R P $ O W R Q / H H V F R P - pilation of reviews of manuscripts submitted to Teaching and Teacher Education, we can begin to see the problems associated with research by scholars who do not know the literature in their fields. For the 58 manuscripts submitted for review over a 1-year period, she identified 369 distinct criticisms in the 142 reviews, which she divided into 13 broad categories. Methodological is- sues were most common, but reviewers also identified theoreti- cal shortcomings (31 timesyf L Q D G H T X D F L H V L Q O L W H U D W X U H U H Y L H w (29yf S D U R F K L D O I R F X V \f, failure to add to the international lit- erature (21yf D Q G I D L O X U H W R O L Q N I L Q G L Q J V W R O L W H U D W X U H \f. In short, it appears that either many of the authors who sub- mit manuscripts to this international journal do not know the literature in their fields or else their knowledge of their fields does not inform the presentation of their manuscripts. Moreover, a better understanding of the research in their field might have aided them with the other methodological problems that the re- viewers identified. We speculate that the shortcomings that edi- tors and reviewers bemoan stem from insufficient preparation in doctoral programs. Methodological training cannot occur in a vacuum, and increased training in research methods alone will not lead to better research. Instead, we must recognize the cen- trality of the literature review in doctoral research preparation and broaden our understanding of what literature reviewing entails. Education Research and Doctoral Preparation There is an emerging consensus that the perceived lack of qual- ity in education research stems from problems with doctoral preparation and that improving doctoral education is key to im- proving education research. Initiatives by several foundations have spawned a small but healthy literature analyzing the prob- lems of doctoral education, describing revised programs, and mak- ing recommendations.' When considering the criteria and standards used to evaluate a dissertation, we need to keep in mind that most people with doctorates in education do not go on to pursue research careers. Most teach, administer, or lead (Passmore, 1980yf < H W D Q \ R Q e earning a doctorate ought to be a steward of the field of educa- tion (Richardson, 2003yf Z L W K D O O W K H U L J K W V D Q G U H V S R Q V L E L O L W L H s thereto appertaining. One of our responsibilities-whether we become a researcher, teacher, administrator, or leader-is to know the literature in our field. And the best avenue for acquir- ing knowledge of the literature (beyond taking courses and com- prehensive examinationsyf L V W K H G L V V H U W D W L R Q O L W H U D W X U H U H Y L H Z . Yet it is apparent that for many, if not most, doctoral candi- dates and dissertation committees, the literature review is of sec- ondary importance. This was not always the case. Until the 19th century, the doctorate was primarily a teaching degree, certify- ing that one had a thorough and sophisticated grasp of a partic- ular field of study. As such, the dissertation and accompanying oral examination served primarily to assess one's suitability as a scholar and a teacher. Graduate education in the United States developed in a period when German universities were ascendant and when "America copied the German version of advanced studies . . . unfortunately the period when the Berlin positivists were in the ascendancy" (Berelson, 1960, p. 12; see also Storr, 1969yf $ V D U H V X O W W K H 8 6 G R F W R U D W H Z D V G H V L J Q H G W R I R F X V R n research training, and the dissertation became a vehicle for demonstrating research prowess.2 Consistent with the assumption that the doctorate is primar- ily for research training, the limited U.S. literature on education doctorates has focused primarily on methodological and episte- mological issues and to a lesser extent on the core and canon of education knowledge. Barger and Duncan (1986yf U D L V H G L I I L F X O t questions about the assumption that doctoral candidates should be expected to do creative scholarly work, and outline what they feel are the psychological, theoretical-methodological, and insti- tutional contexts required for creative work. Based on a collec- tion of anecdotes and experiences, Schoenfeld (1999yf L G H Q W L I L H s a number of difficulties and dilemmas facing doctoral education. Among these are specialization that leads to compartmentaliza- tion, theorizing that leads to superficiality, and simplistic ap- proaches to methodology that hinder a deep understanding of what it means to make and justify a claim about educational phe- nomena. Schoenfeld suggests that many graduates complete their 4~I EDUCATIONAL RESEARCHER This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms degrees unable to identify and frame workable research prob- lems. In a theme issue of Educational Researcher, several authors (Metz, 2001; Page, 2001; Pallas, 2001yf G L V F X V V H G V L P L O D U F K D O - lenges that they had faced in educating doctoral students and methods that they had used in their programs to address those challenges. On the basis of his experience coordinating and teaching in a doctoral program, Labaree (2003yf R X W O L Q H G V R P H R f the general problems facing doctoral education. He framed the problems in terms of a clash between school and university cul- tures that occurs when we ask teachers to shift from a normative to an analytic way of thinking, from a personal to an intellectual relationship with educational phenomena, from a particular to a universal perspective, and from an experimental to a theoretical disposition. An important exception to the emphasis on methodology is Richardson (2003yf Z K R G H Y H O R S V W K H F R Q F H S W R I G R F W R U V R I H G - ucation as stewards of both the field of study and the enterprise of education. She uses this conceptual framework to argue for the knowledge, skills, and dispositions that doctoral programs in ed- ucation should inculcate. The U.S. literature on the education doctorate is reminiscent of the early research on learning to teach (Wideen, Mayer-Smith, & Moon, 1999yf Z L W K O L W W O H R U Q R V X S S R U W I U R P V R O L G G D W D W K e authors rely on their personal prestige to discuss the problems of practice and make recommendations for improvement. But like the literature on learning to teach, the literature on learning to research must move from anecdotes, generalizations, and reports of programs to systematic investigation and recommendations based on evidence. The Literature Review: A Necessary Chore? The perceived lack of importance of the dissertation literature re- view is seen in the paucity of research and publications devoted to understanding it. Doctoral students seeking advice on how to im- prove their literature reviews will find little published guidance worth heeding. Every introductory educational, social, and be- havioral research textbook contains a chapter or section on re- viewing prior research as part of the research process (e.g. Babbie, 1998; Creswell, 2002; Fraenkel & Wallen, 2003; Gay & Airasian, 2000; McMillan & Schumacher, 2001yf 7 K H V H F K D S W H U V W \ S L F D O O y indicate the importance of the literature review, albeit in vague terms, and briefly summarize techniques for searching electronic databases and methods for abstracting prior research. We infer from these chapters and sections that the authors of these text- books acknowledge the importance of the literature review, at least in a salutatory way, but place a much greater emphasis on an understanding of methods of data collection and analysis. In accordance with other textbooks, Creswell (1994yf V X J J H V W s that the literature review should meet three criteria: "to present results of similar studies, to relate the present study to the ongo- ing dialogue in the literature, and to provide a framework for comparing the results of a study with other studies" (p. 37yf 7 o accomplish these criteria Creswell (2002yf U H F R P P H Q G V D I L Y H - step process: "identifying terms to typically use in your literature search; locating literature; reading and checking the relevance of the literature; organizing the literature you have selected; and writing a literature review" (p. 86yf : L W K J X L G H O L Q H V O L N H W K H V H , graduate students could be forgiven for thinking that writing a literature review is no more complicated than writing a high school term paper. These chapters and sections in introductory research text- books are not the extent of the literature, but other sources de- voted to the topic of literature reviewing and dissertation writing are little more sophisticated (e.g., Galvan, 2004; Lester, 2002; Mauch, 1998; Nickerson, 1993; Ogden, 1993; Pan, 2003yf 0 R U e advanced research textbooks and handbooks ignore the subject, focusing entirely on methods of data collection, interpretation, and philosophical issues. In other words, with the very few ex- ceptions noted below, most graduate students receive little or no formal training in how to analyze and synthesize the research lit- erature in their field, and they are unlikely to find it elsewhere. Bruce's (1994yf S K H Q R P H Q R J U D S K L F V W X G \ R I P H W D S K R U V I R U W K e literature review sheds further light on the limitations of pub- lished criteria for reviewing literature. She found that research students perceive themselves in quite diverse metaphorical rela- tionships with the literature in their field, ranging from listing, searching, and surveying to acting as a vehicle for learning, facil- itating, and reporting. Note that the metaphor of vehicle for learning, facilitating, and reporting suggests more sophisticated, developed, and integrated literature reviews. Although a litera- ture review consistent with any of these metaphors can fulfill textbook criteria, candidates may view themselves as anywhere from metaphorically standing aside and haphazardly cataloguing prior findings to critically analyzing and synthesizing the field of study. Bruce's account offers further support for the view that criteria published in popular textbooks are too vague and do not provide clear standards. That doctoral candidates would espouse such naive concep- tions of literature reviewing and perceive it as relatively low in importance would seem to be a product of the culture of doctoral programs in education. Zaporozhetz (1987yf U H S R U W H G W K D W G R F - toral candidates felt their library skills were inadequate, while their faculty advisors admitted expecting their candidates to pos- sess advanced bibliographic skills even though the advisors them- selves had little knowledge of information retrieval. Faculty also ranked the review-of-literature chapter the lowest in importance when considered in relation to the other standard dissertation chapters. Zaporozhetz also reported that most dissertation chairs and students saw the literature review as a relatively routine ac- tivity that doctoral candidates should be able to complete alone with little help from their advisors. And Labaree (2003yf U H P L Q G s us that most doctoral students in education have little formal training in education research and scholarship before they start their doctorate, with their undergraduate and master's degrees usually in other fields or disciplines or focused almost entirely on education practice. Both Zaporozhetz and Labaree note that ed- ucation doctoral students tend to be mature, accomplished pro- fessionals who are committed to improving education practice. Yet these qualities make it more difficult for them to admit that they may lack library search and information synthesis skills and knowledge. We may speculate that, for similar reasons, it is dif- ficult for education faculty to admit to lacking such skills and knowledge-and the skills and knowledge that they can claim in that area probably are tacit and hence difficult to teach. AUGUST/SEPTEMBER2005 IE This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms A product of this doctoral program culture is that the litera- ture review is not valued, and because it is not valued it is rarely an explicit part of doctoral curriculums. Perhaps the conflicting messages and lack of formal training explain why it is not un- heard of for education doctoral candidates to research and write their literature reviews after they have decided on their research problems and methods. This unsystematic approach is not sur- prising when we consider how difficult it is to find a clear artic- ulation of the criteria and standards for quality in a literature review. Students often lack the knowledge and skills even to complete thorough summaries of the existing literature, let alone more sophisticated forms of research synthesis. And, because lit- erature review is not valued, it is also not evaluated. Dissertations pass despite their poor literature reviews, and another generation of education researchers fails to learn that generativity is the core of scholarship-fails to learn what it means to understand and justify an educational idea in a thorough, sophisticated way. Librarians have been aware of these issues for some time and have offered many suggestions for improving the situation (see Libutti & Kopala, 1995, for a reviewyf < H W P D Q \ O L E U D U L D Q V V X I - fer from some of the same naive conceptions about the role of the dissertation literature review as do doctoral candidates and education faculty. Library instruction has tended to focus on the mechanics of database search strategies and on the varieties of in- formation available. Bibliographic skills and knowledge are necessary for ensuring that a researcher can locate and evaluate the available literature, but a literature review should not be understood as merely an ex- haustive summary of prior research. Instead, we need to under- stand that the ability to write a thorough, sophisticated literature review is a form of scholarship requiring a broad range of skills and knowledge-skills and knowledge that we ought to expect of anyone earning a doctorate. The Literature Review: Our Foundation and Inspiration Despite the scant attention paid to literature reviewing in re- search textbooks and programs, a few authors have clearly artic- ulated its centrality in research. Commenting on the importance of reviews, Lather (1999yf D U J X H G W K D W D V \ Q W K H W L F U H Y L H Z V K R X O d serve a critical role in gatekeeping, policing, and leading to new productive work, rather than merely mirroring research in a field. In an editorial in Review ofEducationalResearch, LeCompte and colleagues (2003yf Z U R W H R Q W K H L P S R U W D Q F H R I F R Q Y L Q F L Q J H P H U J - ing scholars that state-of-the-art literature reviews are legitimate and publishable scholarly documents. Too many new scholars believe that empiri- cal research is the only "real" research; they avoid the deep levels of investigation needed to create the kinds of manuscripts sought by RER. This leaves education research without an integrative and critical grounding in prior investigations and weakens subsequent work. (p. 124yf Strike and Posner (1983, pp. 356-357yf I X U W K H U V X J J H V W W K D W a good synthetic review has three characteristics. First, it clarifies and perhaps resolves the problems within a field of study rather than glossing over those problems. Second, it results in a "pro- gressive problem shift" that yields a new perspective on the liter- ature with more explanatory and predictive power than is offered by existing perspectives. Finally, it satisfies the formal criteria of a good theory. Standards such as consistency, parsimony, elegance, and fruitfulness characterize a good synthesis. Lather (1999yf , LeCompte et al. (2003yf D Q G 6 W U L N H D Q G 3 R V Q H U V H H P W R V X J J H V t that reviewing the literature in a field perhaps does require more training than is needed to write a high school term paper and that learning to perform substantive literature reviews should be part of doctoral education. An interesting contrast to the U.S. literature is provided by sev- eral studies from the United Kingdom and Australia, where dis- sertations and research theses are normally adjudicated by outside examiners who submit written reports (Delamont, Atkinson, & Parry, 2000; Johnston, 1997; Nightingale, 1984; Pitkethly & Prosser, 1995; Winter, Griffiths, & Green, 2000yf 1 R W L Q J W K e vagueness and ambiguity of commonly espoused goals for dis- sertations, these studies analyzed the examiners' reports to better understand the criteria and standards of evaluation. Many of these studies indicate that examiners often found problems with literature reviews, although criticisms of methodology were a more serious concern. However, a later study by Mullins and Kiley (2002yf H V W D E - lished a link between methodology and the literature review. They interviewed experienced Australian dissertation examiners and found that examiners typically started reviewing a disserta- tion with the expectation that it would pass; but a poorly con- ceptualized or written literature review often indicated for them that the rest of the dissertation might have problems. On en- countering an inadequate literature review, examiners would proceed to look at the methods of data collection, the analysis, and the conclusions much more carefully. In that way, Mullins and Kiley found that for examiners there was a tacit link between candidates' knowledge of the field and their ability to do sub- stantive, well-justified research. In contrast to the vague suggestions of the U.S. research meth- odology textbooks, Hart (1999yf V X J J H V W V D P X F K P R U H F H Q W U D l role for the literature review in doctoral dissertations. By main- taining that a candidate simply cannot do original or substantial research without a thorough understanding of the field, Hart runs contrary to the assumption that data collection and analy- sis constitute the centerpiece of a doctoral dissertation. In addi- tion to the commonly discussed criteria of summarizing similar studies, linking the dissertation research to ongoing research in the field, and providing a basis for comparing the dissertation findings to prior studies, Hart contends (p. 27yf W K D W D G L V V H U W D - tion literature review should clearly articulate what research needs be done in a field and why it is important, articulate the practical significance of the research, synthesize prior research to gain a new perspective on it, and critically analyze the research methods used.3 He also analyzes the most commonly discussed criteria and subdivides them into more understandable compo- nents. By introducing these new criteria and setting higher ex- pectations, he communicates the importance not only of the literature review in a dissertation but also of the criteria and stan- dards against which it should be judged. It is important to emphasize here that Hart's criteria--sup- ported by Lather (1999yf / H & R P S W H H W D O \f, and Strike and Posner (1983yf J L Y H X V T X L W H D G L I I H U H Q W F R Q F H S W L R Q I U R P W K D t 611 EDUCATIONAL RESEARCHER This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms seen in most published accounts of dissertation literature re- viewing. Hart clearly articulates that doctoral students must be successful scholars-able to critically synthesize ideas and meth- ods in their field-before they are to have any chance of being generative researchers. Contrast this with the most common con- ception, which seems to entail a mechanical process of summa- rizing a supposedly exhaustive collection of prior studies. Piqued by our own experiences with doctoral students' lack of expertise in literature reviewing, we adapted Hart's (1999yf F U L W H U L D W R L Q - vestigate systematically how well doctoral students were learning the skills of scholarship. Standards and Criteria of a Literature Review In our recent study (Boote & Beile, 2004yf Z H X V H G + D U W V \f criteria to develop a framework from which to analyze literature reviews in doctoral dissertations in the field of education. We also sought to understand whether the criteria are reasonable for a dissertation literature review. Hart's criteria were adapted and incorporated into our 12-item scoring rubric, which can be grouped into five categories (see Table 1yf . The first category, "Coverage," consists of a single criterion that was not one of Hart's. Criterion A assessed how well the au- thor of the dissertation justified criteria for inclusion and exclu- sion from review. Cooper (1985yf D U J X H V W K D t coverage is probably the most distinct aspect of literature review- ing. The extent to which reviewers find and include relevant works in their paper is the single activity that sets this expository form apart from all others. How reviewers search the literature and how they make decisions about the suitability and quality of materials involve methods and analytic processes that are unique to this form of scholarship. (p. 12yf Although it is worth noting that Cooper is referring here to lit- erature reviewing as a distinct form of scholarship, we believe that the same expectation should be applied to a literature review that is a precursor to research. We are encouraged in this belief by Cooper's (1985yf R E V H U Y D - tion that there are interesting differences among the ways that authors search the literature and make decisions about suitabil- ity and quality. Relative novices to a topic of the review, mea- sured by the number of previous publications on the topic, tend to be very explicit about their search strategies and criteria and are more likely to use databases and indexes to identify and se- lect research to review. Relative experts, on the other hand, tend to not be as explicit about their search strategies and criteria and often rely on personal communications with leading researchers as their main means of identifying relevant research. We might infer that part of the reason that relative experts on a topic do not need to justify their criteria for inclusion and exclusion is that readers will assume that the well-recognized authors know the literature. Doctoral candidates are novice researchers almost by definition and do not have the luxury of being assumed to know the literature. For that reason we believe that the onus is on doc- toral candidates to convince their readers that they have thor- oughly mined the existing literature and purposefully decided what to review. Hjorland (1988yf S U R Y L G H V D Y L Y L G F D V H V W X G \ R f what happens when a doctoral student lacks sophisticated library search skills; the case study shows the effects on both the origi- nality and the exhaustivity of the resulting dissertation. The dis- sertation that Hjorland analyzed would probably have been ad- equate for most dissertation committees, but the candidate's inability to mine the existing literature led to many erroneous claims about the state of knowledge in the field. Yet library search skills are not enough. Too often, coverage is interpreted by doctoral students as exhaustive coverage of every- thing previously written about their topic (Bruce, 2001 ayf 7 K L s naive approach to searching and selecting prior research can make it very difficult for researchers to critically synthesize the literature in their field, especially when the literature is relatively small or large, or when it is highly fragmented empirically, con- ceptually, or ideologically. Bruce suggests that coverage should be looked at more broadly. He proposes eight criteria: topicality, comprehensiveness, breadth, exclusion, relevance, currency, avail- ability, and authority. Thus, for example, a student reviewing the literature on a topic about which very little has been written may need to broaden the search to examine analogous research in other fields or topics. A student reviewing the literature on a topic about which a great deal has been written may need, in- stead, to focus on the best available evidence or on a smaller number of key conceptual pieces. Whatever the strategy adopted, the burden is on the doctoral candidate to convince the audience that inclusion has been purposeful and thorough. Criterion A is included in our rubric to measure the degree to which selection criteria are clearly justified in the dissertation. The second category, "Synthesis," consists of Criteria B through G and is designed to gauge how well the author summarized, analyzed, and synthesized the selected literature on a topic. The individual criteria ask how well the author (Byf G L V W L Q J X L V K H G Z K D t has been done in the field from what needs to be done, (Cyf S O D F H d the topic or problem in the broader scholarly literature, (Dyf S O D F H d the research in the historical context of the field, (Eyf D F T X L U H G D Q d enhanced the subject vocabulary, (Fyf D U W L F X O D W H G L P S R U W D Q W Y D U L - ables and phenomena relevant to the topic, and (Gyf V \ Q W K H V L ] H d and gained a new perspective on the literature. As Lather (1999yf and Strike and Posner (1983yf V X J J H V W W K L V H Q G H D Y R U V K R X O G H Q - able the author to synthesize the literature, gain a new perspec- tive on it, and clarify what has been done and still needs to be done. Such a synthesis enables the dissertation author to clarify and resolve inconsistencies and tensions in the literature and thereby make a genuine contribution to the state of knowledge in the field, by developing theories with more explanatory and predictive power, clarifying the scope and limitations of ideas, posing fruitful empirical investigations, and/or identifying and pursuing unresolved problems. This kind of theorizing is central to our conception of what it means to earn a doctoral degree. Criteria H and I constitute the third category, "Methodology." Criterion H measures how well the author identified the main methodologies and research techniques that have been used in the field, and analyzed their advantages and disadvantages.4 Criterion I evaluates how well the author's literature review related ideas and theories to research methodologies (a criterion not included in Hart's [1999] listyf $ W P L Q L P X P D Q D X W K R U V K R X O G U H F R J Q L ] H K R w previous researchers' methodological choices affected the research findings. Any sophisticated review of literature should also con- sider the research methods used in that literature and consider the strengths and weaknesses of those research methods in relation the AUGUST/SEPTEMBER 2005 7 This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms Table 1 Literature Review Scoring Rubric Category Criterion 1 2 3 4 1. Coverage A. Justified criteria for inclusion and Did not discuss the Discussed the literature Justified inclusion and exclusion exclusion from review. criteria inclusion or included and of literature exclusion excluded 2. Synthesis B. Distinguished what has been Did not distinguish Discussed what has and Critically examined the state of done in the field from what needs what has and has has not been done the field to be done. not been done C. Placed the topic or problem in the Topic not placed in Some discussion of Topic clearly situated in broader broader scholarly literature broader scholarly broader scholarly scholarly literature literature literature D. Placed the research in the histori- History of topic not Some mention of his- Critically examined history cal context of the field. discussed tory of topic of topic E. Acquired and enhanced the sub- Key vocabulary not Key vocabulary defined Discussed and resolved ambigui- ject vocabulary. discussed ties in definitions F. Articulated important variables Key variables and Reviewed relationships Noted ambiguities in literature and phenomena relevant to phenomena not among key variables and proposed new relationships the topic. discussed and phenomena G. Synthesized and gained a new Accepted literature at Some critique of Offered new perspective perspective on the literature, face value literature 3. Methodology H. Identified the main methodolo- Research methods Some discussion of re- Critiqued research methods Introduced new methods gies and research techniques not discussed search methods used to address problems that have been used in the field, to produce claims with predominant and their advantages and methods disadvantages. I. Related ideas and theories in the Research methods Some discussion of ap- Critiqued appropriateness of field to research methodologies, not discussed propriateness of re- research methods to warrant search methods to claims warrant claims 4. Significance J. Rationalized the practical signifi- Practical significance Practical significance Critiqued practical significance of cance of the research problem. of research not discussed research discussed K. Rationalized the scholarly Scholarly significance Scholarly significance Critiqued scholarly significance of significance of the research of research not discussed research problem. discussed 5. Rhetoric L. Was written with a coherent, Poorly conceptual- Some coherent Well developed, coherent clear structure that supported the ized, haphazard structure review. Note: The column-head numbers represent scores for rating dissertation literature reviews on 3-point and 4-point scales (endnote 4 explains our choice of the two types of scalesyf $ G D S W H G I U R P ' R L Q J a Literature Review: Releasing the Social Science Research Imagination (p. 27yf E \ & K U L V W R S K H U + D U W / R Q G R Q 6 $ * ( 3 X E O L F D W L R Q V & R S \ U L J K W E \ 6 $ * ( 3 X E O L F D W L R Q V $ G D S W H G Z L W K S H U P L V V L R Q . This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms state of the field. In many cases, the body of literature on a topic is limited by the research methods used and advances within the field can be traced back to increased methodological sophistica- tion. Very sophisticated literature reviews might recognize the methodological weaknesses of a field of study and propose new methodologies to compensate for those weaknesses. Criteria H and I measured how well the doctoral candidates identified the main methodologies and research techniques used in the field, analyzed their advantages and disadvantages, and related the ideas and theories in the field to the research methodologies. To score well, the candidates could justify their own methodologi- cal choices and perhaps even suggest and justify new research methods. The fourth category, "Significance," includes Criteria J and K, which measure how well the dissertation rationalized the practi- cal (Jyf D Q G V F K R O D U O \ . \f significance of the research problem. We would expect that, at minimum, a dissertation should discuss both the scholarly and the practical implications of the existing research on a topic and, preferably, note any ambiguities or short- comings in the literature. Some dissertations clearly are more scholarly in their orientation and others are more practical, but we prefer that any dissertation explain both the practical and the scholarly significance and limitations of prior research on the topic (Richardson, 2003yf 7 K L V H [ S H F W D W L R Q D F N Q R Z O H G J H V W K H L P - portance of linking research and practice in the field of education. The final category, "Rhetoric," also consists of a single item, Criterion L, which measures whether the literature review was written with a coherent, clear structure that supported the re- view. This criterion, too, was not included in Hart's list, but it emerged as an important one as we read through our sample of dissertations. Once an author has summarized, analyzed, and synthesized the literature, he or she will want to make some claims about that literature. Those claims should be articulated clearly, and the writing should be organized to support them. This may seem like an obvious point, but our experience even before undertaking this research gave us many examples of liter- ature reviews that lacked rhetorical structure and were very poorly written. Cooper's (1985yf V W X G \ R I J U D G X D W H V W X G H Q W V U H D G L Q J O L W - erature reviews reported that rhetorical structure and organiza- tion were key determinants in how influential and persuasive readers believed the review to be. More generally, Granello (2001yf K D V D U J X H G W K D W I R F X V L Q J R n the formal aspects ofwriting is a means of increasing students' cog- nitive complexity, moving students from lower to higher levels of Bloom's taxonomy of the cognitive domain (see also Libutti & Kopala, 1995yf + D Y L Q J W R R U J D Q L ] H R Q H V W K R X J K W V L V L P S R U W D Q W Q R t merely for persuading an audience but for better understanding what one is writing (Klein, 1999; D. Kuhn, 1992; Rivard, 1994yf . Criterion L measures how well the dissertation articulated clear claims based on its analysis and synthesis of the literature and sup- ported those claims through purposeful organization and cogent writing. Taken together, these twelve criteria and associated stan- dards set ambitious expectations for doctoral dissertation liter- ature reviews. A literature review that meets high standards on these criteria indicates that the doctoral candidate has a thor- ough, sophisticated understanding of a field of study-a pre- condition for substantial, useful research. Literature Review Analysis Findings After developing our rubric, we initially examined 30 dissertations awarded in the year 2000 from three state-funded colleges of ed- ucation in the United States; we selected 12 of those dissertations for full analysis. The three colleges all offered doctoral degrees in addition to significant involvement with preservice teacher edu- cation, had similar rates of acceptance to their graduate pro- grams, and had a comparable number of faculty members. One college was ranked by US News 6& World Report among the top 15 U.S. colleges of education; one was among the top 30; and one was not ranked. Although we are cautious regarding the methods used by US News & World Report to rank schools, our sample represents something of the diversity among state-funded education doctoral programs. From these schools we chose a stratified random sample of thirty dissertations representing the general topics of education leadership, educational psychology, instructional or learning theory, and teacher education. Our analy- sis of dissertation literature reviews supports Schoenfeld's (1999yf contention that doctoral students may not be learning what it means to make and justify educational claims.5 Our findings raise interesting questions about doctoral candi- dates' ability to write a thorough and sophisticated literature re- view and what doctoral dissertation committees will accept as adequate. Although our analysis of scores revealed differences in quality of dissertation literature reviews among the institutions, mean scores across all the institutions were surprisingly low. These results must be interpreted cautiously because of our small sample, but we cannot deny that the worst literature reviews we analyzed were mere disjointed summaries of a haphazard collec- tion of literature. We developed a very clear sense that for many of these doctoral candidates, reviewing prior research on their topic was a hollow exercise from which they learned nothing of substance and which contributed little to their understanding of their research project. Moreover, the common failure to synthe- size literature, critique methodology, or explain scholarly signif- icance supported Schoenfeld's (1999yf D V V H U W L R Q 6 X F K O D F N R f sophistication does not bode well for the students' ability to stay abreast of research in their field as teachers, administrators or leaders-let alone to lead productive research careers after re- ceiving their doctorates. That said, we are happy to report that the best literature re- views were thorough, critical examinations of the state of the field that set the stage for the authors' substantive research proj- ects. These high-quality reviews lead us to believe that our cri- teria and standards are not unreasonable. They also support Hart's (1999yf F O D L P W K D W W K H F U L W H U L D D U H L P S R U W D Q W I R U G R F W R U D l students' understanding of their field and prerequisite for fram- ing fruitful research problems and appropriating sophisticated research methods (Richardson, 2003; Schoenfeld, 1999yf . Perhaps even more remarkable than the differences among in- stitutions is the range of scores and amount of variation within each institution. The variance within all three institutions im- plies that literature reviews were not held to consistent criteria or standards, or that standards for acceptable literature reviews were of little or no importance. We were not surprised by this finding, considering that we suspect that Zaporozhetz' (1987yf U H S R U W L s generally true and that most faculty supervising doctoral disser- tations do not value the role of reviewing literature in a research AUGUST/SEPTEMBER2005 IE This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms project. We are inclined to attribute the differences to faculty expertise and effectiveness at communicating scholarship ex- pectations, and we can only infer that some faculty have higher, albeit uncodified, minimal standards for acceptable work than do others. The assumption that all doctoral candidates are on the cutting edge of current research in their field was not well supported by our study, nor can our study justify the assumption that all doc- toral candidates have learned to critically analyze and synthesize research in their field. Although we must still exercise caution with our small sample from a limited number of schools, our re- sults suggest that not everyone with a doctorate in education un- derstands the norms of scholarly communication or the processes of warranting scholarly claims. The existing literature and our analysis of dissertation literature reviews suggests that the crite- ria and standards for a high-quality literature review are not part of the formal curriculum or graduation expectations of even na- tionally ranked doctoral programs. Doctoral faculty and programs must pay more attention to explicitly teaching and assessing stu- dents on the norms and methods of scholarship and scholarly communication. Refining Our Conception of Literature Reviewing The primary purpose of this article is to highlight the general weakness of dissertation literature reviews and, in so doing, to argue their centrality in preparing doctoral candidates to be bet- ter scholars and researchers. We developed our rubric as a re- search tool to assess suggested criteria but have discovered that many graduate faculty members and graduate students seem more interested in using it as a pedagogical tool to teach or learn literature review skills. As we hear from these colleagues, how- ever, it becomes clear to us that education researchers have quite varied beliefs about the literature review and its role in learning how to do research. One of the most common concerns raised about our research is whether the criteria and standards that we have developed should apply to the two types of doctoral degrees in education and to various dissertation formats. First, should a literature re- view from an Ed.D. dissertation and a literature review from a Ph.D. dissertation be assessed by means of the same criteria? Some might say, for example, that an Ed.D. dissertation should be more concerned with the practical implications of research, whereas a Ph.D. dissertation should be more concerned with its scholarly importance. Although there is much debate about the role and purpose of each degree, we take the position that any- one earning a doctorate in education ought to know the literature in his or her area of specialization-indeed, it is quite unclear to us what, exactly, earning a doctorate might signify if one does not know the literature in one's field. For that reason, we did not dif- ferentiate between Ed.D. and Ph.D. degrees in our study, and we do not believe that the corresponding literature reviews should be held to different standards and criteria. The second concern often raised is whether our criteria ought to apply to the various formats of dissertations. The concern here seems to be that our criteria may inadvertently valorize the tra- ditional five-chapter, empirical dissertation and may be inap- propriate to apply to other dissertation formats. This certainly was not our intent, nor do we think it will be the effect. Paltridge (2002yf G L I I H U H Q W L D W H V D P R Q J I R X U J H Q H U D O G L V V H U W D W L R n formats, each of which is seen among education dissertations. The traditional simple dissertation presents a single study in five chapters: Introduction, Literature Review, Methodology, Results, and Conclusions. Twenty-nine of the 30 dissertations that we sampled used this format. What Paltridge (2002yf F D O O V W K H W U D G L W L R Q D O F R P S O H [ I R U P D W S U H - sents several related studies, each presenting its own introduction, methods, results, and conclusions. The traditional complex dis- sertations that he found tended to use a single literature review for all of the studies. We did not have any dissertations of this type in our sample, but we are aware that they are often used in the field of education, especially for behavioral-science-influenced topics. The topic-based dissertation is also often used in education, especially for theoretical, philosophical, humanities-based, and qualitative dissertations (Paltridge, 2002yf , Q W R S L F E D V H G G L V V H U - tations, authors divide the larger work into chapters that best support the rhetorical structure and often do not use separate chapters for the literature review, methodology, results, or con- clusions. Our sample contained one such dissertation, a narra- tive of a professional development collaboration in which the literature reviewed was interspersed throughout the dissertation. We had no difficulty applying the rubric to this format; indeed, it scored fairly well against our criteria and standards. Finally, the compilation of research articles format for disserta- tions, advocated in these pages by Duke and Beck (1999yf S U H - sents a number of discrete articles often written in the format of journal articles, framed with introductory and concluding sec- tions. Each article must be complete unto itself, including its own literature review. Although this kind of dissertation has been of increasing popularity in the sciences (Dong, 1998yf Z H D U H Q R t certain how often it is used in education. One notable difference between article compilations and other dissertation formats is that dissertations in the compilation format tend to be written "more as 'experts writing for experts', than novices 'writing for admission to the academy"' (Paltridge, 2002, p. 132yf $ V V X F K , they contain much less writing that seems to serve the purpose of merely displaying the author's knowledge. Nothing in our arguments or analysis is intended to justify the all-too-common practice, in a five-chapter, traditional simple dis- sertation, of isolating the literature review in the second chapter. A doctoral candidate who has a thorough, sophisticated under- standing of the literature should clearly be expected to demonstrate an understanding throughout the dissertation, from introduction through conclusion. We can only speculate that the tendency to isolate the literature review reflects doctoral candidates' less-than- thorough understanding of their literature and inability to see how the literature should influence their choice and justification of research topic, choice and justification of methods of data col- lection and analysis, and discussion of the findings, conclusions, and implications. The choice of format for a dissertation ought to fit its rhetorical structure, including the decision whether to write the literature review as a stand-alone chapter or to integrate it throughout the dissertation. Whatever format the author chooses, a thorough, sophisticated review ought to be influential and ev- ident in the entire dissertation. El EDUCATIONAL RESEARCHER This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms We would expect to find significant differences among the dis- sertation literature reviews according to format, including pre- sentation, format, degree of integration, and authorial voice. We can see no reason, however, to suggest that the various disserta- tion formats ought to be judged against different criteria in their literature reviews. A dissertation of any format should demon- strate that the author thoroughly understands the literature in his or her area of specialization. The fact that 29 of the 30 disserta- tions we examined were of the traditional-simple format made our data collection easier but is not relevant to the application of the criteria. A related concern seems to originate from the practice, perhaps common among students, of writing literature reviews as part of dissertation proposals and then using the same literature in the dissertation with little revision. Thus the literature review be- comes a static artifact rather than a dynamic part of the entire dis- sertation. In contrast, we would normally expect candidates to continually revisit their understanding of the literature through- out the dissertation experience. This might mean rereading the literature in light of subsequent findings or analysis, or reading new literature to address emerging findings or ideas. Without viewing the literature review as a dynamic, integral part of the re- search process, we are much more likely to find the problems that Alton-Lee (1998yf L G H Q W L I L H G L Q W K H V X E P L V V L R Q V W R 7 H D F K L Q J D Q d Teacher Education. Several critics have also raised concerns about Criterion A, "Justified criteria for inclusion and exclusion from review," not- ing that it was not stipulated by Hart (1999yf D Q G W K D W H Y H Q W K e best of the dissertations that we analyzed did not score well on this criterion. Of all of the criteria we used in our study, we suspect that this will be most contentious. Our decision to include this criterion was based on our experience working with dissertation students and noting how haphazardly many approached the lit- erature search. We also are aware that review journals increasingly expect authors to describe explicitly how they identified research to indicate the conditions for inclusion and exclusion, implicitly following Cooper's (1985yf U H F R P P H Q G D W L R Q V : H F R Q W L Q X H W o believe that it is important for all researchers, especially novices, to begin to take this more methodical approach to literature re- views. We also contend that its use holds the most potential for improving the quality of dissertation literature reviews, as it forces candidates to be more methodical. Other critics have worried that providing detailed criteria to evaluate the quality of a literature review will lead to yet more dissertations that are formulaic.6 This outcome seems unlikely to us, but we acknowledge that clear criteria alone will never lead to better scholarship. As Bargar and Duncan (1986yf Z U L W H D W K R U - ough understanding and sincere commitment to problems of im- portance can and very often do lead to pedestrian, unimaginative solutions" (p. 35yf / H V V V X F F H V V I X O U H V H D U F K H U V K D Y H S H U K D S V Q H Y H r learned to develop productive research questions because they have superficial understanding of the problems of their field, they tend to follow unproductive habits learned in their dissertation research projects, or they misunderstand the changing norms and expectations of their research community. We need to stress that a good literature review is necessary but not sufficient for good research. A good review of the literature cannot guarantee either a rigorous study or significant findings. Just because authors understands the research that others have done does not mean that they will necessarily be able to collect, analyze, or interpret data well. It certainly does not mean that their interpretation of prior research in the field will lead them to focus on research problems that will yield significant and im- portant studies. Of course, even having a significant insight into the literature in a field does not guarantee that researches will then be able to do significant research on the topic. But, again, it is unlikely if not impossible to do significant research without productive insight into the field. It is productive insight that distinguishes a synthetic review, in Lather's (1999yf V H Q V H I U R P W K H S O R G G L Q J U H V H D U F K V X P P D U L H s that characterize most dissertations. Productive insight can never be routine. But we contend that requiring doctoral candidates to engage in substantive, thorough, sophisticated literature reviews creates and fosters conditions that will greatly increase the likeli- hood of their developing productive insight. Looking Forward Doctoral students must be scholars before they are researchers. First and foremost, a dissertation should demonstrate a thorough and sophisticated grasp of one's field of study; secondarily and antecedently, it should demonstrate the ability to do research that advances the collective understanding of important educa- tion issues. Education research is difficult because of the complex nature of the phenomena studied. In the face of perennial con- cerns about the quality of education research and contemporary pressures to reform it, U.S. education research journals have em- phasized methods of data collection and analysis and related issues of epistemology. In turn, the emerging literature on preparing doc- toral students in education has emphasized methodological so- phistication as the key to improving education research. Yet to try to improve education research by focusing on methodologi- cal sophistication is to put the cart before the horse. Researchers cannot appropriate sophisticated research methods if their understanding of the phenomena they are investigating is rudimentary and unsystematic. To be able to identify workable and potentially important research problems (Richardson, 2003; Schoenfeld, 1999yf W K H \ P X V W E H D E O H W R V K L I W W K H S U R E O H P W R I L Q d perspectives that are progressively more explanatory and insight- ful; they must become more sophisticated theoretically without being superficial (Strike & Posner, 1983yf 7 K L V U H T X L U H V P X F h more than the mere summaries of existing literature that we found in most of the dissertations we analyzed. Doctoral students should be expected to move through Bloom's stages of cognitive development, from comprehending to applying, to analyzing, and on to synthesizing and evaluating (Granello, 2001yf 0 R U H R Y H U , simply expecting students to meet these criteria and standards is not enough-the skills required must be taught explicitly. Yet the most obvious means of improving the situation- adding a class on literature reviewing to doctoral programs-is the least likely to be effective. Such a curricular solution, as Britzman (1991yf Q R W H G D E R X W O H D U Q L Q J W R W H D F K Z R X O G O H D Y H W K H K D U G H V t task-integrating and applying the lessons from various classes in the doctoral program--to those who are least capable of doing it. That is, to review the literature in the way that we have sug- gested here is a very complex task that requires the integration and application of a variety of skills and knowledge that few in- dividual faculty members have mastered. AUGUST/SEPTEMBER2005 IZ This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms For example, this approach to literature reviewing requires ad- vanced bibliographic methods for searching and locating re- search from a variety of sources, skills usually associated with library instruction and the traditional domain of librarians. And, indeed, there is a healthy literature on the importance of doctoral faculty teaming with librarians, because neither alone typically have the skills and knowledge needed (Bailey, 1985; Bruce, 2001b; Caspers & Lenn, 2000; Isbell & Broadsus, 1995; Libutti & Kopala, 1995yf . Yet even this approach is inadequate, because to do the kind of critical synthesis that we suggest also requires diverse skills on the part of doctoral faculty. These include a substantive under- standing of the topic being reviewed, the skills and knowledge required to critically evaluate and synthesize concepts, advanced understanding of writing and rhetoric, and the sophisticated un- derstanding of research methodology that is required to critically evaluate methods used in prior studies and suggest means of overcoming prior methodological limitations. For reasons like these, we suggest that a stand-alone class in literature reviewing is inadequate. Rather, literature reviewing should be a central focus of predissertation coursework, inte- grated throughout the program. Demonstrating the scholarly abilities required for good literature reviewing ought to be a pre- requisite for passing into candidacy. Such a process will require the joint efforts of not only subject experts but also librarians, writing teachers, methodologists, and perhaps others from across campus. The current interest in improving doctoral education and ed- ucation research coincides with changes in instruction on infor- mation access and use. Only recently have libraries redefined library instruction, with its traditional emphasis on mechanical searching skills, to include information literacy, which employs a more conceptual approach to information use. The Association of College and Research Libraries (2000yf G H I L Q H V L Q I R U P D W L R Q O L W - eracy as an intellectual framework for understanding, finding, evaluating, and using information-activities which may be accomplished in part by fluency with information technology, in part by sound in- vestigative methods, but most importantly, through critical dis- cernment and reasoning. (pp. 3-4yf The new focus of libraries on teaching students to critically en- gage with information offers the possibility of successful fac- ulty-librarian collaboration, especially in the realm of graduate literature reviewing and writing. Green and Bowser (2003yf G H - scribe an initiative whereby faculty and librarians guided stu- dents in determining appropriate study subtopics, organizing literature reviews, evaluating resources, and establishing relation- ships with the literature (for other examples, see Bruce, 2001; Caspers & Lenn, 2000; Heller-Ross, 1996; Stein & Lamb, 1998; Wright, 2000yf . Taking the idea a step further, Isbell and Broaddus (1995yf discuss the possibility of integrating writing instruction into the process. Many doctoral students have not explicitly studied writing and rhetoric since their freshman composition classes. Doctoral faculty expect doctoral students to possess not just bibliographic skills but also advanced skills in the mechanics of writing and the art of rhetoric. Yet the writing in many of the dis- sertations we read was little more sophisticated than that of fresh- men. Admittedly, this is part of a larger discussion, but the point is that we cannot expect students to write more sophisticated lit- erature reviews if they lack sophisticated writing skills. Students enter doctoral programs with a range of skills and abilities. Suc- cessful supervision and development of doctoral students entails integrating the expertise of a range of university personnel. Indi- vidual faculty members cannot be responsible for teaching the necessary skills in isolation-these skills must be integrated into the curriculum at the program level, and clearly communicated and evaluated. If the dissertations we reviewed had failed to properly collect and analyze data the education community would blame their authors or their dissertation committees. With few exceptions, the procedures and standards of data collection and analysis used in dissertations are well articulated and widely disseminated. However, we are not so fortunate in the case of literature re- viewing-the methods and expectations of literature reviewing are at best haphazardly described. Cooper (1985, p. 33yf D V V H U W V , "Students in education ... can take five or six statistics or meth- ods courses without ever directly addressing the problems and procedures of literature review." If Cooper's claim remains true, the situation must be remedied. Our concern is that by focusing on methodological issues, the education research community is addressing the symptom rather than the cause. That is, researchers must understand prior research in their field, and its strengths and weaknesses, before they can be expected to choose appropriate methods of data collection and data analysis. Moreover, sophisticated methods of data collection and analysis are of little use if one is studying an unproductive problem. They are also of little use if one lacks the sophisticated understanding of the literature needed to understand the mean- ing of the data. If doctoral programs and dissertation committees have not been attending to the literature review as a key compo- nent of a research project, we might find that increased attention to this aspect of our tradecraft will in turn improve the quality and usefulness of research. Further, if we, the education research community, are to teach our doctoral students, then we must begin to value the literature review in our own work. Imagine if we were to devote one tenth as much energy, care, and thought to being better scholars as we do to developing our methods of data collection and analysis. That we have not done so is a symptom of the broader culture of education research that artificially distinguishes between litera- ture review, on the one hand, and methods and analytic tech- niques, on the other. As a result, empiricism and methodological issues have been ascendant at the expense of scholarship, generativity, and theory building. Theorizing is fundamental to research and scholarship. It is an understanding of the literature that leads to increasingly sophisticated inquiry, connecting research methods and claims with their warrants. That our doctoral candidates often graduate without a sophisticated understanding of the literature in which they are supposed to be expert indicates a failure not only of doc- toral programs but of the education research enterprise in gen- eral. Doctoral students need to see us engaged in systematic analysis and synthesis of the literature if they are going to value those activities for themselves. Dissertation committees must hold 12 EDUCATIONAL RESEARCHER This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms the literature review to standards at least as high as those for methodology-arguably higher. Our failure to do so will leave us with fragmented and disjointed research, unconnected to theory. Requiring doctoral students in education to approach the ex- isting literature in their field in the ways that we have suggested is a means of inculcating the norms and practices of academic culture, with its emphasis on the analytic, the intellectual, the universal, and the theoretical (Labaree, 2003yf : H V X J J H V W W K D t the standards and criteria of good literature reviewing are part of the hidden curriculum of good graduate programs and perhaps part of the tacit knowledge passed on from mentors to candidates. Neither of these propositions can be tested with our current data, and further research will be needed to understand the pattern. In- deed, future research may show that a thorough, sophisticated understanding of the field is what separates the best doctoral can- didates and education researchers from everyone else. NOTES The authors wish to thank Aldrin Sweeney, Bill Gaudelli, Ann Austin, and three anonymous reviewers for their very helpful suggestions. 1 For example, the Carnegie Initiative on the Doctorate (Carnegie Foundation, 2003yf L V L Q Y H V W L J D W L Q J W K H V W U X F W X U H R I W K H G R F W R U D W H D F U R V s six disciplines, including education, and its implications for the future vigor, quality, and integrity of the field. Similar initiatives have been started by the Spencer Foundation (Young, 2001yf W K H 3 H Z & K D U L W D E O e Trust (2001yf W K H : R R G U R Z : L O V R Q 1 D W L R Q D O ) H O O R Z V K L S ) R X Q G D W L R n (2001yf W K H 1 D W L R Q D O 5 H V H D U F K & R X Q F L O \f, and the National Edu- cation Association (Lageman & Shulman, 1999yf , Q W K H & D U Q H J L H ) R X Q - dation initiative, for example, participating departments are from notable schools such as Arizona State University; Indiana University, Bloom- ington; Michigan State University; Ohio State University; the Univer- sity of Colorado, Boulder; the University of Michigan; the University of North Carolina, Chapel Hill; the University of Southern California; and Washington State University. All of the participating departments are housed in nationally ranked schools of education (America's Best Colleges, 2004yf ( P H U J L Q J I U R P W K H V H L Q L W L D W L Y H V U H F H Q W F K D S W H U V E y Schoenfeld (1999yf D Q G 6 L G G O H : D O N H U \f and articles by Labaree (2003yf 0 H W ] \f, Page (2001yf 3 D O O D V \f, and Richardson (2003yf have, in turn, addressed the problems of doctoral education and made suggestions for improvement. This literature has initiated an important dialogue in our field, but it is lacking for at least two reasons. First, although the aforementioned initiatives suggest helpful strategies for improving education research, they ignore the fact that, of the approximately 7,000 recipients of doc- toral degrees awarded in education every year, only a small percentage graduate from these nationally ranked institutions (National Center for Education Statistics, 2002yf : H F D Q Q R W D V V X P H W K D W W K H H [ S H U L H Q F H V R f doctoral students in these resource-rich institutions are representative of the experiences of most doctoral students. As a result, we need to look carefully at any generalizations and recommendations that these initia- tives generate. Second, although the reflections of leading scholars on doctoral education are valuable in initiating conversations about im- proving doctoral education, they are only the beginning and must be followed by systematic examinations of doctoral education. 2 It is worth noting that in the middle 19th century, U.S. universities misunderstood the changes in German universities. At that time the new U.S. graduate research universities emphasized the utility of empirical research (a theme that continues todayyf Z K H U H D V W K H * H U P D Q X Q L Y H U V L - ties emphasized the importance of freely pursuing investigations, both empirical and scholarly, without regard for the immediate needs of so- ciety (Veysey, 1965, p. 126yf . 3 Specifically, Hart (1999yf D U J X H G W K D W W K H G L V V H U W D W L R Q O L W H U D W X U H U H - view plays a central role in 1. distinguishing what has been done from what needs to be done; 2. discovering important variables relevant to the topic; 3. synthesizing and gaining a new perspective; 4. identifying relationships between ideas and practices; 5. establishing the context of the topic or problem; 6. rationalizing the significance of the problem; 7. enhancing and acquiring the subject vocabulary; 8. understanding the structure of the subject; 9. relating ideas and theory to applications; 10. identifying the main methodologies and research techniques that have been used; 11. placing the research in a historical context to show familiarity with state-of-the-art developments. (p. 27yf We found that these criteria gave us a much more robust and thorough set of criteria with which to evaluate the quality of the dissertations we were examining. However, as we tried to operationalize the criteria, we found that we needed to combine some, rewrite others, add several, and reorder the list to group similar ones. For example, we combined Hart's (1999yf & U L W H U L D D Q G L Q W R R X U & U L W H U L R Q - 5 D W L R Q D O L ] H G W K H S U D F W L F D l significance of the research problem." Also, we omitted Hart's Criterion 8 because it was too ambiguous to operationalize and evaluate. 4 Criterion H is the only one that is measured on a 4-point scale. While pilot-testing this rubric, we found that we could not reliably use a 4-point scale in measuring the other criteria, so we converted all of the others to 3-point scales. However, we needed a 4-point scale to include the possibility of an author's proposing and justifying the need for new research methods within a field of study. 5 A complete description of the research methodology can be found in Boote and Beile (2004yf 6 X P P D U \ V W D W L V W L F V D U H L Q F O X G H G K H U H 0 H D n scores for the individual items ranged from a low of 1.08 (SD = .29yf R n Criterion A, "Justified criteria for inclusion and exclusion from review" to a high of 2.33 on three separate criteria: "Placed the research in the historical context of the field" (SD = .78yf $ F T X L U H G D Q G H Q K D Q F H G W K e subject vocabulary" (SD = .49yf D Q G $ U W L F X O D W H G L P S R U W D Q W Y D U L D E O H s and phenomena relevant to the topic" (SD = .49yf , W Z R X O G V H H P W K D W , by common agreement, these latter three criteria are expected in any dis- sertation. And, indeed, these are the most common criteria listed in the introductory research textbooks. On the other hand, we were more trou- bled by the low average scores in criteria such as "Synthesized and gained a new perspective on the literature" (M = 1.42, SD = .67yf , G H Q W L I L H d the main methodologies and research techniques that have been used in the field, and their advantages and disadvantages" (M= 1.92, SD = .79yf , and "Rationalized the scholarly significance of the research problem" (M= 1.92, SD = .79yf . Inter-university differences also revealed potential concerns. Scores on each of the twelve criteria were averaged to arrive at an overall qual- ity score for the literature review. The mean of the literature review qual- ity scores ranged from a low of 1.42 to a high of 2.91 (M= 2.09, SD = .50yf % \ L Q V W L W X W L R Q W K H W R S U D Q N H G F R O O H J H V O L W H U D W X U H U H Y L H Z T X D O L W y scores (M= 2.12, SD= .55yf U D Q J H G I U R P W R W K H P L G W L H U F R O - lege's scores (M= 2.40, SD = .41yf U D Q J H G I U R P W R D Q G W K e nonranked college's scores (M= 1.73, SD = .37yf U D Q J H G I U R P W o 2.25. A Kruskal-Wallis calculation, H(2yf S U H Y H D O H G Q o statistically significant difference among institutions, yet practical dif- ferences are indicated by the nonranked college's average placement of 3.88, as compared with the top-tiered college's average of 6.75 and the mid-tier college's average of 8.88. 6 Passmore (1980yf U H P L Q G V X V W K D W W K H 3 K ' G L G Q R W R I F R X U V H F U H - ate pedantry" (1980, p. 53yf . REFERENCES Alton-Lee, A. (1998yf $ W U R X E O H V K R R W H U V F K H F N O L V W I R U S U R V S H F W L Y H D X - thors derived from reviewers' critical feedback. Teaching and Teacher Education, 14(8yf . AUGUST/SEPTEMBER 2005I13 This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms America's Best Colleges. (2004yf 8 6 1 H Z V H : R U O G 5 H S R U W 5 H W U L H Y H d January 4, 2005, from http://www.usnews.com/usnews/rankguide/ rghome.htm Babbie, E. (1998yf 7 K H S U D F W L F H R I V R F L D O U H V H D U F K W K H G \f. Belmont, CA: Wadsworth. Bailey, B. (1995yf 7 K H V L V S U D F W L F X P D Q G W K H O L E U D U L D Q V U R O H - R X U Q D O R f Academic Librarianship, 11, 79-81. Barger, R. R., & Duncan, J. K. (1986yf & U H D W L Y L W \ L Q G R F W R U D O U H V H D U F K : A reasonable expectation? The Educational Forum, 51(1yf . Barry, C. A. (1997yf , Q I R U P D W L R Q V N L O O V I R U D Q H O H F W U R Q L F Z R U O G 7 U D L Q - ing doctoral research candidates. Journal of Information Science, 23(3yf . Berelson, B. (1960yf * U D G X D W H H G X F D W L R Q L Q W K H 8 Q L W H G 6 W D W H V 1 H Z < R U N : McGraw-Hill. Berliner, D. C. (2002yf ( G X F D W L R Q D O U H V H D U F K 7 K H K D U G H V W V F L H Q F H R I D O O . Educational Researcher, 31(8yf . Boote, D. N., & Gaudelli, W. (2002yf 7 K H X V H D Q G D E X V H R I K L V W R U L F D l educational theorists: Comments on recent Dewey scholarship. In- sights: A publication of the John Dewey Society for the Study ofEduca- tion and Culture, 35(2yf . Boote, D.N., & Beile, P. (2004, Aprilyf 7 K H T X D O L W \ R I G L V V H U W D W L R Q O L W H U - ature reviews: A missing link in research preparation. Paper presented at the annual meeting of the American Educational Research Associ- ation, San Diego, CA. Britzman, D. (1991yf 3 U D F W L F H P D N H V S U D F W L F H $ F U L W L F D O V W X G \ R I O H D U Q L Q g to teach. Albany: State University of New York Press. Bruce, C. S. (1994yf 5 H V H D U F K V W X G H Q W V H D U O \ H [ S H U L H Q F H V R I W K H G L V V H U - tation literature review. Studies in Higher Education, 19(2yf . Bruce, C. S. (2001 layf , Q W H U S U H W L Q J W K H V F R S H R I W K H L U O L W H U D W X U H U H Y L H Z V : Significant differences in research students concerns. New Library World, 102(4yf . Bruce, C. S. (2001byf ) D F X O W \ O L E U D U L D Q S D U W Q H U V K L S V L Q $ X V W U D O L D n higher education: Critical dimensions. Reference Service Review, 29, 106-115. Carnegie Foundation. (2003yf & D U Q H J L H L Q L W L D W L Y H R Q W K H G R F W R U D W H 5 H - trieved January 4, 2005, from http://www.carnegiefoundation.org/ CID/index.htm Caspers, J., & Lenn, K. (2000yf 7 K H I X W X U H R I F R O O D E R U D W L R Q E H W Z H H n librarians and teaching faculty. In D. Raspa & D. Ward (Eds.yf 7 K e collaborative imperative: Librarians and faculty working together in the information universe (p. 148-154yf & K L F D J R $ P H U L F D Q / L E U D U y Association. Cooper, H. M. (1985yf $ W D [ R Q R P \ R I O L W H U D W X U H U H Y L H Z V 3 D S H U S U H V H Q W H G D t the annual meeting of the American Educational Research Association, Chicago. ERIC Document Reproduction Services No. ED254541. Creswell, J. W. (1994yf 5 H V H D U F K G H V L J Q 4 X D O L W D W L Y H D Q G T X D Q W L W D W L Y H D S - proaches. Thousand Oaks, CA: SAGE. Creswell, J. W. (2002yf ( G X F D W L R Q D O U H V H D U F K 3 O D Q Q L Q J F R Q G X F W L Q J D Q d evaluating quantitative and qualitative research. Upper Saddle River, NJ: Merrill Prentice Hall. Delamont, S., Atkinson, P., & Parry, O. (2000yf 7 K H G R F W R U D O H [ S H U L - ence: Success and failure in graduate school. New York: Falmer. Dong, Y. R. (1998yf 1 R Q Q D W L Y H J U D G X D W H V W X G H Q W V W K H V L V G L V V H U W D W L R n writing in science: Self-reports by students and their advisors from two US institutions. English for Specific Purposes, 17(4yf . Duke, N. K., & Beck, S. W. (1999yf ( G X F D W L R Q V K R X O G F R Q V L G H U D O W H U Q D - tive forms for the dissertation. Educational Researcher, 28(3yf . Fraenkel, J. R., & Wallen, N. E. (2003yf + R Z W R G H V L J Q D Q G H Y D O X D W H U H - search in education (5th ed.yf % R V W R Q 0 F * U D Z + L O O + L J K H U ( G X F D W L R Q . Galvan, J. L. (2004yf : U L W L Q J O L W H U D W X U H U H Y L H Z V $ J X L G H I R U V W X G H Q W V R I W K e social and behavioral sciences (2nd ed.yf / R V $ Q J H O H V 3 \ U F ] D N . Gay, L. R., & Airasian, P. W. (2000yf ( G X F D W L R Q D O U H V H D U F K & R P S H W H Q - ciesfor analysis and application. Upper Saddle River, NJ: Merrill. Granello, D. H. (2001yf 3 U R P R W L Q J F R J Q L W L Y H F R P S O H [ L W \ L Q J U D G X D W e written work: Using Bloom's taxonomy as a pedagogical tool to im- prove literature reviews. Counselor Education and Supervision, 40(4yf , 292-307. Grant, C. A., & Graue, E. (1999yf 5 H \fViewing a review: A case history of the "Review of Educational Research." Review ofEducational Re- search, 69(4yf . Hart, C. (1999yf ' R L Q J D O L W H U D W X U H U H Y L H Z 5 H O H D V L Q J W K H V R F L D O V F L H Q F H U H - search imagination. London: SAGE. Hjorland, B. (1988yf , Q I R U P D W L R Q U H W U L H Y D O L Q S V \ F K R O R J \ , P S O L F D W L R Q V R f a case study. Behavioral and Social Sciences Librarian, 6(3-4yf . Isbell, D., & Broaddus, D. (1995yf 7 H D F K L Q J Z U L W L Q J D Q G U H V H D U F K D V L Q - separable: A faculty-librarian teaching team. Reference Services Li- brarian, 48, 5-25. Johnston, S. (1997yf ( [ D P L Q L Q J W K H H [ D P L Q H U V $ Q D Q D O \ V L V R I H [ D P L Q - ers' reports on doctoral theses. Studies in Higher Education, 22(3yf , 333-347. Klein, P. (1999yf 5 H R S H Q L Q J L Q T X L U \ L Q W R F R J Q L W L Y H S U R F H V V H V R I O H D U Q - ing-to-write. Educational Psychology Review, 11, 203-270. Kuhn, D. (1992yf 7 K L Q N L Q J D V D U J X P H Q W + D U Y D U G ( G X F D W L R Q D O 5 H Y L H Z , 62(2yf . Kuhn, T. S. (1970yf 7 K H V W U X F W X U H R I V F L H Q W L I L F U H Y R O X W L R Q V Q G H G \f. Chicago: University of Chicago Press. Labaree, D. F. (2003yf 7 K H S H F X O L D U S U R E O H P V R I S U H S D U L Q J H G X F D W L R Q D l researchers. Educational Researcher, 32(1yf . Lagemann, E. C., & Shulman, L. S. (1999yf , V V X H V L Q H G X F D W L R Q U H V H D U F K : Problems and possibilities. San Francisco: Jossey-Bass. Lather, P. (1999yf 7 R E H R I X V H 7 K H Z R U N R I U H Y L H Z L Q J 5 H Y L H Z R I ( G X - cational Research, 69(1yf . LeCompte, M. D., Klingner, J. K., Campbell, S. A., & Menk, D. W. (2003yf ( G L W R U V L Q W U R G X F W L R Q 5 H Y L H Z R I ( G X F D W L R Q D O 5 H V H D U F K \f, 123-124. Lester, J. D. (2002yf 7 K H H V V H Q W L D O J X L G H 5 H V H D U F K Z U L W L Q J D F U R V V W K H G L V F L - plines. New York: Longman. Libutti, P., & Kopala, M. (1995yf 7 K H G R F W R U D O V W X G H Q W W K H G L V V H U W D - tion, and the library: A review of the literature. Reference Librarian, 48, 5-25. Mauch, J. E. (1998yf * X L G H W R W K H V X F F H V V I X O W K H V L V D Q G G L V V H U W D W L R Q A handbook for students and faculty. New York: M. Dekker. McMillan, J. H., & Schumacher, S. A. (2001yf 5 H V H D U F K L Q H G X F D W L R Q A conceptual introduction (5th ed.yf 1 H Z < R U N / R Q J P D Q . Metz, M. H. (2001yf , Q W H O O H F W X D O E R U G H U F U R V V L Q J L Q J U D G X D W H H G X F D W L R Q : A report from the field. Educational Researcher, 30(5yf . Mullins, G., & Kiley, M. (2002yf , W V D 3 K ' Q R W D 1 R E H O 3 U L ] H + R w experienced examiners assess research theses. Studies in Higher Edu- cation, 27(4yf . National Center for Education Statistics. (2002yf ' L J H V W R I H G X F D W L R Q V W D - tistics, 2001. Washington, DC: U.S. Department of Education. National Research Council. (1999yf , P S U R Y L Q J V W X G H Q W O H D U Q L Q J $ V W U D W H - gic plan for education research and its utilization. Washington, DC: National Academy Press. Nickerson, E.T. (1993yf 7 K H G L V V H U W D W L R Q K D Q G E R R N $ J X L G H W R V X F F H V V I X l dissertations. Dubuque, IA: Kendall/Hunt. Nightingale, P. (1984yf ( [ D P L Q D W L R Q R I U H V H D U F K W K H V H V + L J K H U ( G X F D - tion Research and Development, 3, 137-150. Ogden, E. H. (1993yf & R P S O H W L Q J \ R X U G R F W R U D O G L V V H U W D W L R Q R U P D V W H U s thesis in two semesters or less. Lancaster, PA: Technomic. Page, R. N. (2001yf 5 H V K D S L Q J J U D G X D W H S U H S D U D W L R Q L Q H G X F D W L R Q D O U H - search methods: One school's experience. Educational Researcher, 30(5yf . Pallas, A. M. (2001yf 3 U H S D U L Q J H G X F D W L R Q G R F W R U D O V W X G H Q W V I R U H S L V W H - mological diversity. Educational Researcher, 30(5yf . 14 1 EDUCATIONAL RESEARCHER This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms Paltridge, B. (2002yf 7 K H V L V D Q G G L V V H U W D W L R Q Z U L W L Q J $ Q H [ D P L Q D W L R n of published advice and actual practice. English for Specific Purposes, 21, 125-143. Pan, M. L. (2003yf 3 U H S D U L Q J O L W H U D W X U H U H Y L H Z V 4 X D O L W D W L Y H D Q G T X D Q W L - tative approaches. Los Angeles: Pyrczak. Passmore, J. (1980yf 7 K H S K L O R V R S K \ R I J U D G X D W H H G X F D W L R Q , Q : . . Frankena (Ed.yf 7 K H S K L O R V R S K \ D Q G I X W X U H R I J U D G X D W H H G X F D W L R n (pp. 40-59yf $ Q Q $ U E R U 8 Q L Y H U V L W \ R I 0 L F K L J D Q 3 U H V V . Pew Charitable Trust. (2001yf 5 H H Q Y L V L R Q L Q J W K H 3 K ' 5 H W U L H Y H G - D Q - uary 4, 2005, from http://www.grad.washington.edu/envision/ Pitkethly, A., & Prosser, M. (1995yf ( [ D P L Q H U V F R P P H Q W V R Q W K H L Q - ternational context of PhD theses. In C. McNaught & K. Beattie (Eds.yf 5 H V H D U F K L Q W R K L J K H U H G X F D W L R Q ' L O H P P D V G L U H F W L R Q V D Q G G L Y H U - sions (pp. 129-136yf 0 H O E R X U Q H + L J K H U ( G X F D W L R Q 5 H V H D U F K D Q G ' H - velopment Society of Australasia Victoria. Richardson, V. (2003yf 7 K H 3 K ' L Q H G X F D W L R Q 5 H W U L H Y H G - D Q X D U \ , 2005, from http://www.carnegiefoundation.org/CID/essays/CID_ educ_Richardson.pdf Rivard, L. P. (1994yf $ U H Y L H Z R I Z U L W L Q J W R O H D U Q L Q V F L H Q F H , P S O L F D - tions for practice and research. Journal ofResearch in Science Teach- ing, 31, 969-983. Schoenfeld, A. H. (1999yf 7 K H F R U H W K H F D Q R Q D Q G W K H G H Y H O R S P H Q W R f research skills. In E. C. Lagemann & L. S. Shulman (Eds.yf , V V X H V L n education research: Problems andpossibilities (pp. 166-223yf 6 D Q ) U D Q - cisco: Jossey-Bass. Schools of Education. (2000, April 10yf 8 6 1 H Z V H : R U O G 5 H S R U W . Shulman, L. S. (1999yf 3 U R I H V V L Q J H G X F D W L R Q D O V F K R O D U V K L S , Q ( & . Lagemann & L. S. Shulman (Eds.yf , V V X H V L Q H G X F D W L R Q U H V H D U F K 3 U R E - lems andpossibilities (pp.159-165yf 6 D Q ) U D Q F L V F R - R V V H \ % D V V . Siddle Walker, V. (1999yf & X O W X U H D Q G F R P P L W P H Q W & K D O O H Q J H V I R U W K e future training of education researchers. In E. C. Lagemann & L. S. Shulman (Eds.yf , V V X H V L Q H G X F D W L R Q U H V H D U F K 3 U R E O H P V D Q G S R V V L E L O L W L H s (pp. 224-244yf 6 D Q ) U D Q F L V F R - R V V H \ % D V V . Storr, R. J. (1969yf 7 K H E H J L Q Q L Q J R I J U D G X D W H H G X F D W L R Q L Q $ P H U L F D 1 H w York: Arno Press and New York Press. Strike, K., & Posner, G. (1983yf 7 \ S H V R I V \ Q W K H V L V D Q G W K H L U F U L W H U L D , n S. Ward and L. Reed (Eds.yf . Q R Z O H G J H V W U X F W X U H D Q G X V H , P S O L F D - tions for synthesis and interpretation (pp. 343-362yf 3 K L O D G H O S K L D : Temple University Press. Veysey, L. R. (1965yf 7 K H H P H U J H Q F H R I W K H $ P H U L F D Q X Q L Y H U V L W \ & K L F D J R : University of Chicago Press. Wideen, M. F., Mayer-Smith, J., & Moon, B. (1999yf $ F U L W L F D O D Q D O \ - sis of the research on learning to teach: Making the case for an eco- logical perspective on inquiry. Review ofEducational Research, 68(2yf , 130-178. Winter, R., Griffiths, M., & Green, K. (2000yf 7 K H D F D G H P L F T X D O L - ties of practice: What are the criteria for a practice-based PhD? Stud- ies in Higher Education, 25(1yf . Woodrow Wilson National Fellowship Foundation. (2001yf 7 K H U H V S R Q - sive PhD. Retrieved January 4, 2005, from http://www.woodrow. org/responsivephd/ Young, L. J. (2001yf % R U G H U F U R V V L Q J V D Q G R W K H U M R X U Q H \ V 5 H H Q Y L V L R Q L Q g the doctoral preparation of education researchers. Educational Re- searcher, 30(5yf . Zaporozhetz, L. E. (1987yf 7 K H G L V V H U W D W L R Q O L W H U D W X U H U H Y L H Z + R Z I D F - ulty advisors prepare their doctoral candidates. Unpublished doctoral dissertation, University of Oregon, Eugene. AUTHORS DAVID N. BOOTE is an Assistant Professor of Curriculum Studies in the College of Education, University of Central Florida, Orlando, FL 32816-1250; [email protected]. His research interests include edu- cation research studies, teacher educators, mathematics and physics problem solving, and moral education. PENNY BEILE is an Associate Librarian at the University of Central Florida, P.O. Box 162666, Orlando, FL 32816-2666; pbeile@mail. ucf.edu. Her areas of specialization are the production and use of schol- arly information in education and the information literacy abilities of education students. She is completing the development and validation of the Information Literacy Assessment Scale for Education (ILAS-EDyf . Manuscript received December 10, 2003 Revisions received January 4, April 28, 2005 Accepted May 6, 2005 AUGUST/SEPTEMBER 2005 I5 This content downloaded from 208.95.48.49 on Mon, 21 Feb 2022 02:25:11 UTC All use subject to https://about.jstor.org/terms