Psychology Research Paper

Properties of the Narrative Scoring Scheme Using Narrative Retells in Young School-Age Children John Heilmann East Carolina University, Greenville, NC Jon F. Miller Ann Nockerts University of Wisconsin — Madison Claudia Dunaway San Diego Unified School District, CA Purpose: To evaluate the clinical utility of the narrative scoring scheme (NSS) as an index ofnarrative macrostructure for young school-agechildren.Method: Oral retells of a wordless picture book were elicited from 129 typically developing chil-dren, ages 5 –7. A series of correlations and hierarchical regression equations were completedusing microstructural measures of vocabulary andgrammar to predict NSS scores.Results: While the NSS was significantly corre- lated with age and each of the microstructuralmeasures, the hierarchical regression analysesrevealed a unique relationship between vocabu-lary and narrative macrostructure. Conclusion: The NSS is an efficient and informa- tive tool for documenting children ’s development of narrative macrostructure. The relationshipbetween the NSS and microstructural measuresdemonstrates that it is a robust measure ofchildren ’s overall oral narrative competence and a powerful tool for clinicians and researchers. Theunique relationship between lexical diversity andthe NSS confirmed that a special relationshipexists between vocabulary and narrative organi-zation skills in young school-age children.

Key Words: narrative, language sample analysis, story grammar, vocabulary A nalysis of oral narratives provides a rich source of data that documents children ’s language use in a naturalistic context. Narrative analysis is a highly effective clinical and research tool, as it allows examiners to analyze multiple linguistic features simultaneously using a single short sample. Examiners have the opportunity to doc- ument children ’s productive vocabulary and grammar using microstructural analyses as well as children ’s broader text- level narrative organization skills by utilizing macrostructural analyses (see Westby, 2005). Microstructural analyses primarily focus on children ’s lin- guistic form and content, which are measured within individ- ual utterances. Linguistic form is commonly assessed by analyzing children ’s grammatical and syntactic abilities using mean length of utterance (e.g., Brown, 1973; Miller, 1981) and various measures of sentence complexity (e.g., Nippold, Hesketh, Duthie, & Mansfield, 2005; Schuele & Tolbert, 2001; Scott & Stokes, 1995). Measures of linguistic content are used to document children ’s productive vocabulary skills, which typically calculate children ’s lexical diversity using measures such as type –token ratio (Templin, 1957) and number of dif- ferent words (Miller, 1987; Miller & Klee, 1995). While these measures are the most commonly cited microstructural anal- yses in the literature, this list is in no way exhaustive. Micro- structural analyses continue to be reviewed, critiqued, and reanalyzed (see Justice et al., 2006, for a review and discussion). Macrostructural analyses, on the other hand, examine children ’s language skills beyond the utterance level and document children ’s ability to relate concepts that transcend the individual utterance. Most macrostructural analyses of children ’s narratives are rooted in the story grammar tradi- tion, which proposes that all stories have a setting and episode system (Mandler & Johnson, 1977; Rumelhart, 1975; Stein & Glenn, 1979). The setting provides background information Research American Journal of Speech-Language Pathology Vol. 19 154 –166 May 2010 AAmerican Speech-Language-Hearing Association 154 on the characters and their environment, while the episode system includes three main components that occur in all stories: (a) a problem (initiating event and /or internal response), (b) attempts at solving the problem, and (c) consequences/ outcomes. To be a complete episode, a narrator must include all three of these key components (see Strong, 1998, for a review). These settings and episodes can be combined in an infinite number of ways to create individual stories.

Relationship Between Microstructural and Macrostructural Measures Previous research has demonstrated that lexical and gram- matical gains play an important role in children ’s acqui- sition of narrative proficiency (Berman & Slobin, 1994; Bishop & Donlan, 2005). In their seminal work, Berman and Slobin (1994) documented the role of linguistic forms taking on new functions that aid in the organization of narratives.

This work described the trade-offs that occurred when differ- ent microstructural features were used for different functions, including the organization of narratives. Children ’snarrative organization skills were positively related to advances in use of microstructural features, including grammatical forms (e.g., verb tense, aspect, and voice), lexical forms (e.g., lexical aspect and manner/cause of verbs of motion), and lexico-grammatical features (e.g., locative particles, prepositional phrases, and connectives). This pattern of results is consistent with Slobin ’s long-standing claim that “new forms first express old func- tions, and new functions are first expressed by old forms ” (Slobin, 1973, p. 184). Bishop and Donlan (2005) observed that children ’s ability to encode and retell a story was more strongly related to linguistic ability than nonverbal IQ. Chil- dren ’s microstructures, including complex syntax and relating causal concepts, were more influential in event memory and story organization than their cognitive skills. These data docu- menting the unique relationship between microstructures and macrostructures are contrary to evidence demonstrating that children ’s narrative organization skills emerge from more general cognitive capabilities, including executive function (van den Broek, 1997), and sociolinguistic processes (Eaton, Collis, & Lewis, 1999; McCabe, 1997; Peterson & McCabe, 2004; Quasthoff, 1997a, 1997b).

Narrative Skills of Children With Language Impairment Given the strong relationship between microstructural and macrostructural measures, it is not surprising that children with language impairment have substantial difficulty using appropriate vocabulary and grammar when telling stories (Boudreau & Hedberg, 1999; Gillam & Johnston, 1992; Pearce, McCormack, & James, 2003; Reilly, Losh, Bellugi, & Wulfeck, 2004) in addition to substantial difficulty with text-level orga- nization of narratives (Boudreau & Hedberg, 1999; Manhardt & Rescorla, 2002; Merritt & Liles, 1987; Pearce et al., 2003; Reilly et al., 2004). The presence of both microstructural and macrostructural deficits in children with language impair- ment counters theories that identify the primary cause of language impairment as deficiencies with grammatical com- petence (e.g., Clahsen, 1989; Gopnik & Crago, 1991; Rice, Wexler, & Cleave, 1995). Alternatively, the global narrative difficulties of children with language impairment suggest that their language deficits may be due to broader information- processing deficits, such as a reduced processing capacity (Boudreau, 2007; Colozzo, Garcia, Megan, Gillam, & Johnston, 2006). Further study of the relationship between children ’s microstructural and macrostructural language skills will pro- vide a better understanding of the nature of the impairment.

Better understanding of children ’s narrative skills in general, and macrostructural skills in particular, also has substantial clinical implications. Difficulty with narrative organization can have a dramatic impact on children with language impair- ment, as such discourse-level skills are required to effectively communicate. While vocabulary and grammar deficits limit children ’s ability to produce fully competent utterances, im- paired macrostructural skills affect children ’s ability to gen- erate coherent and age-appropriate extended discourse.

Furthermore, narratives are a major component of the school curriculum, and children are expected to understand and use appropriate narrative form effectively. Therefore, narrative macrostructure skills must be efficiently and accurately docu- mented, and should be considered within an extensive assess- ment protocol for children with language difficulties.

Methods of Measuring Narrative Macrostructure While most narrative macrostructure measures are rooted in the story grammar tradition and share the same underlying principles, the coding of narrative macrostructure varies widely throughout the literature. Some narrative macro- structure coding schemes documented children ’s inclusion of specific story grammar components (e.g., Strong, 1998) or identified the presence or absence of specific story grammar components for a given story (e.g., Berman, 1988; Boudreau & Hedberg, 1999; Miles & Chapman, 2002; Reilly et al., 2004). In these studies, children who produced more story grammar components and /or more advanced story grammar features were thought to have stronger narrative organization skills. The second major class of macrostructural measures used text-level judgments of children ’s narrative proficiency (Applebee, 1978; Hedberg & Westby, 1993; Stein, 1988).

Rather than identifying the presence or absence of specific story grammar components, these measures required holistic judgments by the examiner to rate the quality and develop- mental level of the narrative. The major advantage of the simple story grammar coding procedures is that it facilitates a relatively high level of ac- curacy across coders. There is less room for differences across coders, as the coder is only responsible for identifying the presence or absence of specific story-related themes. The major disadvantage of simple story grammar analyses is that they are potentially limited in their ability to account for the abstract interutterance concepts and qualitative aspects of the narrative, or the story ’s“sparkle ”(Peterson & McCabe, 1983). McFadden and Gillam (1996) demonstrated that holistic rat- ings of children ’s narratives capture the more refined aspects of narratives, such as charm and depth, and that these ratings were better than simple story grammar analyses for documenting differences between children with language impairment and their typically developing peers between the ages of 9;0 (years; months) and 11;7.

Heilmann et al.: Properties of the Narrative Scoring Scheme 155 Additional Skills Required to Tell an Effective Story:

Beyond Story Grammar The development of narratives in children and adults has been studied extensively and has revealed additional areas of advancement beyond inclusion of story grammar features, including children ’s use of literate language and cohesive de- vices (Bamberg & Damrad-Frye, 1991; Halliday & Hasan, 1976; Hedberg & Westby, 1993; Wigglesworth, 1997). Use of literate language occurs when children use abstract language features commonly used by teachers and found in the curri- culum (Westby, 2005). Some key literate language features related to narrative competence include use of metacognitive verbs (e.g., think or know ), metalinguistic verbs (e.g., say or talk ), and elaborated noun phrases (e.g., the boy in the restau- rant with the frog ; see Nippold, 2007, and Westby, 2005, for a review). Bamberg and Damrad-Frye (1991) identified that more sophisticated and later developing narratives included abstract language such as metacognitive and metalinguistic verbs. In their analysis of productions of Frog, Where Are You? (Mayer, 1969), Bamberg and Damrad-Frye identified that these abstract language skills emerged at age 5 years, demonstrated robust development through adulthood, and were essential for relating the hierarchical relationships be- tween events in complex narrative productions. Additional studies have identified that literate language skills were pres- ent in children ’s oral narratives during the preschool years (Curenton & Justice, 2004), developed through the school years and into adolescence (Greenhalgh & Strong, 2001; Nippold, 2007; Pelligrini, Galda, Bartini, & Charak, 1998), and were used less frequently by children with language im- pairment (Greenhalgh & Strong, 2001). Another high-level narrative feature that continues to develop through the school years is the cohesiveness of children ’snar- rative productions (Halliday & Hasan, 1976; Wigglesworth, 1997). To tell a story, narrators must effectively use cohesive devices to carry concepts across individual utterances. Three major categories of cohesive devices are (a) referential cohe- sion, which allows a narrator to maintain appropriate refer- ence to the characters, objects, and locations across utterances using both noun phrases and pronouns; ( b) conjunctive cohe- sion, which allows a narrator to sustain concepts across phrases and utterances using conjunctive words and phrases (e.g., and, but, besides, on the other hand, finally, in addition ); and (c) lexical cohesion, which allows a narrator to effectively use vocabulary to link concepts across utterances. Measures of cohesiveness can be a sensitive index of language use, as children with language impairment have more difficulty with correct use of cohesive ties (Hedberg & Westby, 1993; Liles, 1985; Strong & Shaver, 1991). While cohesion is often con- sidered a microstructural measure, we treated it as a macro- structural measure because aspects of cohesion transcend individual utterances and are necessary for producing coher- ent narratives.

The Narrative Scoring Scheme: A Comprehensive Measure of Narrative Proficiency Our goal in developing the narrative scoring scheme (NSS) was to create a metric that documents the range of skills required for school-age children to effectively tell a coherent and in- teresting story (see Appendix). To extend beyond simple story grammar analyses, the NSS incorporates multiple aspects of the narrative process into a single scoring rubric and provides an overall impression of the child ’s narrative ability. This metric combines both the basic features of the story grammar approaches as well as the higher level narrative skills that continue to develop through the school-age years. In addition to adding higher level narrative skills in the scoring scheme, the NSS uses a combination of discrete coding criteria and examiner judgment. The NSS was created to improve on the simple story grammar measures by requiring the examiners to make interutterance text-level judgments, which have been shown to be more effective than discrete coding schemes in identifying children with language impairment (McFadden & Gillam, 1996). By breaking the judgments into seven skill areas, examiners have the opportunity to reflect on each com- ponent of the narrative process and judge the child ’s profi- ciency in that area. This combination of explicit scoring guidelines and flexibility to allow for examiner judgment reflects the hybrid nature of the NSS. The scores from the seven NSS categories are combined to provide a single com- posite score, which allows the examiner to generate an index of children ’s overall narrative ability. The first step in developing the NSS was establishing the key components from the story grammar literature, which included the story ’sintroduction, the major conflicts and res- olutions ( conflict resolution ), and a conclusion . To document children ’s use of literate language, the NSS includes catego- ries that assess metacognitive and metalinguistic verbs. The mental states component documents children ’s abilities to use metacognitive verbs (e.g., think and know ) to describe the characters ’thoughts and feelings. The character development component of the NSS also documents children ’s literate language skills by measuring the ability to use metalinguistic verbs (e.g., talk and say ), differentiate between main and sup- porting characters, and talk in the first person to depict the characters in the story. The NSS evaluates two separate as- pects of cohesive ties that were adapted from Halliday and Hasan (1976). The referencing component measures aspects of referential cohesion, including appropriate use of pronouns and antecedents. The cohesion component documents the conjunctive and lexical aspects of cohesion, including appro- priate ordering, emphasis of critical events, and transitions between events.

Goals of the Study Children ’s performance on the NSS has been reported in studies that examined the narrative organization skills of na- tive Spanish-speaking children who were learning English as a second language (Miller et al., 2006). The goal of the present study was to describe the NSS from a clinical perspec- tive and to further analyze the linguistic properties of the measure in a group of children who were fluent in English.

Furthermore, the literature has revealed that there is a special relationship between children ’s microstructural and macro- structural language skills (Berman & Slobin, 1994; Bishop & Donlan, 2005). To better understand the linguistic properties of the NSS and to extend our understanding of the relationship 156 American Journal of Speech-Language Pathology Vol. 19 154 –166 May 2010 between microstructural and macrostructural measures, a second goal was to document the relationship between chil- dren ’s vocabulary, grammar, and narrative organization skills. To achieve this, we examined the relationship between mea- sures of vocabulary, grammar, and the NSS. These analyses further clarified the key constructs that the NSS is measuring and also provided additional evidence for the role of vocabu- lary and grammar in the development of young school-age children ’s narrative organization skills. This study addressed the following questions:

1. Are age and measures of vocabulary and grammar signifi- cantly correlated with NSS scores in narrative retells of young school-age children? 2. Are measures of vocabulary uniquely related to NSS scores in narrative retells of young school-age children? 3. Are measures of grammar uniquely related to NSS scores in narrative retells of young school-age children? Method Participants A total of 129 typically developing children age 5 –7 years were recruited for this study. The children were recruited from public schools in the San Diego (CA) City School and Cajon Valley School Districts. Administrators from the two districts assisted with obtaining informed consent from each of the children ’s primary caregivers. The pool of potential partici- pants was reviewed by the school ’s speech-language pathol- ogists (SLPs) and classroom teachers to identify children who qualified for the study. To participate in the study, children were required to have average scores on all summative class- room, district, and state assessments. The classroom teachers reviewed the records for each student to identify him or her as average performing. The child ’s academic record was also reviewed to identify that he or she had no history of language impairment and /or learning disability. While academic data were used as inclusionary criteria for the participants, test scores and descriptions of performance were not recorded and were not available for further analysis. The majority of the participants were native English speakers. A small percentage of the children were Spanish / E nglish bilingual and designated as “fluent English. ”This designation was made by the child ’s respective school district and was based on a passing grade on an English proficiency test and grade-level academics. The SLPs and teachers confirmed that each child met the inclu- sionary criteria and enrolled the eligible children in the study.

The individual children participating in the study provided verbal assent prior to completing the protocol. Table 1 summarizes the demographic data for the partici- pants. Sixty-one percent of the participants were in kinder- garten, 36% of the children were in first grade, and 2% were in preschool. The numbers of male and female participants were roughly equal, with slightly fewer boys than girls. The school SLPs attempted to recruit students who reflected the racial and ethnic diversity of their schools. The SLPs first identified the children ’s ethnicity (Hispanic or not Hispanic) and then documented the non-Hispanic children ’s race. A review of the race and ethnicity data revealed that the sample is a relatively heterogeneous group. Socioeconomic status was measured by calculating the highest number of years of education that the child ’s mother completed. On average, mothers completed 14.4 years of education ( SD = 2.5), with a range of 9 –20 years. The majority of the mothers completed at least some college, while only eight of the children ’s mothers did not complete high school.

Procedure Each participant completed a narrative retell of the word- less picture book Frog, Where Are You? (Mayer, 1969). The purpose of collecting the narratives was to establish a nor- mative database reflecting typically developing children ’s oral narrative skills and to further our understanding of chil- dren ’s developing narrative competence. The head SLP from each district was responsible for training 18 school-based SLPs to elicit the oral narratives. The head SLP and clinicians met on three separate occasions and had the opportunity to practice the protocol with each other several times. The clini- cians read the scripted instructions to the students and cued them to listen to a taped version of the story while following along with the pictures in the book. The students then retold the story using the book as an aid. Examiner prompts were limited to encouragement to begin the story and open-ended cues to continue the retell. The scripted instructions and audiotaped story script were adapted from the Strong Narrative Assess- ment Procedure (Strong, 1998) and were used to facilitate high fidelity among the numerous examiners completing the language sample elicitation.

Transcription and Coding The children ’s narrative productions were digitally recorded and sent to the Language Analysis Lab at the University of Wisconsin — Madison, where they were transcribed by trained research assistants who had at least 10 hr of transcription TABLE 1. Demographic data for all participants. Variable n GradePreschool 3Kindergarten 79First grade 47 SexFemale 69Male 60 Race/ethnicityWhite 87Hispanic 16Other a 15 African American 7No data 4 Note. Ethnicity data were collected for children who were Hispanic or Latino. Race data are provided for all children who were non-Hispanic/non-Latino.aOther races/ethnicities included Arabic (2), Chinese (3), Japanese (2),Korean (1), Filipino (5), Portuguese (1), and Samoan (1).

Heilmann et al.: Properties of the Narrative Scoring Scheme 157 experience using standard coding conventions for Systematic Analysis of Language Transcripts (SALT) software (Miller & Iglesias, 2008). Utterances were segmented into communi- cation units (C-units; Labov & Waletzky, 1967), which in- cluded a main clause and all dependent clauses. The transcripts began and ended with the child ’s first and last utterance, re- spectively. See Miller and Iglesias (2008) for a full review of the transcription conventions. After completing the orthographi c transcription, the research assistant reviewed the transcript and completed the NSS. To score the NSS, the transcriber carefully reviewed the narrative transcript and assigned a score of 0 –5 for each of the seven categories summarized in the Appendix. Categories that could not be scored received a score of zero or NA. Scores of zero were given if the child did something that precluded the ex- aminer from scoring a section of the NSS, such as skipping a part of the story or refusing to complete the task. If there was an error on the part of the examiner (e.g., not following the protocol or problems with the recording), sections of the NSS that were affected were not scored, and the examiner coded the section as not applicable for analysis (i.e., NA). For all other sections, scores of 1 reflected minimal presence/immature performance, scores of 3 reflected emerging skills, and scores of 5 reflected proficient performance. Transcribers also had the opportunity to assign scores of 2 and 4 if performance was judged to be between the major anchors. To improve the accuracy of the scoring procedure, specific guidelines were provided for scores of 1, 3, and 5 (see the Appendix). These guidelines assisted the transcriber in assigning an accurate score that reflected the child ’s performance in each compo- nent of the narrative process and reduced the abstractness of the narrative macrostructure concepts. The scoring across the seven categories received equal weighting because the liter- ature revealed that each of these seven narrative aspects is necessary for telling a well-developed story. Furthermore, keeping the scoring rules stra ightforward and consistent facilitates simple and accurate scoring. A comprehensive training procedure is available on the SALT Web site (www.

saltsoftware.com /training / handcoded / ) that includes an overview of the NSS; scoring tips; a description of how to enter NSS scores into a SALT file; excerpts from samples demonstrating minimal/immature, emerging, and proficient performance across each section of the NSS; and a set of practice transcripts. In addition to the NSS, the transcribers completed coding for the subordination index (SI; Scott & Stokes, 1995; Strong, 1998). The transcribers added a code to each C-unit that sum- marized the number of independent and dependent clauses.

C-units that were incomplete, unintelligible, or nonverbal, or that had an error at the utterance level, were excluded from the analysis. Utterance-level errors included incorrect word order, omission of more than two words in an utterance, and utterances that simply did not ma ke sense. Elliptical responses to examiner questions were also excluded from the SI analysis.

Utterances in which the child inappropriately omitted the sub- ject or copula were coded and received a score of zero. After each C-unit was coded, the SI was generated by dividing the total number of clauses (both main and subordinate) by the total number of C-units. After all the narratives were transcribed and coded, the transcripts were analyzed using SALT (Miller & Iglesias, 2008), which produced a rectan- gular data file summarizing each dependent measure for each of the transcripts. This file was formatted for statistical anal- ysis using SPSS Version 16.0.

Agreement Accuracy of the transcription and coding process was ex- amined at three levels. Protocol accuracy was calculated by the principal investigator, who reviewed 10% of the written transcripts to identify whether the transcribers were adhering to the transcription conventions. Percentage agreement be- tween the transcribers and principal investigator was 98% for segmentation rules, 99% for word-level codes, and 98% for coding of reduplications and reformulations (i.e., mazes).

To determine transcription accuracy and coding agreement, 10% of the narratives were independently transcribed and coded for the NSS and SI by a second research assistant.

Transcription accuracy was calculated by comparing the in- dependent transcripts at the word and morpheme level (94% agreement), utterance segmentation decisions (98% agree- ment), placement of mazes (93% agreement), and utterance types (100% agreement). Calculating agreement for the two coding schemes pre- sented a greater challenge. Simple interrater agreement scores can be misleading, as small differences between coders (e.g., NSS scores of 24 and 25) are treated the same as large dif- ferences between coders (e.g., NSS scores of 16 and 28).

Therefore, agreement for the NSS and SI coding was calcu- lated using Krippendorff ’s alpha, which accounted for both chance agreement and the degree of difference between tran- scribers (Krippendorff, 1980). Alpha values accounting for differences in ordinal data were calculated using the summed NSS and SI scores that were calculated by the two independent transcribers ( a=.92forSI; a= .79 for NSS). Krippendorff established benchmarks for alpha values, with ≥.80 reflecting adequate agreement and values between .67 and .80 reflect- ing acceptable agreement for exploratory research and draw- ing tentative conclusions. For a review of the accuracy and agreement process and a further discussion of Krippendorff ’s alpha, see Heilmann et al. (2008).

Language Sample Measures To test the relationship between microstructural measures and the NSS, we used the following language sample mea- sures that repeatedly have been found to be robust and devel- opmentally sensitive to the population and context used in the present study (i.e., typically developing 5 –7-year-olds who produced oral narratives): Length /productivity. Number of total words ( NTW) is a measure of productivity that documents the amount of infor- mation provided in the story (Allen, Kertoy, Sherblom, & Pettit, 1994; Paul & Smith, 1993). In addition, NTW was used to assist with statistically controlling sample length, which has the potential to affect additional language sample measures. Vocabulary. Number of different words (NDW) is a mea- sure of lexical diversity that provides a robust estimate of children ’s productive vocabulary (Klee, 1992; Miller, 1987; Miller & Klee, 1995) and has been widely used as an index of 158 American Journal of Speech-Language Pathology Vol. 19 154 –166 May 2010 vocabulary skills in studies examining children ’s oral nar- rative skills (Gazella & Stockman, 2003; Humphries, Cardy, Worling, & Peets, 2004; Swanson, Fey, Mills, & Hood, 2005; Uccelli & Páez, 2007). Grammar. Mean length of C-unit (MLCU) measures the average number of morphemes that children use per C-unit and is an index of general grammati cal skills that increases with age through the school-age years, particularly when analyzed using narratives and expositories (Leadholm & Miller, 1992; Nippold et al., 2005). Also, the SI, a measure of clausal density, indicates the average number of subordinate clauses produced per C-unit. Use of subordinate clauses emerges during the preschool years (Diessel & Tomasello, 2000) and continues to develop through the school-age years (Nippold, 2007). Because the productions of oral narratives were relatively short, issues of sample length required special consideration.

For conversational language samples, it is common practice to control sample length across children by using a consistent number of utterances, words, or elapsed time (e.g., 50 utter- ances; Miller, 1981). In the present study, children produced between 12 and 77 utterances. Using a standard transcrip- tion cut of 50 utterances would limit the analyses to 11 tran- scripts. If we chose to maintain 75% of the data, 97 transcripts would remain in the analysis, and each transcript would be cut at 29 utterances and would ultimately withhold 955 utter- ances from the data set. Our goal was to maintain the maxi- mum amount of data available by maintaining the entire sample and to use statistical procedures to account for differences in transcript length. Furthermore, using the entire sample for linguistic analyses, including NDW, is common practice in contemporary studies of children ’s oral narratives (Gazella & Stockman, 2003; Humphries et al., 2004; Swanson et al., 2005; Uccelli & Páez, 2007).

Results The NSS scores were first reviewed to identify how many child and examiner errors precluded scores to be adminis- tered. In the present study, a total of 903 scores were completed using the children ’s narrative transcripts (seven categories were scored across 129 transcripts). Across the 903 sections, only six sections received scores of zero (four stories lacked an in- troduction, and two children omitted a conclusion), and only one of the 903 sections received a score of NA; 99% of the NSS sections were able to be scored correctly, confirming that the training and elicitation procedures facilitated a high level of child compliance and examiner fidelity during the elici- tation process. The skewness statistic was calculated for the NSS to de- termine if there was an unequal distribution of NSS scores across the sample (see Coolican, 2004, for a review). Skew- ness measures of zero indicate a perfectly normal distribution, while skewness values below –0.8 or above 0.8 have been described as “noticeably skewed ”( Bourque & Clark, 1992, p. 69). The skewness statistic for the NSS was –0.5, indicat- ing that scores were more concentrated toward the ceiling but were not noticeably skewed according to the criteria established by Bourque and Clark. Descriptive statistics for age, the NSS, and each of the microstructural measures are presented in Table 2. Table 2 also summarizes the bivariate correlations between the NSS, children ’s age, and each of the microstructures. All correla- tions were significant and were moderate in strength (Cohen, 1988). To further explore the covariance structure between the variables and to identify unique relationships between the microstructures and the NSS, two separate hierarchical re- gression equations were completed. Hierarchical regressions allow examination of variance that is uniquely explained by a given variable. The first hierarchical regression equation is summarized in Table 3. Sample length was first controlled by entering NTW into Model 1. Length was controlled because measures of lexical diversity are inevitably influenced by the NTW in the sample (Malvern & Richards, 2002). That is, the more total words that a child produces, the more opportunity he or she has to produce different words. Model 2 identified the unique relationship between vocab- ulary and NSS after controlling for length. Taken together, NTW and NDW were significantly correlated with NSS scores (r= .58). Adding NDW in Model 2 increased the explained variance from 24% to 33%, a net increase of 9%. A one-way analysis of variance was completed to determine whether the increase in explained variance was significant, and an f2sta- tistic was calculated to estimate its effect size. According to Cohen (1988), effect sizes of 0.02, 0.15, and 0.35 are con- sidered small, medium, and large, respectively. The 9% increase in explained variance was significant, F(1, 124) = 16.5, p< .001, f2= 0.12, demonstrating that NDW was uniquely TABLE 2. Descriptive statistics for language sample measuresand correlations with the narrative scoring scheme (NSS). Measure MSD Range Correlationwith NSS Age 6.0 0.7 5.0 –7.0 .30* Narrative macrostructure(NSS) 19.0 3.0 11.0 –26.0 — Productivity (NTW) 264.1 77.2 133.0 –608.0 .50** Vocabulary (NDW) 92.5 19.0 47.0 –150.0 .58** Grammar (MLCU) 7.0 0.9 4.8 –9.6 .44** Grammar (SI) 1.1 0.1 1.0 –1.3 .35** Note. NTW = number of total words; NDW = number of different words; MLCU = mean length of C-unit; SI = subordination index.

*p= .001. ** p< .001. TABLE 3. Summary of hierarchical regression analysis withgrammatical measures uniquely predicting NSS scores inModel 3.

Model Predictors r Adjusted r2 r2change 1 Productivity (NTW) .50 .24*2 Productivity (NTW) .58 .33* .09* Vocabulary (NDW) 3 Productivity (NTW) .61 .35* .02 Vocabulary (NDW)Grammar (MLCU, SI) *Significant at p< .01. Heilmann et al.: Properties of the Narrative Scoring Scheme 159 related to children ’s NSS scores above and beyond the vari- ability explained by NTW. Model 3 was completed to identify if there was a unique relationship between the two grammatical measures (MLCU and SI) and NSS scores, after controlling for NTW and NDW.

The measures in this final model were significantly correlated with the NSS ( r= .61). Adding MLCU and SI to the third regression equation added an additional 2% prediction to NSS scores, which was not significant, F(2, 122) = 2.9, p=.060, f2= 0.03. In this first hierarchical regression analysis, NDW uniquely predicted NSS scores after controlling for length using NTW. The two grammatical measures, however, did not add unique prediction of children ’s NSS scores. To con- firm that NDW was the major unique predictor of NSS scores, a second hierarchical regression equation was completed and is summarized in Table 4. Again, sample length was controlled by entering NTW in the first model. The two grammatical measures were next entered in the second equation, which resulted in a combined correlation of r= .57. Adding SI and MLCU to NTW increased the explained variance from .24 to .30, documenting that the grammatical measures explain 6% of the variance in NSS after controlling for sample length.

This increase was significant, F(2, 123) = 6.0, p= .002, f2= 0.08. NDW was added to the third model to test the unique prediction of vocabulary on NSS scores. Adding NDW in the third model resulted in a 5% increase in explained variance between the microstructures and NSS, which was significant, F(1, 122) = 8.7, p= .003, f2=0.06. In sum, this series of hierarchical regression equations documented that children ’s use of vocabulary is the major significant and unique microstructural variable in predicting their story organization skills as measured by the NSS. Chil- dren ’s productive grammar, while significantly correlated with NSS scores, did not provide unique prediction of the children ’s narrative macrostructure ability. Discussion Upon establishing a set of reference databases for chil- dren ’s narrative retells, the Language Analysis Lab at the University of Wisconsin — Madison set out to identify a clini- cally useful measure of children ’s narrative organization skills. The NSS was created to bring together the benefits of con- crete scoring criteria combined with judgment of text-level constructs. The NSS also incorporated higher level narrative components, including cohesive markers and measures of literate language, to measure a wider range of skills than tradi- tional story grammar analyses. To determine whether the NSS was developmentally ap- propriate for the children in this study, the skewness statistic was calculated. This analysis revealed that the data were not noticeably skewed according to Borque and Clark ’s(1992) criteria and that the NSS appeared to be a sensitive measure for school-age children who produced an oral retell. (We are cur- rently completing a more thorough investigation of the de- velopmental sensitivity of the NSS as compared to alternative methods of documenting narrative macrostructure skills.) To document the relationship between the microstructural features of language samples and children ’s performance on the NSS, a series of correlation and hierarchical regression analyses were completed. These analyses confirmed that a close relationship existed between children ’s productivity, vocab- ulary, grammar, and narrative macrostructure skills. The correlationanalysesdocumentedthatageandeachofthe microstructural measure s (NTW, NDW, MLCU, and SI) were significantly correlated with children ’s narrative organi- zation skills. It is noteworthy that the correlation between age and NSS was the weakest observed correlation. The socio- cultural theory of narrative macrostructure development pro- poses that children who have more experience with stories will have greater narrative competence (e.g., Eaton et al., 1999; Stein & Glenn, 1979). The children in the present sample spanned 2 years in age. The older children in this sample likely had more experience listening to and telling stories. How- ever, it was the children ’s vocabulary and grammar skills that were most strongly related to their narrative macrostructure scores. While this study did not directly control for the amount of experience the children had with narratives, the data pro- vide some additional evidence for the importance of chil- dren ’s linguistic proficiency in predicting narrative organization skills (Berman & Slobin, 1994; Bishop & Donlan, 2005). Two separate hierarchical regression analyses were com- pleted to identify the unique relationships between each of the microstructures and NSS scores. After controlling for length, the unique relationship between the NSS and mea- sures of vocabulary and grammar was calculated. The anal- yses showed that children ’s productive vocabulary skills were the only unique predictor of narrative organization skills.

Grammatical measures, on the other hand, provided no unique prediction of NSS scores. The unique importance of vocab- ulary in predicting narrative organization skills was a novel finding. Bishop and Donlan (2005) documented that children ’s use of complex syntax and expression of causal concepts uniquely predicted children ’s ability to organize their oral narratives. Bishop and Donlan examined children between 7 and 9 years of age, while the present study investigated children age 5 –7 years. The children in the present study were using minimal complex syntax. As observed in Table 2, SI values averaged 1.1, showing that children produce approx- imately one subordinate clause every 10 utterances. The children ’s use of subordination may have been influenced by the story used in this study; the children ’s SI values, on aver- age, were just slightly lower than the SI value from the story script (SI = 1.15). These low levels of subordination could explain, in part, the modest correlations and lack of a unique relationship between the grammatical measures and the NSS. TABLE 4. Summary of hierarchical regression analysis withvocabulary measures uniquely predicting NSS scores inModel 3.

Model Predictors r Adjusted r2 r2change 1 Productivity (NTW) .50 .24*2 Productivity (NTW) .57 .30* .06* Grammar (MLCU, SI) 3 Productivity (NTW) .61 .35* .05* Grammar (MLCU, SI)Vocabulary (NDW) *Significant at p< .01. 160 American Journal of Speech-Language Pathology Vol. 19 154 –166 May 2010 Relationship Between Vocabulary and the NSS The present study revealed that there is a special and im- portant relationship between narrative organization and vo- cabulary skills that emerges prior to children becoming fully literate. This study demonstrated that the development of story schema and vocabulary acquisition is developing along a similar path. The importance of vocabulary in narrative organization skills is not surprising given the literature de- scribing the development of narrative form. Preschool and young school-age children typically produce narratives that simply chain sequences of events in temporal order (Berman, 1988) and provide simple descriptive sequences (Stein & Glenn, 1979). It is not until the later school-age years that children hierarchically organize the events in their narrative productions (Berman, 1988) and take multiple perspectives to relate the events (Stein & Glenn, 1979). To produce these more advanced narratives, children must use complex syntax (Bamberg & Damrad-Frye, 1991; Bishop & Donlan, 2005).

Before children are proficient in using complex syntax, they likely have to rely on their vocabulary skills to organize their narrative productions. Furthermore, the emerging literacy literature has documented the importance of vocabulary in the development of children ’s narrative and comprehension skills. There is a well-documented relationship between children ’s vocabulary skills and reading comprehension (see Scarborough, 2001). Furthermore, preschool and young school-age chil- dren acquire new vocabulary through repeated exposures to narrative form (Robbins & Ehri, 1994; Senechal & Cornell, 1993) and by receiving adult scaffolding that highlights the story ’s structure (Hargrave & Senechal, 2000; Penno, Wilkinson, & Moore, 2002).

Limitations The results from the hierarchical regression equations could have been affected by the sampling context. Because all of the measures were collected from a single language sam- ple, high internal validity was achieved. While each of the language sample measures theoretically and empirically re- flects its respective linguistic domain, strong intercorrelations between language sample measures have been observed (Miller & Klee, 1995). The hierarchical regression equations did af- ford better understanding of the covariance structure. How- ever, vocabulary and grammatical measures acquired from alternate tasks may provide a more informative test of the relationship between vocabulary, grammar, and narrative macrostructure.

Clinical Implications The NSS was created as a clinically useful comprehensive narrative macrostructure measure. To be clinically feasible, an assessment tool must be able to be completed in a short amount of time to accommodate the busy schedule of the SLP.

In our lab, trained transcribers could complete the NSS using a narrative transcript in approximately 3 min. In addition to efficient scoring, the NSS was developed to facilitate accurate scoring both within and between examiners. The Krippen- dorff alpha analyses revealed that the NSS had lower inter- rater agreement than we would ideally observe and that the NSS alpha was not as high as the alpha for the SI. The dif- ference in alphas between the NSS and SI was not surprising, however, as the NSS requires much greater individual judg- ment when compared to the relatively straightforward scoring rules for the SI. However, recall that these subjective ratings are often the most sensitive when identifying children with language impairment (e.g., McFadden & Gillam, 1996). Our goal is to continue developing new training methods and complete additional research identifying ways to increase coding accuracy for the NSS and other macrostructural anal- yses of children ’s natural language use. Upon scoring the NSS, clinicians are provided with a clini- cally useful benchmark for children ’s overall narrative profi- ciency. The data described in this study used the composite NSS score, which was the summed score for all seven sec- tions of the NSS. We proposed that the NSS composite score provided an index of children ’s overall narrative organization skills. The NSS data described in this study are available as part of the SALT Narrative Story Retell database and can be downloaded free of charge from the SALT Web site. SALT software, used to access the data, can compare a target child ’s NSS scores with those of age-matched peers. In addition to comparing NSS scores to the SALT database, composite NSS scores can be useful for monitoring progress and documenting treatment outcomes by collecting multiple narrative samples from a child and documenting changes in NSS scores over time. The NSS also provides examiners with the opportunity to identify specific aspects of the narrative process that are dif- ficult for a child. Because the NSS separately judges seven aspects of the narrative process, examiners can evaluate per- formance on each section of the NSS to identify areas of strength and areas that require intervention. Compared with narrative macrostructure measures that make holistic text- level judgments of narrative proficiency, there is greater spe- cificity in the NSS scoring procedure. Having a detailed narrative performance profile facilitates a more accurate de- scription of the child ’s performance and can assist in the development of treatment goals. For example, a child who performed poorly on the referencing and cohesion sections of the NSS but did well on the other sections likely has difficulty using cohesive devices. Treatment goals could include im- proving the child ’s use of referential, lexical, and conjunctive cohesion. Increasing our understanding of the relationships between vocabulary, grammar, and narrative macrostructure has im- portant clinical implications for documenting functional out- comes and identifying treatment goals. One goal of language intervention programs is for the therapy tasks to generalize to functional tasks. Telling a well-formed narrative is a func- tional task that is important to children ’s academic success (Bishop & Edmundson, 1987; Griffin, Hemphill, Camp, & Wolf, 2004; O ’Neill, Pearce, & Pick, 2004). The data from the present study demonstrated that vocabulary skills were uniquely related to children ’s story organization skills. Therefore, a treatment procedure that increases a child ’s vocabulary skills would have broader, more functional outcomes if concurrent increases in his or her narrative macrostructure skills were documented. Similarly, we may expect that treatments ad- dressing narrative macrostruc ture skills could also result in Heilmann et al.: Properties of the Narrative Scoring Scheme 161 concurrent increases in vocabulary skills. Clearer understand- ing of the relationship between microstructures and macro- structures will facilitate selection of appropriate treatment goals. Understanding these relationships can aid in identifying the appropriate microstructures to address when implement- ing interventions that focus on linguistic macrostructures, such as narrative organization. It is important to note that the data in this study are purely correlational and that further research is needed before causal relationships between vocabulary, grammar, and narrative macrostructure can be identified. In sum, narrative language assessment is an effective method for documenting children ’s language skills. The NSS was developed by the Language Analysis Lab as a clinically useful index of children ’s narrative organization skills. Given its clinical feasibility and its robust relationship with other linguistic measures, the NSS provides clinicians and re- searchers with an additional tool to document children ’sglobal language skills using a functional and curriculum-oriented task.

Acknowledgments This research was supported by National Institutes of Health Grant 5 T32 DC005459 ( “Interdisciplinary Research Training in Speech-Language Disorders ”) and by the Language Analysis Lab at the University of Wisconsin —Madison. This work was also made possible through close collaboration with two public school districtsin San Diego County: San Diego City Schools and Cajon ValleyUnion Schools. The authors express appreciation to Karen Andriacchiand the research assistants from the Language Analysis Lab for theirassistance with transcription and organization of the data. Portionsof this article were presented at the 2006 American Speech-Language-Hearing Association Annual Convention in Miami, FL.

References Allen, M., Kertoy, M., Sherblom, J., & Pettit, J. (1994). Chil- dren ’s narrative productions: A comparison of personal event and fictional stories. Applied Psycholinguistics, 15, 149 –176. Applebee, A. (1978). The child ’s concept of story: Ages two to seven . Chicago, IL: University of Chicago Press. Bamberg, M., & Damrad-Frye, R. (1991). On the ability to pro- vide evaluative comments: Further explorations of children ’s narrative competencies. Journal of Child Language, 18, 689 –710. Berman, R. (1988). On the ability to relate events in narrative. Discourse Processes, 11, 469 –497. Berman, R., & Slobin, D. (Eds.). (1994). Relating events in nar- rative: A crosslinguistic developmental study . Hillsdale, NJ: Erlbaum. Bishop, D., & Donlan, C. (2005). The role of syntax in encoding and recall of pictorial narratives: Evidence from specific lan-guage impairment. British Journal of Developmental Psychol- ogy, 23, 25–46. Bishop, D., & Edmundson, A. (1987). Language-impaired 4-year- olds: Distinguishing transient from persistent impairment.Journal of Speech & Hearing Disorders, 52, 156 –173. Boudreau, D. (2007). Narrative abilities in children with language impairment. In R. Paul (Ed.), L anguage disorders from a devel- opmental perspective: Essays in honor of Robin S. Chapman( pp. 331 –356). Mahwah, NJ: Erlbaum. Boudreau, D., & Hedberg, N. (1999). A comparison of early literacy skills in children with specific language impairment and their typically developing peers. American Journal of Speech- Language Pathology, 8, 249 –260. Bourque, L., & Clark, V. (1992). Processing data: The survey example . Thousand Oaks, CA: Sage. Brown, R. (1973). A first language: The early stages . Cambridge, MA: Harvard University Press. Clahsen, H. (1989). The grammatical characterization of develop- mental dysphasia. Linguistics, 27, 897 –920. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Colozzo, P., Garcia, R., Megan, C., Gillam, R., & Johnston, J. (2006, June). Narrative assessment in SLI: Exploring interac- tions between content and form . Poster session presented at the annual meeting of the Symposium on Research in ChildLanguage Disorders, Madison, WI. Coolican, H. (2004). Research methods and statistics in psychol- ogy . London, England: Hodder & Stoughton. Curenton, S., & Justice, L. (2004). African American and Cauca- sian preschoolers ’use of decontextualized language: Literate language features in oral narratives. Language, Speech, and Hearing Services in Schools, 35, 240 –253. Diessel, H., & Tomasello, M. (2000). The development of relative clauses in spontaneous child speech. Cognitive Linguistics, 11, 131 –151. Eaton, J., Collis, G., & Lewis, V. (1999). Evaluative explanations in children ’s narratives of a video sequence without dialogue. Journal of Child Language, 26, 699 –720. Gazella, J., & Stockman, I. (2003). Children ’s story retelling under different modality and task conditions: Implications forstandardizing language sampling procedures. American Journal of Speech-Language Pathology, 12, 61–72. Gillam, R., & Johnston, J. (1992). Spoken and written language relationships in language/ learning-impaired and normally achiev-ing school-age children. Journal of Speech and Hearing Research, 35, 1303 –1315. Gopnik, M., & Crago, M. (1991). Familial aggregation of a devel- opmental language disorder. Cognition, 39, 1–50. Greenhalgh, K., & Strong, C. (2001). Literate language features in spoken narratives of children with typical language and childrenwith language impairments. Language, Speech, and Hearing Services in Schools, 32, 114 –125. Griffin, T., Hemphill, L., Camp, L., & Wolf, D. (2004). Oral discourse in the preschool years and later literacy skills. First Language, 24, 123 –147. Halliday, M., & Hasan, R. (1976). Cohesion in English . London, England: Longman. Hargrave, A., & Senechal, M. (2000). A book reading interven- tion with preschool children who have limited vocabularies: Thebenefits of regular reading and dialogic reading. Early Child- hood Research Quarterly, 15, 75–90. Hedberg, N., & Westby, C. (1993). Analyzing storytelling skills: Theory to practice . Tucson, AZ: Communication Skill Builders. Heilmann, J., Miller, J., Iglesias, A., Fabiano-Smith, L., Nockerts, A., & Andriacchi, K. (2008). Narrative transcription accuracy and reliability in two languages. Topics in Language Disorders, 28, 178 –188. Humphries,T.,Cardy,J.,Worling,D.,&Peets,K. (2004). Narrative comprehension and retelling abilities of childrenwith nonverbal learning disabilities. Brain and Cognition, 56, 77–88. Justice, L., Bowles, R., Kaderavek, J., Ukrainetz, T., Eisenberg, S., & Gillam, R. (2006). The Index of Narrative Microstructure: A clinical tool for analyzing school-age children ’s narrative performances. American Journal of Speech-Language Pathol- ogy, 15, 177 –191. 162 American Journal of Speech-Language Pathology Vol. 19 154 –166 May 2010 Klee, T. (1992). Developmental and diagnostic characteristics of quantitative measures of children ’s language production. Topics in Language Disorders, 12, 28–41. Krippendorff, K. (1980). Content analysis: An introduction to its methodology . Thousand Oaks, CA: Sage. Labov, W., & Waletzky, J. (1967). Narrative analysis: Oral ver- sions of personal experience. In J. Helm (Ed.), Essays on the verbal and visual arts ( pp. 12 –44). Seattle: University of Washington Press. Leadholm, B., & Miller, J. F. (1992). Language sample analysis: The W isconsin guide . Madison, WI: Department of Public Instruction. Liles, B. (1985). Cohesion in the narratives of normal and language- disordered children. Journal of Speech and Hearing Research, 28, 123 –133. Malvern, D., & Richards, B. (2002). Investigating accommoda- tion in language proficiency interviews using a new measure oflexical diversity. Language Testing, 19, 85–104. Mandler, J., & Johnson, N. (1977). Remembrance of things parsed: Story structure and recall. Cognitive Psychology, 9, 111 –151. Manhardt, J., & Rescorla, L. (2002). Oral narrative skills of late talkers at ages 8 and 9. Applied Psycholinguistics, 23, 1–21. Mayer, M. (1969). Frog, where are you? New York, NY: Dial Press. McCabe, A. (1997). Developmental and cross-cultural aspects of children ’s narration. In G. Bamberg (Ed.), Narrative develop- ment: Six approaches ( pp. 137 –174). Mahwah, NJ: Erlbaum. McFadden, T., & Gillam, R. (1996). An examination of the quality of narratives produced by children with language disorders.Language, Speech, and Hearing Services in Schools, 27, 48–56. Merritt, D., & Liles, B. (1987). Story grammar ability in children with and without language disorder: Story generation, storyretelling, and story comprehension. Journal of Speech and Hearing Research, 30, 539 –552. Miles, S., & Chapman, R. S. (2002). Narrative content as described by individuals with Down syndrome and typically developingchildren. Journal of Speech, Language, and Hearing Research, 45, 175 –189. Miller, J. F. (1981). Assessing language production in children . Baltimore, MD: University Park Press. Miller, J. F. (1987). A grammatical characterization of language disorder. In J. Martin, P. Fletcher, R. Grunwell, & D. Hall (Eds.),Proceedings of the First International Symposium on SpecificSpeech and Language Disorders in Children ( pp. 100 –113). London, England: Association for All Speech Impaired Children. Miller, J., Heilmann, J., Nockerts, A., Iglesias, A., Fabiano, L., & Francis, D. (2006). Oral language and reading in bilingual children. Learning Disabilities Research & Practice, 21, 30–43. Miller, J. F., & Iglesias, A. (2008). Systematic Analysis of Lan- guage Transcripts (Research Version 9.1) [Computer software].Middleton, WI: SALT Software. Miller, J., & Klee, T. (1995). Computational approaches to the anal- ysis of language impairment. In P. Fletcher & B. MacWhinney(Eds.), The handbook of child language (pp. 545 –572). Oxford, England: Blackwell. Nippold, M. (2007). Later language development: School-age children, adolescents, and young adults (3rd ed.). Austin, TX: Pro-Ed. Nippold, M., Hesketh, L., Duthie, J., & Mansfield, T. (2005). Conversational versus expository discourse: A study of syntacticdevelopment in children, adolescents, and adults. Journal of Speech, Language, and Hearing Research, 48, 1048 –1064. O’Neill, D., Pearce, M., & Pick, J. (2004). Preschool children ’s narratives and performance on the Peabody IndividualizedAchievement Test —Revised: Evidence of a relation between early narrative and later mathematical ability. First Language, 24, 149 –183. Paul, R., & Smith, R. (1993). Narrative skills in 4-year-olds with normal, impaired, and late-developing language. Journal of Speech and Hearing Research, 36, 592 –598. Pearce,W.,McCormack,P.,&James,D. (2003). Exploring the boundaries of SLI: Findings from morphosyntactic andstory grammar analyses. Clinical Linguistics & Phonetics, 17, 325 –334. Pelligrini, A., Galda, L., Bartini, M., & Charak, D. (1998). Oral language and literacy learning in context: The role of socialrelationships. Merrill-Palmer Quarterly, 44, 38–54. Penno, J., Wilkinson, I., & Moore, D. (2002). Vocabulary acquisi- tion from teacher explanation and repeated listening to stories:Do they overcome the Matthew effect? Journal of Educational Psychology, 94, 23–33. Peterson, C., & McCabe, A. (1983). Developmental psycholin- guistics: Three ways of looking at a child ’s narrative .NewYork, NY: Plenum Press. Peterson, C., & McCabe, A. (2004). Echoing our parents: Parental influences on children ’s narration. In M. Pratt & B. Fiese (Eds.), Family stories and the life course: Across time and generations( pp. 27 –54). Mahwah, NJ: Erlbaum. Quasthoff, U. (1997a). An interactive approach to narrative devel- opment. In M. Bamberg (Ed.), Narrative development: Six approaches ( pp. 51 –83). Mahwah, NJ: Erlbaum. Quasthoff, U. (1997b). Were you ever in a situation where you were in serious danger of being killed? Narrator-listener inter-action in Labov and W aletzky ’s narratives. Journal of Narrative & Life History , 7, 121 –128. Reilly, J., Losh, M., Bellugi, U., & Wulfeck, B. (2004). Frog, Where are you? Narratives in children with specific languageimpairment, early focal brain injury and Williams syndrome.Brain & Language, 88, 229 –247. Rice, M. L., Wexler, K., & Cleave, P. L. (1995). Specific language impairment as a period of extended optional infinitive. Journal of Speech and Hearing Research, 38, 850 –863. Robbins, C., & Ehri, L. (1994). Reading storybooks to kinder- garteners helps them learn new vocabulary words. Journal of Educational Psychology, 86, 56–64. Rumelhart, D. (1975). Notes on a schema for stories. In D. G. Bobrow & A. M. Collins (Eds.), Representation and under- standing: Studies in cognitive science ( pp. 211 –236). New York, NY: Academic Press. Scarborough, H. (2001). Connecting early language and literacy to later reading (dis)abilities: Evidence, theory, and practice. InS. Neuman & D. Dickinson (Eds.), Handbook for research in early literacy ( pp. 97 –110). New York, NY: Guilford Press. Schuele, C., & Tolbert, L. (2001). Omissions of obligatory relative markers in children with specific language impairment. Clinical Linguistics & Phonetics, 15, 257 –274. Scott, C., & Stokes, S. (1995). Measures of syntax in school-age children and adolescents. Language, Speech, and Hearing Services in the Schools, 26, 309 –319. Senechal, M., & Cornell, E. (1993). Vocabulary acquisition through shared reading experiences. Reading Research Quarterly, 28, 360 –374. Slobin, D. (1973). Cognitive prerequisites for the development of grammar. In C. Ferguson & D. Slobin (Eds.), Studies of child language development ( pp. 175 –208). New York, NY: Aca- demic Press. Stein, N. L. (1988). The development of storytelling skill. In M. Franklin & S. Barten (Eds.), Child language: A reader ( pp. 282 –297). New York, NY: Oxford University Press. Stein, N., & Glenn, C. (1979). An analysis for story comprehen- sion in elementary school. In R. Freedle (Ed.), New directions Heilmann et al.: Properties of the Narrative Scoring Scheme 163 in discourse processing (Vol. 2, pp. 53 –120). Norwood, NJ: Ablex. Strong, C. (1998). The Strong Narrative Assessment Procedure . Eau Claire, WI: Thinking Publications. Strong, C., & Shaver, J. (1991). Stability of cohesion in the spoken narratives of language-impaired and normally developing school-aged children. Journal of Speech and Hearing Research , 34, 95–111. Swanson, L., Fey, M., Mills, C., & Hood, L. (2005). Use of narrative-based language intervention with children who havespecific language impairment. American Journal of Speech- Language Pathology, 14, 131 –143. Templin, M. (1957). Certain language skills in children: Their development and interrelationships . Minneapolis: University of Minnesota Press. Uccelli, P., & Páez, M. (2007). Narrative and vocabulary devel- opment of bilingual children from kindergarten to first grade:Developmental changes and associations among English andSpanish skills. Language, Speech, and Hearing Services in Schools, 38, 225 –236. van den Broek, P. (1997). Discovering the cement of the universe: The development of event comprehension from childhood toadulthood. In P. van den Broek, P. Bauer, & T. Bourg (Eds.),Developmental spans in event comprehension: Bridging fic-tional and actual events ( pp. 321 –342). Mahwah, NJ: Erlbaum. Westby, C. (2005). Assessing and facilitating text comprehension problems. In H. Catts & A. Kamhi (Eds.), Language and reading disabilities ( pp. 157 –232). Boston, MA: Allyn & Bacon. Wigglesworth, G. (1997). Children ’s individual approaches to the organization of narrative. Journal of Child Language, 24, 279 –309. Received March 24, 2008Revision received December 1, 2008Accepted November 30, 2009DOI: 10.1044/1058-0360(2009/08-0024) Contact author: John Heilmann, East Carolina University, Department of Communication Sciences and Disorders, HealthSciences Building, Room 3310T, Greenville, NC 27858-4353.E-mail: [email protected]. 164 American Journal of Speech-Language Pathology Vol. 19 154 –166 May 2010 Appendix ( p. 1 of 2) The Narrative Scoring Scheme Characteristic Proficient Emerging Minimal/immature Introduction Setting Setting -Child launches into story with no attempt to providethe setting. -Child states general place and providessome detail about the setting (e.g.,reference to the time of the setting — daytime, bedtime, or season). -Child states general setting butprovides no detail. -Setting elements are stated atappropriate place in story. -Description or elements of storyare given intermittentlythrough story. Characters -Child may provide descriptionof specific element of setting(e.g., the frog is in the jar). -Main characters are introducedwith some description ordetail provided. ORCharacters-Characters of story are mentionedwith no detail or description. Characterdevelopment -Main character(s) and all supporting character(s) are mentioned. -Both main and active supportingcharacters are mentioned. -Inconsistent mention is made ofinvolved or active characters. -Throughout story it is clear that childcan discriminate between main andsupporting characters (e.g., moredescription of and emphasis onmain character[s]). -Main characters are not clearlydistinguished from supportingcharacters. -Characters necessary foradvancing the plot arenot present. -Child narrates in first person usingcharacter voice (e.g., “You get out of my tree, ”said the owl). Mental states -Mental states of main and supporting characters are expressed whennecessary for plot developmentand advancement. -Some mental state words areused to develop character(s). No use is made of mental statewords to develop characters. -A variety of mental state words are used. -A limited number of mental statewords are used inconsistentlythroughout the story. Referencing -Child provides necessary antecedents to pronouns. -Referents/antecedents are usedinconsistently. -Pronouns are used excessively. -References are clear throughout story. -No verbal clarifiers are used.-Child is unaware listener is confused. Conflictresolution -Child clearly states all conflicts andresolutions critical to advancingthe plot of the story. -Description of conflicts andresolutions critical toadvancing the plot of thestory is underdeveloped. -Random resolution is stated withno mention of cause or conflict. OR OR -Not all conflicts and resolutionscritical to advancing the plotare present. -Conflict is mentioned withoutresolution.OR-Many conflicts and resolutionscritical to advancing theplot are not present. Cohesion -Events follow a logical order. -Events follow a logical order. -No use is made of smooth transitions. -Critical events are included, while lessemphasis is placed on minor events. -Excessive detail or emphasisprovided on minor eventsleads the listener astray. -Smooth transitions are providedbetween events. OR -Transitions to next event are unclear.OR-Minimal detail is given for critical events.OR-Equal emphasis is placed on all events. Heilmann et al.: Properties of the Narrative Scoring Scheme 165 Appendix ( p. 1 of 2)? (p. NaN ) The Narrative Scoring Scheme Characteristic Proficient Emerging Minimal/immature Conclusion -Story is clearly wrapped up using general concluding statementssuch as “and they were together again happy as could be. ” -Specific event is concluded, but nogeneral statement is made asto the conclusion of the whole story. -Child stops narrating, andlistener may need to ask if thatis the end. Scoring: Each characteristic receives a scaled score of 0 –5. Proficient characteristics = 5; Emerging = 3; Minimal /immature = 1. Scores between (i.e., 2 and 4) are undefined; use judgment. Scores of zero and NA are defined below. A composite is scored by adding the total of the characteristicscores. Highest score = 35.

A score of zero is given for child errors (such as telling the wrong story, conversing with examiner, not completing /refusing task, using wronglanguage and creating inability of scorer to comprehend story in target language, abandoned utterances, unintelligibility, poor performance, orcomponents of rubric are in imitation-only).

A score of NA (nonapplicable) is given for mechanical /examiner/operator errors (such as interference from background noise, issues with recordingsuch as cut-offs or interruptions, examiner quitting before child does, examiner not following protocol, or examiner asking overly specific or lead ing questions rather than open-ended questions or prompts).

Appendix ( p. 2 of 2) The Narrative Scoring Scheme 166 American Journal of Speech-Language Pathology Vol. 19 154 –166 May 2010 Copyright of American Journal of Speech-Language Pathology is the property of American Speech-Language- Hearing Association and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.