Psychology Research Paper

Article Corresponding author:John Heilmann, Department of Communi\fation S\fien\fes an\ud Disorders, \bail Stop #668, East Carolina University, Greenville, NC 27858, USAE-mail: heilmannj@e\fu.edu / $ 1 * 8 $ * ( 7 ( 6 7 , 1 * Language Testing27(4) 603–626 © The Author(s) 2010Reprints and permiss\uion: sagepub.\fo.uk/journalsPermission.navDOI: 10.1177/026553220\u9355669 http://ltj.sagepub.\fom Sensitivity o\f narrative organization measur\yes using narrative retells produ\bed by young s\bhool-age \bhildren John HeilmannEast Carolina University, USA Jon F. MillerUniversity of Wis\fonsin – \badison, USA Ann No\bkerts University of Wis\fonsin – \badison, USA Abstra\bt Analysis of \fhildren’s produ\ftions of oral narratives provides a ri\fh des\fription of \fhildren’s oral language skills. However, measures of narrative organization \fan be dire\ftly affe\fted by both developmental and task-based performan\fe \fonstraints whi\fh \fan make a measure insensitive and inappropriate for a parti\fular population and/or sampling method. This study \friti\fally reviewed four methods of evaluating \fhildren’s narrative organization skills and revealed that the Narrative S\foring S\fheme (NSS) was the most developmentally sensitive measure for a group of 129 5–7-year -old \fhildren who \fompleted a narrative retell. Upon \fomparing the methods of assessing narrative organization skills, the NSS was unique in its in\forporation of higher-level narrative features and its s\foring rules, whi\fh required examiners to make subje\ftive judgments a\fross seven aspe\fts of the narrative pro\fess. The dis\fussion surrounded issues of measuring \fhildren’s narrative organization skills and, more broadly, issues surrounding sensitivity of \friterion referen\fed assessment measu\ures.

Keywords assessment, \feiling effe\fts, narrative, language development, oral language, psy\fhometri\fs 604 Language Testing 27(4\f Introdu\btion Oral narrative skills in c\bildren Assessment of children’s oral narratives is of significant interest to researchers and \fractitioners, as being a \froficient narrator is an im\fortant skill in the life of \boung children. Oral narrative skills are a ke\b com\fonent of most school curricula. In the USA, each state is required to develo\f guidelines describing skills that need to be incor - \forated into the general curriculum. While oral language skills are often not formall\b assessed in the general classroom, the\b are a ke\b com\fonent in each state’s guidelines.

For exam\fle, the state of North Carolina requires teachers to facilitate master\b of nar - rative com\frehension as well as effective abilit\b to \froduce narratives and the com\flex language associated with a literate s\feaking st\ble (North Carolina De\fartment of Public Instruction, 2004). Oral narrative skills have been incor\forated into children’s curricula for good reasons. Being a \froficient narrator is a skill needed to ex\fress one’s intentions and effectivel\b \fartici\fate in classroom activities. In addition, several decades of research have documented the strong link between children’s oral narrative skills and broader curricular requirements. Research on monolingual English-s\feaking children has documented that chil - dren’s oral narrative skills are \fredictive of later reading outcomes (Bisho\f & Edmundson, 1987; Dickinson & McCabe, 2001; Griffin, Hem\fhill, Cam\f, & Wolf, 2004; Hem\fhill & Snow, 1996; Roth, S\feece, Coo\fer, & de la Paz, 1996; Snow, 1983; Snow, Dickinson, Jennings, & Purves, 1991; Tabors, Snow, & Dickinson, 2001).

Additional studies have documented that children’s earl\b narrative com\fetence is related to broader academic outcomes (e.g., Fazio, Naremore, & Connell, 1996; O’Neill, Pearce, & Pick, 2004). Fazio and colleagues found that oral narrative skills were one of the strongest \fredictors for whether or not a child later required academic remediation, while O’Neill et al. identified a strong relationshi\f between \boung children’s oral narrative skills and later mathematical abilit\b. While these studies documenting the relationshi\f between oral narratives and broader reading and aca - demic outcomes are correlational and a causal relationshi\f cannot be assumed, there is a general consensus in the field that oral narrative skills ma\b \fla\b a ke\b role in develo\fing the foundation for higher level academic tasks. It is onl\b natural that oral narrative skills are of interest to those working with children who have language im\fairments (LI). A\f\froximatel\b 7% of monolingual English-s\feaking children ex\ferience significant deficits in their oral language skills des\fite normal cognitive skills (Tomblin, Records, Buckwalter, Zhang, Smith, & O’Brien, 1997). Children with LI have substantial difficult\b \froducing full\b coherent oral narratives (Boudreau & Hedberg, 1999; Fe\b, Catts, Proctor-Williams, Tomblin, & Zhang, 2004; Gillam & Johnston, 1992; Newman & McGregor, 2006; Pearce, McCormack, & James, 2003; Reill\b, Losh, Bellugi, & Wulfeck, 2004). Assessment of children’s narratives can be an effective method of identif\bing the \fresence of LI (Allen, Kerto\b, Sherblom, & Petit, 1994; Paul & Smith, 1993) and \frovides a functional descri\ftion of children’s \ferformance, which ma\b assist in the develo\fment of treatment goals (Miller, Gillam, & Peña, 2001). Heilmann et al. 605 Oral narratives in bilingual\4 c\bildren Clinicians, teachers, and researchers are also interested in better understanding the oral narrative skills of children learning a second language. Just as with monolingual chil - dren, there is a significant \fredictive relationshi\f between oral narrative skills and read - ing outcomes in children learning a second language (August & Shanahan, 2006; Miller, Heilmann, Nockerts, Iglesias, Fabiano, & Francis, 2006; Oller & Pearson, 2002). Miller et al. identified that measures from oral narratives \fredicted reading scores within and across languages in \boung school-age English language learners (ELL). Given the strong relationshi\f between oral narratives and reading, it is im\fortant to ensure that \boung ELLs have sufficient oral narrative skills. Because a dis\fro\fortionatel\b high number of ELLs have \foor reading outcomes when com\fared to their monolingual counter\farts (August & Hakuta, 1997), there is a need for a better understanding of the relationshi\f between oral narrat\give skills and read\ging outcomes. Numerous s\feech and language scholars have recommended using oral narratives in clinical assessments of bilingual children, given the naturalness of the task (e.g., Fiestas & Peña, 2004; Munoz, Gillam, Peña, & Gulle\b-Faehnle, 2003; Uccelli & Páez, 2007).

The act of telling stories is universal across cultures and is an im\fortant instrument for transmitting information (Mandler, Scribner,Cole, & DeForest, 1980); the format of tell - ing a stor\b ma\b be more familiar to children from different cultures than formal language testing. Anal\bsis of oral narratives across multi\fle languages can \frovide an estimate of relative \froficienc\b in each language s\foken and ma\b assist in differentiating language differences, where a children ma\b have limited \froficienc\b in one of their languages s\foken, from global language disorders, where children demonstrate marked deficits in all languages s\foken\g (Gutiérrez-Clellen\g, 2002; Rojas & Igl\gesias, 2009).

Criterion referenced assessment Oral narrative anal\bsis falls under the broader umbrella of criterion referenced (CR) assessment. The goal of CR assessment is to generate a detailed descri\ftion of a child’s \ferformance in a target domain (e.g., narrative organization) and to identif\b if the child’s \ferformance meets develo\fmental ex\fectations (Linn & Gronlund, 2000). While the \fur\fose of standardized tests is to maximize differences between individuals, CR assess - ments afford clinicians the o\f\fortunit\b to obtain more detailed data on s\fecific skill areas (Bachman, 2000). While standardized language tests commonl\b evaluate limited sets of skills that have been stri\f\fed of their communicative context (Marquardt & Gillam, 1999), most CR oral language assessments, including narrative language sam\fle anal\bsis, examine real-life communication situations and do not rel\b on assessment of decontextualized l\ganguage skills. Because CR assessments can generate rich descri\ftive data, the\b are \farticularl\b use - ful when describing the \frofile of a child exhibiting language learning difficulties. While children with LI share the same underl\bing characteristic (i.e., substantial difficult\b with language), there is considerable heterogeneit\b in the \fatterns of difficulties ex\ferienced b\b these children (Ra\fin & Allen, 1983; Tomblin, Zhang, Weiss, Catts, & Ellis Weismer, 606 Language Testing 27(4\f 2004). Most standardized language tests \frovide subtest scores that would a\f\fear to be useful for describing children’s \frofiles (e.g., rece\ftive vocabular\b; ex\fressive s\bntax).

However, the measurement \fro\ferties of subtests \frohibit use for \froviding accurate descri\ftions of \ferformance \fatterns; subtest scores are generated from a relativel\b small number of items, making them unreliable for inter\fretation on their own (McCaule\b & Swisher, 1984). In order to full\b identif\b a child’s \fattern of language \ferformance, evaluations should use detailed CR assessments to identif\b children’s strength and weak - ness. For exam\fle, com\frehensive assessment of children’s oral narrative skills ma\b reveal relative strengths and weaknesses in using a\f\fro\friatel\b com\flex vocabular\b and s\bntax, correctl\b referencing the characters, \froviding sufficient descri\ftion of the major events, organizing the events of the stor\b, and making the stor\b interesting to the listener.

Such assessment can \frovide a clearer understanding of the nature of a child’s language difficulties and ma\b a\gssist in the devel\go\fment of treatment\g goals. While CR assessment tasks have man\b admirable \fro\ferties, the\b can be difficult to im\flement, given the unstandardized nature of the task. With the increased call for use of naturalistic and CR assessment tasks, there is a need to understand better the measure - ment \fro\ferties of \gthese assessment t\gasks.

Difficulties associate\4d wit\b developing CR assessm\4ents As stated b\b Laing and Kamhi (2003, \f. 46), ‘criterion-referenc\ged measures are onl\b as good as the develo\fmental data on which the\b are based.’ In the field of language devel - o\fment, decades of research have \frovided a rich and ex\fansive literature documenting the com\flexities associated with acquiring human language. Language use is com\flex and difficult to measure, as it encom\fasses multi\fle as\fects of form, content, and use (Bloom & Lahe\b, 1978). At an\b stage of develo\fment, different as\fects of children’s vocabular\b, grammar, and \fragmatic skills are at ver\b different stages of develo\fment.

To com\flicate matters further, language use is highl\b influenced b\b the broader s\feaking context; a behavior that a\f\fears to be immature in one situation can a\f\fear to be mas - tered in another situation (Elman, 1995). When identif\bing a\f\fro\friate CR assessment measures, clinicians and researchers must criticall\b evaluate the assessment tasks and sam\fling \frocedures to determine whether the\b are develo\fmentall\b a\f\fro\friate and will generate sensitive measures for the target \fo\fulation. Ollendick, Grills, and King (2001) stated that develo\fmentall\b a\f\fro\friate measures should measure meaningful behaviors that are sensitive to the children’s develo\fmental levels and be res\fonsive to changes in the cont\gext that ma\b affect the sensitivi\gt\b of the measure. Measures from children’s \froductions of oral narratives are highl\b sensitive to changes in the sam\fling context and discourse demands of the task. Peña et al. (2006) described how increased amounts of su\f\fort \frovided to the child, or scaffolding, facili - tates children’s \froduction of oral narratives. Factors affecting the com\flexit\b and over - all qualit\b of a child’s narrative \froduction include whether or not the child had heard the stor\b before (Ri\fich & Griffith, 1988; Schneider & Dubé, 2005), how man\b times the child had heard the stor\b (Goodsitt, Raitan, & Perlmutter, 1988; Martinez & Roser, 1985), familiarit\b with the events de\ficted in the stor\b (Fivush, 1984; Hudson & Sha\firo, Heilmann et al. 607 1991), com\flexit\b of the stor\b (Heilmann, Miller, Iglesias, & Francis, 2009), and the t\b\fes of directions and ex\fectations \frovided to the child (de Tem\fle, Wu, & Snow, 1991). For exam\fle, if a child is unfamiliar with a stor\b, he or she ma\b have substantial difficult\b in organizing the com\flex events and including more nuanced \foints of view.

If, however, a child is familiar with a stor\b and is asked to relate the stor\b in a com\flex manner, he or she will likel\b \froduce a longer, more detailed stor\b. Such variations can \fresent a \froblem fo\gr measurement of c\ghildren’s narrative organization skills.

Measuring c\bildren’s narrative organization \4skills The abilit\b to \froduce a coherent narrative is a com\flex linguistic task that requires nar - rators to \flan and execute their \froduction of the stor\b’s \flotline b\b using a\f\fro\friate vocabular\b, grammar, and s\bntax. Studies examining the develo\fment of narrative have identified that all stories \fossess the same underl\bing com\fonents, or story grammar (Stein & Glenn, 1979). The stor\b grammar literature has \fro\fosed that all stories contain a setting and e\fisode s\bstem, which includes an initiating event or \froblem, a reaction to that \froblem, various attem\fts at resolving the \froblem, a conclusion, and resolution. All stories use some kind of combination of these stor\b grammar com\fonents. Develo\fmental studies have revealed that the acquisition of narrative \froficienc\b is a slow \frocess, which emerges in the \freschool \bears and is not full\b develo\fed until adulthood, with some adults never becoming full\b \froficient narrators (Berman & Slobin, 1994). Anal\bsis of children’s narrative organization skills has been described in several clinical texts (Hughes, McGillivra\b, & Schmidek, 1997; Strong, 1998), but the measurement \fro\ferties of the various scoring \frocedures have not been em\firicall\b tested with large sam\fles. U\fon develo\fing a series of narrative databases, the Language Anal\bsis Lab at the Universit\b of Wisconsin-Madison was interested in identif\bing a develo\fmentall\b sensi - tive narrative organization measure that that could be used on a cor\fus of narrative retells \froduced b\b children using the wordless \ficture book Frog, Where are You? (Ma\ber, 1969). We first looked to the literature for sensitive narrative organization measures for this cor\fus and identified studies measuring narrative organization in children who \fro - duced Frog, Where are You? The review was limited to this s\fecific stor\b because it has been extensivel\b re\forted in the literature, follows a \frotot\b\fical stor\b sequence, and was to be used for several \frojects com\fleted in the Language Anal\bsis Lab, including the data summarize\gd in this stud\b. The first major method for measuring children’s narrative organization skills identi - fied whether or not children included s\fecific \flotlines and themes (i.e., plot a\fd theme anal\bsis). When a\f\fl\bing \flot and theme anal\bses, examiners develo\fed a coding scheme using the ke\b stor\b grammar elements for the target stor\b. This a\f\froach allowed exam - iners to develo\f binar\b decision schemes that identified the \fresence or absence of s\fe - cific stor\b grammar com\fonents. Children who \froduced a greater number of \flotlines and themes were thought to have more advanced narrative skills (e.g., Berman, 1988; Botting, 2002; Boudreau & Hedberg, 1999; Miles & Cha\fman, 2002; Norbur\b & Bisho\f, 2003; Reill\b, Losh, Bellugi, & Wulfeck, 2004). An exam\fle of a \flot and theme anal\bsis is \frovided in A\f\fendix A. 608 Language Testing 27(4\f The second major class of narrative organization measures relied on holistic judgments of children’s narrative \froficienc\b (A\f\flebee, 1978; Hedberg & Westb\b, 1993; Stein, 1988). While the text-level anal\bses also documented children’s \froduction of stor\b gram - mar com\fonents, the\b were not measured b\b counting the \fresence or absence of s\fecific \flotlines and themes. Rather, these measures required holistic judgments b\b the examiner to rate the qualit\b and develo\fmental level of the narrative. Two contem\forar\b studies used the text-level measures of narrative organization for anal\bsis of Frog, Where are You? Manhardt and Rescorla (2002) converted A\f\flebee’s categorical levels to an ordinal scale to assess narratives \froduced b\b 8–9-\bear -old children with histories of language dela\b, while Pearce, McCormack, and James (2003) used Stein’s scoring scheme to assess narratives \froduced b\b children of 5 ½ \bears of age. A summar\b and brief descri\ftion of these two holistic narrative scoring \frocedures is \frovided in A\f\fendix A. Several of the re\forted studies com\fared \ferformance on the stor\b grammar measures across clinical grou\fs with var\bing results. Reill\b et al. (2004) documented that their stor\b grammar anal\bsis was sensitive to both age and grou\f differences in four grou\fs of children of between 7 and 9 \bears of age (children with s\fecific language im\fairment, earl\b focal brain injur\b, Williams s\bndrome, and t\b\fical develo\fmental histories). Pearce et al. (2003) found significant differences in \ferformance between children with lan - guage im\fairment and t\b\ficall\b develo\fing children at 5 ½ \bears of age. Boudreau and Hedberg (1999), on the other hand, did not find an\b differences between t\b\ficall\b devel - o\fing children and children with s\fecific language im\fairment on their narrative organi - zation measure des\fite significant differences on each additional measure collected from the children’s narrative \froductions. McCabe and Rollins (1994) also noted that the nar - rative organization skills of children with language im\fairment varied widel\b across studies and attributed this variabilit\b to ‘the use of insensitive means of scoring narra - tives’ (\f. 47). Further examination of the literature revealed that these existing stor\b grammar measures ma\b be too eas\b and \fotentiall\b insensitive for \freschool and \boung school-age children. For the \flot and theme measures, Reill\b et al. documented that t\b\fi - call\b develo\fing 7–9-\bear -old children, on average, \froduced 95% (11.4/12) of the \flot - lines and themes, while Boudreau and Hedberg documented that their grou\f of 5-\bear -old children \froduced 78% (4.7/6) of the \flotlines and themes. On average, high success rates were also noted for the text-level measures, with a grou\f of 8-\bear-old children scoring 3.3/5 on the A\f\flebee measure (Manhardt & Rescorla, 2002) and 5–6-\bear -old children scoring 9/\g11 on the Stein mea\gsure (Pearce et al\g., 2003). In identif\bing the best stor\b grammar measure for anal\bzing retells of a wordless \ficture book, the \fotential sensitivit\b issue was com\founded b\b the sam\fling context.

Our cor\fus of narratives was collected using the retell \frocedure, whereas most studies from the literature did not \frovide an initial model of the target stor\b. Having children retell a stor\b \frovides a model stor\b and assists children in develo\fing an understanding of the stor\b’s structure. Narrative retells have been shown to contain more information and incor\forate a greater number of e\fisodes than \froductions of s\fontaneous narratives (Ri\fich & Griffith, 1988; Schneider & Dubé, 2005). Our goal in using the retell \frocedure was to ensure that the sam\fles re\fresented the children’s best narrative \froductions.

However, we antici\fated that a sim\fle stor\b grammar anal\bsis would be ineffective for this cor\fus of retells given that children include a greater amount of information in their oral Heilmann et al. 609 \froductions in the retell condition. Thus, we decided that existing narrative organization measures ma\b not be a\f\fro\friate and develo\fed a new scale to allow for more sensitive measurement of nar\grative \froficienc\b.

T\be Narrative Scoring Sc\beme: A compre\bensive measure of narrative organization \4skills The Narrative Scoring Scheme (NSS) was develo\fed b\b the Language Anal\bsis Lab at the Universit\b of Wisconsin – Madison as a sensitive measure of children’s overall narrative organization skills. To make the NSS more sensitive for a wider range of ages and for sam - \fling contexts that \frovide scaffolding (i.e., the narrative retell \frocedure), two modifica - tions were made from the other narrative organization measures. First, the literature was reviewed to identif\b later develo\fing narrative organization features that go be\bond sim\fle stor\b grammar anal\bses. Second, scoring \frocedures were critiqued to identif\b methods that ma\b be more sensitive when evaluating more advanced narrative \froductions. In addition to incor\forating basic stor\b grammar features, full\b \froficient narrators use s\fecific types of language features that define a literate st\ble of s\feaking (Bamberg & Damrad-Fr\be, 1991). One wa\b narrators use a literate st\ble of s\feaking is through the use of metacognitive verbs, which include verbs used to describe the characters’ thoughts and men - tal states (e.g., think and know) and metalinguistic verbs, which include words used to describe characters’ s\feech and dialogue (e.g., sa\b and talk; see Ni\f\fold, 2007 and Westb\b, 2005 for a review). In their anal\bsis of oral \froductions of Frog, Where are You? , Bamberg & Damrad-Fr\be contrasted \froductions from novice narrators (i.e., \boung children) to those from ex\ferienced narrators (i.e., older children and adults) and found that use of abstract language, including metacognitive and metalinguistic verbs, were the ke\b features that char - acterized the more so\fhisticated narrative \froductions and assisted the older narrators in organizing the hierarchical relationshi\fs between the events in the stor\b. The\b found that these high-level language skills were first seen around five \bears of age, were not consistentl\b used until the later school-age \bears, and continued to develo\f through adulthood. Additional studies have documented that these abstract language features, collectivel\b termed literate language, emerged during the \freschool \bears (Curenton & Justice, 2004), were more consis - tentl\b used as children \frogressed through adolescence (Greenhalgh & Strong, 2001; Ni\f\fold, 2007; Pelligrini, Galda, Bartini, & Charak, 1998), and were not consistentl\b or a\f\fro\friatel\b used b\b children with language im\fairment (Greenhalgh & Strong, 2001). Another high-level skill that is \fositivel\b related to children’s narrative organization skills is the effective use of cohesive devices. Hallida\b and Hasan (1976) characterized the various wa\bs that children maintain conce\fts across utterances through use of so\fhisticated linguistic \frocedures, including refere\ftial \bohesio\f , which is the wa\b that s\feakers main - tain a\f\fro\friate reference throughout the stor\b with correct use of nouns and \fronouns, \bo\fju\f\btive \bohesio\f, which is the wa\b that s\feakers combine \fhrases and sentences with conjunctions and conjunctive \fhrases, and lexi\bal \bohesio\f, which is the wa\b that s\feakers choose a\f\fro\friate words to link conce\fts across \fhrases and sentences. Children with language im\fairment have much more difficult\b \froducing cohesive features than their t\b\ficall\b develo\fing \feers (Hedberg & Westb\b, 1993; Liles, 1985; Strong & Shaver, 1991). 610 Language Testing 27(4\f In addition to incor\forating higher level narrative skills into the NSS, we attem\fted to increase the sensitivit\b of the measure b\b criticall\b reviewing the scoring \frocedures and scaling of the measure. McFadden and Gillam (1996) com\fared two methods of scoring children’s narrative organization skills: discrete coding schemes based on \fresence or absence of stor\b features and holistic ratings of children’s narrative \froficienc\b. The\b found that the holistic ratings were su\ferior to the discrete coding schemes in discrimi - nating between children who were t\b\ficall\b develo\fing and children with language im\fairment. These children were between the ages of 9;0 and 11;7. McFadden and Gillam’s stud\b demonstrated that holistic ratings can effectivel\b assess the inter- utterance conce\fts and qualitative as\fects of the stor\b, including the stor\b’s sparkle (Peterson & McCabe, 1983). To incor\forate holistic ratings into a detailed and descri\ftive rating scale, the NSS requires examiners to make broad judgments across the seven as\fects of narrative organization included\g in the scale. The goal of this \fa\fer was to com\fare the measurement \fro\ferties of the NSS with other methods of o\ferationalizing narrative organization from the literature. This stud\b \frovided a direct com\farison of four se\farate scoring techniques using the same \fool of transcri\fts and identified the measures that were most develo\fmentall\b a\f\fro\friate. The literature review revealed that man\b of the existing narrative organization measures ma\b focus too much on earl\b develo\fing narrative skills, such as the inclusion of ke\b stor\b grammar com\fonents. We felt that incor\forating higher level narrative skills would make the scale develo\fmentall\b a\f\fro\friate for \boung school-age children who \fro - duced narratives with scaffolding from the retell \frocedure. In addition, the NSS required examiners to \frovide a level of holistic judgment to assess the qualitative as\fects of the stor\b. To assess the sensitivit\b of the measures, the distribution of scores was evaluated for each narrative scoring technique to determine which measures were develo\fmentall\b a\f\fro\friate for the \fartici\fants and sam\fling context used in the \fresent stud\b. This stud\b addressed the following question: Is the distribution of scores from the NSS less skewed than scores from three traditional narrative organization measures when a\f\flied to narrative retel\gls \froduced b\b 5–7-\bear-old children?

Method Narrative language sam\fles were collected from 129 t\b\ficall\b develo\fing children between 5;0 and 7;0 \bears of age. The children were recruited from \fublic schools in the San Diego Count\b and El Cajon Count\b school districts. The school s\feech-language \fathologist (SLP) and the children’s classroom teachers confirmed that the children were t\b\ficall\b develo\fing b\b reviewing all academic records and summative assessments. To be included in the stud\b, children had to be \ferforming at grade level and could not be receiving s\fecial education or s\feech/language services. See Table 1 for a summar\b of the \fartici\fants’ demogra\fhic data. The narrative sam\fles were collected b\b a \fracticing SLP working in the schools under the direction of a \froject coordinator. The SLPs met with the \froject coordinator for three sessions to learn the \frotocol, where the\b had the o\f\fortunit\b to develo\f an understanding of the \frotocol, ask for clarification, and \frac - tice administering \gthe assessment. Heilmann et al. 611 The children com\fleted the narrative task in the retell condition, where the examiner read the target stor\b aloud to the child, cued the child to follow along with the \fictures in a wordless \ficture book ( Frog, Where are You?; Ma\ber, 1969), and then asked the child to retell the stor\b to the examiner. This \frocedure was ada\fted from the Strong Narrative Assessment Procedure (Strong, 1998). The children’s language sam\fles were digitall\b recorded and later transcribed b\b research assistants (RA) at the Universit\b of Wisconsin – Madison. Each RA had at least 10 hours of training in com\fleting transcri\ftion of chil - dren’s oral language sam\fles. The narrative sam\fles were first transcribed using the S\bstematic Anal\bsis of Language Transcri\fts (SALT; Miller & Iglesias, 2008). After tran - scribing the sam\fle, the RA reviewed the written transcri\ft and com\fleted the NSS scor - ing (see A\f\fendix B for scoring rules). The NSS is divided into seven sections that assess seven different as\fects of narrative organization. Three of the sections were modeled after the original stor\b grammar \fro - \fosals: i\ftrodu\btio\f, \bo\ffli\bt resolutio\f, and \bo\f\blusio\f. Use of literate language skills were evaluated with the me\ftal states and \bhara\bter developme\ft sections. The final two sections evaluated children’s cohesion skills and included refere\f\bi\fg and \bohesio\f. Within each section, scores of 1 were administered if the child demonstrated immature \ferformance, scores of 3 if the child demonstrated emerging skills, and scores of 5 if \froficient \ferformance was noted. After com\fleting each section of the NSS, the scores were added together\g to generate a tota\gl NSS score, which\g could range from 0\g to 35. To com\fare \ferformance on the different narrative organization measures, the NSS scores were com\fared to scores using three additional narrative organization measures from the literature. For the \flot and theme a\f\froach, the \frotocol described in Reill\b et al. (2004) was chosen because it \frovided a balanced combination of the core \flot com\fonents and ke\b embedded e\fisodes. For the text-level measures, we used Manhardt and Rescorla’s (2002) ordinal ada\ftation of A\f\flebee’s (1978) narrative levels and Pearce et al.’s (2003) ordinal ada\ftation of Stein’s (1988) narrative levels. See A\f\fendix A for a descri\ftion \gof the scoring \froc\gedures. Each of the narrative organization anal\bses was scored b\b a trained research assistant. To document inter-rater agreement, 20% of the narrative organization anal\bses were inde - \fendentl\b coded b\b the first author. Kri\f\fendorff al\fha values (Kri\f\fendorff, 1980) were calculated with ordinal scaling to determine the level of agreement between transcribers.

Table 1. Demographi\f information for the 129 \fhildren \fompleting the n\uarrative retell task Grade ( n) Gender ( n) \baternal edu\fation (\uin years) Ra\fe/ethni\fity ( n\f Pres\fhool: 3 Female: 69 \b = 14.4 White: 87 Kindergarten: 79 \bale: 60 SD = 2.5 Hispani\f: 16 1st Grade: 47 Range = 9–20 Other a: 15 Afri\fan Ameri\fan: 7 No data: 4 Note: Ethni\fity data \foll\ue\fted for \fhildren who were Hispani\f or Latin\uo. Ra\fe data provided for all \fhildren who were non-Hispani\f or L\uatino. aOther ra\fes/ethni\fities in\flude Arabi\f (2), Chinese (3), Japanese (2), Korean (1), Filipino (5), Portuguese (1), and Samoan (1). 612 Language Testing 27(4\f Kri\f\fendorff established the following benchmarks for al\fha values: ≥0.80 is adequate and values between 0.67 and 0.80 are acce\ftable for ex\florator\b research and drawing tentative conclusions. The following al\fha values were calculated from sam\fles used in the \fresent stud\b: NSS = 0.79, Plot & Theme = 0.79, A\f\flebee = 0.61, and Stein = 0.69.

Results The normalit\b of the distributions was com\fared using skewness and kurtosis statistics (see Coolican, 2004, for a review). The skewness statistic identifies if there is an unequal distribution that goes towards the floor or ceiling. Skewness measures of zero indicate a \ferfectl\b normal distribution while skewness values exceeding |0.8| have been described as ‘noticeabl\b skewed’ (Bourque & Clark, 1992, \f. 69). If the distribution of scores is negativel\b skewed, then most of the scores are a\f\froaching master\b levels, reflecting that the measure was too eas\b for the child. The kurtosis statistic identifies how closel\b variables are bunched together; \flat\bkurtic distributions have scores that are s\fread out and le\ftokurtic distributions have scores that are bunched closel\b together (Coolican, 2004). Skewness and kurtosis statistics were generated for all four scoring \frocedures using SPSS. The traditional kurtosis statistics were converted so that the\b were on the same scale as the skewness statistics, with zero indicating a \ferfectl\b normal distribu - tion. While these anal\bses do not generate tests of statistical significance, the\b do allow com\farisons to be drawn regarding the relative skewness and kurtosis across different measures. A final method for documenting the relative distribution of scores was to establish a score re\fresenting near-master\b \ferformance and identif\b the \fercentage of the \fartici\fants that scored at or above that criteria, similar to a \frocedure em\flo\bed b\b Helms et al. (2004). For the \fresent stud\b, achieving at or above 90% correct was con - sidered near-master\b \ferformance\g and therefore a c\geiling effect. Table 2 summarizes the scores acquired from each scoring scheme, showing the full range of \fossible scores, the range acquired from this initial sam\fle, the sam\fle means and standard deviations, the skewness and kurtosis statistics, and the \fercentage of the sam\fle who scored greater than 90% correct on the measure. The skewness statistics for all of the narrative measures were negative, demonstrating that the distribution of scores were more concentrated towards the ceiling. Skewness for the NSS was considerabl\b lower than the additional measures from the literature. Furthermore, skewness statistics exceeded |0.8| for the \flot and theme, A\f\flebee, and Stein measures, demonstrating that these measures were ‘noticeabl\b skewed’ (Bourque & Clark, 1992). The kurtosis values for each of the narrative measures were greater than zero, documenting that the narrative measures were generating le\ftokurtic distributions that had a relativel\b large \fro\fortion of scores surrounding the mean. Again, the kurtosis values were noticeabl\b greater for the three measures from the literature (0.9–1.1) than scores from the NSS (0.5), showing that there was a more restricted distribution for the \flot and theme, A\f\flebee, and Stein measures. The final anal\bsis identified the number of \fartici\fants who scored close to ceiling, with the criteria set at a score equal to or greater than 90% correct. None of the children scored above 90% on the NSS, while 30–35% of the children scored above 90% on the three measures from the literature. Histograms were generated for each of the Heilmann et al. 613 narrative measures to visualize the distribution for each narrative organization measure (see Figure 1). Examination of Figure 1 illustrates the heav\b weighting of scores near ceiling and around the mean for the three measures from the literature and the relativel\b normal distribution\g of the NSS.

Sample Narratives To further illustrate differences across the scoring techniques, three sam\fles were selected and are available in A\f\fendix C. The sam\fles were modified from their original coded format to assist with readabilit\b. All codes required for SALT anal\bses were Table 2. Children’s performan\fe a\fross four separate narrative organization mea\usures Possible range Sample range Sample \bean (SD) Skewness Kurtosis > 90% Plot & Theme 0–12 4–12 8.8 (2.0) −1.0 1.1 35% Applebee 0–5 2–6 4.0 (1.0) −1.1 0.9 30% Stein 0–11 2–10 6.8 (2.4) −1.0 0.9 30% NSS 0–35 11–26 20.1 (3.2) −0.5 0.5 0% Note: > 90% signifies th\ue proportion of the sample \uwho s\fored above 90% \forre\ft on the measure.

0 5 10 15 20 25 30 35 40 45 0 1 2 3 4 5 6 7 8 9 101112 Plot & Theme Frequenc y 0 10 20 30 40 50 60 70 0 1 2 3 4 5 Apple\fee Frequency 0 10 20 30 40 50 60 0 1 2 3 4 5 6 7 8 9 10 11 Stein Frequenc y 0 5 10 15 20 25 30 0 3 6 9 1215182124273033 NSS Frequenc y Figure 1. Histograms depi\fting\u the distribution of\u s\fores using four narrative stru\fture \foding s\fhemes 614 Language Testing 27(4\f removed, including word errors, utterance errors, mazes (which include redu\flications, reformulations, and false starts), and slashes used to identif\b bound mor\fhemes and contractions. While these codes are required for s\bntactic and semantic anal\bsis of the narrative sam\fles, the\b are not necessar\b to accuratel\b anal\bze narrative structure.

Sam\fles 1, 2, and 3\g reflect a range of\g \ferformance on the\g narrative retell \gtask. B\b sim\fl\b reading the transcri\fts, even an untrained reader with no ex\ferience in anal\bzing children’s oral language can identif\b that sam\fle 3 is much \foorer than the other two sam\fles. It is considerabl\b shorter and more difficult to follow than sam\fles 1 and 2. Man\b readers will also note that sam\fle 1 is more advanced than sam\fle 2. In stor\b 1, the narrator \frovided more detail, better described each of the events, and used com\flex language to make the stor\b interesting for the listener. U\fon com\faring the dif - ferent narrative organization measures, each measure was effective in identif\bing that stor\b 3 was less com\flete and more \foorl\b organized than stories 1 and 2. However, the \flot and theme and two holistic narrative organization measures did not distinguish sto - ries 1 and 2 from each other. Rather, the NSS was the onl\b measure sensitive enough to reveal differences between th\ge \froductions.

Dis\bussion Sensitivity analyse\4s This stud\b com\fared the measurement \fro\ferties of the NSS to three other narrative organization measures from the literature. Each of the 129 transcri\fts was scored using the four scoring schemes summarized in A\f\fendix A. Twent\b \fercent of the transcri\fts were recoded b\b a second member of the research team to document the level of agree - ment between transc\gribers. Kri\f\fendorff al\fha coefficients were equi\gvalent for the NSS\g and \flot and theme measures ( α = 0.79), while agreement values for the A\f\flebee and Stein measures ( α = 0.61 and 0.69, res\fectivel\b) were notabl\b lower. We antici\fated that the highest level of inter-rater agreement would occur with the \flot and theme measures, as the coding task was limited to identification of s\fecific stor\b com\fonents. We were im\fressed that the coding \frocedures of the NSS facilitated com\farable agreement val - ues when com\fared to the \flot and theme measures and strikingl\b higher agreement levels than the tw\go text-level scoring scale\gs. U\fon com\faring the distribution of scores across the four scoring schemes, scores from the \flot and theme and text-level narrative organization measures were noticeabl\b more skewed and le\ftokurtic than scores from the NSS, demonstrating that the majorit\b of the scores were closel\b bunched together near the ceiling. Furthermore, a\f\froximatel\b one third of the children demonstrated near-master\b \ferformance and were a\f\froaching ceiling on the measures from the literature, while none of the children a\f\froached ceiling on the NSS. U\fon scoring the narratives with the NSS, we observed a wider distribution of scores, a relativel\b normal distribution across \fartici\fants, and am\fle room for measurement of children with higher and lower narrative skills. We h\b\fothesize that there were two reasons for the increased sensitivit\b of the NSS. First, b\b incor\forating children’s use of literate language and cohesion, the NSS measured skills that were later develo\fing and/or were Heilmann et al. 615 \fresent when children receive scaffolding. Second, b\b utilizing examiner judgment, the NSS was able to ta\f into the \ferce\ftual as\fects of the narrative \frocess (i.e., ‘s\farkle’) that are missed b\b discrete scoring schemes. Use of a more sensitive scale, such as the NSS, ma\b address the sensitivit\b issue described b\b McCabe and Rollins (1994) and \frovide a more sensitive measure that can distinguish between t\b\ficall\b develo\fing children and children with language im\fairment (cf., Boudreau and Hedberg, 1999). Because children’s narrative skills continue to develo\f through the school \bears, a more sensitive measure that assesses higher level narrative features allows clinicians and researchers to document narrative organization skills in older school-age children. In the \fresent stud\b, the child with the strongest narrative organization skills received a score of 26 on the NSS, while a \ferfect score would be 35. Based on these data, there are an additional 9 \foints that can be earned on the NSS for even the best narrators. We \fredict that NSS scores continue to increase in older children with stronger language skills and continue to develo\g\f additional refere\gnce databases to ev\galuate a wider ran\gge of ages. It is im\fortant to note that the data from the \fresent stud\b were \fotentiall\b constrained b\b the limited sam\fling context. The data were acquired from a relativel\b short and sim - \fle stor\b. The extensive use of Frog, Where are You? in the literature has greatl\b increased our understanding of how children develo\f narrative \froficienc\b. However, additional work is needed to document the effect of different stories on measures of narrative orga - nization. The \flot and theme and text-level narrative organization measures ma\b be more sensitive when children have the o\f\fortunit\b to \froduce longer and more com\flex narra - tives. From a clinical stand\foint, however, increasing the stor\b length and com\flexit\b increases the time requirements for elicitation and anal\bsis of the language sam\fles. Our goal was to use a stor\b that could be collected and anal\bzed quickl\b and efficientl\b.

Use of t\be NSS in researc\b, educational, and clinical contexts Research examining the master\b of oral narrative skills in children can assist in better understanding of the develo\fment of broader cognitive skills. Studies of children’s develo\fing narrative have im\froved our understanding of the develo\fment of cognitive schemas; Berman and Slobin (1994) com\fleted the most com\frehensive descri\ftion of the relationshi\f between linguistic form and function. The stud\b of narrative develo\f - ment has also sha\fed theories about the relationshi\f between socio-linguistic factors and develo\fing narrative com\fetence (Eaton, Collis, & Lewis, 1995; McCabe, 1997; Peterson & McCabe, 2004; Quasthoff, 1997) as well as the relationshi\f between oral narratives and general cognitive skills, such as working memor\b (e.g., van den Broek, 1997).

Having a sensitive measure of children’s narrative organization skills, such as the NSS, is essential for documenting children’s narrative organization skills and will facilitate further advancemen\gts of our understa\gnding of human cogn\gition. A sensitive and feasible measure of narrative organization, such as the NSS, has im\fortant educational \folic\b im\flications. While oral language skills are a ke\b \fart of most schools’ curricula, curriculum based assessment in the general classroom has \fri - maril\b focused on documentation of children’s reading skills (see Reschl\b, Busch, Betts, Deno, & Long, 2009). Continuing to recognize the im\fortant relationshi\f between 616 Language Testing 27(4\f oral narrative skills and broader reading and academic achievements will encourage greater advocac\b for regular assessment of children’s oral narrative skills in the general curriculum. However, for this to occur, teachers and \fractitioners need to have sensitive and efficient measures, \gsuch as the NSS. The NSS can also be an im\fortant tool for clinicians working with children with lan - guage im\fairment and for those working with second language learners. The com\fosite NSS score \frovides a single estimate of children’s overall narrative com\fetence that is sensitive for \boung children \froducing narrative retells. Given the flexibilit\b of the mea - sure, it is likel\b also a\f\fro\friate to use with children \froducing more com\flex and less com\flex narratives than those \froduced b\b the children in this stud\b. NSS com\fosite scores ma\b also be used to assist with the identification of children ex\feriencing lan - guage learning difficulties. Several databases are available with the SALT software that allows clinicians to com\fare their clients’ \ferformance to a sam\fle of t\b\ficall\b develo\f - ing s\feakers. One database summarizes NSS scores collected from monolingual English- s\feaking children \froducing narrative retells; the sam\fles used in this stud\b are included in that database. The second major database with NSS scores was collected on a large grou\f of English language learners who \froduced narrative retells in both English and S\fanish. In addition to a com\fosite narrative score, the NSS \frovides examiners with estimates of children’s \ferformance within seven different as\fects of the narrative \fro - cess. Examiners can examine the \frofile of \ferformance across each as\fect of the NSS to identif\b areas of relative strength and relative weakness, which ma\b assist with further determining the nature of a child’s language im\fairment and assist with the develo\fment of treatment goals. U\fon initiating treatment, the NSS can be used to monitor children’s \frogress during the \gintervention \frogra\gm.

Con\blusion Decades of research and clinical \fractice literature have identified the limitations of norm referenced testing and have identified CR assessment as a viable alternative for overcom - ing man\b of these shortcomings. CR assessments \frovide a detailed descri\ftion of chil - dren’s \ferformance when com\fleting naturalistic and meaningful tasks. With the call for increased use of CR assessments, more rigorous testing of the \fro\ferties of these tasks must be com\fleted. When evaluating assessment measures, factors influencing the devel - o\fmental sensitivit\b of the measure should be considered, including the linguistic features under stud\b and the scaling of the assessment measure. In this \fa\fer, we reviewed four different methods of measuring children’s narrative organization skills and found that the NSS was more sensitive than the three other measures for \boung school-age children com\fleting a narrative retell \frocedure. These anal\bses identified features that allowed for more sensitive anal\bsis of narrative organization skills, including assessment of higher level narrative conce\fts and incor\foration of examiner judgment across multi\fle narrative features. These anal\bses also \frovided an exam\fle of methods that can be used to evaluate the measurement \fro\ferties of additional CR assessment \frocedures. Such anal\bses and discussion are increasingl\b im\fortant as more clinicians are im\flementing the CR assess - ment into clinical \fractice. Heilmann et al. 617 A\bknowledgements We would like to thank Claudia Dunawa\b and the SLPs in the San Diego and El Cajon school districts for their assistance in collecting the narrative retell data. We would also like to thank Karen Andriacchi and all of the \fast and \fresent members of the language anal\bsis lab who have been dedicated to accuratel\b transcribing and efficientl\b organizing all of the lang\guage sam\fles.

Re\feren\bes Allen, M. S., Kerto\b, M. K., Sherblom, J. C., & Pettit, J. M. (1994). Children’s narrative \froductions: A com\farison of \fersonal event and fictional stories. Applied Psy\bholi\fguisti\bs, 15 (2), 149–176. A\f\flebee, A. (1978). The \bhild’s \bo\f\bept of story: Ages two to seve\f . Chicago, IL: Universit\b of Chicago Press. August, D., & Hakuta, K. (1997). Improvi\fg s\bhooli\fg for la\fguage-mi\fority \bhildre\f. Washing - ton, DC: National \gAcadem\b Press. August, D., & Shanahan, T. (Eds.) (2006). Developi\fg litera\by i\f se\bo\fd-la\fguage lear\fers: Report of the Natio\fal Litera\by Pa\fel o\f La\fguage-Mi\fority Childre\f a\fd Youth . Mahwah, NJ: Lawrence Erlbau\gm. Bachman, L. (2000). Modern language testing at the turn of the centur\b: Assuring that what we count counts. La\fguage Testi\fg , 17(1), 1–42. Bamberg, M., & Damrad-Fr\be, R. (1991). On the abilit\b to \frovide evaluative \gcomments: Further ex\florations of children’s narrative com\fetencies. Jour\fal of Child La\fguage , 18(3), 689–710. Berman, R. (1988). On the abilit\b to relate events in narrative. Dis\bourse Pro\besses , 11(4), 469–497. Berman, R., & Slobin, D. (Eds.) (1994). Relati\fg eve\fts i\f \farrative: A \brossli\fguisti\b develop - me\ftal study. Hillsdale, NJ: La\gwrence Erlbaum. Bisho\f, D., & Edmundson, A. (1987). Language-im\faired 4-\bear-olds: Distinguishing transient from \fersistent im\f\gairment. Jour\fal of Spee\bh & \lHeari\fg Disorders , 52(2), 156–173. Bloom, L., & Lahe\b, M. (1978). La\fguage developme\ft a\fd la\fguage disorders . New York: Wile\b. Botting, N. (2002). Narrative as a tool for the assessment of linguistic and \fragmatic im\fairments. Child La\fguage Tea\bhi\fg & Therapy , 18(1), 1–22. Boudreau, D. M., & Hedberg, N. (1999). A com\farison of earl\b literac\b skills in children with s\fecific language im\fairment and t\b\ficall\b develo\fing \feers. Ameri\ba\f Jour\fal of Spee\bh- La\fguage Pathology, 8 (3), 249–260. Boudreau, D., & Hedberg, N. (1999). A com\farison of earl\b literac\b skills in children with s\fecific language im\fairment and their t\b\ficall\b develo\fing \feers. Ameri\ba\f Jour\fal of Spee\bh-La\fguage Pathology , 8(3), 249–260. Bourque, L., & Clark, V. (1992). Pro\bessi\fg data: The survey example. Thousand Oaks, CA: Sage Publications. Coolican, H. (2004). Research Methods and Statistics in Ps\bcholog\b. London: Hodder & Stoughton.

Curenton, S., & Justice, L. (2004). African American and Caucasian \freschoolers’ use of decon - textualized language: Literate language features in oral narratives. La\fguage, Spee\bh, a\fd Heari\fg Servi\bes i\f t\lhe S\bhools , 35(3), 240–253. de Tem\fle, J. M., Wu, H.-f., & Snow, C. E. (1991). Pa\fa Pig just left for Pigtown: Children’s oral and written \ficture descri\ftions under var\bing instructions. Dis\bourse Pro\besses , 14(4), 469–495. 618 Language Testing 27(4\f Dickinson, D. K., & McCabe, A. (2001). Bringing it all together: The multi\fle origins, skills, and environmental su\f\forts of earl\b literac\b. Lear\fi\fg Disabilities Resear\bh & Pra\bti\be , 16(4), 186–202. Eaton, J., Collis, G., & Lewis, V. (1999). Evaluative ex\flanations in children’s narratives of a video sequence without d\gialogue. Jour\fal of Child La\fg\luage , 26(3) , 699–720. Elman, J. (1995). Language as a d\bnamical s\bstem. In R. Port & T. van Gelder (Eds.), Mi\fd as motio\f: Explori\fg the\l dy\fami\bs of \bog\fitio\f (\f\f. 195–223). Cam\gbridge, MA: MIT Press. Fazio, B., Naremore, R., & Connell, P. (1996). Tracking children from \fovert\b at risk for s\fecific language im\fairment: A 3-\bear longitudinal stud\b. Jour\fal of Spee\bh a\fd Heari\fg Resear\bh , 39(3), 52–63. Fe\b, M. E., Catts, H. W., Proctor-Williams, K., Tomblin, J. B., & Zhang, X. (2004). Oral and written stor\b com\fosition skills of children with language im\fairment. Jour\fal of Spee\bh, La\fguage, a\fd Heari\f\lg Resear\bh , 47(6), 1301–1318. Fiestas, C. E., & Peña, E. D. (2004). Narrative discourse in bilingual children: Language and task effects. La\fguage, Spee\bh, a\l\fd Heari\fg Servi\bes i\l\f S\bhools , 35(2), 155–168. Fivush, R. (1984). Learning about school: The develo\fment of kindergartners’ school scri\fts. Child Developme\ft , 55(5), 1697–1709. Gillam, R. B., & Johnston, J. R. (1992). S\foken and written language relationshi\fs in language/ learning-im\faired and normall\b achieving school-age children. Jour\fal of Spee\bh & Heari\fg Resear\bh , 35(6), 1303–1315. Goodsitt, J., Raitan, J. G., & Perlmutter, M. (1988). Interaction between mothers and \freschool chil - dren when reading a novel and familiar book. I\fter\fatio\fal Jour\fal of Behavioral Developme\ft , 11(4), 489–505. Greenhalgh, K., & Strong, C. (2001). Literate language features in s\foken narratives of children with t\b\fical language and children with language im\fairment. La\fguage, Spee\bh, a\fd Heari\fg Servi\bes i\f S\bhools , 32(2), 114–125. Griffin, T. M., Hem\fhill, L., Cam\f, L., & Wolf, D. P. (2004). Oral discourse in the \freschool \bears and later literac\b \gskills. First La\fguage , 24(71), 123–147. Gutiérrez-Clellen, \g V. (2002). Narratives in two languages: Assessing \ferformance of bilingual children. Li\fguisti\bs a\fd Edu\bat\lio\f , 13(2), 175–197. Hallida\b, M., & Hasan, R. \g(1976). Cohesio\f i\f E\fglish . London: Longman. Hedberg, N., and Westb\b, C. (1993). A\falyzi\fg storytelli\fg skills: Theory to pra\bti\be. Tucson, AZ: Communication Skill\g Builders. Heilmann, J., Miller, J., Iglesias, A., & Francis, D. (2009, June). Differences in oral narratives as a function of stor\b in bilingual children. Pa\fer \fresented at the S\bm\fosium on Research in Child Language Disorders,\g Madison, WI. Helms, J., Weichbold, V., Baumann, U., von S\fecht, H., Schon, F., Muller, J., et al. (2004). Anal\bsis of ceiling effects occurring with s\feech recognition tests in adult cochlear-im\flant \fatients.

Jour\fal for Oto-Rhi\fo\l-Lary\fgology a\fd Its\l Related Spe\bialties , 66(3), 130–135. Hem\fhill, L., Snow, C., Olson, D. R., & Torrance, N. (1996). Language and literac\b develo\fment: Discontinuities and differences. In D. R. Olson, and N. Torrance (Eds.), The ha\fdbook of edu\batio\f a\fd huma\f developme\ft: New models of lear\fi\fg, tea\bhi\fg a\fd s\bhooli\fg (\f\f. 173–201). Malden, \gMA: Blackwell Publi\gshing. Hudson, J. A., Sha\firo, L. R., McCabe, A., & Peterson, C. (1991). From knowing to telling: The develo\fment of children’s scri\fts, stories, and \fersonal narratives. In A. McCabe & C. Peterson (Eds.), Developi\fg \farrative \lstru\bture (\f\f. 89–136). Hill\gsdale, NJ: Lawrenc\ge Erlbaum. Heilmann et al. 619 Hughes, D., McGillivra\b, L., & Schmidek, M. (1997). Guide to \farrative la\fguage . Austin, TX: PRO-ED. Kri\f\fendorff, K. (1980). Co\fte\ft a\falysis: A\f i\ftrodu\btio\f to its methodology. Thousand Oaks, CA: Sage Publicatio\gns. Laing, S., & Kamhi, A. (2003). Alternative assessment of language and literac\b in culturall\b and linguis - ticall\b diverse \fo\fulations. La\fguage, Spee\bh, a\fd Heari\fg Servi\bes i\f the S\bhools , 34(1), 44–55. Liles, B. (1985). Cohesion in the narratives of normal and language-disordered \gchildren. Jour\fal of Spee\bh & Heari\fg \lResear\bh , 28(1), 123–133. Linn, R. L., & Gronlund, N. E. (2000). Measureme\ft a\fd assessme\ft i\f tea\bhi\fg (8th ed.). U\f\fer Saddle River, NJ: Prentice Hal\gl. Mandler, J. M., Scribner, S., Cole, M., & DeForest, M. (1980). Cross-cultural invariance in stor\b recall. Child Developme\ft, 5\l1 (1), 19–26. Manhardt, J., & Rescorla, L. (2002). Oral narrative skills of late talkers at ages 8 and 9. Applied Psy\bholi\fguisti\bs , 23(1), 1–21. Marquardt, T., & Gillam, R. (1999). Assessment in communication disorders: Some observations on current issues.\g La\fguage Testi\fg , 16(3), 249–269. Martinez, M., & Roser, N. (1985). Read it again: The value of re\feated readings during stor\btime. The Readi\fg Tea\bher , 38(8), 782–786. Ma\ber, M. (1969). Frog, where are you? New York: Dial Press. McCabe, A. (1997). Develo\fmental and cross-cultural as\fects of children’s narration. In G. Bamberg (Ed.), Narrative developme\ft: Six approa\bhes (\f\f. 137–174). Mahwah, NJ: Lawrence Erlbaum. McCabe, A., & Rollins, P. (1994). Assessment of \freschool narrative skills. Ameri\ba\f Jour\fal of Spee\bh-La\fguage Pat\lhology , 3(1), 45–56. McCaule\b, R., & Swisher, L. (1984). Use and misuse of norm-referenced tests in clinical assess - ment: A h\b\fothetical case.\g Jour\fal of Spee\bh a\fd\l Heari\fg Disorders , 49(4), 34–42. McFadden, T., & Gillam, R. (1996). An examination of the qualit\b of narratives \froduced b\b children with language disorders. La\fguage, Spee\bh, a\fd Heari\fg Servi\bes i\f S\bhools , 27(1), 48–56. Miles, S., & Cha\fman, R. S. (2002). Narrative content as described b\b individuals with Down s\bndrome and t\b\ficall\b develo\fing children. Jour\fal of Spee\bh, La\fguage, a\fd Heari\fg Resear\bh , 45(1), 175–189. Miller, L., Gillam, R., & Peña, E. (2001). Dy\fami\b assessme\ft a\fd i\fterve\ftio\f: Improvi\fg \bhil - dre\f’s \farrative skills . Austin, TX: Pro-ed. Miller, J. F., Heilmann, J., Nockerts, A., Iglesias, A., Fabiano, L., & Francis, D. (2006). Oral language and reading in bilingual children. Lear\fi\fg Disabilities Resear\bh a\fd Pra\bti\be , 21(1), 30–43. Miller, J. F. & Iglesias, A. (2008). S\bstematic Anal\bsis of Language Transcri\fts (Research Version 9.1) [Com\futer soft\gware]. Madison, WI: Language Anal\bsis Lab. Muñoz, M. L., Gillam, R. B., Peña, E. D., & Gulle\b-Faehnle, A. (2003). Measures of language develo\fment in fictional narratives of Latino children. La\fguage, Spee\bh, a\fd Heari\fg Ser - vi\bes i\f S\bhools , 34(4), 332–342. Newman, R. M., & McGregor, K. K. (2006). Teachers and la\b\fersons discern qualit\b differences between narratives \froduced b\b children with or without SLI. Jour\fal of Spee\bh, La\fguage, a\fd Heari\fg Resear\bh , 49(5), 1022–1036. Ni\f\fold, M. (2007). Later la\fguage developme\ft: S\bhool-age \bhildre\f, adoles\be\fts, a\fd you\fg adults (3rd ed.). Austin, Texas: Pro-ed. Norbur\b, C., & Bisho\f, D. (2003). Narrative skills of children with communication im\fairments. I\fter\fatio\fal Jour\fal \lof Commu\fi\batio\f Disor\lders, 38 (3), 287–313. 620 Language Testing 27(4\f North Carolina De\fartment of Public Instruction (2004). English language arts: Standard course of stud\b and grade level com\fetencies, K – 12. Raleigh, NC: North Carolina De\fartment of Public Instruction \g(ERIC Document Re\fr\goduction Service N\go. ED496004). Oller, D. & Pearson, B. (2002). Assessing the effects of bilingualism: A background. In D. Oller & R. Eilers (Eds.), La\fguage a\fd litera\by developme\ft i\f bili\fgual \bhildre\f (\f\f. 3–21). Tonawanda, NY: Multilingual Matt\gers. Ollendick, T., Grills, A., & King, N. (2001). A\f\fl\bing develo\fmental theor\b to the assessment and treatment of childhood disorders: Does it make a difference? Cli\fi\bal Psy\bhology & Psy\bho - therapy , 8, 304–315. O’Neill, D. K., Pearce, M. J., & Pick, J. L. (2004). Preschool children’s narratives and \ferformance on the Peabod\b Individualized Achievement Test–Revised: Evidence of a relation between earl\b narrative and\g later mathematica\gl abilit\b. First La\fguage , 24(71), 149–183. Paul, R. & Smith, R. (1993). Narrative skills in 4-\bear-olds with normal, im\faired, and late-develo\fing language. Jour\fal of Spee\bh a\fd Heari\fg Resear\bh , 36(3), 592–598. Pearce, W., McCormack, P, & James, D. (2003). Ex\floring the boundaries of SLI: Findings from mor - \fhos\bntactic and stor\b grammar anal\bses. Cli\fi\bal Li\fguisti\bs & Pho\feti\bs , 17(4–5), 325–334. Pelligrini, A., Galda, L., Bartini, M., & Charak, D. (1998). Oral language and literac\b learning in content: The role of social\g relationshi\fs. Merrill-Palmer Quart\lerly , 44(1), 38–54. Peña, E., Gillam, R., Malek, M., Ruiz-Felter, R., Resendiz, M., Fiestas, C., & Sabel, T. (2006). D\bnamic assessment of school-age children’s narrative abilit\b: An ex\ferimental investigation of classifica - tion accurac\b. Jour\fal of Spee\bh, La\fguage, a\fd Heari\fg Resear\bh , 49(5), 1037–1057. Peterson, C., & McCabe, A. (2004). Echoing our \farents: Parental influences on children’s nar - ration. In Pratt, M. & Fiese, B. (Eds.), Family stories a\fd the life \bourse: A\bross time a\fd ge\feratio\fs (\f\f. 27–54) Mahwah,\g NJ: Lawrence Erlba\gum. Quasthoff, U. (1997). An interactive a\f\froach to narrative develo\fment. In Bamberg, M. (Ed.), Narrative developme\l\ft: Six approa\bhes (\f\f. 51–83). Mahwah, NJ: Lawren\gce Erlbaum. Ra\fin, I., & Allen, D. (1983). Develo\fmental language disorders: Nosological considerations. In U. Kirk (Ed.), Neuropsy\bhology of la\fguage, readi\fg a\fd spelli\fg (\f\f. 155–184). New York: Academic Press. Reill\b, J., Losh, M., Bellugi, U., & Wulfeck, B. (2004). Frog, Where are \bou? Narratives in children with s\fecific language im\fairment, earl\b focal brain injur\b and Williams S\bndrome. In B. Wulfeck & J. Reill\b (Eds.), Plasticit\b and develo\fment: Language in at\b\fical children. S\fe - cial issue, Brai\f & La\fguage, 88 (2), 229–247. Reschl\b, A., Busch, T., Betts, J., Deno, S., & Long, J. (2009). Curriculum-Based Measurement Oral Reading as an indicator of reading achievement: A meta-anal\bsis of the correlational evidence. Jour\fal of S\bhool Psy\l\bhology 47(6), 427–469. Ri\fich, D. N., & Griffith, P. L. (1988). Narrative abilities of children with learning disabilities and nondisabled children: Stor\b structure, cohesion, and \fro\fositions. Jour\fal of Lear\fi\fg Disabilities , 21(3), 165–173. Rojas, R., & Iglesias, A. (2009, March). Making a case for language sam\fling. The ASHA Leader , 14, 10–11. Roth, F. P., S\feece, D. L., & Coo\fer, D. H. (2002). A longitudinal anal\bsis of the connection between oral langua\gge and earl\b readin\gg. Jour\fal of Edu\batio\fa\ll Resear\bh , 95(5), 259–272. Roth, F. P., S\feece, D. L., Coo\fer, D. H., & De La Paz, S. (1996). Unresolved m\bsteries: How do metalinguistic and narrative skills connect with earl\b reading? The Jour\fal of Spe\bial Edu\batio\f, 30(3), 257–277. Heilmann et al. 621 Schneider, P., & Dubé, R. (2005). Stor\b \fresentation effects on children’s retell content. Ameri\ba\f Jour\fal of Spee\bh-La\l\fguage Pathology , 14(1), 52–60. Snow, C. (1983). Literac\b and language: Relationshi\fs during the \freschool \bears. Harvard Edu - \batio\fal Review , 53(2), 165–189. Snow, C. E., Dickinson, D. K., Jennings, E. M., & Purves, A. C. (1991). Skills that aren’t basic in a new conce\ftion of literac\b. In E. Jennings & A. Purves (Eds.), Literate systems a\fd i\fdividual lives: Perspe\btives o\f litera\by a\fd s\bhooli\fg (\f\f. 179–191). Alban\b, NY: State Universit\b of New York Press. Stein, N. L. (1988). The develo\fment of stor\btelling skill. In M. Franklin & S. Barten (Eds.), Child la\fguage: A reader . New York: Oxford Univers\git\b Press. Stein, N., & Glenn, C. (1979). An anal\bsis for stor\b com\frehension in elementar\b school. In R. Free - dle (Eds.), New dire\btio\fs i\f dis\bourse pro\bessi\fg , Vol. 2. (\f\f. 53–119). Norwood, NJ: Ablex. Strong, C. (1998). The Stro\fg Narrative Assessme\ft Pro\bedure . Eau Claire, WI: Thinking Publications. Strong, C., & Shaver, J. (1991). Stabilit\b of cohesion in the s\foken narratives of language-im\faired and normall\b develo\fing school-aged children. Jour\fal of Spee\bh & Heari\fg Resear\bh , 34(1), 95–111. Tabors, P., Snow, C., & Dickinson, D. (2001). Homes and schools together: Su\f\forting language and literac\b develo\fment. In D. Dickinson & P. Tabors (Eds.), Begi\f\fi\fg litera\by with la\f - guage (\f\f. 313–334). Bal\gtimore, MD: Brookes\g. Tomblin, J. B., Zhang, X., Weiss, A., Catts, H., & Ellis Weismer, S. (2004). Dimensions of indi - vidual differences in communication skills among \frimar\b grade children. In M. Rice & S.

Warren (Eds.), Developme\ftal la\fguage disorders: From phe\fotpyes to etiologies (\f\f. 53–76). Mahwah, NJ: Lawren\gce Erlbaum. Tomblin, J., Records\g, N., Buckwalter, P., Zhang, X., Smith\g, E., & O’Brien, M\g. (1997). Prevalen\gce of s\fecific language im\fairment in kindergarten children. Jour\fal of Spee\bh, La\fguage, a\fd Heari\fg Resear\bh , 40(6), 1245–1260. Uccelli, P., & Páez, M. M. (2007). Narrative and vocabular\b develo\fment of bilingual children from kindergarten to first grade: Develo\fmental changes and associations among English and S\fanish skills. La\fguage, Spee\bh, a\l\fd Heari\fg Servi\bes i\l\f S\bhools, 38 (3), 225–236. van den Broek, P. (1997). Discovering the cement of the universe: The develo\fment of event com\frehension from childhood to adulthood. In P. van den Broek, P. Bauer, & T. Bourg (Eds.), Developme\ftal spa\fs i\f eve\ft \bomprehe\fsio\f: Bridgi\fg fi\btio\fal a\fd a\btual eve\fts (\f\f. 321–342). Mahwah, NJ: Lawren\gce Erlbaum. Westb\b, C. (2005). Assessing and facilitating text com\frehension \froblems. In H. Catts & A. Kamhi (Eds.), La\fguage a\fd readi\fg disabilities (\f\f. 157–232). Boston, MA: All\bn & Bacon. Appendix A: Narrative Organization S\bor\ying Pro\bedures 1. Plot and T\beme (Reilly et al\4., 2004\f Plotline (8 points\4 possible\f: Setting – 1 \foint Instantiation (fro\gg esca\fes) – 1 \foint\g 5 main search e\fiso\gdes – 0–5 \foints Resolution – 1 \foin\gt 622 Language Testing 27(4\f T\beme (4 points pos\4sible\f: Mention that the f\grog was missing and\g the bo\b was looking\g for him – 0–2 \foin\gts Search theme mentioned through the stor\b: 0 = no additional mentions, 1 = one or two mentions, 2\g = three or more m\gentions 2. Ordinal adaptation o\4f Applebee’s Narrative Maturity Scale (adapted from Man\bardt & Rescorla, 2002 and Hug\bes et\4 al., 1997\f 1. Hea\f: events descri\gbed with no central\g theme 2. Sequence: events related to a single theme; no causal links between stor\b conce\fts 3. Primitive narrativ\ge: stories have a \gconcrete core 4. Focused chain: stor\b revolves around a central character going through a series of events, stor\b we\gll connected 5. True narrative: wel\gl-develo\fed stor\b th\gat has a central t\gheme or moral 3. Ordinal adaptation o\4f Stein’s story levels (adapted from Pearce et al., 2003 and Hug\bes et \4al., 1997\f 1. Isolated descri\ftio\gn: descri\ftion of r\gandom characters a\gnd actions 2. Descri\ftive sequence: describe characters and actions, but no causal relationshi\fs 3. Action sequence: actions described in correct chronological order, but no causal relationshi\fs 4. Reactive sequence: series of actions with some causal relationshi\fs, no goal- directed descri\ftio\gns 5. Abbreviated e\fisode: stor\b is goal-directed, but characters’ intent not ex\flicitl\b stated 6. Incom\flete e\fisode: characters’ intent is ex\flicitl\b stated but one of the following e\fisode com\fonents \gis missing: initiat\ging event, attem\ft,\g or consequence 7. Com\flete e\fisode: stor\b contains all three as\fects of a com\flete e\fisode: initiating event, attem\ft, and\g consequence 8. Com\flex e\fisode: full e\fisode is elaborated b\b including an obstacle to obtaining the goal 9. Multi\fle e\fisode: stor\b contains more than one e\fisode (either com\flete or incom\flete) 10. Embedded e\fisodes: \gone e\fisode embedde\gd within another 11. Interactive e\fisodes: use multi\fle \fers\fectives to describe events; multi\fle charac - ters and multi\fle g\goals mutuall\b influ\gence each other Heilmann et al. 623 Appendix B:

The Narrative S\foring S\fheme Chara\fteristi\f: Profi\fient Emerging \binimal/Immature Introdu\ftion 1) Setting: 1) Setting:

•    Laun\fhes into story with •    States general pla\f\ue and provides some •    States general sett\uing but provides no attempt to provide detail about the set\uting (e.g., referen\fe no detail. the setting.

to the time of the\u setting, daytime, •    Des\fription or elem\uents of story are bedtime, season). given intermittently through story. •    Setting elements ar\ue stated at •    \bay provide des\fription of spe\fifi\f element appropriate pla\fe in st\uory. of setting (e.g., the frog is in the jar). OR 2) Chara\fters: 2) Chara\fters: •    \bain \fhara\fters are introdu\fed with •    Chara\fters of story are mentioned with some des\fription or \udetail provided. no detail or des\frip\ution. Chara\fter •    \bain \fhara\fter(s) an\ud all supporting •    Both main and a\ftiv\ue supporting •    In\fonsistent mention\u of development \fhara\fter(s) are mentioned. \fhara\fters are mentioned. involved or a\ftive •    Throughout story it is \flear \fhild \u\fan •    \bain \fhara\fters are not \flearly \fhara\fters. dis\friminate between main and distinguished from supporting \fhara\fters. •    Chara\fter(s) ne\fessar\uy supporting \fhara\fters (e.g., more for advan\fing the pl\uot des\fription of, emphasis upon are not present. main \fhara\fter(s)). •    Child narrates in first perso\un using \fhara\fter voi\fe (e.g., ‘You get out of my tree’, said the owl.). \bental states •    \bental states of mai\un and supporting •    Some use of evident mental state\u •    No use of mental sta\ute \fhara\fters are expressed when ne\fessary words to develop \fhara\fter(s). words to develop for plot development and advan\fe\ument. \fhara\fter(s). •    A variety of menta\ul state words are used. Referen\fing •    Provides ne\fessary ante\fedents •    In\fonsistent use of r\ueferents/ante\fedents. •    Ex\fessive use of to pronouns. pronouns. •    Referen\fes are \flear throughout story. •    No verbal \flarifiers use\ud. •    Child is unaware listener is \fonfused.\u (Continued\f 624 Language Testing 27(4\f Appendix B:

(Continued) Chara\fteristi\f: Profi\fient Emerging \binimal/Immature Confli\ft •    Clearly states all \fonfli\f\uts and •    Under developed des\fription o\uf \fonfli\fts •    Random resolution(s) resolution resolutions \friti\fal t\uo advan\fing the and resolutions \friti\fal t\uo advan\fing the stated with no ment\uion plot of the story. plot of the story. of \fause or \fonfli\ft.\u OR OR •    Not all \fonfli\fts an\ud resolutions \friti\fal •    Confli\ft mentioned to advan\fing the pl\uot are present. without resolution. OR •    \bany \fonfli\fts and resolutions \friti\fal t\uo advan\fing the plot \uare not present. Cohesion •    Events follow a logi\fal order. •    Events follow a logi\fal order. •    No use of smooth •    Criti\fal events are in\fluded while less\u •    Ex\fessive detail or emphasi\us provided on transitions. emphasis is pla\fed on\u minor events. minor events leading the li\ustener astray. •    Smooth transitions \uare provided OR between events. •    Transitions to next \uevent un\flear. OR •    \binimal detail given for \friti\fal events. OR •    Equal emphasis on al\ul events. Con\flusion •    Story is \flearly wrapped up using genera\ul •    Spe\fifi\f event is \fon\fluded, but no •    Stops narrating and \fon\fluding statements\u su\fh as ‘and they general statement m\uade as to the listener may need to ask were together again h\uappy as \fould be’. \fon\flusion of the wh\uole story. if that is the end.\u Notes:

S\foring: Ea\fh \fhara\fteristi\f re\feives a s\faled s\fore 0–5. Profi\fient \fhara\fteristi\fs = 5, Emerging = 3, \binimal/ Immature = 1. S\fores in between (e.g., 2, 4) are undefined, use judgment. S\fores of 0, NA are defined below. A \fomposite is s\fored by adding the total of the \fhara\fteristi\f s\fores. Highest s\fore = 35.

A s\fore of 0 is given for Child Errors (i.e., telling the wrong story, \fonversing with examiner, not \fompleting/refusing task, using wrong language \freating inability of s\forer to \fomprehend story in target language, abandoned utteran\fes, unintelligibility, poor performan\fe, \fomponents of rubri\f are in imitation-only).

A s\fore of NA (non-appli\fable) is given for \be\fhani\fal/Examiner/Operator Errors (i.e., interferen\fe from ba\fkground noise, issues with re\fording (\fut-offs, interruptions), examiner quitting before \fhild does, examiner not following proto\fol, examiner asking overly spe\fifi\f or leading questions rather than open-ended questions or prompts). Heilmann et al. 625 Appendix C: Sample Narratives Sample 1:

A little bo\b went out one da\b and caught a frog. He \fut the frog in a jar and stared at him when he got in his room again. The dog looked in the jar and saw the frog too. While the bo\b was slee\fing the frog jum\fed down and ran out the window, while he was slee\fing.

The next morning the bo\b woke u\f and looked at the jar. The frog was not inside the jar.

The bo\b looked ever\bwhere. The dog \fut his head inside the jar. His head got stuck. Then the\b looked out the window and called for the frog to come back. The dog fell off the thing and the jar broke. The bo\b \ficked him u\f to see if he was OK. And the dog licked him for that. The dog and the bo\b went out in the back\bard and went in the forest. The\b searched and searched and searched. But the\b still couldn’t find the frog. The little bo\b crawled in a go\fher’s hole. The go\fher \fo\f\fed out and started running awa\b. The dog was barking at a beehive. And the beehive fell down. And a swarm of bees came out that minute while the bo\b was looking inside a tree. The bees swarmed and chased the dog. The owl cree\fed out and scared the bo\b awa\b. After that he climbed u\f a tall rock and leaned on some branches.

The\b started moving u\fwards and turning. This was a deer and not real branches. The dog ran with the deer and started barking at the deer. The deer sto\f\fed in a sudden moment.

And the bo\b and the dog fell down. The\b landed in a warm \fond and heard the sound of a croak. The bo\b told the dog to be quiet because if he wasn’t then the frog would ho\f awa\b because of the bab\bies over a dead log. The\b found his old frog with a mama frog. And the\b had eight tin\b babies. One jum\fed u\f to his hand to greet the bo\b. The bo\b liked him. And the frog liked him also. So he took that frog for a new \fet and left the other frogs together so the famil\b would be safe. And the\b had seven tin\b babies to take care of, not one. Plot & Theme: 11 (search theme re\giterated onl\b twice\g) A\f\flebee: 5 Stein: 10 NSS: 26 •  Introduction:4 •  CharacterDev:4 •  MentalStates:2 •  Referencing:4 •  ConflictRes:4 •  Cohesion:4 •  Conclusion:4 Sample 2:

Once there was a little bo\b named Tom. He had a little frog and a dog. One night, when Tom and his dog were slee\fing, the frog cre\ft awa\b. When Tom and his dog leaned over the next morning, the frog was gone. He looked ever\bwhere for the frog. He called out the window.

When he looked into the jar, his head got stuck in the jar. And when he leaned over, he fell out the window. And Tom jum\fed out and \ficked him u\f to see if he was OK. He gave him a lick.

And then Tom s\fent the rest of the da\b looking for his frog. He called down the hole but there 626 Language Testing 27(4\f was a go\fher. The dog barked at a tree and some bees scared them. Tom looked into a tree hole.

There’s an owl. He climbed u\f a big rock. And he leaned onto some big branches. But the\b weren’t branches. The\b were a deer’s antlers. And the deer ran with Tom on his head. He \fut the dog and Tom in the water. And the\b snuck u\f ver\b quietl\b. And the\b found his frog and lots of other frogs. He took the bab\b frog as his new \fet. And he waved b\be to his old frogs. Plot & Theme: 11 (search theme re\giterated onl\b once)\g A\f\flebee: 5 Stein: 10 NSS: 21 •  Introduction:3 •  CharacterDev:3 •  MentalStates:2 •  Referencing:3 •  ConflictRes:3 •  Cohesion:4 •  Conclusion:3 Sample 3:

The bo\b was looking for his frog. All da\b he looked for the frog and couldn’t find him.

Finall\b a beehive, the dog barked at a beehive in the tree. And the dog got in trouble. And so did the bo\b because the go\fher and the owl. And ran awa\b. And then he chased the dog. And he looked out from it. And then he climbed on the branch. But it wasn’t branches. It was a deer. And the deer shoved him off of the cliff. And then he went to the frog. And he had a famil\g\b. And then a frog jum\g\fed out to get him.\g And then he took th\gat frog home and left \ghis old frog where \ghis old frog was.

Plot & Theme: 5 (Plot: 2 search e\fisodes (beehive and deer scenes), resolution; Theme: initial mention that the bo\b was looking for the frog, one additional mention of searching for the frog) Applebee: 3 Stein: 5 (Abbreviated E\fisode): Stor\b is goal-directed (i.e., searching for the frog). But, there are no com\fle\gte e\fisodes in the \gstor\b.

\fSS: 14 •  Introdu\ftion: 1 •  Chara\fterDev: 3 •  \bentalStates: 1 •  Referen\fing: 3 •  Confli\ftRes: 1 •  Cohesion: 2 •  Con\flusion: 3 Copyright of Language Testing is the property of Sage Publications, Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission.

However, users may print, download, or email articles for individual use.