Summary Analysis: Quality Indicator Instructions Overview Using Odom et al. (2005) and Cook et al. (2015), the candidate will read and analyze the quality indicators presented by the Council for Exc

Summary Analysis: Quality Indicator Template

What is the Purpose of Quality Indicators?

Quality Indicator: 1.0. Context and Setting

QI

1.0. Context and setting: “The study provides sufficient information regarding the critical features of the context or setting.”

Component

“1.1. The study describes critical features of the context or setting relevant to the review; for example, type of program or classroom, type of school (e.g., public, private, charter, preschool), curriculum, geographic location, community setting, socioeconomic status, physical layout.”

After thoughtful consideration, what is one question regarding the implication of the context and setting quality indicator?

Word count:________

Quality Indicator: 2.0. Participants

QI

2.0. Participants: "The study provides sufficient information to identify the population of participants to which results may be generalized and to determine or confirm whether the participants demonstrated the disability or difficulty of focus."

Component

"2.1. The study describes participant demographics relevant to the review (e.g., gender, age/grade, race/ethnicity, socioeconomic status, language status)."

"2.2. The study describes disability or risk status of the participants (e.g., specific learning disability, autism spectrum disorder, behavior problem, at risk for reading failure) and method for determining status (e.g., identified by school using state IDEA criteria, teacher nomination, standardized intelligence test, curriculum-based measurement probes, rating scale)."

After thoughtful consideration, what is one question regarding the implication of the participants quality indicator?

Word count:________

Quality Indicator: 3.0. Intervention Agent

QI

3.0. Intervention agent: "The study provides sufficient information regarding the critical features of the intervention agent."

Component

"3.1. The study describes the role of the intervention agent (e.g., teacher, researcher, paraprofessional, parent, volunteer, peer tutor, sibling, technological device/computer) and, as relevant to the review, background variables (e.g., race/ethnicity, educational background/licensure). "

"3.2. The study describes any specific training (e.g., amount of training, training to a criterion) or qualifications (e.g., professional credential) required to implement the intervention, and indicates that the interventionist has achieved them."

After thoughtful consideration, what is one question regarding the implication of the intervention quality indicator?

Word count:________

Quality Indicator: 4.0. Description of Practice

QI

4.0. Description of practice: "The study provides sufficient information regarding the critical features of the practice (intervention), such that the practice is clearly understood and can be reasonably replicated."

Component

"4.1. The study describes detailed intervention procedures (e.g., intervention components, instructional behaviors, critical or active elements, manualized or scripted procedures, dosage) and intervention agents’ actions (e.g., prompts, verbalizations, physical behaviors, proximity), or cites one or more accessible sources that provide this information."

"4.2. When relevant, the study describes materials (e.g., manipulatives, worksheets, timers, cues, toys), or cites one or more accessible sources providing this information."

After thoughtful consideration, what is one question regarding the implication of the description of practice quality indicator?

Word count:________

Quality Indicator: 5.0. Implementation Fidelity

5.0. Implementation fidelity: "The practice is implemented with fidelity."

Component

"5.1. The study assesses and reports implementation fidelity related to adherence using direct, reliable measures (e.g., observations using a checklist of critical elements of the practice)."

"5.2. The study assesses and reports implementation fidelity related to dosage or exposure using direct, reliable measures (e.g., observations or self-report of the duration, frequency, curriculum coverage of implementation)."

"5.3. As appropriate, the study assesses and reports implementation fidelity (a) regularly throughout implementation of the intervention (e.g., beginning, middle, end of the intervention period), and (b) for each interventionist, each setting, and each participant or other unit of analysis. If either adherence or dosage is assessed and reported, this item applies to the type of fidelity assessed. If neither adherence nor dosage is assessed and reported, this item is not applicable."

After thoughtful consideration, what is one question regarding the implication of the implementation fidelity quality indicator?

Word count:________

Quality Indicator: 6.0. Internal Validity

QI

6.0. Internal Validity: "The independent variable is under the control of experimenter. The study describes the services provided in control and comparison conditions and phases. The research design provides sufficient evidence that the independent variable causes change in the dependent variable or variables. Participants stayed with the study, so attrition is not a significant threat to internal validity."

Component

"6.1. The researcher controls and systematically manipulates the independent variable."

"6.2. The study describes baseline (single-subject studies) or control/comparison (group comparison studies) conditions, such as the curriculum, instruction, and interventions (e.g., definition, duration, length, frequency, learner: instructor ratio). "

"6.3. Control/comparison-condition or baseline-condition participants have no or extremely limited access to the treatment intervention."

"6.4. The study clearly describes assignment to groups, which involves participants (or classrooms, schools, or other unit of analysis) being assigned to groups in one of the following ways:

(a) randomly;

(b) nonrandomly, but the comparison groups are matched very closely to the intervention group (e.g., matched on prior test scores, demographics, a propensity score; see Song & Herman, 2010);

(c) nonrandomly, but techniques are used to measure differences and, if meaningful differences are identified—for example, statistically significant difference, difference greater than 5% of a standard deviation (What Works Clearinghouse, 2011)—to statistically control for any differences between groups on relevant pretest scores or demographic characteristics (e.g., statistically adjust for confounding variable through techniques such as ANCOVA or propensity score analysis); or

(d) nonrandomly on the basis of a reasonable cutoff point (regression discontinuity design).""

"6.5. The design provides at least three demonstrations of experimental effects at three different times."

"6.6. For single-subject research designs with a baseline phase (alternating treatment designs do not require a baseline), all baseline phases include at least three data points (except when fewer are justified by study author due to reasons such as measuring severe or dangerous problem behaviors and zero baseline behaviors with no likelihood of improvement without intervention) and establish a pattern that predicts undesirable future performance (e.g., increasing trend in problem behavior, consistently infrequent exhibition of appropriate behavior, highly variable behavior)."

"6.7. The design controls for common threats to internal validity (e.g., ambiguous temporal precedence, history, maturation, diffusion) so plausible, alternative explanations for findings can be reasonably ruled out. Commonly accepted designs such as reversal (ABAB), multiple-baseline, changing criterion, and alternating treatment address this quality indicator when properly designed and executed, although other approaches can be accepted if study authors justify how they ruled out alternative explanations for findings or control for common threats to internal validity."

"6.8. Overall attrition is low across groups (e.g., <30% in a 1-year study)."

"6.9. Differential attrition (between groups) is low (e.g., ≤10%) or is controlled for by adjusting for noncompleters (e.g., conducting intent-to-treat analysis)."

After thoughtful consideration, what is one question regarding the implication of the internal validity quality indicator?

Word count:________

Quality Indicator: 7.0. Outcome Measures/Dependent Variables

QI

7.0. Outcome measures/dependent variables: "Outcome measures are applied appropriately to gauge the effect of the practice on study outcomes. Outcome measures demonstrate adequate psychometrics."

Component

"7.1. Outcomes are socially important (e.g., they constitute or are theoretically or empirically linked to improved quality of life, an important developmental/learning outcome, or both)."

"7.2. The study clearly defines and describes measurement of the dependent variables."

"7.3. The study reports the effects of the intervention on all measures of the outcome targeted by the review (p levels and effect sizes or data from which effect sizes can be calculated for group comparison studies; graphed data for single-subject studies), not just those for which a positive effect is found."

"7.4. Frequency and timing of outcome measures are appropriate. For most single-subject studies, a minimum of three data points per phase is necessary if a given phase is to be considered as part of a possible demonstration of experimental effect (except when fewer are justified by study author due to reasons such as measuring severe or dangerous problem behaviors and zero baseline behaviors with no likelihood of improvement without intervention). For alternating treatment designs, at least four repetitions of the alternating sequence are required (e.g., ABABABAB; see Kratochwill et al., 2013)."

"7.5. The study provides evidence of adequate internal reliability, interobserver reliability, test-retest reliability, or parallel-form reliability, as relevant (e.g., score reliability coefficient ≥ .80, interobserver agreement ≥ 80%, kappa ≥ 60%)."

"7.6. The study provides adequate evidence of validity, such as content, construct, criterion (concurrent or predictive), or social validity."

After thoughtful consideration, what is one question regarding the implication of the outcome measures/dependent variables quality indicator?

Word count:________

Quality Indicator: 8.0. Data Analysis

QI

8.0. Data Analysis: "Data analysis is conducted appropriately. The study reports information on effect size."

Component

"8.1. Data analysis techniques are appropriate for comparing change in performance of two or more groups (e.g., t tests, ANOVAs / MANOVAs, ANCOVAs / MANCOVAs, hierarchical linear modeling, structural equation modeling). If atypical procedures are used, the study provides a rationale justifying the data analysis techniques."

"8.2. The study provides a single-subject graph clearly representing outcome data across all study phases for each unit of analysis (e.g., individual, classroom, other group of individuals) to enable determination of the effects of the practice. Regardless of whether the study report includes visual or other analyses of data, graphs depicting all relevant dependent variables targeted by the review should be clear enough for reviewers to draw basic conclusions about experimental control using traditional visual analysis techniques (i.e., analysis of mean, level, trend, overlap, consistency of data patterns across phases)."

"8.3. The study reports one or more appropriate effect size statistic (e.g., Cohen’s d, Hedge’s G, Glass’s ∆, η^2) for all outcomes relevant to the review being conducted, even if the outcome is not statistically significant, or provides data from which appropriate effect sizes can be calculated."

After thoughtful consideration, what is one question regarding the implication of the data anlaysis quality indicator?

Word count:________