Glossary: Personalized Research and Assessment Instructions Overview Vocabulary has been linked to academic, reading, and writing success in the post-secondary setting. Language and terminology deve

0% Current Progress 100% 100%

Check for Understanding

Congratulations! You answered 44 questions correctly, which means your score was 100%! This means you have met the recommended 90% or higher criterion before beginning a systematic literature review using the Group Comparison and Single-case Research Design Quality Indicator Matrix Using Council for Exceptional Children 2014 Standards. Enjoy! 100.0%

44/44

1/1

True or false, the Council for Exceptional Children Standards for Evidence-Based Practices in Special Education (hereafter referred to as Standards for EBP; CEC, 2014) is a quality appraisal tool to support the methodological evaluation and categorization of evidence-based practices in special education.

1. True

2. False

1/1

The Standards for EBP include quality indicators.

1. 7

2. 8

3. 11

4. 22

1/1

The Standards for EBP can be used for what type(s) of experimental designs?

1. Conceptual

2. Descriptive

3. Group comparison design

4. Single-case research design

5. Qualitative

1. 1. 1 and 2

   2. 1, 2, and 5

   3. 3 and 4

   4. 3 only

1/1

QI 1.1. The study describes critical features of the context or setting r elevant to the review, is considered met if:

1. Enough context/setting information is reported to determine whether the study should be included in the review

2. At least one context and one setting descriptor is reported

3. Critical features of the physical setting (e.g., number of tables and chairs) were described with sufficient precision to allow replication

4. At least one context and/or setting descriptor is reported

1/1

QI 2.1. The study describes participant demographics relevant to the review, is considered met if:

1. Disability or risk-status and method for determining status was reported for each participant

2. Age or grade, gender, race/ethnicity, socioeconomic status, or language status was reported for each participant

3. At least one demographic element for each participant or a range when a whole class was the unit of analysis was reported

4. Any of the above based on relevance of review

1/1

QI 2.2. The study describes disability or risk status of the participants and method for determining status, is considered met if:

1. Study defined participant’s disability or risk status, including the method of determination used (i.e., standardized assessment, interdisciplinary team)

2. Study reported participating students were identified by teacher nomination

3. Study reported participating students were identified by teacher nomination and included operational definition of nomination process (e.g., criteria to identify students for nomination)

4. Study reported disability status using IDEA classification or Diagnostic and Statistical Manual (DSM) diagnoses for each participant

1. 1. 1 and 2

   2. 1 and 3

   3. 2 and 3

   4. 3 and 4

1/1

QI 3.1. The study describes the role of the intervention agent and, as relevant to the review, background variables, is considered met if:

1. At least one background (demographic) variable (e.g., race/ethnicity, educational background/licensure) was provided

2. Study described intervention agent role (e.g., teacher, researcher, paraprofessional) OR at least one background (demographic) variable (e.g., race/ethnicity, educational background/ licensure)

3. Study described intervention agent role (e.g., teacher, researcher, paraprofessional) AND one or more background variables as relevant to the purpose of the review

4. Study included individual paragraph(s) in method section describing intervention agent

1/1

QI 3.2. The study describes any specific training or qualifications required to implement the intervention, and indicates that the interventionist has achieved them, is considered met if:

1. Study specified training was not required to implement the intervention

2. Study reported (or may be reasonably inferred) an author was the interventionist who designed the intervention (e.g., self-trained by virtue of designing)

3. Study described training, reported training was completed by each interventionist, and the training criterion each interventionist achieved

4. Reviewer considers the intervention reasonably implemented without specific training or qualifications

1. 1. Any of the above

   2. None of the above

1/1

True or false, for QI 4.1. The study describes detailed intervention procedures, is considered met if study does not describe the intervention procedures with sufficient detail to allow for replication but cites one or more accessible sources that provide this information.

1. True

2. False

1/1

True or false, for QI 4.2. When relevant, the study describes materials is not applicable if no materials were mentioned and none appeared to be needed to implement the practice.

1. True

2. False

1/1

QI 5.1. The study assesses and reports implementation fidelity related to adherence using direct, reliable measures, is considered met if:

1. The study assessed implementation fidelity using a direct, reliable measure (or similar use of quantifiable levels) and reported at minimum ≥ 80% levels of implementation fidelity

2. The study assessed implementation fidelity using a direct, reliable measure (or similar use of quantifiable levels) and reported levels of implementation fidelity

3. The study assessed implementation fidelity using a direct, reliable measure (or similar use of quantifiable levels) and reported at minimum ≥ 90% levels of implementation fidelity

4. The study assessed implementation fidelity using a direct, reliable measure (or similar use of qualitative levels)

1/1

QI 5.2. The study assesses and reports implementation fidelity related to dosage or exposure using direct, reliable

measures, may be considered met when dosage was not a reported measure but procedures were clearly described and graph is present allowing reasonable inference of dosage/intensity.

1. True

2. False

1/1

QI 5.3. The study assesses and reports implementation fidelity (a) regularly throughout implementation of the intervention, and (b) for each interventionist, each setting, and each participant or other unit of analysis, is considered met if fidelity of implementation was assessed throughout the intervention (i.e., across all phases, percentage of all sessions throughout the intervention).

1. True

2. False

1/1

QI 5.3. The study assesses and reports implementation fidelity (a) regularly throughout implementation of the intervention, and (b) for each interventionist, each setting, and each participant or other unit of analysis is not applicable (scored NA) if neither adherence (QI 5.1) nor dosage (QI 5.2) was assessed and reported.

1. True

2. False

1/1

QI 6.1. The researcher controls and systematically manipulates the independent variable, is considered met if:

1. The research design provided sufficient evidence that implementation of the practice was under the control of the researcher

2. The research design demonstrated improvements in student outcomes

3. The study assessed and reported implementation fidelity (i.e., QI 5.1 was met) and the research design provided sufficient evidence that implementation of the practice was under the control of the researcher

4. The study assessed and reported implementation fidelity (i.e., QI 5.1 was met) and reported adequate levels of implementation fidelity (e.g., ≥ 80%)

1/1

QI 6.2. The study describes baseline or control/comparison conditions, is only considered met if the study (a) clearly described baseline or control/comparison conditions with replicable precision and (b) assessed and reported implementation fidelity (i.e., QI 5.1 was met).

1. True

2. False

1/1

QI 6.3. Control/comparison-condition or baseline-condition participants have no or extremely limited access to the treatment intervention, is considered met when:

1/1

1. Exposure to the intervention was extremely unlikely for comparison groups or during baseline phases but was not explicitly stated

2. When a validated research design was used that included two or more comparison groups/baseline phases

3. It was explicitly stated the intervention was removed/withdrawn/returned to baseline conditions for A-B-A-B (withdrawal) designs

4. The study assessed and reported implementation fidelity (i.e., QI 5.1 was met)

1. 1. 1 only

   2. 3 only

   3. 1 and 3

   4. 1 and 4

QI 6.4 The study clearly describes assignment to groups, is considered met when authors report group assignment was:

1. 1. 1. Random

      2. Nonrandom, but the comparison groups were matched very closely to the intervention group

      3. Nonrandom, but techniques were used to measure differences and, if meaningful differences were identified, to statistically control for any differences between groups on relevant pretest scores or demographic characteristics

      4. Nonrandom on the basis of a reasonable cutoff point

1. 1. 1. 1. 1 only

         2. 1 and 4

         3. Any of the above

         4. None of the above

1/1

QI 6.5. The design provides at least three demonstrations of experimental effects at three different times, is considered met if the study design allowed for the possibility of three demonstrations of experimental effect, even if a functional relation was not established.

1. True

2. False

1/1

QI 6.6. For single-subject research designs with a baseline phase, all baseline phases include at least three data points and establish a pattern that predicts future performance, is not required under which conditions:

1. Case study designs, which do not require a baseline phase

2. When fewer data points are justified (e.g., ethical reasons)

3. Alternative treatment designs, which do not require a baseline phase

4. When justified by statistical procedures to predict future level, stability, and trend within baseline

1. 1. 2 only

   2. 3 only

   3. 2 and 3

   4. 2, 3 and 4

1/1

True or false, to meet QI 6.7. The design controls for common threats to internal validity, the design must also have assessed and reported implementation fidelity (i.e., QI 5.1 was met).

1. True

2. False

1/1

True or false, to meet QI 6.8. Overall attrition is low across groups, reviewers should consider the context of the studies reviewed when setting a criterion for attrition (e.g., < 30% in a 1-year study)

1. True

1. False

1/1

True or false, QI 6.9. Differential attrition (between groups) is low may be met when differential attrition exceeded reviewer criterion but study demonstrated it had no impact (e.g., demonstrated pretest equivalence of groups after attrition) or was controlled for by using intent-to-treat analysis.

1. True

2. False

1/1

QI 7.1. Outcomes are socially important, is considered met when:

1. The study used a social validity measure with favorable results

2. Determined by a justification or argument in the introduction or discussion supporting the intervention’s goals and/or outcomes

3. Determined through reviewers’ personal opinions regarding the social importance of the goals targeted, procedures used, or outcomes obtained

4. Determined through consensus model between reviewers and a third party not working on the review

1. 1. 1 only

   2. 2 only

   3. 1 and 2

   4. 3 and 4

1/1

True or false, for QI 7.2. The study clearly defines and describes measurement of the dependent variables, if a study measured multiple outcomes, reviewers should consider effects only for outcomes relevant to their review

1. True

2. False

1/1

QI 7.3. The study reports the effects of the intervention on all measures of the outcome targeted by the review, is considered met if:

1. Study reported p levels and effect sizes, or data from which effect sizes can be calculated for group comparison studies

2. Study reported graphed data and effect sizes, or data from which effect sizes can be calculated

3. Study reported findings for all outcomes relevant to the review

4. Study reported graphed data for single-case studies

1. 1. 1 only

   2. 2 only

   3. 1, 3 and 4

   4. Any of the above

1/1

True or false, QI 7.4. Frequency and timing of outcome measures are appropriate, is considered met following the same criteria for single-case or group comparison research designs.

1. True

2. False

1/1

True or false, QI 7.5. The study provides evidence of adequate internal reliability, interobserver reliability, test-retest

reliability, or parallel-form reliability, is considered met when typical criteria for acceptable reliability are met (e.g., score reliability coefficient ≥ .80, inter-observer agreement ≥ 80%, kappa ≥ 60%), but reviewers should specify criteria in the methodology of their review prior to beginning the review.

1/1

1. True

2. False

True or false, regarding QI 7.6. The study provides adequate evidence of validity, because validity can be assessed multiple ways and uniform standards do not exist for validity, QI 7.6 is considered met if either (a) study authors reported adequate validity coefficients (as determined a priori by the review team) or (b) outcomes adequately represented content measured (i.e., content validity; as justified by study authors or as determined by reviewers).

1. 1. True

   2. False

1/1

True or false, regarding QI 8.1. Data analysis techniques are appropriate for comparing change in performance of two or more groups, is considered met when (a) researchers used statistical analyses generally recognized as appropriate for comparing change in the performance of two or more groups, or (b) atypical procedures were used but justified and explained (or publicly available references used) by researchers.

1. True

2. False

1/1

True or false, regarding QI 8.2. The study provides a single-subject graph clearly representing outcome data across all study phases for each unit of analysis to enable determination of the effects of the practice, requires the demonstration of an experimental effect, not just the possibility of an experimental effect.

1. True

2. False

1/1

True or false, regarding QI 8.3. The study reports one or more appropriate effect size statistic, is considered met if study did not provide effect sizes but provided data from which appropriate effect sizes can be calculated by the reviewer.

1. True

2. False

1/1

When setting up the Excel file, the first step is to:

1. Enter participant N, effects, and experiment type on the QI Coding Summary tab

2. List each study's author(s), year, and title on the QI Coding Summary tab in columns B-D

3. List each study's author(s), year, and title on the CEC 2014 Quality Indicators tab across the top, replacing "Reference 1" "Reference 2" etc.

4. Enter reviewer one (R1) and reviewer two (R2) initials on the CEC 2014 Quality Indicators tab, replacing the red R1 and R2 cells, and score all articles

1/1

When setting up the Excel file, the second step is to:

1. Enter participant N, effects, and experiment type on the QI Coding Summary tab

2. List each study's author(s), year, and title on the QI Coding Summary tab in columns B-D

3. List each study's author(s), year, and title on the CEC 2014 Quality Indicators tab across the top, replacing "Reference 1" "Reference 2" etc.

4. Enter reviewer one (R1) and reviewer two (R2) initials on the CEC 2014 Quality Indicators tab, replacing the red R1 and R2 cells, and score all articles

1/1

When setting up the Excel file, the third step is to:

1. Enter participant N, effects, and experiment type on the QI Coding Summary tab

1/1

1. List each study's author(s), year, and title on the QI Coding Summary tab in columns B-D

2. List each study's author(s), year, and title on the CEC 2014 Quality Indicators tab across the top, replacing "Reference 1" "Reference 2" etc.

3. Enter reviewer one (R1) and reviewer two (R2) initials on the CEC 2014 Quality Indicators tab, replacing the red R1 and R2 cells, and score all articles

True or false, when making changes to the order of included articles’ authors, year, and title, it is okay to move them around using cut and paste.

1. 1. True, cut and paste will not affect any formulas in Excel

   2. False, copy and paste must be used in order to not affect formulas in Excel

1/1

On the CEC 2014 Quality Indicators tab where you code articles, some quality indicators are shaded green, which represents:

1. The quality indicator is only applicable to group comparison designs: Green = Group

2. The quality indicator is only applicable to single-case research designs: Sapphire = Single-case

1/1

On the CEC 2014 Quality Indicators tab where you code articles, some quality indicators are shaded sapphire blue, which represents:

1. The quality indicator is only applicable to group comparison designs: Green = Group

2. The quality indicator is only applicable to single-case research designs: Sapphire = Single-case

1/1

When coding a single-case research design study, the quality indicators shaded green should be coded:

1. 0

2. 1

3. N/A (with a slash) because the quality indicator is not applicable to single-case research designs

4. NA (with no slash) because the quality indicator is not applicable to single-case research designs

1/1

When coding a group comparison design study, the quality indicators shaded sapphire blue should be coded:

1. 0

2. 1

3. N/A (with a slash) because the quality indicator is not applicable to group comparison designs

4. NA (with no slash) because the quality indicator is not applicable to group comparison designs

1/1

When entering participant N on the QI Coding Summary tab:

1. Type in the number of participants/cases in the study relevant to the review

2. Type in the number of participants/cases in the study who demonstrated positive outcomes

3. Type in the number of participants/cases in the study who demonstrated either positive or neutral outcomes but do not count those with negative outcomes

4. Type in the number of participants/cases in the study regardless of relevance to the review

1/1

When entering effects on the QI Coding Summary tab:

1. Type in the number of participants who demonstrated positive outcomes in the "Positive" column, the number of participants who demonstrated neutral or mixed outcomes in the "Neutral/Mixed" column, and the number of participants who demonstrated negative outcomes in the "Negative" column

2. Type in either a 0 or 1 in each of the Positive, Neutral/Mixed, and Negative columns to indicate if the study met

(1) or did not meet (0) criterion to be considered as having positive, neutral/mixed, or negative effects, respectively

1/1

True or false, when entering experiment type on the QI Coding Summary tab, type in either a 0 or 1 in the Randomized Group Study, Non-Randomized Group Study, and Single-Case Design columns to indicate if the study design was (1) or was not (0) each design type.

1. True

2. False

1/1

True or false, when QI coding is complete, the Excel formulas indicating if the practice is evidence based or potentially evidence based will always be accurate.

1. True, the Excel formulas use logic checks to calculate the evidence-based status with mathematical accuracy and infallibility

2. False, the Excel formulas are susceptible to data entry errors and negated by any study with negative effects or a ratio of positive to neutral/mixed effects less than 3:1 - the review team must check accuracy of data entry and use judgment to make final conclusions

QI Matrix: Check for Understanding Powered by Qualtrics