Read the article and answer the questions

Program Evaluation Report Critique for Breakout Group Assignment Session 13

PURPOSE: The purpose of this assignment is to provide you with the opportunity to put your skills to the test by critiquing a program evaluation report. You should have already read the article and answered questions about the background and sampling strategy.

DIRECTIONS: Now, we will divide into breakout groups. As a group answer the questions in the outline below fully and completely. DO NOT DIVIDE THE QUESTIONS UP AMONG YOU. EACH PERSON SHOULD BE PART OF THE PROCESS OF RESPONDING TO EACH QUESTION. Explain the rationale for each of your answers. Create a narrative that follows the outline below. Please use headings and subheadings in your paper. (Doing so helps you stay organized, and it helps your instructor find the information he/she is searching for quickly.) If the information to answer the question is not available in the article, please state this and comment on the author’s decision not to include (Does it make sense to leave out? Would including this have helped the reader in understanding the evaluation?)

1. Describe the data collection method. This could include interviews/surveys/questionnaires, focus groups, observations, record reviews, etc. Consider the following questions:

1. 1. Where, when, by whom, and under what conditions were data collected? How might data be biased in any way because of these factors?
  2. What is one strength and one limitation of the data collection strategy?

1. What are the strengths and limitations of the measurement strategy? Consider the following questions:

1. 1. Are operationalizations presented for each process goal/objective and each outcome goal/objective for the program? How clearly are they described? Select one variable and provide its operational definition.
  2. Is measurement purely quantitative, purely qualitative or mixed methods? Explain.
  3. For quantitative measurements, are standardized instruments with demonstrated reliability and validity used? Are any self-constructed measures used? In either case, what have researchers included to describe and/or improve reliability and validity of the measures? For qualitative measurements, are measurements flexible, but still require some specific data elements; that is, systematic, verifiable, consistently measured? What is said about the qualitative measurement reliability and validity?
  4. Do the data come from multiple perspectives (e.g., client, family members, service providers, staff, administration, etc.) or rely on one group of key informants? Explain.

1. Describe the data analysis strategy . Consider the following questions:

1. 1. For quantitative data analysis, what types of statistics were run? Univariates (e.g., frequencies, means, medians, and modes), bi-variates (e.g., chi-squares, t-tests, ANOVAs, correlations), multivariate models (e.g., multiple linear regression or multiple logistical regression)
  2. For qualitative data, were themes developed by two or more coders? Was inter-rate reliability calculated? What steps are discussed to enhance qualitative rigor.

1. If this was a summative evaluation, what are the key threats to internal validity (i.e., confidence that the intervention, as opposed to other factors, is responsible for any changes). Think about history, maturation, testing, instrumentation, statistical regression, experimental mortality, etc. If threats to internal validity are not relevant explain why.
2. If this is a formative evaluation, name one research limitations.
3. For both formative and summative how well does the sample represent the population? Think about sample size, selection biases, etc. What are other limitations of the evaluation?
Findings/Results
1. Consider the following questions:

Provide a brief summary of what was found about one goal/objective of the study..
How are tables, charts, figures, and graphs, used to help the reader understand the overall findings? Be specific.
If there are qualitative findings, how are they presented? Provide an example of how this is done for one qualitative finding.

Conclusions/Implications
1. What conclusions can you draw from this study?
2. Identify one way for each group member to incorporate the findings, inferences, or conclusions from this evaluation into your current or future clinical social work practice.