"FOR NJOSH ONLY"
Name: Date:
UNIT THREE
Validity and Reliability
Introduction
Now that we have begun the research process, it is important for us to understand two important aspects of any research project: validity and reliability. There are literally hundreds of different types of data gathering methods and procedures. As you read through different research projects, you will start to see that authors will use different procedures in different situations. Regardless of the procedure used, reliability and validity are essential to the effectiveness of any research design. In this unit we will begin to define validity and reliability, discuss factors that ensure a valid and reliable research design and data gathering method, identify research designs and instruments that are both reliable and valid.
Objectives
After reading the assigned chapters and participating in discussions related to reliability, validity, research planning and proposals, the student will demonstrate an understanding of quality research design and instrument design by completing assignments provided by the instructor, and conducting research that is both reliable and valid.
Define validity and reliability as they relate to research designs
Define validity and reliability as they relate to research tools
Describe internal and external validity
Describe internal and external reliability
List and describe different factors that indicate a design and instrument are both reliable and valid
Identify a reliable research study and data gathering method and explain what makes both reliable
Identify a valid research study and data gathering method and explain what makes both valid
Design a research study that is both valid and reliable
Assignments
Read pages 3 and 4 attached re Objectives 3.1 through 3.4.
Read pages 21-31, 85-101 in Leedy & Ormond and complete the assignment on page 5 & 6 re Objectives 3.5 and 3.7.
Complete the assignment on page 7 re Objective 3.8.
Acknowledgements
Non-statistical and statistical validity material provided from Dr. Larry Bailey, Southern Illinois University at Carbondale (WED 594 course – Spring 2003)
Leedy, P. D. & Ormrod, J. E. (2005). Practical research: Planning and design (5th ed.). Upper Saddle River, NJ: Pearson Merrill Prentice Hall.
Best, J. W. & Kahn, J. V. (2003). Research in education (9th ed.). Boston: Pearson Education Company.
Wiersma, W. (2000). Research methods in education: An introduction (7th ed.). Needham Heights, MA: Allyn and Bacon.
III–1 Validity and Reliability Defined
While the terms validity and reliability have many different meanings depending on area of study, there is a generally accepted use of the terms in research. As you will read in your textbook, the basic definition of both terms as they relate to research are;
Reliability – is the degree of consistency that the instrument or procedure demonstrates: Whatever it is measuring, it does so consistently. (Concerns the replicability and consistency of the methods, conditions, and results)
Validity – is that quality of a data-gathering instrument or procedure that enables it to measure what it is supposed to measure. (Deals with the accurate interpretability of the results and the generalizability of the results)
So what is the relationship between validity and reliability? As you can see from the above examples, one is not necessarily a condition of the other. Our goal is to have a project that has both high validity and reliability. This would mean that it produces a consistent result in repeated application, and it accurately reflects what we hope to represent.
The problem is usually found in one of the following areas:
Having a measure that has high reliability but low validity - one that is consistent in getting bad information
Low reliability and low validity - inconsistent and not on target
It is not possible to have a measure that has low reliability and high validity – it is impossible to get at what you are interested in if your measure fluctuates wildly
III–2 Internal and External Validity and Reliability
Types of Reliability
Internal Reliability – refers to the extent that data collection, analysis, and interpretations are consistent given the same conditions. For example: If you were to do an observation where five different observers were used to evaluate a particular event, to what extent do the observers agree. If this internal reliability is lacking, the data becomes a function of who collects them rather than what actually happened.
External Reliability – deals with the issue of whether or not independent researchers can replicate studies in the same or similar setting. If they are able to replicate the study, will the results be consistent? To be replicable, a research study must include adequate descriptions of the procedures and conditions of the research.
Types of Validity
Internal Validity – is the extent to which the results of the research study can be interpreted accurately and with confidence. For example: say we want to test a new instructional techniques impact on test scores. The teacher randomly assigns 30 students into two different classes, and implements the technique into one of the classes. At the end of the 16 week period, the teacher administers the same performance test to both classes. This research study has good internal validity because, if the performances on the tests differ, we can say with confidence that the results are because of the different technique. We can say this because the only difference in the two classes is the technique being used. Could we say the same if the study instead used two different schools, different teachers, or different performance tests?
External Validity – is the extent to which research results are generalizable to other populations and/or conclusions. For example: Say we wanted to conduct a study of the current students’ perception of the Masters of Technology (MoT) graduate program here at EIU. We develop a survey with well-constructed and tested items to cover the program characteristics of interest. We randomly select current graduate students in the program and give them the survey. For all students unwilling to complete the survey, we have a procedure for random replacement. There are a total of 55 surveys completed with only 4 replacements. We can say with confidence that the results of the study are generalizable to the population. In this case the population is MoT graduate students at EIU. This study has high or good external validity. Can we generalize these results to all MoT graduate programs in the country? If this were done, we would need to provide an argument for the similarity between the graduate students at EIU and those in other schools. We would also need to address the similarities within the MoT graduate programs.
III–3 Validity and Reliability
Validity and reliability not only relate to our studies design, but also the instruments we use. Read pages 28 through 29 and list below some of the factors that make a measurement instrument valid and reliable.
Measurement Instrument Validity is determined by:
Measurement Instrument Reliability is determined by:
Directions
Read the basic research design scenarios below and determine whether they are internally and externally valid and reliable.
Scenario 1: A study is conducted to determine the perceptions of students about the programs offered at the student recreation center. A survey is constructed by a student worker and sent out to a total of 1500 freshman and sophomore students living on campus. Of the 1500 surveys sent out, only 256 (or 17%) are returned. The study concludes that changes need to be made to the programs offered at the SRC.
Is this study valid? Reliable? Why or why not?
Scenario 2: A study is conducted to determine the overall job satisfaction of student workers at EIU. The researcher decides to use Spector’s (1997) Job Satisfaction Index. The JSS is free to use, is used to measure job satisfaction within public institutions, and has a reliability coefficient of .91. The JSS instrument is administrated to 243 of the 600 student workers (or 41%) at the university during a voluntary exit training session. The study concludes that overall the student workers at EIU are satisfied with their job.
Is this study valid? Reliable? Why or why not?
III–3 Validity and Reliability – cont.
Scenario 3: A study is conducted to evaluate the effectiveness of a particular teaching method on student behavior in a k-8 charter school. The researcher uses observations in order to determine the overall effectiveness. An observation collection instrument is developed and agreed upon by a panel of eight experts. The eight panel members each observe all five k-5 teachers using the teaching method in question at the charter school. The observations show that the experts are in agreement that the method doesn’t have an effect on student behavior in the charter school.
Is this study valid? Reliable? Why or why not?
Scenario 4: A study is conducted to determine the overall effectiveness of a performance improvement tool kit binder developed for a division of the Ford Motor Company. The binder was developed and tested by performance consults and performance technology practitioners for a variety of improvement scenarios. The tool kit binder was then implemented into practice and a survey was developed to analyze customer feedback on the binder. Data were collected during a six month period and concluded that using the tool kit was effective in improving the practitioner’s ability to evaluate and analyze situations.
Is this study valid? Reliable? Why or why not?
III–4 Building Validity and Reliability into a study
Whether we are testing hypotheses, developing a survey, or conducting historical research, the validity and reliability of our research project is extremely important. As you begin to develop your research topic and write the review of literature, critically analyze the research reports you review. Is the author’s research reliable? Is the author’s research valid? What about the collection method?
Your assignment for this unit is to ensure that the research you conduct is both valid and reliable. In order to begin your research on the right track, list below some questions that you need to answer to make sure your research is reliable and valid.
TEC 5143 – Unit 3 Spring 2015
