1. Ways to Increase Reliability
1.1. increase the number of evaluative items
1.1.1. with "a larger sample of the trait being evaluated, chance errors will tend to cancel each other out" (Moore, 2015, p. 266)
1.2. establish optimum item difficulty
1.2.1. "moderate difficulty gives a moderate spread of scores, which allows you to better judge each student's performance in relation to other students" (Moore, 2015, p. 266)
1.3. write clear items and directions
1.3.1. "ambiguities and misunderstood directions lead to irrelevant errors" (Moore, 2015, p. 266)
1.4. administer the evaluative instrument carefully
1.4.1. no distractions by noises and no rush on students (Moore, 2015)
1.5. score objectively
1.5.1. "with subjective data, internal differences within the scorer can result in identical responses or behaviors being scored differently on different occasions" (Moore, 2015, p. 266)
1.6. Teachers can use these steps when they create assessments to help ensure they are reliable. If they are using a pre-made assessment, they should still look through the list and make sure the assessment they are giving follows the guidelines of reliability.
2. Classroom Reliability
2.1. create clear instructions for every assignment (The Graide, 2018)
2.2. write questions that capture the material taught (The Graide, 2018)
2.3. seek feedback "regarding the clarity and thoroughness of the assessment from students and colleagues" (The Graide, 2018, p. 12)
2.4. All forms of assessment in the classroom should be reliable. Students should never purposely be tricked or given assessments that have nothing to do with the content being taught. Practice activities should also be reliable.
3. Reliability of Qualitative Assessments
3.1. interrater consistency
3.1.1. "the calculation of the percent agreement between two or more raters of student performance or products that are more appropriately scored on qualitative criteria" (Mertler, 2017, p. 63)
3.1.2. Examples where it might be used as noted by Mertler (2017)
3.1.2.1. performance assessments
3.1.2.2. portfolios
3.1.2.3. physical performance
3.1.2.4. musical performance
3.1.2.5. essays
3.2. Qualitative assessments are not measurable by statistics as quantitative assessments are. However, it is still important to ensure they are reliable.
4. Reliability & Vailidity
4.1. "Valid test results are also reliable, but reliable test results are not necessarily valid" (Mertler, 2017, p. 64).
4.2. "Validity and reliability (along with fairness) are considered two of the principles of high quality assessments" (The Center, 2018).
4.3. Teachers have a responsibility to ensure the assessments they choose to create, or use in their class, are both valid and reliable. Students deserve to take assessments that align with the learning standards and cover content they have been taught. Parents, school leaders, teachers, and students themselves will have a better picture of where students are at if they are taking reliable and valid assessments.
5. References
5.1. Forrester, E. (2020, August 7). What is Assessment Reliability & Validity? Illuminate Education. What is Assessment Reliability & Validity? - Illuminate Education
5.2. Mertler, C. (2017). Classroom Assessment: A Practical Guide for Educators. Routledge.
5.3. Moore, K. D. (2015). Effective Instructional Strategies: From Theory to Practice (4th ed.). SAGE.
5.4. The Center on Standards and Assessment Implementation. (2018, March). Valid and Reliable Assessments. https://files.eric.ed.gov/fulltext/ED588476.pdf.
5.5. The Graide Network. (2018). Importance of Validity and Reliability in Classroom Assessments. Retrieved from https://www.thegraidenetwork.com/blog-all/2018/8/1/the-two-keys-to-quality-testing-reliability-and-validity
6. Consistency
6.1. "the consistency of measurements when the testing procedure is repeated on a population of individuals or groups" (Mertler, 2017, p. 60)
6.2. "If you test a student today and test a student tomorrow, are the scores going to be similar? If you test one population of students and a different population of students, are you going to see similar scores on the same assessment?" (Forrester, 2020, para. 1)
6.3. Assessment reliability relates to the consistency of scores and interpretation of those scores.
6.4. Checking for the reliability of assessments means that there should be consistency in the scores whether it is a different group of students taking the test or the same students at different times.
7. Test-retest Reliability
7.1. "if a group of students takes a test twice, both the results for individual students, as well as the relationship among students’ results, should be similar across tests" (The Graide, 2018, p. 10)
7.1.1. Steps as noted by Mertler (2017)
7.1.1.1. 1. Give test to a group of students (p. 61)
7.1.1.2. 2. Administer the same test to the same students at another time (p. 61)
7.1.1.3. 3. Find correlations between the scores of the two tests (p. 61)
7.2. coefficient of stability
7.2.1. "if the two sets of scores are very similar (i.e., high-high, low-low) the results can be considered stable or reliable" (Mertler, 2017, p. 61)
7.3. Re-testing students after a short period of time could help teachers see if students are really remembering the material that was covered as well as see who is consistently struggling or exceeding.
8. Alternate-Forms Method
8.1. "consistency of both individual scores and positional relationships" (The Graide, 2018, p. 10)
8.1.1. Steps as noted by Mertler (2017)
8.1.1.1. 1. Administer Form A of the test to a group of students (p. 62)
8.1.1.2. 2. Shortly afterward, administer Form B of the test to the same students (p. 62)
8.1.1.3. 3. Correlate the scores from the two test forms (p. 62)
8.2. Teachers may not choose this method of checking reliability since it requires multiple tests to be given on the same content. However, one area giving multiple tests on the same content would come in handy is remediation of standards. After the content is taught and students practice, they take a formative assessment. Those who did not master the standard would now receive remediation and practice more. They would then take the same or a similar assessment on the same content, to see if they have mastered the standard after receiving additional help.
9. Internal Consistency Method
9.1. "a measure of how the actual content of an assessment works together to evaluate understanding of a concept" (The Graide, 2018, p. 10)
9.2. split-half method
9.2.1. "dividing one test into two comparable halves" (Mertler, 2017, p. 62)
9.3. Kuder-Richardson methods
9.3.1. "an estimate of the reliability equal to the average of all possible split-half combinations" (Mertler, 2017, p. 63)
9.3.2. most widely used
9.3.3. most accurate
9.4. Cronbach's alpha method
9.4.1. "a special case of the KR-21 formula where individual items or tasks may be scored with different point values" (Mertler, 2017, p. 63)