Evaluating Skills

1. Assessment Techniques

1.1. Direct Testing

1.1.1. Certain skills can be directly tested. E.g.: Drawing blood, sorting letters, running, using a power tool (physical)

1.1.2. Primary evaluation is on final outcome as a result of performing the skill (product). How the learner performed can also be evaluated (process)

1.1.3. Guidelines

1.1.3.1. 1. Review task analysis results to determine the steps needed to perform = individual criteria

1.1.3.2. 2. Determine level of proficiency required

1.1.3.3. 3. Determine where, and what testing will require

1.1.3.4. 4. Write test instructions to inform how test will be conducted

1.1.3.5. 5. Establish how results will be recorded

1.1.3.6. 6. Conduct test, judge proficiency, provide feedback

1.2. Performance Ratings

1.2.1. Similar to direct testing, but focus is different.

1.2.2. Focus on process, not product (E.g. Cooking techniques to make souffle, develop plans to print 3D model, debating skills)

1.2.3. Checklists to determine sequence, Rating scales to determine quality (Likert-scale)

1.2.4. Rating scales must be interrater reliable (consistent results): Pilot-test Agree on scale criteria, how it works

1.2.5. Guidelines

1.2.5.1. 1. Five points Likert-scale (or odd numbers)

1.2.5.2. 2. Verbal descriptions for each rating point must be included, not overlapping

1.2.5.3. 3. Clear and distinct wording to express items

1.2.5.4. 4. One idea for each item

1.2.5.5. 5. Pilot test from target group or closely representative group

1.3. Observation and Anecdotal Records

1.3.1. A record resulting from observing the perfomance of a skill (e.g. teacher observation, instructor conducting training, riding along on shift with police)

1.3.2. Steps

1.3.2.1. Carried out in exact or representative setting in which skill will be used

1.3.2.2. Notes taken to reflect how well skill was performed

1.3.2.3. Notes are expanded into narrative describing the performance and areas of improvement

1.3.3. Guidelines

1.3.3.1. 1. Determine what skills to focus on

1.3.3.2. 2. Determine whether recording devices will be used - gain permission

1.3.3.3. 3. Decide if observation will be announced (or not)

1.3.3.4. 4. Notes taken should be brief and direct, no interpretation. Expand on notes ASAP after the observation

1.3.3.5. 5. Determine whether follow-up observation is required

1.3.3.6. 6. If there are multiple observers, ensure interrater reliability

1.4. Portfolios

1.4.1. Collection of artifacts depicting ability E.g: Artist paintings, songs, short stories, creative works, learning products, ISD model

1.4.1.1. Represent important contextualized learning that require complex thinking and expressive skills

1.4.2. Focus: The product learner creates resulting from the skills acquired over a given period of time - show progression of learning

1.4.2.1. Indicate whether skills learnt have been applied

1.4.2.2. Indicate strength and weaknesses

1.4.2.3. Richer and more valid assessment of students' competencies compared to traditional testing

1.4.3. Guidelines

1.4.3.1. 1. Determine skills to cover, period of time, relation to instructional goals and objective

1.4.3.2. 2. Identify how artifacts will be evaluated; develop rubrics

1.4.3.3. 3. Identify how the entire portfolio will be evaluated, how change in skill determined

1.4.3.4. 4. Determine what artifact samples to be included - discuss and meet to give feedback

1.4.3.5. 5. Judge the completed portfolio

1.5. Rubrics

1.5.1. Criterion-referenced assessment

1.5.1.1. Can be used in conjuction with various other techniques

1.5.1.2. Can be used with knowledge, skill, and attitude assessment

1.5.1.3. Guide for learner AND instructor

1.5.2. Benefits

1.5.2.1. Can be matched with standards and objectives

1.5.2.2. Provide much greater level of detail than letter grades or numerical scores

1.5.2.3. Save evaluation time

1.5.2.4. Help make consistent and objective evaluation

1.5.2.5. Guide learners on what to focus on

1.5.3. Rubric generator

1.5.3.1. RubiStar Home

1.5.3.2. Rubric Maker - Create custom assessments

1.5.3.3. Quick Rubric :)

2. How?: Examine actions or behaviours that can be directly observed.

2.1. Performance assessment

2.2. A learner's ability to perform a particular task OR series of related tasks that make up a particular skill

2.3. E.g.: Speaking in an interview

2.3.1. Coherence

2.3.2. Body Language

2.3.3. Fluency

2.3.4. Grammar

2.3.5. Cohesion

2.3.6. Tone

2.3.7. Lexical resources

2.4. General skill = instructional goal Precise skill = instructional objectives

3. Guidelines

3.1. 1. Determine what will be evaluated. Both process and product can be evaluated.

3.2. 2. Evaluating process involves: Following a proper series of steps, using tools or instruments properly, completing the skill in a certain timeframe.

3.3. 3. Evaluating product: Quality and quantity

3.4. 4. Evaluate under the most realistic condition possible