1. What is Evaluation ???
1.1. evaluation technique is a technique to test the usability and functionality of system that
2. Evaluating Designs
2.1. Cognitive Walkthrough
2.1.1. Proposed by Polson et al
2.1.2. evaluates design on how well it supports user in learning task
2.1.3. usually performed by expert
2.1.4. expert ‘walks though’ design to identify potential problems
2.1.5. used forms
2.2. Heuristic Evaluation
2.2.1. Proposed by Nielsen and Molich
2.2.2. design examined by experts
2.2.3. Example heuristics -system behavior is predictable -system behavior is consistent -feedback is provided
2.3. Review-based evaluation
2.3.1. Results from the literature
2.3.2. Model-based evaluation
2.3.3. Cognitive models used to filter design options e.g. GOMS prediction of user performance
3. Evaluating through user Participation
3.1. Laboratory studies
3.1.1. Advantages: -specialist equipment available -uninterrupted environment Disadvantages: -lack of context -difficult to observe several users cooperating Appropriate: -if system location is dangerous or impractical for constrained single user systems to allow controlled manipulation of use
3.2. Field Studies
3.2.1. Advantages: -natural environment -context retained (though observation may alter it) -longitudinal studies possible Disadvantages: -distractions -noise Appropriate: -where context is crucial for longitudinal studies
4. Experimental evaluation
4.1. controlled evaluation of specific aspects of interactive behaviour
4.2. evaluator chooses hypothesis to be tested
5. Analysis - types of test
5.1. parametric : -assume normal distribution -robust -powerful
5.2. non-parametric : -do not assume normal distribution -less powerful -more reliable
5.3. contingency table : -classify data by discrete attributes -count number of data items in each group
6. Query Techniques
6.1. Interviews
6.1.1. analyst questions user on one-to -one basis usually based on prepared questions
6.2. Questionnaires
7. Goals of Evaluation
7.1. assess extent of system functionality
7.2. assess effect of interface on user
7.3. identify specific problems
8. Experimental factors
8.1. Subjects
8.1.1. who – representative, sufficient sample
8.2. Variables
8.2.1. independent variable (IV) -characteristic changed to produce different conditions -e.g. interface style, number of menu items
8.2.2. dependent variable (DV) -characteristics measured in the experiment -e.g. time taken, number of errors.
8.3. Hypothesis
8.3.1. prediction of outcome : -framed in terms of IV and DV -e.g. “error rate will increase as font size decreases”
8.3.2. null hypothesis: -states no difference between conditions -aim is to disprove this -e.g. null hyp. = “no change with font size”
8.4. Experimental design
8.4.1. within groups design: -each subject performs experiment under each condition. -transfer of learning possible
8.4.2. between groups design: -each subject performs under only one condition -no transfer of learning -more users required
9. Observational Methods
9.1. Think Aloud
9.1.1. user observed performing task
9.1.2. Advantages : -simplicity - requires little expertise -can provide useful insight
9.1.3. Disadvantages : -subjective -selective
9.2. Cooperative evaluation
9.2.1. -variation on think aloud -user collaborates in evaluation
9.2.2. Additional advantages ; -less constrained and easier to use -user is encouraged to criticize system -clarification possible
9.3. Protocol analysis
9.3.1. -paper and pencil -audio -video -computer logging -user notebooks -Mixed use in practice -audio/video transcription difficult and -requires skill -Some automatic support tools available
9.4. automated analysis – EVA
9.4.1. -Workplace project -Post task walkthrough
9.4.2. Advantages : -analyst has time to focus on relevant incidents -avoid excessive interruption of task
9.4.3. Disadvantages : -lack of freshness -may be post-hoc interpretation of events
9.5. post-task walkthroughs
9.5.1. -transcript played back to participant for comment -immediately > fresh in mind -delayed > evaluator has time to identify questions
10. Physiological methods
10.1. Eye tracking
10.1.1. head or desk mounted equipment tracks the position of the eye
10.1.2. eye movement reflects the amount of cognitive processing a display requires
10.2. Physiological measurement
10.2.1. emotional response linked to physical changes
10.2.2. these may help determine a user’s reaction to an interface