Evaluation Techniques

Evaluation Technique (Group Izzul

Get Started. It's Free
or sign up with your email address
Rocket clouds
Evaluation Techniques by Mind Map: Evaluation Techniques

1. Definition

1.1. - Tests usability and functionality of system - Occurs in laboratory, field and/or in collaboration with users

2. Goals of Evaluation

2.1. assess extent of system functionality

2.2. assess effect of interface on user

2.3. identify specific problems

3. Experimental evaluation

3.1. Factors

3.1.1. Subjects

3.1.1.1. who – representative, sufficient sample

3.1.2. Variables

3.1.2.1. independent variable (IV) characteristic changed to produce different conditions e.g. interface style, number of menu items dependent variable (DV) characteristics measured in the experiment e.g. time taken, number of errors.

3.1.3. Hypothesis

3.1.3.1. prediction of outcome framed in terms of IV and DV e.g. “error rate will increase as font size decreases” null hypothesis: states no difference between conditions aim is to disprove this e.g. null hyp. = “no change with font size”

3.1.4. Experimental Designs

3.1.4.1. Within groups design -each subject performs experiment under each condition. -transfer of learning possible -less costly and less likely to suffer from user variation. Between groups design each subject performs under only one condition no transfer of learning more users required variation can bias results.

4. Analysis of data

4.1. Before you start to do any statistics: look at data save original data Choice of statistical technique depends on type of data information required Type of data discrete - finite number of values continuous - any value

4.2. Type of test - parametric assume normal distribution robust powerful - non-parametric less powerful more reliable - contingency table classify data by discrete attributes count number of data items in each group

5. Observational Methods

5.1. Think Aloud

5.1.1. user asked to describe what he is doing and why, what he thinks is happening etc. Advantages simplicity - requires little expertise can provide useful insight Disadvantages subjective selective

5.2. Cooperative evaluation

5.2.1. both user and evaluator can ask each other questions throughout Advantages -less constrained and easier to use -user is encouraged to criticize system -clarification possible

5.3. Protocol analysis

5.3.1. paper and pencil – cheap, limited to writing speed

5.3.2. audio – good for think aloud, difficult to match with other protocols

5.3.3. video – accurate and realistic, needs special equipment, obtrusive

5.3.4. computer logging – automatic and unobtrusive, large amounts of data difficult to analyze

5.3.5. user notebooks – coarse and subjective, useful insights, good for longitudinal studies

5.4. Automated analysis

5.4.1. Post task walkthrough user reacts on action after the event used to fill in intention

5.5. Post-task walkthroughs

5.5.1. useful to identify reasons for actions and alternatives considered necessary in cases where think aloud is not possible

6. Evaluating Designs

6.1. Cognitive Walkthrough

6.1.1. Expert ‘walks through’ design to identify potential problems using psychological principles

6.1.2. Forms used to guide analysis

6.2. Heuristic Evaluation

6.2.1. Usability criteria (heuristics) are identified

6.2.2. Design examined by experts to see if these are violated

6.3. Review-based evaluation

6.3.1. Results from the literature used to support or refute parts of design.

6.3.2. Care needed to ensure results are transferable to new design

7. Query Techniques

7.1. Interviews

7.1.1. informal, subjective and relatively cheap

7.1.2. Advantages can be varied to suit context issues can be explored more fully can elicit user views and identify unanticipated problems Disadvantages very subjective time consuming

7.2. Questionnaires

7.2.1. Set of fixed questions given to users Advantages quick and reaches large user group can be analyzed more rigorously Disadvantages less flexible less probing

8. Evaluating through user Participation

8.1. Laboratory studies

8.1.1. Advantages: specialist equipment available uninterrupted environment Disadvantages: lack of context difficult to observe several users cooperating

8.2. Field studies

8.2.1. Advantages: natural environment context retained (though observation may alter it) longitudinal studies possible Disadvantages: distractions noise

9. Physiological methods

9.1. Eye tracking

9.1.1. head or desk mounted equipment tracks the position of the eye eye movement reflects the amount of cognitive processing a display requires

9.2. Physiological measurement

9.2.1. emotional response linked to physical changes

9.2.2. these may help determine a user’s reaction to an interface