Introducing Interactive Multimedia

Evaluating interactive multimedia

Hedberg (1998) suggested a five part rationale for evaluating interactive learning materials:

1. Previous technological innovations in education (radio, film, television) have not been especially successful in their implementation.
2. The applications of materials in new, more complex, approaches to teaching and learning need to be evaluated.
3. Complex technological systems require materials to be tested in operation with a variety of systems.
4. The materials being developed are increasingly complex and their operation is largely untested.
5. There is a growing emphasis on performance and cost effectiveness.

He suggested that the major questions to be asked in evaluation are:

• What actually occurred?
• How can it be improved?
• Did it accomplish the objectives?
• What impact did it have?

Evaluation should not be a one shot process conducted after the project is completed. Rather, there needs to be formative evaluation throughout a project, commencing with the original concept and informing each phase of design and development. During the development stage much of the evaluation will relate to testing and debugging of the software and interface. Issues which emerge during this testing need to be addressed carefully to ensure that the product delivered is as error free and useable as possible.

Summative evaluation should follow completion of the project to determine to what extent the objectives have been met and to inform future development efforts.

A variety of alternative methods including interviews, surveys, focus groups and direct observation may be used to gather evaluative data at various points in the progress of the project. Interpretation of results can also vary according to the assumptions of the evaluation team.

Criteria and metrics for evaluation

Criteria and associated metrics are important for evaluation in a variety of everyday circumstances. In broad terms, criteria identify the characteristics which are considered important for some purpose. Metrics provide a basis for measuring and comparing individual performances or properties relative to the criteria.

For example, in selecting a team for a particular sport several criteria might be considered. Depending upon the sport, they might include accuracy in throwing a ball, speed in running between points, and so on. In order to make judgments relative to the criteria, metrics would be established. Metrics for accuracy in throwing the ball might be based on the frequency with which a target could be hit from a prescribed distance. Metrics for running between points might reflect the times taken to run from one point to the other, turn and return to the original point.

Criteria should be selected according to what are considered to be the most important factors in determining success. For example, ability to run quickly is important in many sports but it might not be a useful criterion for selecting competitors in swimming or archery.

Metrics should be developed to facilitate comparisons which reflect the criteria and are reproducible. They should produce the same results when used by different persons or by the same person on different occasions. If a sports team requires members who can run quickly then an appropriate metric would be the time taken to run some standard distance. Even an experienced coach would probably not wish to select athletes on the basis of personal opinion after simply seeing the athletes.

Criteria and their associated metrics are fundamental to making judgments about the quality of educational multimedia. Additional material about criteria and metrics for evaluation of web sites and multimedia may be found from the sources listed in the reference list below.

References

FET5622 Creating Interactive Multimedia © University of Southern Queensland
Last updated: 08-Jun-2007 by Peter Albion