MEFISTO WP4-1

Wright, P. C.
Concepts for usability evaluation

The aim of this document is to explore a number of issues and concepts related to the Mefisto evaluation activity We take "evaluation" to mean a process of making judgements about a newly designed artefact in terms of the impact on the work system in which it will eventually be deployed. Depending on the circumstances, "judgement" and "impact" can be interpreted in a number of different ways. For instance, we may be in a situation of wanting to select among a number of competing design alternatives, in which case, the impact may be assessed by making some form of comparison, of a qualitative or quantitative nature between the options. In a different situation, however, instead of making comparisons, it may be more important to evaluate whether a single design is acceptable, or to identify ways in which it may be improved. There are many aspects of a designed artefact that can be evaluated. Our primary concern in the Mefisto project is with usability evaluation and in particular the identification of usability issues which have an impact on usability and safety. Many techniques, methods, and tools already exist to support a wide range of empirical and formal evaluation activities. The aim of this work package is not to replicate any of the large body of work on usability evaluation that currently exists. Nor is it the aim of this document to present a comprehensive review. For an extensive survey of evaluation techniques and their applicability to the ATC environment, see Foot et al., (1996) and WP 4.2. In the Mefisto proposal our aim is to develop evaluation techniques appropriate for the ATC prototypes that are the product of the Mefisto design approach. In this document we take on a broader and more significant challenge by exploring not only concepts for the evaluation of prototypes but also concepts for the evaluation of design notations. It is our belief that evaluating Mefisto notations and attempting to link properties of notations to features of a prototype would yield a more significant contribution to the overall aims of the Mefisto project. Broadly our aim is to develop methods for the early formative evaluation of prototypes. We take cognisance of the fact that in safety critical settings such evaluation much take account of the envisioned context of use. To achieve this, are evaluations are based around scenarios of use gleaned from our requirements gathering activities and in the work of Siena on war stories. We also explore the link between this form of evaluation and the formal, mode-based evaluation of task and system specifications. In exploring concepts for the evaluation of notations, we examine the possibility of identifying relatively context-independent properties of notations and combine these with feedback from training sessions to identify important usability characteristics. We also explore how these may be linked to prototype evaluations through an approach similar to quality function deployment. Section 2 identifies a number of dimensions that define a space of possible evaluation activities. Section 3 identifies some of the concepts that the evaluation of ATC prototypes can seek to explore, and that are of particular relevance for the Mefisto project. Section 4 identifies concepts relevant to the evaluation of design notations and techniques. Section 5 concludes some specific proposals for how to progress the evaluation activities in activity 4.2 with the development of evaluation techniques that embody some of the concepts identified here.


Home    Overview    Contact list    Planning of Work    Documents    Workspace    Related Links    Statistics