MEFISTO WP4-2

Fields, B., Goillau, P., Fairburn, C., Wright, P.
Techniques for usability evaluation

There are many possible techniques for usability evaluation (Preece et al, 1994), and those techniques especially relevant to the evaluation of Air Traffic Management (ATM) systems have been extensively reviewed elsewhere, for example in the RHEA and HIFA projects (Foot et al, 1996; Enterkin et al, 1999).). This exercise will not be repeated here. It has also been noted elsewhere (MEFISTO WP4.1 working paper: Concepts for Usability Evaluation) that the term 'Distributed Evaluation' sums up the nature of the evaluation process in MEFISTO: evaluation is distributed in that it takes place in a number of forms and is spread over a number of evaluation events distributed in time and over different locations. Of the available evaluation techniques, the following seem especially relevant to the MEFISTO ATM concepts, the formal methods notations, the process of design, and the final MIDDLES and MIDAS ATM system prototypes:

  1. checklist Walkthrough-style techniques applicable to ATM concept descriptions where a product is still being developed
  2. evaluation techniques for the usability of notations themselves
  3. action research techniques for the design process
  4. classic human factors prototype product evaluation techniques

This note covers the first group of techniques. It is an attempt at applying checklist Walkthrough-style evaluation techniques (classic Cognitive Walkthrough, Information Resources Analysis and the pragmatic MACAW technique) to one of the scenarios (runway incursion) of the MEFISTO Aerodrome case study based on Fiumicino airport, Rome, Italy. The aim is to investigate whether such Walkthrough approaches based on a checklist of questions might be useful tools for evaluating the MIDAS prototype for Fiumicino airport in the context of scenarios such as runway incursion. The description covers both the current system and the envisioned world with an Advanced-Surface Movement Guidance and Control System (A-SMGCS) technology in place as part of MIDAS. In this note we will first describe the Fiumicino aerodrome Case Study. The document then goes on to outline a runway incursion scenario (rather than a single task) as it might happen in the current Fiumicino tower environment. The scenario is then re-worked for a Midas-like technological context. This runway incursion scenario forms the basis for three analytic exercises that will serve to illustrate the application of the Cognitive Walkthrough, Information Resources Analysis, and MACAW walkthrough style techniques. The techniques are contrasted and compared.


Home    Overview    Contact list    Planning of Work    Documents    Workspace    Related Links    Statistics