On the reliability of identifying design moves in protocol analysis

This paper discusses issues and ways of measuring the reliability of segmenting verbal protocols of design activity, a central focus of design research. Reliability is an important issue in distinguishing 'design moves'. In the present study, seven students working for a master in design degree, one graduated designer and two professors segment a 30 min protocol of a product design process into design moves. The intra and inter reliability was calculated for these observers using alpha coefficients. Neither the students', designer's nor professors' segmentation reached the desired cut-off value of 0.8. This negative finding questions the clarity of existing conceptions and urges more concise definitions, better training of analysts, and formulating more decisive instructions. (c) 2013 Elsevier Ltd. All rights reserved.