How Much Method-in-Use Matters? A Case Study of Agile and Waterfall Software Projects and their Design Routine Variation

Development methods are rarely followed to the letter, and, consequently, their effects are often in doubt. At the same time, information systems scholars know little about the extent to which a given method truly influences software design and its outcomes. In this paper, we approach this gap by adopting a routine lens and using a novel methodological approach. Theoretically, we treat methods as (organizational) ostensive routine specifications and deploy routine construct as a feasible unit of analysis to analyze the effects of a method on actual, performed design routines. We formulated a research framework that identifies method, situation fitness, agency, and random noise as main sources of software design routine variation. Empirically, we applied the framework to examine the extent to which waterfall and agile methods induce variation in software design routines. We trace-enacted design activities in three software projects in a large IT organization that followed an object-oriented waterfall method and three software projects that followed an agile method and then analyzed these traces using a mixed-methods approach involving gene sequencing methods, Markov models, and qualitative content analysis. Our analysis shows that, in both cases, method-induced variation using agile and waterfall methods accounts for about 40% of all activities, while the remaining 60% can be explained by a designer's personal habits, the project's fitness conditions, and environmental noise. Generally, the effect of method on software design activities is smaller than assumed and the impact of designer and project conditions on software processes and outcomes should thus not be understated.