Dealing With Data Challenges When Delivering Data-Intensive Software Solutions

The predicted increase in demand for data-intensive solution development is driving the need for software, data, and domain experts to effectively collaborate in multi-disciplinary data-intensive software teams (MDSTs). We conducted a socio-technical grounded theory study through interviews with 24 practitioners in MDSTs to better understand the challenges these teams face when delivering data-intensive software solutions. The interviews provided perspectives across different types of roles including domain, data and software experts, and covered different organisational levels from team members, team managers to executive leaders. We found that the key concern for these teams is dealing with data-related challenges. In this article, we present a theory of dealing with data challenges that explains the challenges faced by MDSTs including gaining access to data, aligning data, understanding data, and resolving data quality issues; the context in and condition under which these challenges occur, the causes that lead to the challenges, and the related consequences such as having to conduct remediation activities, inability to achieve expected outcomes and lack of trust in the delivered solutions. We also identified contingencies or strategies applied to address the challenges including high-level strategic approaches such as implementing data governance, implementing new tools and techniques such as data quality visualisation and monitoring tools, as well as building stronger teams by focusing on people dynamics, communication skill development and cross-skilling. Our findings have direct implications for practitioners and researchers to better understand the landscape of data challenges and how to deal with them.

Automated summary

This article presents a socio-technical grounded theory study conducted through interviews with 24 practitioners in multi-disciplinary data-intensive software teams (MDSTs). The study aims to understand the challenges faced by MDSTs when delivering data-intensive software solutions. The findings highlight the key concern of data-related challenges and provide a theory that explains the challenges, the context in which they occur, the causes, and the consequences. The study also identifies strategies and contingencies applied to address these challenges. The findings have implications for practitioners and researchers in understanding and dealing with data challenges.