Data Imputation for Multivariate Time Series Sensor Data With Large Gaps of Missing Data

Imputation of missing sensor-collected data is often an important step prior to machine learning and statistical data analysis. One particular data imputation challenge is filling large data gaps when the only related data comes from the same sensor station. In this paper, we propose a framework to improve the popular multivariate imputation by chained equations (MICE) method for dealing with missing data. One key strategy we use to improve model accuracy is to reshape the original sensor data to leverage the correlation between the missing data and the observed data. We demonstrate our framework using data from continuous water quality monitoring stations in Vermont. Because of possible irregularly spaced peaks throughout the time series, the reshaped data is split into extreme and normal values and two MICE models are built. We also recommend that sensor-collected data should be transformed to meet the machine learning model assumptions. According to our experimental results, these strategies can improve MICE data imputation model accuracy at least 23% for large data gaps based on R-2 values and are promising to be applied for other data imputation algorithms.

Automated summary

This paper proposes a framework to improve the accuracy of the popular multivariate imputation by chained equations (MICE) method for dealing with missing data. The framework involves reshaping the original sensor data and leveraging the correlation between missing and observed data. Experimental results using water quality monitoring data demonstrate a significant improvement in MICE model accuracy with these strategies.