Clustering-based acceleration for virtual machine image deduplication in the cloud environment

More and more virtual machine (VM) images are continuously created in datacenters. Duplicated data segments may exist in such VM images, and it leads to a waste of storage resource. As a result, VM image deduplication is a common daily activity in datacenters. Our previous work Crab is such a product and it is on duty regularly in our datacenter. The size of VM images is large and the amount of VM images is huge, and it is inefficient and impractical to load massive VM image fingerprints into memory for a fast comparison to recognize duplicated segments. To address this issue, we in this paper propose a clustering-based acceleration method. It uses an improved k-means clustering to find images having high chances to contain duplicated segments. With such a candidate selection phase, only limited VM image candidate fingerprints are loaded into memory. We empirically evaluate the effectiveness, robustness, and complexity of the proposed system. Experimental results show that it significantly reduces the performance interference to hosting virtual machine with an acceptable increase in disk space usage, compared with existing deduplication methods. (C) 2016 Elsevier Inc. All rights reserved.