A MapReduce-supported network structure for data centers

Several novel data center network structures have been proposed to improve the topological properties of data centers. A common characteristic of these structures is that they are designed for supporting general applications and services. Consequently, these structures do not match well with the specific requirements of some dedicated applications. In this paper, we propose a hyper-fat-tree network (HFN): a novel data center structure for MapReduce, a well-known distributed data processing application. HFN possesses the advanced characteristics of BCube as well as fat-tree structures and naturally supports MapReduce. We then address several challenging issues that face HFN in supporting MapReduce. Mathematical analysis and comprehensive evaluation show that HFN possesses excellent properties and is indeed a viable structure for MapReduce in practice. Copyright (c) 2011 John Wiley & Sons, Ltd.