Multi-Framework Reliability Approach

Despite advances in making datacenters dependable, failures still happen. This is particularly onerous for long-running big data applications, where partial failures can lead to significant losses and lengthy recomputations. Big data processing frameworks like Hadoop MapReduce include fault tolerance (FT) mechanisms, but these are commonly targeted at specific system/failure models, and are often redundant between frameworks. This article proposes the paradigm of dependable resources: big data processing frameworks are typically built on top of resource management systems (RMSs), and proposing FT support at the level of such an RMS yields generic FT mechanisms, which can be provided with low overhead by leveraging constraints on resources. We demonstrate our concepts through Guardian, a robust RMS based on Mesos and YARN. Guardian allows frameworks to run their applications with individually configurable FT granularity and degree, with only minor changes to their implementation. We demonstrate the benefits of our approach by evaluating Hadoop, Tez, Spark and Pig on a prototype of Guardian running on Amazon-EC2, improving completion time by around 68 percent in the presence of failures, while maintaining around 6 percent overhead.

Automated summary

This article proposes the paradigm of dependable resources, which provides generic fault tolerance mechanisms by offering fault tolerance support at the level of resource management systems. Through the demonstration of Guardian, the benefits of this concept are shown, improving completion time for big data processing frameworks in the presence of failures while maintaining low overhead.