4.7 Article

Memory Scaling of Cloud-Based Big Data Systems: A Hybrid Approach

Journal

IEEE TRANSACTIONS ON BIG DATA
Volume 8, Issue 5, Pages 1259-1272

Publisher

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TBDATA.2020.3035522

Keywords

Cloud computing; Memory management; Checkpointing; Big Data applications; Runtime; Resource management; Memory management; resource allocation; cloud computing

Ask authors/readers for more resources

This article investigates the selection of memory scaling mechanisms for deploying applications on big data systems in public clouds. By analyzing the trade-off between performance and cost of physical memory and virtual memory, and proposing a just-in-time migration method, this study demonstrates that these techniques can improve performance and reduce costs.
When deploying applications with dynamic and intensive memory footprint to big data systems on public clouds, one important yet challenging question to answer is how to select a specific instance type whose memory capacity is large enough to prevent out-of-memory errors while the cost is minimized without violating performance requirements. The state-of-the-practice solution is trial and error, causing both performance overhead and additional monetary cost. This article investigates two memory scaling mechanisms in public clouds: physical memory (good performance and high cost) and virtual memory (degraded performance and no additional cost). In order to analyze the trade-off between performance and cost of the two scaling options, a performance-cost model is developed that is driven by a lightweight analytic prediction approach through a compact representation of the memory footprint. In addition, for those scenarios when the footprint is unavailable, a meta-model-based prediction method is proposed using just-in-time migration mechanisms. The proposed techniques have been extensively evaluated with various benchmarks and real-world applications on Amazon Web Services: the performance-cost model is highly accurate and the proposed just-in-time migration approach reduces the monetary cost by up to 66 percent.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.7
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

No Data Available
No Data Available