期刊
IEEE ACCESS
卷 9, 期 -, 页码 109413-109431出版社
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/ACCESS.2021.3102645
关键词
Scalability; Throughput; Pipelines; Benchmark testing; Sparks; Measurement; Storms; Apache spark; structured streaming; apache flink; apache kafka; kafka streams; distributed computing; stream processing frameworks; scalability; benchmarking; big data
This study evaluates the scalability of stream processing jobs in four popular frameworks and finds that scaling efficiency is influenced by factors such as cluster layout, scaling direction, framework design, and data characteristics. Recommendations are provided on how to scale clusters effectively.
More and more use cases require fast, accurate, and reliable processing of large volumes of data. To do this, a distributed stream processing framework is needed which can distribute the load over several machines. In this work, we study and benchmark the scalability of stream processing jobs in four popular frameworks: Flink, Kafka Streams, Spark Streaming, and Structured Streaming. Besides that, we determine the factors that influence the performance and efficiency of scaling processing jobs with distinct characteristics. We evaluate horizontal, as well as vertical scalability. Our results show how the scaling efficiency is impacted by many factors including the initial cluster layout and direction of scaling, the pipeline design, the framework design, resource allocation, and data characteristics. Finally, we give some recommendations on how practitioners should undertake to scale their clusters.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据