Journal
SIGMOD '21: PROCEEDINGS OF THE 2021 INTERNATIONAL CONFERENCE ON MANAGEMENT OF DATA
Volume -, Issue -, Pages 1637-1650Publisher
ASSOC COMPUTING MACHINERY
DOI: 10.1145/3448016.3457320
Keywords
-
Categories
Funding
- H2020 project OpertusMundi [870228]
- ICAI AI for Fintech Lab project
- SURF Cooperative
Ask authors/readers for more resources
Stream processing is crucial for modern businesses, but ensuring fast and consistent recovery from failures remains a challenge. Clonos is a fault tolerance approach that enforces causally consistent recovery by tracking nondeterminism within the system, achieving instant, local operator recovery with negligible overhead.
Stream processing lies in the backbone of modern businesses, being employed for mission critical applications such as real-time fraud detection, car-trip fare calculations, traffic management, and stock trading. Large-scale applications are executed by scale-out stream processing systems on thousands of long-lived operators, which are subject to failures. Recovering from failures fast and consistently are both top priorities, yet they are only partly satisfied by existing fault tolerance methods due to the strong assumptions these make. In particular, prior solutions fail to address consistency in the presence of nondeterminism, such as calls to external services, asynchronous timers and processing-time windows. This paper describes Clonos, a fault tolerance approach that achieves fast, local operator recovery with exactly-once guarantees and high availability by instantly switching to passive standby operators. Clonos enforces causally consistent recovery, including output deduplication, by tracking nondeterminism within the system through causal logging. To implement Clonos we re-engineered many of the internal subsystems of a state of the art stream processor. We evaluate Clonos' overhead and recovery on the Nexmark benchmark against Apache Flink. Clonos achieves instant recovery with negligible overhead and, unlike previous work, does not make assumptions on the deterministic nature of operators.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available