Fast and Accurate Cardinality Estimation by Self-Morphing Bitmaps

Estimating the cardinality of a data stream is a fundamental problem underlying numerous applications such as traffic monitoring in a network or a datacenter and query optimization of Internet-scale P2P data networks. Existing solutions suffer from high processing/query overhead or memory in-efficiency, which prevents them from operating online for data streams with very high arrival rates. This paper takes a new solution path different from the prior art and proposes a self-morphing bitmap, which combines operational simplicity with structural dynamics, allowing the bitmap to be morphed in a series of steps with an evolving sampling probability that automatically adapts to different stream sizes. We further generalize the design of self-morphing bitmap. We evaluate the self-morphing bitmap theoretically and experimentally. The results demonstrate that it significantly outperforms the prior art.

Automated summary

This paper proposes a self-morphing bitmap as a new solution for estimating the cardinality of a data stream. It combines operational simplicity with structural dynamics, automatically adapting to different stream sizes. The theoretical and experimental evaluation demonstrates its significant improvement over previous techniques.