Real-Time Analytics: Concepts, Architectures, and ML/AI Considerations

With the advancement in intelligent devices, social media, and the Internet of Things, staggering amounts of new data are being generated, and the pace is continuously accelerating. Real-time analytics (RTA) has emerged as a distinct branch of big data analytics focusing on the velocity aspect of big data, in which data is prepared, processed, and analyzed as it arrives, intending to generate insights and create business value in near real-time. The objective of this paper is to provide an overview of key concepts and architectural approaches for designing RTA solutions, including the relevant infrastructure, processing, and analytics platforms, as well as analytics techniques and tools with the most up-to-date machine learning and artificial intelligence considerations, and position these in the context of the most prominent platforms and analytics techniques. The paper develops a logical analytics stack to support the description of key functionality and relationships between relevant components in RTA solutions based on a thorough literature review and industrial practice. This provides practitioners with guidance in selecting the most appropriate solutions for their RTA problems, including the application of emerging AI technologies in this context. The paper discusses the complex event processing technology that has influenced many recent data streaming solutions in the analytics stack and highlights the integration of machine learning and artificial intelligence into RTA solutions. Some real-life application scenarios in the finance and health domains are presented, including several of the authors' earlier contributions, to demonstrate the utilization of the techniques and technologies discussed in this paper. Future research directions and remaining challenges are discussed.

Automated summary

With the proliferation of intelligent devices, social media, and the Internet of Things, a huge amount of new data is being generated, resulting in an increasing pace. Real-time analytics has emerged as a branch of big data analytics that focuses on the velocity aspect, where data is processed and analyzed as it arrives, aiming to provide insights and create business value in near real-time. This paper provides an overview of key concepts and architectural approaches for designing real-time analytics solutions, including infrastructure, processing and analytics platforms, machine learning, and artificial intelligence. It also presents real-life application scenarios and discusses the integration of machine learning and artificial intelligence into real-time analytics solutions. Future research directions and challenges are also discussed.