Web Observatory Insights: Past, Current, and Future

In the present era of Big Data, with continuously increasing amounts of user-generated content, it is becoming a challenge to understand the relation between the content that is available on the Web and the users who are generating that content. Researchers have come up with many ways to understand today's Web better. One of the recently introduced concepts is a Web observatory (WO). This article provides a deep understanding about web observatories. It discusses the status of existing WO systems. The article investigates and gathers the common practices of WOs. This research has implications for researchers and communities in the adoption of the WO concept. The article highlights the challenges of WOs, such as data crawling, privacy and security. It also provides future research and development directions. The article provides a comparative analysis of existing WOs. It discusses the architecture of WOs. It presents components of a WO in a coherent manner and finally provides insights into challenges and limitations of WOs.