期刊
SOFTWARE-PRACTICE & EXPERIENCE
卷 34, 期 8, 页码 711-726出版社
WILEY
DOI: 10.1002/spe.587
关键词
Web crawling; distributed computing; fault tolerance; consistent hashing
We report our experience in implementing UbiCrawler, a scalable distributed Web crawler, using the Java programming language. The main features of UbiCrawler are platform independence, linear scalability, graceful degradation in the presence of faults, a very effective assignment function (based on consistent hashing) for partitioning the domain to crawl, and more in general the complete decentralization of every task. The necessity of handling very large sets of data has highlighted some limitations of the Java APIs, which prompted the authors to partially reimplement them. Copyright (C) 2004 John Wiley Sons, Ltd.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据