Efficient Distributed Web Crawling Utilizing Internet Resources

Xiao XU  Weizhe ZHANG  Hongli ZHANG  Binxing FANG  

IEICE TRANSACTIONS on Information and Systems   Vol.E93-D   No.10   pp.2747-2762
Publication Date: 2010/10/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.E93.D.2747
Print ISSN: 0916-8532
Type of Manuscript: PAPER
Category: Data Engineering, Web Information Systems
Internet computing,  distributed Web crawling,  DHT,  network coordinate system,  load balancing,  

Full Text: PDF(2.2MB)>>
Buy this Article

Internet computing is proposed to exploit personal computing resources across the Internet in order to build large-scale Web applications at lower cost. In this paper, a DHT-based distributed Web crawling model based on the concept of Internet computing is proposed. Also, we propose two optimizations to reduce the download time and waiting time of the Web crawling tasks in order to increase the system's throughput and update rate. Based on our contributor-friendly download scheme, the improvement on the download time is achieved by shortening the crawler-crawlee RTTs. In order to accurately estimate the RTTs, a network coordinate system is combined with the underlying DHT. The improvement on the waiting time is achieved by redirecting the incoming crawling tasks to light-loaded crawlers in order to keep the queue on each crawler equally sized. We also propose a simple Web site partition method to split a large Web site into smaller pieces in order to reduce the task granularity. All the methods proposed are evaluated through real Internet tests and simulations showing satisfactory results.