Effective web crawling
Source:
SIGIR Forum, ACM Press, Volume 39, Number 1, New York, NY, USA, p.55--56 (2005)
URL:
http://portal.acm.org/citation.cfm?id=1067268.1067287
Keywords:
crawling
Abstract:
The key factors for the success of the World Wide Web are its large size and the lack of a centralized control over its contents. Both issues are also the most important source of problems for locating information. The Web is a context in which traditional Information Retrieval methods are challenged, and given the volume of the Web and its speed of change, the coverage of modern search engines is relatively small. Moreover, the distribution of quality is very skewed, and interesting pages are scarce in comparison with the rest of the content.
