View Single Post
Old 04-22-2017, 12:41 PM   #3
Registered User
Join Date: Apr 2017
Posts: 74
Crawl time is a function of inbound bandwidth to the crawling machines, the processing time on the crawling machine, the time to send the result of the analysis to another machine, and the crawl delay as specified in each sites robots.txt file. Multiply the number of pages at a site times the site's crawl delay to get the total time to crawl the site politely. If images are being downloaded, processed, and uploaded to another machine all sorts of bottlenecks can slow down the download process. Crawling is typically done using parallel machines

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
cultratradseo is offline   Reply With Quote