View Single Post
Old 04-22-2017, 11:41 AM   #3
cultratradseo
Registered User
 
Join Date: Apr 2017
Posts: 85
Crawl time is a function of inbound bandwidth to the crawling machines, the processing time on the crawling machine, the time to send the result of the analysis to another machine, and the crawl delay as specified in each sites robots.txt file. Multiply the number of pages at a site times the site's crawl delay to get the total time to crawl the site politely. If images are being downloaded, processed, and uploaded to another machine all sorts of bottlenecks can slow down the download process. Crawling is typically done using parallel machines
__________________
Cultatrad
cultratradseo is offline   Reply With Quote