|
![]() |
#1 |
Registered User
Join Date: Apr 2017
Posts: 74
|
Crawl time is a function of inbound bandwidth to the crawling machines, the processing time on the crawling machine, the time to send the result of the analysis to another machine, and the crawl delay as specified in each sites robots.txt file. Multiply the number of pages at a site times the site's crawl delay to get the total time to crawl the site politely. If images are being downloaded, processed, and uploaded to another machine all sorts of bottlenecks can slow down the download process. Crawling is typically done using parallel machines
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
![]() |
![]() |
![]() |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Display Modes | Rate This Thread |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Secrete for seo | alina albert | Search Engine Optimization | 87 | 03-11-2018 05:47 AM |
Why to Decrease the Website Load Time on Google | ogeninfosystem | Post your ad here | 0 | 03-16-2017 01:52 AM |
Google website promotion techniques | zamina | Web Promotion | 7 | 09-24-2013 06:00 AM |