|
'Crawler', ‘bot’ or ‘spider’ follows an algorithmic process to determine which sites to crawl and how often. As a search engine's crawler moves through your site it will detect and record any links it finds on these pages and add them to a list that will be crawled later.
|