Site Owners Forums - Webmaster Forums - What is the main usage of search engine spiders?

Site Owners Forums - Webmaster Forums (http://siteownersforums.com/index.php)

- General Discussion (http://siteownersforums.com/forumdisplay.php?f=2)

- - What is the main usage of search engine spiders? (http://siteownersforums.com/showthread.php?t=201808)

tennywilson

09-01-2017 12:18 AM

What is the main usage of search engine spiders?

What is the main usage of search engine spiders?

checkifsccode

09-01-2017 02:25 AM

The spider for the AltaVista search engine and its Web site is called Scooter . Scooter adheres to the rules of politeness for Web spiders that are specified in the Standard for Robot Exclusion (SRE)

All times are GMT -7. The time now is 11:00 PM.