Site Owners Forums - Webmaster Forums - View Single Post

lylevasser12 · 08-31-2012, 07:14 AM

Hello ,
It's a text file which instructs search engine spiders or crawlers on what to do. It tells specific web spiders on which specific web pages to index.

08-31-2012, 07:14 AM	#4
lylevasser12 Registered User Join Date: Aug 2012 Posts: 69	Hello , It's a text file which instructs search engine spiders or crawlers on what to do. It tells specific web spiders on which specific web pages to index.