Thread: Robots.txt
View Single Post
Old 09-28-2012, 12:33 AM   #10
3idatascraping
Registered User
 
Join Date: Aug 2012
Posts: 13
Robot.txt means to tell search engine of which pages you want to crawl or Not.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
3idatascraping is offline   Reply With Quote