View Single Post
Old 04-11-2016, 12:02 AM   #13
Anubhav-soin
Registered User
 
Join Date: Feb 2016
Location: Delhi , India
Posts: 17
Its a file that is used by search engines to find out that which pages in your website you want to crawled.

Note : do not edit the robot.txt yourself if you do not know how to modify it otherwise you good pages might get blocked from search engine.

You can also use site:www.yousite.com in google to find what are the pages that are currently been crawled from your website.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Last edited by Anubhav-soin; 04-11-2016 at 12:03 AM.. Reason: Improved answer
Anubhav-soin is offline   Reply With Quote