|
The robots.txt is for a search engine, visit the site of the time want to see the first file. The robots.txt file told spiders in the server is what document can be looked at.
When a search spider visit a website, it will first check the site to root directory whether there is the robots.txt, if present, search according to the document robot to determine the content of the scope of the visit; If the file does not exist, all the search spiders will be able to access web site was not all password protected pages.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|