|
Robot.txt is a text file kept on the web servers. The robot.txt allows the site admin to permit/allow which Search engines(like google, yahoo,bing) can crawl the site. Some times the admins won't allow any of the search engines to crawl at all.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|