![]() |
#5 |
Registered User
Join Date: Nov 2012
Posts: 76
|
Robots.txt files inform search engine spiders how to interact with indexing your content. If you do not have a robots.txt file, your server logs will return 404 errors whenever a bot tries to access your robots.txt file.
|
![]() |
![]() |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
Robots.txt | sameerseo | Search Engine Optimization | 26 | 09-28-2012 04:18 AM |
robots.txt | wheelbarrow001 | 2 | 07-27-2012 04:10 AM | |
Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
What Is Robots.txt? | samlko | Search Engine Optimization | 13 | 03-09-2012 02:37 PM |