how to work robot txt and crawl
 

how to work robot txt and crawl

new info
 

I recommend you getting a moderate delay as, it will work better for you.
Here is a post on robots.txt. It is actually for a wordpress website, but it explains in detail, and should give you an idea of what you might want to do with yours.
boganmarketing.com/blog/ultimate

Robot.txt is a file to show way to crawler which page is index in their database.

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

Robot.txt is file used for not allow google to crawl particular webpage. :)