Site Owners Forums - Webmaster Forums

Site Owners Forums - Webmaster Forums (http://siteownersforums.com/index.php)
-   Search Engine Optimization (http://siteownersforums.com/forumdisplay.php?f=16)
-   -   how to work robot txt and crawl (http://siteownersforums.com/showthread.php?t=152466)

anjaliexim 02-25-2015 04:15 AM

how to work robot txt and crawl
 
how to work robot txt and crawl

LyleRedus 02-26-2015 10:23 PM

new info
 
I recommend you getting a moderate delay as, it will work better for you.
Here is a post on robots.txt. It is actually for a wordpress website, but it explains in detail, and should give you an idea of what you might want to do with yours.
boganmarketing.com/blog/ultimate

James_Tarrier 02-26-2015 10:31 PM

Robot.txt is a file to show way to crawler which page is index in their database.

jaysh4922 03-01-2015 09:41 PM

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

saisunsolution 03-02-2015 01:36 AM

Robot.txt is file used for not allow google to crawl particular webpage. :)


All times are GMT -7. The time now is 05:28 AM.


Powered by vBulletin Copyright © 2020 vBulletin Solutions, Inc.