02-25-2015, 04:15 AM
|
#1 |
|
Registered User
Join Date: Nov 2014
Posts: 972
|
how to work robot txt and crawl
how to work robot txt and crawl
__________________
Tyre Recycling Plant,waste tyre recycling plant,tyre waste recycling plant,used tyre recycling plant, waste tyre plant,waste tyre to oil, tyre recycling machinery, waste tyre recycling machine, waste tyre recycling machinery, tyre recycling plant manufacturer, tyre recycling plant manufacturers,tyre recycling plant manufacturer in india |
|
|
|
02-26-2015, 10:23 PM
|
#2 |
|
Registered User
Join Date: Feb 2015
Posts: 27
|
new info
I recommend you getting a moderate delay as, it will work better for you.
Here is a post on robots.txt. It is actually for a wordpress website, but it explains in detail, and should give you an idea of what you might want to do with yours. boganmarketing.com/blog/ultimate |
|
|
|
|
02-26-2015, 10:31 PM
|
#3 |
|
Registered User
Join Date: Nov 2014
Location: UK
Posts: 26
|
Robot.txt is a file to show way to crawler which page is index in their database.
__________________
Send Money Now |
|
|
|
|
03-01-2015, 09:41 PM
|
#4 |
|
Registered User
Join Date: Dec 2012
Posts: 657
|
Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.
|
|
|
|
|
03-02-2015, 01:36 AM
|
#5 |
|
Registered User
Join Date: Mar 2015
Posts: 20
|
Robot.txt is file used for not allow google to crawl particular webpage.
|
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Linear Mode
|
|
All times are GMT -7. The time now is 04:49 AM.