02-25-2015, 04:15 AM | #1 |
Registered User
Join Date: Nov 2014
Posts: 972
|
how to work robot txt and crawl
how to work robot txt and crawl
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
02-26-2015, 10:23 PM | #2 |
Registered User
Join Date: Feb 2015
Posts: 27
|
new info
I recommend you getting a moderate delay as, it will work better for you.
Here is a post on robots.txt. It is actually for a wordpress website, but it explains in detail, and should give you an idea of what you might want to do with yours. boganmarketing.com/blog/ultimate |
02-26-2015, 10:31 PM | #3 |
Registered User
Join Date: Nov 2014
Location: UK
Posts: 26
|
Robot.txt is a file to show way to crawler which page is index in their database.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-01-2015, 09:41 PM | #4 |
Registered User
Join Date: Dec 2012
Posts: 657
|
Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-02-2015, 01:36 AM | #5 |
Registered User
Join Date: Mar 2015
Posts: 20
|
Robot.txt is file used for not allow google to crawl particular webpage.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
|
|