how to work robot txt and crawl - Site Owners Forums

anjaliexim · 02-25-2015, 04:15 AM

how to work robot txt and crawl

LyleRedus · 02-26-2015, 10:23 PM

I recommend you getting a moderate delay as, it will work better for you.
Here is a post on robots.txt. It is actually for a wordpress website, but it explains in detail, and should give you an idea of what you might want to do with yours.
boganmarketing.com/blog/ultimate

James_Tarrier · 02-26-2015, 10:31 PM

Robot.txt is a file to show way to crawler which page is index in their database.

jaysh4922 · 03-01-2015, 09:41 PM

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

saisunsolution · 03-02-2015, 01:36 AM

Robot.txt is file used for not allow google to crawl particular webpage.

02-25-2015, 04:15 AM	#1
anjaliexim Registered User Join Date: Nov 2014 Posts: 972	how to work robot txt and crawl how to work robot txt and crawl __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-26-2015, 10:23 PM	#2
LyleRedus Registered User Join Date: Feb 2015 Posts: 27	new info I recommend you getting a moderate delay as, it will work better for you. Here is a post on robots.txt. It is actually for a wordpress website, but it explains in detail, and should give you an idea of what you might want to do with yours. boganmarketing.com/blog/ultimate

02-26-2015, 10:31 PM	#3
James_Tarrier Registered User Join Date: Nov 2014 Location: UK Posts: 26	Robot.txt is a file to show way to crawler which page is index in their database. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

03-01-2015, 09:41 PM	#4
jaysh4922 Registered User Join Date: Dec 2012 Posts: 657	Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-02-2015, 01:36 AM	#5
saisunsolution Registered User Join Date: Mar 2015 Posts: 20	Robot.txt is file used for not allow google to crawl particular webpage. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)