Problem with robots.txt file - Site Owners Forums

basrtaj · 11-18-2016, 02:12 PM

Hi friends !
Please i use wordpress for the first time, but i have a problem with robots.txt in webmaster tool, they show me that i have a problem in my robots.txt
Can you help me???

Ajaysharma · 11-18-2016, 11:45 PM

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth.

Pitradev · 11-19-2016, 12:01 AM

Simply saying, Robot.txt is the text file that can stop the crawling by the search engines.

henrybolt · 11-19-2016, 12:46 AM

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

sudeepkhana · 11-19-2016, 01:33 AM

Robots.txt is a text (not HTML) file you put on your site to tell search robots which pages you would like them not to visit. The robots.txt is a very simple text file that is placed in your root directory.

BryceBrady · 11-21-2016, 12:36 AM

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

basrtaj · 11-22-2016, 04:44 AM

Thank you all, but Should i add it to my website Host or no???

BryceBrady · 11-29-2016, 01:55 AM

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

Radhika Wadhwa · 11-29-2016, 02:11 AM

Simply saying, Robot.txt is the text file that can stop the crawling by the search engines.

basrtaj · 12-31-2016, 07:07 AM

Thanks a lot, it's really helpful!

11-18-2016, 02:12 PM	#1
basrtaj Registered User Join Date: Nov 2014 Posts: 14	Problem with robots.txt file Hi friends ! Please i use wordpress for the first time, but i have a problem with robots.txt in webmaster tool, they show me that i have a problem in my robots.txt Can you help me??? __________________ Hi! Your Bro Mohamed Amine, a simple man who love blogging and search engin SEO. this is one of my websites: To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-18-2016, 11:45 PM	#2
Ajaysharma Registered User Join Date: Nov 2016 Posts: 372	"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-19-2016, 12:01 AM	#3
Pitradev Registered User Join Date: Nov 2015 Posts: 85	Simply saying, Robot.txt is the text file that can stop the crawling by the search engines. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-19-2016, 12:46 AM	#4
henrybolt Registered User Join Date: Oct 2016 Posts: 655	"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-19-2016, 01:33 AM	#5
sudeepkhana Registered User Join Date: Jan 2016 Location: India Posts: 1,258	Robots.txt is a text (not HTML) file you put on your site to tell search robots which pages you would like them not to visit. The robots.txt is a very simple text file that is placed in your root directory. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-21-2016, 12:36 AM	#6
BryceBrady Registered User Join Date: Oct 2016 Posts: 580	"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-22-2016, 04:44 AM	#7
basrtaj Registered User Join Date: Nov 2014 Posts: 14	Thank you all, but Should i add it to my website Host or no??? __________________ Hi! Your Bro Mohamed Amine, a simple man who love blogging and search engin SEO. this is one of my websites: To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-29-2016, 01:55 AM	#8
BryceBrady Registered User Join Date: Oct 2016 Posts: 580	"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-29-2016, 02:11 AM	#9
Radhika Wadhwa Registered User Join Date: Nov 2016 Posts: 42	Simply saying, Robot.txt is the text file that can stop the crawling by the search engines.

12-31-2016, 07:07 AM	#10
basrtaj Registered User Join Date: Nov 2014 Posts: 14	Thanks a lot, it's really helpful! __________________ Hi! Your Bro Mohamed Amine, a simple man who love blogging and search engin SEO. this is one of my websites: To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)