What is robot.txt file and what are the benefits of using it? - Site Owners Forums

Martinricky · 03-05-2016, 01:39 AM

Hello,

What is robot.txt file and what are the benefits of using it?

Thank you

aditya12 · 03-05-2016, 04:23 AM

Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not.

Best schools in Chandigarh Best schools in Mohali Best schools in Panchkula Best schools in Tricity Colleges in Chandigarh Colleges in Mohali Colleges in Tricity Colleges in Panchkula

tenso500 · 03-05-2016, 04:52 AM

The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

Admyrin · 03-05-2016, 05:17 AM

Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not.

langtu1292 · 03-24-2016, 06:46 AM

you can search on google..many thing for you

josechukkiri · 03-24-2016, 07:14 AM

The explanation about robot.txt files are satisfactory to me and thanks to the concerned.

sudeepkhana · 03-25-2016, 05:57 AM

Robot.txt is a text file kept on the web servers. The robot.txt allows the site admin to permit/allow which Search engines(like google, yahoo,bing) can crawl the site. Some times the admins won't allow any of the search engines to crawl at all.

Devin Mataka · 03-25-2016, 06:06 AM

The robots.txt file is a simple text file (no html) that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip. Many webmasters utilize this file to help the search engines index the content of their websites.
If webmasters can tell the search engine spiders to skip pages that they do not consider important enough to be crawled (eg. printable versions of pages, .pdf files etc.), then they have a better opportunity to have their most valuable pages featured in the search engine results pages.The robots.txt file is a simple method of essentially easing the process for the spiders to return the most relevant search results.
Another benefit of having a robots.txt is that you can specify the location of the Google .xml or Yahoo sitemap. This also increases spiderability for the search engines.

rahul3214 · 04-07-2016, 09:42 PM

Robots.txt file is important for seo, search engine spiders first see that where your robots.txt file because they do not know which file crawl or which not. In robots.txt file google know which file crawl.

karanjain · 04-08-2016, 12:27 PM

nice and awesome
IPL 9 - Indian Premier League 2016
IPL Auction 2016 Live Results Shane Watson And Yuvraj Sold On Highest Bidding

albertocosta · 04-09-2016, 04:57 AM

The robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

brknny · 04-10-2016, 11:27 PM

Robots.txt file is a file through which we give instructions to the search engines whether to crawl or not the particular web page.

Anubhav-soin · 04-11-2016, 12:02 AM

Its a file that is used by search engines to find out that which pages in your website you want to crawled.

Note : do not edit the robot.txt yourself if you do not know how to modify it otherwise you good pages might get blocked from search engine.

You can also use site:www.yousite.com in google to find what are the pages that are currently been crawled from your website.

williamstinner · 04-11-2016, 10:51 PM

Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.

jasonroy21 · 11-24-2016, 11:27 PM

The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

03-05-2016, 01:39 AM	#1
Martinricky Registered User Join Date: Aug 2015 Posts: 434	What is robot.txt file and what are the benefits of using it? Hello, What is robot.txt file and what are the benefits of using it? Thank you __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-05-2016, 04:23 AM	#2
aditya12 Registered User Join Date: Jan 2016 Location: India Posts: 271	Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not. Best schools in Chandigarh Best schools in Mohali Best schools in Panchkula Best schools in Tricity Colleges in Chandigarh Colleges in Mohali Colleges in Tricity Colleges in Panchkula __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-05-2016, 04:52 AM	#3
tenso500 Registered User Join Date: Feb 2016 Posts: 51	The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-05-2016, 05:17 AM	#4
Admyrin Registered User Join Date: Feb 2016 Location: Surat Posts: 396	Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-24-2016, 06:46 AM	#5
langtu1292 Registered User Join Date: Mar 2016 Posts: 118	you can search on google..many thing for you __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-24-2016, 07:14 AM	#6
josechukkiri Registered User Join Date: Apr 2015 Location: India Posts: 34	The explanation about robot.txt files are satisfactory to me and thanks to the concerned. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-25-2016, 05:57 AM	#7
sudeepkhana Registered User Join Date: Jan 2016 Location: India Posts: 1,258	Robot.txt is a text file kept on the web servers. The robot.txt allows the site admin to permit/allow which Search engines(like google, yahoo,bing) can crawl the site. Some times the admins won't allow any of the search engines to crawl at all. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-25-2016, 06:06 AM	#8
Devin Mataka Registered User Join Date: Mar 2016 Location: New York Posts: 2	The robots.txt file is a simple text file (no html) that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip. Many webmasters utilize this file to help the search engines index the content of their websites. If webmasters can tell the search engine spiders to skip pages that they do not consider important enough to be crawled (eg. printable versions of pages, .pdf files etc.), then they have a better opportunity to have their most valuable pages featured in the search engine results pages.The robots.txt file is a simple method of essentially easing the process for the spiders to return the most relevant search results. Another benefit of having a robots.txt is that you can specify the location of the Google .xml or Yahoo sitemap. This also increases spiderability for the search engines.

04-07-2016, 09:42 PM	#9
rahul3214 Registered User Join Date: Mar 2016 Posts: 75	Robots.txt file is important for seo, search engine spiders first see that where your robots.txt file because they do not know which file crawl or which not. In robots.txt file google know which file crawl. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-08-2016, 12:27 PM	#10
karanjain Registered User Join Date: Jul 2015 Posts: 12	nice and awesome IPL 9 - Indian Premier League 2016 IPL Auction 2016 Live Results Shane Watson And Yuvraj Sold On Highest Bidding __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-09-2016, 04:57 AM	#11
albertocosta Registered User Join Date: Apr 2016 Posts: 83	The robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

04-10-2016, 11:27 PM	#12
brknny Registered User Join Date: Feb 2016 Posts: 110	Robots.txt file is a file through which we give instructions to the search engines whether to crawl or not the particular web page. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-11-2016, 12:02 AM	#13
Anubhav-soin Registered User Join Date: Feb 2016 Location: Delhi , India Posts: 17	Its a file that is used by search engines to find out that which pages in your website you want to crawled. Note : do not edit the robot.txt yourself if you do not know how to modify it otherwise you good pages might get blocked from search engine. You can also use site:www.yousite.com in google to find what are the pages that are currently been crawled from your website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. Last edited by Anubhav-soin; 04-11-2016 at 12:03 AM.. Reason: Improved answer

04-11-2016, 10:51 PM	#14
williamstinner Registered User Join Date: Mar 2016 Posts: 20	Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.

11-24-2016, 11:27 PM	#15
jasonroy21 Registered User Join Date: Jul 2016 Posts: 755	The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)