05-18-2016, 04:23 AM
|
#1 |
|
Registered User
Join Date: Nov 2015
Posts: 76
|
Is it necessary to use Robots.txt?
Hi,
My question is, Is it necessary to use Robots.txt?
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
05-18-2016, 04:33 AM
|
#2 |
|
Registered User
Join Date: Mar 2016
Posts: 75
|
Yes, we need robots.txt in our website. Google spiders first see in your website robots.txt to which page they crawl or which not.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
05-18-2016, 04:47 AM
|
#3 |
|
Registered User
Join Date: Feb 2014
Posts: 197
|
Robots.txt files are useful:
If you want search engines to ignore any duplicate pages on your website If you don’t want search engines to index your internal search results pages If you don’t want search engines to index certain areas of your website or a whole website If you don’t want search engines to index certain files on your website (images, PDFs, etc.) If you want to tell search engines where your sitemap is located
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
05-18-2016, 11:44 PM
|
#4 |
|
Registered User
Join Date: May 2016
Posts: 86
|
Robot.txt is using for identifying the located map or pages from which the users identify the web page.
|
|
|
|
|
05-18-2016, 11:50 PM
|
#5 |
|
Registered User
Join Date: Jan 2016
Location: India
Posts: 1,258
|
A robots.txt record is utilized to tell search motors which pages you don't need to be slithered/listed.
There are different reasons you may wish to obstruct certain pages on a site : - you know certain pages have copy content - private pages (not for general visibility) - administrator pages - test/draft/under development pages - pages with blunders
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
05-19-2016, 01:07 AM
|
#6 |
|
Registered User
Join Date: Feb 2016
Posts: 454
|
yes its keep your confidential file hide from searchengine spider.
|
|
|
|
|
05-19-2016, 01:31 AM
|
#7 |
|
Registered User
Join Date: Jun 2014
Posts: 209
|
Yes. robots.txt for our website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
05-19-2016, 11:15 PM
|
#8 |
|
Registered User
Join Date: May 2015
Posts: 105
|
It is not important use robots. txt file but it is advisable to use it, if there is any page on your websites which you don't want search engine to crawl. Robots.txt also improves better indexing for the website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
05-28-2016, 12:57 AM
|
#9 |
|
Registered User
Join Date: May 2016
Posts: 551
|
yes, it is must we should include robots.txt
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
05-28-2016, 01:13 AM
|
#10 |
|
Registered User
Join Date: May 2016
Posts: 72
|
yes robots.txt is very important for the site.
|
|
|
|
|
05-28-2016, 10:24 PM
|
#11 |
|
Registered User
Join Date: May 2016
Posts: 14
|
it is a guide for google bots, to crawl your site in your way. you can use it,not crawl any type of file u want.
__________________
pawan kumar SEO guide To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
05-31-2016, 12:01 AM
|
#12 |
|
Registered User
Join Date: May 2016
Posts: 115
|
Robots.txt is a text file that is uploaded to a website's root directory and linked in the html code of the website. The robots.txt is used to provide instructions about the website to webrobots and spiders.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
06-01-2016, 08:58 PM
|
#13 |
|
Registered User
Join Date: May 2016
Posts: 141
|
Google spiders first see in your website robots.txt to which page they crawl or which not.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Crawlers, spider & robots.txt? | jackthomas087 | Search Engine Optimization | 0 | 04-22-2014 03:29 AM |
| Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
| How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
| The robots.txt file | rubberfender | Yahoo | 0 | 12-02-2012 07:32 PM |
| Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
All times are GMT -7. The time now is 02:48 AM.