Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 03-06-2012, 03:35 AM   #1
samlko
Registered User
 
Join Date: Sep 2009
Posts: 67
What Is Robots.txt?

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
samlko is offline   Reply With Quote

Old 03-06-2012, 03:46 AM   #2
josefaryan
Registered User
 
Join Date: Mar 2012
Posts: 259
Robots.txt file can be used to tell search engines that which of your website pages/directories should not be crawled.
josefaryan is offline   Reply With Quote
Old 03-06-2012, 03:52 AM   #3
blueapple
Registered User
 
Join Date: Feb 2012
Posts: 92
Somehow you can say like this but purpose is same to disallow search engine to index the content.
blueapple is offline   Reply With Quote
Old 03-06-2012, 04:06 AM   #4
amitmax
Registered User
 
Join Date: Aug 2010
Posts: 215
Very true.Generally we use it to save unusual crawling of spider on those pages having duplicate content or files which we don't want to share publicly.
Thanks
amitmax is offline   Reply With Quote
Old 03-07-2012, 04:28 AM   #5
Johnmark
Registered User
 
Join Date: Dec 2011
Posts: 153
it gives the instruction to not crawl the specific pages or file mention in txt
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Johnmark is offline   Reply With Quote
Old 03-08-2012, 01:13 AM   #6
jessysmith
Registered User
 
Join Date: Feb 2012
Posts: 27
Robots.txt will inform the search engine what are the pages should be crawled
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
jessysmith is offline   Reply With Quote
Old 03-08-2012, 02:52 AM   #7
andersonsjohn
Registered User
 
Join Date: Jan 2012
Posts: 177
Robots.txt will tell the search engine what to crawl and what not to. This helps if you do not want search engines to crawl and index some of your private data.
andersonsjohn is offline   Reply With Quote
Old 03-08-2012, 03:54 AM   #8
froz
Registered User
 
Join Date: Feb 2012
Posts: 86
Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site..
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
froz is offline   Reply With Quote
Old 03-08-2012, 07:13 AM   #9
jessica733
Registered User
 
Join Date: Jul 2011
Location: Nicosia, Cyprus
Posts: 57
Thanks to share this great information with us, Robot.txt files are very important in on-page field to make our website error free.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
jessica733 is offline   Reply With Quote
Old 03-08-2012, 11:30 PM   #10
wheelbarrow001
Registered User
 
Join Date: Nov 2011
Posts: 35
The robots.txt is for a search engine, visit the site of the time want to see the first file. The robots.txt file told spiders in the server is what document can be looked at.
When a search spider visit a website, it will first check the site to root directory whether there is the robots.txt, if present, search according to the document robot to determine the content of the scope of the visit; If the file does not exist, all the search spiders will be able to access web site was not all password protected pages.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
wheelbarrow001 is offline   Reply With Quote
Old 03-08-2012, 11:40 PM   #11
atoment01
Registered User
 
Join Date: Mar 2012
Posts: 15
robot.txt file is simple text document.
1. open notepad
2. type these lines.
3. save your file as robots.txt (extension should be .txt)
4. placed this file in your root folder
5. you can view it
atoment01 is offline   Reply With Quote
Old 03-09-2012, 03:33 AM   #12
Diane Forster
Registered User
 
Join Date: Aug 2011
Posts: 290
It is a code within the root file that acts to avoid spiders from crawling over the site or page. It is a technique used in SEO to avoid indexing.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Junkies
Diane Forster is offline   Reply With Quote
Old 03-09-2012, 04:04 AM   #13
harryjohn99
Registered User
 
Join Date: Nov 2011
Posts: 51
Robot.txt is the way to stop the search engines to crawl any page of the website. This is required because some time there is private data on some pages of the website and the website owner does not want search engines to crawl those pages and read that useful information. In those cases robot.txt is the file in which all the pages that we don't want to crawl are added and this file is then added to the root folder of the website.

Last edited by harryjohn99; 01-08-2014 at 12:44 AM..
harryjohn99 is offline   Reply With Quote
Old 03-09-2012, 02:37 PM   #14
electra
Registered User
 
Join Date: Mar 2012
Posts: 44
Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

Thanks and Regards
__________________
Need webhosting now? Read
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
or for business hosting read
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
. If You like, please also visit
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
to get some discount.
Thank you
electra is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
sitemap.xml and robots.txt? jamesranatte Search Engine Optimization 5 01-31-2012 11:16 PM


All times are GMT -7. The time now is 08:09 AM.


Powered by vBulletin Copyright © 2020 vBulletin Solutions, Inc.