Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 07-24-2012, 03:36 AM   #1
sameerseo
Registered User
 
Join Date: Mar 2012
Posts: 140
Robots.txt

Do you know anything about robots.txt?
sameerseo is offline   Reply With Quote

Old 07-24-2012, 05:04 AM   #2
wyanesmith1987
Registered User
 
Join Date: Jun 2012
Location: India
Posts: 250
Introduction to "robots.txt"

There is a hidden, relentless force that permeates the web and its billions of web pages and files, unbeknownst to the majority of us sentient beings. I'm talking about search engine crawlers and robots here. Every day hundreds of them go out and scour the web, whether it's Google trying to index the entire web, or a spam bot collecting any email address it could find for less than honorable intentions. As site owners, what little control we have over what robots are allowed to do when they visit our sites exist in a magical little file called "robots.txt."

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

if you want to read more so see this link..
http://www.javascriptkit.com/howto/robots.shtml
wyanesmith1987 is offline   Reply With Quote
Old 07-25-2012, 05:02 AM   #3
C.Rebecca
Registered User
 
Join Date: Dec 2011
Posts: 247
Robots.txt is a text (not html) file you put on your root directory to tell search robots which files to ignore (or alternatively) which files to crawl. It also helps Search Engines to locate the Sitemap of the website and hence crawl the entire website in depth... helping in your rankings and traffic.
__________________
230% more traffic with 12+ Keyword Research Tools
C.Rebecca is offline   Reply With Quote
Old 08-16-2012, 02:32 AM   #4
larsen81
Registered User
 
Join Date: Aug 2012
Posts: 36
Robot.txt is a file where you tell search engines which sections or pages of your site not to index.
larsen81 is offline   Reply With Quote
Old 08-16-2012, 04:25 AM   #5
davikerkrish
Registered User
 
Join Date: Jul 2012
Posts: 111
robots.txt is a text file that placed on your root directory. It can allows or disallows spiders or search engines from indexing the pages.
__________________
dialler | case management software
davikerkrish is offline   Reply With Quote
Old 08-16-2012, 06:05 AM   #6
Paul Jameson
Registered User
 
Join Date: Aug 2012
Posts: 22
Robot.txt is main function which guides search engine to find pages on your website to crawl. It is text file which you place on your root directory.
Paul Jameson is offline   Reply With Quote
Old 08-16-2012, 11:17 PM   #7
rajnish240
Registered User
 
Join Date: Jul 2012
Posts: 89
It is used to hide the privacy policy of a company from the Google's spider.So that your privacy are not visible publicly.
http://doxinh.com/danh-muc/do-lot-cao-cap/ Ao chip cao cap Quan lot doc Do ngu cao cap Do so sinh loai khac cao cap Cho thue trang phuc bieu dien Quan lot nam cao cap
__________________
AD Systems is trusted for LED video displays and showroom info directors. Its scrolling LED signs are highly durable, eco friendly and power saving.To know more visit here http://www.adsystemsled.com.

Last edited by rajnish240; 06-08-2013 at 12:17 PM..
rajnish240 is offline   Reply With Quote
Old 08-16-2012, 11:55 PM   #8
cheskaSEO
Registered User
 
Join Date: Aug 2012
Posts: 38
Quote:
Originally Posted by rajnish240 View Post
It is used to hide the privacy policy of a company from the Google's spider.So that your privacy are not visible publicly.
agree with this post
__________________
Most Affordable SEO Service that will get you a FAST RESULT for $5 CLICK HERE
Get Relevant Backlinks for your website!
website traffic rankings
cheskaSEO is offline   Reply With Quote
Old 08-17-2012, 02:55 AM   #9
Dilli Live
Registered User
 
Join Date: Jul 2012
Posts: 35
robot.txt is a text file which appears in the root directory in your website place. With help of this you can hide your unwanted website link for search engines.


Thanks,
Dilli Live is offline   Reply With Quote
Old 08-17-2012, 03:17 AM   #10
samaustin141
Registered User
 
Join Date: Aug 2012
Posts: 45
It is a text file which instructs search engine spiders or crawlers on what to do. It tells specific web spiders on which specific web pages to index. Robots are configured to read text.It contains restrictions for Web Spiders, telling them where they have permission to search. It is like defining rules for search engine spiders (robots) what to follow and what not to.
samaustin141 is offline   Reply With Quote
Old 08-18-2012, 05:06 AM   #11
AshleyScott
Registered User
 
Join Date: Jul 2012
Posts: 9
Robots. txt file is necessary at time where you want to instruct the crawler for the pages it is allowed to crawl.
__________________
Beaded Jewelry
AshleyScott is offline   Reply With Quote
Old 08-20-2012, 03:42 AM   #12
jiyaalbert
Registered User
 
Join Date: Apr 2011
Posts: 346
The robots.txt file is a set of instructions for robots visiting that index the content of your web site pages. For those spiders that obey the file, it provides a map for what they can, and cannot index. The file must reside in the root directory of your web.
__________________
Paintless Dent Repair
jiyaalbert is offline   Reply With Quote
Old 08-20-2012, 04:48 AM   #13
NateJacobs
Registered User
 
Join Date: Aug 2012
Posts: 9
Robots.txt is main use,if you don't want url indexing in Google,so use robot.txt.Many website owner are using robots.txt,so hacker don't hack site.
__________________
android application development
NateJacobs is offline   Reply With Quote
Old 08-21-2012, 02:07 AM   #14
georgemathew
Registered User
 
Join Date: Dec 2011
Posts: 73
Robots.txt is a text file it can allows or disallows spiders or search engines from indexing the pages.
georgemathew is offline   Reply With Quote
Old 08-23-2012, 10:59 PM   #15
nptifitness
Registered User
 
Join Date: Jun 2012
Posts: 133
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site
nptifitness is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
issue in robots.txt file davikerkrish Search Engine Optimization 0 07-20-2012 12:40 AM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 02:19 AM
Don�t exceed maximum file size for robots.txt file ClaudiaSchayffe Search Engine Optimization 3 04-02-2012 04:28 AM
What Is Robots.txt? samlko Search Engine Optimization 13 03-09-2012 02:37 PM
sitemap.xml and robots.txt? jamesranatte Search Engine Optimization 5 01-31-2012 11:16 PM


All times are GMT -7. The time now is 09:49 AM.


Powered by vBulletin Copyright © 2020 vBulletin Solutions, Inc.