Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 05-06-2014, 11:02 PM   #1
sahithya
Registered User
 
Join Date: Feb 2013
Location: Bangalore
Posts: 1,086
What is the purpose of robots.txt file?

What is the purpose of robots.txt file?
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
sahithya is offline   Reply With Quote

Old 05-06-2014, 11:24 PM   #2
Sarahbriggs
Registered User
 
Join Date: May 2014
Posts: 12
Robots.txt is a text file which you put on your website to tell crawler which pages you would like not to crawl.

You need to create a file and you have to enter that URL which you don't want to crawl. Have a look below.

User Agent: *
Disallow: //admin

Disallow: ///t141187-

Disallow: //file.html

Disallow: //purpose
__________________
Supreme Suppliers & Manufacturers of Plastic Furniture India - Find modern designer plastic moulded good quality Premium chairs/ Furnitures,dining Chair, tables, furniture.
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Sarahbriggs is offline   Reply With Quote
Old 05-06-2014, 11:32 PM   #3
vickyjohn
Registered User
 
Join Date: Nov 2013
Posts: 199
Robots.txt is a file, which user needs to add a webpage that do not want to crawl by the spider or crawler...
eg-
User Agent: *
Disallow: //officeadmin
Type in a notepad file and add it to website root folder ..
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
vickyjohn is offline   Reply With Quote
Old 05-06-2014, 11:48 PM   #4
employmentpakis
Registered User
 
Join Date: Feb 2014
Posts: 51
Robot.txt file is a text file that give instructions to search engine robots which pages you would like them not to visit.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
employmentpakis is offline   Reply With Quote
Old 05-07-2014, 12:43 AM   #5
spyindia01
Registered User
 
Join Date: Jul 2013
Posts: 49
A simple text file that stops Google (and other search engines that recognize the file and its commands) from crawling the site, selected pages in the site, or selected file types in the site.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
spyindia01 is offline   Reply With Quote
Old 05-07-2014, 02:38 AM   #6
ChristinaCa
Registered User
 
Join Date: Mar 2014
Posts: 210
Robot.txt file is a file which contains the webpages which one don't want to be crawled by the crawler.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
ChristinaCa is offline   Reply With Quote
Old 11-27-2017, 04:57 AM   #7
LydiaAaron
Registered User
 
Join Date: Sep 2017
Posts: 113
Robots.txt purpose not index for the any particular web pages in our web sites.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
LydiaAaron is offline   Reply With Quote
Old 11-27-2017, 05:07 AM   #8
Fozia khadim
Registered User
 
Join Date: Nov 2017
Posts: 4
Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robot.txt. Exclusion Protocol tells the robot that it should not visit any pages on the site.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Fozia khadim is offline   Reply With Quote
Old 11-27-2017, 08:00 AM   #9
infiniumtech
Registered User
 
Join Date: Nov 2017
Location: Surat, Gujarat, India
Posts: 36
Robots.txt is a text file webmasters create to instruct web robots. how to crawl pages on their website.
infiniumtech is offline   Reply With Quote
Old 11-27-2017, 08:16 PM   #10
AmandaCherry
Registered User
 
Join Date: Nov 2017
Posts: 342
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
AmandaCherry is offline   Reply With Quote
Old 02-13-2018, 08:30 AM   #11
ROYCPO
Registered User
 
Join Date: Dec 2015
Location: Bangladesh
Posts: 267
Robots.txt is the common name of a text file that is uploaded to a Web site's root directory and linked in the HTML code of the Web site. The robots.txt file is used to provide instructions to the Web site to Web robots and spiders. Web authors can use robots.txt
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Outsource
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
providing
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
services all over the world. Expert in
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Service.
ROYCPO is offline   Reply With Quote
Old 02-13-2018, 10:25 PM   #12
harshithaasin
Registered User
 
Join Date: Oct 2017
Posts: 306
The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
harshithaasin is offline   Reply With Quote
Old 02-24-2018, 08:51 AM   #13
deepakrajput
Registered User
 
Join Date: Feb 2015
Location: CA
Posts: 733
Robots.txt file is set of instructions help to index the website through search engines.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
deepakrajput is offline   Reply With Quote
Old 02-25-2018, 09:53 PM   #14
Kentowin
Registered User
 
Join Date: Feb 2018
Location: Hyderabad
Posts: 47
robots.txt file is the file which instruct search engine which part of your webpage you don't want to get crawl by the search engine spider.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Kentowin is offline   Reply With Quote
Old 02-25-2018, 10:56 PM   #15
harshithaasin
Registered User
 
Join Date: Oct 2017
Posts: 306
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
harshithaasin is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 12:36 PM
The robots.txt file rubberfender Yahoo 0 12-02-2012 07:32 PM
RSS Feed rajivwebads Search Engine Optimization 12 08-31-2012 07:15 AM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 02:19 AM
Don’t exceed maximum file size for robots.txt file ClaudiaSchayffe Search Engine Optimization 3 04-02-2012 04:28 AM


All times are GMT -7. The time now is 09:15 AM.


Powered by vBulletin Copyright © 2020 vBulletin Solutions, Inc.