What is use of robots.txt? - Site Owners Forums

weddingcardswal · 01-28-2016, 05:10 AM

What is use of robots.txt?

towertech · 01-28-2016, 06:26 AM

Hi,

Robots.txt : You can disallow search engine bot via robots.txt

in robots.txt you can put the list of web URL of your website that you don;t want to crawl by Google bot.

Thanks

Nexevo Technolo · 01-28-2016, 07:15 AM

Robots txt file is the used to intimate the information to your website in Crawler.

quantumleap · 01-28-2016, 09:15 PM

Robots.txt file is use to intimate the information about your website to the search engines so that they can easy navigate your website !

austin11 · 01-28-2016, 10:42 PM

Robots.txt file is used to tell search robots which pages you would like them not to visit. By defining a few rules in this text file, you can instruct robots to not crawl certain files, directories within your site, or at all.

Roshantyagi · 01-29-2016, 12:15 AM

The main need of robot.txt.file is to dictate how spiders or robots will crawl your website If on a website you do not use robot.txt file then search engine will crawl all pages on the website.

RH-Calvin · 01-29-2016, 03:56 AM

Robots.txt is a text file inserted into the website. The file lists webpages that needs to be crawled and not allowed to crawl by search engine spiders.

ashwani566 · 01-29-2016, 05:54 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

01-28-2016, 05:10 AM	#1
weddingcardswal Registered User Join Date: Aug 2015 Posts: 58	What is use of robots.txt? What is use of robots.txt? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-28-2016, 07:15 AM	#3
Nexevo Technolo Registered User Join Date: Sep 2015 Location: Bangalore Posts: 548	Robots txt file is the used to intimate the information to your website in Crawler. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-28-2016, 10:42 PM	#5
austin11 Registered User Join Date: Oct 2013 Posts: 77	Robots.txt file is used to tell search robots which pages you would like them not to visit. By defining a few rules in this text file, you can instruct robots to not crawl certain files, directories within your site, or at all. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-29-2016, 12:15 AM	#6
Roshantyagi Registered User Join Date: Jan 2016 Posts: 24	The main need of robot.txt.file is to dictate how spiders or robots will crawl your website If on a website you do not use robot.txt file then search engine will crawl all pages on the website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-29-2016, 03:56 AM	#7
RH-Calvin Registered User Join Date: May 2015 Location: Forum Posts: 2,517	Robots.txt is a text file inserted into the website. The file lists webpages that needs to be crawled and not allowed to crawl by search engine spiders. __________________ █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| $1 VPS Hosting █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Windows Server with Remote Desktop Access █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Servers with Free IPMI Setup

01-28-2016, 06:26 AM	#2
towertech Registered User Join Date: Jan 2016 Posts: 38	Hi, Robots.txt : You can disallow search engine bot via robots.txt in robots.txt you can put the list of web URL of your website that you don;t want to crawl by Google bot. Thanks

01-28-2016, 09:15 PM	#4
quantumleap Registered User Join Date: Jan 2016 Location: Hyderabad Posts: 4,287	Robots.txt file is use to intimate the information about your website to the search engines so that they can easy navigate your website !

01-29-2016, 05:54 AM	#8
ashwani566 Registered User Join Date: Jan 2016 Posts: 13	Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM