Which is better Robots.txt or Meta Robots Tag? - Site Owners Forums

Sunshineskin · 03-06-2018, 09:24 PM

Which is better Robots.txt or Meta Robots Tag?

smmexpertss · 03-06-2018, 11:06 PM

It's a pretty common misconception that the robots.txt will prevent indexing. It's only purpose is actually to prevent crawling, anything disallowed in there is still up for indexing if it's linked to elsewhere. If you want something deindexed, your best bet is the robots meta tag, but make sure you allow crawling of the URLs to give search engine bots an opportunity to see the tag.

matildaeric · 03-06-2018, 11:54 PM

Both are supported by all crawlers which respect webmasters wishes not all do but against them neither technique is sufficient you can use robots.txt rules for general things like disallow whole sections of your.

buyinstagram · 03-07-2018, 08:36 AM

Msnbot, discobot, and Slurp are all called out specifically, so those user-agents will only pay attention to the directives in their sections of the robots.txt file. All other user-agents will follow the directives in the user-agent: * group.

nathanleo · 12-30-2021, 05:54 AM

Robots. txt files are best for disallowing a whole section of a site, such as a category whereas a meta tag is more efficient at disallowing single files and pages. You could choose to use both a meta robots tag and a robots.

03-06-2018, 09:24 PM	#1
Sunshineskin Registered User Join Date: Feb 2018 Location: D 47, Bali Nagar, (Next to Raja Garden community centre) Delhi, 110015, India Posts: 10	Which is better Robots.txt or Meta Robots Tag? Which is better Robots.txt or Meta Robots Tag? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-06-2018, 11:54 PM	#3
matildaeric Registered User Join Date: Jan 2018 Posts: 93	Both are supported by all crawlers which respect webmasters wishes not all do but against them neither technique is sufficient you can use robots.txt rules for general things like disallow whole sections of your. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

03-07-2018, 08:36 AM	#4
buyinstagram Registered User Join Date: Nov 2017 Posts: 253	Msnbot, discobot, and Slurp are all called out specifically, so those user-agents will only pay attention to the directives in their sections of the robots.txt file. All other user-agents will follow the directives in the user-agent: * group. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. / To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. / To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. / To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. / To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-30-2021, 05:54 AM	#5
nathanleo Registered User Join Date: Jul 2017 Posts: 2,060	Robots. txt files are best for disallowing a whole section of a site, such as a category whereas a meta tag is more efficient at disallowing single files and pages. You could choose to use both a meta robots tag and a robots. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Which is better Robots.txt or Meta Robots Tag	doorhardwareuk	Search Engine Optimization	0	07-10-2017 11:17 PM
Which is better Robots.txt or Meta Robots Tag?	mithucarrentals	Search Engine Optimization	0	01-30-2017 04:04 AM
Which is better Robots.txt or Meta Robots Tag?	apranajoshi	Search Engine Optimization	8	08-30-2016 01:26 AM
Which is better Robots.txt or Meta Robots Tag?	riyakhan	Search Engine Optimization	3	12-14-2015 01:01 AM

03-06-2018, 11:06 PM	#2
smmexpertss Registered User Join Date: Sep 2015 Location: UK Posts: 447	It's a pretty common misconception that the robots.txt will prevent indexing. It's only purpose is actually to prevent crawling, anything disallowed in there is still up for indexing if it's linked to elsewhere. If you want something deindexed, your best bet is the robots meta tag, but make sure you allow crawling of the URLs to give search engine bots an opportunity to see the tag.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)