Which is better Robots.txt or Meta Robots Tag? - Site Owners Forums

Sunshineskin · 03-06-2018, 09:24 PM

Which is better Robots.txt or Meta Robots Tag?

smmexpertss · 03-06-2018, 11:06 PM

It's a pretty common misconception that the robots.txt will prevent indexing. It's only purpose is actually to prevent crawling, anything disallowed in there is still up for indexing if it's linked to elsewhere. If you want something deindexed, your best bet is the robots meta tag, but make sure you allow crawling of the URLs to give search engine bots an opportunity to see the tag.

matildaeric · 03-06-2018, 11:54 PM

Both are supported by all crawlers which respect webmasters wishes not all do but against them neither technique is sufficient you can use robots.txt rules for general things like disallow whole sections of your.

buyinstagram · 03-07-2018, 08:36 AM

Msnbot, discobot, and Slurp are all called out specifically, so those user-agents will only pay attention to the directives in their sections of the robots.txt file. All other user-agents will follow the directives in the user-agent: * group.

nathanleo · 12-30-2021, 05:54 AM

Robots. txt files are best for disallowing a whole section of a site, such as a category whereas a meta tag is more efficient at disallowing single files and pages. You could choose to use both a meta robots tag and a robots.

03-06-2018, 09:24 PM	#1
Sunshineskin Registered User Join Date: Feb 2018 Location: D 47, Bali Nagar, (Next to Raja Garden community centre) Delhi, 110015, India Posts: 10	Which is better Robots.txt or Meta Robots Tag? Which is better Robots.txt or Meta Robots Tag? __________________ Skin Spe******t in Delhi Hair Transplant in Delhi Best Skin Doctor in Delhi

03-06-2018, 11:54 PM	#3
matildaeric Registered User Join Date: Jan 2018 Posts: 93	Both are supported by all crawlers which respect webmasters wishes not all do but against them neither technique is sufficient you can use robots.txt rules for general things like disallow whole sections of your. __________________ Natural Handmade Soap \| Handmade Soap \| *Candle Range* \| *Luxury Scented Candles* \| *Sports injuries*

03-07-2018, 08:36 AM	#4
buyinstagram Registered User Join Date: Nov 2017 Posts: 253	Msnbot, discobot, and Slurp are all called out specifically, so those user-agents will only pay attention to the directives in their sections of the robots.txt file. All other user-agents will follow the directives in the user-agent: * group. __________________ BUY INSTAGRAM FOLLOWERS UK / BUY TWITTER FOLLOWERS UK / BUY INSTAGRAM FOLLOWERS / BUY FACEBOOK LIKES UK / BUY YOUTUBE SUBSCRIBERS

12-30-2021, 05:54 AM	#5
nathanleo Registered User Join Date: Jul 2017 Posts: 2,283	Robots. txt files are best for disallowing a whole section of a site, such as a category whereas a meta tag is more efficient at disallowing single files and pages. You could choose to use both a meta robots tag and a robots. __________________ Villa Plots in Chennai Plots Sale in Kelambakkam Premium Villa Plots in Kelambakkam fastest selling villa plots in chennai Villa Plots in Kelambakkam OMR Residential Plots for Sale in OMR Premium Villa Plots in OMR Villa Plots in Kelambakkam Villa Plots Sale in Kelambakkam Plots Sale in Kelambakkam Residential Plots in Kelambakkam Land for Sale Near Chennai Plots for Sale in Chennai Premium Villa Plots OMR luxury villa plots in ECR Chennai

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Which is better Robots.txt or Meta Robots Tag	doorhardwareuk	Search Engine Optimization	0	07-10-2017 11:17 PM
Which is better Robots.txt or Meta Robots Tag?	mithucarrentals	Search Engine Optimization	0	01-30-2017 04:04 AM
Which is better Robots.txt or Meta Robots Tag?	apranajoshi	Search Engine Optimization	8	08-30-2016 01:26 AM
Which is better Robots.txt or Meta Robots Tag?	riyakhan	Search Engine Optimization	3	12-14-2015 01:01 AM

03-06-2018, 11:06 PM	#2
smmexpertss Registered User Join Date: Sep 2015 Location: UK Posts: 447	It's a pretty common misconception that the robots.txt will prevent indexing. It's only purpose is actually to prevent crawling, anything disallowed in there is still up for indexing if it's linked to elsewhere. If you want something deindexed, your best bet is the robots meta tag, but make sure you allow crawling of the URLs to give search engine bots an opportunity to see the tag.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)