Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > General Arena > Social Networks

Notices


Reply
 
Thread Tools Rating: Thread Rating: 5 votes, 2.00 average. Display Modes
Old 07-06-2018, 10:50 PM   #1
riprook7
Registered User
 
Join Date: Nov 2017
Location: Oregon
Posts: 49
Robots.txt

What is Robots.txt file?
riprook7 is offline   Reply With Quote

Old 07-10-2018, 03:23 AM   #2
wellliving
Registered User
 
Join Date: Dec 2015
Location: USA
Posts: 1,168
Hi

The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

https://www.welllivingshop.com/pillows/organic-cotton/
wellliving is offline   Reply With Quote
Old 07-11-2018, 03:05 AM   #3
david1618
Registered User
 
Join Date: Jul 2015
Location: Melbourne, Australia
Posts: 1,232
robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.
david1618 is offline   Reply With Quote
Old 07-11-2018, 03:18 AM   #4
Halfmealf
Registered User
 
Join Date: Jul 2018
Posts: 5,282
byrtbncnjs

jhpvkbxtat



uukycpjzhi

http://gilrs-prostitutki.ru/forum/vi...219636#p219636
http://blackhatcoins.com/showthread....d=1#post168723
http://www.infintytrading.com/vb/sho...d=1#post231929
Halfmealf is online now   Reply With Quote
Old 07-11-2018, 04:24 AM   #11
Briguegoro
Registered User
 
Join Date: Jul 2018
Posts: 5,265
mlfjktcslw

scarrfxvge



siibecchrw

http://www.xmr.dk/viewtopic.php?f=3&...269455#p269455
http://thietbiso.shop/showthread.php...ed=1#post23852
http://sotall.org/communities/viewto...p?f=22&t=27290
Briguegoro is offline   Reply With Quote
Old 07-11-2018, 04:31 AM   #12
Halfmealf
Registered User
 
Join Date: Jul 2018
Posts: 5,282
arcrdqlefz

jmaiwykqlb



qmtaoewwma

http://www.flixbus.scot/viewtopic.php?f=9&t=114036
http://xn--stragraniczna-64c.y0.pl/s...47504#pid47504
http://www.yelingboy.com/forum.php?m...=145156&extra=
Halfmealf is online now   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Robots.txt or Meta Robots.txt??? santoshkumar Search Engine Optimization 11 02-14-2017 03:03 AM
Robots.txt jaysh4922 Search Engine Optimization 15 07-29-2016 05:07 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt Fking General Discussion 1 12-13-2013 10:27 PM
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 12:36 PM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 02:19 AM


All times are GMT -7. The time now is 05:20 PM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.