Site Owners Forums - Webmaster Forums - View Single Post

yasmineassif · 06-02-2016, 05:33 AM

Website owners use the /robots.txt file to give instructions about their website to robots. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

06-02-2016, 05:33 AM	#5
yasmineassif Registered User Join Date: Jun 2016 Location: lebanon Posts: 8	Website owners use the /robots.txt file to give instructions about their website to robots. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.