Thread
:
What is robots.txt?
View Single Post
03-20-2017, 03:30 AM
#
10
emilyoliver
Registered User
Join Date: Sep 2016
Posts: 248
The Robots.txt is the file used to restrict the crawler from crawl the secured files.
emilyoliver
View Public Profile
Find More Posts by emilyoliver