Thread: What is robots.txt?
View Single Post
Old 03-20-2017, 03:30 AM   #10
emilyoliver
Registered User
 
Join Date: Sep 2016
Posts: 248
The Robots.txt is the file used to restrict the crawler from crawl the secured files.
emilyoliver is offline   Reply With Quote