Thread: what is a robots.txt file?
View Single Post
Old 10-10-2019, 10:25 PM   #9
leenahahim
Registered User
 
Join Date: Feb 2016
Posts: 454
It helps to crawl all web crawler from all web pages or specif web crawler from specif web pages.
leenahahim is offline   Reply With Quote