View Single Post
Old 02-05-2017, 02:52 AM   #2
Avinash kumar
Registered User
 
Join Date: Feb 2017
Posts: 167
Slurp is the Yahoo search bot for the crawl and index your web page.
It obeys the Robot Exclusion standard. If you don not want to crawl frequently your web page, create a robots.txt file in the root directory (Home folder of site and add a rule for "User-agent: Slurp"

Code example in a robot.txt file:

User-agent: Slurp<br>disallow:/cgi-bin/
Avinash kumar is offline   Reply With Quote