Site Owners Forums - Webmaster Forums

Site Owners Forums - Webmaster Forums (http://siteownersforums.com/index.php)
-   Yahoo (http://siteownersforums.com/forumdisplay.php?f=46)
-   -   How to slow the crawl rate of yahoo? (http://siteownersforums.com/showthread.php?t=53826)

billmat 01-08-2012 02:06 AM

How to slow the crawl rate of yahoo?
 
Hello members,

How do I slow the crawl rate of yahoo?
its killing my bandwidth 700 bots a day and putting extra strain on my server.

Please tell me How to do this?

Avinash kumar 02-05-2017 02:52 AM

Slurp is the Yahoo search bot for the crawl and index your web page.
It obeys the Robot Exclusion standard. If you don not want to crawl frequently your web page, create a robots.txt file in the root directory (Home folder of site and add a rule for "User-agent: Slurp"

Code example in a robot.txt file:

User-agent: Slurp
disallow:/cgi-bin/


All times are GMT -7. The time now is 05:16 AM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.