How to slow the crawl rate of yahoo?
Hello members,
How do I slow the crawl rate of yahoo? its killing my bandwidth 700 bots a day and putting extra strain on my server. Please tell me How to do this? |
Slurp is the Yahoo search bot for the crawl and index your web page.
It obeys the Robot Exclusion standard. If you don not want to crawl frequently your web page, create a robots.txt file in the root directory (Home folder of site and add a rule for "User-agent: Slurp" Code example in a robot.txt file: User-agent: Slurp disallow:/cgi-bin/ |
All times are GMT -7. The time now is 09:12 PM. |
Powered by vBulletin Copyright © 2020 vBulletin Solutions, Inc.