Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization > Yahoo


Thread Tools Rate Thread Display Modes
Old 01-08-2012, 01:06 AM   #1
Registered User
Join Date: Jan 2012
Posts: 17
How to slow the crawl rate of yahoo?

Hello members,

How do I slow the crawl rate of yahoo?
its killing my bandwidth 700 bots a day and putting extra strain on my server.

Please tell me How to do this?
billmat is offline   Reply With Quote

Old 02-05-2017, 01:52 AM   #2
Avinash kumar
Registered User
Join Date: Feb 2017
Posts: 167
Slurp is the Yahoo search bot for the crawl and index your web page.
It obeys the Robot Exclusion standard. If you don not want to crawl frequently your web page, create a robots.txt file in the root directory (Home folder of site and add a rule for "User-agent: Slurp"

Code example in a robot.txt file:

User-agent: Slurp<br>disallow:/cgi-bin/
Avinash kumar is offline   Reply With Quote

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Yahoo Questions and Answers jaxondavis37 Yahoo 11 05-22-2017 09:16 PM
Yahoo optimization keywords by relevance loganallen48 Yahoo 2 10-09-2014 11:57 PM
New mobile service by Yahoo! gavindmello053 Yahoo 3 01-02-2012 09:14 PM
How can earn money through Yahoo! haroldmacmillan Yahoo 3 12-08-2011 04:46 PM
Optimizing Yahoo andysmith476 Yahoo 0 11-16-2011 01:04 AM

All times are GMT -7. The time now is 02:30 PM.

Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.