05-29-2012, 02:08 AM | #1 |
Registered User
Join Date: Mar 2012
Posts: 87
|
Google question
I worked on a robots.txt for a customer to disallow certain areas of the site from google crawls and indexing... well, not just google but all bots... However, google is ignoring the robots.txt and doing its own thing anyway. Anybody else have this problem?
Google's been using up about 10GB/month on this particular site. Slurp was the worst culprit though topping out about 13GB. The site is heavily trafficked and the bandwidth is consistently going through the roof so any little bit of reduction would help them.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Display Modes | Rate This Thread |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
OPTIMIZE YOUR SITE SUCCESSFULLY FOR Google. | bobo.cx | 25 | 05-10-2017 05:45 AM | |
Google Alerts Question | wryfhk22 | Search Engine Optimization | 1 | 10-02-2014 03:11 AM |
Google notice of unnatural links detected 101 | Mark Jackson | 2 | 11-04-2012 11:01 PM | |
Google lead the search for the socialization of the future development of social sear | Chinese holiday | 0 | 02-12-2012 11:38 PM | |
Question concerning Google and redirects | mjvndhsb | 0 | 10-10-2011 12:15 AM |