Google question
I worked on a robots.txt for a customer to disallow certain areas of the site from google crawls and indexing... well, not just google but all bots... However, google is ignoring the robots.txt and doing its own thing anyway. Anybody else have this problem?
Google's been using up about 10GB/month on this particular site. Slurp was the worst culprit though topping out about 13GB. The site is heavily trafficked and the bandwidth is consistently going through the roof so any little bit of reduction would help them.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|