Should you panic for new Google warnings over media?

[Ruby707]

Hey everyone, Google is warning everyone about Googlebots not being able to access CSS and JS files. I thought this might be a good idea to share with you guys and hear your side of the story.
Couple of days ago we saw a flood of warnings from Google for some of our websites, although these warning were not penalizing any of the websites but we decided to take action and fix the issues.

It all started with some messages in Google Webmaster tools, showing that Googlebot (Google little spiders) cannot access your JavaScript (JS) and/or CSS files. Well previously there was no errors and suddenly Google cannot access the files. The new algorithm Google has deployed, needs access to some deeper levels of a website to be able to index the website even better.

Most developers when they set up robots.txt (a text file to show search engines where they can and cannot see), block (disallow) templates, image galleries and other types of media. But Google does not like that anymore, and wants to dig deeper into the website. How to solve this issue though is not a difficult process.

To allow Googlebot to crawl and see those parts of your website, there are some workarounds but the easiest is to allow Googlebot to index images, media, templates and modules through your robots.txt file. Simply take out “disallow” function from those areas and fetch with Google to test the new robot.txt.

There are some points to consider while fixing this. Do not totally remove or delete your robots.txt, you need a robot.txt file for other bots. The only spiders on internet are not for search engines, and although you cannot stop misuse with robot.txt but it can help prevent some bots accessing specific parts of your website. That being said, robots.txt is not an obligatory command for bots and it is a suggestion, so there are pages disallowed by robots.txt but still being indexed. Therefore do not use this method to hide or secure a certain file or part of your website. If you type in "yourwebsite.co.nz/robots.txt" you will be able to see your robots.txt, and if you are hiding anything, it would show everyone what that would be.
Mobile Application Development Firm | Game Development USA | Mobile Application Developer | Game Developer USA