crawling error? - Site Owners Forums

northerncarspar · 11-04-2015, 09:08 PM

my site is still not crawled? why? when i was developing my eCommerce magento site at that time i uploaded robot.txt in public_html to stop my site to crawl.. but now my site development is completed and i deleted those robot.txt file from public_html and add site in webmaster tool as well submit site map.. but it's not crawling my site... sitemap showing error in indexing... it's showing you submitted robot.txt but as i said that i deleted robot.txt file in public_html.. so please guide me... I feel nervous... what should i do?

smmexpertss · 11-06-2015, 06:08 AM

There’s many reasons you may suffer from 404 page not found crawl errors however the most important thing is you fix them.

If you have too many 404 errors your site visitors will suffer a poor experience and the search engines may penalize you as well.

There’s many ways to check for broken links, some of which i have written about previously.

Rather than create a temporary fix, its better to fix them permanently and Google Webmaster Tools provides one of the best solutions.

You’ll need to have created an account with Google Webmaster Tools before we can proceed.

11-04-2015, 09:08 PM	#1
northerncarspar Registered User Join Date: Oct 2015 Posts: 31	crawling error? my site is still not crawled? why? when i was developing my eCommerce magento site at that time i uploaded robot.txt in public_html to stop my site to crawl.. but now my site development is completed and i deleted those robot.txt file from public_html and add site in webmaster tool as well submit site map.. but it's not crawling my site... sitemap showing error in indexing... it's showing you submitted robot.txt but as i said that i deleted robot.txt file in public_html.. so please guide me... I feel nervous... what should i do?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
What is meant by static 404 error and dynamic 404 error?	regainhairsolut	Search Engine Optimization	0	02-13-2015 05:34 AM
How to fix this warning error from my google web master?	sahithya	Search Engine Optimization	3	12-09-2014 03:14 AM
Regarding Google Crawling	James12smith	Search Engine Optimization	1	12-31-2013 02:43 AM
Gone from Bing; Could it be a System Error?	bobo.cx	Bing	1	06-20-2012 09:51 PM
How to set up of custom html default error pages	jennypretty	Search Engine Optimization	1	01-03-2012 09:21 AM

11-06-2015, 06:08 AM	#2
smmexpertss Registered User Join Date: Sep 2015 Location: UK Posts: 447	There’s many reasons you may suffer from 404 page not found crawl errors however the most important thing is you fix them. If you have too many 404 errors your site visitors will suffer a poor experience and the search engines may penalize you as well. There’s many ways to check for broken links, some of which i have written about previously. Rather than create a temporary fix, its better to fix them permanently and Google Webmaster Tools provides one of the best solutions. You’ll need to have created an account with Google Webmaster Tools before we can proceed.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)