Little Information

[funeral]

All of us are aware with the terms spiders, crawlers and bots (robots), however very few people know the difference among the three.

Here is the short description the reveals the difference:

Spider - a browser like program that downloads web pages.

Crawler � a program that automatically follows all of the links on each web page.

Robots - An automated computer program that visit websites & perform predefined task. They are guided by search engine algorithms & are able to perform different tasks instead of just one crawling task. They can combine the tasks of crawler & spider together and help in indexing and ranking of websites on any particular search engine.