|
Crawlers are known for URL discovery tools. You just give them a webpage to start off from and they will follow all the links which they can find on that particular page. If the links they follow lead them to a page they haven’t been to before, they will follow all the links on that page as well and the process will goes on, so on, in a loop. If you repeat this process enough then eventually you will wind up with a list of all the possible URLs, usually restricted to a given domain (eg. Asos.com). The advantage about crawlers is that they try to visit every page on a website, so they are very complete but the disadvantage about crawlers is that they try to visit every page on a website, so it takes a long time.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|