|
The main need of robot.txt.file is to dictate how spiders or robots will crawl your website If on a website you do not use robot.txt file then search engine will crawl all pages on the website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|