|
1. Identify which directories and files on your web server you want to block from being rawled by the crawler
2. Identify whether or not you need to specify additional instructions for a particular search engine bot beyond a generic set of crawling directives
3. Use a text editor to create the robots.txt file and directives to block content
4. Optional: Add a reference to your sitemap file (if you have one)
5. Check for errors by validating your robots.txt file
6. Upload the robots.txt file to the root directory of your site
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|