|
|
04-05-2019, 12:03 AM
|
#1 |
|
Registered User
Join Date: Feb 2018
Location: Bangalore
Posts: 283
|
Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
Basic format of the robot.txt file. User-agent: [user-agent name] Disallow: [URL string not to be crawled]
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Hybrid Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| robot.txt | Shivam Tripathi | Web Promotion | 5 | 12-16-2018 10:11 PM |
| Brandon Brice is top rated rated Social Media Blogger | deepakrajput | Post your ad here | 0 | 04-24-2015 04:48 AM |
| how to eligible for adsense in blogger | ifte111 | Making money on the web | 2 | 11-01-2014 08:35 AM |
| Per post keywords in Blogger | Shamatae | Search Engine Optimization | 3 | 10-10-2014 01:42 AM |
| Blogger Sitemap through Google Webmaster Tools | Shamatae | 1 | 12-06-2013 01:00 AM | |
All times are GMT -7. The time now is 03:41 PM.