11-03-2014, 06:17 AM
|
#1 |
|
Registered User
Join Date: Mar 2013
Posts: 43
|
How to prevent particular pages from crawler?
Hi,
Can anyone please share the code which helps to prevent the crawling process for specific pages? Thanks, Mollie
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Best clone script of To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. to start a design contest website |
|
|
|
11-12-2014, 02:38 AM
|
#2 |
|
Registered User
Join Date: Nov 2014
Posts: 11
|
Add this code to the webpage header..before </head>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
11-13-2014, 10:47 AM
|
#3 |
|
Registered User
Join Date: Oct 2014
Posts: 32
|
There are many ways to prevent pages from getting crawled by bots by using some codes on the website. For example: one is mentioned above, other we can use robots.txt file, we can server level meta tags which can tell bots not to crawl or index any page etc..
|
|
|
|
|
02-22-2017, 10:54 PM
|
#4 |
|
Registered User
Join Date: Apr 2016
Posts: 542
|
Use robot.txt file for that
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-03-2017, 03:21 AM
|
#5 |
|
Registered User
Join Date: Sep 2016
Posts: 63
|
Use robot.txt file to inform crawler
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-05-2017, 10:36 PM
|
#6 |
|
Registered User
Join Date: Dec 2016
Posts: 377
|
By using Robots.txt file one can prevent pages from crawler.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-06-2017, 10:45 PM
|
#7 |
|
Registered User
Join Date: Jan 2016
Location: Mumbai, India
Posts: 1,064
|
Use robot.txt to prevent your page from crawler
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Google Crawling and Indexation 101 | sinicline | Search Engine Optimization | 20 | 02-05-2016 07:47 PM |
| Can web crawler find unlinked pages of website? | jamesranatte | Review My Website | 1 | 01-02-2014 11:55 PM |
| How to Optimize Dynamic Website or Pages? | kajal.pillai22 | Search Engine Optimization | 11 | 04-20-2013 03:33 AM |
| Web Crawler | rajivwebads | Search Engine Optimization | 12 | 02-23-2013 11:59 AM |
| Unlinked pages | wryfhk22 | 0 | 09-13-2011 01:36 AM | |
All times are GMT -7. The time now is 08:25 AM.