It has become common practice to implement robot.txt files to prevent the search engine spiders from crawling your webpage and having it show up in the search engine results page (SERP). Unfortunately, robots.txt does not prevent your site from showing up in the search engine results. That means your web page is passing on link juice that is unused and wasted. It does prevent your site from being indexed, but it could show a result in the SERP’s. It will not have a description of your site, only a URL and strange title.
Matt Cutt’s from Google explains this in his video
A better way to prevent a web site from being crawled and indexed in the search engines is to use a robots meta tag a X robots-tag http header .htaccess file to your website.
Meta Tag
Adding the met tag < name=”robots” content=”noindex,nofollow”> lets the search engine’s spider know not to index or follow your website and show it in the search results. This is a useful snippet and can be very beneficial. The downside to this is that you have to hand code this into every page on your web site. This can be impossible for larger sites with tens of thousands of pages.
X robots-tag http header .htaccess
The new tag allows you to create a HTTP header called X-Robots-Tag and set the value as you would the meta robots tag value. This will set up the file for the entire site making your life much easier. So, if your site is running on Apache, and mod_headers is enabled (it usually is), you could add the following single line to your .htaccess file: Header set X-Robots-Tag “noindex, nofollow”
Related posts:
About bosmol
Listen to blogs 
Undoubtfully interesting story u have here. I’d like to read something more about this matter. Thanx for sharing that material.
It’s posts like this that keep me coming back and checking this site regularly, thanks for the info!
Nice post and this mail helped me alot in my college assignement. Thanks you as your information.
PageRank is an algorithim google invented that assesses the strength of a website. The algorithim evaluates a site on a scale of n/a to 10 with 0 being the lowest/weakest and 10 being the highest/strongest. We can check our site’s rank by using a checker pagerank tool.
Sonia recently posted..Pagerank Checker
PageRank is an algorithim google invented that assesses the strength of a website. The algorithim evaluates a site on a scale of n/a to 10 with 0 being the lowest/weakest and 10 being the highest/strongest. We can check our site’s rank by using a pagerank checker tool.
Miss Sonia recently posted..Privacy Policy Generator
this is really cool, I like here!
Robots meta tags is followed worldwide and even search engines pay importance to this tag only.
Durable Goods recently posted..Production Capacity and Competition For Durable Goods
Can you give an example of an X robots-tag? This code is added to htaccess?
Thanks
Steve´s last blog ..Free Printable Grocery Coupon Codes