It has become common practice to implement robot.txt files to prevent the search engine spiders from crawling your webpage and having it show up in the search engine results page (SERP). Unfortunately, robots.txt does not prevent your site from showing up in the search engine results. That means your web page is passing on link juice that is unused and wasted. It does prevent your site from being indexed, but it could show a result in the SERP’s. It will not have a description of your site, only a URL and strange title.

Matt Cutt’s from Google explains this in his video

A better way to prevent a web site from being crawled and indexed in the search engines is to use a robots meta tag a X robots-tag http header .htaccess file to your website.

Meta Tag
Adding the met tag < name=”robots” content=”noindex,nofollow”> lets the search engine’s spider know not to index or follow your website and show it in the search results. This is a useful snippet and can be very beneficial. The downside to this is that you have to hand code this into every page on your web site. This can be impossible for larger sites with tens of thousands of pages.

X robots-tag http header .htaccess
The new tag allows you to create a HTTP header called X-Robots-Tag and set the value as you would the meta robots tag value. This will set up the file for the entire site making your life much easier. So, if your site is running on Apache, and mod_headers is enabled (it usually is), you could add the following single line to your .htaccess file: Header set X-Robots-Tag “noindex, nofollow”

Alt Tags (alt=”")
Anchor tags are important for receiving link juice and build your search engine rankings. Without the proper anchor tags; your links will not be as valuable as they can be. Proper keyword research and strategically placing these keywords in the link text is a valuable SEO technique.

There are options for using tags to help the search engines and the disabled comprehend what a links is about and where it is pointing to.

Alt tags are used to explain that an image is to the disabled. The alt actually stands for alternative text and is used for accessibility issues. People who cannot view images on their computer need a way to understand what the image is. The alt text appears instead of the image on a website, so it is good to put descriptive words here not just keyword stuff. The alt tags are part of the W3C requirements for friendly websites. There is some SEO value as well because the search engines spider read the links anchor text and the alt tags, both of which describe the link. Remember keyword stuffing is SPAM and should never be used. alt=”"

Title Tags (title”")
Title Tags are still one of the most important tags in terms of search engine optimization (SEO) and marketing your company to potential searchers. Your websites title tag is what appear in the search engine results when you search for specific keywords. Your title and description shown on the search engine results are actually your title and description tags from your website. Make sure you properly use this tag. You should keep the title tag under 70 characters and the description tag less than 150 characters. Anything more than this is likely to get cut off by the search engines ad looks SPAMMY. This means making it both search engine friendly and human friendly. You want to have keywords targeting your customers, but you also want it to sound smooth and natural when read by humans. This is a difficult balance to achieve with the limited space provided.

Some things to avoid with the title tag are: leaving it bank, using non descriptive words, putting “Welcome to my Website”, and keyword stuffing. Doing this can result in your website being banned from the search engines. Even worse is that when your site does appear in the search engine results it will be inappropriate and useless. You will have a high bounce rate.

In essence using the proper tags is important for website accessibility, search ability, and ultimately usability. Making sure you have done the proper keyword research and implementing the search engine techniques is vital to your websites success.