Syntax of Robots meta tag

<meta name=”robots” content=”…, …” />
<meta name=”googlebot” content=”…, …” />

Robots meta tag can control the behavior of search engine crawling and indexing.

If you want to control only Google Bot they use the syntax <meta name=”googlebot” content=”….,….”>, and the meta name robots will be universal, it will work for all the search engine bot.

You can put several parameters as content.

Example of Single parameter

<meta name=”robots” content=”noindex” />
<meta name=”googlebot” content=”nofoll0w” />

First one will tell all the search bot that, no to index your page. That means that page will not come in the Search Engine page. But search bot may go through all the available links of that page. Say you have a page namely about.html with “noindex”, the page also has several links like “contact.html, product.html”. if you use <meta name=”robots” content=”noindex” />, it will not show the about.html page in the search engine page but the search engine may index the other pages like contact.html, product.html.  Remember, while you use “robots” it will work for all the search engines.

• noindex: prevents the page from being indexed in search result
• nofollow: prevents the bot from following links from this page
• nosnippet: prevents a snippet from being shown in the search results
• noodp: prevents the alternative description from the ODP/DMOZ from being used
• noarchive: prevents Google from showing the Cached link for a page.
• unavailable_after:[date]: lets you specify the exact time and date you want to stop crawling and indexing of this page
• noimageindex: lets you specify that you do not want your page to appear as the referring page for an image that appears in Google search results.

As a side-note, you can now also specify this information in the header of your pages using the “X-Robots-Tag” HTTP header directive. This is particularly useful if you wish to fine-tune crawling and indexing of non-HTML files like graphics or other kinds of documents.

– ref Google