What is robots.txt?
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:

http://www.yourwebsite.com/robots.txt

Examples:
This example allows all robots to visit all files because the wildcard "*" specifies all robots:
User-agent: *
Disallow:


This example keeps all robots out:
User-agent: *
Disallow: /

The next is an example that tells all crawlers not to enter four directories of a website:
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/


>> Read more about Robots.txt Search Engine