When I first learned about the robots.txt file, I didn’t fully understand the reason behind it.  Why wouldn’t I want the search engine robots to index all the pages on my site?  What is the purpose of the robots.txt file anyways?

A robots.txt file is used to restrict access to your site by search engine robots.  Before a bot can access your site, they look for a robots.txt file that prevents them from accessing certain pages.  Keep in mind that some spammers may ignore the robots.txt file.  To avoid this, you should use a password to protect confidential information.

If you want search engines to index everything on your site, then there is really no purpose of a robots.txt file.  You only need the robots.txt file if you have content that you do not want indexed.

If you want to add a robots.txt file, you will need access to the root of your domain.  If this is not possible, you can also restrict access using the robots meta tag.

Here is an example of a robots.txt file that applies to all bots:

User-agent: *
Disallow: /folder1/

The asterisk next to the User-agent is saying that the robots.txt file should be applied to all bots.

If you would like to to specify a specific bot, you can do the following:

User-Agent: Googlebot

To block certain pages, you can do this by using the Disallow line.
You will be able to list a pattern or URL.
The entry should begin with a forward slash.  Here are some examples.
  • To block the entire site, use a forward slash. 
    Disallow: /
  • To block a directory and everything in it, follow the directory name with a forward slash. 
    Disallow: /junk-directory/
  • To block a page, list the page. 
    Disallow: /private_file.html
  • To remove a specific image from Google Images, add the following: 
    User-agent: Googlebot-Image
Disallow: /images/dogs.jpg
  • To remove all images on your site from Google Images: 
    User-agent: Googlebot-Image
Disallow: /
  • To block files of a specific file type (for example, .gif), use the following: 
    User-agent: Googlebot
Disallow: /*.gif$
Adam Jackson
Latest posts by Adam Jackson (see all)