Now don’t freak out! We’re not looking out for the Martian Tripods from H.G. Wells’ masterpiece War of the Worlds or a race of robots disguised as a line of GM vehicles defending Earth from even more robots disguised as vehicles. What we’re looking out for is your website’s robots.txt file.
A robots.txt file is there to direct the search engine bots that routinely crawl your site, telling them where they can and can’t go. That way, you can keep them from indexing
certain pages for whatever reason you may have. It’s like being in on-campus housing in college, when the RA comes in for cleaning checks. Only with a robots.txt file, you can tell your RA not to look in your closet and to completely ignore the bathroom altogether. Nobody likes to clean that.
Why Have One?
There are a few reasons to keep Google or other search engines from indexing certain pages.
- You may have a legitimate case for duplicate content, but not want to be penalized by Google.
- Your site may have areas that are under construction (like that messy bathroom you’re hiding from the RA during cleaning checks).
- You may want to keep your images out of Google’s image directory or keep your company directory from appearing in a web search.
If you’re looking to create your own robots.txt file but don’t know where to start, check out SEOBook’s user-friendly Robots.txt File Generator.
What to Avoid
Keep in mind that a robots.txt file is not meant to be a security measure. As Brett explained in an earlier post, keep your confidential information password protected. The robots.txt file exists to show off the best parts of your website as the search engine bots crawl your site. His post also contains great in-depth instructions on setting up your own robots.txt file.
Also, be sure to format your robots.txt file correctly, as you may accidently keep search engines from indexing your site!
For more information on increasing your site’s ranking, get in touch with your friendly neighborhood SEO team at Leadgenix.