Now don’t freak out! We’re not looking out for the Martian Tripods from H.G. Wells’ masterpiece War of the Worlds or a race of robots disguised as a line of GM vehicles defending Earth from even more robots disguised as vehicles. What we’re looking out for is your website’s robots.txt file.
A robots.txt file is there to direct the search engine bots that routinely crawl your site, telling them where they can and can’t go. That way, you can keep them from indexing