What is robots.txt?

Robots.txt is a standard file that should be installed at the root of your website. It tells search engine crawlers what to index in your site (you might have folders with protected content in, that you don’t want appearing in Google). 

You can verify it exists and whether it’s working at Google Webmaster tools. 

May 25, 2008 | Filed Under Google, Organic Listings, SEO, Software 

Comments

Leave a Reply




*
To prove you're a person (not a spam script), type the security word shown in the picture. Click on the picture to hear an audio file of the word.
Click to hear an audio file of the anti-spam word