Robot.txt Validation

The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages. For those search engine spiders that obey the file, it provides a map for what they can, and cannot index.

Optimization of Robots.txt

Validation and optimization of your websites robots.txt file is extremely important. Robots.txt is mainly used to exclude robots from files or directories and if not setup correctly, you could be eliminating every search engine from spidering and indexing your website.

If you have no robots.txt file and you monitor your server error log, you will find many errors related to the missing robots.txt file. These errors can and should be eliminated - making the error log much more useful for finding real errors.

« View the SEO Packages for more information.