Robots Txt Generator is an convenient-to-use tool to create correct Robots.Txt directives on your site with no trouble copy and tweak Robots.Txt documents from other web sites.
Now, as a search engine spiders crawl a internet site, they start crawling by means of analyzing a robots.Txt file residing on the root domain.