What is robots.txt   The robots.txt file is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl.   Finding your robots.txt file   If you just want a quick … Continue reading How to Create a Robots.txt File for SEO