Welcome again to the SEO tutorial, and today we will see all about robots text.  The robot exclusion standard is known as robots.txt, you can clearly see the extension is ‘.txt’ and that is why you should not confuse it with html or other extensions. The robot exclusion standard has been introduced in 1994, the robots text will simply guide the bots or robots “how to crawl the pages and index those pages on their website”. Let’s see what we will learn from the robots text discussion:

  • What is Robots text?
  • How to use Robots text?
  • Cons of robots text

If you are familiar about the above points, then you can read:

  • How to create robots.txt
  • Tools to check robots.txt

Learn robots text

What is Robots text?

The robots.txt is like a teacher, teachers normally guide or instruct the student, and parents all about studies and how to improve the studies. Same way robots.txt will instruct the search engines, like how to crawl, which pages, sub-directories and images to leave without crawling. In short, we can define it as:

“A text file that will equip webmasters to control the crawling of the search engines.”

Note: The best place for robots.txt file is a root directory of the website.

How to use Robots text

Search engines are like a student who will obey their elders, teachers and mentors. If your site has the robots.txt file, then the search engine spider will first look for the instructions to find out the restriction, and where they are allowed to crawl.

Uses of robots.txt files are:

  • You can tell where about of your sitemap.
  • You can restrict whole or part of a website from indexing.
  • You can stop Google from indexing files as well.
  • Guide spiders to leave duplicate contents as well.

Note:  Some bad web spider will not follow your robots.txt instruction, because they don’t want to.

Cons of Robot text

  • Hackers will use filenames or directories to find the loophole, because of mishandling of robots.txt. So better understand the robots text, and then allow or not allow bots.
  • All robots are not bound to follow your robots.txt, miscreants do not follow law and same apply to these bad robots.


We have just covered robots.txt, and it’s importance as well. The robots text will give you a way to control the spider crawling or restrict few spiders from crawling as well. In case if you don’t have a robots.txt, then bots (spiders) will crawl at their wish on your website.  Even if, they will find duplicate content on your website, they will index it as well.

Note: If you don’t want robots.txt or don’t know how to create robots.txt, at least put your sitemap in the reach of search engines. By doing this, you will let search engines know about your new content or updated content.

What Next? Next we will see the robots.txt checker

If you liked our content, then share it with your friends, like it or bookmark it for your future references. Any query or suggestion, then reach to us and we will assist with your query.