What is Robots.txt
You can also inform search engines optimization about the files and folders that are on your website to evade the use of the robots metatag. As all search engines do not read metatag, the robots metatag can just went ignored. So a better means to inform search engines optimization about your will is to use a robots.txt file.
Robots.txt is a text file which you use to put on your website to inform search robots about your pages that which pages you would like not to open to visit. It is significant that you explain that robots.txt is not method from stopping search engines to crawling your website and the truth that you place a robots.txt file is a bit similar to putting a message just similar to “Please, do not enter without permission ” on an opened door. From this message it clear that a good civilized person never enter without knocking but it doesn’t you search engines optimization can stop thieves from coming in to an open door. So as we suggest you that if you have any sensitive search engines optimization data, it is too inexperienced to faith on robots.txt to guard it from being indexed and put on view in search results.
The place of robots.txt is very significant. It has to be in the major directory since or else user search engines optimization will not be capable to discover it – they don’t search the whole website for a particular file named robots.txt. search engines optimization In its place, they glance first in the main directory and if they do not able to find out it there, they just think that this website does not have a robots.txt file and so they index all they discover along the way.