How Robot Text Files Can Reveal More Than You Want
A robot text file is used to disallow specific or all search engine spider’s access to folders or pages that you do not want indexed.
You may have created a directory or web page for employees or a paid members area that you don’t want listed. The case may be you simply created a private page just for your own personal use. Some webmasters use it to exclude their guest book pages in order to avoid spam and phishing. Whatever the case may be, there are many different reasons why a robots text file can be helpful, however there are other setbacks that may come to play.
Every time I review my web stats for my 404 (Document Not Found), the robot txt. file is the one file that is searched and not found the most often.
Now let’s look at the robot text file in a different light.
Say you are working on a new web script or program, and if your like me, you probably will change the admin directory or other private directory to new location. Now you go ahead and tell the robot txt. file to ‘no-follow’ that specific directory.
The problem lies when the robot file is accessed, it is found by its name ‘robots.txt’. Now lets say a spammer, pirate or even a hacker tries to use that same file against you by simply calling the robot text file in their browser. They will actually see the area you are attempting to keep from prying eyes.
It is always a good idea to always password protect directories you want kept private as a precaution. This should also be applied to directories that already have a login in place. It’s always better to be safe than sorry.
Related Resources:
