Robots.txt is a text file that web developers use to advise crawlers which pages they want them to go to. Robots.txt are usually found in the main directory, otherwise bots will not be able to locate the file. If the bot does not find the file, it will index everything on the website. Robots.txt has many advantages when used properly. It can save your bandwidth, remove unneeded data from the web statistics and save server resources. Robot.txt files can be used to keep sensitive information private.

Writing A Robots.txt File

Writing a Robots.txt file for your website is a fairly simple task. First, you need to create a normal text file and save it as “Robots.txt”. It must be named that file otherwise; the crawlers will not recognize it. As mentioned before, the file must be in the root directory. These two factors are mandatory steps that should not be overlooked or modified.

The structure of a robots.txt file is not flexible. Do not change the order of commands. It is primarily composed of lists of disallowed files, directories and user agents. Basically, the term user-agent refers to the search engine crawler. Disallow refers to the files, folders and items that should not be included in the indexing process. The user is also allowed to insert comments in the lines by inserting a pound sign (#) at the start of the line.

An example of a robot.txt file may look similar to something like this:

User-agent: *
Disallow: /cgi-bin
Disallow: /dir/document.htm

Using an asterisk in the user-agent part of the file by default indicates that all bots are not allowed to index the files listed under disallow. However, this can be modified to specific search engine bots if necessary. To indicate that you do not want any bots to index your server, simply leave a / under disallow.

You can also create more complex robot.txt files that may look something like this:

User-agent: *
Disallow: /temp
Disallow: /images
User-agent: Googlebot
Disallow: /private/reports.html

This example includes multiple elements with specific functions. In the first section, all bots are not allowed to index /temp and /images. In the second section, all bots except Googlebot are not allowed to access the files labeled under disallow.

Many search engine crawlers now offer the option of inserting “Allow:” in the Robots.txt file. This is just the opposite of disallow, which dictates which files can be indexed. It is not mandatory, therefore sticking to the disallow functions is advisable.

The Importance of Robots.txt

Blocking certain files from being seen is advantageous. Besides from saving bandwidth, it allows you to protect private files. If you have duplicate content, using a robots.txt file allows the bot not to index both sites, which is helpful in SEO.

In conclusion, Robot.txt files gives the user more control of the content on the website. The more control a user has over the content, the more effective the functionality of the website can be.

<style> p.example{ font-family:"Times New Roman", Times, serif; margin-left:30px; } </style>Back