Robots.Txt ��-�� - RobotsExclusion/guide

2006-11-03 09:03:26

�� /Robots Exclusion / guide �� 2006-11-03 09:03:26 � 2007-07-01 22:18:30

��:

��, �� – �� «Disallow: /cgi-bin/ /tmp/». �� Disallow �� User-agent �� , �.�. �� .

��:

��, �� – �� «Disallow: /cgi-bin/ /tmp/». �� Disallow bkb User-agent �� , �.�. �� .

2006-08-18 18:44:57

�� /Robots Exclusion / guide �� 2006-08-18 18:44:57 � 2006-11-03 09:03:26

��:

�� – �� .
�� . ��, �� :
�� «/robots.txt». ��, �� robots.txt � �� – �� . �� robots.txt � ��, �� robots.txt, �� . �� -�� Robots.
��, �� – �� «Disallow: /cgi-bin/ /tmp/». �� Disallow bkb User-agent �� , �.�. �� .
�� , �.�. �� “Allow”. �� , �� :

��:

�� – �� .
�� . � ��, �� :
�� «/robots.txt». ��, �� robots.txt � �� – �� . �� robots.txt � ��, �� robots.txt, �� . �� -�� Robots.
��, �� – �� «Disallow: /cgi-bin/ /tmp/». �� Disallow bkb User-agent �� , �.�. �� .
�� , �.�. �� “Allow”. �� , �� :

2006-08-18 09:22:44

�� /Robots Exclusion / guide �� 2006-08-18 09:22:44 � 2006-08-18 18:44:57

��:

��

�� «/robots.txt».

��


User-agent: Yandex
====��������� � ���������� ��� ����� ����� ������====
��� �������� ��������, �.�. �� ��������� ���������� "Allow". ������ ����� ����� ����������� ��� ����� ����� ����, ������� �� ������ ��������� � ���������� � ������������� � ��������� �� ����������:
Disallow: /docs/

�� :
Disallow: /private.html
Disallow: /foo.html
Disallow: /bar.html%%

��:

The Robot will simply look for a «/robots.txt» URL on your site, where a site is defined as a HTTP server running on a particular host and port number. For example:

To exclude all robots from the entire server

To allow all robots complete access

Or create an empty «/robots.txt» file.

To exclude all robots from part of the server

To exclude a single robot

To allow a single robot


User-agent: WebCrawler
====To exclude all files except one====
This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "docs", and leave the one file in the level above this directory:
Disallow: /~joe/docs/

Alternatively you can explicitly disallow all disallowed pages:
Disallow: /joe/private.html
Disallow: /joe/foo.html
Disallow: /joe/bar.html%%

2006-08-17 09:03:22

�� /Robots Exclusion / guide �� 2006-08-17 09:03:22 � 2006-08-18 09:22:44

��:

�� , �� robots.txt �� .

�� robots.txt

� �� robots.txt �� :
� �� .
��, �� – �� «Disallow: /cgi-bin/ /tmp/». �� Disallow bkb User-agent �� , �.�. �� .
�� . «��» (*) � �� User-agent �� «�� ». �� «Disallow: *.gif» �� «User-agent: Ya*" �� .
�� robots.txt �� , �� . �� :

��:

So, you need to provide the «/robots.txt» in the top-level of your URL space. How to do this depends on your particular server software and configuration.
For most servers it means creating a file in your top-level server directory. On a UNIX machine this might be /usr/local/etc/httpd/htdocs/robots.txt

What to put into the robots.txt file

The «/robots.txt» file usually contains a record looking like this:
In this example, three directories are excluded.
Note that you need a separate “Disallow” line for every URL prefix you want to exclude — you cannot say «Disallow: /cgi-bin/ /tmp/». Also, you may not have blank lines in a record, as they are used to delimit multiple records.
Note also that regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning “any robot”. Specifically, you cannot have lines like «Disallow: /tmp/*" or «Disallow: *.gif”.
What you want to exclude depends on your server. Everything not explicitly disallowed is considered fair game to retrieve. Here follow some examples:

2006-08-17 08:54:34

�� /Robots Exclusion / guide �� 2006-08-17 08:54:34 � 2006-08-17 09:03:22

��:

��

||URL �� |URL �� robots.txt ||
�� «/robots.txt». ��, �� robots.txt � �� – �� . �� robots.txt � ��, �� robots.txt, �� . �� -�� Robots.
�� , �� URL-� �� , � �� «/robots.txt» �� .
||�� robots.txt||
||http://www.w3.org/admin/robots.txt|�� ||
||http://www.w3.org/~timbl/robots.txt|�� ||
||ftp://ftp.w3.com/robots.txt|�� ftp||
||http://www.w3.org/Robots.txt|�� ||

��:

Introduction

||URL �� |�� URL �� robots.txt ||
Note that there can only be a single «/robots.txt» on a site. Specifically, you should not put “robots.txt” files in user directories, because a robot will never look at them. If you want your users to be able to create their own “robots.txt”, you will need to merge them all into a single «/robots.txt”. If you don't want to do this your users might want to use the Robots META Tag instead.
Also, remeber that URL's are case sensitive, and «/robots.txt» must be all lower-case.
||Pointless robots.txt URLs||
||http://www.w3.org/admin/robots.txt|comment||
||http://www.w3.org/~timbl/robots.txt|comment||
||ftp://ftp.w3.com/robots.txt|comment||

2006-08-17 08:37:13

�� /Robots Exclusion / guide �� 2006-08-17 08:37:13 � 2006-08-17 08:54:34

��:

�� – �� -�� robots.txt.
�� – �� .
�� . � ��, �� :
�� , �� , �� «/robots.txt». �� , �� , �� .

�� robots.txt

�� URL «/robots.txt», �� – �� .
||URL �� |�� URL �� robots.txt ||

��:

This guide is aimed at Web Server Administrators who want to use the Robots Exclusion Protocol.
Note that this is not a specification — for details and formal syntax and definition see the specification.
The Robots Exclusion Protocol is very straightforward. In a nutshell it works like this:
When a compliant Web Robot vists a site, it first checks for a «/robots.txt» URL on the site. If this URL exists, the Robot parses its contents for directives that instruct the robot not to visit certain parts of the site.
As a Web Server Administrator you can create directives that make sense for your site. This page tells you how.

Where to create the robots.txt file

||Site URL |Corresponding robots.txt URL ||

2006-08-17 08:14:15

�� /Robots Exclusion / guide �� 2006-08-17 08:14:15 � 2006-08-17 08:37:13

��:

What to put into the robots.txt file

The «/robots.txt» file usually contains a record looking like this:


User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

In this example, three directories are excluded.
Note that you need a separate “Disallow” line for every URL prefix you want to exclude — you cannot say «Disallow: /cgi-bin/ /tmp/». Also, you may not have blank lines in a record, as they are used to delimit multiple records.
Note also that regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning “any robot”. Specifically, you cannot have lines like «Disallow: /tmp/*" or «Disallow: *.gif”.
What you want to exclude depends on your server. Everything not explicitly disallowed is considered fair game to retrieve. Here follow some examples:

To exclude all robots from the entire server


User-agent: *
Disallow: /

To allow all robots complete access


User-agent: *
Disallow:

Or create an empty «/robots.txt» file.

To exclude all robots from part of the server


User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/

To exclude a single robot


User-agent: BadBot
Disallow: /

To allow a single robot


User-agent: WebCrawler
Disallow:
User-agent: *
Disallow: /

To exclude all files except one

This is currently a bit awkward, as there is no “Allow” field. The easy way is to put all files to be disallowed into a separate directory, say “docs”, and leave the one file in the level above this directory:


User-agent: *
Disallow: /~joe/docs/

Alternatively you can explicitly disallow all disallowed pages:


User-agent: *
Disallow: /~joe/private.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html

2006-08-17 08:12:10

�� /Robots Exclusion / guide �� 2006-08-17 08:12:10 � 2006-08-17 08:14:15

��:

||Pointless robots.txt URLs||
||http://www.w3.org/admin/robots.txt|comment||
||http://www.w3.org/~timbl/robots.txt|comment||
||ftp://ftp.w3.com/robots.txt|comment||

��:

||Pointless robots.txt URLs ||
||http://www.w3.org/admin/robots.txt |comment ||
||http://www.w3.org/~timbl/robots.txt |comment ||
||ftp://ftp.w3.com/robots.txt |comment ||

2006-08-17 08:10:09

�� /Robots Exclusion / guide �� 2006-08-17 08:10:09 � 2006-08-17 08:12:10

��:

||Pointless robots.txt URLs ||
||http://www.w3.org/admin/robots.txt |comment ||
||http://www.w3.org/~timbl/robots.txt |comment ||
||ftp://ftp.w3.com/robots.txt |comment ||

Robots.Txt ��-������ - RobotsExclusion/guide

2006-11-03 09:03:26

2006-08-18 18:44:57

2006-08-18 09:22:44

��������� ���� ���� ��� ���������� ����� ��������

��������� ���� ������� ������������� ���� ����

������� �� ���������� ������ ��������� ���������

��������� ���������� ����� ������ ��� ������ ������

��������� ���������� ����� ������ ������ � ��������� ���� ���������

To exclude all robots from the entire server

To allow all robots complete access

To exclude all robots from part of the server

To exclude a single robot

To allow a single robot

2006-08-17 09:03:22

��� ������ � ���� robots.txt

What to put into the robots.txt file

2006-08-17 08:54:34

��������

Introduction

2006-08-17 08:37:13

��� ��������� ���� robots.txt

Where to create the robots.txt file

2006-08-17 08:14:15

What to put into the robots.txt file

To exclude all robots from the entire server

To allow all robots complete access

To exclude all robots from part of the server

To exclude a single robot

To allow a single robot

To exclude all files except one

2006-08-17 08:12:10

2006-08-17 08:10:09

Robots.Txt ��-�� - RobotsExclusion/guide

��

��

��

��

��

�� robots.txt

��

�� robots.txt