Robots meta tag and X-Robots-Tag HTTP header

You can specify for the robots the rules for loading and indexing certain site pages in one of the following ways:

Place the robots meta tag inside the head element of the HTML code of the page.
Configure the X-Robots-Tag HTTP header for a specific URL on your site's server.

Note

If the page is prohibited in the robots.txt file, the meta tag or header directive does not apply.

By default, the search robots take into account the meta tag and header. You can specify directives for certain robots.

Directives supported by Yandex

Directive	Description	Robots meta tag	X-Robots-Tag header
`noindex`	Prohibits indexing the page text. The page won't be included in the search results.
`nofollow`	Prohibits following the links on the page. The robot will not follow links when crawling the site, but it can find out about them from other sources. For example, on other pages or sites.
`none`	Similar to `noindex` and `nofollow` directives.
`noarchive`	Prohibits showing the link to the saved copy on the search results page.
`noyaca`	Prohibits using an automatically generated description.		—
`index` \| `follow` \| `archive`	Cancels the corresponding prohibiting directives.		—
`all`	Allows you to index text and links on the page, similar to `index` and `follow` directives.		—

The robot uses allow directives by default, so you can omit them if there are no other directives. Allow directives have priority over prohibiting directives if there's a combination of both.

Robots from other search engines and services may interpret the directives in a different way.

Example:

Robots meta tag

X-Robots-Tag header

An element that disables page indexing.

<html>
    <head>
        <meta name="robots" content="noindex" />
    </head>
    <body>...</body>
</html>

An HTTP response with a header that prohibits indexing the page.

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
X-Robots-Tag: noindex

Specifying multiple directives

Robots meta tag

X-Robots-Tag header

You can specify multiple directives separated by commas.

<meta name="yandex" content="noindex, nofollow" />

You can pass multiple headers in a single response and a list of comma-separated directives.

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
X-Robots-Tag: noindex, nofollow
X-Robots-Tag: noarchive

If conflicting directives are specified for the Yandex robot, it will consider a positive value. Example of meta tag directives:

<meta name="robots" content="all"/>
<meta name="robots" content="noindex, follow"/>
<!--The indexing bot will choose the “all” value. Text and links will be indexed.-->
<meta name="robots" content="all"/>
<meta name="robots" content="noarchive"/>
<!--Text and links will be indexed, but in the search results there will be no link 
to the saved copy of the page.-->

Instructions for specific robots

You can use the robots meta tag to give a directive to the Yandex robots only. Example:

<meta name="yandex" content="noindex" />

If you list common directives and directives for the Yandex robots, the search engine will take into account all of them.

<meta name="robots" content="noindex" />
<meta name="yandex" content="nofollow" />

The Yandex robot will treat these directives as noindex, nofollow.

Contact support

If the pages don't appear in the search results for a long time or were excluded from them, provide examples of such pages in the form.

You can also go to

Was the article helpful?

Meta tags that Yandex takes into account

About the IndexNow protocol