How do I prohibit indexing of the site or pages?

You can use the robots.txt file, HTML markup or authorization on the site to hide the site content from indexing,

Prohibit indexing of a website, section, or a page

If some pages or sections of the site shouldn't be indexed (for example, the ones that contain technical or confidential information), use the following methods to restrict access to them:

  • In the robots.txt file, specify the Disallow directive.

  • Specify the robots meta tag with the noindex or none directive in the HTML code of the site pages. For more information, see Robots meta tag and HTTP header X-Robots-Tag section.

  • Use authorization on the site. We recommend this method to hide the home page from indexing. If the home page is prohibited in the robots.txt file but there are links leading to it from other sites, the page can be included in the search results.

    Note

    To prevent unauthorized users from accessing restricted pages, configure the server’s HTTP response code for such pages to 404 Not Found, 403 Forbidden, or 410 Gone.

Prohibit indexing of part of the page text

There are several ways to hide part of the text from indexing:

  • Add the noindex element in the HTML code of the page. For example:

    <noindex>text that shouldn't be indexed</noindex>
    

    The element is not sensitive to nesting — it can be located anywhere in the HTML code of the page. If there is no closing tag on the page, the entire contents of the page is considered hidden. Don't create multiple nested noindex tags because markup will ignore everything after the first closing tag.

    You can use the tag in the following format if necessary for making the site code valid:

    <!--noindex-->text that should not be indexed<!--/noindex-->
    
  • Add the noscript element in the HTML code of the page. For example:

    <noscript>text that shouldn't be indexed</noscript>
    

    The noscript element, just like noindex, prohibits indexing but hides the site's content from the user if their browser supports JavaScript.

    Note

    JavaScript is supported by all popular browsers if this feature is not specifically disabled by the user.

    You can view the report on JavaScript support in Yandex Metrica.

We recommend using the rel attribute. Attribute values indicate various link types, which help the search engine better recognize website contents.

  • rel="ugc". Use it if there's a forum on your website or an opportunity for visitors to leave feedback and you aren't sure about the quality of links that they provide.
  • rel="sponsored". Use it if the link is related to advertising or points to an ad placement or a placement under the terms and conditions of a partner program with another website.
  • rel="nofollow". Use it to prevent the robot from following the link regardless of its type.

You can combine multiple values. Example:

<a href="url" rel="nofollow,sponsored">link text</a>
or
<a href="url" rel="nofollow sponsored">link text</a>

Values of the rel attribute are perceived by the robot as a recommendation to ignore the link.

To hide all links on the page from indexing, in the HTML code of the page, specify the robots meta tag with the nofollow directive. The robot will not follow links when crawling the site, but it can find out about them from other sources. For example, on other pages or sites.

When using any of the listed instructions, the link can be processed by the robot and displayed in Yandex Webmaster as internal or external. The display or absence of the link in Yandex Webmaster doesn't mean that the search algorithms take it into account.

Learn more

Contact support



You can also go to