How do I remove pages from the search?

Sometimes you need to exclude a site page from search results, for example, if it contains confidential information, is a duplicate of another page, or was deleted from the site.

Step 1. Prohibit the page or directory indexing

There are several ways to do this:

If the page is removed from the site

  • Add the Disallow directive in the robots.txt file.

  • Configure the server so that when the robot accesses the page URL it sends HTTP status with the 404 Not Found, 403 Forbidden, or 410 Gone code. For the user convenience, we recommend setting up a redirect with HTTP 301 code.

    To check whether the instructions in the robots.txt file are correct, use the Robots.txt analysis tool.

    Note

    When crawling the site, the robot first accesses the robots.txt file if there is one, and then crawls the site pages. Sequential crawling may take some time. If you want to delete many pages from the search, add the Disallow directive for the desired pages.

If the page should not be displayed in the search

  • Specify the robots meta tag with the noindex directive.

  • Configure the server so that when the robot accesses the page URL it sends HTTP status with the 404 Not Found, 403 Forbidden, or 410 Gone code.

    For details, see Robots meta tag and X-Robots-Tag HTTP header.

    Note

    Do not restrict such pages in robots.txt, or the Yandex bot can’t index them and detect your instructions.

The robot's behavior when excluding a page with different methods

Exclusion method

The robot's behavior

Prohibition in the robots.txt file

The page will be removed from the search database within a week after the indexing bot discovers your instructions. However, the page may sometimes appear in search results (for example, if other resources link to it).

HTTP status with the 404, 403 or 410 code

The page will be removed from the search database within a week after the indexing bot discovers your instructions.

The robots meta tag with the noindex directive

The URL of the removed page will appear in the list of excluded pages in Yandex Webmaster. Go to Indexing → Searchable pages to see it.

Excluding pages that violate copyright from the search isn't the robot's priority task. To exclude a page from the search, use the methods described in this section.

Pages excluded from search results may be displayed in Yandex Webmaster until the next site crawl.

We recommend viewing How to remove a page from the search.

Step 2. Speed up the page removal

To speed up the page removal from the search, tell Yandex to remove it, without waiting for the planned robot crawl.

  • If your site hasn't been added or verified in Yandex Webmaster:

    1. Go to the Remove pages from search results page in Yandex Webmaster.
    2. Enter the URL of the page to exclude in the field, for example http://example.com/page.html.
    3. Click the Remove button.

    To exclude multiple pages from the search, remove them one by one.

  • If your site was added to Yandex Webmaster and you verified your site management rights:

    1. Go to Tools → Remove pages from search results.
    2. Set the radio button to By URL.
    3. Enter the page URL in the field, for example http://example.com/page.html.
    4. Click the Remove button.

    You can specify up to 500 URLs per site per day.

You can delete all site pages, individual directories, or pages with the specified parameters in the URL if your site is added to Yandex Webmaster and you verified your site management rights.

  1. In Yandex Webmaster, go to Tools → Remove pages from search results.

  2. Set the radio button to By prefix.

  3. Specify the prefix:

    What to delete

    Example

    Site directory

    http://example.com/catalogue/

    All site pages

    http://example.com/

    URL with parameters

    http://example.com/page?

    You can send up to 20 prefixes per site per day.

  4. Click the Remove button.

Note

The indexing bot will remove the pages specified in Yandex Webmaster only if the Disallow directive is set for these pages in the robots.txt file.

After a URL is submitted to Yandex Webmaster, you can track status changes in Tools → Remove pages from search results:

Status

Description

“In the delete queue”

The robot checks the server response and if the page is prohibited from indexing. The check can take several minutes.

“In progress”

The robot checked the page. The page will be removed from search results within 24 hours.

“Deleted”

The page was removed from the search results.

“Rejected”

The page is allowed for indexing or when the robot accesses the page URL, the server response is different from 404 Not Found, 403 Forbidden or 410 Gone.

How do I return a page to the search results?

Remove the indexing prohibition: the Disallow directive in the robots.txt file or the noindex meta tag. The pages return to the search results when the robot crawls the site and finds out about the changes. This may take up to three weeks.

Questions and answers

I configured redirect to the new pages, but the old ones are still displayed in the search results

If you use a redirect, the robot will gradually track redirects and the old pages will disappear from the search results as it crawls the site. For the robot to learn about the changes faster, send the pages for reindexing.

If the page URLs change with the change of the site's domain name, it may take more than a month to update the search data. Check if the site addresses are configured correctly.

Contact support

Tell us what your question is about so we can direct you to the right specialist:

If you followed the instructions when deleting the site pages and more than two weeks passed after the site was crawled by the Yandex robot, fill out the form below:

If you used a redirect, submit pages for reindexing.

The pages are sent for reindexing, but aren't removed from the search for more than two weeks



You can also go to