How do I lower a load on the site or server?

The indexing robot plans site page visits by itself, adjusting the load on the site or server automatically depending on how many new or already indexed pages of the site need to be crawled.

Sometimes the number of robot requests can increase dramatically, for example, if the robot finds out about a new site section, changes in the site structure, or new page URLs.

To find out the cause of increased robot requests to the site:

  1. Check the number of pages loaded by the robot and the pages participating in the search. To do this, open the My sites page in Yandex Webmaster. If the number of loaded pages is much larger than the number of pages in the search, the robot probably finds a large number of similar pages on the site or duplicate pages that shouldn't be indexed.

  2. Go to Indexing → Crawl statistics and see which pages the indexing bot has accessed recently. You can also find this information in the server logs.

    Pay attention to the pages with GET parameters. Most likely, they contain identical content. For example, if the site has filters by folder, the pages may contain a parameter like ?price=1,000&max_price=1,000,000. Or if the site has a search function, ?search=<search phrase>.

    Even though the robot crawls pages with different URLs, but with the same content, they can't participate in the search results.

  3. Add the Clean-param directive to the robots.txt file. This way you can direct the robot to the necessary URL among the duplicates and the load will gradually drop. This directive also allows you to transfer certain accumulated metrics to the main URL or website.

    To quickly reduce the load on the robot, add the Disallow directive to robots.txt. The robot will stop accessing specified pages within 24 hours after making changes to the file.

If you believe that the load is not related to extra pages, you can configure the server to respond with the 429 HTTP status code. This response will indicate that the server experiences difficulties with the load. The Yandex robot will reduce the number of requests. Or you can change site crawl rate yourself.

Learn more

Contact support



You can also go to