How do I lower the load on a site or server?

The indexing robot plans site page visits by itself, adjusting the load on the site or server automatically depending on how many new or already indexed pages of the site need to be crawled.

Sometimes the number of robot requests can increase dramatically, for example, if the robot finds out about a new site section, changes in the site structure, or new page URLs.

To find out the cause of increased robot requests to the site:

  1. Check the number of pages loaded by the robot and the pages participating in the search. To do this, open the My sites page in Yandex.Webmaster. If the number of loaded pages is much larger than the number of pages in the search, the robot probably finds a large number of similar pages on the site or duplicate pages that shouldn't be indexed.



  2. On the Indexing → Crawl statistics page, check which pages the robot recently accessed. You can also find this information in the server logs.

    Pay attention to the pages with GET parameters. They likely contain identical content. For example, if the site has filters by folder, the pages may contain a parameter like ?price=1000&max_price=1000000. Or, if the site has a search, ?search=<search phrase>.

    Even though the robot crawls pages with different URLs, since the content is the same, they can't participate in the search results.

  3. Add the Clean-param directive to the robots.txt file. This way you can direct the robot to the necessary URL among the duplicates and the load will gradually reduce. This directive also allows you to transfer certain accumulated metrics to the main URL or website.

    To quickly reduce the load on the robot, add the Disallow directive to robots.txt. The robot will stop accessing specified pages within 24 hours of making changes to the file.

If you don't think that the load has to do with extra pages, you can configure the server to respond with the 429 HTTP status code. This response will indicate that the server experiences difficulties with the load. The Yandex robot will reduce the number of requests. Or you can change the site crawl rate yourself.