Possible issues
- Robots.txt file not found
- Errors were detected in the robots.txt file
- No Sitemap files are currently used by the indexing bot
- Errors found in Sitemap files
- Incorrect display settings for non-existent or deleted files and pages
- The title element and the description metatag are missing
- Pages have the same titles and descriptions
- The favicon file is not accessible to the robot
This section contains solutions to common problems from the “Possible” category identified during site diagnostics in Yandex Webmaster. Issues in this category may affect the quality and speed of site indexing.
Tip
Configure notifications about the results of site monitoring.
Robots.txt file not found
The indexing robot requests the robots.txt file several times a day and updates information about it in its database. It the robot cannot load the file, a warning appears in Yandex Webmaster.
In the service, check the availability of the robots.txt file. If the file is still unavailable, add it. If you can't do this yourself, contact your hosting provider or domain name registrar. After the file is added, the data in Yandex Webmaster will be updated within a few days.
Errors were detected in the robots.txt file
Check the robots.txt file of your site. To fix errors, see the directive descriptions.
No Sitemap files are currently used by the indexing bot
The Sitemap file it is an auxiliary tool for indexing a site. It allows you to regularly inform the robot about new pages on the site. This warning appears if the robot is not using any Sitemap
files for the site.
For the robot to start using the created file, add it to Yandex Webmaster and wait for the robot to process the file. This can take up to two weeks. After that, the warning will disappear.
Errors found in Sitemap files
Check the Sitemap file of your site. The check may detect the error “Unknown tag”. This means that the file contains elements that are not supported by Yandex. The unsupported elements are ignored by the robot during Sitemap
processing, but data from supported elements is still taken into account. You don't have to change the contents of the file. Learn more about supported Sitemap elements.
Incorrect display settings for non-existent or deleted files and pages
Probably, the site is incorrectly configured to return the 404 HTTP code (resource not found/Not Found). In this case, the search bot may receive a 200 OK response when accessing the site via any link and index a non-existent page. This will lead to a large number of duplicate pages, complicating and slowing down the indexing of the site and increasing the server load.
To check the server response, use the Page status check tool. Enter the address of a page that does not exist on your site. You can use any combination of letters and numbers, for example, https://example.com/kugbkkrfck
.
If it is impossible to set the recommended 404 response code:
-
Set any other 4xx code, except 429.
We do not recommend using 429 and 5xx response codes, as they tell the bot that the server is experiencing difficulties. This may slow down the indexing of the site.
-
Use the noindex directive or set up a 301 redirect.
When using
noindex
and 301 redirect, error notifications may appear in Yandex Webmaster, but the pages will not be included in the search, and it will not affect the site’s ranking. These methods are the least preferable, as they increase the server load and slow down the indexing of the site.
The title element and the description metatag are missing
The title
element and description
metatag are used for generating a good site description in search results. For details, see Snippet.
If either or both of these elements are missing from your website, add them to the HTML code of the pages and save the changes. If the elements have already been added, wait for the robot to recrawl the pages. The error message will disappear after reindexing. Learn more
Pages have the same titles and descriptions
This problem appears if the title or description is duplicated on many pages of the website. The title
element and the description
meta tag should be informative and appealing and reflect the page content. This will help users find the search results they are looking for.
See examples of pages with duplicate titles or descriptions that Yandex Webmaster found while the robot crawled the website. To fix them, follow the recommendations for:
The problem will stop being displayed when the robot learns about changes on the website. To make this happen faster, send the most important pages to be reindexed or configure page crawling with a Yandex Metrica tag.
The favicon file is not accessible to the robot
If this issue is displayed for your site, it means the favicon is not appearing in search results. In Yandex Webmaster, go to Website optimization → Site diagnostics (Possible problems section) and check the reason why the indexing bot couldn't load the file, then follow the instructions:
Error |
Solution |
The file responds with an HTTP code other than 200 OK |
Check the server response. The response should be 200 OK. For other response statuses, see Server response check. |
The file redirects to another URL |
|
Incorrect data type |
Check the |
A small image displayed in the snippet in Yandex search results, next to the site address in the browser's address bar, and next to the site name in Favorites or Bookmarks in the web browser.