Robots.txt analysis
- How to check the file
- How to find out if the robot will crawl a specific URL
- How to track file changes
- FAQ
The Robots.txt analysis tool helps you to check if the robots.txt file is correct. You can enter the file content, validate it and then copy to robots.txt.
This tool will also help you to track changes to the file and download its specific version.
How to check the file
- If the site was added to Yandex.Webmaster and the site management rights were verified
-
The file content appears on the
page as soon as the rights to manage the website are verified.If the content is shown on the Robots.txt analysis page, click Check.
- If the site has not been added to Yandex.Webmaster
-
- Go to the Robots.txt analysis page.
- In the Site to be checked field, enter the address of your website. For example, https://example.com.
- Click the icon. The content of robots.txt and the results of the analysis will be shown below.
In the sections intended for the Yandex robot (User-agent: Yandex or User-agent: *), the validator checks the directives using the robots.txt terms of use. The other sections are checked against the standard.
- Warnings. They report a deviation from the rules that can be corrected by the tool itself. Warnings also indicate a potential problem with typos or inaccuracies in the directives.
- Errors in the file. This means that the tool cannot process a string, section, or the entire file due to severe syntax errors in the directives.
How to find out if the robot will crawl a specific URL
When your robots.txt file is uploaded to Yandex.Webmaster, the Robots.txt analysis page displays the Check if URLs are allowed block.
- In the URL list field, enter the address of the page you want to check. You can specify the URL completely or relatively to the website root directory. For example, https://example.com/page/ or /page/.
- Click Check.
If the URL allows indexing by Yandex bots, the icon will appear next to it. Otherwise, the address will be highlighted in red.
How to track file changes
To learn about changes to the robots.txt file promptly, please configure notifications.
Yandex.Webmaster regularly checks the file for updates and saves the versions along with the date and time of the change. To view them, go to
.- You added the website to Yandex.Webmaster and verified your right to manage the website.
- Yandex.Webmaster keeps the information on changes in robots.txt.
- View current and previous versions of the file
-
In the robots.txt version list, choose the file version. A field below shows robots.txt along with the parsing results.
- Download the selected file version
-
- In the robots.txt version list, choose the file version.
- Click the Download button. The file will be saved on your device in TXT format.
FAQ
Most likely, you included a mirror in the list of your site URLs. For example, http://example.com instead of http://www.example.com (technically, these are two different URLs). Technically, these are two different URLs. The URLs in the list must belong to the site for which robots.txt is checked.