A/B experimentsβ
Step 1. Formulate a hypothesis
Use experiments to check which Yandex Direct campaign settings are most effective for your business. For example, you can compare campaigns with different creatives, promos, and landing pages, or check how changes to various settings influence the campaign's effectiveness.
It's best to test only one hypothesis related to one parameter within a single experiment.
Step 2. Create test campaigns
Experiments only allow comparing ad campaigns or different settings within the same campaign. Create two identical ad campaigns that differ only in the setting that you want to test. If you want to test different bids within a single campaign, add the same campaign to both variants.
A stopped campaign is launched automatically after you add an experiment to it.
Step 3. Set up an experiment in Yandex Direct
Create an experiment under Library → A/B experimentsβ.
Fill in the Parameters
-
Choose the primary metric for the experiment. The primary metric reflects the nature of your hypothesis but doesn't affect what results are included in the report: all available metrics will be calculated regardless.
-
Specify how many days should pass from the start of the experiment before you plan to analyze the report. This duration represents how many days you expect to wait before seeing tangible results. Not that the experiment will still continue running until you stop it.
-
Select a Yandex Metrica tag and the goal for which you want to collect conversion statistics. By default, all conversions count toward the "Conversions" and "Conversion rate %" metrics.
Choose ad campaign variants
Decide how many variants you need to test. The report with the experiment results compares all variants with the control variant.
-
Select the desired ad campaigns for the control and test variants. A single variant can have one or several campaigns.
-
Specify the sample percentage for each variant. We recommend dividing the audience into equal samples. Make sure the campaign budgets are proportional to their respective sample sizes.
-
Enable bid adjustments if necessary. For instance, you may need them if you want to test different bids within one campaign. In this case, add the same campaign to the control and test variants and set up bid adjustments.
Step 4. Assess the experiment statistics
To open the experiment report, under Library, click A/B experimentsβ → Experiment name → Report or go to the Report tab in the experiment settings.
The report contains values for all metrics. Your primary metric will appear first.
The report shows metric values for each test variant. You can expand them to see the values for each campaign. The report is based on the data available in the Report Wizard.
The cells in the results table are color-coded based on their p-value. P-values less than 0.2 have a dim highlight, while those less than 0.05 are highlighted in bright color. The cell highlight color also depends on the effectiveness of the result. For example, if the number of impressions increases, the corresponding cell turns green, and if the CPA rises, the cell is red.
Note
For the "Conversions", "Conversion (%)" and "Goal cost" metrics, the cells are highlighted in color only for data received after January 1, 2025.
The displayed percentage difference is relative to the control group.
Each table cell contains two metric values: below is the actual absolute value, and above is that value multiplied by the sampling factor to represent 100%.
Step 5. Conclude the experiment
Analyze the results, decide on the best variant, and finish the experiment. There are two ways to do this:
-
By default, the report selects the variant with the greatest improvement in the primary metric. If you're satisfied with the result, click Launch for 100%. The selected campaigns will continue running without any sample size restrictions, and the rest will be stopped.
-
In the table with the list of experiments or in the upper-right corner of the report, click Stop experiment. Select all of the campaigns or some of them: they'll continue running with the current settings and without any sample size restrictions. If you clear all the boxes, all campaigns from the experiment will stop.
Alert
To experiment with adjustments, first stop the experiment, then manually add the desired adjustment in the campaign settings.
Questions?
Alert
Our customer service department can only help you with the campaigns created under the same username you use to contact us. You can check your username by opening ya.ru in another browser tab. Our team can access your data only when processing your request.