“Yandex Direct, experiments” report
Experiments help you test different versions of ads and landing pages, as well as different forecasts for advertising in Yandex Direct. You can test any campaign settings, campaign types, and media plans.
The report helps you determine how well an experiment works: you can compare test groups with each other and with the control group, if you use it in the experiment, by a variety of metrics (for example, bounces and time on the site), as well as by conversion rate. In this way you can use the report to analyze the result of an experiment based on a set of metrics.
If you need a clear answer based on CPA, use the A/B testing reliability calculator. You can change the significance level (p-value). In the report “Yandex.Direct, experiments” the p-value is 68%.
To view the report, go to Reports → Sources → Yandex Direct, experiments.
Conditions for getting the report
To run a test, create an experiment in Yandex Audience. Specify how many segments you want and what proportions to use to divide the audience. Then configure ad campaigns for the experiment in Yandex Direct.
If you deleted an experiment in Yandex Audience, statistics on it and its segments are available in Yandex Metrica for the period prior to deletion.
How to conduct an experiment
Before launching the experiment, give clear names to experimental campaigns in Yandex Direct. This will make it easier for you to analyze statistics in Yandex Metrica.
In the name, specify the experiment and group that the campaign is targeted for. For example, if one of the groups in your experiment is a control group (you will compare the results of the other groups with it), this should be written in the name of the campaign aimed at it.
After you have launched an experimental campaign in Yandex Direct, you can check that the experiment has actually started. To do this, in the Yandex.Metrica report “Direct, experiments” create a segment with all the campaigns that are involved in the experiment. If the report shows sessions and other statistics for these campaigns, then the experiment was launched.
Tip
Save the created segment — you will need it later when viewing statistics.
Why do I need to create a segment
After you create an experiment in Yandex Audience, the system randomly divides Yandex users into experimental groups. The experiment may include users who visited the advertiser’s site and users who didn’t. Users who visited the site may have gotten there some way other than via a Yandex Direct ad. To select only sessions that came from ads, create a segment with the relevant Yandex Direct campaigns.
Evaluating experiment results
Note
We recommend that you start evaluating data using the report “Yandex.Direct, experiments” two weeks after launching an experiment.
In the “Yandex.Direct, experiments” report:
- Select the experiment you want to analyze.
- Select the segment with the Yandex Direct campaigns that you created for the experiment. If you don’t have a segment yet, create one following our recommendations.
- Compare the segments of the experiment. Click Comparison mode and select a control group to compare the results of other groups to.
Note
To compare the effectiveness of an experimental and control campaign, you can add previously created goals to the report.
In comparison mode, some metrics are highlighted in color:
- Green or red marks if there is enough data to evaluate the result of the experiment. Green means that this metric was significantly better than in the control group. Red means it is significantly worse.
- Gray means there’s enough data, but the values of metrics in the control group don’t differ from the test group.
No color is shown if the comparison mode isn’t available for the selected metric or there isn’t enough data accumulated. Wait a while longer or start an experiment with other settings that allow you to collect more traffic.
Metrics that are highlighted in color
- Pageviews
- Sessions
- Converted sessions
- Users
- Converted users
- Goal completions
- Conversions per user
- Conversions
- Conversion rate, users
- Bounces
- Product views
- Users who have viewed the item
- Items added to basket
- Items removed from basket
- Number of users who added the product to the basket
- Products purchased
- Number of visitors who bought the product
- Percentage of missed calls
- Percentage of first-time callers
- Number of calls from unique phone numbers
You can add other metrics to the report. Only those in the list above will be highlighted in color.
Data interpretation
You can use the report to compare ad campaigns by different metrics. This helps you understand according to which metrics one campaign should be more effective than another.
To verify the results of an experiment, you can take the margin of error into account. This is the possible deviation of the metric’s actual value from its true value, or what it would strive for if the experimental campaign was conducted indefinitely. To account for the margin of error, click Comparison mode and enable the display of the margin of error.
The margin of error in the report is relative. Its indicator reflects the percentage of the value’s possible spread. The lower the margin of error, the higher the probability that the metric value is accurate. The higher it is, the greater the deviation. You can assume that with other things being equal, the indicators are better for an advertising campaign where the margin of error is lower.
Report structure and settings
The data in the report is grouped by the name of the experiment created in Yandex Audience. You can only add the groups and segments that were created for conducting the experiment to the report.
Some metrics are calculated and displayed only when the condition is met:
Metric |
Criterion |
Converted sessions |
|
Сonverted users |
|
Conversions |
|
Conversions per user |
|
Conversion rate |
|
User conversion rate |
|
Item views |
Pass e-commerce data to Yandex Metrica. |
Users who have viewed the item |
|
Items added to basket |
|
Items removed from basket |
|
Number of users who added an item to their basket |
|
Items purchased |
|
Number of users who bought items |
|
Percentage of missed calls |
Pass information about calls to Yandex Metrica. |
Percentage of first-time callers |
|
Number of calls from unique phone numbers |
You can view general statistics for an experiment in other reports on Yandex.Direct, for example, Yandex.Direct, summary:
The name of the experiment is displayed as a grouping condition in Yandex Metrica. For example, you can view statistics on an ad campaign as part of an experiment.
A segment created in Yandex Audience is displayed as a segmentation condition in Yandex Metrica. For example, you can only view the session depth of those users who were in the selected segment of the experiment.
Questions and answers
How to compare bounce rate and conversion
To compare the bounce rate and conversion in experimental and control campaigns, switch the desired column to relative values in comparison mode (). You will see the percentage by which the metric value in the experimental campaign differs from the control campaign, and not the share of this metric in “Total and average,” as in other Yandex Metrica reports.
For example, if you see 100% in the Bounce rate column for an experimental campaign, this means that the bounce rate is the same in the control and experimental groups.
How metrics are compared
Yandex Metrica uses different methods for verifying the statistical significance of metrics based on the behavior of each of them. The comparison takes the spread of the experimental and control groups into account, not the absolute values.
Useful links |
Online training |