Setup tools

If you need a more detailed configuration when creating an experiment, use the additional setup tools.

Sample size calculator

Using the sample size calculator, you can assess whether the experiment's result will be statistically significant with the given audience size.

This tool calculates the MDE to show the smallest change in a metric. You can use it to determine the likelihood of capturing small changes in the experiment with the given data and error levels. The smaller the MDE, the more subtle changes your experiment can detect. To reduce the MDE, increase the sample size or the experiment duration. If the MDE is high, you'll only be able to spot significant changes. You can conduct such an experiment on a small audience.

Fill in the calculator fields so that they match the information about your app and experiment.

• Total monthly visits: Visits to the experimental pages that met the conditions of the experiment, not accounting for the sample sizes. To calculate the value of this field, use data from Yandex Metrica.

• Duration: Duration of the experiment in days. Corresponds to the duration in the Date range field.

• Audience share: Sample size, corresponds to the Audience share field.

• Variants: Number of variants in the experiment, can take values from 2 to 26.

• Metric conversion: Percentage of users converted in the control variant. What action counts as a conversion depends on the exact metric. If you want to get statistically significant results for all metrics, specify the minimum conversion here.

Borders of the detected effect show which indicators will be statistically significant. The values outside these borders will be significant: lower than the red border and greater than the green border. Indicator values in-between can be random deviations from the conversion of the control variant. If this range is too broad and you're interested in less pronounced changes in metrics, try adjusting the conditions of your experiment.

Experiment testing

To make sure that your changes are applied, under Experiment testing, click Test next to each variant. The site version associated with the selected variant opens at a link generated by Varioqub. You can share this link within 24 hours of its generation. Clicking Test again generates a new link.

Editing a running experiment

You can edit a running experiment.

1. In Yandex Metrica, go to Varioqub → Experiments.

2. Click → Edit next to the name of the experiment you want to edit.

This will open a page where you can edit the parameters you set when creating the experiment.

Disabling variants

When editing a running experiment, you can disable its variants.

To do this, select the variant you want to disable and click Disable next to its name.

After disabling, users who were initially included in this variant will no longer see the changes from the experiment, as if they were never part of it at all. Nothing changes for users from other variants, and they'll still remain in their variant.

Sample experiments

The examples for connecting Varioqub and creating experiments are available on GitHub.

FAQ and troubleshooting

I'm trying to create an experiment, but all the fields are locked.

What if I want to use conditions that are not available in the Varioqub interface?

How do I disable a specific experiment type for an entire site or page?

Contact support

Minimum Detectable Effect.