Creating an experiment
To create an experiment:
- Go to AppMetrica.
- Go to the A/B experiments page and click Create experiment.
Adding conditions
By default, experiments are only limited in time. You can further restrict your experiment's conditions by adding rules. Added conditions are joined using the AND operator.
-
Audience share: Percentage of the total audience across all experimental variants (set to 100% be default). You can change this value to run the experiment only on a portion of your app's audience.
-
Mobile OS: Operating system your app is built for.
-
OS versions: Version of the user's operating system.
This value must only contain digits separated by dots. To specify a single version, set the same value for the start and end of the range. For Android, specify the API level.
-
Languages: Device language of the users who will see your configuration.
-
Regions: Region set in the device settings of the users.
-
App versions: Version or range of versions of your app that will show the configuration to users.
-
Custom parameters: You can define your own parameters as key-value pairs. For example,
param=value
.To apply multiple conditions in Varioqub, separate them with commas:
aparam=avalue,bparam=bvalue
(the conditions are joined by a logical AND). Condition blocks are combined by the OR operator.Allowed characters for parameters and their values
=
— Placed between the parameter name and its value. If absent, only the parameter's presence is taken into account. Example:param=value
?
— Matches any character between zero or one time. Example:pa?am=va?ue
*
— Matches any characters between zero or unlimited times. Example:p*m=v*e
+
— Matches any characters between one or unlimited times. Example:p+m=v+e
!
— Means negation of a condition or exclusion of a parameter from influencing the experiment. Example:!param=val
Warning
Only use negation
!
as the first character in a key.- Correct example:
!param=value
. - Incorrect examples:
param!=value
orparam=!value
.
Usage examples
Usage example
What's taken into account in the experiment
param
Only the presence of the parameter, regardless of its value.
param=value
Exact match of the parameter and its exact value.
!param
Absence of the parameter with any value.
*param*=value
All parameters that contain the
param
fragment with thevalue
value.!*param*=value
The value of any parameter with the
param
fragment shouldn't bevalue
.!*param*=*value*
The value of any parameter with the
param
fragment shouldn't containvalue
.!*param*
Absence of a parameter containing the
param
fragment.Note
This option is available for paid AppMetrica plans.
Intersections with experiments
Intersection is when two or more experiments run concurrently and may influence each other.
For example, if you're conducting an experiment where you change the color of the Buy button and simultaneously testing a new product description, you may see changes that come from both experiments, which can impact the results. If you want to conduct the experiment independently, select the experiments you want to exclude in the list.
This way, the experiment you're creating won't overlap with the excluded ones. The new experiment will also not affect the excluded experiments' results.
The sum of the percentages (shares) of the audiences of the new and excluded experiments can't exceed 100%. For example, if the audience of the new experiment is 30%, then the audience of the excluded experiments can't exceed 70% so that the shares add up to 100%.
Note
This option is available for paid AppMetrica plans.
Metrics selection
Choose the key metrics by which you'll determine the experiment's success. You can also select up to 10 secondary metrics for advanced analysis (this option is available for paid AppMetrica pricing plans).
For the Conversion to event..., Conversion from event to event..., and Step-by-step conversion from event to event... metrics, you can specify additional parameters in the input field with parameters. You can specify up to 5 levels of parameters for each event: just type a parameter name and press Enter. The total length of each parameter in the event mustn't exceed 50 characters.
The metrics you select will be included in the experiment results report.
Setting up an experimental variant
You can set up multiple variants to use in the experiment. You can either use the version of your app without the changes as the control variant or customize changes for the control variant.
Experimental changes applied to the control and experimental variants can be configured with flags. You don't need to make changes to the app itself. The flags accept String values.
- Select the variant for which you want to apply the experimental changes.
- Set the flags with your modified parameters.
Sample size calculator
Note
This option is available for paid AppMetrica plans.
Using the sample size calculator, you can assess whether the experiment's result will be statistically significant with the given audience size.
This tool calculates the MDE to show the smallest significant change in a metric. You can use it to determine the likelihood of capturing small changes in the experiment with the given data and error levels. The smaller the MDE, the more subtle changes your experiment can detect. To reduce the MDE, increase the sample size or the experiment duration. If the MDE is high, you'll only be able to spot significant changes. You can conduct such an experiment on a small audience.
Fill in the calculator fields so that they match the information about your experiment.
-
Total users per month: Consider your experiment conditions but not the sample size here. Specify the latter in Audience share. To calculate the value for this field, use data from AppMetrica.
-
Duration: Duration of the experiment in days. Corresponds to the duration in the Date range field.
-
Audience share: Sample size, corresponds to the Audience share field.
-
Variants: Number of variants in the experiment, can take values from 2 to 26.
-
Metric conversion: Percentage of users converted in the control variant. What action counts as a conversion depends on the exact metric. If you want to get statistically significant results for all metrics, specify the minimum conversion here.
Borders of the detected effect show which indicators will be statistically significant. The values outside these borders will be significant: lower than the red border and greater than the green border. Indicator values in-between can be random deviations from the conversion of the control variant. If this range is too broad and you're interested in less pronounced changes in metrics, try adjusting the conditions of your experiment.
Launching an experiment
Warning
You won't be able to change the conditions, metrics, and variants for your experiment after you launch it.
To launch your experiment, click Save and run. Familiarize yourself with the brief information about the experiment and click Run if everything is okay.
Once the experiment is launched, you'll be able to view a preliminary report on the A/B experiments tab.
Checking values
You can check how your changes were applied on different devices. To do this:
- Make sure that the Varioqub SDK is installed on the device.
- Use the SDK to get the device ID:
- For Android, use the Varioqub.getId() method.
- For iOS, use VarioqubFacade.shared.varioqubId.
- Specify the retrieved ID under Experiment testing and click Activate. The changes will become available after some time.
They will be deactivated automatically one day after activation. You can deactivate the ID manually.
Stands for the minimum detectable effect.