Dynamic overlap

Theory

Dynamic overlap lets you change overlap depending on:
  • How well the performers do on the task.
  • How well performers' responses match each other.

First the task is distributed with minimal overlap. As performers complete the task, their responses are assigned confidence. If it is less than specified in the confidence in aggregated response field, overlap will be increased until confidence reaches the set value, or overlap reaches the maximum.

Dynamic overlap uses 5 main parameters:
  1. Overlap is the minimum number of performers who should complete each task in the pool.

  2. Maximum overlap is the maximum number of performers who should complete each task in the pool.

  3. Confidence in aggregated response is the percentage of confidence required to stop showing the task to performers.

    When confidence reaches the required level, the system considers the response correct and stops collecting other performers' responses.

  4. Skill reflects how much you trust the performer's responses.

  5. Output fields for aggregation are output data fields that are used to calculate confidence in the response.

How does it work?

Terms

  • — a performer's accuracy
  • — smoothing constant
  • — the most popular response
  • — the probability that the estimate is correct
  • — confidence in the aggregate response

A performer's accuracyis calculated as:

Where:

is a smoothing constant (starting from 0.5) if there are not enough responses to control tasks.

If there are several estimates, the most popular response is determined by adding togetherof the performers who selected each response option. The response with the largest total is considered more correct. Let's call this estimate.

Using Bayes' theorem, we calculate the posterior probability that the estimateis correct.

A uniform distribution of estimates is assumed a priori. For thethe a priori probability is calculated as

where:

is the number of response options.

Next, we calculate the probability that the estimateis correct.

If the performer responded, then the probability of this is equal to the performer's accuracy. If they responded differently, then the probability of this is:

Where:

is the remaining probability;

is the number of remaining responses.

It ensures that the probability of an error is distributed evenly among the remaining estimates.

We take all performers' responses and, for example, optionand calculate the probability that performers will select this response, provided that the correct response is:

func z_prob(x int) : float {
    d = 1,0
    for w[i]: workers
         if answers[w[i]] == z[x]
            d *= q[i]
         else
            d *= (1 - q[i]) / (Y - 1)
    return d
}

Next, using Bayes' theorem, we calculate the probability that the responseis correct:

r = 0
for z[i]: answer_options
    r += z_prob(i) * (1 / Y)

eps = z_prob(j) * (1 / Y) / r

Next, we look at. If the expected result is less than the value ofwe increase overlap until the result is equal to the confidence in the aggregated response or overlap reaches the maximum.

Practice

Before setting up dynamic overlap, you need to prepare the output data fields.

You can use the following fields:
  • Strings and numbers with allowed values.

    The allowed value must match the value parameter in the corresponding interface element.

  • Boolean.
  • Integers with minimum and maximum values. The maximum difference between them is 32.

    If there are too many possible responses in the output field, the dynamic overlap mechanism won't be able to aggregate the data.

You can't use the following fields:
  • Array.
  • File.
  • Coordinates.
  • JSON objects.
Setting up overlap
  1. Create a pool and fill in the parameters.
  2. Set minimum overlap in the Overlap field.
  3. Turn on the Dynamic overlap option.
  4. Set Max overlap.
  5. Specify Confidence in the aggregated response. The higher the value, the more matching performers' responses are needed for the aggregated response. Set this value to at least 70.
  6. Select Skill for weighting responses.
  7. Check Output data for aggregation.
  8. Save the pool.
  9. Upload tasks via smart mixing.
  10. After the pool is completed, start Aggregation of results by skill.
Note. Approximate budget in the pool statistics is calculated based on minimum overlap.
Example

Configuring dynamic overlap for simple classification.

The result has three allowed values: OK, BAD, and 404.

Task 1

Performer A with a skill value of 70 responded with OK.

Performer B with a skill value of 90 also responded with OK.

The confidence in the aggregated OK response is 97.7. The task is considered completed.

Task 2

Performer A with a skill value of 70 responded with OK.

Performer B with a skill value of 90 responded with BAD.

The confidence in the aggregated BAD response is 76.1. The overlap is increased by 1.

Performer C with a skill value of 80 responded with BAD.

The confidence in the aggregated OK response is 96.2. The task is considered completed.

Tips and recommendations

  • Don't use the Recompletion of assignments from banned users quality control rule with dynamic overlap. This increases the maximum overlap value.

  • Set confidence in the aggregated response at between 70 and 95 percent.
    • When the value is < 70%, the average accuracy of the aggregated response isn't sufficient to trust the performers' responses.

    • When the value is > 90%, the average accuracy of the aggregated response doesn't increase, and the labelling costs increase.

    Optimal confidence is at 80% for the best “price/quality” ratio.

  • Upload tasks via smart mixing. Otherwise, the tasks are assigned with minimum overlap.

    This is necessary for overlap to increase because control and training tasks uploaded via smart mixing have infinite overlap.

  • Use a skill calculated as the percentage of correct responses to control tasks. This will give you the most accurate results.

  • If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.

Troubleshooting

Can I change overlap after the pool is started?

Yes. Open edit mode for the pool and set a new overlap value. You don't need to restart the pool. Updating the settings is usually fast, but if there are many tasks, it may take several minutes.

With dynamic overlap, is it possible that the pool will close before the tasks for minimal overlap run out? The overlap increased, but the pool is closed, and I need to start it manually.

Yes, this might happen. You must set an adequate pool closing interval.