Dynamic overlap

Overlap is the number of performers who should complete each task in the pool. To save money, you can set up dynamic overlap.

Dynamic overlap lets you change overlap depending on:
  • How well the performers cope with the task.
  • How well performers' responses match.
Tip. Don't use the Recompletion of assignments from banned users quality control rule with dynamic overlap. This increases the maximum overlap value.
  1. How dynamic overlap works
  2. Preparing output data fields
  3. Setting up dynamic overlap
  4. Example

How dynamic overlap works

To set up dynamic overlap, use:
Maximum overlap
The maximum number of performers to assign the same task to.
Skill

Skill reflects how much you trust the performer's responses. To configure dynamic overlap correctly, use the skill calculated as the percentage of correct responses in control tasks.

Confidence in response

The confidence in the performer's response depends on the skill value.

Confidence in an aggregated response is calculated based on the confidence in each response using a probability algorithm.

Output fields for aggregation
Output data fields to use for aggregating responses. These fields must have fixed values.

First the task is distributed with minimal overlap. If, after this, the confidence in the aggregated response doesn't reach the specified value, the overlap is increased until:

  • The overlap reaches the maximum or
  • The confidence in the aggregated response reaches the specified value.

Preparing output data fields

To compare the performers' responses, specify fixed values for the output data fields to use for aggregating responses.

Output data fields that can be aggregated:
  • Strings and numbers with allowed values.
  • Boolean.
  • Integers with minimum and maximum values. The maximum difference between them is 32.
Note. Arrays can't be aggregated.

To add a set of fixed values for output data fields:

  1. Go to project editing and scroll to the Specifications section.
  2. Hover the mouse over the output data field and click .
  3. Add allowed values.
    Example
  4. Save the field.
  5. Save the project.

    Attention. If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.

Setting up dynamic overlap

  1. Set the minimum overlap in the Overlap field.
  2. Turn on the Dynamic overlap option.
  3. Set Max overlap.
  4. Specify Confidence of aggregated answer. The higher the value, the more matching performers' responses are needed for the aggregated response. Set this value to at least 70.
  5. Choose Skill for users' scoring. This defines the weight of the performer's response.
  6. Check Output fields for answers agreement.
  7. Save the pool.
  8. Upload tasks via smart mixing, otherwise the pool tasks are assigned with minimal overlap.
  9. After the pool is completed, start Aggregation of results by skill.
Note. Approximate budget in the pool statistics is calculated based on the minimal overlap.

Example

Configuring dynamic overlap for simple classification.

The result has three allowed values: OK, BAD, and 404.

Task 1

Performer A with a skill value of 70 responded with OK.

Performer B with a skill value of 90 also responded with OK.

The confidence in the aggregated OK response is 97.7. The task is considered completed.

Task 2

Performer A with a skill value of 70 responded with OK.

Performer B with a skill value of 90 responded with BAD.

The confidence in the aggregated BAD response is 76.1. The overlap is increased by 1.

Performer C with a skill value of 80 responded with BAD.

The confidence in the aggregated OK response is 96.2. The task is considered completed.