Toloka for requesters

Quality control

How do I set quality control in a pool correctly?

The settings of quality control rules depend on the type of tasks. General recommendations:

  • Use one or more ways to control quality of answers.

  • Counting fast answers makes sense for most of the tasks.

  • If the user has to choose between options (for example, ticking checkboxes), tcheck the answers using majority vote or control tasks.

  • If the user has to provide a response as a text, link or upload a foto, the best way to control quality is reviewing assignments. You can have assignments review completed by users. Create a task with a question (for example, Is this phrase translated correctly?) and possible responses (for example, yes/no). Set up overlap and majority vote checking.

  • If a task is more like an opininon poll (for example, choosing nice pictures from a row), majority vote does not suit to control quality. Make control tasks with artificial examples where the choice is evident.

Should I create a skill for every pool?

It is better to use one skill in a project. You can choose the way to calculate the skill:

  • Calculate the skill for each pool separately. The current skill value will be the value of the skill in the last pool the user was completing. This option is good if:

    • The pools are intended for different groups of users (for example, there are filters by city or country).

    • Pools are started one by one and you don't want to take into account the answers in the previous pools to calculate the skill in the current pool.

    This option is on by default when you add quality control rule to a pool. For a golden set rule, leave the History size field empty.

  • Calculate skill based on all tasks in a project This option is good if the pools are small and you don't need to have skill calculated for each pool.

    This option is available only for golden set skills. To use it, fill in the History size field in quality control rules in pools.

Can I use the same skill in different projects?

You shouldn't do it. If a user copes well with one task, it doesn't mean they will be as successful with another. Besides, using skills with old filter settings you restrict the number of available users to complete the tasks.