Quality control

    How do I set quality control in a pool correctly?

    The settings for quality control rules depend on the type of tasks. General recommendations:

    • Use one or more ways to control the quality of responses.

    • Counting fast answers makes sense for most of the tasks.

    • If the user has to choose between options (for example, by selecting checkboxes), check the answers using majority vote or control tasks.

    • If the user has to provide a response as a text, URL, or uploaded photo, the best way to control quality is by reviewing assignments. You can have other users review assignments. Create a task with a question (for example, "Is this phrase translated correctly?") and possible responses (such as Yes or No). Set up overlap and majority vote checking.

    • If the task is more like an opinion poll (for example, choosing the best pictures from a set), majority vote is not a good way to control quality. Make control tasks with artificial examples where the choice is obvious.

    Should I create a skill for every pool?

    It is better to use one skill in a project. You can choose the way to calculate the skill:

    • Calculate the skill for each pool separately. The current skill value will be the value of the skill in the last pool the user was completing. This option is good if:

      • The pools are intended for different groups of users (for example, there are filters by city or country).

      • Pools are started one by one and you don't want to take into account the answers in the previous pools to calculate the skill in the current pool.

      This option is on by default when you add a quality control rule to a pool. For a golden set rule, leave the History size field empty.

    • Calculate skill based on all tasks in a project This option is good if the pools are small and you don't need to have skill calculated for each pool.

      This option is available only for golden set skills. To use it, fill in the History size field in quality control rules in pools.

    Can I use the same skill in different projects?

    This isn't a good idea. If a user handles one task well, it doesn't mean they will be as successful with a different task. In addition, using skills with old filter settings restricts the number of available users to complete the tasks.