Reviewing assignments

Assignment review is a pool setting that allows you to check responses and no to pay for poorly completed tasks.

You can find the Review methods in the Working with results → Reviewing performers' responses section.

How does it work?

You set up the assignment review for your pool in the project.

As the performers complete the tasks, their responses come for review. You can accept or reject a performer's response. If you reject responses, the performer can:

  • Challenge the review result by writing a message to the requester — submit an appeal.

  • Write to the platform support service.

  • Put a low rating to your project.

To get fewer appeals and maintain a high rating for your projects:
  • Write a clear instruction with a clear algorithm of actions, requirements for results and possible reasons for their rejection.

  • Check the responses well — according to your own instructions.

  • Reject response pages if the result is largely inconsistent with the instructions.

    Example

    If the task is to mark up 20 pictures according to the instructions, and the performer completed 18 of them correctly, and 2 were incorrect, the result is considered incorrect to a small extent, or mostly correct.

  • Explain the mistake to the performer so they can learn and do better next time.

You can review responses manually or with the help of performers by posting a separate review task.

Pool status and the review results

30 days after the tasks are completed the pool is automatically sent to the archive. It isn't possible to change the review result in an archive pool, so you won't be able to accept a previously rejected assignment.

Rejecting responses

  • If you reject responses, assignments aren't sent automatically to other performers.
  • You can't reject (or accept) one response on the page — only the entire page with all responses on it.
  • If the performer is banned by the quality control rules, their responses still need to be checked.
Note. Rejecting the performer's responses in one task doesn't result in rejecting responses of this performer.

How do I turn on assignment review?

Take the following steps before starting the pool:

  1. Choose the Non-automatic acceptance parameter on the pool page in edit mode.

  2. Specify the Review period — the number of days for the review (from 1 to 21 days from the task completion date). The performer will see the deadline in the task information on the main Yandex.Toloka page.

    Tip. Set the review period up to 14 days. Performers are reluctant to accept assignments with a longer review period.
  3. Save the settings — the Save button is at the bottom of the pool parameters page.

As the performers start completing the tasks, you will receive answers.

Attention.
  • Set the deadline so that you have enough time to review the responses. When the period ends, responses that haven't been reviewed are accepted and paid for automatically.
  • Explain the acceptance criteria in the instructions for performers. This will improve the quality of the results, as well as reduce the number of appeals and questions about rejection reasons from the performers.

More settings

Send tasks with rejected responses for re-check

If responses are rejected, Yandex.Toloka can automatically send the task to other performers. To set it:

  1. Add the Recompletion of rejected assignments quality control rule to the pool.
  2. Set the following values in the rule block:

Filter out performers if their responses are often rejected

You can also filter out performers based on the percentage of rejected responses.

You can restrict the performers' access to tasks:

  • Block them (set up a quality control rule).
  • Suspend their access to projects (set up a quality control rule).
  • Assign them a skill and use it as a filter in other projects (set up filters in pool parameters).

Learn more about this in the Review results rule settings.