Reviewing assignments

You can review assignments manually and reject them if they weren't completed satisfactorily. Rejected tasks are not paid.

Reviewing assignments may take from 1 to 21 days (this period is set by the requester). When the period ends, assignments that haven't been reviewed are automatically approved. The user can challenge the results by writing to the requester. The decision can be changed within 30 days from when the assignment was sent for review.

You can also filter out users based on the prportion of rejected assignments.

Setting up assignments review for a pool

To be able to review assignments when they are completed, make the settings before starting the pool:

  1. Choose the Offline accept parameter on the pool page in edit mode.

  2. In Deadline, enter the number of days for reviewing (from 1 to 21 days). The user will see the deadline in the task information on the main Toloka page.

  3. Save your settings.

Users' responses will become available for reviewing as assignments are completed.

Note.

Explain the acceptance criteria in the instructions for users.

Resending assignments for completion

If you reject an assignment, Toloka can resend it for completion to other users. To set up automated resending of assignments, add After review re-assessment quality control rule to the pool.

Performing the review

You can review assignments online or upload review results in TSV file.

To accept or reject one task:

  1. Click the Review assignments button on the pool page.

  2. Choose an assignment.

    If responses include files uploaded by the users, click the Actions → Download attachments button to download them. To download all files in submitted assignments in a ZIP archive, click Download results → Download attachments on the assignments review page.

  3. Review the assignment, click Accept or Decline. For rejected assignements, enter a comment (explain why you declined it).

To accept or reject multiple assignments at once, select the tasks in the list and click Accept or Decline.