You can review tasks manually and reject them if they weren't completed satisfactorily. Rejected tasks are not paid.
Reviewing tasks may take from 1 to 21 days (this period is set by the requester). When the period ends, tasks that haven't been reviewed are automatically approved. The user can challenge the results by writing to the requester. The decision can be changed within 30 days from when the task was sent for review.
To use offline acceptance, make the settings before starting the pool:
Choose the Offline accept parameter on the pool page in edit mode.
In Deadline, enter the number of days for reviewing (from 1 to 21 days). The user will see the deadline in the task information on the main Toloka page.
Save your settings.
Users' responses will become available for reviewing as tasks are completed.
Explain the acceptance criteria in the instructions for users.
If you reject a task suite, Toloka can resend it for completion to other users. To set up automated resending of task suites, add After review re-assessment quality control rule to the pool.
To go to the list of unreviewed tasks, click Review assignments on the pool page. You can review the tasks online or in a TSV file.
To review tasks online:
Choose a task and review it.
Click Accept or Decline. For rejected tasks, enter a comment (explain why you declined it).
You can also accept or reject multiple tasks at once: select the tasks in the list and click Accept or Decline.
To use a TSV file for review:
Click Import/Export on the task review page.
Select “Submitted” to download the responses that you haven't checked yet.
Open the file with responses. Fill in the fields (the content of the task fields in each task suite should be the same):
ASSIGNMENT:assignment_id — ID of the assigned task suite.
ACCEPT:verdict — Review result:
“+” if you accept the responses.
“-” if you reject the responses.
ACCEPT:comment — Comments for users if responses were rejected (for example, specify which part of the instructions wasn't followed).
Upload the edited TSV file to Toloka (Import/ Export → Upload results).