Reviewing performers' responses

You started a pool with non-automatic acceptance, and the performers completed all your tasks. What next?

  • Review the performers' responses before the end of the Review period you specified. At the end of the period, unchecked responses will be accepted automatically.
  • If a performer didn't complete the task well, reject their response.
  • The performer can challenge your decision by submitting an appeal.

How do I review the tasks?

In the Toloka interface. This option is suitable for a small number of tasks.

To accept or reject responses:

If you set non-automatic acceptance in the pool settings, you need to review the performers' responses within the time limit set in the Deadline field. Performers may challenge a rejected response by submitting an appeal.

You can review assignments online or upload review results in a TSV file.

Note. Note that you can't change the task status if the task pool was archived.
Online review of one task.

To accept or reject one task:

  1. Click the Review assignments button on the pool page.

  2. Choose an assignment.

    If responses include files uploaded by the performers, click the Actions → Download attachments button to download them. To download all files from submitted responses in a ZIP archive, click Download results → Download attachments on the assignments review page.

  3. Check the responses, click Accept or Decline. For rejected responses, enter a comment (specify the reason).

In the TSV file with the results. You will need the file to filter the results and process them programmatically.
Note.

This review method is also suitable when the result is not displayed in the interface.

Example 1: The task uses JS code, and the response contains initially hidden fields.

Example 2: The task where you need to download files attached by the performer (for example, video or audio recordings) to check the responses.

To accept or reject responses:

  1. Click the Review assignments button on the pool page.

  2. To download responses that you didn't review yet: on the pool page, click Download results. In the window that opens:

    1. In Status leave only the Submitted option enabled.

    2. In Columns leave only the assignment ID option enabled.

    3. Disable the Separate assignments with empty row option.

    4. Click Download results.

  3. If your task requires performers to submit photos, videos or audio files, download them for review in an archive.

    Click Download results → Download attachments.

    Note.

    What does the response archive contain?

    The system automatically assigns names to files when they are uploaded by the performer. You can rename the files manually or with a script if necessary.

    • Response archive

      • Folder1_ID_ responses

        • File1_ID_attachments

        • File2_ID_attachments

      • Folder2_ID_ responses

    You can match responses and attachments by their IDs. Attachment ID is included in the TSV file, in the column with the output field where the file was uploaded.

    • TSV file

      • Column1_with_output_field

        • ID_attachments

      • Column2_with_output_field

        • ID_attachments

  4. Open the file with responses.

    For a single response ID, you need to fill in the verdict and comment only once. The rest should be left empty or deleted.

    • If there is one task on the page, add a verdict and a comment to each response ID (if there is a deviation).
    • If there is more than one task on the page, then different tasks on the page will have the same response ID.

    Explanation

    • ASSIGNMENT:assignment_id — ID of the assigned task suite.

    • ACCEPT:verdict — Review result:

      • “+” if you accept the responses.

      • “-” if you reject the responses.

    • ACCEPT:comment — Comments for performers if responses were rejected (for example, specify which part of the instructions wasn't followed).

  5. Upload the edited TSV file to Toloka (Import/ Export → Upload results).

Delegate assignment review to other performers.

Place a separate task for performers to review the responses. To learn how to do it, see the last project in the Selecting an image area. It implements assignment review with the help of performers.

Note. Note that you can't change the task status if the task pool was archived.

Appeal

Within 7 days after the review, the performer can challenge the result by filing an appeal.

You will receive a message from them that contains:

  • Toloker ID (assignment_id) in the subject.
  • Explanation of why responses should be accepted (optional).

To consider an appeal, you need the response ID. If the performer wrote to you from the rejected response page, the response ID is automatically attached to their message as a link. If there is no ID, request it from the performer.

Do the following:

  • If the responses were mistakenly rejected, accept them.
  • Consider all appeals and respond on time — within 9 days (from the moment when the task was rejected).

Troubleshooting

Can I ask a performer to redo the task if they made mistakes in it?

No. After sending a task, the performer can't make any changes to it. You can add tasks that were completed incorrectly to a new pool.

Can I fix something in a completed task myself?

No, you can't fix anything in the task itself. However, you can do this manually in the results file.

What should I do if I want to accept a completed task but the pool is already archived or the performer wrote to me after the allowed time?

Simply give the performer a separate reward without changing the task status. You can't change the task status in the pool in this case.

What should I do if I rejected a task for a reason that isn't specified in the instructions?

Accept the task and update the instructions. Otherwise, you violate the Requester Agreement that requires to clearly state the task requirements and the results expected from the user.

Can I reject part of the responses on the page and accept part of them?

No. For example, there are 10 tasks in a suite that costs $0.10, and the performer did 2 of them incorrectly.

You can't accept the correct answers and pay for this part ($0.08). Response pages are accepted or rejected in their entirety.