Tips for designing tasks

To get more accurate responses, think through the structure and wording of the tasks and the settings for responses:

Make a task as simple as possible. Divide complex tasks into several projects.

Performers do better with short tasks of the same type. Therefore try to divide a large task into several simple ones, creating a separate project for each task. Example

If your task consists of several stages, create a separate project for each stage.

For example, if you need to have road signs recognized, make a separate project for each of the stages: filtering out photos without road signs, selecting road signs in a photo, recognizing road signs, and reviewing the results.

Prepare good instructions.

Instructions for completing a task should be:

  • Thorough. Describe all possible situations the performer may face.

  • Concise. Use concise wording and simple sentences.

  • Structured. Give step-by-step instructions (if possible). Use lists and formatting to make the text easier to read.

  • Visual. Provide examples. Add images to them (if the tasks have them).

    For example, if a task requires evaluating the quality of an image, put both high-quality and low-quality images in the instructions. If the task is to identify the type of clothing in a picture, describe the possible options in the instructions and illustrate the text with images.

If you review completed assignments using non-automatic acceptance, list the evaluation criteria in the instructions.

Create a simple and user-friendly task interface
  • To help performers work faster, add hotkeys to the task interface.
  • To ensure that performers make fewer mistakes, add response validation.
  • Don't overload the interface. If there are too many elements in the interface, you might want to divide the project into several smaller projects.
  • Check the performance of projects and their display on computer and mobile device screens. Use the sandbox to tune a task.
Allow enough time for completing tasks.

Haste lowers the quality of responses, so you should allow a little extra time to complete the task.

Make training tasks.

Ask performers to go through the training tasks before beginning the pool tasks. Include disputable cases in the training tasks.

Select performers

Make sure to specify the language of the task instruction and text using filters. Add a filter by skill and the Speed/quality balance slider so that only performers with a high rating can participate in the project. For tasks to be completed in mobile apps, set the Client = Mobile Toloka filter.

Add a test based on the results of the training task to select performers. To do this, assign skills depending on whether their responses are correct.

Choose the appropriate type of quality control.

If the task contains a simple question with a multiple-choice response and is completed fairly quickly (1-10 minutes), it is best to run the task with overlap and use majority vote checking, a golden set, a captcha, and restriction of quick responses.

If the task doesn't have clearly defined response options (for example, it requires creating or translating a text or transcribing an audio recording), you can use the following verification methods:

  • Check the responses yourself using the non-automatic acceptance option.

  • Submit the responses to Toloka performers for checking. Create a task with a question (for example, “Is this phrase translated correctly?”) and possible responses (for example, “yes”/“no”). Set up overlap and majority vote checking.

Set a fair price and award bonuses
Set a fair price: the more complicated the task is and the more time the performer spends, the higher the price should be. Use dynamic pricing to motivate diligent performers.
You can give bonuses to performers for completing tasks well. This improves motivation and makes performers approach your tasks with more care.
Keep in touch with performers
Proper communication with performers keeps them interested in the project and lets you quickly detect any issues with the tasks. Answer users' questions through private messages.