Tips for designing tasks
To get more accurate responses, think through the structure and wording of the tasks and the settings for responses:
- Make a task as simple as possible. Divide complex tasks into several projects.
Performers do better with short, uniform tasks. Therefore, if your task is large or consists of several stages, divide it in several simple tasks and create a project for each task or each stage.
For example, if you need to have road signs recognized, make a separate project for each of the stages: filtering out photos without road signs, selecting road signs in photos, recognizing road signs, and reviewing the results.
- Write good instructions
Instructions for completing a task should be:
Complete. Describe all possible situations the performer may face.
Concise. Use clear wording and simple sentences.
Structured. Give step-by-step instructions (if possible). Use lists and formatting to make the text easier to read.
Illustrative. Provide examples. Add images to them (if the tasks have them).
For example, if a task requires evaluating the quality of an image, put both high-quality and low-quality images in the instructions. If the task is to identify the type of clothing in a picture, describe the possible options in the instructions and illustrate the text with images.
If you review completed assignments using non-automatic acceptance, list the evaluation criteria in the instructions.
- Create a simple and user-friendly task interface
- To help performers work faster, add hotkeys to the task interface.
- To ensure that performers make fewer mistakes, add response validation.
- Don't overload the interface. If there are too many elements in the interface, you might want to divide the project into several smaller projects.
- Check the performance of projects and their display on computer and mobile device screens. Use the sandbox to tune a task.
- Allow enough time for completion.
Haste lowers the quality of responses, so you should allow a little extra time to complete the task.
- Make training tasks.
- Select performers
Make sure to specify the language of the task instruction and text using filters. Add a skill filter and set the Speed/quality balance slider so that only performers with a high rating can participate in the project. For tasks to be completed in mobile apps, set the Client = Mobile Toloka filter.
- Choose the appropriate type of quality control.
If the task contains a simple question with a multiple-choice response and is completed fairly quickly (1-10 minutes), it is best to run the task with overlap and use majority vote check, control tasks, captcha, and restriction of fast responses.
If the task doesn't have clearly defined response options (for example, it requires creating or translating a text or transcribing an audio recording), you can use the following review methods:
Check the responses yourself using the non-automatic acceptance option.
- Set a fair price and issue rewards
- Set a fair price: the more complicated the task is and the more time the performer spends, the higher the price should be. Use dynamic pricing to motivate diligent performers.
- You can issue rewards to performers for completing tasks well. This improves motivation and encourages performers to approach your tasks with care.
- Keep in touch with performers
- Proper communication with performers keeps them interested in the project and lets you quickly detect any issues with the tasks. Answer users' questions through private messages.
Always use one or more ways to control quality of answers.
Counting fast responses makes sense for most tasks.
If the user has to provide a response as a text or link or upload a photo, the best way to control quality is by reviewing assignments. You can outsource task acceptance to performers. Create a task with a question (for example, “Is this phrase translated correctly?”) and possible responses (for example, “yes”/“no”). Set up overlap and majority vote check.
The number of tasks depends on how difficult and time-consuming the tasks are. Keep the size reasonably small. Large task suites are unpopular, partly because they are inconvenient for performers (for example, if the internet connection is unstable).
The settings for quality control rules depend on the type of tasks. General recommendations: