Select your cookie preferences

We use essential cookies and similar tools that are necessary to provide our site and services. We use performance cookies to collect anonymous statistics, so we can understand how customers use our site and make improvements. Essential cookies cannot be deactivated, but you can choose “Customize” or “Decline” to decline performance cookies.

If you agree, AWS and approved third parties will also use cookies to provide useful site features, remember your preferences, and display relevant content, including relevant advertising. To accept or decline all non-essential cookies, choose “Accept” or “Decline.” To make more detailed choices, choose “Customize.”

Review Policies - Amazon Mechanical Turk

Review Policies

Using Amazon Mechanical Turk Review Policies you can evaluate Worker submissions against a defined set of criteria. You specify the Review Policy(s) that you want to use when you call the CreateHIT operation.

There are two types of Review Policies, Assignment-level and HIT-level:

  • An Assignment-level Review Policy is applied as soon as a Worker submits an assignment. For more information, see Assignment Review Policies.

  • A HIT-level Review Policy is applied when a HIT becomes reviewable. For more information, see HIT Review Policies.

You can select from a set of pre-defined Review Policies. One Review Policy leverages known answers or gold standards within a Human Intelligence Task (HIT) and has Mechanical Turk calculate a Worker’s performance on these known answers. You can specify an action for Mechanical Turk to take automatically based on Worker performance against the known answer.

Mechanical Turk has Review Policies that calculate consensus/agreement among multiple Workers performing the same HITs. For instance, you can specify a Review Policy that measures agreement on work items within the HIT and authorizes Mechanical Turk to keep asking additional Workers to work on the HIT, until a certain level of agreement is achieved. Once the required level of agreement is achieved, the results are returned to you for immediate use.

Review Policies that track Worker performance on your known answers and agreement with other Workers give you information you can use to manage your Workers. For more information about using Review Policies, see Review Policy Use Cases.

PrivacySite termsCookie preferences
© 2025, Amazon Web Services, Inc. or its affiliates. All rights reserved.