When your application creates a HIT using the CreateHIT operation, the HIT becomes available for Workers to find on the Mechanical Turk web site. If a Worker has Qualifications that meet the HIT's Qualification requirements, the Worker can preview the HIT, then select the "Accept HIT" button to begin work.
When a Worker accepts a HIT, Mechanical Turk creates an assignment to track the work to completion. The assignment belongs exclusively to the Worker, and guarantees that the Worker will be allowed to submit results any time until the HIT's AssignmentDurationInSeconds has elapsed, and still be eligible for the reward.
A Worker completes an assignment by entering values into the question form and selecting the "Submit HIT" button. The results are stored for later retrieval by your application. The Worker continues working on other HITs, or ends the session.
If a Worker decides not to complete a HIT after accepting it, the Worker may select the "Return HIT" button. Unless the HIT has expired-- that is, unless the HIT's LifetimeInSeconds has elapsed since the HIT was created-- the returned HIT becomes available for another qualified Worker to accept. The assignment ends, and is never returned to the Requester application.
If a Worker does not submit results before the assignment's deadline-- that is, before the HIT's AssignmentDurationInSeconds has elapsed since the Worker accepted the HIT-- the HIT is considered abandoned. The effect is similar to if the Worker explicitly returned the HIT, except that abandonments and returns are tracked as separate statistics in the Worker's profile.
Tip:
A Worker's HIT return and abandonment rates are tracked by Mechanical Turk as system Qualifications. You can create HITs that use Qualification requirements based on these numbers. Abandonments and returns are tracked as separate Qualifications.By default, a HIT has at most one assignment. When a Worker accepts the HIT, an assignment is created, and the HIT is no longer available for other Workers to accept. If the Worker returns or abandons the HIT, the assignment is removed, and the HIT becomes available again.
A HIT can be created to accept multiple assignments by specifying a MaxAssignments parameter greater than 1 to the CreateHIT operation. Such a HIT will remain available for Workers to accept as long as the number of assignments, in progress or submitted, is less than MaxAssignments.
A HIT is only available for Workers to accept until the HIT's LifetimeInSeconds elapses, from the time the HIT was created. Once this time elapses, the HIT expires. Such a HIT is no longer available, even if the number of assignments is less than MaxAssignments. Workers with assignments in progress are allowed to continue working on the assignments as long as the AssignmentDurationInSeconds has not elapsed, even after the HIT expires. If a Worker returns or abandons the HIT after the HIT has expired, the HIT is not made available to other Workers.
When either all of a HIT's assignments have been submitted by Workers, or the HIT has expired and all assignments have either been submitted, returned or abandoned, the HIT is considered "reviewable." Once a HIT is reviewable, the Requester can retrieve and process the results.
The GetReviewableHITs operation returns the HIT IDs for all of your HITs that are reviewable. The GetAssignmentsForHIT operation takes the ID of a reviewable HIT and returns the completed assignments, with the answers submitted by Workers. Your application can use these operations to poll for and retrieve results for HITs.
Once results for an assignment have been retrieved, you approve or reject the assignment. Your application calls the ApproveAssignment operation to approve the assignment, or the RejectAssignment operation to reject the assignment.
Note:
A call to ApproveAssignment will return an error if the Requester's account does not have sufficient funds to pay the Worker and the listing fee at the time of the call.An assignment should always be approved, unless it is clear the Worker did not follow the instructions in the HIT. The method your application uses to decide if an assignment should be approved will depend on the nature of your application. For example, you may request 3 assignments for each HIT then check to see if the results match, and if they do, approve all 3 assignments. If they don't match, have those results checked by a human operator. In some cases, it may be appropriate to automatically approve all assignments, then control the quality of answers using Qualification requirements.
Every HIT has an AutoApprovalDelayInSeconds, an amount of time after an assignment is submitted at which point Mechanical Turk will automatically approve the assignment if it has not already been explicitly approved or rejected. If not specified in the call to CreateHIT, this is set to the maximum, equivalent to 30 days. The maximum value assures that the Worker will get paid even if the Requester is unavailable. You may wish to specify a shorter auto-approval delay if the Workers are trusted (such as by having been vetted with Qualification requirements) and to assure Workers that they will be paid in a timely manner.
When you call the ApproveAssignment operation, Mechanical Turk automatically processes payment of the reward to the Worker, and payment of the HIT listing fee, using the money in your Amazon.com account. You will not be charged for the reward, nor for the listing fee, if the results are rejected.
Once all assignments have been approved or rejected, your application calls the DisposeHIT operation. This removes the HIT from the list of HITs returned by GetReviewableHITs, and tells Mechanical Turk the data is no longer needed.
If a HIT has expired, or the maximum number of assignments have all been submitted, the HIT will no longer be available for Workers to accept. If the HIT has not gathered a satisfactory result, you may extend either the expiration date or the number of assignments, or both, using the ExtendHIT operation. With appropriate values, extending the HIT will make it available again.
The DisableHIT operation allows you to completely withdraw a HIT from the system, even before it has expired. Assignments that have been submitted (but not yet explicitly approved or rejected) will be approved automatically. Assignments in progress are allowed to complete, then approved automatically if submitted by the Worker. (Assignments returned or abandoned after the HIT is disabled are simply discarded.) The HIT and all submitted assignment data are disposed. A disabled HIT cannot be re-enabled.
Disabling a HIT is useful if the HIT's question data is incorrect and would cause Workers to submit bad results. Because Workers expect to be paid for correctly answering the question, even if the question is not the one you intended to ask, DisableHIT approves all assignments that have been submitted, and all assignments in progress that are submitted before their deadline. You will be charged for these assignments if you disable the HIT.
Tip:
Assignment data cannot be retrieved until the HIT enters the "reviewable" status, when it is no longer possible for Workers to submit more results. Because it is only necessary to disable a HIT to prevent Workers from submitting results, it is likely you will only wish to disable a HIT prior to it becoming "reviewable". This means you will not get to see the results for submitted assignments prior to disabling the HIT.
If you let a HIT collect results and become reviewable, then retrieve results, then extend the HIT (with the ExtendHIT operation), you can change your mind about the extension and disable the HIT. In this case, you will not be able to retrieve results for assignments submitted between the call to ExtendHIT and the call to DisableHIT. Any assignments that have not been explicitly approved or rejected will be approved.
The NotifyWorkers operation lets you send e-mail to Workers who have completed your HITs in the past. Using the Worker ID found in the assignment data you get back for a HIT, you can send the Worker a message without having to know their name or e-mail address.