Task Configuration (AMI 2.0)

Two configuration options determine how many tasks are run per node, one for mappers and the other or reducers. They are:

Amazon Elastic MapReduce (Amazon EMR) provides defaults that are entirely dependent on the Amazon EC2 instance type. The following table shows the default settings for job flows launched with AMI 2.0.

	Note
	The number of default mappers is based on the memory available on each Amazon EC2 instance type. If you increase the default number of mappers, you also need to modify the task JVM settings to decrease the amount of memory allocated to each task. Failure to modify the JVM settings appropriately could result in out of memory errors.

When your job flow runs, Hadoop creates a number of map and reduce tasks. These determine the number of tasks that can run simultaneously during your job flow. Run too few tasks and you have nodes sitting idle, run too many and there is significant framework overhead.

Amazon EMR determines the number of map tasks from the size and number of files of your input data. You configure the reducer setting. There are four settings you can modify to adjust the reducer setting.

The parameters for configuring the reducer setting are described in the following table.

The two tasksperslot parameters are unique to Amazon Elastic MapReduce (Amazon EMR). They only take effect if mapred.*.tasks is not defined. The order of precedence is:

You can configure the amount of heap space for tasks as well as other JVM options with the mapred.child.java.opts setting. Amazon EMR provides a default -Xmx value in this spot, with the defaults per instance type shown in the following table.

You can start a new JVM for every task, which provides better task isolation, or you can share JVMs between tasks, providing lower framework overhead. If you are processing many small files, it makes sense to reuse the JVM many times to amortize the cost of start-up. However, if each task takes a long time or processes a large amount of data, then you might choose to not reuse the JVM to ensure all memory is freed for subsequent tasks.

Use the mapred.job.reuse.jvm.num.tasks option to configure the JVM reuse settings.

$ ./elasticmapreduce --create --alive --name "JVM infinite reuse" \
--bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop \
--bootstrap-name "Configuring infinite JVM reuse" \
--args "-m,mapred.job.reuse.jvm.num.tasks=-1"

	Note
	Amazon EMR sets the value of mapred.job.reuse.jvm.num.tasks to 20, but you can override it with a bootstrap action. A value of -1 means infinite reuse within a single job, and 0 means do not reuse tasks.

In a distributed environment, you are going to experience random delays, slow hardware, failing hardware, and other problems that collectively slow down your job flow. This is known as the stragglers problem. Hadoop has a feature called speculative execution that can help mitigate this issue. As the job flow progresses, some machines complete their tasks. Hadoop schedules tasks on nodes that are free. Whichever task finishes first is the successful one and the other tasks are killed. This feature can substantially cut down on the run time of jobs. The general design of a mapreduce alogorithm is such that the processing of map tasks is meant to be idempotent. If, however, you are running a job where the task execution has side effects (for example, a zero reducer job that calls an external resource) is it important to disable speculative execution.

You can enable speculative execution for mappers and reducers independently. By default, Amazon EMR enables it for mappers and reducers in AMI 2.0. You can override these settings with a bootstrap action. For more information on using bootstrap actions, refer to Bootstrap Actions.

$ ./elasticmapreduce --create --alive --name "Reducer speculative execution" \
 --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop \
--bootstrap-name "Disable reducer speculative execution" \
--args "-m,mapred.reduce.tasks.speculative.execution=false"