Associated AWS Product Concepts

Amazon EMR enables you to choose the number and kind of Amazon EC2 instances that comprise the cluster that processes your job flow. Amazon EC2 offers four basic types.

	Note
	Amazon EMR does not support micro instances at this time.

The following table describes all the instance types that Amazon EMR supports.

*Cluster Compute Instances are available only in the US-East (Northern Virginia) Region.

**Cluster GPU instances have 22 GB, with 1 GB reserved for GPU operation.

The practical limit of the amount of data you can process depends on the number and type of Amazon EC2 instances selected as your cluster nodes, and on the size of your intermediate and final data. This is because the input, intermediate, and output data sets reside on the cluster nodes while your job flow runs. For example, the maximum amount of data that you can process on a 20-node cluster is 34 TB (20 Extra Large instances x 1.69 TB of hard disk per Amazon EC2 instance = 34 TB).

The default maximum number of Amazon EC2 instances you can specify is 20. If you need more instances, you can make a formal request. For more information, go to the Request to Increase Amazon EC2 Instance Limit Form.

Reserved Instances provide guaranteed capacity and are an additional Amazon EC2 pricing option. You make a one-time payment for an instance to reserve capacity and reduce hourly usage charges. Reserved Instances complement existing Amazon EC2 On-Demand Instances and provide an option to reduce computing costs. As with On-Demand Instances, you pay only for the compute capacity that you actually consume, and if you don't use an instance, you don't pay usage charges for it.

To use a Reserved Instance with Amazon EMR, launch your job flow in the same Availability Zone as your Reserved Instance. For example, let's say you purchase one m1.small Reserved Instance in US-East. If you launch a job flow that uses two m1.small instances in the same Availability Zone in Region US-East, one instance is billed at the Reserved Instance rate and the other is billed at the On-Demand rate. If you have a sufficient number of available Reserved Instances for the total number of instances you want to launch, you are guaranteed capacity. Your Reserved Instances are used before any On-Demand Instances are created.

You can use Reserved Instances by using either the Amazon EMR console, the command line interface (CLI), Amazon EMR API actions, or the AWS SDKs.

Elastic IP addresses are static IP addresses designed for dynamic cloud computing. An Elastic IP address is associated with your account, not a particular instance. You control the addresses associated with your account until you choose to explicitly release them.

You can associate one Elastic IP address with only one job flow at a time. To ensure our customers are efficiently using Elastic IP addresses, we impose a small hourly charge when IP addresses associated with your account are not mapped to a job flow or Amazon EC2 instance. When Elastic IP addresses are mapped to an instance, they are free of charge.

For more information about enabling Elastic IP addresses with Amazon EMR, see Using Elastic IP Addresses. For more information about using IP addresses in AWS, go to the Using Elastic IP Addresses section in the Amazon Elastic Compute Cloud User Guide.

When Amazon EMR starts an Amazon EC2 instance, it uses a 2048-bit RSA key pair that you have named. Amazon EC2 stores the public key. Amazon EMR stores the private key and uses the private key to validate all requests.

The key pair ensures that only you can access your job flows. When you launch an instance using your key pair name, the public key becomes part of the instance metadata. This allows you to access the cluster node securely.

Although specifying the key pair is optional, we strongly recommend that you use key pairs. This key pair becomes associated with all of the nodes created to process your job flow. The key pair name creates a handle you can use to access the master node in the Hadoop cluster. With the key pair name, you can log in to the master node without using a password, enabling you to monitor the progress of your job flows. On the master node, you can retrieve detailed job flow processing status and statistics.

For more information on how to create and use an Amazon EC2 key pair with Amazon EMR, see "Creating an Amazon EC2 Key Pair" in the Getting Started Guide.

Amazon EMRrequires Amazon S3 buckets to hold the input and output data of your Hadoop processing. Amazon EMR uses the Amazon S3 Native File System for Hadoop processing. Amazon S3 uses the hostname method for accessing data, which places restrictions on bucket names used in Amazon EMR job flows.

For more information on creating Amazon S3 buckets for use with Amazon EMR, see Setting Up Your Environment to Run a Job Flow. For more information on Amazon S3 buckets, go to Working with Amazon S3 Buckets in the Amazon S3 Developer Guide.

Amazon Elastic MapReduce (Amazon EMR) supports Amazon S3 multipart upload through the AWS SDK for Java. Multipart upload lets you upload a single object as a set of parts. You can upload these object parts independently and in any order. If transmission of any part fails, you can retransmit that part without affecting other parts. After all parts of your object are uploaded, Amazon S3 assembles the parts and creates the object.

For more information about enabling multipart uploads with Amazon EMR, see Multipart Upload. For more information on Amazon S3 multipart uploads, go to Uploading Objects Using Multipart Upload in the Amazon S3 Developer Guide.