Running Job Flows on an Amazon VPC

To allow users to launch job flows in a Amazon VPC without the ability to modify the Amazon VPC

1. Create the Amazon VPC and launch Amazon EMR into a subnet of that Amazon VPC using an account with permissions to administer Amazon VPC and Amazon EMR.

2. Create a second user account with permissions to call the RunJobFlow, DescribeJobFlows, TerminateJobFlows, and AddJobFlowStep actions in the Amazon EMR API. You should also create an IAM policy that allows this user to launch EC2 instances. An example of this is shown below.

   {
     "Statement": [
       {
         "Action": [
   	     "ec2:AuthorizeSecurityGroupIngress",
   	     "ec2:CancelSpotInstanceRequests",
   	     "ec2:CreateSecurityGroup",
   	     "ec2:CreateTags",
   	     "ec2:DescribeAvailabilityZones",
   	     "ec2:DescribeInstances",
   	     "ec2:DescribeSecurityGroups",
   	     "ec2:DescribeSpotInstanceRequests",
   	     "ec2:ModifyImageAttribute",	
   	     "ec2:ModifyInstanceAttribute",
   	     "ec2:RequestSpotInstances",
   	     "ec2:RunInstances",
   	     "ec2:TerminateInstances"  
         ],
         "Effect": "Allow",
         "Resource": "*"
       },
       {
         "Action": [
            "elasticmapreduce:AddInstanceGroups",
            "elasticmapreduce:AddJobFlowSteps", 
            "elasticmapreduce:DescribeJobFlows",
            "elasticmapreduce:ModifyInstanceGroups",
            "elasticmapreduce:RunJobFlow"
   	     "elasticmapreduce:TerminateJobFlows"
         ],
         "Effect": "Allow",
         "Resource": "*"
       }
     
     }
   				

   Users with the IAM permissions set above will be able to launch job flows within the VPC subnet, but will not be able to change the Amazon VPC configuration.

   	Note
   	You should be cautious when granting ec2:TerminateInstances permissions because this action gives the recipient the ability to shut down any Amazon EC2 instance in the account, including those outside of Amazon EMR.

To create a VPC subnet to run Amazon EMR job flows

1. Sign in to the AWS Management Console and open the Amazon VPC console at https://console.aws.amazon.com/vpc/.

2. Create an Amazon VPC by clicking Get started creating a VPC. Make sure that the Region drop-down box is set to the same Region where you'll be running your job flow. In this example, we're creating a Amazon VPC in the US East (Virginia) Region.

   

3. Choose the VPC configuration by selecting one of the radio buttons.

   If the data used in the job flow is available on the Internet (eg: Amazon S3, Amazon RDS) select VPC with a Single Public Subnet Only.

   

   If the data used in the job flow is stored locally (eg: an Oracle database) select VPC with Public and Private subnets and Hardware VPN Access.

   

4. Confirm the Amazon VPC settings. In order to work with Amazon EMR the Amazon VPC must have both an Internet Gateway and a subnet.

   

5. A dialog box confirms that the Amazon VPC was successfully created. Click Close.

   

To find the Amazon VPC subnet identifier

• Click on Subnets in the navigation menu of the Amazon VPC console. The right pane displays information about the Amazon VPC, including its subnet identifier.

  

To launch a job flow on an Amazon VPC using the Amazon EMR console

1. In the Amazon EMR console, click the Create New Job Flow button.

   

2. Follow the instructions in the Create a New Job Flow wizard, selecting options that match the job flow you want to launch.

3. When you reach the ADVANCED OPTIONS page, choose the Amazon VPC subnet you created previously from the Amazon VPC Subnet Id drop-down box. If you have not created a Amazon VPC subnet, click on the Create a VPC link underneath the drop-down box to open the Amazon VPC console and create a Amazon VPC and subnet.

   

4. Continue the Create a Job Flow Wizard until it is complete and the job flow is launched. It will be launched within the Amazon VPC subnet you specified in Step 3.

To launch a job flow on an Amazon VPC using the CLI

• Once your Amazon VPC is configured, you can launch Amazon EMR job flows on it by using the --subnet argument and specifying the subnet address. This is illustrated in the following example, which creates a long-running job flow on the specified VPC subnet.

  elastic-mapreduce --create --alive --subnet subnet-identifier
  					

To launch a job flow on an Amazon VPC using the API

• Once your Amazon VPC is configured, you can launch Amazon EMR job flows on it by providing the VPC subnet identifier as the value for Ec2SubnetId, an optional String parameter on the JobFlowInstancesConfig structure.

  https://elasticmapreduce.amazonaws.com?
              Operation=RunJobFlow&
              Name=MyJobFlowName&
              LogUri=s3n%3A%2F%2Fmybucket%2Fsubdir&
              Instances.MasterInstanceType=m1.small&
              Instances.SlaveInstanceType=m1.small&
              Instances.InstanceCount=4&
              Instances.Ec2KeyName=myec2keyname&
              Instances.Placement.AvailabilityZone=us-east-1a&
              Instances.KeepJobFlowAliveWhenNoSteps=true&
              Instances.Ec2SubnetId=subnet-identifier&
              Steps.member.1.Name=MyStepName&
              Steps.member.1.ActionOnFailure=CONTINUE&
              Steps.member.1.HadoopJarStep.Jar=MyJarFile&
              Steps.member.1.HadoopJarStep.MainClass=MyMailClass&
              Steps.member.1.HadoopJarStep.Args.member.1=arg1&
              Steps.member.1.HadoopJarStep.Args.member.2=arg2&
              AWSAccessKeyId=AWS Access Key ID&
              SignatureVersion=2&
              SignatureMethod=HmacSHA256&
              Timestamp=2009-01-28T21%3A48%3A32.000Z&
              Signature=calculated value