Did this page help you? Yes No Tell us about it...

Changing the Number of Spot Instances in a Job Flow

With some restrictions, you can modify the number of Spot Instances in a job flow. For example, you cannot change the number of instances in the master instance group, it always has one instance. To prevent data loss, you can add, but not remove core nodes from an instance group. Task nodes do not store state, and so they can be added or removed from a running job flow.

You can only define an instance group as running as Spot Instances when it is created, so for example, if you launched the core instance group as on-demand, you would not be able to change its market type to Spot Instances later.

Note

It’s possible to automatically modify the number of slave nodes in a job flow between job flow steps. You just include a predefined step in your workflow that changes the number of requested Spot Instances. For example:

elastic-mapreduce --jobflow j-XXXXXXXXXXXX \
--jar s3://us-east-1.elasticmapreduce/libs/resize-job-flow/0.1/resize-job-flow.jar \
--args --modify-instance-group,task,--instance-count,4

The following examples show how to add task and core instance groups to a running job flow by increasing the number of instances in an instance group. For task nodes, you can also decrease the number of instances. The procedure is the same as shown in the examples that follow, but instead of specifying a larger number of instances than is currently running, you would specify a smaller number.

	Note
	If you are running a job flow that contains only a master node, you cannot add instance groups to that job flow. A job flow must have one or more core instances for you to be able to add or modify instance groups.

	Note
	Adding nodes to a running job flow is not currently supported in the Amazon EMR console.

CLI

To add Spot Instances to a running job flow

You can use the following command to add a task instance group of Spot Instances to a running job flow:

elastic-mapreduce --jobflow j-xxxxxxxxxxxxx --add-instance-group task  --instance-type m1.small \
--instance-count 5 --bid-price 0.05

CLI

To increase or decrease the number of Spot Instances in a running job flow

You can change the number of requested Spot Instances in a running job flow by calling the set-num-core-group-instances or set-num-task-group-instances on the command line. Note that you can only increase the number of CORE instances in your job flow while you can increase or decrease the number of TASK instances. Setting the number of TASK InstanceGroup instances to zero will remove all Spot Instances.
```
elastic-mapreduce --jobflow j-xxxxxxxxxxxxx --set-num-task-group-instances 5
					
```
This will change the number of requested TASK InstanceGroup instances to 5.

Java SDK

To add Spot Instances to a running job flow

To specify that an instance group should be launched as Spot Instances, set the withBidPrice and withMarket properties on the InstanceGroupConfig object that you instantiate for the instance group. The following code creates a task instance group of type m1.large with an instance count of 10. It specifies $0.35 as the maximum bid price, and will run as Spot Instances.
When you make the call to modify the instance group, pass this object instance in.
```
InstanceGroupConfig instanceGroupConfig = new InstanceGroupConfig()
	.withInstanceCount(10)
	.withInstanceRole(“TASK”)
	.withInstanceType(“m1.large”)
	.withMarket("SPOT")
	.withBidPrice(“0.35”);
					
```

API

To add Spot Instances to a running job flow

The following sample request increases the number of task nodes in the task instance group to eight and requests that they be launched as Spot Instances with an hourly bid price of .35.

The following is an example of the request you would send in to Amazon EMR.

Sample Request

https://elasticmapreduce.amazonaws.com?Operation=ModifyInstanceGroups
&InstanceGroups.member.1.InstanceGroupId=i-3UN6WX5RRO2AG 
&InstanceGroups.member.1.InstanceRequestCount=8
&InstanceGroups.member.1.InstanceRole=TASK
&InstanceGroups.member.1.Market=SPOT
&InstanceGroups.member.1.BidPrice=.35
&AuthParams

Sample Response

<ModifyInstanceGroupsResponse xmlns="http://elasticmapreduce.amazonaws.com/doc/2009-03-31">
   <ResponseMetadata>
      <RequestId>
         2690d7eb-ed86-11dd-9877-6fad448a8419
      </RequestId>
   </ResponseMetadata>
</ModifyInstanceGroupsResponse>

Document Conventions
Terms of Use

Did this page help you? Yes No Tell us about it...