Did this page help you? Yes No Tell us about it...

How to Create a Cascading Job Flow

Cascading is an open-source Java library that provides a Query API, a Query planner, and a job scheduler for creating and running Hadoop MapReduce applications. Applications developed with Cascading are compiled and packaged into standard Hadoop-compatible JAR files similar to other native Hadoop applications. A Cascading job flow is treated as a custom JAR in the Amazon EMR console.

This section covers the basics of creating a Cascading job flow using a custom JAR file in Amazon Elastic MapReduce (Amazon EMR). You'll step through how to create a job flow with either the Amazon EMR console, the CLI, or the Query API. Before you create your job flow you'll need to create objects and permissions; for more information see Setting Up Your Environment to Run a Job Flow.

The examples that follow are based on the Amazon EMR sample: LogAnalyzer for Amazon CloudFront. LogAnalyzer is implemented using Cascading. This sample generates usage reports containing total traffic volume, object popularity, a breakdown of traffic by client IP address, and edge location. Reports are formatted as tab-delimited text files, and saved to the Amazon S3 bucket that you specify.

In this example, the Java JAR is located in an Amazon S3 bucket at elasticmapreduce/samples/cloudfront/logprocessor.jar. The input data is located in the Amazon S3 bucket s3n://elasticmapreduce/samples/cloudfront/input. The output is saved to an Amazon S3 bucket you created as part of Setting Up Your Environment to Run a Job Flow.

Amazon EMR Console

This example describes how to use the Amazon EMR console to create a job flow using a custom JAR file.

To create a job flow using Cascading

Sign in to the AWS Management Console and open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/.
Click Create New Job Flow.
In the DEFINE JOB FLOW page, enter the following:
1. Enter a name in the Job Flow Name field.
  We recommended you use a descriptive name. It does not need to be unique.
2. Select Run your own application.
3. Select Custom JAR in the drop-down list.
4. Click Continue.

In the SPECIFY PARAMETERS page, enter values in the boxes using the following table as a guide, and then click Continue.

Field	Action
JAR Location*	Specify the URI where your script resides in Amazon S3. The value must be in the form `BucketName`/`path`/`ScriptName`.
JAR Arguments*	Enter a list of arguments (space-separated strings) to pass to the JAR file.

* Required parameter

In the CONFIGURE EC2 INSTANCES page, select the type and number of instances, using the following table as a guide, and then click Continue.

	Note
	Twenty is the default maximum number of nodes per AWS account. For example, if you have two job flows running, the total number of nodes running for both job flows must be 20 or less. If you need more than 20 nodes, you must submit a request to increase your Amazon EC2 instance limit. For more information, go to the Request to Increase Amazon EC2 Instance Limit Form.

Field	Action
Instance Count	Specify the number of nodes to use in the Hadoop cluster. There is always one master node in each job flow. You can specify the number of core and tasks nodes.
Instance Type	Specify the Amazon EC2 instance types to use as master, core, and task nodes. Valid types are m1.small (default), m1.large, m1.xlarge, c1.medium, c1.xlarge, m2.xlarge, m2.2xlarge, and m2.4xlarge, cc1.4xlarge, cg1.4xlarge.
Request Spot Instances	Click this checkbox to run master, core, or task nodes on Spot Instances. For more information, see Lowering Costs with Spot Instances

* Required parameter

In the ADVANCED OPTIONS page, set additional configuration options, using the following table as a guide, and then click Continue.

Field Action

Amazon EC2 Key Pair Optionally, specify a key pair that you created previously. For more information, see Create an Amazon EC2 Key Pair and PEM File. If you do not enter a value in this field, you cannot SSH into the master node.

Amazon VPC Subnet Id Optionally, specify a VPC subnet identifier to launch the job flow in an Amazon VPC. For more information, see Running Job Flows on an Amazon VPC.

Amazon S3 Log Path Optionally, specify a path in Amazon S3 to store the Amazon EMR log files. The value must be in the form BucketName/path. If you do not supply a location, Amazon EMR does not log any files.

Enable Debugging

Select Yes to store Amazon Elastic MapReduce-generated log files. You must enable debugging at this level if you want to store the log files generated by Amazon EMR.

If you select Yes, you must supply an Amazon S3 bucket name where Amazon Elastic MapReduce can upload your log files.

For more information, see Troubleshooting.

	Important
	You can enable debugging for a job flow only when you initially create the job flow.

Keep Alive Select Yes to cause the job flow to continue running when all processing is completed.

In the BOOTSTRAP ACTIONS page, select Proceed with no Bootstrap Actions, and then click Continue.
For more information about bootstrap actions, see Bootstrap Actions.
In the REVIEW page, review the information, edit as necessary to correct any of the values, and then click Create Job Flow when the information is correct.
After you click Create Job Flow your request is processed; when it succeeds, a message appears.
Click Close.
The Amazon EMR console shows the new job flow starting.

CLI

This example describes how to use the CLI to create a job flow using Cascading.

To create a job flow using Cascading

Use the information in the following table to create your job flow:

If you are using... Enter the following...

Linux or UNIX

If you are using...	Enter the following...
Linux or UNIX	& ./elastic-mapreduce --create --name "Test Cascading" \ --JAR elasticmapreduce/samples/cloudfront/logprocessor.jar \ --args "-input,s3n://elasticmapreduce/samples/cloudfront/input,\ -start,any,-end,2010-12-27-02 300,-output,\ s3n://`myawsbucket`/cloudfront/output/2010-12-27-02,\ -overallVolumeReport,-objectPopularityReport,-clientIPReport,\ -edgeLocationReport"
Microsoft Windows	c:\ ruby elastic-mapreduce --create --name "Test Cascading" --JAR elasticmapreduce/samples/cloudfront/logprocessor.jar --args \ "-input,s3n://elasticmapreduce/samples/cloudfront/input,\ -start,any,-end,2010-12-27-02,300,-output,\ s3n://`myawsbucket`/cloudfront/output/2010-12-27-02,\ -overallVolumeReport,-objectPopularityReport,-clientIPReport,\ -edgeLocationReport"

& ./elastic-mapreduce --create --name "Test Cascading" \
 --JAR elasticmapreduce/samples/cloudfront/logprocessor.jar \
--args "-input,s3n://elasticmapreduce/samples/cloudfront/input,\
-start,any,-end,2010-12-27-02 300,-output,\
s3n://myawsbucket/cloudfront/output/2010-12-27-02,\
-overallVolumeReport,-objectPopularityReport,-clientIPReport,\
-edgeLocationReport"

Microsoft Windows

c:\ ruby elastic-mapreduce --create --name "Test
Cascading" --JAR
elasticmapreduce/samples/cloudfront/logprocessor.jar --args \
"-input,s3n://elasticmapreduce/samples/cloudfront/input,\
-start,any,-end,2010-12-27-02,300,-output,\
s3n://myawsbucket/cloudfront/output/2010-12-27-02,\
-overallVolumeReport,-objectPopularityReport,-clientIPReport,\
-edgeLocationReport"

The output looks similar to the following.

Created job flow JobFlowID

By default, this command launches a job flow to run on a single-node cluster using an Amazon EC2 m1.small instance. Later, when your steps are running correctly on a small set of sample data, you can launch job flows to run on multiple nodes. You can specify the number of nodes and the type of instance to run with the --num-instances and --instance-type parameters, respectively.

API

This section describes the Amazon EMR API Query request parameters you need to create a Cascading job flow. For an explanation of the parameters unique to RunJobFlow, go to RunJobFlow in the Amazon Elastic MapReduce (Amazon EMR) API Reference. The response includes a <JobFlowID>, which you use in other Amazon EMR operations, such as when describing or terminating a job flow. For this reason, it is important to store job flow IDs.

The Args argument contains location information for your input data, output data, and args, as shown in the following example.

         "Name": "Cascading job flow",
"HadoopJarStep": 
{
"Jar":"s3n://elasticmapreduce/samples/cloudfront/logprocessor.jar",
"Args":
[
"-input",
"s3n://elasticmapreduce/samples/cloudfront/input",
"-start",
"any",
"-end",
"2010-12-27-02 300",
"-output",
"s3n://myawsbucket/cloudfront/output/2010-12-27-02",
"-overallVolumeReport",
"-objectPopularityReport",
"-clientIPReport",
"-edgeLocationReport"
]
}

	Note
	All paths are prefixed with their location. The prefix “s3://” refers to the s3n file system. If you use HDFS, prepend the path with `hdfs:///`. Make sure to use three slashes (///), as in `hdfs:///home/hadoop/sampleInput2/`.

Document Conventions
Terms of Use

Did this page help you? Yes No Tell us about it...