         | Did this page help you? Yes No Tell us about it... |
Distributed Copy Using S3DistCp
Apache DistCp is an open-source tool you can use to copy large amounts of data. DistCp uses MapReduce to copy in a distributed manner—sharing the copy, error handling, recovery, and reporting tasks across several servers. S3DistCp is an extension of DistCp that is optimized to work with Amazon Web Services, particularly Amazon Simple Storage Service (Amazon S3). Using S3DistCp, you can efficiently copy large amounts of data from Amazon S3 into HDFS where it can be processed by your Amazon Elastic MapReduce (Amazon EMR) job flow. You can also use S3DistCp to copy data between Amazon S3 buckets or from HDFS to Amazon S3. For more information about the Apache DistCp open source project, go to http://hadoop.apache.org/common/docs/current/distcp.html.
S3DistCp is stored as a JAR file on Amazon S3. It's stored at s3://region.elasticmapreduce/libs/s3distcp/1.0/s3distcp.jar, where
region is one of the regions supported by Amazon EMR, such as us-west-1. For a complete list of the available regions,
go to the Regions and Endpoints documentation.
If S3DistCp is unable to copy some or all of the specified files, it fails and return a non-zero error code. If this occurs, S3DistCp does not clean up any partially copied files.
S3DistCp Options
When you call S3DistCp, you can specify options that change how it copies and compresses data. These are described in the following table.
| Option | Description | Required |
|---|---|---|
--src |
Location of the data to copy. This can be either an HDFS or Amazon S3 location.
Example: | Yes |
--dest |
Destination for the data. This can be either an HDFS or Amazon S3 location.
Example: | Yes |
--srcPattern |
A regular expression that filters the copy operation to a subset of the data at
Example: | No |
--groupBy |
A regular expression that causes S3DistCp to concatenate files that match the expression. For example, you could use this option to combine all the log files written in one hour into a single file. The concatenated filename is the value matched by the regular expression for the grouping.
When Example: | No |
--targetSize |
The size, in mebibytes (MiB), of the files to create based on the
If the files concatenated by
Example: | No |
--outputCodec |
Specifies the compression codec to use for the copied files. This can take the values:
Example: | No |
--deleteOnSuccess |
If the copy operation is successful, this option causes S3DistCp to delete the copied files from the source location. This is useful if you are copying output files, such as log files, from one location to another as a scheduled task, and you don't want to copy the same files twice.
Example: | No |
--disableMultipartUpload |
Disables the use of multipart upload. For more information about multipart upload, see Multipart Upload.
Example: | No |
--multipartUploadChunkSize |
The size, in MiB, of the multipart upload part size. By default it will use multipart upload when writing to Amazon S3. The default chunk size is 16 MiB.
Example: | No |
In addition to the options above, S3DistCp implements the Tool interface which means that it supports the generic options.
Adding S3DistCp as a Step in a Job Flow
You can call S3DistCp by adding it as a step in your job flow.
To add a S3DistCp step to a job flow using the CLI
Add a step to the job flow that calls S3DistCp, passing in the parameters that specify how S3DistCp should perform the copy operation. For more information about adding steps to a job flow, see Add Steps to a Job Flow.
The following example copys daemon logs from Amazon S3 to hdfs:///output.
In this CLI command:
--jobflowspecifies the job flow to add the copy step to.--jaris the location of the S3DistCp JAR file.--argsis a comma-separated list of the option name-value pairs to pass in to S3DistCp. For a complete list of the available options, see S3DistCp Options.
elastic-mapreduce --jobflow
jobflow-identifier--jar \ s3://region.elasticmapreduce/libs/s3distcp/1.0/s3distcp.jar \ --args 'S3DistCp-options'![[Note]](http://docs.amazonwebservices.com/docs-common/images/note.png)
Note The
--argsparameter is a convenient way to pass in several--argparameters at one time. It splits the string passed in on comma (,) characters to parse them into arguments. If the value of a S3DistCp option contains a comma, you cannot use--args, and must instead individual--argparameters. In the following example,--srcPatternis set to'.*[a-zA-Z,]+'using individual --arg parameters.elastic-mapreduce --jobflow
jobflow-identifier--jar \ s3://region.elasticmapreduce/libs/s3distcp/1.0/s3distcp.jar \ --arg --src --arg s3://myawsbucket/logs/j-3GY8JC4179IOJ/node/ \ --arg --dest --arg hdfs:///output \ --arg --srcPattern --arg '.*[a-zA-Z,]+'The following example illustrates how to copy log files stored in an Amazon S3 bucket into HDFS. In this example the
--srcPatternoption is used to limit the data copied to the daemon logs.elastic-mapreduce --jobflow j-3GY8JC4179IOJ --jar \ s3://us-east-1.elasticmapreduce/libs/s3distcp/1.0/s3distcp.jar \ --args '--src,s3://myawsbucket/logs/j-3GY8JC4179IOJ/node/,--dest,hdfs:///output,\ --srcPattern,.*daemons.*-hadoop-.*'
The next example loads Amazon CloudFront logs into HDFS. In the process it changes the compression format from Gzip (the CloudFront default) to LZO. This is useful because data compressed using LZO can be split into multiple maps as it is decompressed, so you don't have to wait until the compression is complete as you do with Gzip. This provides better performance when analyzing the data using Amazon EMR. This example also improves performance by using the regular expression specified in the
--groupByoption to combine all the logs for a given hour into a single file. Amazon EMR job flows are more efficient when processing a few, large, LZO-compressed files than when processing many, small, Gzip-compressed files.elastic-mapreduce --jobflow j-3GY8JC4179IOK --jar \ s3://us-east-1.elasticmapreduce/libs/s3distcp/1.0/s3distcp.jar \ --args '--src,s3://myawsbucket/cf,--dest,hdfs:///local,\ --groupBy,XABCD12345678.([0-9]+-[0-9]+-[0-9]+-[0-9]+).*,--targetSize,128,\ --outputCodec,lzo,--deleteOnSuccess'
Consider the case in which the preceding example is run over the following CloudFront log files.
s3://myawsbucket/cf/XABCD12345678.2012-02-23-01.HLUS3JKx.gz s3://myawsbucket/cf/XABCD12345678.2012-02-23-01.I9CNAZrg.gz s3://myawsbucket/cf/XABCD12345678.2012-02-23-02.YRRwERSA.gz s3://myawsbucket/cf/XABCD12345678.2012-02-23-02.dshVLXFE.gz s3://myawsbucket/cf/XABCD12345678.2012-02-23-02.LpLfuShd.gz
S3DistCp copies, concatenates, and compresses the files into the following, where the file name is determined by the match made by the regular expression.
hdfs:///local/2012-02-23-01.lzo hdfs:///local/2012-02-23-02.lzo
To add a S3DistCp step to a job flow using the API
Send a request similar to the following example, where the arguments specified by
Steps.member.1.HadoopJarStep.Args.memberalternate the argument name and value, where the value is URL encoded.https://elasticmapreduce.amazonaws.com? JobFlowId=
jobflow-identifier& Steps.member.1.Name="S3DistCp Step"& Steps.member.1.ActionOnFailure=CONTINUE& Steps.member.1.HadoopJarStep.Jar=s3://us-east-1.elasticmapreduce/libs/s3distcp/1.0/s3distcp.jar& Steps.member.1.HadoopJarStep.Args.member.1=--src& Steps.member.1.HadoopJarStep.Args.member.2=s3%3A%2F%2Fbucket%2Fcf& Steps.member.1.HadoopJarStep.Args.member.3=--dest& Steps.member.1.HadoopJarStep.Args.member.4=hdfs%3A%2F%2F%2Flocal& Steps.member.1.HadoopJarStep.Args.member.5=--srcPattern& Steps.member.1.HadoopJarStep.Args.member.6=.%2A%5Ba-zA-Z%5D%2B& Steps.member.1.HadoopJarStep.Args.member.7=--groupBy& Steps.member.1.HadoopJarStep.Args.member.8=.%2A%5Ba-zA-Z%5D%2B& Operation=AddJobFlowSteps& AWSAccessKeyId=access-key-identifier& SignatureVersion=2& SignatureMethod=HmacSHA256& Timestamp=2011-12-28T21%3A51%3A51.000Z& Signature=calculated-value
