Examples of Amazon S3 Export Manifest Options

The operations manifest option specifies a list of one or more export operations. AWS Import/Export exports keys in alphabetical order. That is, it carries out the operations in the order listed and then in the alphabetical order of the keys affected by each export operation. The files on the storage device remain in strict alphabetical order even if the data to export exceeds the capacity of the storage device; AWS Import/Export does not, for example, squeeze in an extra file or two that are out of order just because there is room on the storage device. This alphabetical order enables you to continue the export easily: create a new export job using the key of the last file exported as the beginMarker in your new export job.

The operations subfields prefix, beginMarker, and endMarker limit the data exported from a bucket.

	Note
	It’s possible to construct an export operation that includes no keys. For example, if you specify image for prefix and foo for endMarker, there would be no matching keys because foo comes alphabetically before image.

To export multiple subsets of data from one or more buckets, specify multiple exportBucket operations followed by appropriate export options. Each dash (-) after operations specifies a new set of data to export. Each entry requires an exportBucket manifest option. The following example shows four data sets to export from three different buckets: mybucket, lost, and threescompany.

operations:
  - exportBucket: mybucket
    prefix: images
    beginMarker: images/starwars/jedi.jpg
    endMarker: images/starwars/masterwindoo.jpg
  - exportBucket: mybucket
    prefix: html
  - exportBucket: lost
  - exportBucket: threescompany
    prefix: images
    endMarker: images/characters/mrfurley.jpg

AWS Import/Export uses the entire key when naming the objects exported to your storage device. For example, AWS Import/Export would map the key starwars.s3.amazonaws.com/images/jedi.jpg to the following file on your storage device /starwars/images/jedi.jpg. For Microsoft Windows, AWS Import/Export would map the same key to \starwars\images\jedi.jpg.

The following sections talk in depth about some of the export fields you use with operations.

The following export manifest options enable you to export a subset of the keys in a bucket (specified by exportBucket):

The manifest options beginMarker, and endMarker, and prefix can be used together or separately.

You can specify more than one bucket in your CreateJob request using the export manifest options operations and exportBucket. However, all buckets must be in the same Amazon S3 region. In addition, you must have read permissions on each bucket and each object.

The order in which AWS Import/Export exports keys is the order of the operations you specify in the operations manifest option, and then within each operation, by the alphabetical order of the keys to be exported. In the previous procedure, for example, AWS Import/Export exports all of the keys from mybucket in alphabetical order, and then from lostbucket in alphabetical order.

Note

AWS Import/Export exports keys in parallel, but the finished order of files on the storage device is by operation and then alphabetical. This functionality becomes important when the storage device has insufficient capacity to hold all of the data that needs to be exported. This ordering enables you to easily continue the export with a new export job using the key of the last key exported as your beginMarker in your new export job. For more information about how AWS Import/Export orders keys during export, see Operations Export Manifest Option.

AWS Import/Export gives you the ability to specify the location for the exported files on your storage device using the export manifest options operations and targetDirectory.

The recoveryDirectory manifest option specifies the directory that contains all of the files that AWS Import/Export had to rename because AWS Import/Export could not use the key for the filename on the storage device.

Files that are written to the recoveryDirectory directory start in a subdirectory named 0000, are numbered sequentially from 0 to 9999, and then placed in a new subdirectory named 0001, and so on until all the files are stored. If you set the value of recoveryDirectory to Export-Recovery/, you might see the following hierarchy of files and directories.

EXPORT-RECOVERY/
  0000/
    0000
    0001
    …
    9999
  0001/
  …
  9999/

This naming scheme repeats until one of the following conditions occurs:

	Note
	The default recoveryDirectory name is EXPORT-RECOVERY.

The logPrefix option defines a string AWS Import/Export inserts between the bucket name and log report name to prevent name collisions. The log file name always ends with the phrase export-log- followed by your JobId.

Example path and file name without logPrefix:

http://mybucket.s3.amazonaws.com/export-log-53TX4.

Example path and file name with logPrefix set to logs/:

http://mybucket.s3.amazonaws.com/logs/export-log-53TX4

	Note
	We do not include a slash (/) automatically. If you don't include the slash at the end of the value for logPrefix, the value is concatenated to the log file name. For example, if your logPrefix is taxonomy and your root directory contains the file phylum.jpg, your key would become taxonomyphylum.jpg instead of taxonomy/phylum.jpg.

logPrefix + export-log-JOBID cannot be longer than 1024 bytes. If it is, AWS Import/Export returns an InvalidManifestField error from the CreateJob action.

When returning your device to a U.S. address from a U.S. region bucket, you can use the serviceLevel manifest option to upgrade your shipping. The default serviceLevel option is standard shipping, which includes free ground domestic shipping. When you specify expeditedShipping, your device is delivered using two-day shipping for an additional fee. For information about return shipping fees and services, go to the AWS Import/Export Calculator.