Export, Import, Query, and Join Tables in Amazon DynamoDB Using Amazon EMR

In the following sections, you will learn how to use Amazon Elastic MapReduce (Amazon EMR) with a customized version of Hive that includes connectivity to Amazon DynamoDB to perform operations on data stored in Amazon DynamoDB, such as:

To perform each of the tasks above, you'll launch an Amazon EMR job flow, specify the location of the data in Amazon DynamoDB, and issue Hive commands to manipulate the data in Amazon DynamoDB.

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. Developers can create a database table and grow its request traffic or storage without limit. DynamoDB automatically spreads the data and traffic for the table over a sufficient number of servers to handle the request capacity specified by the customer and the amount of data stored, while maintaining consistent, fast performance. Using Amazon EMR and Hive you can quickly and efficiently process large amounts of data, such as data stored in Amazon DynamoDB. For more information about Amazon DynamoDB go to the Amazon DynamoDB Developer Guide.

Apache Hive is a software layer that you can use to query map reduce job flows using a simplifed, SQL-like query language called HiveQL. It runs on top of the Hadoop architecture. For more information about Hive and HiveQL, go to the HiveQL Language Manual.

There are several ways to launch an Amazon EMR job flow: you can use the Amazon EMR console, the Amazon EMR command line interface (CLI), or you can program your job flow using the AWS SDK or the API. You can also choose whether to run a Hive job flow interactively or from a script. In this section, we will show you how to launch an interactive Hive job flow from the Amazon EMR console and the CLI.

Using Hive interactively is a great way to test query performance and tune your application. Once you have established a set of Hive commands that will run on a regular basis, consider creating a Hive script that Amazon EMR can run for you. For more information about how to run Hive from a script, go to How to Create a Job Flow Using Hive.

Warning

Amazon EMR read or write operations on an Amazon DynamoDB table count against your established provisioned throughput, potentially increasing the frequency of provisioned throughput exceptions. For large requests, Amazon EMR implements retries with exponential backoff to manage the request load on the Amazon DynamoDB table. Running Amazon EMR jobs concurrently with other traffic may cause you to exceed the allocated provisioned throughput level. You can monitor this by checking the ThrottleRequests metric in Amazon CloudWatch. If the request load is too high, you can relaunch the job flow and set the Read Percent Setting or Write Percent Setting to a lower value to throttle the Amazon EMR operations. For information about Amazon DynamoDB throughput settings, see Provisioned Throughput.