Pipelining and $AWSHistory

For AWS service calls that return collections, the objects within the collection are enumerated to the pipeline. Result objects that contain additional fields beyond the collection and which are not paging control fields have these fields added as Note properties for the calls. These Note properties are logged in the new $AWSHistory session variable, should you need to access this data. The $AWSHistory variable is described in the next section.

Examples

The following example returns a list of AWS Regions and your Amazon EC2 machine images (AMIs) in each Region.

PS > Get-AWSRegion | % { Echo $_.Name; Get-EC2Image -Owner self -Region $_ }

The following example stops all Amazon EC2 instances in the current default region.

PS > Get-EC2Instance | Stop-EC2Instance

Because collections enumerate to the pipeline, the output from a given cmdlet might be $null, a single object, or a collection. If it is a collection, you can use the .Count property to determine the size of the collection. However, the .Count property is not present when only a single object is emitted. If your script needs to determine, in a consistent way, how many objects were emitted, you can check the EmittedObjectsCount property of the last command value in $AWSHistory.

$AWSHistory

To better support pipelining, output from AWS cmdlets is not reshaped to include the service response and result instances as Note properties on the emitted collection object. Instead, for those calls that emit a single collection as output, the collection is now enumerated to the PowerShell pipeline. This means that the AWS SDK response and result data cannot exist in the pipe, because there is no containing collection object to which it can be attached.

Although most users probably won't need this data, it can be useful for diagnostic purposes, because you can see exactly what was sent to and received from the underlying AWS service calls made by the cmdlet.

Starting with version 1.1, this data and more is now available in a new shell variable named $AWSHistory. This variable maintains a record of AWS cmdlet invocations and the service responses that were received for each invocation. Optionally, this history can be configured to also record the service requests that each cmdlet made. Additional useful data, such as the overall execution time of the cmdlet, can also be obtained from each entry. For security reasons, requests and responses that contain sensitive data aren't recorded by default. However, the history can be configured to override this behavior if needed. For more information, see the Set-AWSHistoryConfiguration cmdlet shown below.

Each entry in the $AWSHistory.Commands list is of type AWSCmdletHistory. This type has the following useful members:

Note that the $AWSHistory variable is not created until an AWS cmdlet making a service call is used. It evaluates to $null until that time.

Set-AWSHistoryConfiguration

A cmdlet invocation can hold zero or more service request and response entries. To limit memory impact, the $AWSHistory list keeps a record of only the last five cmdlet executions by default; and for each, the last five service responses (and if enabled, last five service requests). You can change these default limits by running the Set-AWSHistoryConfiguration cmdlet. It allows you to both control the size of the list, and whether service requests are also logged:

PS > Set-AWSHistoryConfiguration -MaxCmdletHistory <value> -MaxServiceCallHistory <value> -RecordServiceRequests -IncludeSensitiveData

All parameters are optional.

The MaxCmdletHistory parameter sets the maximum number of cmdlets that can be tracked at any time. A value of 0 turns off recording of AWS cmdlet activity. The MaxServiceCallHistory parameter sets the maximum number of service responses (and/or requests) that are tracked for each cmdlet. The RecordServiceRequests parameter, if specified, turns on tracking of service requests for each cmdlet. The IncludeSensitiveData parameter, if specified, turns on tracking of service responses and requests (if tracked) that contain sensitive data for each cmdlet.

If run with no parameters, Set-AWSHistoryConfiguration simply turns off any prior request recording, leaving the current list sizes unchanged.

To clear all entries in the current history list, run the Clear-AWSHistory cmdlet.

$AWSHistory Examples

Enumerate the details of the AWS cmdlets that are being held in the list to the pipeline.

PS > $AWSHistory.Commands

Access the details of the last AWS cmdlet that was run:

PS > $AWSHistory.LastCommand

Access the details of the last service response received by the last AWS cmdlet that was run. If an AWS cmdlet is paging output, it may make multiple service calls to obtain either all data or the maximum amount of data (determined by parameters on the cmdlet).

PS > $AWSHistory.LastServiceResponse

Access the details of the last request made (again, a cmdlet may make more than one request if it is paging on the user's behalf). Yields $null unless service request tracing is enabled.

PS > $AWSHistory.LastServiceRequest

Automatic Page-to-Completion for Operations that Return Multiple Pages

For service APIs that impose a default maximum object return count for a given call or that support pageable result sets, all cmdlets "page-to-completion" by default. Each cmdlet makes as many calls as necessary on your behalf to return the complete data set to the pipeline.

In the following example, which uses Get-S3Object, the $c variable contains S3Object instances for every key in the bucket test, potentially a very large data set.

PS > $c = Get-S3Object -BucketName test

If you want to retain control of the amount of data returned, you can use parameters on the individual cmdlets (for example, MaxKey on Get-S3Object) or you can explicitly handle paging yourself by using a combination of paging parameters on the cmdlets, and data placed in the $AWSHistory variable to get the service's next token data. The following example uses the MaxKeys parameter to limit the number of S3Object instances returned to no more than the first 500 found in the bucket.

PS > $c = Get-S3Object -BucketName test -MaxKey 500

To know if more data was available but not returned, use the $AWSHistory session variable entry that recorded the service calls made by the cmdlet.

If the following expression evaluates to $true, you can find the next marker for the next set of results using $AWSHistory.LastServiceResponse.NextMarker:

$AWSHistory.LastServiceResponse -ne $null && $AWSHistory.LastServiceResponse.IsTruncated

To manually control paging with Get-S3Object, use a combination of the MaxKey and Marker parameters for the cmdlet and the IsTruncated/NextMarker notes on the last recorded response. In the following example, the variable $c contains up to a maximum of 500 S3Object instances for the next 500 objects that are found in the bucket after the start of the specified key prefix marker.

PS > $c = Get-S3Object -BucketName test -MaxKey 500 -Marker $AWSHistory.LastServiceResponse.NextMarker