Question: How do I troubleshoot issues with UNLOAD in Amazon Redshift?

"I'm trying to unload data from my Amazon Redshift cluster to Amazon Simple Storage Service (Amazon S3). However, I'm getting an error. How I troubleshoot this?"

Answer: "I'm trying to unload data from my Amazon Redshift cluster to Amazon Simple Storage Service (Amazon S3). However, I'm getting an error. How I troubleshoot this?Short descriptionWhen unloading data from your Amazon Redshift cluster to your Amazon S3 bucket, you might encounter the following errors:DB user is not authorized to assume the AWS Identity and Access Management (IAM) Role errorerror:  User arn:aws:redshift:us-west-2:<account-id>:dbuser:<cluster-identifier>/<dbuser username> is not authorized to assume IAM Role arn:aws:iam::<account-id>:role/<Role name>403 Access Denied error[Amazon](500310) Invalid operation: S3ServiceException:Access Denied,Status 403,Error AccessDenied,Note: Use the UNLOAD command with the SELECT statement when unloading data to your S3 bucket. Unload the text data in either a delimited or fixed-width format (regardless of the data format used while being loaded). You can also specify whether a compressed gzip file should be filed.Specified unload destination on S3 is not emptyERROR: Specified unload destination on S3 is not empty. Consider using a different bucket / prefix, manually removing the target files in S3, or using the ALLOWOVERWRITE option.This error might happen when you're trying to run the same UNLOAD command and unloading files in the same folder where data files are already present.For example, you get this error if you run the following command twice:unload ('select * from test_unload')   to 's3://testbucket/unload/test_unload_file1' iam_role 'arn:aws:iam::0123456789:role/redshift_role';ResolutionDB user is not authorized to assume the AWS IAM Role errorIf the database user isn't authorized to assume the IAM role, then check the following:Verify that the IAM role is associated with your Amazon Redshift cluster.Verify that there are no trailing spaces in the IAM role used in the UNLOAD command.Verify that the IAM role assigned to the Amazon Redshift cluster is using the correct trust relationship.403 Access Denied errorIf you receive a 403 Access Denied error from your S3 bucket, confirm that the proper permissions are granted for your S3 API operations:"Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "s3:Get*",               "s3:List*",        "s3:PutObject"      ],      "Resource": [        "arn:aws:s3:::your_bucket",        "arn:aws:s3:::your_bucket/*"      ]    }  ]}If you're using server-side encryption with S3-managed encryption keys, then your S3 bucket encrypts each of its objects with a unique key. As an additional safeguard, the key itself is also encrypted with a root key that is regularly rotated.To encrypt the unloaded data at rest, verify that the s3:x-amz-server-side-encryption parameter in your S3 bucket policy is set to "true":"Condition": {   "Null": {           "s3:x-amz-server-side-encryption": "true"           }Confirm that KMS_KEY_ID is encrypted so that you can UNLOAD the encrypted data to your S3 bucket:unload ('select * from a') to 's3://mybucket/test/varchar/' iam_role 'arn:aws:iam::xxxxxx:role/RedshiftS3Access' ALLOWOVERWRITE CSVKMS_KEY_ID '11111111111111111111111111'encrypted;If your S3 bucket is encrypted with a root key from AWS Key Management Service (AWS KMS), then add these permissions to your IAM policy:"Action": ["kms:Encrypt","kms:Decrypt","kms:ReEncrypt*","kms:GenerateDataKey*","kms:DescribeKey"]Specified unload destination on S3 is not emptyThe specified unload destination on S3 is not empty error happens if you're trying to UNLOAD files in a folder where the same file name exists.Choose one of the following options to resolve the error message:Create a new Amazon S3 bucket or use a different S3 bucketChange the S3 key prefixClean up the files present in the destination Amazon S3 bucketUse the ALLOWOVERWRITE option with the UNLOAD commandCreate a new Amazon S3 bucket or use a different S3 bucket1.    Create a new S3 bucket or identify a different existing S3 bucket.2.    Update the bucket policy of your S3 bucket for your new or existing bucket. The following example policy must be added as a permissions policy:{    "Version": "2012-10-17",    "Id": "ExamplePolicy01",    "Statement": [        {            "Sid": "ExampleStatement01",            "Effect": "Allow",            "Principal": {                "AWS": "arn:aws:iam::0123456789:user/<UserName>"            },            "Action": [                "s3:GetObject",                "s3:GetBucketLocation",                "s3:ListBucket"            ],            "Resource": [                "arn:aws:s3:::testbucket/*",                "arn:aws:s3:::testbucket",                "arn:aws:s3:::testbucket2/*",  --> Additional permission for new bucket                "arn:aws:s3:::testbucket2"                   ]        }    ]}3.    Use the new bucket path in the UNLOAD command. In the following example, s3://testbucket2/unload/test_unload_file1 is the new bucket path:iam_role 'arn:aws:iam::0123456789:role/redshift_role';unload ('select * from test_unload')   to 's3://testbucket2/unload/test_unload_file1'Change the S3 key prefixChange the S3 Bucket Key prefix in the S3 path when using the UNLOAD command. In the following example, unload2 is the changed prefix:iam_role 'arn:aws:iam::0123456789:role/redshift_role';unload ('select * from test_unload')   to 's3://testbucket/unload2/test_unload_file1'For more information, see Organizing objects using prefixes.Clean up the files present in the destination Amazon S3 bucketUse the CLEANPATH option to remove existing files located in the Amazon S3 path specified in the TO clause before unloading files to the specified location. If you include the PARTITION BY clause, existing files are removed from only the partition folders. This allows the Amazon S3 bucket to receive new files generated by the UNLOAD operation.Important: Files removed using the CLEANPATH option are permanently deleted and can't be recovered.To use the CLEANPATH option, you must have the s3:DeleteObject permission on the Amazon S3 bucket. The following example policy must be added as a permissions policy:{    "Version": "2012-10-17",    "Id": "ExamplePolicy01",    "Statement": [        {            "Sid": "ExampleStatement01",            "Effect": "Allow",            "Principal": {                "AWS": "arn:aws:iam::0123456789:user/<UserName>"            },            "Action": [                "s3:GetObject",                "s3:GetBucketLocation",                "s3:ListBucket",                "s3:DeleteObject"  --> Additional Action added for delete object            ],            "Resource": [                "arn:aws:s3:::testbucket/*",                "arn:aws:s3:::testbucket"            ]        }    ]}Note: You can't specify the CLEANPATH option if you specify the ALLOWOVERWRITE option.Use the ALLOWOVERWRITE option with the UNLOAD commandThe UNLOAD command fails if it finds files that it might possibly overwrite. If ALLOWOVERWRITE is specified, UNLOAD overwrites existing files, including the manifest file. In the following example, the previous files are overwritten and the bucket is updated with the new records.iam_role 'arn:aws:iam::0123456789:role/redshift_role' allowoverwrite;unload ('select * from test_unload')   to 's3://testbucket/unload/test_unload_file1'To use the ALLOWOVERWRITE option, you must have the s3:PutObject permission on the Amazon S3 bucket. The following example policy must be added as a permissions policy:{    "Version": "2012-10-17",    "Id": "ExamplePolicy01",    "Statement": [        {            "Sid": "ExampleStatement01",            "Effect": "Allow",            "Principal": {                "AWS": "arn:aws:iam::0123456789:user/<UserName>"            },            "Action": [                "s3:GetObject",                "s3:GetBucketLocation",                "s3:ListBucket",                "s3:PutObject"  --> Additional Action added for overwriting objects            ],            "Resource": [                "arn:aws:s3:::testbucket/*",                "arn:aws:s3:::testbucket"            ]        }    ]}Related informationAuthorizing Amazon Redshift to access other AWS services on your behalfHow to prevent uploads of unencrypted objects to Amazon S3Queries appear to hang and sometimes fail to reach the clusterUnloading encrypted data filesPolicies and permissions in Amazon S3Follow"


Question: "When I try to access an S3 object, I get the error "Request has expired." Why?"

"I'm trying to access an object in Amazon Simple Storage Service (Amazon S3), but I'm getting the error message "Request has expired." How can I fix this?"

Answer: "I'm trying to access an object in Amazon Simple Storage Service (Amazon S3), but I'm getting the error message "Request has expired." How can I fix this?ResolutionAmazon S3 returns this error message when you try to access a presigned URL past the specified expiration date of the URL.For example, in the following response, the expiration date of the presigned URL is October 28, 2022. You'll get an error message if you try to access this URL after that date.<Error>    AccessDenied    <Message>Request has expired</Message>    <Expires>2022-10-28T07:13:14Z</Expires>    <ServerTime>2022-10-28T20:03:02Z</ServerTime>    <RequestId>87E1D2CFAAA7F9A6</RequestId>    <HostId>    A9BEluTV2hk3ltdFkixvQFa/yUBfUSgDjptwphKze+jXR6tYbpHCx8Z7y6WTfxu3rS4cGk5/WTQ=    </HostId></Error>To resolve this issue, the owner of the object must generate a new presigned URL with a new expiration date. If you own that object, see presign or Sharing objects using presigned URLs for more information.Note: The presigned URL might expire before the configured expiration date. This is because the AWS Identity and Access Management (IAM) entity STS token has expired before the presigned URL expiration date. For more information about STS session duration, see Comparing the AWS STS API operations.Related informationGenerating a presigned URL to upload an objectFollow"


Question: How can I find out which user launched an EC2 instance in my account?

I want to identify the user that launched an Amazon Elastic Compute Cloud (Amazon EC2) instance in my account. How can I view this information?

Answer: "I want to identify the user that launched an Amazon Elastic Compute Cloud (Amazon EC2) instance in my account. How can I view this information?ResolutionAWS CloudTrail allows you to view the last 90 days of recorded API activity and events in an Amazon Web Services (AWS) Region. When viewing your CloudTrail event history, you can apply one attribute filter and a time range filter to narrow the results.Before you begin, identify the instance ID for the instance that you want to investigate. Follow these steps:Open the Amazon EC2 console, and then choose Instances from the navigation pane.Select the instance, and note the Instance ID from the Detailstab.To view CloudTrail events and identify the user information associated with the instance launch, follow these steps:Open the CloudTrail console, and then choose Event history from the navigation pane.For Filter, choose Resource name from the dropdown menu.For Enter resource name, enter the instance ID.(Optional) For Time range, select a time range.In the list of results, look for RunInstances in the Event name column, and then select that result to expand the event details.Choose View event to view the event details, including information about who initiated the instance launch request.Related informationViewing CloudTrail events in the CloudTrail consoleFollow"


Question: How do I create a WorkSpaces image?

I want to create a custom Windows or Linux Amazon WorkSpaces image. How can I do that?

Answer: "I want to create a custom Windows or Linux Amazon WorkSpaces image. How can I do that?ResolutionBefore you create a WorkSpaces image, review the requirements and best practices for creating custom images:Requirements to create Windows custom imagesRequirements to create Amazon Linux custom imagesBest practicesCreate a custom WorkSpaces imageRun the Image Checker.Create an image from WorkSpaces, and then create a custom bundle from the image. For instructions, see Create a custom image and custom bundle.Related informationAdminister your WorkSpacesCreate a custom WorkSpaces image and bundleFollow"


Question: How do I update the AWS CloudFormation cfn-response module for AWS Lambda functions running on Python 2.7/3.6/3.7?

I want to update the AWS CloudFormation cfn-response module for AWS Lambda functions running on Python 2.7/3.6/3.7.

Answer: "I want to update the AWS CloudFormation cfn-response module for AWS Lambda functions running on Python 2.7/3.6/3.7.ResolutionNote: The following steps are applicable only to Lambda functions running on Python 2.7/3.6/3.7. The following commands are applicable to Linux and macOS environments. Syntax may vary on Windows PowerShell.Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.1.    To find the stacks that contain custom resources, run the following command:aws cloudformation list-stacks --region us-east-1 | grep -oE 'arn:[^"]+' | while read arn; do aws cloudformation list-stack-resources --stack-name $arn --region us-east-1 | grep -E '(Custom::)|(::CustomResource)' | awk '{print $2}' | while read resource; do if [[ -n $resource ]]; then echo $arn; echo $resource; fi; done; doneYou should see output similar to the following example output:arn:aws:cloudformation:us-east-1:123456789012:stack/TestStack/3497b950-55f1-11eb-aad4-124a026c8667"ResourceType": "AWS::CloudFormation::CustomResource",2.    To find the Lambda function associated with the custom resource, run the following command to check the custom resource’s ServiceToken property from the stack’s template:aws cloudformation get-template --stack-name TestStack | jq -r .TemplateBodyNote: The command in step 2 previews the stack’s template by using the jq option (from the jq website) to format the response.You should see output similar to the following example output:Resources:  MyCustomResource:    Type: AWS::CloudFormation::CustomResource    Properties:      ServiceToken: !GetAtt MyFunction.Arn      Name: "John"  MyFunction:    Type: AWS::Lambda::Function    Properties:      Handler: index.handler      Role: !GetAtt MyRole.Arn      Runtime: python3.7      Code:        ZipFile: |          import cfnresponse          def handler(event, context):            responseData = {'Message': 'Hello {}!'.format(event['ResourceProperties']['Name'])}            cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID")  MyRole:    Type: AWS::IAM::Role    Properties:      AssumeRolePolicyDocument:        Statement:          - Effect: Allow            Principal:              Service:                - lambda.amazonaws.com            Action:              - sts:AssumeRole      ManagedPolicyArns:        - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"Outputs:  Result:    Value: !GetAtt MyCustomResource.MessageNote: The template that you get from the output for step 2 is an example of a minimal template for a Lambda-backed custom resource. The property ServiceToken: !GetAtt MyFunction.Arn is in the MyCustomResource section. The value resolved by the !GetAtt MyFunction**.Arn** of the ServiceToken property is either the Amazon Resource Name (ARN) of the Amazon Simple Notification Service (Amazon SNS) topic or the Lambda function.3.    In the template from step 2, identify where your Lambda function is defined.If your Lambda function is in the same stack as the custom resource, skip to step 4. For example, the Fn::GetAtt function in step 2 shows that the Lambda function is defined in the same template as the custom resource.If the ServiceToken property points to a hardcoded ARN, then the Lambda function could be in another stack. If the ServiceToken  property is resolved through Fn::Import, then use the list-exports API in AWS CloudFormation to look up the value. For example:aws cloudformation list-exports --region us-east-1{    "Exports": [        {            "ExportingStackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/SomeOtherStack/481dc040-b283-11e9-b1bd-12d607a4fd1c",            "Value": "arn:aws:lambda:us-east-1:123456789012:function:SomeOtherStack-MyFunction-5ZE2CQO8RAA9",            "Name": "MyExport"        }    ]}Then, check for function tags located in a separate stack by using list-tags to locate the AWS CloudFormation stack ARN. For example:aws lambda list-tags --resource arn:aws:lambda:us-east-1:123456789012:function:TestStack-MyFunction-5ZE2CQO8RAA9 | grep stack-idYou receive output similar to the following:"aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-1:123456789012:stack/TestStack/3497b950-55f1-11eb-aad4-124a026c8667"Note: You can also find function tags in the AWS Lambda console.4.    To allow AWS CloudFormation to load the latest cfn-response module in your Lambda function, update the inline source code of your Lambda function. For example:Code:        ZipFile: |          import cfnresponse          def handler(event, context):            responseData = {'Message': 'Hello {}!'.format(event['ResourceProperties']['Name'])}            cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID")Note: See step 2 for an example template that has a Lambda function with inline source code.Now, the following cfn-response module code example is loaded by AWS CloudFormation into your Lambda function. For example:from botocore.vendored import requestsimport json SUCCESS = "SUCCESS"FAILED = "FAILED" def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False):    responseUrl = event['ResponseURL']     print(responseUrl)     responseBody = {}    responseBody['Status'] = responseStatus    responseBody['Reason'] = 'See the details in CloudWatch Log Stream: ' + context.log_stream_name    responseBody['PhysicalResourceId'] = physicalResourceId or context.log_stream_name    responseBody['StackId'] = event['StackId']    responseBody['RequestId'] = event['RequestId']    responseBody['LogicalResourceId'] = event['LogicalResourceId']    responseBody['NoEcho'] = noEcho    responseBody['Data'] = responseDataNote: For more information, see the code examples in the "Module source code" section of the cfn-response module.The cfn-response module code example uses botocore.requests in the Lambda function’s deployment package.To update the cfn-response module to the latest version that uses urllib3, update the function’s inline code in the AWS CloudFormation template. Do this by adding a comment to the inline Lambda function's code. For example:ZipFile: |           import cfnresponse           def handler(event, context):+            # This comment was added to force an update on this function's code             responseData = {'Message': 'Hello {}!'.format(event['ResourceProperties']['Name'])}             cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID")   MyRole:5.    Save any changes to the template that contains your Lambda function.6.    Update your stack.The cfn-response module is modified after the stack is finished updating.Note: If your function's code resides in an Amazon Simple Storage Service (Amazon S3) bucket or Amazon Elastic Container Registry (Amazon ECR) image, you must update the module yourself to include the version with urllib3. To get the source code for the latest version of the cfn-response module, see cfn-response module.Note: If a new Python or JavaScript runtime introduces a breaking change, you must update the cfn-response module. Instead of updating the ZipFile again, you can automatically attach the newest version of the cfn-response module whenever a function’s Runtime property is updated.Follow"


Question: "How do I use CloudTrail to search for actions on a specific resource, such as who deleted an EBS volume on an EC2 instance?"

"How do I use AWS CloudTrail to search a given resource for information, such as all event names for a resource, or who deleted an Amazon Elastic Block Store (Amazon EBS) volume or snapshot on an Amazon Elastic Compute Cloud (Amazon EC2) instance?"

Answer: "How do I use AWS CloudTrail to search a given resource for information, such as all event names for a resource, or who deleted an Amazon Elastic Block Store (Amazon EBS) volume or snapshot on an Amazon Elastic Compute Cloud (Amazon EC2) instance?Short descriptionYou can use AWS CloudTrail to log and monitor account activity across your AWS infrastructure. To do so, you must have CloudTrail turned on, and enable logging in the AWS Region where the EC2 resource you want to query resides.ImportantThe rate of lookup requests is limited to one request per second per account. If this limit is exceeded, a throttling error occurs.Events that occurred during a selected time range aren't available for lookup if CloudTrail logging wasn't enabled when the events occurred.ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.The following examples are run from the AWS CLI. Note that most of these commands use jq.Use yum to install jq in Amazon Linux.yum install jqNote: There are jq install options for other platforms.List all event names for a given resourceThe following example uses an EBS volume ID to get a list of recent events for any API action. Replace the AttributeValue of vol-0f59a355c2example and the --region of us-east-1 with your own values.$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=vol-0f59a355c2example --query 'Events[].{username:Username,time:EventTime,event:EventName,eventid:EventId,accesskey:AccessKeyId,resource:(Resources[0].ResourceName)}' --output table --region us-east-1Note: CloudTrail might take longer than expected to populate recent calls.The same example can be run using different resource IDs, such as an EBS snapshot. Replace the AttributeValue of snap-0993c0d9a8example and the --region of us-east-1 with your values.$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=snap-0993c0d9a8example --query 'Events[].{username:Username,time:EventTime,event:EventName,eventid:EventId,accesskey:AccessKeyId,resource:(Resources[0].ResourceName)}' --output table --region us-east-1List events for a specific API action for a given resourceThe following example shows how to get a list of recent events for the DeleteVolume API action for an EBS volume. Replace the AttributeValue of vol-0f59a355c2example with your volume ID.$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=vol-0f59a355c2example --query 'Events[].{username:Username,time:EventTime,event:EventName,eventid:EventId,accesskey:AccessKeyId,resource:(Resources[0].ResourceName)}' --output json --region us-east-1 | jq -r '.[] | select(.event == "DeleteVolume")'The following is an example of the resulting output:{  "username": "jdoe",  "eventid": "e3ec4051-9999-4e87-9999-9cc72example",  "resource": "vol-0f59a355c2example",  "accesskey": "ASIAXUZVKEUACEXAMPLE",  "time": 1550191014,  "event": "DeleteVolume"}List a specific event name for all resourcesThe following example lists deleted EBS volumes by using the DeleteVolume event name as a filter. Replace --region us-east-1 with your Region.$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteVolume --query 'Events[].{username:Username,time:EventTime,event:EventName,eventid:EventId,accesskey:AccessKeyId,resource:(Resources[0].ResourceName)}' --output table --region us-east-1List terminated EC2 instances for all resourcesThe following example lists recent EC2 TerminateInstances. Replace –region us-east-1 with your Region.$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances --query 'Events[].{username:Username,time:EventTime,event:EventName,eventid:EventId,accesskey:AccessKeyId,resource:(Resources[0].ResourceName)}' --output table --region us-east-1View details for a specific event idAfter you find an event ID, you can view the details about this event. Replace the AttributeValue of 0840b15f-75b5-4082-a194-86e15example with your event ID and replace --region us-east-1 with your Region.$ aws cloudtrail lookup-events --query "Events[0].CloudTrailEvent" --output text --lookup-attribute AttributeKey=EventId,AttributeValue=0840b15f-75b5-4082-a194-86e15example --region us-east-1 | jq -r '.'The following is the example output:{  "eventVersion": "1.05",  "userIdentity": {    "type": "AssumedRole",    "principalId": "AROAJ3THTCWDOKEXAMPLE:jdoe",    "arn": "arn:aws:sts::52570EXAMPLE:assumed-role/Admin/jdoe",    "accountId": "52570example",    "accessKeyId": "ASIAXUZVKEUAKEXAMPLE",    "sessionContext": {      "attributes": {        "mfaAuthenticated": "false",        "creationDate": "2019-02-14T23:55:15Z"      },      "sessionIssuer": {        "type": "Role",        "principalId": "AROAJ3THTCWDOKEXAMPLE",        "arn": "arn:aws:iam::52570EXAMPLE:role/Admin",        "accountId": "52570EXAMPLE",        "userName": "Admin"      }    }  },  "eventTime": "2019-02-15T00:48:05Z",  "eventSource": "ec2.amazonaws.com",  "eventName": "DeleteVolume",  "awsRegion": "us-east-1",  "sourceIPAddress": "999.999.999.999",  "userAgent": "aws-cli/1.16.999 Python/2.7.15 Darwin/17.7.0 botocore/1.12.91",  "requestParameters": {    "volumeId": "vol-0c50d65c6eexample"  },  "responseElements": {    "_return": true  },  "requestID": "a8a43ccd-736d-4b09-ba75-24b9cexample",  "eventID": "0840b15f-75b5-4082-a194-86e15example",  "eventType": "AwsApiCall",  "recipientAccountId": "52570EXAMPLE"}Specify a date rangeYou can use the --start-time and --end-time parameters to specify a date range of events. The listed events occurred after the start-time, as well as up to, and including, the end-time.The default start time is the earliest date that data is available within the last 90 days. The default end time is the time of the event that occurred closest to the current time.If the specified start time is after the specified end time, an InvalidTimeRangeException error is returned.$ aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteVolume --query 'Events[].{username:Username,time:EventTime,event:EventName,eventid:EventId,accesskey:AccessKeyId,resource:(Resources[0].ResourceName)}' --output table --region us-east-1 --start-time 2019-01-01T13:00Z --end-time 2019-03-01T14:00ZNote: Choose your preferred timestamp from the list at Valid timestamp formats.Related informationAWS CLI command referenceAmazon EC2 API referenceFollow"


Question: How do I use environment variables from an Elastic Beanstalk instance shell?

I want to use environment variables from an AWS Elastic Beanstalk instance shell.

Answer: "I want to use environment variables from an AWS Elastic Beanstalk instance shell.Short descriptionIn Elastic Beanstalk, environment variables aren't exposed to the operating system.To use a utility with environment variables from the operating system when the environment variables are set as Elastic Beanstalk environment variables, you must use a .ebextension that exports the environment variables to operating system variables.The following resolution shows you how to:1.    Use the get-config utility to read your Elastic Beanstalk environment variables.2.    Use the jq utility (from the jq website) to transform the get-config output from JSON to shell variables.3.    Save the variables on the /etc/profile.d/sh.local file. This file is read by Bash to export variables to the operating system.Resolution1.    Create a .ebextension file in your application source bundle and include the following:commands:    setvars:        command: /opt/elasticbeanstalk/bin/get-config environment | jq -r 'to_entries | .[] | "export \(.key)=\"\(.value)\""' > /etc/profile.d/sh.localpackages:    yum:        jq: []Note: The configuration file in step 1 is called setvars.config.2.    Save the .ebextension file, and then deploy it to your Elastic Beanstalk environment.3.    To test if the variables are being exported, connect to your instance using SSH, and then run the following command. Before testing, close any existing sessions and then reconnect using SSH.env | grep VARIABLE_NAMEImportant: For step 3, set VARIABLE_NAME to a variable defined in your environment.The output shows if your environment variables are set correctly. In the following example output, a variable called RDS_PORT is defined in the Elastic Beanstalk environment.$ env|grep RDS_PORTRDS_PORT=3306Note: Because you're using the commands on the .ebextension, you can update the sh.local file only with a new deployment. If you add or change a variable in the environment, then you must create a new deployment before the variable is exported to the operating system.Follow"


Question: Why is my AWS OpsWorks Stacks instance not starting and stuck in the "setup_failed" state?

"One of my Amazon Elastic Compute Cloud (Amazon EC2) instances that's managed by AWS OpsWorks Stacks is stuck in the "setup_failed" state. Why won't my OpsWorks Stacks instance enter the "online" state, and how do I troubleshoot the issue?"

Answer: "One of my Amazon Elastic Compute Cloud (Amazon EC2) instances that's managed by AWS OpsWorks Stacks is stuck in the "setup_failed" state. Why won't my OpsWorks Stacks instance enter the "online" state, and how do I troubleshoot the issue?Short descriptionAn OpsWorks Stacks instance can enter the setup_failed state and not start during a setup lifecycle event for a variety of reasons. However, the problem is usually because one of the following resources is misconfigured:Custom recipesApplicationsCustom Berkshelf cookbooksCustom JSON that's used to override stack attributesFor more information on how to troubleshoot instances in the setup_failed state, see Logging in to a failed instance in the OpsWorks User Guide.Note: For instances stuck in the start_failed state, see Why is my AWS OpsWorks Stacks instance not starting and stuck in the "start_failed" state?ResolutionVerify that your instance's Amazon Elastic Block Store (Amazon EBS) volume is configured correctlyReview Registering Amazon EBS volumes with a stack and make sure that your instance meets all of the requirements listed.Verify that your custom recipe is configured correctlyReview your instance's Chef logs to identify what's causing the issue. Then, debug your custom recipe based on the error message that's recorded in the logs.For more information, see Recipes in the OpsWorks User Guide.Verify that your instance has enough memory to allocate during the setup lifecycle eventFor instructions on how to review your instance's memory metrics, see Monitoring Stacks using Amazon CloudWatch.Follow"


Question: My Amazon S3 bucket has default encryption using a custom AWS KMS key. How can I allow users to download from and upload to the bucket?

I set up my Amazon Simple Storage Service (Amazon S3) bucket to use default encryption with a custom AWS Key Management Service (AWS KMS) key. I want an AWS Identity and Access Management (IAM) user to be able to download from and upload to the bucket. How can I do that?

Answer: "I set up my Amazon Simple Storage Service (Amazon S3) bucket to use default encryption with a custom AWS Key Management Service (AWS KMS) key. I want an AWS Identity and Access Management (IAM) user to be able to download from and upload to the bucket. How can I do that?ResolutionThe IAM user and the AWS KMS key belong to the same AWS account1.    Open the AWS KMS console, and then view the key's policy document using the policy view. Modify the key's policy to grant the IAM user permissions for the kms:GenerateDataKey and kms:Decrypt actions at minimum. You can add a statement like the following:{  "Sid": "ExampleStmt",  "Action": [    "kms:Decrypt",    "kms:GenerateDataKey"  ],  "Effect": "Allow",  "Principal": {    "AWS": "arn:aws:iam::111122223333:user/Jane"  },  "Resource": "*"}Note: This example policy includes only the minimum permissions required for an individual IAM user to download and upload to an encrypted S3 bucket. You can modify or expand the permissions based on your use case.2.    Open the IAM console. Add a policy to the IAM user that grants the permissions to upload and download from the bucket. You can use a policy that's similar to the following:Note: For the Resource value, enter the Amazon Resource Name (ARN) for the bucket with a wildcard character to indicate the objects in the bucket.{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "ExampleStmt",      "Action": [        "s3:GetObject",        "s3:PutObject"      ],      "Effect": "Allow",      "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"    }  ]}Important: The S3 permissions granted by the IAM user policy can be blocked by an explicit deny statement in the bucket policy. Be sure to review the bucket policy to confirm that there aren't any explicit deny statements that conflict with the IAM user policy.The IAM user is in a different account than the AWS KMS key and S3 bucketImportant: You can grant cross-account access for a customer managed AWS KMS key, but not for an AWS managed AWS KMS key. The key policy of an AWS managed AWS KMS key can't be modified.1.    Open the AWS KMS console, and then view the key's policy document using the policy view. Modify the key's policy to grant the IAM user permissions for the kms:GenerateDataKey and kms:Decrypt actions at minimum. You can add a statement that's similar to the following:{  "Sid": "ExampleStmt",  "Action": [    "kms:Decrypt",    "kms:GenerateDataKey"  ],  "Effect": "Allow",  "Principal": {    "AWS": "arn:aws:iam::111122223333:user/Jane"  },  "Resource": "*"}Note: This example policy includes only the minimum permissions required for an individual IAM user to download and upload to an encrypted S3 bucket. You can modify or expand the permissions based on your use case.2.    Open the IAM console from the account that the IAM user belongs to. Add a policy to the IAM user that grants the permissions to upload and download from the bucket. The policy must also work with the AWS KMS key that's associated with the bucket.For cross-account scenarios, consider granting s3:PutObjectAcl permissions so that the IAM user can upload an object. Then, grant the bucket's account full control of the object (bucket-owner-full-control). Additionally, consider granting s3:ListBucket permissions, which is required for running a sync operation, or a recursive copy operation. You can use a policy that's similar to the following:Note: For the first Resource value, enter the ARN for the bucket with a wildcard character to indicate the objects in the bucket. For the second Resource value, enter the ARN for the bucket. For the third Resource value, enter the AWS KMS key's ARN.{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "DownloadandUpload",      "Action": [        "s3:GetObject",        "s3:GetObjectVersion",        "s3:PutObject",        "s3:PutObjectAcl"      ],      "Effect": "Allow",      "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"    },    {      "Sid": "ListBucket",      "Action": [        "s3:ListBucket"      ],      "Effect": "Allow",      "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET"    },    {      "Sid": "KMSAccess",      "Action": [        "kms:Decrypt",        "kms:GenerateDataKey"      ],      "Effect": "Allow",      "Resource": "arn:aws:kms:example-region-1:123456789098:key/111aa2bb-333c-4d44-5555-a111bb2c33dd"    }  ]}3.    Open the Amazon S3 console from the account that owns the S3 bucket. Update the bucket policy to grant the IAM user access to the bucket. You can use a policy like the following:Note: For the Principal values, enter the IAM user's ARN. For the first Resource value, enter the ARN for the bucket with a wildcard character to indicate the objects in the bucket. For the second Resource value, enter the ARN for the bucket.{  "Id": "Policy1584399307003",  "Version": "2012-10-17",  "Statement": [    {      "Sid": "DownloadandUpload",      "Action": [        "s3:GetObject",        "s3:GetObjectVersion",        "s3:PutObject",        "s3:PutObjectAcl"      ],      "Effect": "Allow",      "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",      "Principal": {        "AWS": [          "arn:aws:iam::111122223333:user/Jane"        ]      }    },    {      "Sid": "ListBucket",      "Action": [        "s3:ListBucket"      ],      "Effect": "Allow",      "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET",      "Principal": {        "AWS": [          "arn:aws:iam::111122223333:user/Jane"        ]      }    }  ]}Related informationEditing keysaws s3 cpUsing IAM policies with AWS KMSAmazon S3: Allows read and write access to objects in an S3 bucketDisabling ACLs for all new buckets and enforcing Object OwnershipFollow"


Question: How can I use a Lambda function to stop an Amazon RDS instance for longer than seven days?

I want to use AWS Lambda to stop an Amazon Relational Database Service (Amazon RDS) for longer than the seven-day duration.

Answer: "I want to use AWS Lambda to stop an Amazon Relational Database Service (Amazon RDS) for longer than the seven-day duration.Short descriptionBy default, you can stop an Amazon RDS database instance for up to seven days at a time. After seven days, the instance restarts so that it doesn't miss any maintenance updates.To stop your instance for more than seven days, you can use Step Functions to automate the workflow without missing a maintenance window.Note: For an alternative resolution, see How can I use Step Functions to stop an Amazon RDS instance for longer than seven days?ResolutionConfigure IAM permissionsCreate an AWS Identity and Access Management (IAM) policy to allow Lambda to start and stop the instance and retrieve information on the instance.1.    Open the IAM console.2.    In the navigation pane, choose Policies.3.    Choose Create Policy.4.    Choose the JSON tab.5.    To grant the required IAM permissions, enter the following policy under the JSON tab:{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "VisualEditor0",            "Effect": "Allow",            "Action": [                "rds:StartDBCluster",                "rds:StopDBCluster",                "rds:ListTagsForResource",                "rds:DescribeDBInstances",                "rds:StopDBInstance",                "rds:DescribeDBClusters",                "rds:StartDBInstance"            ],            "Resource": "*"        }    ]}6.    Choose Next: Tags.7.    (Optional) To add a tag, choose Add tag, and then enter the appropriate values for the Key and Value - optional fields.8.    Choose Next: Review.9.    On the Review policy page, for Name, enter the name for your policy. Review the Summary section to see the permissions that are granted by your policy.10.    Choose Create policy.For more information, see Creating policies using the JSON editor.Create an IAM role, and then attach the required policies1.    Open the IAM console.2.    In the navigation pane, choose Roles.3.    Choose Create role.4.    For Select type of trusted entity, select AWS service.5.    Under Or select a service to view its use cases, select Lambda.6.    Choose Next: Permissions.7.    For Filter-policies, enter the name of the policy that you created in the previous section. After it appears, select the policy. For Filter-policies, enter AWSLambdaBasicExecutionRole. After it appears, select the AWSLambdaBasicExecutionRole managed policy.8.    Choose Next: Tags.9.     (Optional) To add a tag, enter the appropriate values for the Key and Value (optional) fields.10.    Choose Next: Review.11.    On the Create role page, for Role name, enter the name for the role that you're creating.12.    Choose Create role.For more information, see Creating a role for an AWS service (console).Add tags for DB instances1.    Open the Amazon RDS console.2.    In the navigation pane, choose Databases.3.    Choose the DB instance that you want to start and stop automatically.4.    In the details section, scroll down to the Tags section.5.    Under the Tags tab, choose Add. For Tag key, enter autostart. For Value, enter yes. Choose Add to save your changes.6.    Choose Add again. For Tag key, enter autostop. For Value, enter yes. Choose Add to save your changes.For more information, see Adding, listing, and removing tags.Create a Lambda function to start the tagged DB instances1.    Open the Lambda console.2.    In the navigation pane, choose Functions.3.    Choose Create function.4.    Choose Author from scratch.5.    For Function name, enter the name of your function.6.    For Runtime, select Python 3.7.7.    For Architecture, keep the default selection of x86_64.8.    Expand Change default execution role.9.    For Execution role, select Use an existing role.10.    For Existing role, select the IAM role that you created.11.    Choose Create function.12.    Choose the Code tab.13.    In the Code source editor, delete the sample code and enter the following code:import boto3rds = boto3.client('rds')def lambda_handler(event, context):    #Start DB Instances    dbs = rds.describe_db_instances()    for db in dbs['DBInstances']:        #Check if DB instance stopped. Start it if eligible.        if (db['DBInstanceStatus'] == 'stopped'):            try:                GetTags=rds.list_tags_for_resource(ResourceName=db['DBInstanceArn'])['TagList']                for tags in GetTags:                #if tag "autostart=yes" is set for instance, start it                    if(tags['Key'] == 'autostart' and tags['Value'] == 'yes'):                        result = rds.start_db_instance(DBInstanceIdentifier=db['DBInstanceIdentifier'])                        print ("Starting instance: {0}.".format(db['DBInstanceIdentifier']))            except Exception as e:                print ("Cannot start instance {0}.".format(db['DBInstanceIdentifier']))                print(e)                if __name__ == "__main__":    lambda_handler(None, None)14.    Choose File, choose Save, and then choose Deploy.15.    Choose the Configuration tab, choose General configuration, and then choose Edit.16.    Under Timeout, complete the following fields:For min, select 0.For sec, select 10.17.    Choose Save.Create a Lambda function to stop the tagged DB instancesTo create a Lambda function to stop the tagged DB instances, see the previous section Create a Lambda function to start the tagged DB instances. Follow the same steps, but use different code for step 12.In the Code source editor, delete the sample code and enter the following code:import boto3rds = boto3.client('rds')def lambda_handler(event, context):    #Stop DB instances    dbs = rds.describe_db_instances()    for db in dbs['DBInstances']:        #Check if DB instance is not already stopped        if (db['DBInstanceStatus'] == 'available'):            try:                GetTags=rds.list_tags_for_resource(ResourceName=db['DBInstanceArn'])['TagList']                for tags in GetTags:                #if tag "autostop=yes" is set for instance, stop it                    if(tags['Key'] == 'autostop' and tags['Value'] == 'yes'):                        result = rds.stop_db_instance(DBInstanceIdentifier=db['DBInstanceIdentifier'])                        print ("Stopping instance: {0}.".format(db['DBInstanceIdentifier']))            except Exception as e:                print ("Cannot stop instance {0}.".format(db['DBInstanceIdentifier']))                print(e)                if __name__ == "__main__":    lambda_handler(None, None)Perform function testingFor tagged DB instances that are in the Stopped state, complete the following steps to perform function testing:1.    Open the Lambda Functions list.2.    Choose the function that you created to start the DB instances.3.    Choose Actions, and then choose Test.4.    Under the Test tab, for Name, enter the name of your event.5.    Choose Save changes, and then choose Test.Create the scheduleYou can create rules to set up a schedule. For example, if your weekly maintenance window for the tagged DB instances is Sunday 22:00 - 22:30, then you can create the following rules:Automatically start the DB instance 30 minutes before the maintenance window begins.Automatically stop the DB instance 30 minutes after the maintenance window ends.To create the rule to automatically start the DB instance 30 minutes before the maintenance window, complete the following steps:1.    Open the Lambda Functions list.2.    Choose the function that you created to start the DB instances.3.    Under Function overview, choose Add trigger.4.    Select EventBridge (CloudWatch Events), and then select Create a new rule.5.    For Rule name, enter the name of the rule that you want to create.6.    For Schedule Expression, add a cron expression for the automated schedule (Example: cron(30 21 ? * SUN *)).7.    Choose Add.Use the same instructions to create another rule to automatically stop the DB instance 30 minutes after the maintenance window. Be sure to change the name of the rule and the cron expression for the automated schedule accordingly (Example: cron(00 23 ? * SUN *)).Related informationImplementing DB instance stop and start in Amazon RDSStopping an automatically started database instance with Amazon RDSHow can I stop an Amazon Aurora cluster for longer than seven days?Follow"


Question: How can I download and access AWS Artifact SOC reports?

"I requested access to an AWS Artifact Service Organization Control (SOC) report, but didn't receive a response, or I received a message similar to the following:"You need additional approval from AWS to access this report.""

Answer: "I requested access to an AWS Artifact Service Organization Control (SOC) report, but didn't receive a response, or I received a message similar to the following:"You need additional approval from AWS to access this report."ResolutionOpen the AWS Artifact console. Then, follow these instructions to configure and submit an access request for AWS Artifact documents and open SOC reports.AWS Identity and Access Management (IAM) user permissionsBy default, only the AWS account root user has permissions to access AWS Artifact compliance documents (also known as audit artifacts or reports). To provide AWS Artifact permissions to an IAM entity using IAM policies, see Identity and access management in AWS Artifact. If you provided IAM permissions and get an error message, see the "error message in AWS Artifact console" section in the AWS Artifact FAQ.Important:If multi-factor authentication (MFA) is enabled for an IAM user with an enforce MFA policy attached, you must sign in using MFA.For AWS Organizations accounts, be sure that service control policies (SCP) do not deny access to AWS Artifact reports.I submitted the AWS Artifact access form and received approval, but still can't access the report or didn't receive a responseAfter you submit the AWS Artifact Access Request Form for access to a SOC report, access is granted within one business day. If you can't access AWS Artifact documents after one business day, be sure that your information in the form is correct.If the information in the form is correct and your request to access AWS Artifact wasn't approved, submit the AWS Compliance Contact Us form.Open the SOC report in the downloadAWS Artifact reports contain a copy of the signed NDA and the document that you downloaded. To access the SOC report in the AWS Artifact download, choose the paperclip icon in the PDF. If you don't see the AWS Artifact report download, see Troubleshooting.For more information, see How to open AWS Artifact reports.Related informationHow can I download and share AWS Artifact documents with regulators and auditors, or with others in my company?SOC Report FAQLearn how to use AWS Artifact to retrieve your compliance reportsFollow"


Question: How can I create a Client VPN endpoint using certificate-based authentication?

I want to access my AWS Resources using AWS Client VPN. How can I create a Client VPN endpoint using certificate-based authentication?

Answer: "I want to access my AWS Resources using AWS Client VPN. How can I create a Client VPN endpoint using certificate-based authentication?ResolutionThe Client VPN endpoint is the server where all Client VPN sessions are terminated. The endpoint, managed by AWS, establishes a secure Transport Layer Security (TLS) connection between your VPC and the OpenVPN-based client. To create a Client VPN endpoint using certificate-based authentication, follow these steps:Generate server and client certificates and keysTo authenticate the clients, you must generate the following, and then upload them to AWS Certificate Manager (ACM):Server and client certificatesClient keysCreate a Client VPN endpointWhen you create a Client VPN endpoint, specify the Server Certificate ARN provided by ACM. You also must choose a Client IPv4 CIDR, which is the IP address range assigned to the clients after the VPN is established. Note that the IP address range can't overlap with the VPC CIDR block.You can enable client connection logging with CloudWatch Logs and specify custom DNS servers for clients to use. You can also enable split-tunnel on the VPN endpoint, and then select UDP or TCP as the transport protocol.Enable VPN connectivity for clientsTo enable clients to establish a VPN session, you must associate a target network with the Client VPN endpoint. A target network is a subnet in a VPC. One subnet association is sufficient for clients to access a VPC's entire network, if authorization rules permit this. You can associate additional subnets to provide high availability if an Availability Zone goes down.Authorize clients to access VPC resources or any other networkTo authorize clients to access the VPC, create an authorization rule. The authorization rule specifies the clients that can access the VPC.You can also enable access to additional networks, such as AWS services, peered VPCs, on-premises networks, or the internet. For each additional network, you must add a route to the Client VPN endpoint route table and then configure an authorization rule to give clients access.To authorize clients to access your VPC and different networks, see Add an authorization rule for the VPC.Download the Client VPN endpoint configuration fileThe final step is to download and prepare the Client VPN endpoint configuration file. Provide this file to clients so that they can upload the configuration settings into their VPN client application.Related informationGetting started with Client VPNFollow"


Question: How can I load data to Amazon Redshift from my Salesforce account using Amazon AppFlow?

I want to extract data from my Salesforce account and load it to Amazon Redshift.

Answer: "I want to extract data from my Salesforce account and load it to Amazon Redshift.Short descriptionTo extract data from your Salesforce account and load it to Amazon Redshift, you must:Allow Salesforce edition to provide API access to Amazon AppFlow.Check that your Salesforce account meets the requirements to load data to Amazon Redshift.Run Amazon AppFlow on demand to test the connectivity and flow, end to end.Note: To load your Salesforce data to Amazon Redshift, Amazon Redshift must have a public security group.ResolutionLog in to your Salesforce account.Choose Setup, and then, in the search window, enter Company Information.Under Organization edition, note the edition of Salesforce that you're using. Note: Salesforce Enterprise, Unlimited, Developer, and Performance editions provide API access, but Professional and Essentials editions don't. Check the latest information on which editions provide API access before continuing.In the navigation pane, choose Users, and then choose Profiles.Choose System administrator, and then choose the System Administrator link.On the next page, under Administrative Permissions, select the check box for API enabled.Log in to the Amazon AppFlow console, and choose Create flow.Under Flow details, enter a name and description for your flow.Under Source name, type Salesforce, and then, under Choose Salesforce connection, choose Create new connection.Under Salesforce environment, choose Production, and then enter a name for your connection.Choose Continue. This opens the Salesforce login page. Enter your user ID and password to continue. After you log in and allow access, the connection from Amazon AppFlow to Salesforce is established.On the Configure flow page in Salesforce, choose the object that you want to migrate.In the Destination details section, for Destination name, search for and then choose Amazon Redshift.Under New connection, choose Create new connection.On the Connect to Amazon Redshift page, enter the details of your Amazon Redshift connection. Be sure that Amazon Redshift has a public security group.Attach this AWS Key Management Service (AWS KMS) decrypt policy to the AWS Identity and Access Management (IAM) role for Amazon Redshift:{    "Effect": "Allow",    "Action": "kms:Decrypt",    "Resource": "*" }Choose the Amazon Simple Storage Service (Amazon S3) bucket that you want Amazon AppFlow to use when migrating data to Amazon Redshift.Under Choose Amazon Redshift object, enter the Amazon Redshift schema name. Then, under Choose Redshift table, enter the Amazon Redshift table name.Under Error handling, choose the Amazon S3 bucket that you want to write records to if an issue stops records being written to the destination.On the Map data fields page, choose the object fields that you want to migrate from the destination to the source. Then, choose Map fields.Review all your details, and then choose Run flow.After running the flow, check the records in the destination table.Related informationWhat is Amazon AppFlow?Supported source and destination applicationsFollow"


Question: How do I manage my AWS service quotas?

"How do I increase my service quotas, and how do I manage my service quotas as I scale up?"

Answer: "How do I increase my service quotas, and how do I manage my service quotas as I scale up?ResolutionAWS maintains service quotas (formerly called service limits) for each account to help guarantee the availability of AWS resources and prevent accidental provisioning of more resources than needed.Increase your service quotasYou can use Service Quotas console to view and request increases for most AWS quotas.Important: Most service quotas are specific to an AWS Region. Select the AWS Region where you require the quota increase in.For all other quota increase requests, create a case in the AWS Support console, and then choose Service Limit Increase.Note: You aren't charged if your service quotas are increased. You're charged only if you launch or use AWS resources or services.Monitor and proactively manage your service quotasYou can see your quota utilization in the Service Quotas console. You can also create Amazon CloudWatch alarms for supported services. For more information, see Service Quotas and Amazon CloudWatch Alarms.Related informationAWS service quotasFollow"


Question: Should I use GetMetricData or GetMetricStatistics for CloudWatch metrics?

"I want to retrieve data points from my Amazon CloudWatch metrics. Which API should I use, GetMetricData or GetMetricStatistics?"

Answer: "I want to retrieve data points from my Amazon CloudWatch metrics. Which API should I use, GetMetricData or GetMetricStatistics?Short descriptionIt's a best practice to use the GetMetricData API instead of GetMetricStatistics, because you can retrieve data faster at scale with GetMetricData. GetMetricData also supports metric math, and it returns ordered, paginated results. This difference in functionality is reflected in the cost of each service. GetMetricStatistics is included in the free tier of CloudWatch, for up to 1 million API requests, while GetMetricData is not. Refer to the chart below and the CloudWatch pricing chart to see which API is more appropriate for you.Metrics per callData points per callSupports metric mathReturns ordered and paginated resultsIncluded with free tier limitsGetMetricData500100,800YesYes*NoGetMetricStatistics11,440NoNoYes*To return the next set of data points, you must iterate over the data using the NextToken (--next-token) provided in the response.The service quotas for the GetMetricData API are:50 transactions per second (TPS).180,000 data points per second (DPS), if the StartTime in the API request is less than or equal to three hours from the current time.396,000 DPS, if the StartTime is more than three hours from the current time.ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Use the following example call as a reference for making your own GetMetricData API calls in the AWS CLI.1.    Create an input parameter for your GetMetricData API call (metric-data-queries.json). The input parameter has two custom metrics  (Invocations and Errors) and one metric (ErrorRate) calculated by the metric math of the other two metrics (ErrorRate).$ cat metric-data-queries.json[    {        "Id": "e1",        "Expression": "m1 / m2",        "Label": "ErrorRate"    },    {        "Id": "m1",        "MetricStat": {            "Metric": {                "Namespace": "MyApplication",                "MetricName": "Errors",                "Dimensions": [                    {                        "Name": "FunctionName",                        "Value": "MyFunc"                    }                ]            },            "Period": 300,            "Stat": "Sum",            "Unit": "Count"        },        "ReturnData": false    },    {        "Id": "m2",        "MetricStat": {            "Metric": {                "Namespace": "MyApplication",                "MetricName": "Invocations",                "Dimensions": [                    {                        "Name": "FunctionName",                        "Value": "MyFunc"                    }                ]            },            "Period": 300,            "Stat": "Sum",            "Unit": "Count"        },        "ReturnData": false    }]2.    Publish sample metric data as custom metrics using PutMetricData. For example:$ aws cloudwatch put-metric-data --namespace MyApplication --metric-name Invocations --dimensions FunctionName=MyFunc --value 10 --unit Count --timestamp 2018-06-19T04:00:00Z$ aws cloudwatch put-metric-data --namespace MyApplication --metric-name Invocations --dimensions FunctionName=MyFunc --value 10 --unit Count --timestamp 2018-06-19T04:05:00Z$ aws cloudwatch put-metric-data --namespace MyApplication --metric-name Invocations --dimensions FunctionName=MyFunc --value 10 --unit Count --timestamp 2018-06-19T04:10:00Z$ aws cloudwatch put-metric-data --namespace MyApplication --metric-name Invocations --dimensions FunctionName=MyFunc --value 10 --unit Count --timestamp 2018-06-19T04:15:00Z$ aws cloudwatch put-metric-data --namespace MyApplication --metric-name Invocations --dimensions FunctionName=MyFunc --value 10 --unit Count --timestamp 2018-06-19T04:20:00Z$ aws cloudwatch put-metric-data --namespace MyApplication --metric-name Errors --dimensions FunctionName=MyFunc --value 3 --unit Count --timestamp 2018-06-19T04:00:00Z$ aws cloudwatch put-metric-data --namespace MyApplication --metric-name Errors --dimensions FunctionName=MyFunc --value 6 --unit Count --timestamp 2018-06-19T04:05:00Z$ aws cloudwatch put-metric-data --namespace MyApplication --metric-name Errors --dimensions FunctionName=MyFunc --value 2 --unit Count --timestamp 2018-06-19T04:10:00Z$ aws cloudwatch put-metric-data --namespace MyApplication --metric-name Errors --dimensions FunctionName=MyFunc --value 9 --unit Count --timestamp 2018-06-19T04:15:00Z$ aws cloudwatch put-metric-data --namespace MyApplication --metric-name Errors --dimensions FunctionName=MyFunc --value 1 --unit Count --timestamp 2018-06-19T04:20:00ZNote: You can also publish up to 20 metrics using a single PutMetricData API call under the same namespace. To do so, use the --metric-data option in the PutMetricData call.3.    Run the command aws cloudwatch get-metric-data with your input parameters.4.    Review the output. In this example, five data points are calculated using metric math and returned as a time-ordered result. m1 and m2 are not included in the response, because ReturnData is set to false.$ aws cloudwatch get-metric-data --metric-data-queries file://./metric-data-queries.json --start-time 2018-06-19T04:00:00Z --end-time 2018-06-19T04:30:00Z{    "MetricDataResults": [        {            "Timestamps": [                "2018-06-19T04:20:00Z",                "2018-06-19T04:15:00Z",                "2018-06-19T04:10:00Z",                "2018-06-19T04:05:00Z",                "2018-06-19T04:00:00Z"            ],            "StatusCode": "Complete",            "Values": [                0.1,                0.9,                0.2,                0.6,                0.3            ],            "Id": "e1",            "Label": "ErrorRate"        }    ]}Related informationget-metric-data AWS CLI referenceget-metric-statistics AWS CLI referenceFollow"


Question: How do I set up a deny list in my Amazon Connect contact center that allows agents to flag numbers in real time?

I want agents in my Amazon Connect contact center to be able to add numbers to a deny list in real time. I also want to be able to add numbers to the list manually. How do I set that up?

Answer: "I want agents in my Amazon Connect contact center to be able to add numbers to a deny list in real time. I also want to be able to add numbers to the list manually. How do I set that up?Short descriptionTo allow agents in your Amazon Connect contact center to add numbers to a deny list in real time, do the following:Create an Amazon DynamoDB table that holds denied numbers.Create an AWS Identity and Access Management (IAM) role that an AWS Lambda function can assume to either look up or add denied numbers in Dynamo DB.Create a Lambda function (function A) that stores the denied numbers in the DynamoDB table.Create a second Lambda function (function B) that queries the DynamoDB table to check if an incoming call is from a number on the deny list.Add the Lambda functions to your Amazon Connect instance.Create an inbound contact flow associated with a claimed number that does the following:Invokes function B to check an incoming call against the deny list in DynamoDB.Then, if the number isn't on the deny list, transfers the call to your regular customer queue flow.-or-(If the number is on the deny list) Plays an audio prompt and disconnects the call.Create a transfer to queue flow that invokes function A when an agent adds a number to the deny list.Create a quick connect that allows agents to add numbers to the deny list using the Contact Control Panel (CCP).Note: You can also manually add numbers to the deny list by editing the DynamoDB table directly.ResolutionImportant: Make sure that you follow these steps in the same AWS Region that your Amazon Connect instance is in.Create a DynamoDB table that holds denied numbers1.    Open the DynamoDB console.2.    Choose Create DynamoDB table. Then, do the following:For Table name, enter DenylistingTable.In the Partition key panel, for Primary key, enter ContactNumber.For Data type, choose String.Choose Create.Note: For more information, see Write data to a table using the console or AWS CLI.Create an IAM role that a Lambda function can assume to either look up or add denied numbers in DynamoDB1.    Create an IAM role that uses the following JSON policy:Important: Make sure that the AWSLambdaBasicExecutionRole is included in the policy along with the DynamoDB permissions. If the AWSLambdaBasicExecutionRole isn't included in the policy, then the function won't invoke.Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.Permission is hereby granted, free of charge, to any person obtaining a copy of thissoftware and associated documentation files (the "Software"), to deal in the Softwarewithout restriction, including without limitation the rights to use, copy, modify,merge, publish, distribute, sublicense, and/or sell copies of the Software, and topermit persons to whom the Software is furnished to do so.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR APARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHTHOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTIONOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THESOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "VisualEditor0",            "Effect": "Allow",            "Action": "cloudwatch:GetInsightRuleReport",            "Resource": "arn:aws:cloudwatch:*:*:insight-rule/DynamoDBContributorInsights*"        },        {            "Sid": "VisualEditor1",            "Effect": "Allow",            "Action": "iam:PassRole",            "Resource": "*",            "Condition": {                "StringLike": {                    "iam:PassedToService": [                        "application-autoscaling.amazonaws.com",                        "application-autoscaling.amazonaws.com.cn",                        "dax.amazonaws.com"                    ]                }            }        },        {            "Sid": "VisualEditor2",            "Effect": "Allow",            "Action": "iam:CreateServiceLinkedRole",            "Resource": "*",            "Condition": {                "StringEquals": {                    "iam:AWSServiceName": [                        "replication.dynamodb.amazonaws.com",                        "dax.amazonaws.com",                        "dynamodb.application-autoscaling.amazonaws.com",                        "contributorinsights.dynamodb.amazonaws.com",                        "kinesisreplication.dynamodb.amazonaws.com"                    ]                }            }        },        {            "Sid": "VisualEditor3",            "Effect": "Allow",            "Action": [                "lambda:CreateFunction",                "cloudwatch:DeleteAlarms",                "sns:Unsubscribe",                "dynamodb:*",                "lambda:GetFunctionConfiguration",                "datapipeline:CreatePipeline",                "kinesis:ListStreams",                "logs:CreateLogStream",                "kinesis:DescribeStreamSummary",                "resource-groups:GetGroup",                "cloudwatch:DescribeAlarmsForMetric",                "lambda:DeleteFunction",                "sns:Subscribe",                "iam:GetRole",                "application-autoscaling:RegisterScalableTarget",                "sns:ListSubscriptionsByTopic",                "datapipeline:ListPipelines",                "dax:*",                "lambda:ListFunctions",                "sns:CreateTopic",                "application-autoscaling:DeleteScalingPolicy",                "cloudwatch:GetMetricStatistics",                "logs:CreateLogGroup",                "resource-groups:CreateGroup",                "application-autoscaling:DescribeScalingPolicies",                "lambda:ListEventSourceMappings",                "application-autoscaling:PutScalingPolicy",                "cloudwatch:DescribeAlarms",                "resource-groups:ListGroupResources",                "ec2:DescribeSubnets",                "lambda:DeleteEventSourceMapping",                "datapipeline:ActivatePipeline",                "resource-groups:GetGroupQuery",                "tag:GetResources",                "sns:DeleteTopic",                "cloudwatch:GetMetricData",                "sns:ListTopics",                "sns:SetTopicAttributes",                "lambda:CreateEventSourceMapping",                "datapipeline:DescribePipelines",                "cloudwatch:ListMetrics",                "cloudwatch:DescribeAlarmHistory",                "application-autoscaling:DescribeScalingActivities",                "kms:DescribeKey",                "datapipeline:PutPipelineDefinition",                "application-autoscaling:DescribeScalableTargets",                "datapipeline:QueryObjects",                "iam:ListRoles",                "datapipeline:DescribeObjects",                "kinesis:DescribeStream",                "sns:ListSubscriptions",                "resource-groups:ListGroups",                "datapipeline:GetPipelineDefinition",                "logs:PutLogEvents",                "ec2:DescribeSecurityGroups",                "resource-groups:DeleteGroup",                "cloudwatch:PutMetricAlarm",                "ec2:DescribeVpcs",                "kms:ListAliases",                "datapipeline:DeletePipeline",                "application-autoscaling:DeregisterScalableTarget"            ],            "Resource": "*"        }    ]}2.    Attach the following, second JSON policy to the IAM role:Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.Permission is hereby granted, free of charge, to any person obtaining a copy of thissoftware and associated documentation files (the "Software"), to deal in the Softwarewithout restriction, including without limitation the rights to use, copy, modify,merge, publish, distribute, sublicense, and/or sell copies of the Software, and topermit persons to whom the Software is furnished to do so.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR APARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHTHOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTIONOF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THESOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "lambda:*",                "dynamodb:*"            ],            "Resource": "*"        }    ]}Create a Lambda function (function A) that stores denied numbers in the DynamoDB tableUse the following Python code to create a Lambda function. Attach the following permissions to the role associated with the lambda function:{        "Version": "2012-10-17",        "Statement": [{            "Sid": "VisualEditor0",            "Effect": "Allow",            "Action": "dynamodb:PutItem",            "Resource": "arn:aws:dynamodb:<region>:<account-number>:table/DenylistingTable"        }]    }Create a second Lambda function (function B) that queries the Dynamo DB table to check if an incoming call is from a number on the deny listCreate a second Lambda function with the following permissions:{"Version": "2012-10-17","Statement": [{"Sid": "VisualEditor0","Effect": "Allow","Action": "dynamodb:Query","Resource": "arn:aws:dynamodb:<region>:<account-number>:table/DenylistingTable"}]}Add the Lambda functions (function A and function B) to your Amazon Connect instance1.    Open the Amazon Connect console.2.    In the Instance Alias column, choose the name of your Amazon Connect instance.3.    In the left navigation pane, choose Contact flows.4.    In the AWS Lambda section, choose the Function dropdown list. Then, choose the name of function A. Note: The Function dropdown list names the functions that are in the same AWS Region as your Amazon Connect instance only. If no functions are listed, then choose Create a new Lambda function to create a new function in the correct Region.5.    Choose Add Lambda Function. Then, confirm that the Amazon Resource Name (ARN) of the function is added under Lambda Functions.6.    Add function B to your Amazon Connect instance by repeating steps 1-5. For step 4, make sure that you choose the name of function B.For more information, see Invoke AWS Lambda functions.Create an inbound contact flowCreate a new inbound contact flow associated with a claimed number. After you choose Create contact flow, the contact flow designer opens.In the contact flow designer, do the following:Note: The following is an example of a basic inbound contact flow. You can add or edit blocks for your use case.Add a Set logging behavior blockImportant: If you haven't already, turn on contact flow logging for your instance.To turn on contact flow logs in your Amazon Connect CloudWatch Logs, use a Set logging behavior block.1.    Choose Set.2.    Drag and drop a Set logging behavior block onto the canvas to the right of the Entry point block.3.    Choose the block title (Set logging behavior). The block's settings menu opens.4.    Choose Enable.Add an Invoke AWS Lambda function blockTo invoke function B and check if an incoming call is from a number on the deny list, use an Invoke AWS Lambda function block.1.    Choose Integrate.2.    Drag and drop an Invoke AWS Lambda function block onto the canvas to the right of the Set logging behavior block.3.    Choose the block title (Invoke AWS Lambda function). The block's settings menu opens.4.    Choose the name of function B.5.    Choose Save.Add a Check contact attributes blockTo confirm if the number was added to the deny list or not, use a Check contact attributes block.1.    Choose Branch.2.    Drag and drop a Check contact attributes block onto the canvas to the right of the Invoke AWS Lambda function block.3.    Choose the block title (Check contact attributes). The block's settings menu opens.4.    For Type, choose External.5.    For Attribute, choose Success.6.    Choose Add another Parameter twice so that two parameters are created.7.    Under Conditions to check, choose Equals for both parameters. Then, for the value of the first parameter, enter true. For the value of the second parameter, enter false.8.    Choose Save.Add a Play prompt blockTo play a prompt that informs agents when a deny list request error occurs, use a Play prompt block.Choose Interact.2.    Drag and drop a Play prompt block onto the canvas to the right of the Invoke AWS Lambda function block. Then, connect the Play prompt block to the Error output of the Invoke AWS Lambda function block.3.    Choose the block title (Play prompt). The block's settings menu opens.4.    For Prompts, choose Text-to-speech. Then, choose one of the following: Enter text allows you to play text as a lifelike audio message. -or- Enter dynamically allows you to upload .wav files to play a recorded audio message.5.    Choose Save.Add a Transfer to queue blockTo end the contact flow and transfer the call to your regular customer queue flow, use a Transfer to queue block.1.    Choose Terminate/Transfer.2.    Drag and drop a Transfer to queue block onto the canvas to the right of the Play prompt block.Note: No settings need to be configured for the Transfer to queue block for this use case.Add a Disconnect blockTo disconnect a caller from the contact flow if the incoming call is from a number that's on the deny list, add a Disconnect block.1.    Choose Terminate/Transfer.2.    Drag and drop a Disconnect block onto the canvas to the right of the Transfer to queue block.Note: No settings need to be configured for the Disconnect block for this use case.Activate the contact flow1.    To save a draft of the flow, Choose Save.2.    To activate the flow, Choose Publish.Create a Transfer to queue flow that allows agents to add numbers to the deny list in real timeCreate a new Transfer to queue flow. After you choose Create contact flow, the contact flow designer opens.In the contact flow designer, do the following:Add a Play prompt blockTo let agents know that a number is in the process of being added to the deny list, use a Play prompt block.1.    Choose Interact.2.    Drag and drop a Play prompt block onto the canvas to the right of the Entry point block.3.    Choose the block title (Play prompt). The block's settings menu opens.4.    For Prompts, choose Text-to-speech. Then, choose either Enter text or Enter dynamically to create an audio message.5.    Choose Save.Add an Invoke AWS Lambda function blockTo invoke function A when an agent adds a number to the deny list using the CCP, use an Invoke AWS Lambda function block.1.    Choose Integrate.2.    Drag and drop an Invoke AWS Lambda function block onto the canvas to the right of the Play prompt block.3.    Choose the block title (Invoke AWS Lambda function). The block's settings menu opens.4.    Choose the name of function A.5.    In the Function input parameters section, choose Add a parameter and choose Use attribute. Then, do the following: For Destination key, enter CustomerNumber. For Type, choose System. For Attribute, choose Customer Number.6.    Choose Save.Add a second Play prompt blockTo let agents know that a number was added to the deny list, use a second Play prompt block.1.    Choose Interact.2.    Drag and drop a Play prompt block onto the canvas to the right of the Invoke AWS Lambda function block. Then, connect the Play prompt block to the Success output of the Invoke AWS Lambda function block.3.    Choose the block title (Play prompt). The block's settings menu opens.4.    For Prompts, choose Text-to-speech. Then, choose either Enter text or Enter dynamically to create an audio message.5.    Choose Save.Add a third Play prompt blockTo let agents know that a number wasn't added to the deny list because of an error, use a third Play prompt block.1.    Choose Interact.2.    Drag and drop a Play prompt block onto the canvas to the right of the Invoke AWS Lambda function block. Then, connect the Play prompt block to the Error output of the Invoke AWS Lambda function block.3.    Choose the block title (Play prompt). The block's settings menu opens.4.    For Prompts, choose Text-to-speech. Then, choose either Enter text or Enter dynamically to create an audio message.5.    Choose Save.Add a Disconnect blockTo disconnect agents from the contact flow, add a Disconnect block.1.    Choose Terminate/Transfer.2.    Drag and drop a Disconnect block onto the canvas to the right of the three Play prompt blocks.3.    Connect all the flow's branches to the Disconnect block.Note: No settings need to be configured for the Disconnect block for this use case.Activate the Transfer to queue flow1.    To save a draft of the flow, choose Save.2.    To activate the flow, choose Publish.Create a quick connect that allows agents to add numbers to the deny list using the CCP1.    Create a quick connect using the Amazon Connect console that has the following settings: For Name, enter a name for the quick connect. For example: Deny list. For Type, choose Queue. For Destination, enter the name of the queue that you want to have the deny list functionality.2.     Add the quick connect to the queues assigned to the agents that you want to have access to the deny list feature.Now, when an agent is on a call, they can use the deny list quick connect in the CCP to add the number to the deny list. The next time a call comes in to your Amazon Connect contact center from that number, the call will be automatically disconnected without being placed in a queue.Follow"


Question: Why am I receiving an "EFS Disk Quota Exceeded" error when connecting to my file system?

I'm receiving a "Disk quota exceeded" error when attempting to connect to my Amazon Elastic File System (Amazon EFS) file system. How can I troubleshoot this error and prevent it from happening again?

Answer: "I'm receiving a "Disk quota exceeded" error when attempting to connect to my Amazon Elastic File System (Amazon EFS) file system. How can I troubleshoot this error and prevent it from happening again?ResolutionThe Disk quota exceeded error message occurs when a limit on the disk is exceeded. This occurs when you hit the Network File System (NFS) client limit, because Amazon EFS doesn't support user disk quotas. Or, this error occurs when a user tries to open a file on an EFS file system that reached its limit. For more information, see Quotas for NFS clients.You might also receive the following error messages when you exceed a limit:LockObtainFailedException: This error might occur if the number of locks is exceeded.write.lock: java.io.IOException: This error might occur if you exceed an NFS client quota when using a Java-based application.Prevent disk quota exceeded errorsTo prevent exceeding disk quotas, do the following:Identify a specific limit. You can use performance monitoring tools such as nfsstat and nfsiostat. For more information on these tools, see the Red Hat page Using nfsstat and nfsiostat to troubleshoot NFS performance issues on Linux. Modify your application so that it doesn't exceed specific limits, such as for user accounts, open files, and process locks.Split your workload across more instances. Depending on your NFS client, there are various file usage limits that might result in a Disk quota exceeded error. However, there's only one limit shared across all clients that leads to this error. Specifically, a file can have up to 512 locks across all instances connected and users accessing that file. Therefore, by splitting the workload across more instances, you reduce the chances of reaching an instance's limit.If you continue to receive disk quota exceeded errors after following the steps above, contact the AWS support team for additional troubleshooting.Follow"


Question: How can I resolve the "HIVE_METASTORE_ERROR" error when I query a table in Amazon Athena?

I get the error "HIVE_METASTORE_ERROR" when I query my Amazon Athena table.

Answer: "I get the error "HIVE_METASTORE_ERROR" when I query my Amazon Athena table.Short descriptionSee the following types of "HIVE_METASTORE_ERROR" errors and their causes:HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: Error: : expected at the position 1234 of struct<test_column> but '/' is found. (Service: null; Status Code: 0; Error Code: null; Request ID: null): A column name in the queried table includes a special character, or a column name in the partition schema includes a special character. Athena doesn't support special characters other than underscore. For more information, see Names for tables, databases, and columns.HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: Required Table Storage Descriptor is not populated:The StorageDescriptor parameter contains information about the physical storage of the table. You get this error if one or more partitions on the table don't have the partition location set because of corrupt partitions.HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: java.io.IOException: Response payload size (11112222 bytes) exceeded maximum allowed payload size (6291556 bytes): You use an AWS Lambda function to run Athena queries against a cross-account AWS Glue Data Catalog or an external Hive metastore. However, Lambda has an invocation payload limit of 6 MB. You get this error when the size of the object that's returned from Lambda is more than 6 MB. The Lambda payload limit is a hard limit and can't be increased. For more information, see Lambda quotas.ResolutionHIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: Error: : expected at the position 1234 of struct<test_column> but '/' is found. (Service: null; Status Code: 0; Error Code: null; Request ID: null)To resolve this error, complete the following steps:To replace the special character in the column name with an underscore, run the following custom script on your data:import restring = open('a.txt').read()new_str = re.sub('/', '_', string)open('b.txt', 'w').write(new_str)Edit the existing schema of the table from the AWS Glue console, and then replace '/' with any other character that's supported by Athena.HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: Required Table Storage Descriptor is not populatedTo resolve this error, choose one or more of the following solutions:If your table is already partitioned, and the data is loaded in Amazon Simple Storage Service (Amazon S3) Hive partition format, then load the partitions. Run a command that's similar to the following example:Note: Be sure to replace doc_example_table with the name of your table.MSCK REPAIR TABLE doc_example_tableIf the MSCK REPAIR TABLE command doesn’t resolve the issue, then drop the table and create a new table with the same definition. Then, run the MSCK REPAIR TABLE command on the new table.Create a separate folder in the Amazon S3 bucket, and then move the data files that you want to query into that folder. Create an AWS Glue crawler that points to that folder instead of the bucket.HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: java.io.IOException: Response payload size (11112222 bytes) exceeded maximum allowed payload size (6291556 bytes)To resolve this error, choose one or more of the following solutions:Upload the Lambda function's response payload as an object into an Amazon S3 bucket. Then, include this object as payload in the Lambda function response. For information on generating a presigned URL for your object, see Sharing objects using presigned URLs.If your table is partitioned, and your use case permits, query only the specific partition.When you create the Lambda function, specify the spill location in Amazon S3. Responses that are larger than the threshold spill into the specified S3 location.Related informationTroubleshooting in AthenaFollow"


Question: Why doesn't Amazon CloudFront cache files in the amount of time I specified?

"I want to specify the amount of time that my Amazon CloudFront distribution caches objects, so I added Cache-Control or Expires headers on the origin. However, CloudFront doesn't cache objects for the amount of time I specified, or CloudFront doesn't cache at all. Why is this happening?"

Answer: "I want to specify the amount of time that my Amazon CloudFront distribution caches objects, so I added Cache-Control or Expires headers on the origin. However, CloudFront doesn't cache objects for the amount of time I specified, or CloudFront doesn't cache at all. Why is this happening?ResolutionBe sure that the directives that you set on the Cache-Control or Expires headers don't conflict with each other. As a best practice, use the Cache-Control max-age directive instead of the Expires header. If you specify values for both, then CloudFront uses only the value that you set for Cache-Control max-age.Additionally, verify that the Minimum TTL, Default TTL, and Maximum TTL that you set on the CloudFront distribution don't conflict with the Cache-Control or Expires headers.If these cache-duration settings conflict with each other, then CloudFront might not cache objects for the set time, or CloudFront may not cache at all. Avoid these examples of cache-duration settings that conflict with each other:If you set the Maximum TTL to 5 minutes (300 seconds) and the Cache-Control max-age header to 1 hour (3600 seconds), then CloudFront caches the object for 5 minutes instead of 1 hour.If you set the Cache-Control max-age header to 3 hours and the Expires header to 1 month, then CloudFront caches the object for 3 hours instead of 1 month.If you set the Default, Minimum, and Maximum TTL to 0 seconds, then CloudFront always verifies that it has the most recent content from the origin.Important: If you update an origin asset with a caching header, then CloudFront reflects the change only after CloudFront makes a new request to your origin. CloudFront makes a new request to your origin when the cached asset expires in the edge locations. Then, your distribution receives a new request for the asset. To force CloudFront to reflect the update before the cached asset is set to expire, invalidate the cache.Related informationObject cachingFollow"


Question: How can I create a backup of my AWS DMS task settings using the AWS CLI and jq?

I want to take a backup of my AWS Database Migration Service (AWS DMS) replication task settings to restore at a later time or to use as a template for other tasks. How can I do this?

Answer: "I want to take a backup of my AWS Database Migration Service (AWS DMS) replication task settings to restore at a later time or to use as a template for other tasks. How can I do this?Short descriptionUse the AWS Command Line Interface (AWS CLI) and jq command-line JSON processor to create a backup of your AWS DMS replication task settings.Run the AWS CLI describe-replication-tasks command to get your replication task settings.Use the jq command-line JSON processor to filter and transform the task settings.Save the JSON.ResolutionNote: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.PrerequisitesInstall the AWS CLI.Download the jq tool.Get the replication task settingsTo get your AWS DMS replication task settings, run the describe-replication-tasks command using the AWS CLI and use the --filters flag. By using the filters flag, you return only the task that matches one of these items:The replication task IDThe replication task ARNRun this query to get the replication task settings for your AWS DMS task using the replication task ID. Replace <dms_task_id> with your AWS DMS replication task ID.aws dms describe-replication-tasks --filter Name=replication-task-id,Values=<dms_task_id> --output jsonRun this query to get the replication task settings for your AWS DMS task using the replication task ARN. Replace <dms_task_arn> with your AWS DMS replication task ARN.aws dms describe-replication-tasks --filter Name=replication-task-id,Values=<dms_task_arn> --output jsonFilter and transform the JSON output using jqNext, filter and transform the JSON output from the previous step. You need to filter and transform this JSON to make it compatible for restoring at a later time.1.    Use the jq command-line JSON processor to make these modifications to the output JSON:Add the key CdcStartPosition, and then set it to RecoveryCheckpoint.Remove this data:ReplicationTaskArnReplicationTaskStartDateReplicationTaskStatsStatusStopReasonReplicationTaskCreationDateCloudWatchLogGroupCloudWatchLogStreamRecoveryCheckpoint2.    Run one of these commands to pipe the output JSON to jq, and then filter and transform the replication task data:Using the AWS DMS replication task ID:aws dms describe-replication-tasks --filter Name=replication-task-id,Values=<dms_task_id> --output json | jq '.ReplicationTasks | .[] | .TableMappings |= fromjson | .ReplicationTaskSettings |= fromjson | .["CdcStartPosition"] = .RecoveryCheckpoint | delpaths([    ["ReplicationTaskArn"],["ReplicationTaskStartDate"],["ReplicationTaskStats"],["Status"],["StopReason"],["ReplicationTaskCreationDate"],["ReplicationTaskSettings","Logging","CloudWatchLogGroup"],["ReplicationTaskSettings","Logging","CloudWatchLogStream"],["RecoveryCheckpoint"]])    | .TableMappings |= tostring |.ReplicationTaskSettings |= tostring'Using the AWS DMS replication task ARN:aws dms describe-replication-tasks --filter Name=replication-task-arn,Values=<dms-task-arn> --output json | jq '.ReplicationTasks | .[] | .TableMappings |= fromjson | .ReplicationTaskSettings |= fromjson | .["CdcStartPosition"] = .RecoveryCheckpoint | delpaths([    ["ReplicationTaskArn"],["ReplicationTaskStartDate"],["ReplicationTaskStats"],["Status"],["StopReason"],["ReplicationTaskCreationDate"],["ReplicationTaskSettings","Logging","CloudWatchLogGroup"],["ReplicationTaskSettings","Logging","CloudWatchLogStream"],["RecoveryCheckpoint"]])    | .TableMappings |= tostring |.ReplicationTaskSettings |= tostring'Save the output to a JSON fileTo save the resulting output as a backup, redirect it to a JSON file. For example, this command saves the task settings for an AWS DMS task with the task ID of dms_original_task to a JSON file named task_backup.json:aws dms describe-replication-tasks --filter Name=replication-task-id,Values=original-dms-task --output json | jq '.ReplicationTasks | .[] | .TableMappings |= fromjson | .ReplicationTaskSettings |= fromjson | .["CdcStartPosition"] = .RecoveryCheckpoint | delpaths([ ["ReplicationTaskArn"],["ReplicationTaskStartDate"],["ReplicationTaskStats"],["Status"],["StopReason"],["ReplicationTaskCreationDate"],["ReplicationTaskSettings","Logging","CloudWatchLogGroup"],["ReplicationTaskSettings","Logging","CloudWatchLogStream"],["RecoveryCheckpoint"]]) | .TableMappings |= tostring |.ReplicationTaskSettings |= tostring' > task_backup.jsonThe resulting JSON file shows an output similar to this:{  "ReplicationTaskIdentifier": "original-dms-task",  "SourceEndpointArn": "arn:aws:dms:us-east-1:xxxxxxxxxxxx:endpoint:QKMQN2TCULCPYIK3CMGF2CRDC5DIWZ433BCKHZY",  "TargetEndpointArn": "arn:aws:dms:us-east-1:xxxxxxxxxxxx:endpoint:6LEFAZCDPQD2HWHJIXJYUU4PGMTKA2AKKFN4KCA",  "ReplicationInstanceArn": "arn:aws:dms:us-east-1:xxxxxxxxxxxx:rep:C4CAB7ANKQU2AU7WFH4BYCJSLPS3YYUGCTHL3UA",  "MigrationType": "full-load-and-cdc",  "TableMappings": "{\"rules\":[{\"rule-type\":\"selection\",\"rule-id\":\"293111713\",\"rule-name\":\"293111713\",\"object-locator\":{\"schema-name\":\"%\",\"table-name\":\"%\"},\"rule-action\":\"include\",\"filters\":[]}]}",  "ReplicationTaskSettings": "{\"Logging\":{\"EnableLogging\":true,\"LogComponents\":[{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TRANSFORMATION\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"SOURCE_UNLOAD\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"IO\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TARGET_LOAD\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"PERFORMANCE\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"SOURCE_CAPTURE\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"SORTER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"REST_SERVER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"VALIDATOR_EXT\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TARGET_APPLY\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TASK_MANAGER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"TABLES_MANAGER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"METADATA_MANAGER\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"FILE_FACTORY\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"COMMON\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"ADDONS\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"DATA_STRUCTURE\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"COMMUNICATION\"},{\"Severity\":\"LOGGER_SEVERITY_DEFAULT\",\"Id\":\"FILE_TRANSFER\"}]},\"StreamBufferSettings\":{\"StreamBufferCount\":3,\"CtrlStreamBufferSizeInMB\":5,\"StreamBufferSizeInMB\":8},\"ErrorBehavior\":{\"FailOnNoTablesCaptured\":true,\"ApplyErrorUpdatePolicy\":\"LOG_ERROR\",\"FailOnTransactionConsistencyBreached\":false,\"RecoverableErrorThrottlingMax\":1800,\"DataErrorEscalationPolicy\":\"SUSPEND_TABLE\",\"ApplyErrorEscalationCount\":0,\"RecoverableErrorStopRetryAfterThrottlingMax\":true,\"RecoverableErrorThrottling\":true,\"ApplyErrorFailOnTruncationDdl\":false,\"DataTruncationErrorPolicy\":\"LOG_ERROR\",\"ApplyErrorInsertPolicy\":\"LOG_ERROR\",\"EventErrorPolicy\":\"IGNORE\",\"ApplyErrorEscalationPolicy\":\"LOG_ERROR\",\"RecoverableErrorCount\":-1,\"DataErrorEscalationCount\":0,\"TableErrorEscalationPolicy\":\"STOP_TASK\",\"RecoverableErrorInterval\":5,\"ApplyErrorDeletePolicy\":\"IGNORE_RECORD\",\"TableErrorEscalationCount\":0,\"FullLoadIgnoreConflicts\":true,\"DataErrorPolicy\":\"LOG_ERROR\",\"TableErrorPolicy\":\"SUSPEND_TABLE\"},\"ValidationSettings\":{\"ValidationPartialLobSize\":0,\"PartitionSize\":10000,\"RecordFailureDelayLimitInMinutes\":0,\"SkipLobColumns\":false,\"FailureMaxCount\":10000,\"HandleCollationDiff\":false,\"ValidationQueryCdcDelaySeconds\":0,\"ValidationMode\":\"ROW_LEVEL\",\"TableFailureMaxCount\":1000,\"RecordFailureDelayInMinutes\":5,\"MaxKeyColumnSize\":8096,\"EnableValidation\":true,\"ThreadCount\":5,\"RecordSuspendDelayInMinutes\":30,\"ValidationOnly\":false},\"TTSettings\":{\"TTS3Settings\":null,\"TTRecordSettings\":null,\"EnableTT\":false},\"FullLoadSettings\":{\"CommitRate\":10000,\"StopTaskCachedChangesApplied\":false,\"StopTaskCachedChangesNotApplied\":false,\"MaxFullLoadSubTasks\":8,\"TransactionConsistencyTimeout\":600,\"CreatePkAfterFullLoad\":false,\"TargetTablePrepMode\":\"DROP_AND_CREATE\"},\"TargetMetadata\":{\"ParallelApplyBufferSize\":0,\"ParallelApplyQueuesPerThread\":0,\"ParallelApplyThreads\":0,\"TargetSchema\":\"\",\"InlineLobMaxSize\":0,\"ParallelLoadQueuesPerThread\":0,\"SupportLobs\":true,\"LobChunkSize\":0,\"TaskRecoveryTableEnabled\":false,\"ParallelLoadThreads\":0,\"LobMaxSize\":32,\"BatchApplyEnabled\":false,\"FullLobMode\":false,\"LimitedSizeLobMode\":true,\"LoadMaxFileSize\":0,\"ParallelLoadBufferSize\":0},\"BeforeImageSettings\":null,\"ControlTablesSettings\":{\"historyTimeslotInMinutes\":5,\"HistoryTimeslotInMinutes\":5,\"StatusTableEnabled\":false,\"SuspendedTablesTableEnabled\":false,\"HistoryTableEnabled\":false,\"ControlSchema\":\"\",\"FullLoadExceptionTableEnabled\":false},\"LoopbackPreventionSettings\":null,\"CharacterSetSettings\":null,\"FailTaskWhenCleanTaskResourceFailed\":false,\"ChangeProcessingTuning\":{\"StatementCacheSize\":50,\"CommitTimeout\":1,\"BatchApplyPreserveTransaction\":true,\"BatchApplyTimeoutMin\":1,\"BatchSplitSize\":0,\"BatchApplyTimeoutMax\":30,\"MinTransactionSize\":1000,\"MemoryKeepTime\":60,\"BatchApplyMemoryLimit\":500,\"MemoryLimitTotal\":1024},\"ChangeProcessingDdlHandlingPolicy\":{\"HandleSourceTableDropped\":true,\"HandleSourceTableTruncated\":true,\"HandleSourceTableAltered\":true},\"PostProcessingRules\":null}",  "CdcStartPosition": "checkpoint:V1#102#00000000/B803EBA0#0#0#*#0#277"}You can now edit and restore the JSON file using the create-replication-task AWS CLI command. For example, this command creates an AWS DMS replication task with a task ID of restored-dms-task using the task_backup.json file:aws dms create-replication-task --replication-task-identifier restored-dms-task --cli-input-json file://task_backup.jsonNote: When you include the --replication-task-identifier in the command, then you override the --replication-task-identifier value in the task_backup.json file.Related informationSpecifying task settings for AWS Database Migration Service tasksCreating tasks for ongoing replication using AWS DMSFollow"