Question: How do I troubleshoot a 502: "The request could not be satisfied" error from CloudFront?

"I configured an Amazon CloudFront distribution with a custom domain. When requesting the alternative Canonical Name record (CNAME) domain over CloudFront, I get a 502 Error response with the message "The request could not be satisfied.""

Answer: "I configured an Amazon CloudFront distribution with a custom domain. When requesting the alternative Canonical Name record (CNAME) domain over CloudFront, I get a 502 Error response with the message "The request could not be satisfied."Short descriptionA 502 error occurs when CloudFront is unable to connect to the origin. See the following sections for the causes of the error, and how to troubleshoot.ResolutionCloudFront can't establish a TCP connection with the origin serverBy default, CloudFront connects to the origin over port 80 (for HTTP) and port 443 (for HTTPS). If the origin doesn't allow traffic over these ports, or blocks the CloudFront IP address's connection, then the TCP connection fails. The failure produces a 502 error. To resolve, confirm that the CloudFront distribution's Protocol setting is set to the correct port for HTTP or HTTPS connections.To test port connectivity, run the following command:telnet ORIGIN_DOMAIN/ORIGIN_IP PORTNote: For ORIGIN_DOMAIN, enter the ID of your origin domain. For ORIGIN_IP, enter the IP address of your origin. For PORT, enter the port number you're using to connect to the origin.SSL/TLS negotiation with the origin server failedIf the SSL/TLS transaction fails, then the connection between CloudFront and the origin fails and produces a 502 error. See the following sections for causes of an SSL/TLS transaction failure, and how to resolve them.SSL certificate doesn't match the domain nameThe SSL certificate at the origin must include or cover one of the following domain names:The origin domain name in the certificate's Common Name field or Subject Alternative Names field.The host header's domain name for incoming viewer host headers that are forwarded to the origin in the CloudFront distribution.To check for the Common Name and Subject Alternative Names in the certificate, run the following command:$ openssl s_client -connect DOMAIN:443 -servername SERVER_DOMAIN | openssl x509 -text | grep -E '(CN|Alternative)' -A 2Note: For DOMAIN, enter the origin domain name. For SERVER_DOMAIN enter the origin domain name. Or, if the viewer host header is forwarded to the origin), for SERVER_DOMAIN, enter the incoming host header value.If the following are true, then configure the cache policy or origin request policy to include the host header:The SSL certificate's common name or subject alternate name (SAN) includes the viewer host header value.The host header is not forwarded to the origin.Origin Certificate is expired, not trusted, or self-signedThe certificate installed the custom origin must be signed by a trusted Certificate Authority. The certificate authorities trusted by CloudFront are found on the Mozilla included CA certificate list on the Mozilla website.CloudFront doesn't support self-signed certificates for SSL set up with the origin. Self-signed certificates are issued by the organizations themselves or generated locally on a web-server instead of being issued by a trusted Certificate Authority.To check if your origin certificate is expired, run the following OpenSSL command. In the output, find the Not Before and Not After parameters. Confirm that the current date and time is within the certificate's validity period.$ openssl s_client -connect DOMAIN:443 -servername SERVER_DOMAIN | openssl x509 -text | grep Validity -A 3Note: For DOMAIN, enter the origin domain name. For SERVER_DOMAIN enter the origin domain name. Or, if the viewer host header is forwarded to the origin), for SERVER_DOMAIN, enter the incoming host header value.Missing intermediate CA certificates or the incorrect order of intermediate certificates causes failure between HTTPS communication and the origin. To check the certificate chain, run the following command.$ openssl s_client -showcerts -connect DOMAIN:443 -servername SERVER_DOMAINNote: For DOMAIN, enter the origin domain name and for SERVER_DOMAIN enter the origin domain name. Or, if the viewer host header is forwarded to the origin), for SERVER_DOMAIN, enter the incoming host header value.The origin's cipher suit isn't supported by CloudFrontSSL/TLS transactions between CloudFront and Origin fail if there is no common negotiated ciphers suite. To confirm you're using the correct cipher suite, see Supported protocols and ciphers between CloudFront and the origin.You can also use an SSL server test tool to check if your origin domain name is included on the list of supported ciphers.CloudFront can't resolve the origin IP addressIf CloudFront can't resolve the origin domain, then it returns 502 error. To troubleshoot this, use a dig/nslookup command to check if the origin domain resolves to an IP.Linux:$ dig ORIGIN_DOMAIN_NAMEWindows:nslookup ORIGIN_DOMAIN_NAMENote: For ORIGIN_DOMAIN_NAME, enter the origin domain name.If successful, then the command returns the IP of the origin domain name. Use a DNS checker tool to check for DNS resolution across different geographies.The error is caused by upstream originThe custom origin defined in the CloudFront distribution can be a proxy, Content Delivery Network (CDN) hostname, or load balancer connected to the actual origin. If any of these intermediary services fails to connect to the origin, then a 502 error is returned to CloudFront. To resolve this, work with your service provider.The Lambda@Edge function associated with the CloudFront distribution failed validationIf the Lambda@Edge function returns a response to CloudFront that's not valid, then CloudFront returns a 502 error. To resolve this, check your Lambda@Edge function for the following common issues:Returned JSON objectMissing required fieldsInvalid object in the responseAdding or updating disallowed or read-only headersExceeding the maximum body sizeCharacters or values that aren't validFor more information, see Testing and debugging Lambda@Edge functions.Follow"


Question: How do I troubleshoot issues related to scheduled tasks in Amazon ECS?

"I have scheduled my Amazon Elastic Container Service (Amazon ECS) task to run periodically. However, my Amazon ECS task isn't triggered. I'm not getting execution logs or the history of the tasks in the cluster."

Answer: "I have scheduled my Amazon Elastic Container Service (Amazon ECS) task to run periodically. However, my Amazon ECS task isn't triggered. I'm not getting execution logs or the history of the tasks in the cluster.ResolutionWhen you use a scheduled Amazon ECS task, Amazon CloudWatch Events calls the RunTask API to Amazon ECS to run the tasks on your behalf.Your scheduled Amazon ECS task might not be invoked due to the following reasons:The Amazon EventBridge time or cron expression is configured incorrectly.The EventBridge rule doesn't invoke the target.The RunTask API failed to run.The container exit due to application issues or resource constraints.Check whether the EventBridge cron expression is configured incorrectlyTo get the EventBridge cron expression, run the following AWS Command Line Interface (AWS CLI) command:$ aws events describe-rule --name "example-rule" --region example-regionIn the output of the command, you can view the configured EventBridge cron expression in the parameter ScheduleExpression. Be sure that you set the schedule for the rule in UTC+0 time zone.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.Check whether the rule doesn't invoke the targetUse the Amazon CloudWatch metrics generated by EventBridge to view the rule performance. Invocation datapoints indicate that the target was invoked by the rule. If FailedInvocations data points are present, then there is an issue invoking the target. FailedInvocations represent a permanent failure and might be the result of incorrect permissions or a misconfiguration of the target.To review the CloudWatch metrics for the EventBridge rule, do the following:Open the CloudWatch console.In the navigation pane, choose Metrics, and then choose All metrics.Choose Events.Choose By Rule Name.Select the TriggerRules, Invocations, and FailedInvocations (if available) metrics for the EventBridge rule that's configured to run the ECS task.Choose the Graphed metrics tab.For all metrics listed, select SUM for Statistic.If FailedInvocations data points are present, there might be an issue related to inadequate target permissions. Be sure that EventBridge has access to invoke your ECS task. Verify that the EventBridge AWS Identity and Access Management (IAM) role has the required permissions. For more information, see Amazon ECS CloudWatch Events IAM Role.Check whether the RunTask action failed to runTo verify if the RunTask API failed to run, search in AWS CloudTrail event history for RunTask within the time range of when the scheduled ECS task was expected to be invoked.To find if the scheduled task wasn't invoked because the RunTask action failed, do the following:Open the AWS CloudTrail console.In the navigation pane, choose Event history.In the Event history page, for Lookup attributes, select Event name.For Enter an event name, enter RunTask.Choose the time range in the time range filter based on when the scheduled ECS task was expected to run.Note: The preset values for time range are 30 minutes, 1 hour, 3 hours, and 12 hours. To specify a custom time range, choose Custom.From the results list, choose the event that you want to view.Scroll to Event record on the Details page to view the JSON event record.Look for errorMessage or responseElements.failures.reason elements in the JSON event record.These elements in the JSON event record display the reason for the scheduled ECS task not being invoked.For examples of RunTask API failure reasons and their causes, see API failure reasons.Check whether the container exited after the task ranThe Amazon ECS tasks might be stopped even after the task runs successfully due to application issues or resource constraints. For more information, see How do I troubleshoot issues with containers exiting in my Amazon ECS tasks?Related informationHow do I troubleshoot Amazon ECS tasks on Fargate that stop unexpectedly?Follow"


Question: Why can't I view conversation logs for Amazon Lex in CloudWatch?

I am unable to view the conversation logs for Amazon Lex in Amazon CloudWatch.

Answer: "I am unable to view the conversation logs for Amazon Lex in Amazon CloudWatch.Short descriptionThere are several reasons why you might not see your Amazon Lex conversation logs in CloudWatch. For example, you might not have the right permissions configured to allow Amazon Lex use CloudWatch logs. Or, you might have enabled COPPA on your bot which doesn't allow you to use the conversation logs feature.Use the troubleshooting steps in this article to find the root cause of this issue.ResolutionAdd an IAM role and policy to Amazon LexCheck if you have granted the correct permissions to allow your Amazon Lex bot to log to CloudWatch. To log conversation logs, Amazon Lex needs to use CloudWatch logs and access Amazon Simple Storage Service (Amazon S3) buckets to store conversation logs. Follow these steps to add the required AWS Identity and Access Management (IAM) roles and policies using the Amazon Lex console.1.    Open the Amazon Lex console, and choose the bot that you want to edit.2.    Choose Settings, and then choose Conversation logs.3.    Choose the settings icon, and then choose IAM role.4.    Add an IAM role with trust relationship similar to this:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Service": "lex.amazonaws.com"   //For V2 "Service": "lexv2.amazonaws.com"      },      "Action": "sts:AssumeRole"    }  ]}Attach an IAM policy to the role that allows logging of conversation text to CloudWatch logs:{  "Version": "2012-10-17",  "Statement": [      {          "Effect": "Allow",          "Action": [              "logs:CreateLogStream",              "logs:PutLogEvents"          ],          "Resource": "arn:aws:logs:region:account-id:log-group:log-group-name:*"      }  ]}6.    Add an IAM policy to the role that allows audio logging to an S3 bucket:{  "Version": "2012-10-17",  "Statement": [      {          "Effect": "Allow",          "Action": [              "s3:PutObject"          ],          "Resource": "arn:aws:s3:::bucket-name/*"      }  ]}Review your COPPA settingsCheck if COPPA is enabled for your bot. If you enabled COPPA, then you can't use the conversation logs feature for that bot.Using Amazon Lex V1To check this setting, check the general settings of your bot using the Amazon Lex console.1.    Open the Amazon Lex console, and then choose Return to the V1 console.2.    Choose the bot that you want to edit.2.    Choose Settings, and then choose General.3.    Choose COPPA.Use Amazon Lex V21.    Open the Amazon Lex V2 console, and choose bot versions.2.    Choose the version that you want to use, and then choose COPPA.3.    If COPPA is enabled for a version you want to use, you can't disable it. Instead, go to Draft versions, and choose COPPA. You can now change COPPA to no, and publish a new version.Further troubleshooting steps1.    Check that the log group you're using is in the same Region as your Amazon Lex bot.2.    Check that the bot alias that you're using and the alias that you specified logging for are the same. Conversation logs are configured according to bot alias, so it's important that they match.3.    Check that you aren't using $LATEST alias or a test bot that Amazon Lex provides for testing. You can't log conversations for either of these.4.    Check that you haven't enabled AI services opt out policies in your AWS organization. If you enable opt-out policies, then Amazon Lex doesn't log conversation logs.Related informationConversation logsIAM policies for conversation logsMonitoring with conversation logsFollow"


Question: I'm trying to upload a large file using the Amazon S3 console. Why is the upload failing?

"I'm trying to upload a large file (1 GB or larger) to Amazon Simple Storage Service (Amazon S3) using the console. However, the upload persistently fails and I'm getting timeout errors. How do I resolve this?"

Answer: "I'm trying to upload a large file (1 GB or larger) to Amazon Simple Storage Service (Amazon S3) using the console. However, the upload persistently fails and I'm getting timeout errors. How do I resolve this?ResolutionFor large files, Amazon S3 might separate the file into multiple uploads to maximize the upload speed. The Amazon S3 console might time out during large uploads because of session timeouts. Instead of using the Amazon S3 console, try uploading the file using the AWS Command Line Interface (AWS CLI) or an AWS SDK.Note: If you use the Amazon S3 console, the maximum file size for uploads is 160 GB. To upload a file that is larger than 160 GB, use the AWS CLI, AWS SDK, or Amazon S3 REST API.AWS CLIFirst, install and configure the AWS CLI. Be sure to configure the AWS CLI with the credentials of an AWS Identity and Access Management (IAM) user or role. The IAM user or role must have the correct permissions to access Amazon S3.Important: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.To upload a large file, run the cp command:aws s3 cp cat.png s3://docexamplebucketNote: The file must be in the same directory that you're running the command from.When you run a high-level (aws s3) command such as aws s3 cp, Amazon S3 automatically performs a multipart upload for large objects. In a multipart upload, a large file is split into multiple parts and uploaded separately to Amazon S3. After all the parts are uploaded, Amazon S3 combines the parts into a single file. A multipart upload can result in faster uploads and lower chances of failure with large files.For more information on multipart uploads, see How do I use the AWS CLI to perform a multipart upload of a file to Amazon S3?AWS SDKFor a programmable approach to uploading large files, consider using an AWS SDK, such as the AWS SDK for Java. For an example operation, see Upload an object using the AWS SDK for Java.Note: For a full list of AWS SDKs and programming toolkits for developing and managing applications, see Tools to build on AWS.Related informationUsing Amazon S3 with the AWS CLIFollow"


Question: How do I integrate Amazon Connect with Zendesk?

I want to use Amazon Connect with Zendesk. How do I set that up?

Answer: "I want to use Amazon Connect with Zendesk. How do I set that up?Short descriptionInstall and configure the Amazon Connect for Zendesk app in your Zendesk Support account, then integrate the app with Amazon Connect. After integration, you can create contact flows to use Amazon Connect with Zendesk ticketing.For this setup, you need the following:An Amazon Connect instance.A Zendesk Support account with a Zendesk Talk Partner Edition plan, or a Zendesk trial account.ResolutionGet your instance's access URL1.    Open the Amazon Connect console.2.    Under Instance Alias, choose your instance's alias.3.    On the Overview pane, copy the Access URL for your instance.Important: If the URL looks like the following, then copy through the .com only: https://instance-name.awsapps.com/connect/login (For example: https://instance-name.awsapps.com)-or-If the URL looks like the following, then copy the complete URL: https://instance-name.my.connect.awsInstall the Amazon Connect for Zendesk app1.    Sign in to your Zendesk account.2.    On the Amazon Connect app page of the Zendesk Marketplace website, choose Install.3.    In the APP INSTALLATION dialog box, confirm that your Zendesk account is selected. Then, choose Install.4.    In your Zendesk, under INSTALLATION, do the following:For Amazon Connect URL, paste the access URL for your instance that you copied earlier.For Default entry point phone number, enter a number.For Contact Flow attribute containing Zendesk Ticket Number, copy the value "zendesk_ticket". You need this attribute value later in this setup.(Optional) Update the other settings based on your use case.Note: You can change these settings in your Zendesk at any time after installation.5.    Choose Install.Integrate Zendesk with Amazon Connect1.    Refresh the Zendesk account's browser tab. The Amazon Connect app icon appears at the top-right of your Zendesk instance.2.    Select the Amazon Connect app icon at the top-right corner. Two URLs appear.3.    Copy the two URLs one at a time. Then, add them to the Approved domains list in the Approved origins section of your Amazon Connect instance by doing the following:Open the Amazon Connect console. Then, in the left navigation pane, choose Approved origins.Under Approved origins, choose Add domain. An Add domain text box appears.In the Add domain text box, under Enter domain URL, paste one of the URLs that you copied earlier.Choose Add.Important: You can add one URL at a time to the Approved domains list only. You must repeat step 3 of this section to add the second URL.4.    In a new browser tab, log in to the Amazon Connect dashboard either as a created user or by choosing Log in for emergency access.5.    In your Zendesk, select the Amazon Connect app icon. The Amazon Connect Contact Control Panel (CCP) opens in Zendesk.(Optional) Create a contact flow for Zendesk ticketingFollow these steps to create an example contact flow. This contact flow prompts callers for input (for example, an account number or ticket number). Then, the flow automatically generates a Zendesk ticket with the caller's information.Note: It's a best practice to include Check hours of operation and Check staffing blocks before transferring a call to an agent. These blocks verify that the call is within working hours and that agents are staffed to service the call. However, these blocks are not required to achieve the desired functionality of this setup. For more information, see Best practices for Amazon Connect.Create a contact flow1.    In the Amazon Connect console, under Access URL, choose the access URL for your instance.2.    Log in to your instance using the administrator account.3.    In the left navigation bar, pause on Routing. Then, choose Contact flows.4.    Under Contact flows, choose a template, or choose Create contact flow to design a contact flow from scratch. For more information, see Create a new contact flow.5.    In the contact flow designer, next to Save, choose the arrow icon. Then, choose Save as.Note: This option saves your contact flow as a new version, instead of overwriting the original.6.    In the Save as text box, enter a New name and Description for your contact flow.7.    Choose Save as.Add a Store customer input block1.    In the contact flow designer, choose Interact. Then, drag and drop a Store customer input block onto the canvas.2.    Choose the block title (Store customer input). The block's settings menu opens.3.    Under Prompt, do the following:Choose Text to speech (Ad hoc).For Enter text, enter a message that asks the customer for their input. For example, "Please enter a valid Zendesk ticket number followed by #." For more information, see Create prompts.4.    Under Customer input, do the following:Choose Custom.(Optional) For Maximum Digits, enter a maximum number of characters that callers can enter.(Optional) For Delay between entry, enter a number of seconds that callers have between entering each digit before timeout.For Encrypt entry (recommended), select the check box and configure encryption. For more information, see Encrypt customer input and Creating a secure IVR solution with Amazon Connect.5.    Choose Save.For more information, see Use Amazon Connect contact attributes.Add a Set contact attributes block1.    In the contact flow designer, choose Set. Then, drag and drop a Set contact attributes block onto the canvas.2.    Choose the block title (Set contact attributes). The block's settings menu opens.3.    Under Attribute to save, choose Use attribute. Then, do the following:For Destination key, enter "zendesk_ticket".For Type, choose System.For Attribute, choose Stored customer input.4.    Choose Save.For more information, see Use Amazon Connect contact attributes.Set up call transfersAdd a Set working queue block and a Transfer to queue block.For more information, see Set up contact transfers.Add a Disconnect / hang up blockFrom the contact flow designer, choose Terminate / Transfer. Then, drag and drop a Disconnect / hang up block onto the canvas.Connect the blocksIn the contact flow designer, drag the arrows from each block to the block that you want to perform the next action. All connectors must be connected to a block before you can publish the contact flow.As you connect the contact blocks, make sure that you do the following for each block:Connect Start or Success to the next block in the following order:Entry point > Store customer input > Set contact attributes > Set working queue > Transfer to queueConnect Error and At capacity to the Disconnect / hang up block.Save and publish the contact flow1.    Choose Save to save a draft of the flow.2.    Choose Publish to activate the flow immediately.For more information, see Create a new contact flow.Related informationZendesk (AWS Partner Network)Setup pre-built integrationsFollow"


Question: How do I Provide IAM users with a link to assume an IAM role?

How do I provide AWS Identity and Access Management (IAM) users links to an IAM role?

Answer: "How do I provide AWS Identity and Access Management (IAM) users links to an IAM role?Short descriptionYou can provide the IAM user with the URL link of the IAM role that you want them to assume.ResolutionFollow these instructions to get the IAM role link that you want the IAM user to assume. The following example uses IAM user Bob.1.    If you haven't already done so, follow the instructions for creating an IAM user in your AWS account. Then, follow the instructions for creating an IAM role.2.    Choose the role name. In the Summary pane, in Link to switch roles in console, copy the link. The link looks similar to the following: https://signin.aws.amazon.com/switchrole?roleName=YOURROLE&account=1234567890123.    Provide the link to IAM user Bob.4.    Bob opens the IAM console, and then pastes the link into the browser window.5.    The Switch Role page opens for Bob. In Display Name, enter Bob.6.    (Optional) Choose a Color for Bob.7.    Choose Switch Role.Bob now assumes the IAM role that you provided.For more information, see Things to know about switching roles in the console.Related informationGranting permissions to an IAM user to switch rolesSwitching to a role (console)Follow"


Question: "How do I troubleshoot "Neither the global service principal states.amazonaws.com, nor the regional one is authorized to assume the provided role" errors in AWS Step Functions?"

"When I try to run my AWS Step Functions state machine, I receive the following error: "Neither the global service principal states.amazonaws.com, nor the regional one is authorized to assume the provided role." How do I troubleshoot the issue?"

Answer: "When I try to run my AWS Step Functions state machine, I receive the following error: "Neither the global service principal states.amazonaws.com, nor the regional one is authorized to assume the provided role." How do I troubleshoot the issue?ResolutionVerify that the AWS Identity and Access Management (IAM) role that your state machine assumes has the required trust relationships configuredOne of the following must be listed as a trusted entity in the IAM role's trust policy:An AWS Regional endpoint: states.<region>.amazonaws.comThe AWS global endpoint: states.amazonaws.comTo review and edit the trust policy of the IAM role that your state machine assumes, follow the instructions in Modifying a role trust policy (console). For more information, see How AWS Step Functions works with IAM.Note: When the StartExecution API action is called, Step Functions uses the IAM role that's associated with the state machine during the duration of the action's runtime. If the IAM role that the state machine assumes is changed during the action's runtime, then the IAM role isn't used on that API action.Verify that the IAM role that your state machine assumes still exists1.    Open the Step Functions console.2.    In the left navigation pane, choose State machines.3.    Select the name of your state machine.4.    In the Details section, choose the link under IAM role ARN. If the IAM role exists, the role opens in the IAM console. If the IAM role doesn't exist, the IAM console opens a page that says No Entity Found.If the IAM role that your state machine assumes doesn't exist, create a new IAM role with a different name that includes the required permissions. Then, configure your state machine to assume the new IAM role that you created. For more information, see How AWS Step Functions works with IAM.Important: The new IAM role that you create must have a different name than the previous IAM role.Follow"


Question: Why are emails failing to deliver when I send Amazon SES emails using the SendTemplatedEmail operation?

"I'm using the SendTemplatedEmail operation to send messages from my Amazon Simple Email Service (Amazon SES) account. However, Amazon SES doesn't deliver some emails."

Answer:
"I'm using the SendTemplatedEmail operation to send messages from my Amazon Simple Email Service (Amazon SES) account. However, Amazon SES doesn't deliver some emails.ResolutionWhen you use an email template, Amazon SES validates that the template data you send includes the required variables in the template. If the template data contains non-compliant variables or is missing variables, then Amazon SES can't deliver the email. This is called a Rendering Failure.Use Amazon Simple Notification Service (Amazon SNS) to set up Rendering Failure event notifications. Review the Rendering Failure event notifications to find out why Amazon SES doesn't deliver an email when you use the SendTemplatedEmail operation.After you set up Rendering Failure event notifications, you receive an Amazon SNS notification when a templated email delivery fails. The notification error message includes information about the template variable that led to the Rendering Failure.For example, the following template contains the variables name and favoritecolor:{  "Template": {    "TemplateName": "ExampleTemplate",    "SubjectPart": "Hello, {{name}}!",    "HtmlPart": "
Hello {{name}},

Your favorite color is {{favoritecolor}}.
",    "TextPart": "Dear {{name}},\r\nYour favorite color is {{favoritecolor}}."  }}If you send the following template data, then Amazon SES can't deliver the email. This is because the favoritecolor variable is missing from the template.Important: Including extra variables that aren't present in the template, such as favoritenumber, doesn't cause an error. However, all variables that you include in the template must have an exact case-sensitive counterpart in the template data. See the following example:"TemplateData": "{ \"name\":\"Jane\", \"favoritenumber\": \"10\" }"With Rendering Failure event notifications, you receive a failure notification that's similar to the following message:{    "eventType": "Rendering Failure",    "mail": {        "timestamp": "2019-09-09T04:38:19.788Z",        "source": "sender@example.com",        "sourceArn": "arn:aws:ses:us-west-2:1234567890123:identity/sender@example.com",        "sendingAccountId": "1234567890123",        "messageId": "01010161a734a0eb-a706827a-3bda-490f-8eaa-63cf4b00d10c-000000",        "destination": [            "receiver@example.com"        ],        "headersTruncated": false,        "tags": {            "ses:configuration-set": [                "RenderFailure"            ]        }    },    "failure": {        "errorMessage": "Attribute 'favoritecolor' is not present in the rendering data.",        "templateName": "ExampleTemplate"    }}To avoid Rendering Failures, follow these guidelines:Check the capitalization of variable names in your template data. Variable names in the template are case sensitive.Verify that your template data includes all the variables in the template.Related informationUsing templates to send personalized email with the Amazon SES APIFollow"


Question: How do I resolve the "cannotpullcontainererror" error for my Amazon ECS tasks on Fargate?

I want to resolve the "cannotpullcontainererror" error so that I can start my Amazon Elastic Container Service (Amazon ECS) tasks on AWS Fargate.

Answer: "I want to resolve the "cannotpullcontainererror" error so that I can start my Amazon Elastic Container Service (Amazon ECS) tasks on AWS Fargate.Short descriptionThe "cannotpullcontainererror" error can prevent tasks from starting. To start an Amazon ECS task on Fargate, your Amazon Virtual Private Cloud (Amazon VPC) networking configurations must allow your Amazon ECS infrastructure to access the repository where the image is stored. Without the correct networking, the image can't be pulled by Amazon ECS on Fargate and the container can't start.ResolutionConfirm that your VPC networking configuration allows your Amazon ECS infrastructure to reach the image repositoryThe route tables associated to the subnets that your task is created in must allow your Amazon ECS infrastructure to reach the repository endpoint. The endpoint can be reached through an internet gateway, NAT gateway, or VPC endpoints.If you're not using AWS PrivateLink, then complete the following steps:Open the Amazon VPC console.In the navigation pane, choose Subnets.Select the subnet that your ECS Fargate task is using.Choose the Route Table tab.In the Destination column, confirm that the default route (0.0.0.0/0) of the route table allows public internet access. This access can be either through a NAT gateway or an internet gateway.Important: The NAT gateway or internet gateway must be the target of the default route. For example route tables, see Example routing options. If you're not using a NAT gateway or internet gateway, then make sure that your custom configuration allows public internet access.If you're using an internet gateway (public subnets), then confirm that the task has a public IP assigned to it. To do this, launch your ECS task with Auto-assign public IP set to ENABLED in the VPC and security groups  section when you create the task or service.If you're using PrivateLink, confirm that the security groups for your VPC endpoints allow the Fargate infrastructure to use them.Note: Amazon ECS tasks hosted on Fargate using version 1.3.0 or earlier require the com.amazonaws.region.ecr.dkrAmazon Elastic Container Registry (Amazon ECR) VPC endpoint and the Amazon Simple Storage Service (Amazon S3) gateway endpoint. Amazon ECS tasks hosted on Fargate using version 1.4.0 or later require both the com.amazonaws.region.ecr.dkr and com.amazonaws.region.ecr.api Amazon ECR VPC endpoints and the Amazon S3 gateway endpoint.Open the Amazon VPC console.In the navigation pane, choose Endpoints.Select the endpoint from the list of endpoints, and then choose the Subnets tab. The VPC endpoints com.amazonaws.region.ecr.dkr and com.amazonaws.region.ecr.api for Amazon ECR will display in the list of subnets and associated with the Fargate subnets. You also see the Amazon S3 gateway on the list of subnets.Note: If a subnet isn't listed, choose Manage Subnets. Next, select the subnet based on its Availability Zone. Then, choose Modify Subnets.Choose the Policy tab, and then confirm that the correct policy requirements are met.To confirm that the security group attached to the com.amazonaws.region.ecr.api and com.amazonaws.region.ecr.dkr VPC endpoints allows incoming connections on port 443 from the Amazon ECS tasks for Fargate, select the endpoint from the list of endpoints.Choose the Security Groups tab.For Group ID, choose the security group ID.Choose the Inbound rules tab, and then confirm that you can see the rule that allows 443 connections from your ECS tasks on Fargate.Check the VPC DHCP Option SetOpen the Amazon VPC console.In the navigation pane, choose Your VPCs.Select the VPC that contains your Fargate task.On the Details tab, note the setting for DHCP options set.In the navigation pane, choose DHCP Options Sets.Select the DHCP options set that you noted in step 4.Choose Actions, and then choose View details.Confirm that Domain name servers is set to AmazonProvidedDNS. If it isn't set to AmazonProvidedDNS, then configure conditional DNS forwarding.Check the task execution role permissionsOpen the IAM console. To confirm that the security group attached to the com.amazonaws.region.ecr.api and com.amazonaws.region.ecr.dkr VPC endpoints allows incoming connections on port 443 from the Amazon ECS tasks for Fargate, select the endpoint from the list of endpointsIn the navigation pane, choose Roles.Select the task execution role that your Fargate tasks are using.Confirm that the task execution role has the permissions to pull an image from Amazon ECR.Check that the image existsOpen the Amazon ECR console.Select the Amazon ECR repository that your Fargate task should be pulling the image from.Confirm that the URI and the tag in Amazon ECR are the same as what's specified in the task definition.Note: If you're not using Amazon ECR, then make sure that you see image:tag in the specified image repository.Follow"


Question: Why is my AWS DMS task that uses PostgreSQL as the source failing with all of the replication slots in use?

"I have an AWS Database Migration Service (AWS DMS) task that uses an Amazon Relational Database Service (Amazon RDS) DB instance that is running PostgreSQL as the source. My task is failing, all of the replication slots are in use, and I received an error message. Why is my task failing, and how do I resolve these errors?"

Answer: "I have an AWS Database Migration Service (AWS DMS) task that uses an Amazon Relational Database Service (Amazon RDS) DB instance that is running PostgreSQL as the source. My task is failing, all of the replication slots are in use, and I received an error message. Why is my task failing, and how do I resolve these errors?Short descriptionFor Amazon RDS for PostgreSQL instances, AWS DMS uses native replication slots to perform the logical replication for change data capture (CDC).The number of replication slots that a PostgreSQL instance has is controlled by the max_replication_slots parameter. By default, there are five replication slots for RDS PostgreSQL instances. If you exceed the maximum number of replication slots, then you see log entries like these:Messages[SOURCE_CAPTURE ]E: Failed (retcode -1) to execute statement [1022502] (ar_odbc_stmt.c:2579)[SOURCE_CAPTURE ]E: RetCode: SQL_ERROR SqlState: 53400 NativeError: 1 Message: ERROR: all replication slots are in use;To resolve these errors, remove the used replication slots or increase the value of the max_replication_slots parameter.ResolutionRemove used replication slotsIf you are running multiple AWS DMS tasks or you have old tasks running on same DB instance, then remove used replication slots. These replication slots continue to occupy space, so by removing then you make the slots available for new tasks.First, identify the maximum number of replication slots. Then, remove or "drop" the used replication slots.Run this query to check the maximum number of replication slots:SELECT * FROM pg_replication_slots;     slot_name    |    plugin     | slot_type | datoid | database | active |  xmin  | catalog_xmin | restart_lsn -----------------+---------------+-----------+--------+----------+--------+--------+--------------+-------------old_and_used_slot | test_decoding | logical   |  12052 | postgres | f      |        |          684 | 0/16A4408Run this query to drop a used replication slot:SELECT pg_drop_replication_slot('old_and_used_slot');Note: Replace old_and_used_slot with the name of your replication slot.Increase the value of the max_replication_slots parameterModify the DB parameter in the custom DB parameter groups that is attached to the Amazon RDS DB instance. Then, increase the value of the max_replication_slots parameter. This is a static parameter, so be sure to reboot the DB instance after changing the parameter value.After you remove the used replication slots or increase the value of the max_replication_slots parameter, restart the task.Related informationPostgreSQL on Amazon RDSUsing a PostgreSQL database as an AWS DMS sourcePostgreSQL documentation for Logical decoding examplesFollow"


Question: How do I rebalance the uneven shard distribution in my Amazon OpenSearch Service cluster?

"The disk space in my Amazon OpenSearch Service domain is unevenly distributed across the nodes. As a result, the disk usage is heavily skewed."

Answer: "The disk space in my Amazon OpenSearch Service domain is unevenly distributed across the nodes. As a result, the disk usage is heavily skewed.Short descriptionDisk usage can be heavily skewed because of the following reasons:Uneven shard sizes in a cluster. Although OpenSearch Service evenly distributes the number of shards across nodes, varying shard sizes require different amounts of disk space.Available disk space on a node. For more information, see Disk-based shard allocation on the Elasticsearch website.Incorrect shard allocation strategy. For more information, see Demystifying OpenSearch Service shard allocation.To rebalance the shard allocation in your OpenSearch Service cluster, consider the following approaches:Check the shard allocation, shard sizes, and index sharding strategy.Be sure that shards are of equal size across the indices.Keep shard sizes between 10 GB to 50 GB for better performance.Add more data nodes to your OpenSearch Service cluster.Update your sharding strategy.Delete the old or unused indices to free up disk space.ResolutionCheck the shard allocation, shard sizes, and index sharding strategyTo check the number of shards allocated to each node and the amount of disk space used on each node, use the following API:GET _cat/allocation?vTo check the shards allocated to each node and the size of each shard, use the following API:GET _cat/shards?vNote: This API shows that the size of shards can vary for different indices.The uneven sharding strategy for indices can cause data skewness. In this case, shards of bigger indices reside on only a few nodes. To check the sharding strategy for indices. use the following API:GET _cat/indices?vBe sure that shards are of equal size across the indicesIf the index size varies significantly, then use the rollover index API to create a new index when certain index sizes are reached. Or, you can use the Index State Management (ISM) to create a new index for OpenSearch Service versions 7.1 and later. For more information about rolling an alias using ISM, see rollover on the Open Distro website.Keep shard sizes between 10 GB to 50 GB for better performanceIf you have a large class of instances, then use the Petabyte scale for Amazon OpenSearch Service to determine shard sizes. For example, an OpenSearch Service domain with several i3.16xlarge.search instances can support shard sizes of up to 100 GB because there are more resources available. For more information about sharding strategy, see Choosing the number of shards.Add more data nodes to your OpenSearch Service clusterIf your OpenSearch Service cluster has reached high disk usage levels, then add more data nodes to your cluster. The addition of data nodes also adds more resources to improve cluster performance.Note: OpenSearch Service doesn't automatically rebalance the cluster when there's a lack of available storage space. As a result, if a data node runs out of unused storage space, then the cluster blocks any writes. For more information about disk space management, see How do I troubleshoot low storage space in my Amazon OpenSearch Service domain?Update your sharding strategyBy default, Amazon OpenSearch Service has a sharding strategy of 5:1, where each index is divided into five primary shards. Within each index, each primary shard also has its own replica. OpenSearch Service automatically assigns primary shards and replica shards to separate data nodes, and makes sure that there's a backup in case of failure.To modify OpenSearch Service default behavior, design your indices so that shards are distributed equally by size:For existing indices, use the reindex API to change the number of primary shards. The _reindex API can be used to merge smaller indices into a bigger index, or it can be used to split up the bigger index. When the bigger index is split into more primary shards, the shard sizes are decreased.For new indices, use the index template API to define the number of primary and replica shards.Then, update the indices settings for your shards. For more information, see Update index settings API on the Elasticsearch website.Delete the old or unused indices to free up disk spaceOpenSearch Service or Elasticsearch version 6.8 or later support ISM. With ISM, you can define custom management policies so that old or unused indices are deleted after an established duration.Related informationCalculating storage requirementsGet started with Amazon OpenSearch Service: How many shards do I need?Follow"


Question: How do I deploy artifacts to Amazon S3 in a different AWS account using CodePipeline and a canned ACL?

I want to deploy artifacts to an Amazon Simple Storage Service (Amazon S3) bucket in a different account. Is there a way to do that using AWS CodePipeline with an Amazon S3 deploy action provider and a canned Access Control List (ACL)?

Answer: "I want to deploy artifacts to an Amazon Simple Storage Service (Amazon S3) bucket in a different account. Is there a way to do that using AWS CodePipeline with an Amazon S3 deploy action provider and a canned Access Control List (ACL)?ResolutionNote: The following example procedure assumes the following:You have two AWS accounts: A development account and a production account.The input bucket in the development account is called codepipeline-input-bucket (with versioning activated).The default artifact bucket in the development account is called codepipeline-us-east-1-0123456789.The output bucket in the production account is called codepipeline-output-bucket.You're deploying artifacts from the development account to an S3 bucket in the production account.You're using a canned ACL to provide the bucket owner in the production account with access to the objects owned by the development account.Note: To deploy artifacts and set the production account as object owner, see How do I deploy artifacts to Amazon S3 in a different account using CodePipeline?Create a CodePipeline in the development account1.    Open the CodePipeline console. Then, choose Create pipeline.2.    For Pipeline name, enter a name for your pipeline. For example: crossaccountdeploy.Note: The Role name text box is populated automatically with the service role name AWSCodePipelineServiceRole-us-east-1-crossaccountdeploy. You can also choose another, existing service role.3.    Expand the Advanced settings section.4.    For Artifact store, choose Default location.Note: You can select Custom location if that's necessary for your use case.5.    For Encryption key, select Default AWS Managed Key.6.    Choose Next.7.    On the Add source stage page, for Source provider, choose Amazon S3.8.    For Bucket, enter the name of your development input S3 bucket. For example: codepipeline-input-bucket.Important: The input bucket must have versioning activated to work with CodePipeline.9.    For S3 object key, enter sample-website.zip.Important: To use an example AWS website instead of your own website, see Tutorial: Create a pipeline that uses Amazon S3 as a deployment provider. Then, search for "sample static website" in the Prerequisites of the 1: Deploy Static Website Files to Amazon S3 section.10.    For Change detection options, choose Amazon CloudWatch Events (recommended).11.    Choose Next.12.    On the Add build stage page, choose Skip build stage. Then, choose Skip.13.    On the Add deploy stage page, for Deploy provider, choose Amazon S3.14.    For Region, choose the AWS Region that your output S3 bucket is in. For example: US East (N. Virginia).15.    For Bucket, enter the name of your production output S3 bucket. For example: codepipeline-output-bucket.16.    Select the Extract file before deploy check box.Note: If needed, enter a path for Deployment path.17.    Expand Additional configuration.18.    For Canned ACL, choose bucket-owner-full-control.Note: The bucket-owner-full-control gives the bucket owner in the production account full access to the objects deployed and owned by the development account. For more information, see Canned ACL.19.    Choose Next.20.    Choose Create pipeline. The pipeline runs, but the source stage fails. The following error appears: "The object with key 'sample-website.zip' does not exist."The Upload the sample website to the input bucket section of this article describes how to resolve this error.Configure a CodePipeline service role with an AWS Identity and Access Management (IAM) policy that adds S3 access for the output bucket of the production account1.    Open the IAM console in the development account.2.    In the navigation pane, choose Policies. Then, choose Create policy.3.    Choose the JSON tab. Then, enter the following policy into the JSON editor:Important: Replace codepipeline-output-bucket with your production output S3 bucket's name.{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": "s3:*",      "Resource": "arn:aws:s3:::codepipeline-output-bucket/*"    },    {      "Effect": "Allow",      "Action": "s3:ListBucket",      "Resource": "arn:aws:s3:::codepipeline-output-bucket"    }  ]}4.    Choose Review policy.5.    For Name, enter a name for the policy. For example: prodbucketaccess.6.    Choose Create policy.7.    In the navigation pane, choose Roles.8.    From the list of roles, choose AWSCodePipelineServiceRole-us-east-1-crossaccountdeploy. This is the CodePipeline service role.Note: You can use your own service role, if required for your use case.9.    Choose Attach polices.10.    Select the policy that you created (prodbucketaccess). Then, choose Attach policy to grant CodePipeline access to the production output S3 bucket.Configure the output bucket in the production account to allow access from the development account1.    Open the Amazon S3 console in the production account.2.    In the Bucket name list, choose your production output S3 bucket. For example: codepipeline-output-bucket.3.    Choose Permissions. Then, choose Bucket Policy.4.    In the text editor, enter the following policy, and then choose Save:Important: Replace dev-account-id with your development environment's AWS account ID. Replace codepipeline-output-bucket with your production output S3 bucket's name.{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "",      "Effect": "Allow",      "Principal": {        "AWS": "arn:aws:iam::dev-account-id:root"      },      "Action": "s3:Put*",      "Resource": "arn:aws:s3:::codepipeline-output-bucket/*"    },    {      "Sid": "",      "Effect": "Allow",      "Principal": {        "AWS": "arn:aws:iam::dev-account-id:root"      },      "Action": "s3:ListBucket",      "Resource": "arn:aws:s3:::codepipeline-output-bucket"    }  ]}Upload the example website to the input bucket1.    Open the Amazon S3 console in the development account.2.    In the Bucket name list, choose your development input S3 bucket. For example: codepipeline-input-bucket.3.    Choose Upload. Then, choose Add files.4.    Select the sample-website.zip file that you downloaded.5.    Choose Upload to run the pipeline. When the pipeline runs, the following occurs:The source action selects the sample-website.zip from the development input S3 bucket (codepipeline-input-bucket). Then, the source action places the zip file as a source artifact inside the default artifact bucket in the development account ( codepipeline-us-east-1-0123456789).In the deploy action, the CodePipeline service role ( AWSCodePipelineServiceRole-us-east-1-crossaccountdeploy) uses its access to deploy to the production output S3 bucket ( codepipeline-output-bucket). The deploy action also applies the canned ACL bucket-owner-full-control.Note: The development account is the owner of the extracted objects in the production output S3 bucket ( codepipeline-output-bucket). The bucket owner in the production account also has full access to the deployed artifacts.Follow"


Question: How can I use IAM policies to grant user-specific access to specific S3 folders?

I want to use IAM user policies to restrict access to specific folders within Amazon Simple Storage Service (Amazon S3) buckets.

Answer: "I want to use IAM user policies to restrict access to specific folders within Amazon Simple Storage Service (Amazon S3) buckets.Short descriptionYou can use AWS Identity and Access Management (IAM) user policies to control who has access to specific folders in your Amazon S3 buckets.ResolutionSingle-user policy - This example policy allows a specific IAM user to see specific folders at the first level of the bucket and then to take action on objects in the desired folders and subfolders. This example uses an IAM user named David and a bucket named my-company with the following structure:/home/Adele/ /home/Bob/ /home/David/ /restricted/ /root-file.txt{ "Version":"2012-10-17", "Statement": [   {     "Sid": "AllowUserToSeeBucketListInTheConsole",     "Action": ["s3:ListAllMyBuckets", "s3:GetBucketLocation"],     "Effect": "Allow",     "Resource": "*"   },  {     "Sid": "AllowRootAndHomeListingOfCompanyBucket",     "Action": ["s3:ListBucket"],     "Effect": "Allow",     "Resource": ["arn:aws:s3:::my-company"],     "Condition":{"StringEquals":{"s3:prefix":["","home/"],"s3:delimiter":["/"]}}    },   {     "Sid": "AllowListingOfUserFolder",     "Action": ["s3:ListBucket"],     "Effect": "Allow",     "Resource": ["arn:aws:s3:::my-company"],     "Condition":{"StringLike":{"s3:prefix":["home/David/*"]}}   },   {     "Sid": "AllowAllS3ActionsInUserFolder",     "Effect": "Allow",     "Action": ["s3:*"],     "Resource": ["arn:aws:s3:::my-company/home/David/*"]   } ]}The Amazon S3 console uses the slash (/) as a special character to show objects in folders. The prefix (s3:prefix) and the delimiter (s3:delimiter) help you organize and browse objects in your folders.Multiple-user policy - In some cases, you might not know the exact name of the resource when you write the policy. For example, you might want to allow every user to have their own objects in an Amazon S3 bucket, as in the previous example. However, instead of creating a separate policy for each user that specifies the user's name as part of the resource, you can create a single group policy that works for any user in that group.You can do this by using policy variables, which allow you to specify placeholders in a policy. When the policy is evaluated, the policy variables are replaced with values that come from the request itself.This example shows a policy for an Amazon S3 bucket that uses the policy variable ${aws:username}:{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "AllowUserToSeeBucketListInTheConsole",      "Action": [        "s3:ListAllMyBuckets",        "s3:GetBucketLocation"      ],      "Effect": "Allow",      "Resource": "*"    },    {      "Sid": "AllowRootAndHomeListingOfCompanyBucket",      "Action": [        "s3:ListBucket"      ],      "Effect": "Allow",      "Resource": [        "arn:aws:s3:::my-company"      ],      "Condition": {        "StringEquals": {          "s3:prefix": [            "",            "home/"          ],          "s3:delimiter": [            "/"          ]        }      }    },    {      "Sid": "AllowListingOfUserFolder",      "Action": [        "s3:ListBucket"      ],      "Effect": "Allow",      "Resource": [        "arn:aws:s3:::my-company"      ],      "Condition": {        "StringLike": {          "s3:prefix": [            "home/${aws:username}/*"          ]        }      }    },    {      "Sid": "AllowAllS3ActionsInUserFolder",      "Effect": "Allow",      "Action": [        "s3:*"      ],      "Resource": [        "arn:aws:s3:::my-company/home/${aws:username}/*"      ]    }  ]}Note: Only StringLike recognizes an asterisk (*) as wildcard. StringEquals doesn't. For more information, see string condition operators.Related informationControlling access to a bucket with user policiesAmazon S3 condition key examplesBucket policy examplesFollow"


Question: Why can't I connect to an Amazon EC2 instance within my Amazon VPC from the internet?

I'm unable to connect to an Amazon Elastic Compute Cloud (Amazon EC2) instance within an Amazon Virtual Private Cloud (Amazon VPC) from the internet. How can I fix this?

Answer: "I'm unable to connect to an Amazon Elastic Compute Cloud (Amazon EC2) instance within an Amazon Virtual Private Cloud (Amazon VPC) from the internet. How can I fix this?Short descriptionProblems connecting to Amazon EC2 instances in Amazon VPC are usually related to the configuration of security groups, network access control lists (ACLs), or route tables.ResolutionBefore you start, be sure that your Amazon EC2 instance is passing system status checks and instance status checks.Check security groupsBe sure that the security groups associated with the elastic network interface of the instance allow connections from the required ports.Important: In a production environment, enable only a specific IP address or range of addresses to access your instance. For testing purposes, you can specify a custom IP address of 0.0.0.0/0 to enable all IP addresses to access your instance using SSH or RDP.Note: You don't need to configure security group egress rules, because security groups are stateful.Check network ACLsCheck your network ACLs for the following:Be sure that the network ACLs associated with your VPC subnet allow traffic through the required ports.Note: For more information, see Recommended network ACL rules for your VPC and Adding and deleting rules.Be sure that both inbound and outbound traffic are allowed.Note: Network ACLs are stateless. Responses to allowed inbound traffic are subject to the rules for outbound traffic, but responses to allowed outbound traffic are subject to the rules for inbound traffic.Be sure to open only ephemeral ports in outbound ACLs.Note: It's a best practice to allow only the ports that you need.Important: If you're still not certain what is blocking traffic from accessing your instance, consider enabling VPC flow logs. Flow logs capture IP address traffic that flows through your VPC. If you see rejected traffic in your flow logs, be sure to check your security groups and network ACL settings again.Check route tablesTo check if an internet gateway is attached to your VPC, complete the following steps:Sign in to the Amazon VPC console.On the navigation pane, in the Virtual Private Cloud section, choose Internet Gateways.In the search box, search for the internet gateway attached to your VPC. You can also use the search bar on the page to search for your Attached VPC ID (for example, vpc-xxxxxxxx).Note the ID of the internet gateway (for example, igw-xxxxxxxx).If an internet gateway is already attached to your VPC, complete the following steps:Check your VPC's route tables for a route to your internet gateway. Look for a route entry whose Target is the ID internet gateway attached to your VPC (for example, igw-xxxxxxxx), and whose Destination is 0.0.0.0/0.If the route doesn't exist, add a route entry with the internet gateway as the Target and 0.0.0.0/0 as the Destination.Be sure that the subnet route table also has a route entry to the internet gateway. If this entry doesn't exist, the instance is in a private subnet and is inaccessible from the internet.Note: Be sure that operating system-level (OS-level) route tables allow traffic from the internet. Use the command route -n (Linux instances) or netstat -rn (Linux or Windows instances), depending on your configuration.Check IP addressesCheck if a public IP address is assigned to your VPC instance, or an Elastic IP address is attached to the network interface of the instance. If a public IP address or elastic IP address isn't assigned to the network interface of the instance, then assign one.Note: For more information, see Working with IP addresses and Working with elastic IP addresses.Be sure that the OS-level software or firewalls on the instance allow traffic through the required ports.Related informationComparison of security groups and network ACLsFollow"


Question: How do I delete Amazon S3 objects and buckets?

I have Amazon Simple Storage Service (Amazon S3) objects and buckets that I don't need. How do I delete these resources?

Answer: "I have Amazon Simple Storage Service (Amazon S3) objects and buckets that I don't need. How do I delete these resources?ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.To delete individual S3 objects, you can use the Amazon S3 console, the AWS CLI, or an AWS SDK.To delete multiple S3 objects using a single HTTP request, you can use the AWS CLI, or an AWS SDK.To empty an S3 bucket of its objects, you can use the Amazon S3 console, AWS CLI, lifecycle configuration rule, or AWS SDK.To delete an S3 bucket (and all the objects that it contains), you can use the Amazon S3 console, AWS CLI, or AWS SDK.Related informationBuckets overviewAmazon S3 objects overviewFollow"


Question: How can I configure a CloudWatch subscription filter to invoke my Lambda function?

I want to configure an Amazon CloudWatch subscription filter to invoke my AWS Lambda function.

Answer: "I want to configure an Amazon CloudWatch subscription filter to invoke my AWS Lambda function.Short descriptionWith Amazon CloudWatch Logs, you can use a subscription filter that sends log data to your Lambda function. CloudWatch Logs subscription filters are base64 encoded and compressed with the GZIP format.Before you create your Lambda function, calculate the volume of log data that will be generated. Be sure to create a function that can manage the volume amount. If the function doesn't have enough volume, then the log stream is throttled. For more information, see Lambda quotas.Note: Streaming large amounts of CloudWatch Logs data might result in high usage charges. It's a best practice to use AWS Budgets to track spending and usage. For instructions, see How can I use AWS Budgets to track my spending and usage?ResolutionCreate a CloudWatch Logs subscription filter that sends log data to your AWS Lambda function.Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.1.    To provide CloudWatch Logs permission to invoke your Lambda function, run the AWS CLI command add-permission similar to the following:aws lambda add-permission \    --function-name "helloworld" \    --statement-id "helloworld" \    --principal "logs.amazonaws.com" \    --action "lambda:InvokeFunction" \    --source-arn "arn:aws:logs:region:123456789123:log-group:YourLogGroup:*" \    --source-account "123456789012"Important: Replace "helloworld" with your Lambda function name, "YourLogGroup" with your log group, and the example account number with your account.2.    Create a subscription filter using the AWS CLI command put-subscription-filter to send log events that contains a keyword. In the following example, the keyword "ERROR" is used for the Lambda function:Important: Replace "YourLogGroup" with your log group and the example account number with your account.aws logs put-subscription-filter \    --log-group-name YourLogGroup \    --filter-name demo \    --filter-pattern "ERROR" \    --destination-arn arn:aws:lambda:region:123456789123:function:helloworldThe CloudWatch log group "YourLogGroup" invokes the Lambda function when it receives a log event that contains the keyword "ERROR" similar to the following:{  "awslogs": {    "data": "H4sIAAAAAAAAAHWPwQqCQBCGX0Xm7EFtK+smZBEUgXoLCdMhFtKV3akI8d0bLYmibvPPN3wz00CJxmQnTO41whwWQRIctmEcB6sQbFC3CjW3XW8kxpOpP+OC22d1Wml1qZkQGtoMsScxaczKN3plG8zlaHIta5KqWsozoTYw3/djzwhpLwivWFGHGpAFe7DL68JlBUk+l7KSN7tCOEJ4M3/qOI49vMHj+zCKdlFqLaU2ZHV2a4Ct/an0/ivdX8oYc1UVX860fQDQiMdxRQEAAA=="  }}Related informationFilter and pattern syntaxFollow"


Question: Why is my AWS Glue ETL job running for a long time?

My AWS Glue job is running for a long time.-or-My AWS Glue straggler task is taking a long time to complete.

Answer: "My AWS Glue job is running for a long time.-or-My AWS Glue straggler task is taking a long time to complete.Short descriptionSome common reasons why your AWS Glue jobs take a long time to complete are the following:Large datasetsNon-uniform distribution of data in the datasetsUneven distribution of tasks across the executorsResource under-provisioningResolutionEnable metricsAWS Glue provides Amazon CloudWatch metrics that can be used to provide information about the executors and the amount of done by each executor. You can enable CloudWatch metrics on your AWS Glue job by doing one of the following:Using a special parameter: Add the following argument to your AWS Glue job. This parameter allows you to collect metrics for job profiling for your job run. These metrics are available on the AWS Glue console and the CloudWatch console.Key: --enable-metricsUsing the AWS Glue console: To enable metrics on an existing job, do the following:Open the AWS Glue console.In the navigation pane, choose Jobs.Select the job that you want to enable metrics for.Choose Action, and then choose Edit job.Under Monitoring options, select Job metrics.Choose Save.Using the API: Use the AWS Glue UpdateJob API with --enable-metrics as the DefaultArguments parameter to enable metrics on an existing job.Note: AWS Glue 2.0 doesn't use YARN that reports metrics. This means that you can't get some of the executor metrics, such as numberMaxNeededExecutors and numberAllExecutor, for AWS Glue 2.0.Enable continuous loggingIf you enable continuous logging in your AWS Glue job, then the real-time driver and executor logs are pushed to CloudWatch every five seconds. With this real-time logging information, you can get more details on the running job. For more information, see Enabling continuous logging for AWS Glue jobs.Check the driver and executor logsIn the driver logs, check for tasks that ran for a long time before they were completed. For example:2021-04-15 10:53:54,484 ERROR executionlogs:128 - g-7dd5eec38ff57a273fcaa35f289a99ecc1be6901:2021-04-15 10:53:54,484 INFO [task-result-getter-1] scheduler.TaskSetManager (Logging.scala:logInfo(54)): Finished task 0.0 in stage 7.0 (TID 139) in 4538 ms on 10.117.101.76 (executor 10) (13/14)...2021-04-15 12:11:30,692 ERROR executionlogs:128 - g-7dd5eec38ff57a273fcaa35f289a99ecc1be6901:2021-04-15 12:11:30,692 INFO [task-result-getter-3] scheduler.TaskSetManager (Logging.scala:logInfo(54)): Finished task 13.0 in stage 7.0 (TID 152) in 4660742 ms on 10.117.97.97 (executor 11) (14/14)In these logs, you can view that a single task took 77 minutes to complete. Use this information to review why that particular task is taking a long time. You can do so by using the Apache Spark web UI. The Spark UI provides well-structured information for different stages, tasks and executors.Enable the Spark UIYou can use the Spark UI to troubleshoot Spark jobs that run for a long time. By launching the Spark history server and enabling the Spark UI logs, you can get information on the stages and tasks. You can use the logs to learn how the tasks are executed by the workers. You can enable Spark UI using the AWS Glue console or the AWS Command Line Interface (AWS CLI). For more information, see Enabling the Apache Spark web UI for AWS Glue jobs.After the job is complete, you might see driver logs similar to the following:ERROR executionlogs:128 - example-task-id:example-timeframe INFO [pool-2-thread-1] s3n.MultipartUploadOutputStream (MultipartUploadOutputStream.java:close(414)): close closed:false s3://dox-example-bucket/spark-application-1626828545941.inprogressAfter analyzing the logs for the job, you can launch the Spark history server either on an Amazon Elastic Compute Cloud (Amazon EC2) instance or using Docker. Open the UI, and navigate to the Executor tab to check whether a particular executor is running for a longer time. If so, the uneven distribution of work and under-utilization of available resources could be caused due to a data skew in the dataset. In the Stages tab, you can get more information and statistics on the stages that took a long time. You can find details on whether these stages involved shuffle spills that are expensive and time-consuming.Capacity planning for data processing units (DPUs)If all executors contribute equally for the job, but the job still takes a long time to complete, then consider adding more workers to your job to improve the speed. DPU capacity planning can help you to avoid the following:Under-provisioning that might result is slower execution timeOver-provisioning that incurs higher costs, but provides results in the same amount of timeFrom CloudWatch metrics, you can get information on the number of executors used currently and the maximum number of executors needed. The number of DPUs needed depends on the number of input partitions and the worker type requested.Keep the following in mind when you define the number of input partitions:If the Amazon Simple Storage Service (Amazon S3) files aren't splittable, then the number of partitions is equal to the number of input files.If the Amazon S3 files are splittable, and the data is unstructured/semi-structured, then the number of partitions is equal to the total file size / 64 MB. If the size of each file is less than 64 MB, then the number of partitions is equal to the number of files.If the Amazon S3 files are splittable and the data is structured, then the number of partitions is equal to the total file size / 128 MB.Do the following to calculate the optimal number of DPUs:For example, suppose that the number of input partitions is 428. Then, you can calculate the optimal number of DPUs by the following formula:Maximum number of executors needed = Number of input partitions / Number of tasks per executor = 428/4 = 107Keep in mind the following:The Standard worker type supports 4 tasks per executorG.1X supports 8 tasks per executorG.2X supports 16 tasks per executorThe standard worker type has two executors, including one driver, in one node. One of these executors is a driver in Spark. Therefore, you need 108 executors.The number of DPUs needed = (No. of executors / No. of executors per node) + 1 DPU = (108/2) + 1 = 55.Related informationSpecial parameters used by AWS GlueMonitoring jobs using the Apache Spark web UIMonitoring for DPU capacity planningFollow"


Question: How do I associate a target network with a Client VPN endpoint?

I need to enable my clients to establish a VPN session with an AWS Client VPN endpoint so that they can access network resources. How do I associate a target network with a Client VPN endpoint?

Answer: "I need to enable my clients to establish a VPN session with an AWS Client VPN endpoint so that they can access network resources. How do I associate a target network with a Client VPN endpoint?Short descriptionA target network is a subnet in a VPC. When you associate a subnet with a Client VPN endpoint, clients can establish a VPN session. You can associate multiple subnets with a Client VPN endpoint. Note that all subnets must be from the same VPC.Before associating a target network to a Client VPN endpoint, consider the following:The clients will be able to establish a VPN connection to the Client VPN endpoint only after a target network is associated with the Client VPN endpoint.Associating a single target network allows you to establish a VPN session with the Client VPN endpoint. However, it's a best practice to associate at least two target networks from two different Availability Zones for redundancy.The subnet that you associate as the target must have a CIDR block with at least a /27 bitmask (for example, 192.168.0.0/27). Also, there must be at least eight available IP addresses in the subnet.When you associate a subnet with a Client VPN endpoint, the local route of the VPC in which the associated subnet is provisioned is automatically added to the Client VPN endpoint's route table.ResolutionAssociate a target network with a Client VPN endpointOpen the Amazon VPC console.In the navigation pane, choose Client VPN Endpoints.Select the Client VPN endpoint to associate with the target network.Choose Associations, and then choose Associate.For VPC, choose the VPC in which the subnet is provisioned.For Subnet to associate, choose the subnet to associate with the Client VPN endpoint.Choose Associate.Apply a security group to a target networkWhen you associate the first target network with a Client VPN endpoint, the default security group of the VPC is applied in the associated subnet. After you associate the first target network, you can change the security groups that are applied to the Client VPN endpoint. The security group rules that are required depend on the type of VPN access you want to configure.Open the Amazon VPC console.In the navigation pane, choose Client VPN Endpoints.Select the Client VPN endpoint to which you plan to apply the security groups.Choose Security Groups, select the current security group, and then choose Apply Security Groups.Select the new security groups in the list, and then choose Apply Security Groups.(Optional) Disassociate a target network from a Client VPN endpointAfter making sure that there are no clients connected to the Client VPN endpoint, you can disassociate unwanted target networks. You need at least one target network for the clients to establish a connection to the Client VPN endpoint. When you disassociate all target networks, the Client VPN endpoint removes the route that was automatically created when the target networks were associated.Open the Amazon VPC console.In the navigation pane, choose Client VPN Endpoints.Select the Client VPN endpoint with which the target network is associated.Choose Associations.Select the target network to disassociate.Choose Disassociate, and then choose Yes, Disassociate.Follow"


Question: How do I resolve the error "FAILED: ParseException line 1:X missing EOF at '-' near 'keyword'" in Athena?

"When I run an MSCK REPAIR TABLE or SHOW CREATE TABLE statement in Amazon Athena, I get an error similar to the following: "FAILED: ParseException line 1:X missing EOF at '-' near 'keyword'"."

Answer: "When I run an MSCK REPAIR TABLE or SHOW CREATE TABLE statement in Amazon Athena, I get an error similar to the following: "FAILED: ParseException line 1:X missing EOF at '-' near 'keyword'".ResolutionYou get this error when the database name specified in the DDL statement contains a hyphen ("-"). AWS Glue allows database names with hyphens. However, underscores (_) are the only special characters that Athena supports in database, table, view, and column names.In the following example, the database name is alb-database1. When you run MSCK REPAIR TABLE or SHOW CREATE TABLE, Athena returns a ParseException error:Your query has the following error(s):FAILED: ParseException line 1:7 missing EOF at '-' near 'alb'This query ran against the "alb-database1" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: cc5c1234-4c12-4dcb-a123-bff954b305eb.To resolve this issue, recreate the database with a name that doesn't contain any special characters other than underscore (_).Related informationNames for tables, databases, and columnsFollow"


Question: How do I resolve ORA-00018 or ORA-00020 errors for an Amazon RDS for Oracle DB instance?

I'm trying to connect as the primary user or DBA user on an Amazon Relational Database Service (Amazon RDS) for Oracle DB instance. But I receive one of these errors:ORA-00018 maximum sessions exceededORA-00020 maximum processes exceededHow do I resolve these errors?

Answer: "I'm trying to connect as the primary user or DBA user on an Amazon Relational Database Service (Amazon RDS) for Oracle DB instance. But I receive one of these errors:ORA-00018 maximum sessions exceededORA-00020 maximum processes exceededHow do I resolve these errors?Short descriptionThese errors can be caused by either a planned scaling exercise or an unplanned event that causes a large number of DB connections. In these cases, many client sessions are initiated against a DB instance, and one of the following database limits is reached:PROCESSES – the maximum number of user processes allowed.SESSIONS – the maximum number of user sessions allowed.If the maximum connections are reached because of a planned scaling exercise, then increase SESSIONS or PROCESSES, or both. This accommodates your application’s new scale. These two parameters are not dynamic, so modify the parameters and then reboot the instance.If the maximum connections are reached because of an unplanned event, then identify the cause of the event and take appropriate action.For example, your application might overwhelm the database when response times increase because of locking or block contention. In this case, increasing SESSIONS or PROCESSES only increases the number of connections before maxing out again. This can worsen issues caused by contention. This in turn can prevent the Amazon RDS monitoring system from logging in, performing health checks, or taking corrective action, such as rebooting.ResolutionOracle user PROFILEOne of the most common causes for the ORA-18 and ORA-20 is the large number of idle connections. Idle connections remain in a database without being properly closed by either the application or by database administrators. The increase of IDLE connections can cause the database to reach the maximum limit of SESSIONS/PROCESSES parameter. As a result, no new connections are allowed. It's a best practice to set a PROFILE for application connections with a limited IDLE_TIME value.In Oracle databases, each user is assigned to a PROFILE. An Oracle PROFILE is a set of resources that is assigned to each user attached to this PROFILE. One of those resources is IDLE_TIME. IDLE_TIME specifies in minutes the amount of continuous inactivity that's allowed during a session before the query is killed by the database. Long-running queries and other operations are not subject to this limit.This example shows how to create a profile with maximum IDLE_TIME of 30 minutes and assign it to an application user. Any IDLE connection for more than 30 minutes is automatically killed by the database:Create a PROFILE with a limited IDLE_TIME parameter:SQL> select count(*) from v$session where type= 'BACKGROUND';Assign this PROFILE to a specific user:  SQL> ALTER USER <username> PROFILE <profile_name>;Scaling RDS instance sizeThe maximum number of SESSIONS/PROCESSES might be reached because of an increase of the incoming workload from application users. By default, in RDS for Oracle, both parameter limits are calculated based on a pre-defined formula that depends on the DB instance class memory. It's not a best practice to manually modify SESSIONS/PROCESSES parameter in this case. Instead, scale up/down the instance based on the workload.RDS default setting for PROCESSES/SESSIONS parameter are calculated using these formulas:PROCESSES= LEAST({DBInstanceClassMemory/9868951}, 20000)SESSIONS = Oracle Default = (1.5 * PROCESSES) + 22Manually setting either PROCESSES or SESSIONS parameters beyond their default limit might cause an increase in memory consumption. As a result, the database might crash due to out of memory issues. Also, manually setting PROCESSES or SESSIONS might cause configuration mismatch when scaling up/down the instance. This happens because the PROCESSES and SESSIONS parameters don't depend on the allocated instance memory anymore.LICENSE_MAX_SESSIONSThe LICENSE_MAX_SESSIONS parameter specifies the maximum number of concurrent user sessions allowed. This doesn't apply to Oracle background processes or to users with RESTRICTED SESSION privileges, including users with the DBA role. Set LICENSE_MAX_SESSIONS to a lower value than SESSIONS and PROCESSES. This causes the client connections get an ORA-00019 error instead of an ORA-18 or ORA-20 error. An ORA-00019 error doesn’t apply to users who have RESTRICTED SESSION privileges. So, primary and RDSADMIN users are able to log on to the DB instance and perform administrative troubleshooting and corrective actions. Also, Amazon RDS monitoring can continue to connect to the database by using RDSADMIN to perform health checks.Note: LICENSE_MAX_SESSIONS was originally intended to limit the usage based on the number of concurrent sessions. Oracle no longer offers licensing based on the number of concurrent sessions, and LICENSE_MAX_SESSIONS is deprecated. But, you can still use the parameter if you use Oracle versions up to 19c. Also, application users shouldn't be granted either the DBA role or RESTRICTED SESSION privileges. For more information, see the Oracle documentation for LICENSE_MAX_SESSIONS.LICENSE_MAX_SESSIONS is a dynamic parameter, so it can be set without restarting the DB instance. For more information, see Working with parameter groups.LICENSE_MAX_SESSIONS parameter can be set as a formula, similar to the PROCESSES parameter. It's a best practice to set the LICENSE_MAX_SESSIONS parameter to be based on a formula rather than a static value. This helps to avoid mis-configuration when scaling up/down the instance size. For example:To set it same as PROCESSES parameter: LICENSE_MAX_SESSIONS= LEAST({DBInstanceClassMemory/9868951}, 20000)To set it to 4/5 of PROCESSES parameter: LICENSE_MAX_SESSIONS= LEAST({DBInstanceClassMemory/12336188}, 20000)Using DEDICATED sessionsIf you use DEDICATED sessions, your client connections might exceed the limit of the PROCESSES parameter (ORA-20). If your client connections exceed the limit, set the value of the LICENSE_MAX_SESSIONS below PROCESSES:LICENSE_MAX_SESSIONS = maximum number of client connections only.PROCESSES = LICENSE_MAX_SESSIONS + all background processes, including parallel queries, DBA users including primary users, and a buffer. A buffer allows for unexpected background processes that might occur later. To see how many background processes you have now, run a query like this:SQL> select count(*) from v$session where type= 'BACKGROUND';Note: SESSIONS, which defaults to (1.5 * PROCESSES) + 22, should be sufficient. For more information, see the Oracle documentation for SESSIONS.To manually connect to your instance to verify the SESSIONS, run a command like this:SQL> select name, value from v$parameter where upper(name) in ('SESSIONS','PROCESSES','LICENSE_MAX_SESSIONS');NAME   VALUE------------------------------ ------------------------------processes   84sessions   148license_max_sessions   0Using SHARED sessionsIf you use SHARED sessions, your client connections might exceed the limit of the SESSIONS parameter (ORA-0018). If your client connections exceed the limit, set the PROCESSES parameter to a higher value.LICENSE_MAX_SESSIONS = maximum number of client connections only.PROCESSES = all background processes, including parallel queries, and DBA users including primary users, and a buffer. Be sure to include settings of SHARED_SERVERS and DISPATCHERS with the count of background processes.SESSIONS = (1.5 * PROCESSES) + 22If you use SHARED servers, and you receive a max processes (ORA-20) error instead of max sessions (ORA-18) error, then your dispatchers might be overwhelmed. When dispatchers are overwhelmed, the connections are forced to come in as DEDICATED. Increase the number of DISPATCHERS to allow more sessions to connect shared. The SHARED_SERVERS parameter might also need to be increased.To check if you are using SHARED or DEDICATED servers, run a command like this:SQL> select decode(server, 'NONE', 'SHARED', server) as SERVER, count(*)from v$session group by decode(server, 'NONE', 'SHARED',server)Related informationBest practices for working with OracleFollow"


Question: How can I create and use partitioned tables in Amazon Athena?

I want to create partitioned tables in Amazon Athena and use them to improve my queries.

Answer: "I want to create partitioned tables in Amazon Athena and use them to improve my queries.Short descriptionBy partitioning your Athena tables, you can restrict the amount of data scanned by each query, thus improving performance and reducing costs. Partitioning divides your table into parts and keeps related data together based on column values. Partitions act as virtual columns and help reduce the amount of data scanned per query.Consider the following when you create a table and partition the data:You must store your data on Amazon Simple Storage Service (Amazon S3) buckets as a partition.Include the partitioning columns and the root location of partitioned data when you create the table.Choose the appropriate approach to load the partitions into the AWS Glue Data Catalog. The table refers to the Data Catalog when you run your queries.Use partition projection for highly partitioned data in Amazon S3.ResolutionHere are a few things to keep in mind when you create a table with partitions.Store on Amazon S3The data must be partitioned and stored on Amazon S3. The partitioned data might be in either of the following formats:Hive style format (Example: s3://doc-example-bucket/example-folder/year=2021/month=01/day=01/myfile.csv)Note: The path includes the names of the partition keys and their values (Example: year=2021)Non-Hive style format (Example: s3://doc-example-bucket/example-folder/2021/01/01/myfile.csv)Include partitioning information when creating the tableThe CREATE TABLE statement must include the partitioning details. Use PARTITIONED BY to define the partition columns and LOCATION to specify the root location of the partitioned data. Run a query similar to the following:CREATE EXTERNAL TABLE doc-example-table (first string,last string,username string)PARTITIONED BY (year string, month string, day string)STORED AS parquetLOCATION 's3://doc-example-bucket/example-folder'Replace the following in the query:doc-example-table with the name of the table that you are creatingdoc-example-bucket with the name of the S3 bucket where you store your tableexample-folder with the name of your S3 folderfirst, last, and username with the names of the columnsyear, month, and day with the names of the partition columnsLoad partitions into the Data Catalog with an approach that's appropriate for your use caseAfter creating the table, add the partitions to the Data Catalog. You can do so using one of the following approaches:Use the MSCK REPAIR TABLE query for Hive style format data: The MSCK REPAIR TABLE command scans a file system, such as Amazon S3, for Hive-compatible partitions. The command compares them to the partitions that are already present in the table and then adds the new partitions to the Data Catalog. Run a command similar to the following:Note: This approach isn't a best practice if you have more than a few thousand partitions. Your DDL queries might face time out issues. For more information, see Why doesn't my MSCK REPAIR TABLE query add partitions to the AWS Glue Data Catalog?MSCK REPAIR TABLE doc-example-tableUse the ALTER TABLE ADD PARTITION query for both Hive style and non-Hive style format data: The ALTER TABLE ADD PARTITION command adds one or more partitions to the Data Catalog. In the command, specify the partition column name and value pairs along with the Amazon S3 path where the data files for that partition are stored. You can add one partition or a batch of partitions per query by running commands similar to the following:ALTER TABLE doc-example-table ADD PARTITION (year='2021', month='01', day='01') LOCATION 's3://doc-example-bucket/example-folder/2021/01/01/'ALTER TABLE doc-example-table ADDPARTITION (year='2021', month='01', day='01') LOCATION 's3://doc-example-bucket/example-folder/2021/01/01/'PARTITION (year='2020', month='06', day='01') LOCATION 's3://doc-example-bucket/example-folder/2020/06/01/'Use the AWS Glue crawler for both Hive and non-Hive style format data: You can use the Glue crawler to automatically infer table schema from your dataset, create the table, and then add the partitions to the Data Catalog. Or, you can use the crawler to only add partitions to a table that's created manually with the CREATE TABLE statement. To use the crawler to add partitions, when you define the crawler, specify one or more existing Data Catalog tables as the source of the crawl instead of specifying data stores. The crawler crawls the data stores specified by the Data Catalog tables. No new tables are created. Instead, the manually created tables are updated, and new partitions are added. For more information, see Setting crawler configuration options.Use partition projection for highly partitioned data in Amazon S3: When you have highly partitioned data in Amazon S3, adding partitions to the Data Catalog can be impractical and time consuming. Queries against a highly partitioned table don't complete quickly. In such cases, you can use the partition projection feature to speed up query processing of highly partitioned tables and automate partition management. In partition projection, partition values and locations are calculated from configuration rather than read from a repository, such as the Data Catalog. This means that there's no need to add partitions to the Data Catalog with partition projection. Because in-memory operations are usually faster than remote operations, partition projection can reduce the runtime of queries against highly partitioned tables. The partition projection feature currently works with enumerated values, integers, dates, or injected partition column types. For more information, see Partition Projection with Amazon Athena.Related informationWhy do I get zero records when I query my Amazon Athena table?Follow"


Question: Why did my AWS DMS task fail when using Binary Reader for Amazon RDS for Oracle?

Why did my AWS Database Migration Service (AWS DMS) task fail when using Binary Reader for Amazon Relational Database Service (Amazon RDS) for Oracle?

Answer: "Why did my AWS Database Migration Service (AWS DMS) task fail when using Binary Reader for Amazon Relational Database Service (Amazon RDS) for Oracle?Short descriptionDuring the change data capture (CDC) phase, Oracle provides two methods to read the redo logs: Oracle LogMiner and Binary Reader. The Oracle LogMiner is an SQL interface that accesses the online and archived redo logs. The Binary Reader is an AWS DMS feature that reads and parses the redo logs directly.When using Binary Reader for migrations with a lot of changes, there is lower impact on the Oracle source when compared to using Oracle LogMiner. This is because the archive logs are copied and then parsed on the replication instance. For migrations that have a lot of changes, Binary Reader usually has better CDC performance than Oracle LogMiner. Be sure to provision sufficient network bandwidth to avoid network performance bottlenecks.If the prerequisites aren't met when using Binary Reader, you can receive two types of errors:Permissions errorsExtra connection attribute errorsResolutionPermissions errorsAWS DMS uses the Binary Reader by creating directories on the source database. So, the AWS DMS user account must have the required privileges to access the source Oracle endpoint and create necessary directories. If AWS DMS doesn't have permissions, then you see log entries similar to this:Messages"[SOURCE_CAPTURE ]E: OCI error 'ORA-00604: error occurred at rMeecursive SQL level 1 ORA-20900: Invalid path used for directory: /rdsdbdata/log/arch ORA-06512: at "RDSADMIN.RDSADMIN", line 321 ORA-06512: at line 2' [1022307] (oradcdc_bfilectx.c:164)"To resolve these errors, use the Amazon RDS primary user as the AWS DMS user. The directories are created automatically when an AWS DMS task starts running. If the directories aren't created, first log in to the Oracle database using the primary user. Then, run these commands to test if these directories can be created:SQL> exec rdsadmin.rdsadmin_master_util.create_archivelog_dir; SQL> exec rdsadmin.rdsadmin_master_util.create_onlinelog_dir;Check the results by querying the all_directories table:SQL> select directory_path from all_directories where directory_name in ('ONLINELOG_DIR','ARCHIVELOG_DIR');DIRECTORY_PATH--------------------------------------------------------------------------------/rdsdbdata/log/arch/rdsdbdata/log/onlinelogAfter you create the required directories ONLINELOG_DIR and ARCHIVELOG_DIR, restart your AWS DMS task.Extra connection attribute errorsIf you use the Binary Reader but you're missing the necessary extra connection attributes for your Oracle source, then you see this log entry:Messages"[TASK_MANAGER ]E: ORA-00604: error occurred at recursive SQL level 1 ORA-20900: Invalid path used for directory: awsdms_dir_test ORA-06512: at "RDSADMIN.RDSADMIN", line 321 ORA-06512: at line 2 ; Invalid RDS Oracle binary reader db settings, replacePathPrefix should be set to TRUE and usePathPrefix should be set to '/rdsdbdata/log/'; Invalid RDS Oracle binary reader db settings, useAlternateFolderForOnline should be set to TRUE; Invalid RDS Oracle binary reader db setting, oraclePathPrefix should not be empty; Invalid RDS Oracle binary reader db settings; Failed while preparing stream component 'st_0_4MGMBIOJCILNOU3UHICCDBCNFQ'.; Cannot initialize subtask; Stream component 'st_0_4MGMBIOJCILNOU3UHICCDBCNFQ' terminated [1020418] (replicationtask.c:2680)"To use the Binary Reader to capture change data for an Amazon RDS for Oracle source, add these extra connection attributes to the source endpoint:useLogMinerReader=N;useBfile=Y;replacePathPrefix=true;usePathPrefix=/rdsdbdata/log/;useAlternateFolderForOnline=true;oraclePathPrefix=/rdsdbdata/db/ORCL_A/;accessAlternateDirectly=falseRelated informationUser account privileges required on an AWS-managed Oracle source for AWS DMSAccessing online and archived redo logsMigrate an on-premises Oracle database to Amazon RDS for PostgreSQL using an Oracle bystander and AWS DMSFollow"


Question: Why are the emails that I send using Amazon SES failing with the error message "Email rejected per DMARC policy"?

I'm sending emails from Amazon Simple Email Services (Amazon SES) using a verified email address. But the emails are failing with the error message "Email rejected per DMARC policy".

Answer: "I'm sending emails from Amazon Simple Email Services (Amazon SES) using a verified email address. But the emails are failing with the error message "Email rejected per DMARC policy".Short descriptionThe following are common reasons why you can receive a Domain-based Message Authentication and Conformance (DMARC) failure:You have a "reject" DMARC policy on your domain, and your email address isn't authenticated through Sender Policy Framework (SPF) or DomainKeys Identified Mail (DKIM). To comply with DMARC, you must authenticate your email messages through SPF or DKIM, or both.Your email address is verified, but your domain isn't. To resolve this issue, you must verify your domain using DKIM to comply with DMARC.ResolutionAuthenticate your email messages through SPF or DKIMYou have a DMARC policy on your domain that's similar to the following:v=DMARC1; p=reject; rua=mailto:dmarcreports@example.com;When you have a DMARC policy that controls your domain's outbound email traffic and your email address isn't authenticated, your email is rejected.To resolve this issue, authenticate your email identity with DKIM or SPF, and comply with DMARC. For step-by-step instructions, see What do I do if my Amazon SES emails fail DMARC validation for SPF alignment or DKIM alignment?Verify your domain using DKIM to comply with DMARCIf you're sending email using an email address that's verified on a domain that's not verified, then your email fails DMARC with no DKIM authentication. Follow these instructions to verify your domain, Verifying a DKIM domain identity with your DNS provider. For methods on how to authenticate your email through DKIM see, Authenticating email with DKIM in Amazon SES.Note:You can't authenticate an email address that's on a domain that's not verified.When you send email using a separately verified email address on a DKIM-configured domain, Amazon SES automatically authenticates the message. However, if you turned off DKIM on a separately verified email address that's on a DKIM-configured domain, then your message isn't authenticated.Related informationWhy are the emails that I send using Amazon SES getting marked as spam?How do I enable DKIM for Amazon SES?Why is DKIM domain failing to verify on Amazon SES?Troubleshooting DKIM problems in Amazon SESFollow"


Question: How do I troubleshoot Amazon MQ broker connection issues?

I can't connect to my Amazon MQ broker using the web console or wire-level endpoints. How do I troubleshoot the issue?

Answer: "I can't connect to my Amazon MQ broker using the web console or wire-level endpoints. How do I troubleshoot the issue?Short descriptionAmazon MQ broker connection issues can occur for many reasons and can result in many types of errors. The following are the most common reasons why Amazon MQ returns a connection-related error:Misconfigured local firewalls and routing tablesMisconfigured security groups and network access control lists (NACLs)Connecting to the wrong port numberDNS resolution issuesUnsupported SDK clientsIncorrect broker visibility settingsMisconfigured SSL/TLSOut-of-date broker endpoint certificatesThe following are examples of some of the error messages that Amazon MQ can return when there's a broker connection issue:SSLHandshakeExceptionJMSExceptionUnknownHostExceptionSocketTimeoutExceptionResolutionTo troubleshoot Amazon MQ broker connection issues, do the following.Verify that your local firewalls and routing tables are configured correctlyFor brokers with public accessibilityConfirm the following:Your application has internet access.Your local firewalls and route tables allow public internet traffic.Your subnet NACLs and broker's security group allow communication through ports that are supported by Amazon MQ.The broker in your Amazon Virtual Private Cloud (Amazon VPC) is in a public subnet with a default route to an internet gateway. For more information, see Enable internet access in the Amazon VPC User Guide.For private brokersConfirm the following:AWS Cloud service-specific configurations associated with your Amazon MQ broker are set up correctly.Your local firewalls and route tables allow inbound and outbound traffic between any associated virtual private networks (VPNs).Your subnet NACLs and broker's security group allow communication through ports that are supported by Amazon MQ.Test your Amazon MQ broker's network connectivityFollow the instructions in step 5 of I can't connect to my broker web console or endpoints in the Amazon MQ Developer Guide.Note: If the nslookup command request times out, or no values are present in the answer section of the command output, do the following:Clear your local DNS cache.Restart your local networking equipment and client.Verify that your local DNS servers are configured correctly.Verify that you're using a supported SDK clientMake sure that you're using the most recent SDK client implementation that's compatible with the broker engine that you're using.For Amazon MQ for ActiveMQ, you must use the AMQP 1.0 protocol. For Amazon MQ for RabbitMQ, you must use the AMPQ 0-9-1 protocol.Make sure that you're using SSL/TLS to communicate with AWS resourcesNote: Amazon MQ doesn't support Mutual Transport Layer Security (TLS) authentication currently.The use of incorrect SSL/TLS versions and unsupported cipher suits results in SSL errors. For more information, see Data protection in Amazon MQ in the Amazon MQ Developer Guide.For a list of supported ciphers for RabbitMQ and ActiveMQ, see Encryption in transit in the Amazon MQ Developer Guide.Note: Some client libraries require SSL to be activated explicitly. The following are examples of explicit SSL activation for the Java and Ruby programing languages.(Java) Explicit SSL activation code exampleConnectionFactory factory=new ConnectionFactory();factory.useSslProtocol()(Ruby) Explicit SSL activation code exampleconn = Bunny.new(:tls => true, :verify_peer  => false)Verify that you're Amazon MQ broker endpoint certificate is up to dateFor more information, see the SSL exceptions section of the Amazon MQ Developer Guide.For instructions on how to tell if the Amazon Trust Services Certificate Authorities (CAs) are in your trust store, see the following article: How to prepare for AWS's move to its own certificate authority.Follow"