Question: How can I prevent an increasing backlog of messages in my Amazon SQS queue?

"The backlog of messages in my Amazon Simple Queue Service (Amazon SQS) queue is increasing. Why does this happen, and how can I prevent the backlog from increasing?"

Answer: "The backlog of messages in my Amazon Simple Queue Service (Amazon SQS) queue is increasing. Why does this happen, and how can I prevent the backlog from increasing?Short descriptionStandard and FIFO SQS queuesThe backlog of messages increases under the following circumstances:Producers send messages at a faster rate than the messages are consumed.Consumers dont delete messages within the visibility timeout period. When the SQS queue is polled, messages reappear in the queue.FIFO SQS queuesFor FIFO (First-In-First-Out) SQS queues, an increase in the backlog of messages can also result from either of the following:The FIFO queue 20,000 message buffer limit.A message that belongs to a message group is stuck with a consumer, preventing other messages from the same message group from processing.ResolutionFollow these best practices to prevent the backlog of messages from increasing.Standard and FIFO SQS queuesSet the SQS queue optimum visibility timeout to allow the consumer to delete messages after processing them within the visibility timeout period. If you dont know how long it takes to process a message, then create a heartbeat for your consumer process. Specify the initial visibility timeout (for example, 2 minutes). Then, if the consumer needs more time to process the message, continue to increase the visibility timeout using the ChangeMessageVisibility API call.Increase the batch size when you make ReceiveMessage API calls. Set the MaxNumberOfMessages parameter value to more than 1 and up to a maximum of 10.Monitor the SQS queue metric Approximate Number of Messages Visible. This metric allows you to see if producers start to produce messages at a higher rate than consumers can consume the messages. To scale horizontally, increase the number of consumers or clients that consume the SQS queue, or increase the number of threads that poll the queue.FIFO SQS queues20,000 message bufferFIFO queues allow a maximum of 20,000 inflight messages. Inflight messages include those received from a queue by a consumer, but not yet deleted from the queue. If you reach the 20,000 quota, Amazon SQS doesnt return error messages. A FIFO queue looks through the first 20,000 messages to determine available message groups. If you have a backlog of messages in a single message group, you cant consume messages from other message groups until you successfully consume messages from the backlog.Scaling over message groupsMessages that belong to the same message group are processed one by one, in the order relative to the message group. When receiving messages with multiple message group IDs, Amazon SQS tries first to return as many messages with the same message group ID as possible. This allows other consumers to process messages with a different message group ID. When messages that belong to a specific message group ID are invisible, no other consumer can process messages with the same message group ID. However, consumers can process messages from other message groups. Try increasing the number of message groups where the order isnt important.Follow"


Question: How can I use Amazon S3 to host a static website using API Gateway as the proxy?

I want to use Amazon API Gateway as a proxy for my static website hosted with Amazon Simple Storage Service (Amazon S3). How can I configure this?

Answer: "I want to use Amazon API Gateway as a proxy for my static website hosted with Amazon Simple Storage Service (Amazon S3). How can I configure this?Short descriptionYou can use Amazon S3 to host static websites. You can also configure API Gateway as a proxy for the Amazon S3 static website using either HTTP proxy integration or AWS service integration.ResolutionHTTP proxy integrationIf your Amazon S3 static website is publicly accessible, then you can use HTTP Integration and provide the S3 static website URL for the API Gateway.If you haven't already done so, follow the tutorial to configure a static website on Amazon S3.Create a REST API1.    Open the API Gateway console.2.    For REST API, choose Build.3.    For API name, enter a name for your REST API.4.    Choose the Endpoint Type dropdown list, choose your endpoint type, and then choose Create API.5.    In the navigation pane, choose Resources under your API name.6.    Choose the Actions dropdown list, and then choose Create Resource.7.    For Resource Name, enter a name. For example, key.8.    For Resource Path, enter a path parameter. For example, {key}.9.    Choose Create Resource.Create the method for the root resource1.    Choose the resource name {key}, choose the Actions dropdown list, and choose Create Method.2.    From the {key} dropdown list, choose GET, and then choose the check mark icon.3.    For Integration type, choose HTTP, select Use HTTP Proxy integration, and keep HTTP method as GET.4.    For Endpoint URL, enter http://.s3-website-.amazonaws.com/{key} and then choose Save.Deploy the REST API1.    From the Actions dropdown list, choose Deploy API.2.    From the Deployment stage dropdown list, choose [New Stage].3.    For Stage name, enter a name. For example, Dev.4.    Choose Deploy. From the Dev Stage Editor, note the Invoke URL. You'll need it later to test your API.5.    Test the API proxy for your Amazon S3 website using the curl command similar to the following:curl -X GET https://<api_id>.execute-api.<region>.amazonaws.com/index.htmlAWS service integrationConfigure the Amazon S3 static websiteIf your Amazon S3 static website is blocked for public access, configure the website so that it can be accessed only from the API proxy.1.    If you haven't already done so, follow the tutorial to configure a static website on Amazon S3.Note: Skip Step 3 of the tutorial and keep the default Block all public access setting enabled.2.    Modify the bucket policy in Step 4 to allow API proxy to access only the Amazon S3 bucket similar to following example:{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "APIProxyBucketPolicy",      "Effect": "Allow",      "Principal": {        "Service": "apigateway.amazonaws.com"      },      "Action": "s3:GetObject",      "Resource": "arn:aws:s3:::<bucket_name>/*",      "Condition": {        "ArnLike": {          "aws:SourceArn": "arn:aws:execute-api:<region>:<account>:<api_id>/*/GET/"        }      }    }  ]}3.    Create an AWS Identity and Access Management (IAM) policy with permission to the GetObject API to the Amazon S3 bucket.{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "s3:GetObject"      ],      "Resource": "*"    }  ]}4.    Create an IAM role and attach the IAM policy created in the previous step to this new role.5.    Copy the IAM roles ARN. You'll need this in another step.6.    The IAM role must contain following trust policy for API Gateway to assume the role:{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "",      "Effect": "Allow",      "Principal": {        "Service": "apigateway.amazonaws.com"      },      "Action": "sts:AssumeRole"    }  ]}Create a REST API proxy for the Amazon S3 service1.    Open the API Gateway console.2.    Choose Create API.3.    Choose REST API, and then choose Build.4.    For API name, enter a name for your REST API.5.    For Endpoint Type, choose the endpoint type depending on where the majority of client traffic originates from.Note: It's a best practice to use edge-optimized endpoints for public services accessed from the internet. Regional endpoints are used primarily for APIs that are accessed from within the same AWS Region.6.    Choose Create API.Create GET method for Root resourceThe root method is configured for proxy requests to Amazon S3 static websites.1.    With root resource "/" selected, choose the Actions dropdown list, and then choose Create Method.2.    Select GET from the dropdown list, and then choose the check mark icon.3.    For Integration type, choose AWS Service.4.    Choose the AWS Region dropdown list, and then choose your Region.5.    Choose the AWS Service dropdown list, and then choose Simple Storage Service (S3).6.    For AWS Subdomain, leave this field blank.7.    For HTTP method, select GET.8.    For Action Type, choose Use path override.9.    For Path override, enter the Amazon S3 bucket path similar to <bucket_name>/index.html.10.  For Execution role, paste the IAM role ARN that you created previously.11.  Choose Save.This setup integrates the frontend API request GET https://your-api-host/stage/ to the S3 backend GET https://your-s3-host/index.html.Create an API resource objectTo access specific objects from a S3 bucket, create a resource named {object} that maps the object path in the frontend API request. For example, GET https://your-api-host/stage/home.html to the S3 backend GET https://your-s3-host/home.html.1.    From Resources, choose the Actions dropdown list, and then choose Create Resource.2.    For Resource Name, enter a name. For example, {object}.3.    For Resource Path, enter a path. For example, {object}.4.    Choose Create Resource.Configure a GET method for the resource1.    With {object} resource selected, choose the Actions dropdown list, and then choose Create Method.2.    Choose GET from the dropdown list, and then choose the check mark icon.3.    For Integration type, choose AWS Service.4.    Choose the AWS Region dropdown list, and then choose your Region.5.    Choose the AWS Service dropdown list, and then choose Simple Storage Service (S3).6.    For AWS Subdomain, leave this field blank.7.    For HTTP method, choose GET.8.    For Action Type, choose Use path override.9.    For Path override, enter your Amazon S3 bucket path similar to <bucket_name>/{object.10.  For Execution role, paste the IAM role ARN that you created previously.11.  Choose Save.12.  Choose GET for the Resource name {object}, then choose Integration Request.13.  Expand URL Path Parameters, enter a Name value object and Mapped from value method.request.path.object, select the check box to save.Set up the response header mappings for the GET methodMap the backend content type header parameter value to the frontend counterpart. This makes sure that the browser processes the response successfully with the content type.1.    Choose GET, under Resources /, choose Method Response.2.    Expand the arrow next to HTTP Status 200, under Response Headers for 200, chose Add Header.3.    Choose the header Name as Content-Type, and then choose the check box to save.4.    Choose Method Execution.5.    Choose Integration Response, expand the arrow next to HTTP status regex.6.    Expand Header Mappings, for Content-Type, enter the mapping value as integration.response.header.Content-Type.7.    Repeat steps 1-6 for GET method under Resource /{object}.8.    Repeat the steps in the Deploy the REST API section.Test the API proxyYou can test the API proxy for your Amazon S3 static website using a browser or a curl command similar to the following:Root (/) resourcecurl -X GET https://API_ID.execute-api.region.amazonaws.com/<stage_name>/{Object} resourcecurl -X GET https://API_ID.execute-api.region.amazonaws.com/<stage_name>/home.htmlRelated informationTutorial: Create a REST API as an Amazon S3 proxy in API GatewayHosting a static website using Amazon S3Creating a REST API in Amazon API GatewayDeploying a REST API from the API Gateway consoleFollow"


Question: How do I back up my Amazon EBS volumes?

"I want to back up my Amazon Elastic Block Store (Amazon EBS) volumes, but I don't know how to do it."

Answer: "I want to back up my Amazon Elastic Block Store (Amazon EBS) volumes, but I don't know how to do it.ResolutionTo back up your Amazon EBS volumes, use Amazon EBS snapshots.Use the Amazon Elastic Compute Cloud (Amazon EC2) console, or the AWS Command Line Interface (AWS CLI) to create a snapshot. Make sure that you create the snapshot in the same AWS Region as the volume. You can also create multi-volume snapshots that are point-in-time snapshots of volumes that are attached to an Amazon EC2 instance. To automate snapshot creation and retention, you can use Amazon Data Lifecycle Manager.Note: Creating snapshots incurs charges. You're billed for the amount of data that you store. For more information, see How is my Amazon EBS snapshot bill calculated?Follow"


Question: How can I identify a DOM selector or XPath to use in a CloudWatch canary GUI Workflow Builder?

"I'm trying to create a canary in Amazon CloudWatch using the GUI Workflow Builder. I'm receiving errors that indicate that I selected an incorrect DOM element, so I need to find a DOM selector or XPath."

Answer:
"I'm trying to create a canary in Amazon CloudWatch using the GUI Workflow Builder. I'm receiving errors that indicate that I selected an incorrect DOM element, so I need to find a DOM selector or XPath.Short descriptionIf you selected an incorrect DOM element when creating a canary with the GUI Workflow Builder, you might encounter the following errors:"Error: Node is either not visible or not an HTMLElement.""No test result returned. Error: Navigation failed because browser has disconnected!""TimeoutError: Waiting for selector""TimeoutError: Waiting for XPath"ResolutionDetermine the correct selector typeThe GUI Workflow Builder has five actions. The required selector type depends on the action and the runtime version that you use.Runtime versions use Node.js and Puppeteer, and the Verify text action uses the XPath selector. The following actions use the DOM selector:ClickVerify selectorInput textClick with navigationFor runtime versions that use Python and Selenium WebDriver, all actions use the XPath selector.Identify the correct selectorDOM selectorsTo find a unique attribute for your DOM:Open the context (right-click) menu for the element (such as a button, input field, div, span, or h1).Choose Inspect element (Mozilla Firefox) or Inspect (Google Chrome).Note the unique DOM attribute that you can use to identify your element (such as id, class, or name).If your DOM doesn't have a unique attribute, use one of the following methods to find the element:Use descendant selectors to match elements based on the given path or attribute. You must use the exact path. For example, to find
 test1 test2 test3
, use div.new > a:first-child.Use attribute selectors to match elements based on the presence or value of a given attribute. For example, to find
 <input name="login"/>
, use div.user-panel.main input[name='login'].You can also get a unique DOM selector for your DOM element using your internet browser's Developer mode.Chrome:Open the context (right-click) menu for the element, and then choose Inspect.On the Elements tab, open the context (right-click) menu for the highlighted line.Choose Copy, and then choose Copy selector.Firefox:Open the context (right-click) menu for the element, and then choose Inspect Elements.On the Inspector tab, open the context (right-click) menu for the highlighted line.Choose Copy, and then choose CSS selector.Note: If there are multiple instances of the selector in the HTML document, then the canary selects the first element from the returned array. Avoid selecting elements that contain the display:none; CSS attribute. These elements are invisible and cause the selector to fail.XPath selectorsGet a unique XPath selector for your DOM element by using your internet browser's Developer mode.Chrome:Open the context (right-click) menu for the element, and then choose Inspect.On the Elements tab, open the context (right-click) menu for the highlighted line.Choose Copy, and then choose Copy XPath.Firefox:Open the context (right-click) menu for the element, and then choose Inspect Elements.On the Inspector tab, open the context (right-click) menu for the highlighted line.Choose Copy, and then choose XPath.Note: Before adding the XPath selector that you copied to the canary, remove the "//" prefix. The canary adds the "//" prefix to your code by default.Test the new selector and confirm that it's correctOpen Console mode in your internet browserChrome:At the top right of the browser, choose Customize.Expand the More tools option, and then choose Developer tools.With DevTools open in the browser, choose the Console panel.Firefox:At the top right of the browser, choose Open menu.Choose Web Developer, and then choose Web Console.Note: In some versions of Firefox, the Web Developer menu is located in the Tools menu.Test the DOM selectorThe syntax to test the DOM selector query in the browser console is: document.querySelectorAll("[AttributeName='AttributeValue']"). For example, if you have a div element with id='username', use the following pattern: document.querySelectorAll("[name='username']").Test the XPath selectorThe syntax to test the XPath query in the browser console is: $x("XPATH_QUERY"). For example, if you have a div element with id='welcome' that contains an h1 element, use the following pattern: $x("//div[@id='welcome']//h1). When you add this path to the canary, omit the "//" prefix.Add the new selector to the CloudWatch canaryOpen the CloudWatch console.For Synthetics, choose Canaries.Choose Create Canary.Choose Use a blueprint, and then choose GUI Workflow Builder.Refer to the example in GUI Workflow Builder to complete a blueprint that uses the newly created selector.Follow"


Question: How do I resolve the error message "OOM command not allowed" for an ElastiCache for Redis cluster node?

I received the "OOM command not allowed when used memory greater than 'maxmemory'" error when querying my Amazon ElastiCache for Redis cluster node? How do I resolve this?

Answer: "I received the "OOM command not allowed when used memory greater than 'maxmemory'" error when querying my Amazon ElastiCache for Redis cluster node? How do I resolve this?Short descriptionAn OOM error occurs when an ElastiCache for Redis cluster can't free any additional memory.ElastiCache for Redis implements the maxmemory-policy that's set for the cache node’s parameter group when out of memory. The default value (volatile-lru) frees up memory by evicting keys with a set expiration time (TTL value). When a cache node doesn't have any keys with a TTL value, it returns an error instead.To resolve this error and to prevent clients from receiving OOM command not allowed error messages, do some combination of the following:Set a TTL value for keys on your node.Update the parameter group to use a different maxmemory-policy parameter.Delete some existing keys manually to free up memory.Choose a larger node type.Note: The exact combination of the resolutions you use depends on your particular use case.ResolutionSet a TTL value for keys on your nodeYou can set a TTL value for keys on your node. The default volatile-lru memory management policy then evicts expired keys to free up memory.If you set a TTL value, you can also use the volatile-ttl setting. This setting makes space by evicting only those keys with an expire time set, starting with the keys with the lowest TTL.Choose a different maxmemory-policy settingSet the maxmemory-policy for a cache node parameter group to one of the following values:allkeys-lru: Make space by evicting less recently used keys first.allkeys-random: Make space by evicting random keys.volatile-random: Make space by evicting random keys with an expire set.Additionally, Redis 4.0 offers least frequently used (LFU) eviction modes:allkeys-lfu: Make space by evicting the key that was accessed the fewest number of times.volatile-lfu: Make space by evicting keys with an expire set, starting with the one that was accessed the fewest number of times.Delete existing keys to free up memoryYou can delete existing keys to free up memory using the DEL command. For more information, see Del key [key...] on the redis.io website.Important: Deleting existing keys temporarily resolves OOM errors. Be sure to also reconsider your memory management strategy.Choose a larger node typeEach node type has a different amount of available memory. You can't modify the amount of available memory on a node. However, you can use a larger node type with more storage.Important: Using a larger node type temporarily resolves OOM errors. Be sure to also reconsider your memory management strategy.Related informationRedis 4.0.10 parameter changesRedis node-type specific parametersKey eviction - Overview of Redis key eviction policies (LRU, LFU, etc.)Follow"


Question: How do I automatically create tables in Amazon Athena to search through AWS CloudTrail logs?

"I want to search through a large collection of AWS CloudTrail logs. To avoid errors, I don't want to manually create Amazon Athena tables."

Answer: "I want to search through a large collection of AWS CloudTrail logs. To avoid errors, I don't want to manually create Amazon Athena tables.ResolutionUse the CloudTrail console to automatically create the Athena tables. For more information, see AWS CloudTrail log search using Amazon Athena.Create the Athena table1.    Open the CloudTrail console, and then choose Trails from the navigation pane. Note the S3 bucket name.2.    In the navigation pane, choose Event history, and then choose Create Athena table.3.    In the Create a table in Amazon Athena window, open the Storage location menu, and then choose the Amazon Simple Storage Service (Amazon S3) bucket with the CloudTrail log files.Note: You can improve search performance for specific time frames in a single AWS region using partition projection for CloudTrail logs with Athena.4.    Choose Create table.5.    Choose Go to Athena.Note: You might receive the error "Your account does not have sufficient permissions to create tables in Amazon Athena." If you do, then attach the AmazonAthenaFullAccess managed policy by following the instructions at adding AWS Identity and Access Management (IAM) policies (console).You can use one or more of the following sample queries with your logs. Replace your_athena_tablename with the name of your Athena table, and access_key_id with your 20-character access key. Your access key usually begins with the characters AKIA or ASIA.Run the Athena query1.    Open the Athena console, choose New query, and then choose the dialog box to clear the sample query.2.    Enter your query and then choose Run Query.Display all recorded AWS API activity for a specific access keySELECT eventTime, eventName, userIdentity.principalIdFROM your_athena_tablename WHERE userIdentity.accessKeyId like 'access_key_id'Identify any security group changes for your EC2 instanceSELECT eventname, useridentity.username, sourceIPAddress, eventtime, requestparametersFROM your_athena_tablenameWHERE (requestparameters like '%sg-5887f224%' or requestparameters like '%sg-e214609e%' or requestparameters like '%eni-6c5ca5a8%')and eventtime > '2017-02-15T00:00:00Z'order by eventtime asc;Display any console logins over the last 24 hoursSELECT useridentity.username, sourceipaddress, eventtime, additionaleventdataFROM your_athena_tablename WHERE eventname = 'ConsoleLogin'and eventtime >= '2017-02-17T00:00:00Z'and eventtime < '2017-02-18T00:00:00Z';Display any failed console sign-in attempts over the last 24 hoursSELECT useridentity.username, sourceipaddress, eventtime, additionaleventdataFROM your_athena_tablenameWHERE eventname = 'ConsoleLogin'and useridentity.username = 'HIDDEN_DUE_TO_SECURITY_REASONS'and eventtime >= '2017-02-17T00:00:00Z'and eventtime < '2017-02-18T00:00:00Z';Identify CloudTrail monthly cost increasesSELECT eventName,count(eventName) AS NumberOfChanges,eventSourceFROM your_athena_tablename WHERE eventtime >= '2019-01-01T00:00:00Z'and eventtime < '2019-01-31T00:00:00Z'GROUP BY eventName, eventSourceORDER BY NumberOfChanges DESCNote: The eventtime defined values are limited to a single month. To get results for multiple months, run additional queries.Identify missing IAM permissionsSELECT count (*) as TotalEvents, useridentity.arn, eventsource, eventname, errorCode, errorMessageFROM your_athena_tablenameWHERE (errorcode like '%Denied%' or errorcode like '%Unauthorized%')AND eventtime >= '2019-10-28T00:00:00Z'AND eventtime < '2019-10-29T00:00:00Z'GROUP by eventsource, eventname, errorCode, errorMessage, useridentity.arnORDER by eventsource, eventnameIf the results don't display the failed API call, widen the query range similar to the following:SELECT count (*) as TotalEvents, useridentity.arn, eventsource, eventname, errorCode, errorMessageFROM your_athena_tablenameWHERE errorcode <> ''AND eventtime >= '2019-10-28T00:00:00Z'AND eventtime < '2019-10-29T00:00:00Z'GROUP by eventsource, eventname, errorCode, errorMessage, useridentity.arnORDER by eventsource, eventnameRelated informationQuerying AWS CloudTrail logsAdding and removing IAM identity permissionsHow can I troubleshoot IAM permission access denied or unauthorized errors?Follow"


Question: How do I troubleshoot the error "unable to pull secrets or registry auth" in Amazon ECS?

I received one of the following errors when I launched an Amazon Elastic Container Service (Amazon ECS) task: "ResourceInitializationError: unable to pull secrets or registry auth: pull command failed: : signal: killed" or "ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried."

Answer: "I received one of the following errors when I launched an Amazon Elastic Container Service (Amazon ECS) task: "ResourceInitializationError: unable to pull secrets or registry auth: pull command failed: : signal: killed" or "ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried."Short descriptionThe AWS Fargate platform version 1.4.0 uses the task elastic network interface to pull the image and secrets. All network traffic flows through the elastic network interface within your Amazon Virtual Private Cloud (Amazon VPC). You can view this traffic through your Amazon VPC Flow Logs. However, the task uses your network configuration instead of using the elastic network interfaces that Fargate owns. This is because the elastic network interfaces are placed within your Amazon VPC.The Amazon ECS container agent uses the task execution AWS Identity and Access Management (IAM) role to get information from the following services:AWS Systems Manager Parameter StoreAWS Secrets ManagerIf you encrypt data using a customer managed AWS Key Management Service (AWS KMS) key, then grant the following permissions to the task execution IAM role: ssm:GetParameterssecretsmanager:GetSecretValuekms:Decrypt.Use the AWSSupport-TroubleshootECSTaskFailedToStart runbook to troubleshoot the Amazon ECS tasks that fail to start. This automation reviews the following configurations:Network connectivity to the configured container registryMissing IAM permissions required by the task execution roleVirtual private cloud (VPC) endpoint connectivitySecurity group rule configurationAWS Secrets Manager secrets referencesLogging configurationIf the runbook's output doesn't provide recommendations, then use the manual troubleshooting approaches in the following section. ResolutionImportant:Use the runbook in the same AWS Region where your ECS Cluster resources are located.When using the runbook, you must use the most recently failed Task ID. If the failed task is part of the Amazon ECS service, then use the most recently failed task in the service. The failed task must be visible in ECS:DescribeTasks during automation execution. By default, stopped ECS tasks are visible for 1 hour after entering the Stopped state. Using the most recently failed task ID prevents the task state cleanup from interrupting the analysis during the automation.To run the AWSSupport-TroubleshootECSTaskFailedToStart runbook, complete the following steps:Open the AWS Systems Manager console.In the navigation pane, under Change Management, choose Automation.Choose Execute automation.Choose the Owned by Amazon tab.Under Automation document, search for TroubleshootECSTaskFailedToStart.Select the AWSSupport-TroubleshootECSTaskFailedToStart card.Note: Make sure that you select the radio button on the card and not the hyperlinked automation name.Choose Next.Note: After execution, analysis results are populated in the Global output section. However, wait for the status of the document to move to Success. Also, watch for any exceptions in the Output section.For Execute automation document, choose Simple execution.In the Input parameters section, for AutomationAssumeRole, enter the ARN of the role that allows Systems Manager Automation to perform actions.Note: Be sure that either the AutomationAssumeRole or the IAM user or role has the IAM required permissions to run the AWSSupport-TroubleshootECSTaskFailedToStart runbook. If you don't specify an IAM role, then Systems Manager Automation uses the permissions of the IAM user or role that runs the runbook. For more information about creating the assume role for Systems Manager Automation, see Task 1: Create a service role for Automation.For ClusterName, enter the cluster name where the task failed to start.For TaskId, enter the identification for the task that most recently failed.Choose Execute. Based on the output of the automation, use one of the following manual troubleshooting steps.Check the routes from your subnets to the internetIf you have a Fargate task in a public subnet, then verify that your task has an assigned public IP address. Also, confirm that the task has a default route (0.0.0.0/0) to an internet gateway. When you launch a new task or create a new service, turn on Auto-assign public.If you use the following configurations, then don't use the internet gateway in the public subnet to reach the Secrets Manager or Systems Manager. Instead, use an Amazon VPC endpoint:The Secrets Manager or Systems Manager VPC endpoints are in a public subnet.You turned on AmazonProvidedDNS in your Amazon VPC DHCP settings.Note: You can't turn on Auto-assign public for existing tasks. For existing services, you can use only the AWS Command Line Interface (AWS CLI) to reconfigure the services. You can't use the AWS Management Console. If you used an AWS CloudFormation stack to create your Amazon ECS service, then modify the NetworkConfiguration property AWS::ECS::Service to update the service.If you have a Fargate task in a private subnet, then verify that your task has a default route (0.0.0.0/0) to the internet connectivity source. The internet connectivity source can be a NAT gateway, AWS PrivateLink, or other source.If you use a NAT gateway, then place your NAT gateway in a public subnet. For more information, see Architecture with an internet gateway and a NAT gateway.If you use PrivateLink, then be sure that your Fargate infrastructure can use the security groups for your Amazon VPC endpoints.If you use a custom name domain server, then confirm the DNS query's settings. The query must have outbound access on port 53 using UDP and TCP protocol. Also, it must have HTTPS access on port 443. Check your network ACL and security group settingsVerify that your network access control list (network ACL) and security groups don't block outbound access to port 443 from the subnet. For more information, see Control traffic to resources using security groups.Note: Fargate tasks must have outbound access to port 443 to allow outgoing traffic and access Amazon ECS endpoints.Check your Amazon VPC endpointsIf you use PrivateLink, then you must create the required endpoints.The following endpoints are required for Fargate platform versions 1.4.0 or later:com.amazonaws.region.ecr.dkrcom.amazonaws.region.ecr.apiS3 gateway endpointcom.amazonaws.region.logsFor more information, see Considerations for Amazon ECR VPC endpoints.Note: If your task definition uses Secrets Manager, Systems Manager parameters, or Amazon CloudWatch Logs, then you might need to define endpoints. For more information, see the following documentation:Using an AWS Secrets Manager VPC endpointCreating the VPC endpoints for Amazon ECSUsing CloudWatch Logs with interface VPC endpointsIf you use PrivateLink, then check that Amazon VPC's security group allows the correct traffic. The group must allow traffic from the Fargate task security group or Fargate task VPC CIDR range on TCP port 443.To confirm that the Fargate infrastructure has service access, check the VPC endpoint policies and endpoint policies for Amazon Simple Storage Solution (Amazon S3).Check your IAM roles and permissionsThe task execution role grants the required permissions to the Amazon ECS container and Fargate agents to make API calls for the task. Fargate requires this role when you take the following actions:Pull a container image from Amazon Elastic Container Registry (Amazon ECR).Use the awslogs log driver.Use private registry authentication.Use Secrets Manager secrets or Systems Manager Parameter Store parameters to reference sensitive data.If your use case involves any of the preceding scenarios, then define the required permissions in your task execution role. For a complete list of required permissions, see Amazon ECS task execution IAM role.Check the referenced sensitive information in the Amazon ECS task definitionCheck if the secret and parameter names match the referenced names in your Amazon ECS task definition. Then, check if the values in the container definition in your task definition match the values in your Amazon ECS task definition. For more information, see How can I pass secrets or sensitive information securely to containers in an Amazon ECS task?If the Systems Manager Parameter Store parameter and task are in the same Region, then use the full ARN or the name of the secret. If the parameter exists in a different Region, then you must specify the full ARN.To check the Systems Manager parameter name and ARN, complete the following steps:Note: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.Open the AWS Systems Manager console.In the navigation pane, choose Parameter Store, and then confirm your Parameter Store name.To get the parameter's ARN, use the AWS CLI to run the following command. Replace name_of_parameter_store_secret with your Parameter Store secret name.$ aws ssm get-parameter —name <name_of_parameter_store_secret> —with-decryptionNote: Parameters that reference Secrets Manager secrets can't use the Parameter Store versioning or history features. For more information, see Restrictions.Related informationChecking stopped tasks for errorsFollow"


Question: Why can't I load website content uploaded to my Amazon S3 bucket from another AWS account?

"I'm using an Amazon Simple Storage Service (Amazon S3) bucket to store content for my website. A user from another AWS account uploaded an object for the website to my bucket. My bucket policy is correct, but the object won't load on the website. How can I fix this?"

Answer: "I'm using an Amazon Simple Storage Service (Amazon S3) bucket to store content for my website. A user from another AWS account uploaded an object for the website to my bucket. My bucket policy is correct, but the object won't load on the website. How can I fix this?Short descriptionIf another AWS account uploads an object to your bucket, then you won't own the object by default. As a result, you might not be able to read the object. The uploading account must explicitly grant you (the bucket owner) permissions to the object.Additionally, a bucket policy doesn't apply to objects in the bucket that are owned by other accounts. Therefore, the bucket policy that grants read access to your website's users doesn't automatically apply to objects uploaded by another account.To fix the loading issue, change the object's access control list (ACL) using one of these ways:The object owner grants the object public read access.The object owner grants the bucket owner full control of the object.Note the following when granting bucket access to another account:Objects uploaded to a bucket by another account won't be readable by the bucket's account by default. The account that uploaded the object must explicitly update the ACL to grant read permissions.Objects uploaded to a bucket by another account won't automatically inherit the permissions defined in the bucket policy. The bucket owner must take ownership of the object for the bucket policy to apply.If you want to allow another account to access your bucket, it's a best practice that you use a bucket policy. A bucket policy is a more centralized and comprehensive way to manage permissions.If you want to allow another account to upload objects to your bucket, create an AWS Identity and Access Management (IAM) role. The IAM role must be created from your AWS account, and configured so that another AWS account can assume the role. When the other account uses the IAM role to upload objects, your account then owns the objects because the role belongs to your account. For an example cross-account configuration using an IAM role, see Bucket owner granting cross-account permission to objects it does not own.ResolutionThe object owner grants the object public read accessThe account that uploaded the object can grant the object public read access by running this AWS Command Line Interface (AWS CLI) command:aws s3api put-object-acl --bucket docexamplebucket --key example.jpg --acl public readFor the value of --bucket, enter the name of the bucket that stores your website content.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.To restrict traffic to an object, the bucket owner can use a bucket policy that only allows requests coming from certain IP addresses or VPCs.For example, this bucket policy denies access to objects in docexamplebucket unless the request is from the IP addresses listed as "aws:SourceIp". Or, the request must be from the VPC specified as "aws:sourceVpc":{    "Version": "2012-10-17",    "Id": "Policy1415115909152",    "Statement": [        {            "Sid": "Deny-Access-Except-For-Trusted-IPs-and-VPC",            "Effect": "Deny",            "Principal": "*",            "Action": "s3:GetObject",            "Resource": "arn:aws:s3:::docexamplebucket/*",            "Condition": {                "NotIpAddress": {                    "aws:SourceIp": [                        "1.1.1.1/32",                        "2.2.2.2/32",                        "3.3.3.3/32"                    ]                },                "StringNotEquals": {                    "aws:sourceVpc": "vpc-12345abc"                }            }        }    ]}The object owner grants the bucket owner full control of the objectObject owners can automatically grant the bucket owner full control of an object by enabling S3 Object Ownership on the destination bucket. This feature grants the bucket owner control when the object owner uploads an object, and includes the bucket-owner-full-control canned ACL.After the bucket owner configures their S3 Object Ownership to "bucket owner preferred", the bucket owner can add another bucket policy. This additional bucket policy must require all Amazon S3 PUT operations to include the bucket-owner-full-control canned ACL. This ACL grants the bucket owner full control of new objects. For more information, see Controlling ownership of uploaded objects using S3 Object Ownership.To grant the bucket owner full control of the object, use the following CLI syntax when uploading objects:aws s3api put-object --bucket docexamplebucket --key example.jpg --acl bucket-owner-full-controlThe bucket owner now owns the object, and the object inherits the permissions set on the bucket policy.For existing objects that were uploaded before enabling S3 Object Ownership, the object owner must grant the bucket owner permissions on the object. Then, the bucket owner must copy over the object itself.To grant the bucket owner full control of the object, use the following CLI syntax with the account that uploaded the object:aws s3api put-object-acl --bucket docexamplebucket --key example.jpg --acl bucket-owner-full-controlTo inherit object ownership of the object, the bucket owner must copy over the object itself like this:aws s3 cp s3://docexamplebucket/example.jpg s3://docexamplebucket/example.jpg --acl bucket-owner-full-controlThe bucket owner now owns the object, and the object inherits the permissions set on the bucket policy.Require that cross-account uploads grant the bucket owner full control of the objectTo require that all uploads (s3:PutObject) from another account grant the bucket owner full control of the object, use a bucket policy like this:{    "Id": "Policy1541018284691",    "Version": "2012-10-17",    "Statement": [        {            "Sid": "RequireBucketOwnerFullControlOnPuts",            "Principal": {                "AWS": [                    "arn:aws:iam::111122223333:user/iam_user"                ]            },            "Action": [                "s3:PutObject"            ],            "Effect": "Allow",            "Resource": "arn:aws:s3:::docexamplebucket/*",            "Condition": {                "StringEquals": {                    "s3:x-amz-acl": "bucket-owner-full-control"                }            }        }    ]}With this bucket policy, the user from the other account (111122223333) can upload to the bucket only when the object's ACL is specified. The object's ACL must grant the bucket owner full control. The user must then upload objects like this:aws s3 cp path/to/local/file s3://docexamplebucket --acl bucket-owner-full-controlRelated informationHow Amazon S3 authorizes a requestGuidelines for using the available access policy optionsPermissions required for website accessControlling access to a bucket with user policiesFollow"


Question: Why isn't my domain record in Route 53 that points to an Amazon S3 bucket or static website resolving?

"I have a domain record in Amazon Route 53 that points to an Amazon Simple Storage Solution (Amazon S3) bucket or static website. However, the domain isn't resolving. How can I troubleshoot this issue?"

Answer: "I have a domain record in Amazon Route 53 that points to an Amazon Simple Storage Solution (Amazon S3) bucket or static website. However, the domain isn't resolving. How can I troubleshoot this issue?ResolutionNote: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.Review your Route 53 configurationConfirm that the alias or CNAME record for the domain is pointing to the correct website endpoint. Depending on your Region, your Amazon S3 website endpoint must follow one of the following formats:s3-website dash (-) Region — http://bucket-name.s3-website-Region.amazonaws.coms3-website dot (.) Region — http://bucket-name.s3-website.Region.amazonaws.comReview your Amazon S3 bucket configurationWhen you configure an Amazon S3 bucket for website hosting, you must give the bucket the same name as the record that you want to use to route traffic to the bucket. For example, to route traffic for "example.com" to an Amazon S3 bucket that's configured for website hosting, the bucket name must be "example.com".Important: Amazon S3 website endpoints don't support HTTPS. For information about using HTTPS with an Amazon S3 bucket, see:How do I use CloudFront to serve HTTPS requests for my Amazon S3 bucket?Requiring HTTPS for communication between viewers and CloudFrontReview your name server configurationUsing the following command, determine whether the correct name servers are in use and whether the domain is resolving to an IP address:dig <domain_name> +traceThe DNS query must be directed to the correct set of name servers to answer the DNS query.Note: The Resolving IP address isn't static.Example output:; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.2 <<>> swapnil.live +trace;; global options: +cmd.            518400    IN    NS    D.ROOT-SERVERS.NET..            518400    IN    NS    E.ROOT-SERVERS.NET..            518400    IN    NS    F.ROOT-SERVERS.NET..            518400    IN    NS    G.ROOT-SERVERS.NET..            518400    IN    NS    H.ROOT-SERVERS.NET..            518400    IN    NS    I.ROOT-SERVERS.NET..            518400    IN    NS    J.ROOT-SERVERS.NET..            518400    IN    NS    K.ROOT-SERVERS.NET..            518400    IN    NS    L.ROOT-SERVERS.NET..            518400    IN    NS    M.ROOT-SERVERS.NET..            518400    IN    NS    A.ROOT-SERVERS.NET..            518400    IN    NS    B.ROOT-SERVERS.NET..            518400    IN    NS    C.ROOT-SERVERS.NET.;; Received 239 bytes from 172.31.0.2#53(172.31.0.2) in 0 mslive.            172800    IN    NS    demand.beta.aridns.net.au.live.            172800    IN    NS    demand.alpha.aridns.net.au.live.            172800    IN    NS    demand.delta.aridns.net.au.live.            172800    IN    NS    demand.gamma.aridns.net.au.live.            86400    IN    DS    27304 8 1 BB1914F4690FDC1C882FA6CEA84D14FCA6B9D38Dlive.            86400    IN    DS    27304 8 2 F76B40CAC1A4F9D7B2E3ED67602D7F934FD45D9064CE78880FB73D09 B1ACD107live.            86400    IN    RRSIG    DS 8 1 86400 20210427170000 20210414160000 14631 . YbQEzenUx/Dp6C8UcX9fsFqDjyc+eDgcU6oJRZUT63sxJfdatMpHYwWT NCHA8uUTV8W+OVacXR9yduTWgt9jUdAzEHFeg0r4j1PPIDDE7mOB2UUF Fe8lEkny+1bRwXIh/9jsQ0/W5BNIBoSPHRpydE1B36i31V9BLp7KTwWJ sefnFqB8ZL1uZ8rSCw9sVRisegSvcDWZtD7huqkGVcWScLPDVr8/Qj6P 25zEKh6wNPHZDpmIdxvujPFnt7tdY95Jz6P8G7cJwXYdG6ChW7My4XJ5 NRzNjVuFDmWTQSn5S1uEXYL/160ArtHa+M/BE2PKXRxSZa+7djKP9Ida L3qdOQ==;; Received 708 bytes from 192.5.5.241#53(F.ROOT-SERVERS.NET) in 1 msswapnil.live.        86400    IN    NS    ns-1744.awsdns-26.co.uk.        <--- These are the name servers that are in use.swapnil.live.        86400    IN    NS    ns-863.awsdns-43.net.swapnil.live.        86400    IN    NS    ns-385.awsdns-48.com.swapnil.live.        86400    IN    NS    ns-1051.awsdns-03.org.swapnil.live.        86400    IN    DS    45482 13 2 D64D695A19C2BEB920B376E48403B5A77A2816DB5A4AE4B7D37BF719 34F85852swapnil.live.        86400    IN    RRSIG    DS 8 2 86400 20210509045136 20210409041400 12795 live. c71wlpB/k+75nWuVYGcp08yYqZSfeLeuaABBXZM6YFvK6HcMZzzaBkOZ bOEm9s37tfnPBDbFBAhzZaMkz6Ef8RJFnkEYqjaqTwxD/vz+Ztwb9ECB EpnXGut0iZICGpFF7Gg5nZeWDbNs/ihC0QGe63Npqful+ry0uBhwnl1I PwKbGd1VC5NGnO7jNTRwgIIGQnS5a3tFZd0HFb/U+O+cCA==;; Received 425 bytes from 37.209.194.7#53(demand.beta.aridns.net.au) in 0 msswapnil.live.        5    IN    A    52.216.134.18swapnil.live.        5    IN    RRSIG    A 13 2 5 20210415060418 20210415040413 26088 swapnil.live. mgzBciE3HMEg2ecZEIsZIa4l5UpDRi11sEe6ZFlh0d9DGormSdv5HhER zm4qcdb5IXDVe7GIQ+zp2J+5F+lYYQ==;; Received 165 bytes from 205.251.193.129#53(ns-385.awsdns-48.com) in 15Test the backend server responseUsing the following command, test the response of the backend server:curl -IvkL <domain_name>Example output:curl -IvkL swapnil.live* Rebuilt URL to: swapnil.live/*   Trying 52.216.25.195...            <----- Domain resolved to an IP address* TCP_NODELAY set* Connected to swapnil.live (52.216.25.195) port 80 (#0)        <--- Connected to S3 bucket> HEAD / HTTP/1.1> Host: swapnil.live> User-Agent: curl/7.61.1> Accept: */*>< HTTP/1.1 403 Forbidden                <----- Response from S3 bucketHTTP/1.1 403 Forbidden< x-amz-error-code: AccessDeniedx-amz-error-code: AccessDenied< x-amz-error-message: Access Deniedx-amz-error-message: Access Denied< x-amz-request-id: 4ANTJRWVRT5489ZWx-amz-request-id: 4ANTJRWVRT5489ZW< x-amz-id-2: HcuMJslI/Biry9uJYhx9hSyybld0CrEeSagaNqcFJqzormpeup2aPKzW58OPgIQDZGG2rn3RPI4=x-amz-id-2: HcuMJslI/Biry9uJYhx9hSyybld0CrEeSagaNqcFJqzormpeup2aPKzW58OPgIQDZGG2rn3RPI4=< Date: Thu, 15 Apr 2021 05:09:16 GMTDate: Thu, 15 Apr 2021 05:09:16 GMT< Server: AmazonS3                      <----- Server = Amazon S3Server: AmazonS3<* Connection #0 to host swapnil.live left intactRelated informationHosting a static website using Amazon S3Tutorial: Configuring a static website on Amazon S3Configuring a static website using a custom domain registered with Route 53Follow"


Question: How do I create a backup VPN for my AWS Site-to-Site VPN connection using the same transit gateway?

I want to create a backup AWS Virtual Private Network (AWS VPN) for my AWS Site-to-Site connection using the same transit gateway.

Answer: "I want to create a backup AWS Virtual Private Network (AWS VPN) for my AWS Site-to-Site connection using the same transit gateway.Short descriptionWith dynamic routing, you can use the Border Gateway Protocol (BGP) parameters such as local preference, AS_Path, and multi-exit discriminator (MED) values with your VPN connection. A backup AWS Site-to-Site VPN connection is preferred when primary AWS VPN tunnels are down.Note: The AWS Site-to-Site VPN connection must be dynamic, not static. This is because you can't use the same LAN routes in the routing table for more than one transit gateway attachment. Static routes are then blocked in the transit gateway routing table.ResolutionFollow these instructions to create a backup AWS VPN using transit gateway:Note: Equal Cost Multipath (ECMP) doesn't need to be enabled or disabled on the transit gateway. The BGP attribute values that's configured for the prefixes determine the tunnel for the ingress and egress direction.1.    Follow the instructions to create a transit gateway attachment to a VPN.2.    For Customer Gateway, choose Existing, and then select the primary AWS VPN gateway ID.3.    For Routing options, choose Dynamic.4.    Follow the instructions to view your VPN attachments and note the new VPN attachment.5.    Follow the instructions to associate a transit gateway route table with the new VPN attachment.To be sure that the preferred tunnel sends traffic from AWS to an on-premises network, see How do I configure my Site-to-Site VPN connection to prefer tunnel A over tunnel B?Related informationExample customer gateway device configurations for dynamic routing (BGP)Follow"


Question: Why isn't the unified CloudWatch agent pushing log events?

"I'm using the unified CloudWatch agent to push logs to Amazon CloudWatch Logs. However, I can't see the logs in the CloudWatch Logs console. Why isn't the unified CloudWatch agent pushing log events?"

Answer: "I'm using the unified CloudWatch agent to push logs to Amazon CloudWatch Logs. However, I can't see the logs in the CloudWatch Logs console. Why isn't the unified CloudWatch agent pushing log events?Short descriptionThe following issues can prevent the unified CloudWatch agent from pushing log events:Out-of-sync metadata caused by creating an Amazon Machine Image (AMI) after the CloudWatch agent is installedUsing an outdated version of the CloudWatch agentFailure to connect to the CloudWatch Logs endpointIncorrect account, Region, or log group configurationsInsufficient AWS Identity and Access Management (IAM) permissionsCloudWatch agent run errorsTimestamp issuesResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Important: The following troubleshooting steps don't apply to Container Insights. For details on troubleshooting Container Insights, see Troubleshooting Container Insights and How can I troubleshoot pod status in Amazon EKS?Review the installation method for the CloudWatch agentIt's a best practice to install the CloudWatch agent at launch using AWS CloudFormation, AWS Systems Manager Agent (SSM Agent), user data scripts, or the AWS CLI. It is also a best practice to create an AMI before installing the CloudWatch agent. AMIs typically capture unique information from the original instance. Metadata becomes out of sync, and this state can lead to the CloudWatch agent not working as intended. Out-of-sync metadata is the reason that many Windows instances require Sysprep when working with AMI. For more information, see How can I use Sysprep to create and install custom reusable Windows AMIs in Amazon EC2?Confirm that you're using the latest version of the CloudWatch agentDownload and review the README files for the CloudWatch agent release notes and latest version number. If you're using an older version of the CloudWatch agent, be sure to upgrade. The latest version might include updates that resolve the issue that you're experiencing.Test connectivity to your CloudWatch Logs endpointTest connectivity to the CloudWatch Logs endpoint using either of the following commands:telnet logs.<region>.amazonaws.com 443nc -zv logs.<region>.amazonaws.com 443If you get connectivity failures, be sure that:The security group and network access control list (ACL) rules allow connectivityYour instance can reach the public endpoint using an internet gateway or a network address translation (NAT) gatewayIf you're using VPC endpoints, the endpoint resolves to a VPC IP and the endpoint security group allows access from the source instanceReview your account, Region, and log group configurationsIn the CloudWatch agent configuration file:Be sure that the specified Region matches the console RegionVerify that logs are checked in the correct accountOptionally, you can use the common-config.toml file to override system defaults for the CloudWatch agent. These system defaults include the proxy, Region, and credential information for the agent. The file is available in the following locations.Linux:/opt/aws/amazon-cloudwatch-agent/etc/common-config.toml-or-/etc/amazon/amazon-cloudwatch-agent/common-config.tomlWindows:$Env:ProgramData\Amazon\AmazonCloudWatchAgent\common-config.tomlCheck your IAM permissionsThe CloudWatch agent uses credentials from either the IAM user or IAM role policy to push log events to the CloudWatch service. Before a log event can be published, you must create a log group and log stream. If there's no log group or log stream, the CloudWatch agent creates them.Confirm that your policy includes the following IAM permissions:"logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents","logs:DescribeLogStreams"Add any missing IAM permissions to the user policy or the role policy.Note: When creating IAM roles and users, it's a best practice to use the CloudWatchAgentServerPolicy and CloudWatchAgentAdminPolicy policies created by Amazon rather than custom policies.Resolve CloudWatch agent run errorsVerify that the CloudWatch agent is running. If the agent isn't running, check the log file for errors and resolve them. Log files are located in the following locations.Linux:/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.logWindows:$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.logNote: Logs might be specified in a custom logfile location. Check the agent configuration file to identify any custom log locations.In the agent configuration file, enable verbose debug logging using the debug parameter. If you're using the run_as_user parameter, confirm that the user has permissions to the log location path. Without the necessary permissions, the CloudWatch agent can't write logs to the location.Resolve timestamp issuesCheck for log event timestamps that are older than 14 days or more than two hours in the future. The PutLogEvents command doesn't allow log batches in either time frame.Also, verify that the system time service on the instance is correctly configured. For more information, see Set the time for your Linux instance or Set the time for a Windows instance.Related informationTroubleshooting the CloudWatch agentFollow"


Question: How do I troubleshoot problems while connecting to Amazon RDS for PostgreSQL or an Aurora PostgreSQL cluster using IAM authentication?

"I want to connect to my Amazon Relational Database Service (Amazon RDS) for PostgreSQL instance or an Amazon Aurora PostgreSQL-Compatible cluster. When I try to connect using AWS Identity Access Management (IAM) authentication, I get a "PAM authentication failed for user" error."

Answer: "I want to connect to my Amazon Relational Database Service (Amazon RDS) for PostgreSQL instance or an Amazon Aurora PostgreSQL-Compatible cluster. When I try to connect using AWS Identity Access Management (IAM) authentication, I get a "PAM authentication failed for user" error.Short descriptionYou might not be able to connect to your Amazon RDS for PostgreSQL or Aurora DB instance using IAM authentication for one of these reasons:IAM authentication is turned offInsufficient IAM role permissions or incorrect policyDatabase user is improperly configuredIncorrect connection stringIncorrect IAM entityResolutionIAM authentication is turned offBy default, IAM authentication is turned off. Review the configuration settings for your RDS for PostgreSQL instance or Aurora PostgreSQL cluster. Make sure that IAM authentication is turned on. From the Amazon RDS console, you can modify the Aurora cluster or RDS instance by choosing Database Authentication. Then, choose IAM database authentication and Continue to update your configuration settings.Note: When you choose Apply Immediately when updating your cluster configuration settings, all pending modifications are applied immediately. This action doesn't result in downtime.Insufficient IAM role permissionsTo successfully connect to your DB instance using IAM database authentication, you must have access to rds-db:connect action. The rds-db:connect action allows connections to the DB instance.For example:{    "Version": "2012-10-17",    "Statement": [    {        "Effect": "Allow",        "Action": [            "rds-db:connect"        ],        "Resource": [            " arn:aws:rds-db:region:account-id:dbuser:(DbiResourceId for RDS PostgreSQL or DbClusterResourceId for Aurora PostgreSQL)/db_user_name"        ]    }]Note: Replace db_user_name with the database account user that's associated with the IAM authentication.Also, make sure that you're using the correct resource ID (instead of specifying only the ARN). To find a DB instance's resource ID, choose the DB instance or cluster in the Amazon RDS console. Then, choose the Configuration tab to view the resource ID.For more information about the elements listed in the example IAM policy, see Creating and using an IAM policy for IAM database access.Note: IAM authentication doesn't support service-specific context keys that can be used in the condition element of policy statements. IAM authentication also doesn't support all global condition context keys. For more information about global condition context keys, see AWS global condition context keys in the IAM User Guide.Database user is improperly configuredTo use IAM authentication with PostgreSQL, grant the database user the rds_iam role. To confirm that this role is associated with the user, run this command:Note: This command runs only in the PostgreSQL client.\duYour output looks similar to this:List of roles   Role name    |                         Attributes                         |                          Member of                          -----------------+------------------------------------------------------------+-------------------------------------------------------------db_user_name    |                                                            | {rds_iam}                                postgres        | Create role, Create DB                                    +| {rds_superuser}                          | Password valid until infinity                              |You can also use this query, which runs on any platform:SELECT r.rolname,ARRAY(SELECT b.rolnameFROM pg_catalog.pg_auth_members mJOIN pg_catalog.pg_roles b ON (m.roleid = b.oid)WHERE m.member = r.oid) memberofFROM pg_catalog.pg_roles rWHERE r.rolname !~ '^pg_'ORDER BY 1;To create a user and grant the rds_iam role, run this command:CREATE USER db_user_name WITH LOGIN; GRANT rds_iam TO db_user_name;Incorrect connection stringAuthentication tokens consist of several characters that can be unwieldy to use on the command line. Save the token to an environment variable instead, and then use that variable when you connect to your PostgreSQL DB instance.For example:export RDSHOST='aurorapostgresql.cluster-abcdef12ghi.us-east-1.rds.amazonaws.com'export PGPASSWORD="$(aws rds generate-db-auth-token --hostname $RDSHOST --port 5432 --region us-east-1 --username db_user_name)"psql "host=$RDSHOST port=5432 sslmode=verify-full sslrootcert=global-bundle.pem dbname=postgres user=db_user_name password=$PGPASSWORD"For more information about how to connect to a PostgreSQL DB instance using an environment variable, see Connecting to an Aurora PostgreSQL cluster.If you get an error similar to the one in this example, then the client is trying to connect to the DB instance without SSL.FATAL: no pg_hba.conf entry for host "host.ip", user "<username>", database "postgres", SSL off FATAL: password authentication failed for user "db_user_name"IAM database authentication requires an SSL connection, and all data transmitted to and from your DB instance is encrypted. To avoid this error, make sure that the connection is using SSL. It's a best practice to use the sslmode option as verify-full and provide the certificate in the sslrootcert option in your connection string.To download the latest CA certificate, see Using SSL/TLS to encrypt a connection to a DB cluster.Incorrect IAM entityMake sure that the authentication token is being generated by the correct IAM role. To check the assumed role of the current IAM user, run this command:aws sts get-caller-identityIf the assumed role isn't correct, you can switch to the correct IAM user/role or assume the correct role. Be sure that you are using the correct IAM role to generate the authentication token.If you still receive an error similar to "PAM authentication failed for your user", check whether the AWS account is part of an AWS Organizations organization. If the account is part of an organization, then add rds-db:* to organization's SCP. For more information, see Creating, updating, and deleting service control policies.If the account is part of an organization, check whether there is a hierarchy of the IAM user that doesn't have the rds-db permission. For more information, see How to use service control policies to set permission guardrails across accounts in your AWS Organization.If you are using permissions boundaries for IAM entities, make sure that rds-db:connect action is allowed for your IAM user/role. For more information on permissions boundaries, see Permissions boundaries for IAM entities. To modify permissions boundaries for a user, see Changing the permissions boundary for a user.Follow"


Question: Why do I get the "webidentityerr" error when using AWS Load Balancer Controller in Amazon EKS?

"When I try to use the AWS Load Balancer Controller in Amazon Elastic Kubernetes Service (Amazon EKS), I receive a "WebIdentityErr" error message."

Answer: "When I try to use the AWS Load Balancer Controller in Amazon Elastic Kubernetes Service (Amazon EKS), I receive a "WebIdentityErr" error message.Short descriptionWhen you use the AWS Load Balancer Controller in Amazon EKS, you might see the following error:"failed to find existing LoadBalancer due to WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403"The error occurs for the following reasons:Incorrect service account configurationsIncorrect trust relationship of the AWS Identity and Access Management (IAM) role that you use in the service accountWith AWS Load Balancer Controller, worker nodes perform the tasks. You must use IAM permissions to grant these worker nodes access to the Application Load Balancer or Network Load Balancer resources. To set up the IAM permissions, use IAM roles for the service account. Or, attach the IAM permissions directly to the worker node's IAM roles. For more information, see Load Balancer Controller installation on the Kubernetes website.ResolutionIncorrect service account configurationsTo check if you properly configured your service account, complete the following steps:1.    Verify the service account name that's defined in your deployment:kubectl describe deploy LOAD_BALANCER_CONTROLLER -n kube-system | grep -i "Service Account"2.    Describe the service account:kubectl describe sa LOAD_BALANCER_CONTROLLER -n kube-system3.    Verify the service account annotation for the IAM role:Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxx:role/AMAZON_EKS_LOAD_BALANCER_CONTROLLER_ROLE4.    If this annotation is missing or incorrect, then update the annotation. Make sure that you properly associated the IAM role to a service account:kubectl annotate serviceaccount -n SERVICE_ACCOUNT_NAMESPACE SERVICE_ACCOUNT_NAME \ eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME5.    Delete and recreate any existing Amazon EKS pods that are associated with the service account. Be sure to apply the credential environment variables:kubectl delete pods AWS_LOAD_BALANCER_CONTROLLER_POD -n KUBBE_SYSTEMIncorrect trust relationship between the IAM role used and service accountThe following examples show some common mistakes when establishing the trust relationship between your IAM role and service account:Example 1: The IAM role or trust relationship isn't properly defined for the "sts:AssumeRoleWithWebIdentity" actionVerify that the trust relationship is properly defined for the sts:AssumeRoleWithWebIdentity action (and not the sts:AssumeRole action).The following example is a trust relationship that isn't properly defined:{      "Sid": "",      "Effect": "Allow",      "Principal": {        "Service": "xxxxx.amazonaws.com"      },      "Action": "sts:AssumeRole"}To correct this issue, define the trust relationship for the sts:AssumeRoleWithWebIdentity action:{    "Version": "2012-10-17",    "Statement": [      {        "Effect": "Allow",        "Principal": {          "Federated": "arn:aws:iam::AWS_ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC-PROVIDER_ID"        },        "Action": "sts:AssumeRoleWithWebIdentity",        "Condition": {          "StringEquals": {            "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC_PROVIDER_ID:sub": "system:serviceaccount:kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",            "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"          }        }      }    ]  }Note: Be sure to replace all variables with your own values.To use the same IAM role for multiple clusters in one account, define the trust relationship as in the following example:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Federated": "arn:aws:iam::AWS-ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC-PROVIDER_ID"      },      "Action": "sts:AssumeRoleWithWebIdentity",      "Condition": {        "StringEquals": {          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:sub": "system:serviceaccount: kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"        }      }    },    {      "Effect": "Allow",      "Principal": {        "Federated": "arn:aws:iam::AWS_ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID"      },      "Action": "sts:AssumeRoleWithWebIdentity",      "Condition": {        "StringEquals": {          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID:sub": "system:serviceaccount: kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"        }      }    }  ]}Example 2: Incorrect OIDC provider ID when creating an Amazon EKS clusterProperly create and verify an OpenID Connect (OIDC) provider for your Amazon EKS cluster. Verify that the OIDC provider ID and the associated AWS Region are correctly listed. Otherwise, you receive a WebIdentityErr error.Example 3: Service account name or its namespace not correctly enteredWhen you update your AWS Load Balancer Controller deployment, enter the correct service account name and its namespace.Example 4: Missing "sts.amazonaws.com" steps from the trust relationshipIf the service role that's associated with your EKS pod can't perform the STS operation on the AssumeRoleWithWebIdentity action, then update the trust relationship. To perform an STS operation, the trust relationship must include sts.amazonaws.com:"Condition": {        "StringEquals": {          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:sub": "system:serviceaccount:kube-system:LOAD_BALANCER_CONTROLLER_ACCOUNT",          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"        }}For more information about IAM conditions with multiple keys or values, see Creating a condition with multiple keys or values.Follow"


Question: How do I troubleshoot Amazon ECS tasks stopping or failing to start while my container exits?

"My Amazon Elastic Container Service (Amazon ECS) container exits unexpectedly, and tasks stop or fail to start. How can resolve the issue?"

Answer: "My Amazon Elastic Container Service (Amazon ECS) container exits unexpectedly, and tasks stop or fail to start. How can resolve the issue?Short descriptionYour containers can exit due to application issues, resource constraints, or other issues.For task failure due to image issues, see How do I resolve the "Image does not exist" error when my tasks fail to start in my Amazon ECS cluster?For AWS Fargate tasks that fail due to network configuration or resource constraint issues, see Stopped tasks error codes.ResolutionTo identify why your tasks stopped, follow these troubleshooting steps:Check for diagnostic information in the service event log.Check stopped tasks for errors. Note: You can see stopped tasks in the returned results for at least one hour.If you already have a log driver configured, then check your application logs for application issues. Otherwise, use the log configuration options in your task definition to send logs to a custom log driver for the container. For example, you can send the logs to Amazon CloudWatch or use a supported log driver. Note the following information on logs, depending on your task's launch type:For ECS tasks other than Fargate: If you're using the default json-file logging driver with the Amazon Elastic Compute Cloud (Amazon EC2) launch type, run the docker logs yourContainerID command. This command checks the Docker logs of the container on your ECS container instance.For Fargate tasks: Captured logs show the command output that you see in an interactive terminal if you run the container locally, in the STDOUT and STDERR I/O streams. The awslogs log driver passes these logs from Docker to Amazon CloudWatch Logs.To address memory constraint issues, follow the instructions at How can I allocate memory to tasks in Amazon ECS?If the awslogs log driver is configured in your task definition, then check Viewing awslogs container logs in CloudWatch Logs.Follow"


Question: How can I stop my Amazon ECS service from failing to stabilize in AWS CloudFormation?

My Amazon Elastic Container Service (Amazon ECS) service fails to stabilize in AWS CloudFormation. I get the following error: "Service arn:aws:ecs:us-east-accountID:service/ServiceName did not stabilize."

Answer: "My Amazon Elastic Container Service (Amazon ECS) service fails to stabilize in AWS CloudFormation. I get the following error: "Service arn:aws:ecs:us-east-accountID:service/ServiceName did not stabilize."Short descriptionA service created in Amazon ECS fails to stabilize if it isn't in the state specified by the AWS CloudFormation template. To confirm that a service launched the desired number of tasks with the desired task definition, AWS CloudFormation makes repeated DescribeService API calls. These calls check the status of the service until the desired state is met. The calling process can take up to three hours. Then, AWS CloudFormation times out, and returns the "Service ARN did not stabilize" message. While AWS CloudFormation checks the status of the service, the stack that contains the service remains in the CREATE_IN_PROGRESS or UPDATE_IN_PROGRESS state and can't be updated.If you can't fix the underlying issue with your Amazon ECS service tasks immediately and you don't want to wait for the DescribeService API calls to time out, then you can manually force the state of the Amazon ECS service resource in AWS CloudFormation into a CREATE_COMPLETE state. To do this, manually set the desired count of the service to zero in the Amazon ECS console to stop running tasks. AWS CloudFormation then considers the update as successful, because the number of tasks equals the desired count of zero.Important: Manually forcing AWS CloudFormation into a CREATE_COMPLETE state isn't a best practice for production services, because all tasks are stopped, and doing this can cause a production outage.Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.ResolutionVerify resource creation1.    In your AWS CloudFormation template, create an AWS::ECS::Service resource. For example:Resources:ECSServiceA: Type: AWS::ECS::ServiceProperties:  DesiredCount: 1  Cluster: awsExampleECSCluster  LaunchType: EC2  ServiceName: "MyNginxService2"  TaskDefinition: NginxTask:12.    Open the AWS CloudFormation console, and then select your stack.3.    Choose the Events tab, and then verify that your resource is being created.Update the desired count of the serviceYou can update the desired count of the service to your original value with either the AWS CLI or Amazon ECS console.Using the AWS CLI:1.    To describe the service and list the service events, run the following command:aws ecs describe-services --cluster awsExampleECSCluster --services MyNginxService22.    To update the desired count of the service, run the following command:aws ecs update-service --cluster awsExampleECSCluster --service MyNginxService2 --desired-count 03.    Update --desired-count to your original value.Using the Amazon ECS console:1.    Open the Amazon ECS console.2.    In the navigation pane, choose Clusters, and then select the cluster that contains the Amazon ECS service that you created.3.    On the Clusters page, choose the cluster that contains the Amazon ECS service that you created.4.    On the page for the cluster that you selected, in the Service Name column, choose your service.5.    Choose the Events tab, and then choose Update.6.    On the Configure service page, for Number of tasks, enter 0.7.    Choose Next step to step through to the end of the Update Service wizard, and then choose Update Service.The service now reaches a steady state and transitions the Amazon ECS service resource in AWS CloudFormation to CREATE_COMPLETE or UPDATE_COMPLETE.Important: To make your AWS CloudFormation stack sync with the Amazon ECS service properties after you fix the issue with the underlying tasks, you must manually change the desired count (DesiredCount) back to the original value from your template.Related informationUpdating a serviceupdate-serviceservices-stableFollow"


Question: How do I troubleshoot 504 errors returned while using a Classic Load Balancer?

"I see HTTP 504 errors in Classic Load Balancer access logs, Amazon CloudWatch metrics, or when connecting to my service through a Classic Load Balancer."

Answer: "I see HTTP 504 errors in Classic Load Balancer access logs, Amazon CloudWatch metrics, or when connecting to my service through a Classic Load Balancer.ResolutionAn HTTP 504 error is an HTTP status code that indicates a gateway or proxy has timed out. When troubleshooting, investigate the following:Check your load balancer’s idle timeout, and then modify if necessaryThe most common reason for an HTTP 504 error is that a corresponding instance didn't respond to the request within the configured idle timeout. By default, the idle timeout for Classic Load Balancer is 60 seconds.If CloudWatch metrics are turned on, check CloudWatch metrics for your load balancer. When latency data points are equal to your configured load balancer timeout value, and HTTPCode_ELB_5XX has data points, at least one request has timed out.To resolve this, you can do one of two things:Modify the idle timeout for your load balancer so that the HTTP request is completed within the idle timeout period.Tune your application to respond more quickly.Make sure that your backend instances keep connections openWhen the backend instance closes a TCP connection to the load balancer before it reaches its idle timeout value, an HTTP 504 error might display.To resolve this, activate keep-alive settings on your backend instances, and then set the keep-alive timeout to a value greater than the load balancer’s idle timeout.(Apache only) Turn off TCP_DEFER_ACCEPTWhen TCP_DEFER_ACCEPT is activated for Apache backend instances, the load balancer initiates a connection by sending a SYN to the backend instance. The backend instance then responds with a SYN-ACK, and the load balancer sends an empty ACK to the backend instance.Because the last ACK is empty, the backend doesn't accept the ACK, and instead resends a SYN-ACK to the load balancer. This can result in a subsequent SYN retry timeout. When FIN or RST aren’t sent before the backend instance closes the connection, the load balancer considers the connection established, but it's not. Then, when the load balancer sends requests through this TCP connection, the backend responds with an RST, generating a 504 error.To resolve this, set both AcceptFilter http and AcceptFilter https to none.(Apache only) Turn off the event MPM, and optimally configure the prefork and worker MPMsIt’s a best practice to turn off event MPM on backend instances that are registered to a load balancer. Because the Apache backend dynamically closes connections, these closed connections might result in HTTP 504 errors.For optimal performance when using the prefork and worker MPMs, and presuming the load balancer is configured with a 60-second idle timeout, use these values:mod_prefork MPMmod_worker MPMTimeout6565KeepAliveTimeout6565KeepAliveOnOnMaxKeepAliveRequests100000AcceptFilter httpnonenoneAcceptFilter httpsnonenoneRelated informationMonitor your Classic Load BalancerTroubleshoot a Classic Load Balancer: HTTP errorsWhat are the optimal settings for using Apache or NGINX as a backend server for ELB?Follow"


Question: How do I initiate rolling updates when there are no changes to the launch configuration in CloudFormation?

"I want to initiate rolling updates in my Auto Scaling group on every AWS CloudFormation stack update, without modifying the launch configuration."

Answer: "I want to initiate rolling updates in my Auto Scaling group on every AWS CloudFormation stack update, without modifying the launch configuration.Short descriptionYou can initiate rolling updates for an Auto Scaling group only if you meet specific conditions of the UpdatePolicy attribute.To initiate rolling updates, you can create a toggle parameter in the launch configuration of your CloudFormation template. However, if you change the value of the toggle parameter during a stack update, then the UserData property is modified. Any modification to UserData requires replacement. CloudFormation detects the modification to UserData, and then replaces the LaunchConfiguration resource. This replacement initiates the Auto Scaling rolling update, as defined by the UpdatePolicy attribute.ResolutionThe following steps assume that your AutoScalingRollingUpdate policy is configured for your Auto Scaling group, and your Auto Scaling group is configured to reference LaunchConfiguration.Important: Make sure that you don't disrupt other elements in the UserData property when you add the toggle parameter to your template. Also, don't add the toggle parameter before cfn-signal.1.    In your CloudFormation template, define Toggle as the parameter.JSON:"Parameters": {        "Toggle": {            "Type": "String",            "AllowedValues": ["true","false"],            "Default": "true"        }    }YAML:Parameters:  Toggle:    Type: String    AllowedValues:      - 'true'      - 'false'    Default: 'true'2.    In the launch configuration of your template, reference the toggle parameter in the UserData property, and then launch your stack. See the following JSON and YAML examples.JSON:"LaunchConfig" : {  "Type" : "AWS::AutoScaling::LaunchConfiguration",  "Properties" : {     "ImageId" : { "Ref" : "ImageID" },     "UserData" : { "Fn::Base64" : { "Ref" : "Toggle" }         ...         ...     },          "InstanceType" : { "Ref" : "Type" }  }}YAML:LaunchConfig:  Type: 'AWS::AutoScaling::LaunchConfiguration'  Properties:    ImageId: !Ref ImageID    UserData:      'Fn::Base64': !Ref Toggle      ...      ...    InstanceType: !Ref TypeImportant: To initiate rolling updates when you update your stack, change the value of the parameter from true to false or false to true, depending on its initial setting.Note: You can use the toggle parameter solution on properties where an update requires replacement, such as LaunchConfigurationName, and for resources such as, AWS::EC2::LaunchTemplate.Follow"


Question: Can I safely delete the empty files with the _$folder$ suffix that appear in my Amazon S3 bucket when I use Amazon EMR with Amazon S3?

"When I use Amazon EMR to transform or move data into or out of Amazon Simple Storage Service (Amazon S3), several empty files with the "_$folder$" suffix appear in my S3 buckets. What are these files, and is it safe to delete them?"

Answer: "When I use Amazon EMR to transform or move data into or out of Amazon Simple Storage Service (Amazon S3), several empty files with the "_$folder$" suffix appear in my S3 buckets. What are these files, and is it safe to delete them?ResolutionThe "_$folder$" files are placeholders. Apache Hadoop creates these files when you use the -mkdir command to create a folder in an S3 bucket. Hadoop doesn't create the folder until you PUT the first object. If you delete the "_$folder$" files before you PUT at least one object, Hadoop can't create the folder. This results in a "No such file or directory" error.In general, it's a best practice not to delete the "_$folder$" files. Doing so might cause performance issues for the Amazon EMR job. The exception is if you manually delete the folder from Amazon S3 and then try to recreate the folder in an Amazon EMR job or with Hadoop commands. If you don't delete the "_$folder$" files before you try to recreate the folder, you get the "File exists" error.Related informationUpload data to Amazon S3Configure an output locationFollow"


Question: How can I troubleshoot problems using Amazon Data Lifecycle Manager?

"My Amazon Data Lifecycle Manager policy is in an error state, or does not act as expected regarding snapshots. How can I troubleshoot these issues?"

Answer: "My Amazon Data Lifecycle Manager policy is in an error state, or does not act as expected regarding snapshots. How can I troubleshoot these issues?Short descriptionThe following are common reasons that your lifecycle policy is in an error state, or fails to create or copy Amazon Elastic Block Store (Amazon EBS) snapshots:The lifecycle policy isn't turned on.There are incorrect permissions on the policy.You're using an AWS Identity and Access Management (IAM) role other than the default AWSDataLifecycleManagerDefaultRole, and there are issues with trust relationships.There are duplicate tags on the policy.A tag defined in the policy is already in use.Your resources are encrypted.ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, then make sure that you’re using the most recent AWS CLI version.Snapshots aren't created as expectedIf snapshots aren't created, then verify that the lifecycle policy is turned on.1.    Open the Amazon Elastic Compute Cloud (Amazon EC2) console.2.    Select Lifecycle Manager, and then verify that the policy State is set to ENABLED.3.    If the policy isn't set to ENABLED, then choose Actions, Modify Snapshot Lifecycle Policy.Note: If the policy state is ERROR, see the following section, DLM policy is in the ERROR state.4.    Select Enable policy, and then select Update policy.Note: It can take up to an hour after creating a lifecycle policy for snapshots to be created. After at least an hour has passed, open the Amazon EC2 console. Then, select Snapshots to verify that snapshots are being created.Unable to copy snapshots between RegionsIf the copied snapshot copied is encrypted, then the user must have access to the source and destination AWS Key Management Service (AWS KMS) key. For more information, see Determining access to an AWS KMS key.The lifecycle policy is in an error stateA lifecycle policy in the error state can be caused by one or more of these issues:There is a problem with your resource tags.The Amazon Data Lifecycle Manager permissions aren't correct.The IAM permissions aren't correct.In addition, if you're using a custom IAM role, a trust relationship might not be attached to the role.View information about what caused the error state by checking Amazon CloudWatch Events. The following are common errors and resolutions:Duplicate tag keyIf there are duplicate tags in your lifecycle policy, then a CloudTrail Event similar to the following appears. In the following example, the tag key Name is duplicated in the policy.CreateSnapshot @2018-12-24T20:25:58.000Z UTC"errorCode": "Client.InvalidParameterValue", "errorMessage": "Duplicate tag key 'Name' specified.",     "requestParameters": {        "volumeId": "vol-xxxxxxxxxxxx",        "description": "Created for policy: policy-xxxxschedule: First Schedule",1.    Open the Amazon EC2 console.2.    Select Lifecycle Manager.3.    Select your lifecycle policy, and then choose Actions, Modify Lifecycle Policy.4.    In the Tag created EBS snapshots section, change the Key on the duplicated tag to a unique name.5.    Select Update policy.Tag (Name) is already defined in resource id vol-xxxxxxxxxxxxIf a tag that's defined in your lifecycle policy is already in use in a different lifecycle policy, then you might have an issue if:The lifecycle policy is in the same account, andThe lifecycle policy is for the same resource.In this case, a CloudTrail Event similar to the following appears:CreateSnapshots---------------------------------------------------------------------------------    "eventVersion": "1.05",    "userIdentity": {        "type": "AssumedRole",   "eventTime": "2020-01-xxxxxxxx",    "eventSource": "ec2.amazonaws.com",    "eventName": "CreateSnapshots",    "awsRegion": "us-east-1",    "sourceIPAddress": "dlm.amazonaws.com",    "userAgent": "dlm.amazonaws.com",    "errorCode": "Client.InvalidParameterCombination",    "errorMessage": "Tag (Name) is already defined in resource id vol-xxxxxxxx.",    "requestParameters": {"requestParameters": {        "CreateSnapshotsRequest": {            "Description": "Created for policy: policy-xxxxxxxschedule: Default Schedule",            "InstanceSpecification": {                "ExcludeBootVolume": false,                "InstanceId": "i-xxxxxxx"            },A volume or instance can have more than one policy associated with it, but tags can't be duplicated across policies. For more information, see Considerations for Amazon Data Lifecycle Manager.To correct this error, do the following:1.    View your lifecycle policies to determine which tag is duplicated.2.    Create a new lifecycle policy using a different tag, or edit your current lifecycle policy to use a different tag.Client.AuthFailureThe "Client.AuthFailure" error might occur if the custom lifecycle policy or the IAM user don't have permissions set correctly. The following is an example of a Client.AuthFailure caused by an inaccessible key:"Client.AuthFailure","errorMessage": "The specified keyIdarn:aws:kms:us-west-1:xxxxxxxxxxxxx:key/4ad6a1d7-53ac-45a3-8f08-e6eccc948fdd is not accessible",For instructions on setting permissions for Amazon Data Lifecycle Manager, see Permissions for Amazon Data Lifecycle Manager.For instructions on setting permissions for IAM users to use Amazon Data Lifecycle Manager, see Permissions for IAM users.Related informationAmazon Data Lifecycle Manager API referenceAWS CLI Command reference - dlmFollow"


Question: Why did my Auto Scaling group scale down?

My Auto Scaling group scaled down without my intervention. Why did this happen?

Answer: "My Auto Scaling group scaled down without my intervention. Why did this happen?Short descriptionScale-downs are user initiated, or triggered by configured scale-down policies and scheduled scaling. When a scale-down occurs, an instance is terminated according to the configured termination policy.ResolutionView your Auto Scaling group scaling history in the Amazon EC2 console, using the AWS Command Line Interface (AWS CLI), or using the AWS API.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.Policy-based scale downIf your Auto Scaling group scales down because of a scale-down policy, a message similar to the following appears in the scaling history:At 2016-05-08T13:55:14Z a monitor alarm My-Scale-Down-Alarm in state ALARMtriggered policy Decrease Group Size changing the desired capacity from 4 to 3.You can adjust your Auto Scaling group scaling policy using the Amazon EC2 console or with the AWS CLI put-scaling-policy command. To configure when a scale-down occurs, adjust the associated Amazon CloudWatch alarm. Or, create a new alarm and then associate the new alarm with the Auto Scaling group scaling policy.User-initiated scale downA scale-down triggered by a user displays an event similar to the following in the scaling history:At 2016-05-13T15:03:47Z a user request update of AutoScalingGroup constraintsto min: 12, max: 20, desired: 13 changing the desired capacity from 14 to 13.You can determine the user that made the API call by viewing your AWS CloudTrail logs.Note: CloudTrail must be configured and enabled before you can begin recording API calls.Scheduled scalingA scale-down initiated by a scheduled scale down action displays an event similar to the following in the scaling history:At 2016-02-12T16:01:25Z a scheduled action update of AutoScalingGroup constraintsto min: 1, max: 5, desired: 2 changing the desired capacity from 3 to 2. At2016-02-12T16:01:25Z the scheduled action ScaleDown executed. Setting max sizefrom 1 to 5. Setting desired capacity from 3 to 2.To view scheduled scaling using the AWS CLI, run the following command. Replace MY-ASG-NAME with the name of your Auto Scaling group.aws autoscaling describe-scheduled-actions --auto-scaling-group-name MY-ASG-NAMEYou can also view and manage scheduled scaling using the Amazon EC2 console. For more information, see Create and manage scheduled actions (console).For more information about schedule-based Amazon EC2 Auto Scaling, see Scheduled scaling.Related informationTutorial: Set up a scaled and load-balanced applicationMonitoring CloudWatch metrics for your Auto Scaling groups and instancesGetting Amazon SNS notifications when your Auto Scaling group scalesLogging Amazon EC2 Auto Scaling API calls with AWS CloudTrailAWS CLI autoscaling commandsFollow"


Question: How can I troubleshoot issues with EventBridge rules?

"An event occurred that matched my Amazon EventBridge rule. However, my rule isn't functioning correctly. How can I troubleshoot this?"

Answer: "An event occurred that matched my Amazon EventBridge rule. However, my rule isn't functioning correctly. How can I troubleshoot this?Short descriptionDetermine if the problem is that the rule isn't triggering or if the target wasn't invoked. After you determine the source of the problem, you can validate the incoming event, or you can validate the target.ResolutionVerify if the rule isn't being triggered or if the target isn't being invokedTo do this, use the corresponding EventBridge metrics to review on the rule performance.The TriggeredRules metric illustrates the number of times a rule matched an event or was executed. This metric is useful in confirming whether a scheduled rule ran or if a rule matched a specific event. After the rule successfully triggered, EventBridge forwards the event to the target.An Invocations datapoint is generated when a rule invokes a target. EventBridge makes multiple attempts if it encounters difficulty delivering the event to the target. A FailedInvocations datapoint is issued when EventBridge permanently fails to invoke the target. FailedInvocations indicate problems with the target configuration.Review Amazon CloudWatch metrics for the EventBridge rule by doing the following:Open the CloudWatch console.Select All Metrics.Select the AWS/Events namespace.Select the TriggerRules, Invocations, and FailedInvocations (if available) metrics for the rule in question. These metrics can be viewed with the SUM statistic.Validate the incoming eventFor event-based rules, the event pattern must be configured to match the desired event. You can validate the event pattern using the EventBridge console during rule creation. EventBridge also provides the TestEventPattern API for event pattern validation.If the event in question is captured by AWS CloudTrail, then you can retrieve the event from CloudTrail. Then confirm that the provided event pattern is correct.Note that some AWS services are only available in the Region us-east-1. For example, IAM API calls are only published in us-east-1. This means that the corresponding EventBridge rule must be created in the same Region.Validate the targetWhen rules are created using the EventBridge console, the console automatically adds the required permissions to the following:The IAM role associated with the EventBridge rule.The resource policy associated with the target.If the rule is deployed using AWS SDK, the AWS CLI, or AWS CloudFormation, then you must explicitly configure the permissions.EventBridge must be given the appropriate access to invoke the target. Depending on the target, confirm that the appropriate IAM role or resource policy has the correct permissions. FailedInvocations datapoints generate due to inadequate target permissions.If there are no FailedInvocations datapoints, then EventBridge delivered the event to the target successfully. However, the target might have encountered its own issues. For example, an AWS Lambda target might have encountered errors or throttling independent of EventBridge. For the timestamp when the target was invoked by the EventBridge rule, review the target's CloudWatch metrics and any relevant logs.An Amazon Simple Queue Service (Amazon SQS) dead-letter queue (DLQ) can be associated with the target. Any events that failed to be delivered to the target are sent to the dead-letter queue. This is useful to see more details on failed events. For example, incorrectly structuring the event with Input Transformer can result in input validation errors on the target.Related informationTroubleshooting Amazon EventBridgeWhy wasn't my Lambda function triggered by my CloudWatch Events rule?Follow"


Question: How do I set up a NAT gateway for a private subnet in Amazon VPC?

I have Amazon Elastic Compute Cloud (Amazon EC2) instances in a private subnet of my Amazon Virtual Private Cloud (Amazon VPC). How can I configure these instances to communicate securely with the internet?

Answer: "I have Amazon Elastic Compute Cloud (Amazon EC2) instances in a private subnet of my Amazon Virtual Private Cloud (Amazon VPC). How can I configure these instances to communicate securely with the internet?Short descriptionA network address translation (NAT) gateway allows EC2 instances to establish outbound connections to resources on internet without allowing inbound connections to the EC2 instance. It's not possible to use the private IP addresses assigned to instances in a private VPC subnet over the internet. Instead, you must use NAT to map the private IP addresses to a public address for requests. Then, you must map the public IP address back to the private IP addresses for responses.ResolutionCreate a public VPC subnet to host your NAT gateway.Create and attach an internet gateway to your VPC.Create a custom route table for your public subnet with a route to the internet gateway.Verify that the network access control list (ACL) for your public VPC subnet allows inbound traffic from the private VPC subnet. For more information, see Work with network ACLs.Create a public NAT gateway then create and associate your new or existing Elastic IP address.Update the route table of your private VPC subnet to point internet traffic to your NAT gateway.Test your NAT gateway by pinging the internet from an instance in your private VPC subnet.Related informationMonitor NAT gateways with Amazon CloudWatchFollow"


Question: How can I sign or share a compliance report that requires an NDA using AWS Artifact?

I need to sign or share a nondisclosure agreement (NDA) with Amazon Web Services (AWS) for security compliance or auditing purposes.

Answer: "I need to sign or share a nondisclosure agreement (NDA) with Amazon Web Services (AWS) for security compliance or auditing purposes.Short descriptionYou and your AWS Partner Network (APN) can use an AWS account to access reports requiring an NDA using AWS Artifact.ResolutionFollow the instructions for downloading reports in AWS Artifact.Reports requiring an NDA are accessible by signing the terms and conditions in AWS Artifact. You must accept the AWS Artifact NDA to access and download confidential documents in AWS Artifact. You must sign the NDA each time that you access confidential reports.If you have an existing NDA with Amazon covering the same confidential information as provided in AWS Artifact, then your existing NDA applies. You can use your existing NDA instead of the AWS Artifact NDA.For more information, see AWS Artifact Agreements.Note: Reports requiring approval from Amazon are reviewed and approved within 24 hours. For more information, see Getting Started with AWS Artifact.Related informationHow can I download and share AWS Artifact documents with regulators and auditors, or with others in my company?AWS Artifact FAQsFollow"