Question: How can I get my Amazon ECS tasks running using the Amazon EC2 launch type to pass the Application Load Balancer health check in Amazon ECS?

An Application Load Balancer health check for an Amazon Elastic Compute Cloud (Amazon EC2) instance in Amazon Elastic Container Service (Amazon ECS) is returning an unhealthy status. I want my EC2 instance to pass the health check.

Answer: "An Application Load Balancer health check for an Amazon Elastic Compute Cloud (Amazon EC2) instance in Amazon Elastic Container Service (Amazon ECS) is returning an unhealthy status. I want my EC2 instance to pass the health check.Short descriptionWhen your Amazon ECS task fails the load balancer health check, you receive one of the following errors from your Amazon ECS service event message:(service AWS-service) (port 8080) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789) due to (reason Health checks failed with these codes: [502 or 504]) or (request timeout)(service AWS-Service) (port 8080) is unhealthy in target-group tf-20190411170 due to (reason Health checks failed)(service AWS-Service) (instance i-1234567890abcdefg) (port 443) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789) due to (reason Health checks failed)You might also receive the following error from your Amazon ECS task console:Task failed ELB health checks in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789)If you're getting the error (service AWS-Service) (task c13b4cb40f1f4fe4a2971f76ae5a47ad) failed container health checks, then see How do I troubleshoot the container health check failures for Amazon ECS tasks?Note: An Amazon ECS task can return the unhealthy status for many reasons. If the following steps don't resolve your issue, then see Troubleshooting service load balancers.Tip: To find out why your ECS task was stopped, see Checking stopped tasks for errors.ResolutionTo troubleshoot your load balancer health check issues on your Amazon ECS task and pass the Application Load Balancer health check, check the following:Connectivity between your load balancer and Amazon ECS taskHealth check settings of your target groupStatus and configuration of the application in your ECS containerStatus of the container instanceCheck the connectivity between your load balancer and Amazon ECS taskTo make sure that your load balancer is allowed to perform health checks on your Amazon ECS tasks, be sure of the following.The security groups attached to your load balancer and container instance or the ECS task elastic network interface for awsvpc network mode are configured correctlyIt's a best practice to configure different security groups for your load balancer and container instance or task elastic network interface. With this approach, you allow all traffic between your load balancers and container instances or task elastic network interface. You can also enable your container instances to accept traffic on the port that's specified for the task.Confirm that the security group associated with your load balancer allows egress traffic to your container instances or task elastic network interface on the registered port. Confirm that the same is true for the health check port associated with your container instance, if applicable.Confirm that the security group associated with your container instance or task elastic network interface allows all ingress traffic on the task host port range from the security group associated with your load balancer. To check the security group associated with your load balancer, see Security groups for your Application Load Balancer.Important: When you use dynamic port mapping, the service is exposed on the dynamic port (typically ports 32768 - 65535) rather than on the host port. In this case, confirm that your container instance security group reflects the ephemeral port range in the ingress rules for the load balancer as a source.Your load balancer is configured in the same Availability Zone as your container instance or ECS task elastic network interface for awsvpc network modeWhen you enable an Availability Zone for your load balancer, Elastic Load Balancing creates a load balancer node in the Availability Zone. If you register targets in an Availability Zone, but don't enable the Availability Zone, then these registered targets do not receive traffic. For more information, see Availability Zones and load balancer nodes.To find out the Availability Zones that your load balancer is configured for, do the following:1.    Open the Amazon EC2 console.2.    In the navigation pane, under Load Balancing, choose Load Balancers.3.    Select the load balancer that you're using for your Amazon ECS service.4.    On the Description tab, you can view the Availability Zones under the Availability Zones field.Note: For an Application Load Balancer, you can enable or disable the Availability Zones at any time. For a Network Load Balancer, you can't disable the Availability Zones after you enable it, but you can enable additional Availability Zones.If you're using Application Load Balancers, then cross-zone load balancing is always turned on. If you're using Network Load Balancers, then cross-zone load balancing is turned off, by default. After you create the Network Load Balancer, you can turn cross-zone load balancing on or off at any time. For more information, see How Elastic Load Balancing works.To find out the Availability Zones that your container instances are configured for, do the following:1.    Open the Amazon EC2 console.2.    In the navigation pane, under Auto Scaling, choose Auto Scaling Groups.3.    Select the container instance Auto Scaling group that's associated to your cluster.4.    On the Details tab, under Network, verify that the Availability Zones listed match the Availability Zones listed for your load balancer.To modify the Availability Zones of your cluster, open the AWS CloudFormation console, choose the CloudFormation stack for your cluster, and update the subnets configuration.To find out the Availability Zones that your task elastic network interface for awsvpc is configured for, do the following:1.    Open the Amazon ECS console.2.    In the navigation pane, choose Clusters, and then select the cluster that contains your service.3.    On the Services tab of your cluster's page, in the Service Name column, select the service that you want to check.4.    Choose Details, and then choose Allowed subnets to view the subnets that are enabled for the service.5.    You can view the subnets in the Amazon VPC console.6.    Verify that the Availability Zones of the subnets match the Availability Zones listed for your load balancer.Note: You can't change the subnet configuration of an Amazon ECS service from the Amazon ECS console. You can do so using the AWS Command Line Interface (AWS CLI) update-service command.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.The network access control list (ACL) associated with the subnets of your load balancer and ECS container instances or ECS task elastic network interface for awsvpc network mode are configured correctlyThe subnets for your load balancer and your container instance or task elastic network interface might be different. To make sure that traffic is allowed between these subnets, check the following:Be sure that the network ACL associated with the subnets for your load balancer allows ingress traffic on the ephemeral ports (1024 - 65535) and listener port. Verify that the network ACL also allows egress traffic on the health check and ephemeral ports.Be sure that the network ACL associated with the subnets for your container instance or task elastic network interface for awsvpc mode allows ingress traffic on the health check port. Verify that the network ACL allows egress traffic on the ephemeral ports.For more information about network ACLs, see Work with network ACLs.Check the health check settings of your target groupTo be sure that the health check settings for your target group are configured correctly, do the following:1.    Open the Amazon EC2 console.2.    In the navigation pane, under Load Balancing, choose Target Groups.3.    Select your target group.Important: Use a new target group. Avoid adding targets to the target group manually, because Amazon ECS automatically registers and de-registers containers with the target group.4.    On the Health checks tab, do the following:Check that the Port and Path fields are configured correctly. If the Port field isn't configured correctly, then your load balancer might de-register the container.For Port, be sure that you choose traffic port.Note: If you choose Override, then confirm that the port specified matches the task host port.For Timeout, be sure that the response timeout value is set correctly.Note: The response timeout is the amount of time that your container has to return a response to the health check ping. If this value is lower than the amount of time required for a response, then the health check fails.Check the status and configuration of the application in your ECS containerConfirm that the application in your ECS container responds to your load balancer health checkTo make sure that the application in your ECS container responds to your load balancer health check correctly, do the following:Check that the ping port and the health check path for your target group are configured correctly.Monitor the CPU and memory utilization metrics for the ECS service. For example, high CPU can make your application unresponsive and cause a 502 error or timeout.Define a minimum health check grace period. This setting instructs the service scheduler to ignore the Elastic Load Balancing health checks for a pre-defined time period after a task has been instantiated. Your Amazon ECS task might require a longer health check grace period for registering the Network Load Balancer.Check your application logs for application errors. For more information, see Viewing awslogs container logs in CloudWatch Logs.Confirm that the application in your ECS container returns the correct response codeWhen the load balancer sends an HTTP GET request to the health check path, the application in your ECS container is expected to return the default 200 OK response code.Note: If you use an Application Load Balancer, you can update the Matcher setting to a response code other than 200. For more information, see Health checks for your target groups.1.    Connect to your container instance using SSH.2.    (Optional) Install curl with the command appropriate for your system.For Amazon Linux and other RPM-based distributions, run the following command:sudo yum –y install curlFor Debian-based systems (such as Ubuntu), run the following command:sudo apt-get install curl3.    To get the container ID, run the following command:docker psNote: The port for the local listener is displayed in the command output under PORTS at the end of the sequence.4.    To get the IP address of the container, run the docker inspect command:Be sure to replace 112233445566 with the ID number of the container.$ IPADDR=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' 112233445566)Note: The IP address of the container is saved in IPADDR. Use this command only if you're using the BRIDGE network mode.If you're using awsvpc network mode, then use the task IP address assigned to the task elastic network interface. If you're using the HOST network mode, then use the IP address of the host that the task is exposed through.5.    To get the status code, run a curl command that includes IPADDR and the port of the local listener. For example, if you run the curl command on a container listening on port 8080 with the health check path of /health, the command must return the response code 200 OK:curl -I http://${IPADDR}:8080/healthIf you receive a non-HTTP error message, then your application isn't listening to the HTTP traffic. If you receive an HTTP status code that's different from what you specified in the Matcher setting, then your application is listening to the HTTP traffic, but not returning a status code for a healthy target.Check the status of your container instanceSuppose that you get the following event message from your AWS ECS service event:(service AWS-Service) (instance i-1234567890abcdefg) (port 443) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789) due to (reason Health checks failed)Check the status of your container instance by viewing the status check on the Amazon EC2 console. If your instance fails the system status checks, then try stopping and starting your instance.Related informationConfigure a target group for routingService load balancingFollow"


Question: How do I implement pagination in Amazon DynamoDB?

"When I try to fetch items from an Amazon DynamoDB table using a Query or Scan operation, the response doesn't return the complete results."

Answer: "When I try to fetch items from an Amazon DynamoDB table using a Query or Scan operation, the response doesn't return the complete results.ResolutionThis issue occurs under the following conditions:Your initial Scan or Query operation contains LastEvaluatedKey in the response.You didn't perform a subsequent operation using this key as ExclusiveStartKey for the next request.You can resolve this issue by implementing pagination for your DynamoDB table. Pagination is the process of sending subsequent requests to continue when a previous request is incomplete. A Query or Scan operation in DynamoDB might return results that are incomplete and require subsequent requests to attain the entire result set. This is because DynamoDB paginates the results from a Query or Scan operation and returns a maximum of 1MB of data in a single operation. This is a hard limit in DynamoDB. With pagination, the results from the Scan and Query operations are divided into pages of data that are 1 MB or less in size.To paginate your results and retrieve them one page at a time, check the low-level result from a Scan or Query operation to see if the result contains the LastEvaluatedKey element. You can get the remaining results of the initial request using another Scan or Query operation with the same parameters, but using the LastEvaluatedKey value as the ExclusiveStartKey parameter. If the result doesn't include the LastEvaluatedKey value, then there are no items to be retrieved.For more information, see Paginating table query results and Paginating the results.To implement pagination in Boto3 using LastEvaluatedKey, use the following code sample:from __future__ import print_function  # Python 2/3 compatibilityimport boto3from botocore.exceptions import ClientError# Create Clientsession = boto3.session.Session()dynamoDbClient = session.client('dynamodb')table_name = 'AmazonBins'# Track number of Items readitem_count = 0try:    # Get the first 1MB of data    response = dynamoDbClient.scan(        TableName=table_name    )except ClientError as error:    print("Something went wrong: ")    print(error.response['ResponseMetadata'])# Track number of Items readitem_count += len(response['Items'])# Paginate returning up to 1MB of data for each iterationwhile 'LastEvaluatedKey' in response:    try:        response = dynamoDbClient.scan(            TableName=table_name,            ExclusiveStartKey=response['LastEvaluatedKey']        )        # Track number of Items read        item_count += len(response['Items'])    except ClientError as error:        print("Something went wrong: ")        print(error.response['ResponseMetadata'])print("Total number of items found: {}".format(item_count))Follow"


Question: How do I add routes to the main route table in my VPC with CloudFormation?

I want to add routes to the main route table when I create my Amazon Virtual Private Cloud (Amazon VPC) in AWS CloudFormation.

Answer: "I want to add routes to the main route table when I create my Amazon Virtual Private Cloud (Amazon VPC) in AWS CloudFormation.Short descriptionWhen you use CloudFormation to create an Amazon VPC, CloudFormation doesn't recognize the main route table that's created by default. Therefore, information about the route table can't be passed between your Amazon VPC and CloudFormation. As a result, you can't add or remove routes from the main route table because you can't reference the route table from your CloudFormation template.ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you're using the most recent AWS CLI version.To resolve this issue, you can use an AWS Lambda-backed custom resource in the CloudFormation template. The CloudFormation stack creates an Amazon VPC. Then, the custom resource uses an AWS Lambda function to retrieve the main route table ID that's associated with your Amazon VPC.Create a CloudFormation stack using the following RouteTable-template.yml template:AWSTemplateFormatVersion: 2010-09-09Description: Template to add routes to default/main routetable of VPCResources:  MyVPC:    Type: 'AWS::EC2::VPC'    Properties:      CidrBlock: 10.0.0.0/16      Tags:        - Key: Env          Value: Test  LambdaIAMRole:    Type: 'AWS::IAM::Role'    DependsOn: MyVPC    Properties:      AssumeRolePolicyDocument:        Version: 2012-10-17        Statement:          - Effect: Allow            Principal:              Service:                - lambda.amazonaws.com            Action:              - 'sts:AssumeRole'      Path: /      Policies:        - PolicyName: root          PolicyDocument:            Version: 2012-10-17            Statement:              - Effect: Allow                Action:                  - 'ec2:Describe*'                Resource: '*'              - Effect: Allow                Action:                  - 'logs:CreateLogGroup'                  - 'logs:CreateLogStream'                  - 'logs:PutLogEvents'                Resource: 'arn:aws:logs:*:*:*'  LambdaFunction:    Type: 'AWS::Lambda::Function'    Properties:      Handler: index.lambda_handler      Role: !GetAtt LambdaIAMRole.Arn      Runtime: python3.9      Timeout: 50      Code:        ZipFile: |            from __future__ import print_function            import json            import boto3            import urllib3            import cfnresponse            SUCCESS = "SUCCESS"            FAILED = "FAILED"            http = urllib3.PoolManager()            print('Loading function')            ec2 = boto3.client('ec2')            def lambda_handler(event, context):                print("Received event: " + json.dumps(event, indent=2))                responseData={}                try:                    if event['RequestType'] == 'Delete':                        print("Request Type:",event['RequestType'])                        print("Delete Request - No Physical resources to delete")                    elif event['RequestType'] == 'Create':                        print("Request Type:",event['RequestType'])                        VPCID=event['ResourceProperties']['VPCID']                        RouteTableID=get_vpc(VPCID)                        responseData={'RouteTableID':RouteTableID}                        print("Sending response to custom resource")                    elif event['RequestType'] == 'Update':                        print("Request Type:",event['RequestType'])                        VPCID=event['ResourceProperties']['VPCID']                        RouteTableID=get_vpc(VPCID)                        responseData={'RouteTableID':RouteTableID}                        print("Sending response to custom resource")                    responseStatus = 'SUCCESS'                    print("responseStatus: " + responseStatus)                    cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID")                except Exception as e:                    print('Failed to process:', e)                    responseStatus = 'FAILURE'                    responseData = {'Failure': 'Something bad happened.'}                    cfnresponse.send(event, context, cfnresponse.FAILURE, responseData, "CustomResourcePhysicalID")            def get_vpc(VPCID):                response = ec2.describe_route_tables (                  Filters=[                    {                      'Name': 'association.main',                      'Values': [ 'true' ]                    },                    {                      'Name': 'vpc-id',                      'Values': [ VPCID ]                    }                  ]                )                print("Printing the VPC Route Table ID ....")                RouteTableID=response['RouteTables'][0]['RouteTableId']                print(RouteTableID)                return RouteTableID            def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False):                responseUrl = event['ResponseURL']                print(responseUrl)                responseBody = {'Status': responseStatus,                                'Reason': 'See the details in CloudWatch Log Stream: ' + context.log_stream_name,                                'PhysicalResourceId': physicalResourceId or context.log_stream_name,                                'StackId': event['StackId'],                                'RequestId': event['RequestId'],                                'LogicalResourceId': event['LogicalResourceId'],                                'Data': responseData}                json_responseBody = json.dumps(responseBody)                print("Response body:\n" + json_responseBody)                headers = {                    'content-type' : '',                    'content-length' : str(len(json_responseBody))                }                try:                    response = http.request('PUT', responseUrl, headers=headers, body=json_responseBody)                    print("Status code: " + response.reason)                except Exception as e:                    print("send(..) failed executing requests.put(..): " + str(e))  Lambdatrigger:    Type: 'Custom::RouteTableLambda'    Properties:      ServiceToken: !GetAtt LambdaFunction.Arn      VPCID: !Ref MyVPC  MyInternetGateway:    Type: 'AWS::EC2::InternetGateway'    Properties:      Tags:        - Key: Env          Value: Test  AttachGateway:    Type: 'AWS::EC2::VPCGatewayAttachment'    Properties:      VpcId: !Ref MyVPC      InternetGatewayId: !Ref MyInternetGateway  MyRoute:    Type: 'AWS::EC2::Route'    Properties:      RouteTableId: !GetAtt Lambdatrigger.RouteTableID      DestinationCidrBlock: 0.0.0.0/0      GatewayId: !Ref MyInternetGatewayOutputs:  RouteTableID:    Value: !GetAtt Lambdatrigger.RouteTableIDThe data that the custom resource returns contains the main route table ID. You can reference the ID using GetAtt in AWS::EC2::Route to add routes to the main route table. Also, the outputs of the stack display the route table ID.The following is an example of the SUCCESS response body that the custom resource sends to the CloudFormation stack:{  "Status": "SUCCESS",  "Reason": "See the details in CloudWatch Log Stream: 2022/08/31/[$LATEST]c48b90efb3944c11ad3fb6e1ce5e1f45",  "PhysicalResourceId": "CustomResourcePhysicalID",  "StackId": "arn:aws:cloudformation:us-west-1:XXXX:stack/VPC-RT/06c957b0-297e-11ed-afb5-02ca6fd67f8d",  "RequestId": "55c0f2b8-3044-47f7-aba4-84502b4ef632",  "LogicalResourceId": "Lambdatrigger",  "NoEcho": false,  "Data": {    "RouteTableID": "rtb-0fba8d15701234567a"  }}Follow"


Question: Why can't I use an IAM role for the service account in my Amazon EKS pod?

"I try to use an AWS Identity and Access Management (IAM) role for a service account. But my Amazon Elastic Kubernetes Service (Amazon EKS) pod fails to assume the assigned IAM role. I receive an authorization error. Or, my pod tries to use the default IAM role assigned to the Amazon EKS node instead of the IAM role assigned to my pod."

Answer: "I try to use an AWS Identity and Access Management (IAM) role for a service account. But my Amazon Elastic Kubernetes Service (Amazon EKS) pod fails to assume the assigned IAM role. I receive an authorization error. Or, my pod tries to use the default IAM role assigned to the Amazon EKS node instead of the IAM role assigned to my pod.Short descriptionTo resolve issues using the IAM roles for service accounts feature, try the following:Verify that you have an IAM OpenID Connect (OIDC) identity provider for your Amazon EKS clusterValidate your IAM role policies and trust policy configurationConfirm that your service account exists and has a properly formatted annotation for the IAM role's ARNUse a test pod to verify that the service account is workingNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.ResolutionVerify that you have an IAM OIDC identity provider for your Amazon EKS clusterCreate an IAM OIDC provider for your cluster, if you don't already have one. You must have an OIDC identity provider for your cluster to use an IAM role for your service account.Next, verify that the OIDC identity provider is configured correctly:1.    Open the IAM console, and then choose Identity providers from the navigation pane.2.    In the Provider column, identify and note the OIDC provider URL.3.    In a separate tab or window, open the Amazon EKS console, and then choose Clusters from the navigation pane.4.    Choose your cluster, and then choose the Configuration tab.5.    In the Details section, note the value of the OpenID Connect provider URL property.6.    Verify that the OIDC provider URL from the Amazon EKS console (step 5) matches the OIDC provider URL from the IAM console (step 2).If the OIDC provider URL for your Amazon EKS cluster doesn't match any of the OIDC provider URLs in the IAM console, then you must create a new IAM OIDC provider.Validate your IAM role policies and trust policy configurationYour IAM role might not have the full range of permissions that you expect are assigned to that role. Your IAM role's trust relationship policy could also have syntax errors, if you created your IAM role using the AWS Management Console or AWS CLI.To validate your IAM role policies and check for syntax errors in your trust policy, do the following:1.    Open the IAM console.2.    In the navigation pane, choose Roles, and then choose your role.3.    Choose the Permissions tab on your role's page, and then verify that all your required permissions are assigned to the role.4.    Choose the Trust Relationships tab, and then choose Edit trust relationship.5.    In the policy document for your trust relationship, verify that the format of your policy matches the format of the following JSON policy:"Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Federated": "arn:aws:iam::your-account-id:oidc-provider/oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER"      },      "Action": "sts:AssumeRoleWithWebIdentity",      "Condition": {        "StringEquals": {          "oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER:sub": "system:serviceaccount:your-namespace:your-service-account"        }      }    }  ]}In the preceding JSON policy, review the format of the Federated property line and the StringEquals property line. In the Federated line, confirm that your AWS Region code (your-region-code), account ID (your-account-id), and unique OIDC identifier (EXAMPLE_OIDC_IDENTIFIER) are formatted correctly. In the StringEquals line, confirm that your Region code (your-region-code), OIDC unique identifier (EXAMPLE_OIDC_IDENTIFIER), Kubernetes namespace (your-namespace), and Kubernetes service account name (your-namespace) are formatted correctly.6.    If you edit your policy document to correct formatting errors, then choose Update Trust Policy.Confirm that your service account exists and has a properly formatted annotation for the IAM role's ARN1.    Confirm that your Kubernetes service account exists:$ kubectl get serviceaccount YOUR_ACCOUNT_NAME -n YOUR_NAMESPACE -o yamlNote: Replace YOUR_ACCOUNT_NAME with your account name. Replace YOUR_NAMESPACE with your namespace.If the preceding command doesn't return a service account name, then create a service account (from the Kubernetes website).2.    Confirm that your service account has the name that you expect and that its role-arn annotation is correctly formatted. For example:apiVersion: v1kind: ServiceAccountmetadata:  annotations:    eks.amazonaws.com/role-arn: arn:aws:iam::012345678912:role/my-example-iam-role  name: my-example-serviceaccount  namespace: my-test-namespaceUse a test pod to verify that the service account is workingYou can verify if the service account is working correctly by running a test pod. Then, check if the pod can mount environment variables correctly and can assume the specified IAM role.Note: You can experience an issue related to your application's credentials even if the IAM roles for service accounts feature work correctly. You can also experience this issue if your pod has the correct environment variables. To prevent this issue, you can use an AWS CLI container image for verification.1.    Create a local YAML file called awscli-pod.yaml. For example:apiVersion: v1kind: Podmetadata:  name: awscli  labels:    app: awsclispec:  serviceAccountName: YOUR_SERVICE_ACCOUNT  containers:  - image: amazon/aws-cli    command:      - "sleep"      - "604800"    imagePullPolicy: IfNotPresent    name: awscli  restartPolicy: AlwaysNote: Replace YOUR_SERVICE_ACCOUNT with your Kubernetes service account name.2.    Create the test pod (from the YAML file) in your namespace:$ kubectl apply -f ./awscli-pod.yaml -n YOUR_NAMESPACENote: Replace YOUR_NAMESPACE with your namespace.3.    Confirm that the awscli pod has the correct environment variables:$ kubectl exec -n YOUR_NAMESPACE awscli env | grep AWSOutput:AWS_ROLE_ARN=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAMEAWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token4.    Confirm that the test pod is using the correct IAM role:$ kubectl exec -it awscli -n YOUR_NAMESPACE -- aws sts get-caller-identityOutput:{    "UserId": "REDACTEDY471234567890:botocore-session-1632772568",    "Account": "012345678912",    "Arn": "arn:aws:sts::012345678912:assumed-role/your-iam-role/botocore-session-1632772568"}Note the Arn value, including the IAM role name (your-iam-role), that you receive in the output from the command in step 4.5.    Delete the awscli pod after you verify the IAM role:$ kubectl delete -f ./awscli-pod.yaml -n YOUR_NAMESPACEIf the awscli pod shows the correct IAM role, then the IAM roles for service accounts feature are working correctly.The preceding steps confirm that the IAM token is correctly mounted to the pod. If your application still can't use the token file correctly, then there is likely an issue at the application or SDK level. This issue is likely related to how the application ingests AWS credentials. For more information, see Using the Default Credential Provider Chain and Credentials (from the Boto3 website).Follow"


Question: How can I troubleshoot Application Load Balancer session stickiness issues?

"I have an Application Load Balancer that uses duration-based stickiness or application-based stickiness sessions. The load balancer is configured to route all user session requests to the same target. However, user session requests are getting routed to different targets."

Answer: "I have an Application Load Balancer that uses duration-based stickiness or application-based stickiness sessions. The load balancer is configured to route all user session requests to the same target. However, user session requests are getting routed to different targets.Short descriptionSticky sessions use cookies to help the client maintain a connection to the same instance over a cookie's lifetime. Using sticky sessions configures a load balancer to bind user sessions to a specific instance. This means that all requests from a user during a session are sent to the same instance. However, this assumption about the connection can cause imbalances over time.A sticky session can fail if:The registered target didn't generate a cookieThe client didn't return the cookie in the request headerThe cookies aren't formatted correctlyThe duration-based session has passedThe session request is passing through multiple load balancersThe target health status changed to unhealthyAn AWS service disabled stickinessNote: Before you begin, make sure that you review the requirements and considerations section in Sticky sessions for your Application Load Balancer.ResolutionApplication-based session stickiness1.    Check for HTTP errors with your load balancer. For instructions, see Troubleshoot your Application Load Balancers.2.    To check for cookies placed on the backend instance or load balancer, run a curl command similar to the following:Note: Replace the DNS name with your load balancer name.[ec2-user@ip-172-31-22-85 ~]$ curl -vko /dev/null internal-EXAMPLE-ELB-1430759361.eu-central-1.elb.amazonaws.comNote: The Linux curl utility can also be installed on Windows OS.Example output:...< Set-Cookie: PHPSESSID=k0qu6t4e35i4lgmsk78mj9k4a4; path=/< Set-Cookie:AWSALBAPP-0=438DC7A50C516D797550CF7DE2A7DBA19D6816D5E6FB20329CD6AEF2B40030B12FF2839757A60E2330136A2182D27D049FB9D887FBFE9E80FB0724130FB3A86A4B0BAC296FDEB9E943EC9272FF52F5A8AEF373DF33;PATH=/...3.    To verify that the registered target generated an application cookie, send an HTTP request to the instance IP address similar to the following:[ec2-user@ip-172-31-22-85 ~]$ curl -vko /dev/null 172.31.30.168Example output:...< Set-Cookie: PHPSESSID=5pq74110nuir60kpapj04mglg4; path=/...4.    Verify that the cookie name generated by the registered target is the same name as the cookie on the load balancer from steps 2 and 3.5.    If the target generated an application cookie and the load balancer generated an AWSALBAPP cookie, make sure that the client sends both cookies in subsequent requests. If the client fails to include either the application or the AWSELB cookie, then stickiness fails. To verify that the client sends both the application and AWSALBAPP cookies, take a packet capture on the client. Use the tcpdump or Wireshark utility for analysis, or use the browsers web-debugging tool to retrieve the cookie information in the request header.Note: If you are working with AWS Support, you can gather this information by generating a HAR file. HAR files can capture sensitive information, such as user names, passwords, and keys, so be sure to remove any sensitive information from a HAR file.6.    Check which backend instance that the load balancer routed the request to. You can display the instance ID using Instance metadata by running a script similar to the following:<?php $instance_id =file_get_contents('http://169.254.169.254/latest/meta-data/instance-id');echo "instance id = " . $instance_id . "\\xA";?>7.    Review your Elastic Load Balancing access logs to check whether requests from the same user are routed to different registered targets. For instructions, see Access logs for your Application Load Balancer.8.    Verify that the health status of all targets under the target group where stickiness is enabled are healthy. If the target health status changes to “unhealthy,” then stickiness breaks and the load balancer stops routing requests to that target. A new healthy target is then automatically selected by the load balancer and establishes a sticky session. The load balancer continues to route requests to the new target even if the health status is "unhealthy." For more information on health checks, see Health checks for your target groups.9.    Check for any AWS services, for example, Amazon Elastic Kubernetes Service (Amazon EKS), that might have disabled the stickiness in your load balancer. Check the View CloudTrail events with the API name “ModifyTargetGroupAttributes” and attribute “stickiness.enabled”.Duration-based session stickiness1.    Run a command similar to the following with the load balancer DNS name to check for an AWSALB cookie in the response:[ec2-user@ip-172-31-22-85 ~]$ curl -vko /dev/null internal-TESTELB-1430759361.eu-central-1.elb.amazonaws.comExample output:...< Set-Cookie: AWSALB=438DC7A50C516D797550CF7DE2A7DBA19D6816D5E6FB20329CD6AEF2B40030B12FF2839757A60E2330136A2182D27D049FB9D887FBFE9E80FB0724130FB3A86A4B0BAC296FDEB9E943EC9272FF52F5A8AEF373DF33;PATH=/...Note: If there isn't an AWSALB cookie in the response, then there is no stickiness between the client and the backend instance.2.    If the load balancer generated an AWSALB cookie, then be sure that the client sends this cookie in subsequent requests. If the client fails to include the AWSALB cookie, then stickiness doesn't work. To verify that the client sends the AWSALB cookie, take a packet capture on the client. Use the tcpdump or Wireshark utility for analysis, or use the browsers web-debugging tool to retrieve the cookie information in the request header.Note: If you are working with AWS Support, you can gather this information by generating a HAR file. HAR files can capture sensitive information, such as user names, passwords, and keys, so be sure to remove any sensitive information from a HAR file.3.    Check the duration configured on the load balancer. If the cookie expiration passed, then client sessions no longer stick to the registered target until a new cookie is issued by the load balancer.4.    If your request is passing through multiple load balancers, verify that stickiness is enabled on only one load balancer. If more than one load balancer generates a cookie, then the load balancer replaces the original cookie and stickiness fails.For Classic Load Balancers, see How can I troubleshoot the Classic Load Balancer session stickiness feature?Related informationWhy is Elastic Load Balancing unequally routing my load balancer traffic?Configure sticky sessions for your Classic Load BalancerHow do I set up weighted target groups for my Application Load Balancer?Follow"


Question: Why is there a discrepancy between my CloudWatch metrics and AWS CLI storage metrics for Amazon S3?

I'm seeing a discrepancy between the Amazon CloudWatch metrics and AWS Command Line Interface (AWS CLI) storage metrics for Amazon Simple Storage Service (Amazon S3). Why is there such a large difference in reported storage size between the two sources?

Answer: "I'm seeing a discrepancy between the Amazon CloudWatch metrics and AWS Command Line Interface (AWS CLI) storage metrics for Amazon Simple Storage Service (Amazon S3). Why is there such a large difference in reported storage size between the two sources?Short descriptionIf there's a discrepancy between your CloudWatch storage metrics for Amazon S3 and the metrics calculated using the AWS CLI, check for the following:Object versioning.Note: The object versioning feature in Amazon S3 retains multiple versions of an object in your bucket. By default, Amazon S3 object versioning is disabled on buckets, and you must explicitly enable the feature. Additionally, the AWS CLI storage metric calculations only count the newest version and size of each object that is stored in the bucket.Incomplete multipart uploads.Note: Incomplete multipart uploads aren't included in the AWS CLI storage calculations, but are calculated as storage in your CloudWatch metrics.To identify the cause of the reporting discrepancy, check whether you've enabled object versioning and look for any multipart uploads in your bucket. These two factors can result in an increased value of the calculated bucket size in CloudWatch. For more information, see Amazon S3 CloudWatch daily storage metrics for buckets.Tip: If you do have incomplete multipart uploads in Amazon S3, consider creating a lifecycle configuration rule. This lifecycle configuration rule automatically cleans up any incomplete parts, lowering the cost of data storage. Note that lifecycle rules operate asynchronously, so there might be a delay with the operation. However, as soon as the objects are marked for deletion, you aren't billed for storage even if the object isn't removed yet.Additionally, the Amazon S3 monitoring metrics are recorded daily, and therefore might not display the most updated information. Meanwhile, CloudWatch monitors your AWS resources and applications in real time.ResolutionDaily storage metrics in CloudWatchIn CloudWatch, the BucketSizeBytes metric captures all Amazon S3 and Amazon S3 Glacier storage types, object versions, and any incomplete multipart uploads. This value is calculated by summing up all object sizes, metadata in your bucket (both current and noncurrent objects), and any incomplete multipart upload sizes. For example, the BucketSizeBytes metric will calculate the amount of data (in bytes) that are stored in an S3 bucket in all these object storage classes:S3 StandardS3 Intelligent-TieringS3 Standard-IAS3 One Zone-IAS3 Reduced Redundancy StorageS3 Glacier Deep ArchiveS3 GlacierAdditionally, the NumberOfObjects metric in CloudWatch contains the total number of objects that are stored in a bucket for all storage classes. This value counts all objects in the bucket (both current and noncurrent), along with the total number of parts for any incomplete multipart uploads. The NumberOfObjects metric also calculates the total number of objects for all versions of objects in your bucket. For example, if you have two versions of the same object, then the two versions will be counted as two separate objects. For more information, see Metrics and dimensions.Daily storage calculations using the AWS CLITo calculate the storage metrics for Amazon S3 using the AWS CLI, use the following command syntax:aws s3 ls --summarize --human-readable --recursive s3://bucketname | grep -i totalThis command syntax calculates the total number and size of objects in your Amazon S3 bucket. However, note that only the current version of each object that is stored in the bucket (and its size) are calculated. Multipart uploads, delete markers, and noncurrent versions of each object aren't calculated into the total bucket size or the total number of objects.Incomplete multipart uploadsTo review the list of incomplete multipart uploads, run the list-multipart-uploads command:aws s3api list-multipart-uploads --bucket <bucket-example>Then, list all the objects in the multipart upload, using the list-parts command and your UploadId value:aws s3api list-parts --bucket <bucket-example> --key large_test_file --upload-id <examplevalue>Creating a lifecycle ruleTo automatically delete multipart uploads, create a lifecycle configuration rule. Follow these steps:1.    Open the Amazon S3 console.2.    Choose the Management tab.3.    Choose Create new policy.4.    Add the name of the policy.5.    Choose Select - Delete expired delete markers or incomplete multipart uploads.6.    (Optional) If your bucket isn't versioned, choose Delete incomplete multipart uploads.Object versioningTo review and audit your Amazon S3 bucket for different versions of objects, use the Amazon S3 inventory list. An Amazon S3 inventory list file contains a list of the objects in the source bucket and metadata for each object. The inventory list file will capture metadata such as bucket name, object size, storage class, and version ID.Related informationExample 8: Lifecycle configuration to abort multipart uploadsExpiring objectsFollow"


Question: Can I schedule my Savings Plan to renew automatically?

I have a Savings Plan that expires soon. Can I schedule my Savings Plan to renew automatically?

Answer: "I have a Savings Plan that expires soon. Can I schedule my Savings Plan to renew automatically?ResolutionYou can queue your Savings Plan purchases for a specific date and time in the future. To renew your Savings Plans automatically, you can queue a Saving Plans purchase for the date and time that your existing Saving Plan expires. On the scheduled date and time, AWS automatically purchases the Savings Plan for you using your account's default payment method. For instructions, see Queuing a Savings Plan purchase on the Cart page.To view owned Savings Plans and when they expire, check the AWS Cost Management console, then choose Inventory, located under Savings Plans in the navigation pane.You can also set Savings Plans alerts to remind you when your Savings Plans are about to expire. However, Savings Plan alerts don't automatically renew or purchase Savings Plans for you. You can queue your Savings Plans purchases to ensure uninterrupted coverage.Related informationCan I schedule my EC2 Reserved Instance to renew automatically?Why aren't my Savings Plans applying to my AWS billing in the way that I expected?What should I consider before purchasing a Savings Plan?How do I save money on my AWS usage using a Savings Plan?Follow"


Question: How can I use snapshots and clones with AWS Storage Gateway to recover my volume data?

I want to use snapshots and clones to recover my data to an AWS Storage Gateway volume.

Answer: "I want to use snapshots and clones to recover my data to an AWS Storage Gateway volume.ResolutionCached volumes are iSCSI-attached block storage volumes that store volume data. Cached volumes provide a managed cache of local disk data attached to the Storage Gateway appliance by your hypervisor. For more information, see Cached volumes architecture.You can protect Storage Gateway cached volumes using Amazon Elastic Block Store (Amazon EBS) snapshots and clones.Note: Amazon EBS stored volumes can be up to 16 TiB, and Storage Gateway cached volumes can be up to 32 TiB. If your Storage Gateway volume is larger than 16 TiB, you can't create an EBS volume from a snapshot. For more information, see Quotas for volumes.Take an EBS snapshot of your gateway cached volume, and then use the snapshot to create a new volumeTake an EBS snapshot of your gateway cached volume1.    Open the Storage Gateway console. In the navigation pane, choose Volumes. Then, select the volume that you want to take a snapshot of.2.    Choose Actions, and then choose Create EBS snapshot.3.    In Snapshot description, enter a name for the snapshot. Then, choose Create EBS snapshot. Wait for the snapshot's Status to change from Pending to Available.Note: The time it takes for the snapshot to become Available depends on the size of the volume.4.    (Optional) To take snapshots on a reoccurring schedule, choose Actions, Edit snapshot schedule, and then choose the Start time and Recurrence. Enter a Description, and then choose Save.5.    In EBS snapshots, select the name of your snapshot. Choose the Description tab, and then copy the Snapshot ID.6.    From the navigation bar of the console, in the Region selector, select the AWS Region that your snapshot is in.Note: To restore the snapshot to a volume in a different Region, first copy the snapshot to the new Region. Then, restore the snapshot to a volume in that Region. For more information, see Copy a snapshot.7.    In the navigation pane, choose Volumes, and then choose Create volume.Choose the Gateway that you want to use for the volume, and then enter the Capacity. In Volume contents, select Based on EBS snapshot, and then enter the EBS snapshot ID that you copied from step 5. For iSCSI target, enter a name, and then choose Create Volume.9.    In Configure CHAP authentication, choose Skip.Use the snapshot to create a new volume1.    Open the Amazon Elastic Compute Cloud (Amazon EC2) console. In the navigation pane, choose Volumes, and then choose Create Volume.2.    In the Create Volume wizard page, for Volume Type, choose General Purpose SSD, Provisioned IOPS SSD, Cold HDD, Throughput Optimized HDD, or Magnetic.3.    Choose your Snapshot ID, and then for Size (GiB), enter the size. Note: If you specify both a volume size and a snapshot ID, the size must be equal to or greater than the snapshot size.4.    In the IOPS field, enter the maximum number of input/output operations per second (IOPS) that the volume can support.5.    Select your Availability Zone, and then choose Create Volume. Note: EBS volumes can be attached only to EC2 instances in the same Availability Zone.(Optional) Create a clone of your volume from the most recent recovery pointReview the following considerations for recovering a new volume using a clone:Clones don't require you to create and store EBS snapshots beforehand.For recovery purposes, it might be faster to create and access the clone. With a clone, Storage Gateway creates the new volume instantly, and then copies the data from the initial volume in the background.Note: It's a best practice to use EBS snapshots for maintaining specific recovery points in time, or to recover directly to an EBS volume.1.    Open the Storage Gateway console, and then choose Create volume.2.    In Gateway, choose the Volume gateway, and then enter the Capacity.3.    In Volume contents, choose Clone from last volume recovery.4.    In Source volume, select the volume that you want to clone.5.    In iSCSI target,enter a name, and then choose Create Volume.6.    In Configure CHAP authentication, choose Skip.7.    In the navigation pane, choose Volumes, and then verify that the new volume is available and ready to use.Related informationCreate Amazon EBS snapshotsManaging your volume gatewayFollow"


Question: Why is Amazon Inspector not scanning my Amazon EC2 instances?

"I turned on Amazon Inspector but it's not scanning my Elastic Compute Cloud (Amazon EC2) instance. The Amazon Inspector dashboard status is "Unmanaged EC2 instance", "Unsupported OS", "Internal Error", "Pending initial scan" or "No Inventory"."

Answer: "I turned on Amazon Inspector but it's not scanning my Elastic Compute Cloud (Amazon EC2) instance. The Amazon Inspector dashboard status is "Unmanaged EC2 instance", "Unsupported OS", "Internal Error", "Pending initial scan" or "No Inventory".Short descriptionAmazon Inspector uses AWS Systems Manager and the AWS Systems Manager Agent (SSM Agent) to scan software applications installed on your Amazon EC2 instances. The telemetry data collected by the SSM agent is then scanned by Amazon Inspector for software vulnerabilities. You can use the Amazon Inspector dashboard to monitor the status for your Amazon EC2 instances. For more information, see Scanning Amazon EC2 instances with Amazon Inspector.If Amazon Inspector isn't scanning your Amazon EC2 instances, make sure that the:SSM Agent is up to date.Amazon EC2 instance is running.Operating system is supported.Connectivity to Systems Manager is configured.Systems Manager associations and software application are configured.ResolutionCheck the SSM Agent versionAmazon Inspector must have the SSM Agent running to scan Amazon EC2 instances. If you are using an earlier version of the SSM Agent, then you might need to update it to successfully scan Amazon EC2 instances. It's a best practice to automate the process of updating the SSM Agent. For instructions, see Automatically updating SSM Agent.To update the SSM Agent manually, subscribe to SSM Agent notifications. Then, update the SSM Agent using Run Command. You can also subscribe to SSM Agent release notes on the GitHub website.Check that the Amazon EC2 instance is runningThe "EC2 instance stopped" status means that Amazon Inspector paused scanning for the instance because the instance is in a stopped state. Any existing findings persist until the instance is terminated. If the instance is restarted, Amazon Inspector automatically resumes scanning for the instance. To restart an Amazon EC2 instance, see Stop and start your instances.Check that the operating system is supportedThe "Unsupported OS" status means that the Amazon EC2 instance uses an operating system or architecture that Amazon Inspector doesn't support. For a table that lists the supported operating systems for scanning EC2 instances, see Supported operating systems: Amazon EC2 scanning.To check your operating system version, follow these steps for Linux or Windows:Linux OSRun the following command:cat /etc/os-releaselsb_release -ahostnamectlWindows OSChoose the Windows logo key + R, enter msinfo32 in the Open box, and then choose OK.Check connectivity to Systems ManagerNote: If your Amazon EC2 instance doesn't appear in the Systems Manager console, then additional configuration might be required. For more information, see Why is my EC2 instance not displaying as a managed node or showing a "Connection lost" status in Systems Manager?Open the Systems Manager console in the same Region as Amazon Inspector and your Amazon EC2 instance.In the navigation pane, choose Fleet Manager.In Managed nodes, check the SSM Agent ping status. If the status is Online, then your Amazon EC2 instance is connected to the SSM Agent.If the SSM Agent ping status is Connection Lost, then make sure that your Amazon EC2 instance meets the Systems Manager prerequisites. If you're using SSM Agent version 3.1.501.0 or higher, you can use the ssm-cli command line tool for further diagnosing and troubleshooting. For instructions, see Troubleshooting Amazon EC2 managed instance availability using ssm-cli.You can also run the AWSSupport-TroubleshootManagedInstance Systems Manager Automation document to confirm whether the instance meets the prerequisites to be listed as a managed instance. For more information, see AWSSupport-TroubleshootManagedInstance.Check the Systems Manager associations and software applicationAmazon Inspector requires a Systems Manager State Manager association in your account to collect the software application inventory. Amazon Inspector automatically creates an association called InspectorInventoryCollection-do-not-delete. The "No inventory" status means that Amazon Inspector couldn't find the software application inventory to scan for your Amazon EC2 instance.Check the associations statusOpen the Systems Manager console in the same Region as Amazon Inspector and your Amazon EC2 instance.In the navigation pane, choose State Manager.In Associations, make sure that the InspectorInventoryCollection-do-not-delete association exists and the Status is Success.If the InspectorInventoryCollection-do-not-delete association doesn't exist, then run the AWS-GatherSoftwareInventory document on all Amazon EC2 instances. Choose the Association id for the Amazon EC2 instance that didn't scan, and then choose the Execution history tab for more details.If the InspectorInventoryCollection-do-not-delete association Status is Failed, choose the Association id, and then choose Apply association now.Check the InspectorInventoryCollection-do-not-delete association Status again to confirm that it changed from Failed to Success.Note: For Windows, the Amazon Inspector SSM plugin is required to scan Windows EC2 instances. When EC2 scanning is activated, Amazon Inspector creates the new SSM associations InspectorDistributor-do-not-delete, InspectorInventoryCollection-do-not-delete, and InvokeInspectorSsmPlugin-do-not-delete for your Windows resources. If any of these associations Status is Failed, try reapplying the association. If the InspectorSsmPlugin.exe file is deleted, the InspectorDistributor-do-not-delete SSM association will reinstall the plugin at the next Windows scan. For more information, see Scanning Windows EC2 instances with Amazon Inspector.Verify that the software application exists in the nodeMake sure that there are software packages in the inventory for your Amazon EC2 instance.Open the Systems Manager console in the same Region as Amazon Inspector and your Amazon EC2 instance.In the navigation pane, choose Fleet Manager.In Managed nodes, choose your Node ID, and then choose the Inventory tab to check for software applications.Check the software application inventory rateIt's a best practice to set the inventory collection rate to run every 30 minutes. Edit the InspectorInventoryCollection-do-not-delete association and set the cron expression rate for 30 minutes.Related informationAssessing Amazon Inspector coverage of your AWS environmentHow do I set up Amazon Inspector Classic to run security assessments on my Amazon EC2 instances?Follow"


Question: How can I push custom metrics to CloudWatch?

How can I push custom metrics to Amazon CloudWatch?

Answer: "How can I push custom metrics to Amazon CloudWatch?Short descriptionAWS services push data points to CloudWatch by default. However, there might be instances where you must calibrate the performance of your resources based on metrics that aren't supported by AWS services. In these cases, you can push custom metrics to CloudWatch using the unified CloudWatch agent or the API.Important: Custom metrics are charged according to their storage and API use.ResolutionPush custom metrics using the CloudWatch agentThe unified CloudWatch agent collects system-level metrics and sends them to CloudWatch as custom metrics. You can use the agent to push custom metrics from:Linux or Windows serversAmazon Elastic Compute Cloud (Amazon EC2) instances or on-premises serversFor Linux, refer to the list of supported metrics.Sample agent configuration file metric block for disk metrics (Linux):"disk":     {        "measurement": [                    "used_percent"        ],        "resources": [                    "*"        ],        "drop_device":         true      }For Windows, you can reference any counter mentioned in the Windows Performance Monitor in the agent configuration file.Sample agent configuration file metric block for Processor Counter (Windows):"Processor": {        "measurement": [          {"name": "% Idle Time", "rename": "CPU_IDLE", "unit": "Percent"},          "% Interrupt Time",          "% User Time",          "% Processor Time"        ],        "resources": [          "*"        ],        "append_dimensions": {          "d1": "win_foo",          "d2": "win_bar"        }      }You can also use "StatsD" and "collectd" protocols to retrieve custom metrics from your applications or services. Then, the metrics are pushed through the agent. StatsD is supported on both Linux and Windows servers. collectd is supported only on Linux servers. To use these protocols:1.    Install the CloudWatch unified agent.2.    Assign a role or credentials to the instance with CloudWatch permissions.3.    Create the CloudWatch agent configuration file.4.    Start the agent.Push custom metrics using PutMetricDataIf your use case doesn't support the use of the CloudWatch unified agent, you can use the PutMetricData API to push custom metrics to CloudWatch.For example, to push connections for a specific port as a custom metric, you can retrieve the values locally and transfer them in the API:total_conn=$(netstat -an | grep <port> | wc -l) aws cloudwatch put-metric-data   --namespace "totalconn"   --metric-name <port> --dimensions Instance=<InstanceId> --value $  total_connSimilarly, you can use AWS SDKs to use the PutMetricData API and send custom metrics to CloudWatch. Create a local script that runs periodically to send custom metrics. The API is optimal for pushing different metrics and multiple values for the following reasons:You can use up to 1,000 different metrics in a single API.Using the Values and Counts method, you can publish up to 150 values per metric with one PutMetricData request.You can use up to 30 dimensions per metric.Each PutMetricData request is limited to 1 MB for HTTP POST requests.For example, instead of making a separate API call for each metric, you can push multiple metrics using a single API call:aws cloudwatch put-metric-data --namespace "Usage Metrics" --metric-data file://metric.jsonwhere metric.json is:[  {    "MetricName": "DiskMetric",    "Value": <value_derived_by_some_calculation>,    "Unit": "Count"  },  {    "MetricName": "MemoryMetric",        "Value": <value_derived_by_some_calculation>,    "Unit": "Count"  }]This approach typically reduces costs because custom metrics are charged based on the number of APIs and storage. This approach might also help reduce throttling on the PutMetricData API.Follow"


Question: How do I troubleshoot Amazon Connect CCP softphone errors?

I'm receiving softphone errors from my Amazon Connect Contact Control Panel (CCP) that say it can't connect to an endpoint.

Answer: "I'm receiving softphone errors from my Amazon Connect Contact Control Panel (CCP) that say it can't connect to an endpoint.Short descriptionThere are three types of Amazon Connect CCP endpoint errors:Signaling endpoint errorsMedia endpoint errorsAPI endpoint errorsThese errors appear in the Amazon Connect CCP logs as softphone errors, similar to the following examples.Note: You can use the CCP Log Parser to turn your raw CCP logs into a more read-friendly format. For more information, see CCP Log Parser on GitHub.Example Amazon Connect CCP softphone errorsSignaling endpoint errorsMedia endpoint errorsAPI endpoint errors"softphone ERROR {your contact ID} SESSION Failed connecting to signaling server Timeout" "softphone ERROR Softphone error occurred : signalling_connection_failure URL {signaling endpoint URL} cannot be reached.""softphone WARN {your contact ID} SESSION ICE collection timed out" "softphone ERROR {your contact ID} SESSION No ICE candidate"  "softphone INFO {your contact ID} SESSION ConnectSignalingAndIceCollectionState => FailedState"  "softphone ERROR Softphone error occurred : ice_collection_timeout Ice collection timedout.""ccp ERROR 'acceptContact' API request failed: [object Object]"  "ccp TRACE AWSClient: < Operation 'acceptContact' failed: {"message":"Network Failure","code":"NetworkingError","time":"....""Softphone errors occur when your Amazon Connect CCP can't communicate with one or more of the following required endpoints.Required Amazon Connect CCP endpointsEndpoint typeDomain/URL allow listPortSignalingFor CCP v1: rtc*.connect-telecom.{region}.amazonaws.com For CCP v2: *.transport.connect.{region}.amazonaws.com For more information, see Upgrade to the latest CCP.443(TCP)MediaTurnNlb-*.elb.{region}.amazonaws.com3478(UDP)API*.telemetry.connect.{region}.amazonaws.com New: additional domains to add to your allow list {myInstanceName}.my.connect.aws/ccp-v2 {myInstanceName}.my.connect.aws/api *.static.connect.aws Previously required: {myInstanceName}.awsapps.com/connect/ccp-v2 {myInstanceName}.awsapps.com/connect/api *.cloudfront.net443(TCP)Important: Make sure that you replace {region} in each domain with the location of your Amazon Connect instance. For more information, see Set up your network.ResolutionConfirm that nothing is blocking traffic to the endpoint domain listed in the error message. For more information, see Replace Amazon EC2 and CloudFront IP range requirements with a domain allow list.Confirm that the required port for the endpoint domain listed in the error message is open on your network firewall. For more information, see Replace Amazon EC2 and CloudFront IP range requirements with a domain allow list.If you're allowing IP address ranges, then confirm that all IP addresses used by Amazon Connect are on the required allow lists. For more information, see Allow IP address ranges.Confirm that your network connection is stable enough to allow the Amazon Connect CCP to maintain connection with the required endpoints.Related informationTroubleshooting issues with the CCPSet up your networkFollow"


Question: How do I troubleshoot Amazon ECS tasks for Fargate that are stuck in the Pending state?

My Amazon Elastic Container Service (Amazon ECS) task that's running on AWS Fargate is stuck in the PENDING state.

Answer: "My Amazon Elastic Container Service (Amazon ECS) task that's running on AWS Fargate is stuck in the PENDING state.ResolutionCheck what routes to the internet your subnets are usingFor Fargate tasks in a public subnet:Verify that your Fargate task has an assigned public IP address and a default route (0.0.0.0/0) to an internet gateway. To do this, select the Enable auto-assign public IPv4 address check box when you launch your task or create a new service. For more information, see Public IPv4 addresses.Note: You can't select the Enable auto-assign public IPv4 address check box for existing tasks or services.For Fargate tasks in a private subnet:Verify that your Fargate task has a default route (0.0.0.0/0) to either a NAT gateway, AWS PrivateLink, or another source of internet connectivity.If you're using a NAT gateway, then place your NAT gateway in a public subnet. For more information, see Architecture with an internet gateway and a NAT gateway.If you're using AWS PrivateLink, then confirm that your Fargate infrastructure is allowed to use security groups for your Amazon Virtual Private Cloud (Amazon VPC) endpoints.Check your network access control list and security group settingsVerify that your network access control list (network ACL) and security groups don't block outbound access to port 443 from the subnet. For more information, see Control traffic to resources using security groups.Note: Fargate tasks must have outbound access to port 443 to activate outgoing traffic and reach Amazon ECS endpoints.Check your VPC endpointsIf you're using AWS PrivateLink, confirm that you have the required endpoints.Required endpoints for Fargate platform versions 1.3.0 or earlier:com.amazonaws.region.ecr.dkrS3 gateway endpointRequired endpoints for Fargate platform versions 1.4.0 or later:com.amazonaws.region.ecr.dkrcom.amazonaws.region.ecr.apiS3 gateway endpointNote: If your task definition uses AWS Secrets Manager, SSM parameters, or Amazon CloudWatch Logs, you might need to define endpoints. For more information, see Using an AWS Secrets Manager VPC endpoint and Using CloudWatch Logs with interface VPC endpoints.While using PrivateLink, confirm that the security groups for your VPC endpoints allow the Fargate infrastructure to use these security groups.Check your AWS Identity and Access Management (IAM) roles and permissionsThe task execution role grants the Amazon ECS container and Fargate agents permission to make AWS API calls on your behalf. This role is required by Fargate when you:Pull a container image from Amazon Elastic Container Registry (Amazon ECR)Use the awslogs log driverUse private registry authenticationReference sensitive data using Secrets Manager secrets or AWS Systems Manager Parameter Store parametersIf your use case involves any of the preceding scenarios, confirm that you have the right permissions defined in your task execution role. For a complete list of required permissions, see Amazon ECS task execution IAM role.Check for issues with pulling the imageIf you receive a cannotpullcontainer error for your Fargate task, then complete the steps in How do I resolve the "cannotpullcontainererror" error for my Amazon ECS tasks on Fargate?VPC in dual stack modeWhen using a VPC in dual-stack mode with Fargate, you can configure your VPC with an internet gateway or an outbound-only internet gateway for tasks that are assigned an IPv6 address to access the internet. For more information, see Using a VPC in dual-stack mode.Note: To troubleshoot your issue, you can also use Amazon ECS Exec to retrieve the logs from the container instance of your task or service.Container dependency is definedA container dependency that's defined in the Task Definition can lead to the Fargate task to be in PENDING state indefinitely. Example: If containerA depends on certain state of containerB, containerA is expected to stay in PENDING state until the containerB reaches that state. But if containerB never reaches the desired state, then the task stays in PENDING state indefinitely. Be sure that you have the dependencies appropriately or evaluate the dependencies.For more information, see Container dependency.Follow"


Question: Why doesn't my EFS file system automatically mount when rebooting my EC2 instance?

I've added an Amazon Elastic File System (Amazon EFS) mount entry in the /etc/fstab on my Amazon Elastic Compute Cloud (Amazon EC2) instance. Amazon EFS doesn’t automatically mount on reboot. How can I fix this?

Answer: "I've added an Amazon Elastic File System (Amazon EFS) mount entry in the /etc/fstab on my Amazon Elastic Compute Cloud (Amazon EC2) instance. Amazon EFS doesn’t automatically mount on reboot. How can I fix this?Short descriptionAmazon EFS auto mounts on reboot using the Amazon EFS mount helper or the Linux NFS client. Auto mounting might fail for the following reasons:The Amazon EFS mount helper isn't installed.The Linux NFS client isn't installed.The _netdev option isn't included in the EFS mount entry in /etc/fstab.ResolutionAmazon EFS mount helperThe Amazon EFS mount helper is a part of amazon-efs-utils. This package is available for installation on Amazon Linux and Amazon Linux 2 AMIs.Check if the amazon-efs-utils package is installed on the instance:$ rpm -qa amazon-efs-utilsIf the package isn't installed, then run the following command to install it Amazon Linux and Amazon Linux 2 AMIs:$ sudo yum install -y amazon-efs-utilsFor more information, see Installing the Amazon EFS client on Amazon Linux and Amazon Linux 2.To install amazon-efs-utils on other Linux distributions, see Installing the Amazon EFS client on other Linux distributions.Linux NFS clientVerify that the Linux NFS client is installed on the instance.Note: Use the Linux NFS client only when encryption or access points aren't used.If the package isn't installed, then run the following commands to install it:Amazon Linux 1, Amazon Linux 2, RHEL$ sudo yum -y install nfs-utilsUbuntu$ sudo apt-get -y install nfs-commonRun one of the following commands to start the service:$ sudo service nfs start-or-$ sudo service nfs-server startFor more information see, Installing the NFS client._netdev optionThe automount could fail if _netdev option isn't used in the EFS mount entry in /etc/fstab. The _netdev option prevents the operating system from mounting EFS until the network is enabled as EFS required network access.Related informationRecommended NFS mount optionsFollow"


Question: How can I install Python libraries on my EMR clusters?

I want to install external Python libraries on my Amazon EMR clusters

Answer: "I want to install external Python libraries on my Amazon EMR clustersShort descriptionYou can install Python libraries using a bootstrap action.EMR uses puppet, the deployment mechanism used by Apache BigTop, to configure and initialize applications on instances. Instance-controller is EMR's software component that runs on every instance of the cluster. Instance-controller initializes and then provisions instances based on the instance configuration.The instance-controller runs the provision-node script at /usr/share/aws/emr/node-provisioner/bin/provision-node to start NodeProvisioner at cluster startup. NodeProvisioner provisions all of the EMR distribution's applications for the node and cluster configuration. NodeProvisioner is treated as a final bootstrap action that runs after all other bootstrap actions are run on each node of the cluster.ResolutionIn the latest EMR clusters, bootstrap actions run before Amazon EMR installs any applications specified at cluster creation. The bootstrap action runs before cluster nodes begin processing data. If you add nodes to a running cluster, then bootstrap actions also run on those nodes in the same way. You can create custom bootstrap actions and specify applications to install when you create your cluster. For more information, see Create bootstrap actions to install additional software.Troubleshoot libraries installed by bootstrap actions that are overridden by default librariesLibraries installed using bootstrap actions might be overridden by Amazon EMR default libraries. The bootstrap script runs before cluster creation and before node provisioning. So, libraries might be overridden by the default version.To avoid this issue, create a delayed bootstrap action or a second stage bootstrap action as running code. Or, install packages after receiving the message NODEPROVISIONSTATE SUCCESSFUL.The following bootstrap script upgrades the library after the application provisioning stage. You can add this script as a bootstrap script that runs in the background and exits successfully so that cluster provisioning continues. This script continues to monitor node provisioning and upgrades the library after provisioning.The following example script upgrades the NumPy version:#!/bin/bashwhile true; doNODEPROVISIONSTATE=\` sed -n '/localInstance [{]/,/[}]/{/nodeProvisionCheckinRecord [{]/,/[}]/ {   /status: / { p }    /[}]/a   }  /[}]/a}'  /emr/instance-controller/lib/info/job-flow-state.txt | awk ' { print \$2 }'\`        if [ "\$NODEPROVISIONSTATE" == "SUCCESSFUL" ]; then                     sleep 10;                echo "Running my post provision bootstrap"                #your code here                #Below example lines                #sudo python3 -m pip uninstall numpy==1.16.5 (this is default version of numpy)                #sudo python3 -m pip install --upgrade numpy==1.20.1 (new version of numpy)                exit;        fisleep 10;doneNote: In some cases, YARN containers running a Python package might not use an updated package that can be installed using the preceding resolution. If the container isn't running an updated package, you see module not found errors when trying to install. This is because the YARN NodeManager process is responsible for launching containers. The NodeManager's containers might already be running or allocated before the NODEPROVISIONSTATE is successful. This issue is often seen in multi-tenant clusters that have frequent auto scaling.You can avoid module not found errors by polling the state of the nodemanager service. Then, run the desired bootstrap action as soon as the nodemanager starts.Follow"


Question: How can I troubleshoot issues with my Route 53 failover routing policy?

"I configured an Amazon Route 53 failover routing policy. However, when I test the DNS resolution, I see unexpected results."

Answer: "I configured an Amazon Route 53 failover routing policy. However, when I test the DNS resolution, I see unexpected results.Short descriptionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.When you associate health checks with only the primary failover record, the following scenarios occur:If the primary record is unhealthy, then a DNS query returns the secondary record.If there's no health check for the secondary record, then the secondary record is always treated as healthy.If the primary health check becomes healthy again, then Route 53 fails over to the primary record and provides a response to a DNS query.When you associate health checks with both the primary and secondary failover records, the following scenarios occur:If the primary record is healthy, then a DNS query returns the primary record.If the primary record is unhealthy and the secondary record is healthy, then a DNS query returns the secondary record.If both records are unhealthy, then a DNS query returns the primary record.When you configure the secondary record, adding a health check is optional. When there's no health check for the secondary record and the primary record is unhealthy, Route 53 responds to DNS queries using the secondary record. This applies even when the resources that are mapped to the secondary record are unhealthy. If the secondary record doesn't have a health check associated with it, then Route 53 returns the secondary record.Suppose that you configure the health check for an alias record type and set Evaluate target health (ETH) to true. In this case, Route 53 checks the health of the resource that the alias record references before it returns the alias record. If you associate a health check with alias record, then the health check and your alias’s ETH must be healthy to return the alias record.Note: If your alias record set points towards another record (target) in same hosted zone, then the target record must have an associated health check. Otherwise, Route 53 considers the alias record healthy and includes it with possible responses to queries.Resolution1.    To test the configuration of your record set, use the DNS checking tool.2.    To query the DNS configuration, use the dig or nslookup tools. Be sure to replace the placeholder URLs in the following commands with your corresponding values:$ dig abc.example.com$ nslookup abc.example.com3.    Using the information that you find in steps 1-2, determine whether the issue relates to the primary or secondary record.4.    Check the health check configuration to determine if health checks are reporting as healthy. For more information, see How Amazon Route 53 checks the health of your resources. If you identify failing health checks, then see Viewing health check status and the reason for health check failures.5.    If you see that the health check is healthy but you still see unexpected results, then check the record resolution. To do this, send the query to one of your domain hosted zone authoritative name servers or another public resolver:$ dig abc.example.com @AuthoritativeNameServer$ nslookup abc.example.com AuthoritativeNameServer$ dig abc.example.com @DNSResolver_IP$ nslookup abc.example.com DNSResolver_IPNote: Replace AuthoritativeNameServer with your name server. Replace DNSResolver_IP with your public resolver IP address.If you see an expected output, then the problem is most likely an intermediate or local DNS resolver cache issue.Related informationConfiguring DNS failoverHow Amazon Route 53 determines whether a health check is healthyAdvanced configuration ("Monitor an endpoint" only)Follow"


Question: How do I configure the default logging settings for AWS IoT Core?

I want to configure the default logging settings for AWS IoT logs in AWS IoT Core to reduce both costs and data traffic.

Answer: "I want to configure the default logging settings for AWS IoT logs in AWS IoT Core to reduce both costs and data traffic.Short descriptionNote: This article relates to only V2 of AWS IoT Core logs.It's a best practice to set default logging to a low verbosity level for your AWS IoT Core and configure resource-specific logging to a higher verbosity level. Log verbosity levels include DISABLED (lowest), ERROR, WARN, INFO, and DEBUG (highest).Important: Depending on your AWS IoT Core fleet size, turning on more verbose log levels can incur high costs and make troubleshooting more difficult. Turning on more verbose log levels also creates higher data traffic. It's a best practice to use INFO or DEBUG only as a temporary measure when troubleshooting. After troubleshooting is complete, reset logging levels back to a less verbose setting.ResolutionPrerequisiteMake sure that you have the AWS Command Line Interface (AWS CLI) installed locally with IoT admin permission credentials. The default AWS Region for AWS CLI must point towards the targeted AWS Region. You must have clients connected to and interacting with your AWS IoT Core endpoints, either as registered or non-registered IoT things.Note: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.Set the default logging levelsBy setting the default logging level for your AWS IoT logs to ERROR or WARN, you can review log errors or warnings when they occur.Note: If you reach out to AWS Support for assistance, you must provide AWS IoT Core logs for them to troubleshoot. This information is considered customer data and can't be accessed by AWS Support.1.    Run the list-v2-logging-levels AWS CLI command to find your current applied logging level configurations:aws iot list-v2-logging-levelsYou receive a NotConfiguredException error if AWS IoT logs isn't configured because no logging levels have been set.If AWS IoT Core logs is configured but turned off, then you receive an output similar to the following:{ "logTargetConfigurations": [ { "logTarget": { "targetType": "DEFAULT" }, "logLevel": "DISABLED" } ]}2.    Run the following command to confirm the AWS Identity and Access Management (IAM) role currently configured for AWS IoT Core logs:aws iot get-v2-logging-optionsYou receive an output similar to the following if an IAM role is configured:{"roleArn": "arn:aws:iam::123456789012:role/service-role/iot-log-role","defaultLogLevel": "DISABLED","disableAllLogs": false}Note: If there isn't an IAM role configured, then AWS IoT Core can't send logs to Amazon CloudWatch and this feature isn't available for you to use.3.    Set AWS IoT Core default logging to one of the less verbose logging levels such as ERROR or WARN. This can be done using the AWS IoT console or AWS CLI:Note: If you are troubleshooting an issue and don't have specific client attributes, then you can temporarily set the default logging level to DEBUG. Setting the level to DEBUG generates significant costs and data traffic, depending on the fleet size and the duration that it spent at this level.AWS IoT Console1.    Sign in to the AWS IoT console. Note: Make sure to choose the AWS Region you want to configure.2.    In the navigation pane, choose Settings.3.    Under Logs, choose Manage Logs.4.    Under Log role, choose Create Role. Enter a name for the role, then choose Create.5.    Under Log level, choose one of the less verbose logging levels (ERROR or WARN).6.    Choose Update.7.    Open the IAM console.8.    In the navigation pane, choose Roles.9.    Enter the name of log role that you created in the search box, and then select the role.10.    Under Summary, note the ARN of the log role. It will look similar to arn:aws:iam::123456789012:role/service-role/iot-log-role.AWS CLI1.    Create an IAM role with a specific IAM policy to be used for your AWS IoT Core logs service.2.    Run the following command to set the default logging level to WARN:aws iot set-v2-logging-options \ --role-arn arn:aws:iam::123456789012:role/service-role/iot-log-role \ --default-log-level WARNNote: In this example, replace --role-arn with the IAM role's ARN you created. Replace WARN with the logging level that you want to set as the default level.3.    Run the following command to confirm that the changes are reflected in your logging levels:aws iot list-v2-logging levelsThe output looks similar to the following message:{ "logTargetConfigurations": [ { "logTarget": { "targetType": "DEFAULT" }, "logLevel": "WARN" } ]}4.    Run the following command to confirm the changes are reflected on your logging options:aws iot list-v2-logging-optionsThe output looks similar to the following message:{ "roleArn": "arn:aws:iam::123456789012:role/service-role/iot-log-role", "defaultLogLevel": "WARN", "disableAllLogs": false}Monitor generated logsIt's a best practice to monitor your IoT logs for issues or problems. You can use either the Amazon CloudWatch Logs Console or the AWS CLI to monitor your AWS IoT Core logs. For more information, see the "Monitoring log entries" section of How do I best manage the logging levels of my AWS IoT logs in AWS IoT Core?Related informationMonitoring AWS IoTHow do I configure logging levels dynamically for specific resources in AWS IoT Core?How do I configure logging levels manually for specific resources in AWS IoT Core?Follow"


Question: How do I troubleshoot Network Manager registration and events monitoring issues?

I am unable to register my transit gateway to the global network and monitor the global network with Amazon CloudWatch Events. How do I troubleshoot this?

Answer: "I am unable to register my transit gateway to the global network and monitor the global network with Amazon CloudWatch Events. How do I troubleshoot this?ResolutionTo troubleshoot AWS Network Manager registration and events monitoring issues, do the following:Check your global network configuration and transit gateway registrationFirst, be sure you already created a global network. To create a global network as an AWS Identity and Access Management (IAM) user, you must have the service-linked role (SLR) named AWSServiceRoleForNetworkManager. For more information, see AWS Network Manager service-linked roles. For instructions on creating service-linked role, see Using service-linked roles.Then, confirm that you registered the transit gateway with your global network using the Network Manager console or the AWS Command Line Interface (AWS CLI). If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Note: You must specify the US West (Oregon) Region if you're using AWS CLI or AWS SDK.If the transit gateway isn't located in the same AWS account as the global account, then confirm the following:The transit gateway and global accounts are part of the same AWS Organization. For more information, see Manage multiple accounts in Network Manager with AWS Organizations.Trusted access is turned on to deploy the required SLRs and custom IAM roles to the transit gateway account. Trusted access is required for the management account or delegated administrator account to assume these roles.Multi-account access is turned on. It's a best practice to turn on multi-account access using the Network Manager console. The Network Manager console automatically creates all required roles and permissions for trusted access and allows for registering delegated administrators.Check your Amazon CloudWatch Log Insights configurationConfirm that you have onboarded with CloudWatch Logs Insights. To confirm that you have onboarded with CloudWatch Logs Insights, run the following command:aws logs describe-resource-policies --region us-west-2Then, verify a CloudWatch resource policy with the name DO_NOT_DELETE_networkmanager_TrustEventsToStoreLogEvents is created in the US West (Oregon) Region. The following resources must also be present:A CloudWatch Event rule with the name DO_NOT_DELETE_networkmanager_rule in the US West (Oregon) Region.A CloudWatch Logs log group with the name /aws/events/networkmanagerloggroup in the US West (Oregon) RegionThe CloudWatch Event rule is configured with the CloudWatch Logs log group as a target.If you're unable to onboard to CloudWatch Logs Insights, check that the IAM user or role has the following permissions to perform this action:{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "events:PutTargets",                "events:DescribeRule",                "logs:PutResourcePolicy",                "logs:DescribeLogGroups",                "logs:DescribeResourcePolicies",                "events:PutRule",                "logs:CreateLogGroup"            ],            "Resource": "*"        }    ]}Note: You must specify the US West (Oregon) Region if you're using AWS CLI or AWS SDK.For adding or modifying role permissions, see Adding permissions to a user (console) or Modifying a role permissions policy (console).Check your CloudWatch Events monitoring configurationFirst, be sure that you have created a global network and onboarded with CloudWatch Logs Insights.Note: Monitoring events are captured only after registering the transit gateway in the global network. Any changes made to the transit gateway before registration won't show up under events monitoring.If you're still unable to monitor events, then confirm the following:The CloudWatch Event rule with the name DO_NOT_DELETE_networkmanager_rule was invoked for each captured event. This action must be performed in the US West (Oregon) Region.The FailedInvocations graph for event rule DO_NOT_DELETE_networkmanager_rule is 0. Locate the FailedInvocations graph by accessing the event rule with the name DO_NOT_DELETE_networkmanager_rule, and then choose the Monitoring tab.If there are successful rule invocations matching captured events, then confirm that these events are present in the CloudWatch Logs log group with the name /aws/events/networkmanagerloggroup in the US West (Oregon) Region.Follow"


Question: What are the differences between Standard and Convertible Reserved Instances?

I'm interested in using an Amazon Elastic Compute Cloud (Amazon EC2) Reserved Instance. What are the differences between a Standard Reserved Instance and a Convertible Reserved Instance?

Answer: "I'm interested in using an Amazon Elastic Compute Cloud (Amazon EC2) Reserved Instance. What are the differences between a Standard Reserved Instance and a Convertible Reserved Instance?ResolutionThe size of the discount you receive by using Reserved Instances depends on whether you're using a Standard Reserve Instance or a Convertible Reserve Instance:Standard RIs provide the largest discount for a specific instance type over a long-term contract.Convertible RIs provide a smaller discount, but let you change to certain other instance types during the contract. You can also change other attributes during the contract. For more information, see Exchange Convertible Reserved Instances.For a comparison of Standard and Convertible Reserved Instances, see Types of Reserved Instances (offering classes).Related informationReserved InstancesFollow"


Question: Why does my Amazon SageMaker endpoint go into the failed state when I create or update an endpoint?

I want to troubleshoot why the creation or update of my Amazon SageMaker endpoint has failed.

Answer: "I want to troubleshoot why the creation or update of my Amazon SageMaker endpoint has failed.ResolutionWhen the creation or update of your SageMaker endpoint fails, SageMaker provides the reason for the failure. Use either of the following options to review this reason:Check the endpoint in the SageMaker Console. The reason for the failure is reported in the console.Run the AWS Command Line Interface (AWS CLI) command describe-endpoint. Check the FailureReason field to know the reason for the failure.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.The following are some of the failure reasons and their resolution methods.Unable to provision requested ML compute capacity due to InsufficientInstanceCapacity errorYou might get the following error when you try to create an endpoint:Unable to provision requested ML compute capacity due to InsufficientInstanceCapacity errorThis error occurs when AWS doesn't have sufficient capacity to provision the instances requested for your endpoint.You can resolve this error by trying one or more of the following approaches:Wait for a few minutes and try again because capacity can shift frequently.If you are using multiple instances for your endpoint, try to create the endpoint with a smaller number of instances. If you have Auto Scaling configured, SageMaker can scale up or down as required and as capacity permits.Try a different instance type that supports your workload. After creating an endpoint, update the endpoint with the desired instance type. Because SageMaker uses a blue/green deployment method to maximize availability, you can transition to a new instance type without affecting your current production workloads.The container for production variant <variant> did not pass the ping health check. Please check CloudWatch logs for this endpoint.Containers used for SageMaker endpoints must implement a web server that responds to the /invocations and /ping endpoints. When you create an endpoint, SageMaker starts sending periodic GET requests to the /ping endpoint after the container starts.At a minimum, a container must respond with an HTTP 200 OK status code and an empty body to indicate that the container is ready to accept inference requests. This error occurs when SageMaker doesn't get consistent responses from the container within four minutes after the container starts up. SageMaker doesn't consider that the endpoint is healthy because the endpoint doesn't respond to the health check. Therefore, the endpoint is marked as Failed.Health check might also fail when you use one of AWS Deep Learning Containers images. These images use either TorchServe or Multi Model Server to serve the models that implement the HTTP endpoints for inference and health checks. These frameworks check whether the model is loaded before responding to SageMaker with a 200 OK response. If the server is unable to see that the model is loaded, then the health check fails. A model might not load for many reasons, including memory usage. The corresponding error messages are logged in to Amazon CloudWatch logs for the endpoint. If the code loaded into the endpoint caused the failure (for example, model_fn for PyTorch), then the errors are logged in to AWS CloudTrail. To increase the verbosity of these logs, update the SAGEMAKER_CONTAINER_LOG_LEVEL environmental variable for the model with the log levels for Python logging.A health check request must receive a response within two seconds to be successful. Be sure to test the response by starting your model container locally and sending a GET request to the container to check the response.Failed to extract model data archive for containerSageMaker expects a TAR file with the model data for use in your endpoint. After SageMaker downloads the TAR file, the data archive is extracted. This error might occur if SageMaker can't extract this data archive. For example, SageMaker can't extract the data archive if the model artifact contains symbolic links for files located in the TAR file.When you create an endpoint, make sure that the model artifacts don't include symbolic links within the TAR file. To check if the TAR file includes symbolic links, extract the model data, and then run the following command inside the artifacts:find . -type l -lsThis command returns all the symbolic links found after searching through the current directory and any of its subdirectories. Replace any link that's returned with the actual copies of the file.CannotStartContainerErrorThis error occurs when SageMaker fails to start the container to prepare the container for inference.When SageMaker starts the endpoint, your container is started with the following command:docker run <image-id> serveWhen this command is run, your container must start the serving process.To resolve this error, use local mode for the SageMaker Python SDK. Or, try running your inference image with the docker run command. The SageMaker Python SDK loads up your model similar to a SageMaker endpoint. However, Docker doesn't load the model unless you configure the command or container to do so. You can use a command similar to the following to load your model locally:docker run -v $(pwd)/test_dir:/opt/ml -p 8080:8080 --rm ${image} serveFollow"


Question: "Why is my Storage Gateway file share stuck in the CREATING, UPDATING, or DELETING state?"

"My AWS Storage Gateway file share is stuck in the CREATING, UPDATING, or DELETING state."

Answer: "My AWS Storage Gateway file share is stuck in the CREATING, UPDATING, or DELETING state.Short descriptionThe status of a file share summarizes the health of the file share. There are several reasons why a Storage Gateway file share might be stuck in the creating, updating, or deleting state:The AWS Identity and Access Management (IAM) role given to the file share doesn't grant sufficient access. This IAM role must have permissions for your Amazon Simple Storage Service (Amazon S3) bucket. Additionally, the IAM role's trust policy must grant the Storage Gateway service permissions to assume the role.AWS Security Token Service (AWS STS) is deactivated for the AWS Region that the file gateway is activated in.There's an error in the S3 bucket name that's associated with the file share.When you delete a file share from the file gateway, the share is removed from the associated S3 bucket. However, the contents of the bucket aren't deleted. If data was uploading to the bucket from the gateway when the share was deleted, then the delete process completes only after all the data is uploaded. During this process, the file share shows a DELETING status until after all the data has been uploaded.ResolutionConfirm that the Storage Gateway service has permissions for assuming the IAM role associated with the file share1.    Open the IAM console.2.    From the navigation pane, choose Roles.3.    Choose the IAM role that's associated with your file share.4.    Choose the Trust relationships tab.5.    Confirm that Storage Gateway is listed as a trusted entity. If Storage Gateway isn't a trusted entity, choose Edit trust relationship, and then add the following policy:{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "",      "Effect": "Allow",      "Principal": {        "Service": "storagegateway.amazonaws.com"      },      "Action": "sts:AssumeRole"    }  ]}Prevent a file share stuck in the CREATING or UPDATING statesTo avoid a file share stuck in the CREATING or UPDATING states, create a file share with sufficient access, and then verify that AWS STS is turned on:1.    Open the Storage Gateway console.2.    In the navigation pane, choose File shares, and then choose your File share ID.3.    Choose Actions, choose Delete file share, select the check box to confirm deletion, and then choose Delete.4.    Open the IAM console.5.    In the navigation pane, choose Account settings.6.    In Security Token Services Regions, verify that the Region Status is Active for the location where you want to create the file share.7.    From the Storage Gateway console, choose Create file share.8.    Select your Gateway, enter a name in the Amazon S3 bucket field, and then choose Next.9.    Under Review, accept the defaults, and then choose Create file share.Note: This process might take several minutes to complete.10.    Choose your File share ID.11.    In the Details tab, note the sample mount commands for your OS distribution.Note: If the file share is not available, check the S3 bucket policy to verify access.Delete a file share stuck in the DELETING stateImportant: Follow these steps only if you don't want to wait for all in-progress uploads to finish writing to the S3 bucket.1.    Open the Storage Gateway console.   From the navigation pane, choose File shares. Then, choose the file share ID that you want to delete.3.    Choose the Details tab, and then review the This file share is being deleted message.4.    Verify the ID of the file share in the message. Then, select the confirmation box.Warning: You can't undo the force delete operation.5.    Choose Force delete now.Note: If you're using the new console, and the force delete option isn't available, then use the old console. Or you can use the AWS Command Line Interface (AWS CLI) delete-file-share command with force-delete set to true.Related informationCreate a file shareTroubleshooting file share issuesFollow"