Question: Why is my Amazon OpenSearch Service domain upgrade taking so long?

"I'm trying to upgrade my Amazon OpenSearch Service cluster, but the upgrade is taking a long time. Why is this happening and how can I better monitor my cluster upgrade status in OpenSearch Service?"

Answer: "I'm trying to upgrade my Amazon OpenSearch Service cluster, but the upgrade is taking a long time. Why is this happening and how can I better monitor my cluster upgrade status in OpenSearch Service?Short descriptionWhen you make a configuration change in OpenSearch Service, a blue/green deployment process is used. In a blue/green deployment, two production environments are run (one is live, while the other is idle). The two production environments are then switched according to software updates. For OpenSearch Service, a new environment is created during domain updates, and users are routed to the new production environment after the updates are complete. This behavior minimizes the downtime and maintains the original environment in case a deployment is unsuccessful.To better monitor your cluster upgrade status in OpenSearch Service, monitor your blue/green deployment process at each stage:Creation of new nodesData migrationRemoval of old nodesResolutionRetrieving all cluster snapshots and node IDsBefore a migration, OpenSearch Service takes an automated snapshot of your cluster when it passes the eligibility test. During a snapshot, the progress status might show "null" or 0%. After the snapshot is taken, the percentage value is updated. The time it takes to complete a snapshot can vary depending on the storage space. Because snapshots are taken incrementally, your snapshot can take longer to complete if there are significant changes in your data from the previous automated snapshot.The following _snapshot request retrieves all currently running snapshots with detailed status information:GET /_snapshot/_statusFor more information about the snapshot APIs, see Monitor snapshot progress on the Elasticsearch website.To retrieve all currently running snapshots in your cluster, use the current parameter:GET /_snapshot/<snapshot-repository>/_currentTo obtain the IDs of all data nodes, use the cat nodes API:GET _cat/nodesYou can use the node IDs to identify what nodes are old, and what nodes are new. An increasing number of shards on the new nodes indicates a smooth migration. Eventually, all the shards will move to the new nodes and the old nodes will be empty.Monitoring the blue/green deployment processWhen your cluster enters the blue/green deployment process, the new nodes (in the green environment) appear. The shards are then migrated from the old nodes (in the blue environment). After the data migration or shard reallocation is complete, your old cluster is torn down.You can monitor the blue/green deployment process in its three stages: new nodes, data migration, and removal of old nodes.Stage 1: Creation of new nodesYou can monitor the Nodes cluster metric in Amazon CloudWatch to obtain the node count. Or, you can use the cat nodes API to list all the nodes in your cluster:GET /_cat/nodes?v&prettyBecause you're only updating your cluster version, as soon as the node counts (old nodes and new nodes) increase, the process is complete. Afterwards, you might see your OpenSearch Service domain returning to an "Active" state (after being in the “Processing” state). For clusters with dedicated nodes, you can see that the node count increases to the sum of old and new nodes. The leader nodes of the older configuration will shut down and the node count will decrease by the number of leader nodes. For example, an OpenSearch Service cluster with three dedicated leader nodes will decrease by three nodes.Stage 2: Data migrationAs soon as the first stage is complete, the shard migration begins. During the data migration, the shard count for older nodes decreases, and the shard count for newer nodes increases. You can use the cat allocation API to see how many shards are allocated to each node:GET /_cat/allocation?v&prettyFor more information, see cat allocation on the Elasticsearch website.Stage 3: Removal of old nodesAfter all the shards are migrated to the new nodes, older nodes are removed from your cluster. The node count then returns to the original node count that you configured. At this stage, the blue/green deployment and update process are complete.Follow"


Question: Why won't my pods connect to other pods in Amazon EKS?

My pods won't connect to other pods in Amazon Elastic Kubernetes Service (Amazon EKS).

Answer: "My pods won't connect to other pods in Amazon Elastic Kubernetes Service (Amazon EKS).Short descriptionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you're using the most recent AWS CLI version.If your pods can't connect with other pods, then you might receive the following errors, depending on your application.If the security group from a worker node doesn't allow internode communication, then you receive the following error:curl: (7) Failed to connect to XXX.XXX.XX.XXX port XX: Connection timed outIf the DNS isn't working, then you receive the following error:curl nginx  curl: (6) Could not resolve host: nginxIf the DNS is working but there's a pod connectivity issue, then you receive the following error:Error: RequestError: send request failed caused by: Post  dial tcp 1.2.3.4.5:443: i/o timeoutIf you try to resolve the DNS for the pod that isn't exposed through a service, then you receive the following error:kubectl exec -it busybox -- nslookup nginx Server:   10.100.0.10Address:  10.100.0.10:53** server can't find nginx.default.svc.cluster.local: NXDOMAIN*** Can't find nginx.svc.cluster.local: No answer*** Can't find nginx.cluster.local: No answer*** Can't find nginx.ap-southeast-2.compute.internal: No answer*** Can't find nginx.default.svc.cluster.local: No answer*** Can't find nginx.svc.cluster.local: No answer*** Can't find nginx.cluster.local: No answer*** Can't find nginx.ap-southeast-2.compute.internal: No answerTo resolve these errors, check that your environment is set up correctly:Your security groups meet Amazon EKS guidelines.The network access control list (network ACL) doesn't deny the connection.Your subnet has a local route for communicating within your Amazon Virtual Private Cloud (Amazon VPC).There are enough IP addresses available in the subnet.Your security groups for pods allow pods to communicate with each other.Your worker nodes have IP forwarding on.You meet the networking requirements for Kubernetes (excluding any intentional NetworkPolicy).Your pods are correctly using DNS to communicate with each other.Your pods are scheduled and in the RUNNING state.You have the recommended version of the Amazon VPC Container Network Interface (CNI) plugin for Kubernetes.ResolutionYour security groups meet Amazon EKS guidelinesTo restrict the traffic that you allow on the security group of your worker node, create inbound rules. Create these rules for any protocol or any ports that your worker nodes use for internode communication.It's a best practice to allow all traffic for the security group of your worker node. You don't need to change security group rules every time a new pod with a new port is created.For more information, see Amazon EKS security group requirements and considerations.The network ACL doesn't deny the connection1.    Confirm that traffic between your Amazon EKS cluster and VPC CIDR flows freely on your network ACL.2.    (Optional) To add an additional layer of security to your VPC, consider setting up network ACLs with rules similar to your security groups.Your subnet has a local route for communicating within your VPCConfirm that your subnets have the default route for communication within the VPC.There are enough IP addresses available in the subnetConfirm that your specified subnets have enough available IP addresses for the cross-account elastic network interfaces and your pods.For more information, see Amazon EKS VPC and subnet requirements and considerations.To check for available IP addresses, run the following AWS CLI command:$ aws ec2 describe-subnets --subnet-id YOUR-SUBNET-ID --query 'Subnets[0].AvailableIpAddressCount'Your security groups for pods allow pods to communicate with each otherIf you use security groups for pods or CNI custom networking, then you can allocate any security groups to pods. In this scenario, confirm that the security groups allow communication between pods correctly.Your worker nodes have IP forwarding onIf you use a custom AMI, then you must make sure that net.ipv4.ip_forward kernel variable is turned on. To verify this setting on a worker node, run either of the following commands:# sysctl net.ipv4.ip_forward            # cat /proc/sys/net/ipv4/ip_forwardIf the output is 0, use either of the following commands to active the net.ipv4.ip_forward kernel variable.# sysctl -w net.ipv4.ip_forward=1            # echo 1 > /proc/sys/net/ipv4/ip_forwardFor Amazon EKS AMIs in containerd runtime, see lines 184-188 of the install-worker.sh script (on GitHub) for the setting's activation. Because containerd is the default runtime in Amazon EKS versions 1.24 and newer, this step is required to troubleshoot pod-to-pod network connectivity.You meet the networking requirements for Kubernetes (excluding any intentional NetworkPolicy)Confirm that you meet the Kubernetes networking requirements (from the Kubernetes website).By default, pods are not isolated. Pods accept traffic from any source. Pods become isolated by having a NetworkPolicy that selects them.Note: For NetworkPolicy configurations, see Installing the Calico network policy engine add-on.Your pods are correctly using DNS to communicate with each otherYou must expose your pods through a service first. If you don't, then your pods can't get DNS names and can be reached only by their specific IP addresses.The following example output shows an attempt to resolve the DNS name for the nginx service. In this case, the ClusterIP 10.100.94.70 is returned:$ kubectl run nginx --image=nginx --replicas=5 -n webdeployment.apps/nginx created$ kubectl expose deployment nginx --port=80 -n webservice/nginx exposed$ kubectl get svc -n webNAME    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGEnginx   ClusterIP   10.100.94.70   <none>        80/TCP    2s# kubectl exec -ti busybox -n web -- nslookup nginxServer:    10.100.0.10Address 1: 10.100.0.10 ip-10-100-0-10.ap-southeast-2.compute.internalName:      nginxAddress 1: 10.100.94.70 ip-10-100-94-70.ap-southeast-2.compute.internalIf your pods still fail to resolve DNS, see How do I troubleshoot DNS failures with Amazon EKS?Note: For more information, see Pods, Service, and Headless Services from the Kubernetes website.Your pods are scheduled and in the RUNNING stateConfirm that your pods are scheduled and in the RUNNING state.To troubleshoot your pod status, see How can I troubleshoot the pod status in Amazon EKS?You have the recommended version of the Amazon VPC CNI plugin for KubernetesIf you don't have the recommended version of the Amazon VPC CNI plugin for Kubernetes, then upgrade to the latest version.If you have the recommended version but are experiencing issues with it, then see How do I resolve kubelet or CNI plugin issues for Amazon EKS?Follow"


Question: How do I resolve the error ConnectTimeoutError when connecting to an Amazon EMR cluster from my Amazon SageMaker Studio notebook?

I want to troubleshoot the connection timeout error that I receive when trying to connect to an Amazon EMR cluster from my Amazon SageMaker Studio notebook.

Answer: "I want to troubleshoot the connection timeout error that I receive when trying to connect to an Amazon EMR cluster from my Amazon SageMaker Studio notebook.ResolutionThe connection timeout error might occur due to network configuration issues related to the following for SageMaker Studio or the Amazon EMR cluster:Amazon Virtual Private Cloud (Amazon VPC)SubnetsSecurity groupsBe sure that the following prerequisites for the connection are met:SageMaker Studio is launched in VPC only mode.The Amazon EMR cluster and SageMaker Studio notebook are launched in the same VPC. If they are in different VPCs, then they are connected through a VPC peering connection. If VPC peering is used for your cross-Region use case, then you must configure the config file /etc/sparkmagic/config.json manually. This is because the Amazon EMR cluster's discovery functionality doesn't support cross-Region connection currently. For more information, see Build Amazon SageMaker notebooks backed by Spark in Amazon EMR.The Amazon EMR cluster is launched with Apache Spark and Apache Livy applications installed.To resolve connection timeout errors, do the following:1.    Perform the following checks:Be sure that the security groups or network access control lists (ACLs) are configured correctly to allow traffic on port 8998. Perform this check for both the SageMaker Studio notebook and Amazon EMR cluster.Be sure that the security group for SageMaker Studio has an inbound rule to allow NFS traffic over port 2049 between the domain and Amazon Elastic File System (Amazon EFS) volume.Be sure that the EMR cluster master node security group has an inbound rule for Custom TCP over port 8998. This rule can either specify the Studio's security group or a CIDR that includes the Studio's subnet.Note: If you're using VPC peering, then see Update your security groups to reference peer security groups. This documentation provides information on how to specify a security group from a cross-account VPC. If you have a VPC peering connection between Amazon EMR and Studio subnets, the respective route tables must route traffic to each other. If they don't route the traffic properly, then you get the ConnectTimeoutError.2.    If you set up your private subnet in VPC only mode without a NAT gateway, then create the following AWS PrivateLink interface endpoints for EMR and AWS Security Token Service (AWS STS), respectively:**com.amazonaws..elasticmapreduce****com.amazonaws..sts**These endpoints must be created under the VPC that's used with the EMR cluster and SageMaker Studio.AWS STS is a global service. Therefore, you might get the ConnectTimeoutError when you try to connect to a cross-account Amazon EMR cluster from Studio in a Region other than us-east-1:ConnectTimeoutError: Connect timeout on endpoint URL: "https://sts.amazonaws.com/"To resolve this error, set the environment variable AWS_STS_REGIONAL_ENDPOINTS to regional within the Jupyter notebook before running the connect command:%env AWS_STS_REGIONAL_ENDPOINTS=regional%load_ext sagemaker_studio_analytics_extension.magics%sm_analytics emr connect --cluster-id example-cluster-id --auth-type None  --assumable-role-arn arn:aws:iam::example-cross-account:role/example-role-nameFor more information on Regional endpoints, see Managing AWS STS in an AWS Region and AWS STS Regionalized endpoints.3.    Check whether the connection works by opening your Studio notebook, selecting Sparkmagic kernel, and then running the following command in the cell:For connections within the same account:%local!sm-sparkmagic connect --cluster-id <cluster-id>For cross-account connections:%local# If needed, use STS Regional endpoint%env AWS_STS_REGIONAL_ENDPOINTS=regional !sm-sparkmagic connect --cluster-id <cluster-id> --role-arn arn:aws:iam::<cross-account>:role/<role-name>-or-Run the following command from the notebook terminal on the EMR master node private IP:curl <EMR-Master-Private-IP>:8998/sessions -vPerform the following checks to make sure that the configurations are accurate:SSH into EMR cluster to fetch the PID of the Livy service:ps -ef | grep livyCheck the port that the Livy service is running on:sudo netstat -anp | grep <PID>Be sure that the service is running on the default port (8998) for Livy.Related informationCreate and manage Amazon EMR Clusters from SageMaker Studio to run interactive Spark and ML workloads – Part 1Create and manage Amazon EMR Clusters from SageMaker Studio to run interactive Spark and ML workloads – Part 2Build Amazon SageMaker notebooks backed by Spark in Amazon EMRFollow"


Question: How do I troubleshoot "ThrottlingException" and "Rate exceeded" errors in Amazon SNS?

"When I use an Amazon Simple Notification Service (Amazon SNS) API action, I get a "ThrottlingException" error. The error message says "Rate exceeded." Why am I receiving these errors and how do I resolve them?"

Answer: "When I use an Amazon Simple Notification Service (Amazon SNS) API action, I get a "ThrottlingException" error. The error message says "Rate exceeded." Why am I receiving these errors and how do I resolve them?Short descriptionAmazon SNS API calls can't exceed the maximum allowed API request rate for each AWS account and Region. If API requests exceed this API throttling quota, further API calls are throttled and Amazon SNS returns a ThrottlingException error.To prevent or mitigate ThrottlingException errors in Amazon SNS, you can implement one or more of the best practices outlined in this article.Amazon SNS ThrottlingException error message examples"An error occurred (ThrottlingException) when calling the Subscribe operation: Rate exceeded""An error occurred (ThrottlingException) when calling the ListOriginationNumbers operation: Rate exceeded""An error occurred (ThrottlingException) when calling the ListPhoneNumbersOptedOut operation: Rate exceeded""An error occurred (ThrottlingException) when calling the publish operation: Rate exceeded""An error occurred (ThrottlingException) when calling the GetSMSAttributes operation: Rate exceeded"Note: Amazon SNS API calls are throttled to improve service performance. For more information about Amazon SNS API throttling quotas, see Amazon SNS endpoints and quotas.ResolutionBest practices for preventing ThrottlingException errors in Amazon SNSReduce the frequency of the API calls.Stagger the intervals of the API calls.Implement error retries and exponential backoff.If you still receive ThrottlingException errors after implementing these best practices, request a service quota increase for the API action that's receiving the error.To request an Amazon SNS service quota increaseFollow the instructions in AWS service quotas. When you enter your Use case description, make sure that you include the time frame of the throttling issue and the reason for the quota increase request.Important: You can request service quota increases for Soft Amazon SNS quotas only. You can't increase Hard Amazon SNS quotas. For more information, see Amazon SNS endpoints and quotas.Follow"


Question: Does using the IAM Identity Center affect my IAM identities or federation configuration?

"I want to use the AWS IAM Identity Center (successor to AWS Single Sign-On) to provide access for users to our AWS accounts and applications. I want to know if using the IAM Identity Center affects my AWS Identity and Access Management (IAM) identities (users, groups, and roles)."

Answer: "I want to use the AWS IAM Identity Center (successor to AWS Single Sign-On) to provide access for users to our AWS accounts and applications. I want to know if using the IAM Identity Center affects my AWS Identity and Access Management (IAM) identities (users, groups, and roles).Short descriptionYou can use the IAM Identity Center or IAM to federate your workforce into AWS accounts and applications.IAM federation allows you to activate a separate SAML 2.0 or an OIDC IdP for each AWS account and user attributes for access control. You can use identity providers instead of creating IAM users in your AWS account. For more information, see Identity providers and federation.IAM Identity Center uses IAM service-linked roles. You don't have to manually add permissions with service-linked roles. For more information, see Using service-linked roles for IAM Identity Center.ResolutionThe IAM Identity Center is independent of identity federation configured using IAM. Using the IAM Identity Center doesn't impact IAM identities or your federation configuration.The IAM Identity Center uses the service-linked role AWSServiceRoleForSSO to grant permissions to manage AWS resources. The AWSServiceRoleForSSO role created in AWS accounts only trusts the IAM Identity Center service that's similar to the following IAM trust policy:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Service":"sso.amazonaws.com"      },      "Action": "sts:AssumeRole"    }  ]}The IAM roles created by the service-linked role AWSServiceRoleForSSO have an IAM trust policy that's similar to the following:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Federated": "arn:aws:iam::AWS-account-ID:saml-provider/AWSSSO_ec48a2d3f5dc369d_DO_NOT_DELETE"      },      "Action": "sts:AssumeRoleWithSAML",      "Condition": {        "StringEquals": {          "SAML:aud": "https://signin.aws.amazon.com/saml"        }      }    }  ]}Note: This IAM policy only trusts the SAML provider created automatically by the IAM Identity Center.With IAM federation, you must manually create IAM roles in your AWS accounts using a trust policy that's similar to the following:{  "Version": "2012-10-17",  "Statement": [{    "Effect": "Allow",    "Principal": {"Federated": "arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:saml-provider/MYIDP"},    "Action": "sts:AssumeRoleWithSAML",    "Condition": {"StringEquals": {      "saml:edupersonorgdn": "ExampleOrg",      "saml:aud": "https://signin.aws.amazon.com/saml"    }}  }]}Note: Only IAM entities in your organization with this policy attached are able to access your AWS accounts.To configure the IAM identity Center, see How do I get started with using IAM Identity Center and access the AWS access portal?Related informationHow to create and manage users within AWS IAM Identity CenterHow do I assign user access to cloud applications in the IAM Identity Center?How do I use IAM Identity Center permission sets?Identity federation in AWSFollow"


Question: How do I create an Amazon EKS cluster and node groups that don't require access to the internet?

I want to create an Amazon Elastic Kubernetes Service (Amazon EKS) cluster and node groups with PrivateOnly networking. I don't want to use an internet gateway or network address translation (NAT) gateway.

Answer: "I want to create an Amazon Elastic Kubernetes Service (Amazon EKS) cluster and node groups with PrivateOnly networking. I don't want to use an internet gateway or network address translation (NAT) gateway.Short descriptionUse AWS PrivateLink to create an Amazon EKS cluster and its node group without using a route to the internet.ResolutionCreate an Amazon Virtual Private Cloud (Amazon VPC) for your Amazon EKS cluster1.    Create an AWS CloudFormation stack using the following template:https://amazon-eks.s3.us-west-2.amazonaws.com/cloudformation/2020-06-10/amazon-eks-fully-private-vpc.yamlThe stack creates a VPC with three PrivateOnly subnets and VPC endpoints for the required services. A PrivateOnly subnet has a route table with a default local route and no access to the internet.Important: Although the AWS CloudFormation template creates the VPC endpoints with a full access policy, you can further restrict the policy based on your requirements.Tip: To review all the VPC endpoints after the stack is created, open the Amazon VPC console, and then choose Endpoints from the navigation pane.2.    Open the AWS CloudFormation console.3.    In the navigation pane, choose Stacks.4.    Choose your stack, and then choose the Outputs tab. This tab has information on your subnets that you'll need later, including the VPC ID.Configure the Amazon EKS cluster config file, and then create the cluster and node group1.    In the following config file, update the AWS Region and three PrivateOnly subnets that you created in the Create a VPC for your Amazon EKS cluster section. You can also modify other attributes, or add more attributes, in the config file. For example, you can update name, instanceType, or desiredCapacity.Example config file:apiVersion: eksctl.io/v1alpha5kind: ClusterConfigmetadata:  name: prod  region: us-east-1   nodeGroups:  - name: ng-1    instanceType: m5.xlarge    desiredCapacity: 2    privateNetworking: truevpc:  subnets:    private:      us-east-1a:        id: "subnet-0111111111111111"      us-east-1b:        id: "subnet-0222222222222222"      us-east-1c:        id: "subnet-0333333333333333"  clusterEndpoints:    publicAccess:  true    privateAccess: trueIn the preceding config file, for nodeGroups, set privateNetworking to true. For clusterEndpoints, set privateAccess to true.Important: The eksctl tool isn't required for the resolution. You can use other tools or the Amazon EKS console to create the Amazon EKS cluster and nodes. However, if you use other tools or the console to create a node, you must pass the CA certificate and API server endpoint of the Amazon EKS cluster as arguments, while calling the bootstrap script for the node.2.    To create an Amazon EKS cluster and node group based on the updated config file in step 1, run the following command:$ eksctl create cluster -f config.yamlNote: Some eksctl versions can cause errors with creating the Kubeconfig file when the file mentions client.authentication.k8s.io/v1alpha1. You can update Kubeconfig file by running the update-kubeconfig command (make sure that you are using the latest AWS CLI version for this).The preceding command creates an Amazon EKS cluster and node group in a PrivateOnly network using AWS PrivateLink with no access to internet. The process takes approximately 30 minutes.The output is similar to the following:Output[ℹ]  eksctl version 0.21.0[ℹ]  using region us-east-1[✔]  using existing VPC (vpc-01234567) and subnets (private:[subnet-01111111111111111 subnet-02222222222222222 subnet-03333333333333333] public:[])[!]  custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets[ℹ]  nodegroup "ng-1" will use "ami-0ee0652ac0722f0e3" [AmazonLinux2/1.16][ℹ]  using Kubernetes version 1.16[ℹ]  creating EKS cluster "prod" in "us-east-1" region with un-managed nodes[ℹ]  1 nodegroup (ng-1) was included (based on the include/exclude rules)[ℹ]  will create a CloudFormation stack for cluster itself and 1 nodegroup stack(s)[ℹ]  will create a CloudFormation stack for cluster itself and 0 managed nodegroup stack(s)[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-east-1 --cluster=prod'[ℹ]  CloudWatch logging will not be enabled for cluster "prod" in "us-east-1"[ℹ]  you can enable it with 'eksctl utils update-cluster-logging --region=us-east-1 --cluster=prod'[ℹ]  Kubernetes API endpoint access will use provided values {publicAccess=true, privateAccess=true} for cluster "prod" in "us-east-1"[ℹ]  2 sequential tasks: { create cluster control plane "prod", 2 sequential sub-tasks: { update cluster VPC endpoint access configuration, create nodegroup "ng-1" } }[ℹ]  building cluster stack "eksctl-prod-cluster"[ℹ]  deploying stack "eksctl-prod-cluster"[ℹ]  building nodegroup stack "eksctl-prod-nodegroup-ng-1"[ℹ]  --nodes-min=2 was set automatically for nodegroup ng-1[ℹ]  --nodes-max=2 was set automatically for nodegroup ng-1[ℹ]  deploying stack "eksctl-prod-nodegroup-ng-1"[✔]  all EKS cluster resources for "prod" have been created[✔]  saved kubeconfig as "/Users/garpunee/.kube/config"[ℹ]  adding identity "arn:aws:iam::444444444444:role/eksctl-prod-nodegroup-ng-1-NodeInstanceRole-H37FWX4H84GH" to auth ConfigMap[ℹ]  nodegroup "ng-1" has 0 node(s)[ℹ]  waiting for at least 2 node(s) to become ready in "ng-1"[ℹ]  nodegroup "ng-1" has 2 node(s)[ℹ]  node "ip-192-168-254-139.ec2.internal" is ready[ℹ]  node "ip-192-168-60-191.ec2.internal" is ready[ℹ]  kubectl command should work with "/Users/<>/.kube/config", try 'kubectl get nodes'[✔]  EKS cluster "prod" in "us-east-1" region is readyNote: You can also create managed or unmanaged node groups within your cluster using either the AWS Management Console or eksctl. For more information on eksctl, see Managing nodegroups on the Weaveworks website.Related informationModifying cluster endpoint accessUsing config files (eksctl)Follow"


Question: How do I troubleshoot load balancer errors for Amazon ECS tasks on Fargate?

I'm getting load balancer errors with my Amazon Elastic Container Service (Amazon ECS) task on AWS Fargate.

Answer: "I'm getting load balancer errors with my Amazon Elastic Container Service (Amazon ECS) task on AWS Fargate.ResolutionCheck AWS Identity and Access Management (IAM) permissionsThe ecsServiceRole role allows Amazon ECS services to register Amazon ECS containers on Fargate with the load balancer.Confirm that you have the correct ElasticLoadBalancer permissions. For more information, see Service Scheduler IAM Role.Check your container security group ingress rulesIf your container is mapped to port 80, then your container security group must allow inbound traffic on port 80 for the load balancer health checks to pass.Configure your load balancer for all the Availability Zones of the serviceConfigure your load balancer to use all the Availability Zones in an AWS Region, or at least all the Availability Zones for your containers.If a service uses a load balancer and starts a task located in an Availability Zone that the load balancer isn't configured to use, then the task never passes the health check and the task is stopped.Check if your load balancer health check is configured correctlyThe load balancer health check parameters can be too restrictive or point to resources that don't exist. Unhealthy tasks are removed from the load balancer.Confirm that the following parameters are configured correctly for your service load balancer:Ping PortPing PathResponse TimeoutHealth Check IntervalUnhealthy ThresholdFor more information, review the Elastic Load Balancing load balancer health check misconfigured section of Troubleshooting service load balancers.For more information on health check failures caused by load balancers, see How do I troubleshoot health check failures for Amazon ECS tasks on Fargate?Follow"


Question: Why can't I find an Amazon EC2 instance that I launched on my account?

"I launched an Amazon Elastic Compute Cloud (Amazon EC2) instance, but I can't see it in the Amazon EC2 console."

Answer: "I launched an Amazon Elastic Compute Cloud (Amazon EC2) instance, but I can't see it in the Amazon EC2 console.ResolutionYou can use the AWSSupport-ListEC2Resources AWS Systems Manager Automation document to list your active EC2 resources. This automation document returns information on EC2 instances and related resources. These resources can be Amazon Elastic Block Store (Amazon EBS) volumes, Elastic IP addresses, or Amazon EC2 Auto Scaling groups from specified AWS Regions. By default, the information is gathered from all Regions and appears in the output. To run this automation, follow the instructions in Run an automation.The following are some common reasons why you might not see an EC2 instance in the console:The instance is running, but the console is set to a different Region: Because each EC2 instance is launched in a specific Region, you can view only one Region at a time in the console. Be sure that you select the correct Region based on the output of the automation document.Another account user shut down or terminated the instance: If you don't see the EC2 instance in the automation document, then check with other users of your account to see if they terminated the instance.The instance is running on another AWS account: If you don't see the EC2 instance in the output of the automation document, then run the automation on your other AWS accounts to find the instance.The instance was running, but a process or service shut down the instance: If you don't see the EC2 instance in the output of the automation document, then a process or service might have terminated the instance. Some AWS services, such as AWS Auto Scaling, AWS Elastic Beanstalk, Elastic Load Balancing, or AWS Lambda, can start, stop, and terminate instances automatically. For more information, see the service documentation.The instance was launched as a Spot Instance, and the current Spot price exceeds your bid: If you don't see your Spot Instance in the output of the automation document, then it's possible that the Spot instance was terminated. Spot Instances terminate automatically when your bid price is lower than the current Spot price. For current Spot prices, see Amazon EC2 Spot Instances pricing.Related informationWhy do I have running EC2 instances that I didn't launch?What do I do if I receive a bill from AWS but can't find the resources related to the charges?Follow"


Question: Why is the option "Recover this instance" disabled when creating an alarm in the EC2 console?

"I'm trying to create an Amazon CloudWatch alarm in the Amazon Elastic Compute Cloud (Amazon EC2) console. I want to recover my EC2 instance if it's impaired by an underlying hardware failure, and then send a notification when the alarm triggers. However, on the Create Alarm page, the "Recover this instance" option under "Take the action" is grayed out (disabled). The alarm notification options are also disabled. How do I turn these options back on?"

Answer: "I'm trying to create an Amazon CloudWatch alarm in the Amazon Elastic Compute Cloud (Amazon EC2) console. I want to recover my EC2 instance if it's impaired by an underlying hardware failure, and then send a notification when the alarm triggers. However, on the Create Alarm page, the "Recover this instance" option under "Take the action" is grayed out (disabled). The alarm notification options are also disabled. How do I turn these options back on?Short descriptionIf CloudWatch determines that an EC2 instance uses instance store volumes, the Recover this instance option is disabled. The recover action is not supported for EC2 instances that use instance store volumes, including those launched with block device mappings for instance store volumes. CloudWatch reads the metadata associated with each EC2 instance to verify that it doesn’t use instance store volumes before enabling the recover action.Note: The Recover action is supported only on certain instance types. Make sure that you launch a supported instance type. For more information, see Recover your instance. The EC2 recover alarm action is valid only for the StatusCheckFailed_System metric.ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Before proceeding, see Create alarms that stop, terminate, reboot, or recover an instance. Be sure that you meet all the prerequisites for creating CloudWatch alarms.Find ephemeral instance store volumes in EC2 Linux instances1.     Connect to your Linux instance.2.     Use the following command to retrieve metadata from your instance.curl http://169.254.169.254/latest/meta-data/block-device-mapping/3.     Use the output to identify instance store volumes (whose names begin with "ephemeral") mapped to the instance. For example, there are two instance store volumes ("ephemeral0" and "ephemeral2") in this output:[ec2-user ~]$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/amiephemeral0ephemeral2rootFind ephemeral instance store volumes in EC2 Windows instancesUsing a browser:1.     Connect to your Windows instance.2.     Open a browser and connect to:http://169.254.169.254/latest/meta-data/block-device-mapping/3.     Use the information returned to identify instance store volumes (whose names begin with "ephemeral") mapped to the instance.Using Windows PowerShell:1.     Connect to your Windows instance in PowerShell.2.     Run the following command:PS C:\> Invoke-RestMethod -uri http://169.254.169.254/latest/meta-data/block-device-mapping/3.     Use the output to identify instance store volumes (whose names begin with "ephemeral") mapped to the instance. For example, there is one instance store volume ("ephemeral0") in this output:amiephemeral0rootswapFor more information, see View the instance block device mapping for instance store volumes.Find ephemeral instance store volumes in AMIs for EC2 instancesReview the block device mappings for the Amazon Machine Image (AMI) for your EC2 instance. Check for block device instance store volumes whose "Block Devices" field includes "=ephemeral#", where "#" is an integer from 0-23.Block device mapping can occur even if the instance type doesn’t officially support instance store volumes. This action can occur when you use one of the following to launch an EC2 instance (Windows or Linux):The AWS CLIAn SDKA service such as AWS CloudFormation When you use the Amazon EC2 console to launch an instance type that doesn’t support instance store volumes, Amazon EC2 ignores modified block device mappings.You can enumerate the EBS volumes in an AMI’s block device mapping using the Amazon EC2 console or the AWS CLI.For example, the following describe-images command returns the BlockDeviceMapping for the AMI:aws ec2 describe-images --image-ids ami-xxxxxxxxExample output with instance store volumes whose "VirtualName" begins with "ephemeral":aws ec2 describe-images --image-ids ami-11112222"BlockDeviceMappings": [   {      "DeviceName": "/dev/xvda",      "Ebs": {         "DeleteOnTermination": true,         "SnapshotId": "snap-        ",         "VolumeSize": 8,         "VolumeType": "gp2",         "Encrypted": false      }   },   {      "DeviceName": "/dev/sdb",      "VirtualName": "ep0hemeral0"   },   {      "DeviceName": "/dev/sdc",      "VirtualName": "ephemeral2"   }],Remove or override any references to instance store volumes in the AMIAfter you’ve confirmed that your EC2 instance uses instance store volumes, you must remove or override any references to instance store volumes in the AMI. Then, you must relaunch the instance because instance store volumes can’t be added or removed from an existing EC2 instance.Use one of the following methods to remove or override references to instance store volumes:Remove references to instance store volumes from the AMI. Then, relaunch the instance.Launch the EC2 instance from the Amazon EC2 console to override the AMI for instance types that don’t support instance volume stores.Launch the EC2 instance from the AWS CLI, and then manually override each device in the AMI that references an instance store volume. For example, if you have devices "/dev/sdb" and "/dev/sdc" that map to instance store volumes, launch your EC2 instance using the run-instances command:aws ec2 run-instances --image-id ami-11112222 --subnet-id subnet-11112222 --instance-type t2.medium --key-name my_key --associate-public-ip-address --block-device-mappings  '[{"DeviceName": "/dev/sdb","NoDevice": ""},{"DeviceName": "/dev/sdc","NoDevice": ""}]'For an EC2 instance launched by CloudFormation, create a CloudFormation template to override mapping of any devices to instance store volumes. For example:{  "AWSTemplateFormatVersion": "2010-09-09",  "Resources": {   "I4EY8I": {    "Type": "AWS::EC2::Instance",    "Properties": {     "ImageId" : "ami-11112222",     "InstanceType" : "t2.medium",     "SubnetId" : "subnet-11112222",     "KeyName" : "my_key",   "BlockDeviceMappings" : [     {       "DeviceName":"/dev/sdb",       "NoDevice": {}     },     {       "DeviceName":"/dev/sdc",       "NoDevice": {}     }  ]   }  } }}For an EC2 instance launched by an SDK, check the relevant SDK documentation for how to override mapping devices to instance store volumes.Related informationUsing Amazon CloudWatch alarmsFollow"


Question: How do I set up email notifications for AWS Trusted Advisor in my account?

I want to be notified of the status of my Trusted Advisor checks by email.

Answer: "I want to be notified of the status of my Trusted Advisor checks by email.ResolutionYou can enable weekly email notifications from Trusted Advisor in the Preferences page of the Trusted Advisor console.For more information on Trusted Advisor notifications, see How does the Trusted Advisor notification feature work? in AWS Support FAQs.You can retrieve and refresh Trusted Advisor results programmatically. For more information, see AWS Support API Reference.Related informationAWS Trusted AdvisorFollow"


Question: How do I troubleshoot AWS VPN tunnel inactivity or tunnel down on my customer gateway device?

"I'm observing inactivity, instability, or intermittent connectivity issues with the AWS Virtual Private Network (VPN) tunnels on my customer gateway device."

Answer: "I'm observing inactivity, instability, or intermittent connectivity issues with the AWS Virtual Private Network (VPN) tunnels on my customer gateway device.ResolutionCommon reasons for AWS VPN tunnel inactivity or instability on a customer gateway device include the following:Problems with Internet Protocol Security (IPsec) dead peer detection (DPD) monitoringIdle timeouts due to low traffic on a VPN tunnel or vendor-specific customer gateway configuration issuesRekey issues for phase 1 or phase 2A policy-based VPN connection on the customer gateway device is causing intermittent connectivity issuesCheck DPD settingsIf a VPN peer doesn't respond to three successive DPDs, then the peer is considered dead, and the tunnel is closed.If your customer gateway device has DPD turned on, then be sure that the following are true:It's configured to receive and respond to DPD messagesIt isn't too busy to respond to DPD messages from AWS peersIt isn't rate limiting DPD messages because IPS features are turned on in the firewallIt doesn't have internet transit issuesTroubleshoot idle timeoutsIf you're experiencing idle timeouts that are caused by low traffic on a VPN tunnel, then check the following:Confirm that there's constant bidirectional traffic between your local network and your virtual private cloud (VPC). If necessary, create a host that sends ICMP requests to an instance in your virtual private cloud (VPC) every 5 seconds.Review your VPN device's idle timeout settings using information from your device's vendor. If there's no traffic through a VPN tunnel for the duration of your vendor-specific VPN idle time, then the IPsec session ends. Check your vendor documentation for your specific device.Troubleshoot rekey issues for phase 1 or phase 2If you're experiencing rekey issues that are caused by phase 1 or phase 2 mismatch on a VPN tunnel, then check the following:Review the phase 1 or phase 2 lifetime fields on the customer gateway. Make sure that these fields match the AWS parameters. It's a best practice to uncheck parameters in the VPN tunnel options that aren't needed with the customer gateway for the VPN connection.Make sure that perfect forward secrecy (PFS) is activated on the customer gateway device. PFS is activated on the peer on the AWS side, by default.Make sure that inbound traffic to UDP ports 500 [IKE], 4500 [NAT-T], and IP 50 [ESP] on the customer gateway allow rekeys for the AWS endpoint.Note: The IKEv2 lifetime value field is independent of peers. So, if you set a lower lifetime value, then the peer always initiates the rekey.For more information, see Tunnel options for your Site-to-Site VPN connection and Your customer gateway device.Troubleshoot intermittent connectivity issuesIntermittent connectivity issues might be caused by the policy based configuration on your customer gateway device. You might also experience intermittent connectivity issues because you're using multiple encryption domains or proxy-IDs.Limit the number of encryption domains (networks) that have access to your VPC. If you have more than one encryption domain behind your VPN's customer gateway, then configure them to use a single security association. To check if multiple security associations exist for your customer gateway, see Troubleshooting your customer gateway device.Configure your customer gateway device to allow any network behind the customer gateway (0.0.0.0/0) with a destination of your VPC Classless Inter-Domain Routing (CIDR) to pass through the VPN tunnel. This configuration uses a single security association, which improves tunnel stability. This configuration also allows networks that aren't defined in the policy to access the VPC.For more information, see How do I troubleshoot connection problems between an AWS VPN endpoint and a policy-based VPN?Related informationThe VPN tunnel between my customer gateway and my virtual private gateway is Up, but I am unable to pass traffic through it. What can I do?Follow"


Question: "How can I create an IAM policy to explicitly grant permissions for an IAM user, group, or role to create and manage EC2 instances in a specified VPC using tags?"

"How can I create an AWS Identity and Access Management (IAM) policy that explicitly grants an IAM user, group, or role permissions to create and manage Amazon Elastic Compute Cloud (Amazon EC2) instances in a specified VPC? The policy must limit permissions so that the IAM entity can create EC2 instances with specific tags and manage those EC2 instances in a VPC by using those tags."

Answer: "How can I create an AWS Identity and Access Management (IAM) policy that explicitly grants an IAM user, group, or role permissions to create and manage Amazon Elastic Compute Cloud (Amazon EC2) instances in a specified VPC? The policy must limit permissions so that the IAM entity can create EC2 instances with specific tags and manage those EC2 instances in a VPC by using those tags.Short descriptionAmazon EC2 provides limited supported resource-level permissions, but there are several actions, resources, and conditions to consider. Certain Amazon EC2 API actions, such as launching an EC2 instance, can be controlled through the VPC ARN using tags to control the instances.ResolutionApply a custom IAM policy to restrict the permissions of an IAM user, group, or role for creating EC2 instances in a specified VPC with tags. Use policy condition "ec2:ResourceTags" to limit control to instances. This policy grants permissions to launch EC2 instances in a designated VPC with a unique tag. You can then manage those EC2 instances using restrictive tags.Create a managed policy to apply to the IAM entities that launch your instances1.    Open the IAM console, choose Policies, and then choose Create Policy.2.    Choose the JSON tab, and then enter this custom policy:{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "GrantIAMPassRoleOnlyForEC2",      "Action": [        "iam:PassRole"      ],      "Effect": "Allow",      "Resource": [        "arn:aws:iam::ACCOUNTNUMBER:role/ROLENAME",        "arn:aws:iam::ACCOUNTNUMBER:role/ROLENAME"      ],      "Condition": {        "StringEquals": {          "iam:PassedToService": "ec2.amazonaws.com"        }      }    },    {      "Sid": "ReadOnlyEC2WithNonResource",      "Action": [        "ec2:Describe*",        "iam:ListInstanceProfiles"      ],      "Effect": "Allow",      "Resource": "*"    },    {      "Sid": "ModifyingEC2WithNonResource",      "Action": [        "ec2:CreateKeyPair",        "ec2:CreateSecurityGroup"      ],      "Effect": "Allow",      "Resource": "*"    },    {      "Sid": "RunInstancesWithTagRestrictions",      "Effect": "Allow",      "Action": "ec2:RunInstances",      "Resource": [        "arn:aws:ec2:REGION:ACCOUNTNUMBER:instance/*",        "arn:aws:ec2:REGION:ACCOUNTNUMBER:volume/*"      ],      "Condition": {        "StringEquals": {          "aws:RequestTag/TAG-KEY": "TAG-VALUE"        }      }    },    {      "Sid": "RemainingRunInstancePermissionsNonResource",      "Effect": "Allow",      "Action": "ec2:RunInstances",      "Resource": [        "arn:aws:ec2:REGION::image/*",        "arn:aws:ec2:REGION::snapshot/*",        "arn:aws:ec2:REGION:ACCOUNTNUMBER*:network-interface/*",        "arn:aws:ec2:REGION:ACCOUNTNUMBER*:key-pair/*",        "arn:aws:ec2:REGION:ACCOUNTNUMBER*:security-group/*"      ]    },    {      "Sid": "EC2RunInstancesVpcSubnet",      "Effect": "Allow",      "Action": "ec2:RunInstances",      "Resource": "arn:aws:ec2:REGION:ACCOUNTNUMBER:subnet/*",      "Condition": {        "StringEquals": {          "ec2:Vpc": "arn:aws:ec2:REGION:ACCOUNTNUMBER:vpc/VPC-ID"        }      }    },    {      "Sid": "EC2VpcNonResourceSpecificActions",      "Effect": "Allow",      "Action": [        "ec2:DeleteNetworkAcl",        "ec2:DeleteNetworkAclEntry",        "ec2:DeleteRoute",        "ec2:DeleteRouteTable",        "ec2:AuthorizeSecurityGroupEgress",        "ec2:AuthorizeSecurityGroupIngress",        "ec2:RevokeSecurityGroupEgress",        "ec2:RevokeSecurityGroupIngress",        "ec2:DeleteSecurityGroup",        "ec2:CreateNetworkInterfacePermission",        "ec2:CreateRoute",        "ec2:UpdateSecurityGroupRuleDescriptionsEgress",        "ec2:UpdateSecurityGroupRuleDescriptionsIngress"      ],      "Resource": "*",      "Condition": {        "StringEquals": {          "ec2:Vpc": "arn:aws:ec2:REGION:ACCOUNTNUMBER:vpc/VPC-ID"        }      }    },    {      "Sid": "AllowInstanceActionsTagBased",      "Effect": "Allow",      "Action": [        "ec2:RebootInstances",        "ec2:StopInstances",        "ec2:TerminateInstances",        "ec2:StartInstances",        "ec2:AttachVolume",        "ec2:DetachVolume",        "ec2:AssociateIamInstanceProfile",        "ec2:DisassociateIamInstanceProfile",        "ec2:GetConsoleScreenshot",        "ec2:ReplaceIamInstanceProfileAssociation"      ],      "Resource": [        "arn:aws:ec2:REGION:ACCOUNTNUMBER:instance/*",        "arn:aws:ec2:REGION:ACCOUNTNUMBER:volume/*"      ],      "Condition": {        "StringEquals": {          "ec2:ResourceTag/TAG-KEY": "TAG-VALUE"        }      }    },    {      "Sid": "AllowCreateTagsOnlyLaunching",      "Effect": "Allow",      "Action": [        "ec2:CreateTags"      ],      "Resource": [        "arn:aws:ec2:REGION:ACCOUNTNUMBER:instance/*",        "arn:aws:ec2:REGION:ACCOUNTNUMBER:volume/*"      ],      "Condition": {        "StringEquals": {          "ec2:CreateAction": "RunInstances"        }      }    }  ]}3.    Replace the ACCOUNTNUMBER,  REGION, TAG-KEY, TAG-VALUE, VPC-ID, and ROLENAME parameters with values from your environment.4.    (Optional) If you're assigning this policy to only IAM users or groups, then you can replace the TAG-KEY or TAG-VALUE parameters with the IAM policy variable ${aws:username}. This policy variable allows the IAM service to prepopulate these parameters with the friendly name of the calling IAM user. This step allows IAM users to launch an instance in only the specified VPC, and allows the IAM users to control their own instances.5.    Choose Review policy, and then for Name, enter a name. For example, enter "VPC_Lockdown_VPC-ID", where "VPC-ID" is the ID of the VPC that you are applying this policy to.6.    Choose Create policy.Note: Some items must be replaced with specific resources from your environment. For more information, see Amazon Resource Names (ARNs).Attach the policy to a user, group, or role1.    In the IAM console navigation pane, choose Users, Groups, or Roles.2.    Choose the user, group, or role that you are attaching the policy to.3.    Choose Attach policies.4.    Enter the name of the policy that you created in the search box, and then choose your policy. For example, enter "VPC_Lockdown_VPC-ID".5.    Choose Attach Policy.An IAM entity with this custom policy attached can sign in to the AWS Management Console, open the Amazon EC2 dashboard, and then launch an EC2 instance after specifying the subnet, VPC, and tag.This policy restricts the following actions using the policy condition "ec2:ResourceTags":Starting the instanceStopping the instanceRebooting the instanceTerminating the instanceAttaching a volume to the instanceDetaching a volume from the instanceDisassociating the IAM instance profile from the instanceReplacing the IAM instance profile association for the instanceGetting a console screenshot of the instanceThis policy restricts the following actions against the specified VPC:Deleting security groupsCreating and delete routesDeleting route tablesDeleting network ACLsDeleting ACL entriesAuthorizing or revoke security group ingress and egress rulesCreating network interface permissionsUpdating security group description for ingress and egress rulesFollow"


Question: How can I attach an IAM managed policy to an IAM role in AWS CloudFormation?

I want to add an existing or new AWS Identity and Access Management (IAM) managed policy to a new or existing IAM role in AWS CloudFormation.

Answer: "I want to add an existing or new AWS Identity and Access Management (IAM) managed policy to a new or existing IAM role in AWS CloudFormation.Short descriptionTo add an existing or new IAM managed policy to a new IAM role resource, use the ManagedPolicyArns property of resource type AWS::IAM::Role. To add a new IAM managed policy to an existing IAM role resource, use the Roles property of resource type AWS::IAM::ManagedPolicy.Your IAM managed policy can be an AWS managed policy or a customer managed policy.Important: You can attach a maximum of 10 managed policies to an IAM role or user. The size of each managed policy can't exceed 6,144 characters. For more information, see IAM and STS quotas.Based on your scenario, complete the steps in one of the following sections:Add an existing IAM managed policy to a new IAM roleAdd a new IAM managed policy to a new IAM roleAdd a new IAM managed policy to an existing IAM roleResolutionAdd an existing IAM managed policy to a new IAM role1.    In your AWS CloudFormation template, create a parameter or parameters that you can use to pass in the Amazon Resource Name (ARN) of your IAM managed policy. See the following JSON and YAML examples.JSON:{    "AWSTemplateFormatVersion": "2010-09-09",    "Parameters": {        "awsExampleManagedPolicyParameterOne": {            "Type": "String",            "Description": "ARN of the first IAM Managed Policy to add to the role"        },        "awsExampleManagedPolicyParameterTwo": {            "Type": "String",            "Description": "ARN of the second IAM Managed Policy to add to the role"        }    }}YAML:Parameters:  awsExampleManagedPolicyParameterOne:    Type: String    Description: 'ARN of the first IAM Managed Policy to add to the role'  awsExampleManagedPolicyParameterTwo:    Type: String    Description: 'ARN of the second IAM Managed Policy to add to the role'2.    In the Resources section of your template, for the resource of type AWS::IAM::Role, set Ref to the parameters that you created in step 1. For this example, these are the awsExampleManagedPolicyParameterOne and awsExampleManagedPolicyParameterTwo parameters. See the following JSON and YAML examples.JSON:{    "Resources": {        "RootRole": {            "Type": "AWS::IAM::Role",            "Properties": {                "AssumeRolePolicyDocument": {                    "Version": "2012-10-17",                    "Statement": [                        {                            "Effect": "Allow",                            "Principal": {                                "Service": [                                    "ec2.amazonaws.com"                                ]                            },                            "Action": [                                "sts:AssumeRole"                            ]                        }                    ]                },                "Path": "/",                "ManagedPolicyArns": [                    {                        "Ref": "awsExampleManagedPolicyParameterOne"                    },                    {                        "Ref": "awsExampleManagedPolicyParameterTwo"                    }                ]            }        }    }}YAML:Resources:  RootRole:    Type: 'AWS::IAM::Role'    Properties:      AssumeRolePolicyDocument:        Version: '2012-10-17'        Statement:          - Effect: Allow            Principal:              Service:                - ec2.amazonaws.com            Action:              - 'sts:AssumeRole'      Path: /      ManagedPolicyArns:        - !Ref awsExampleManagedPolicyParameterOne        - !Ref awsExampleManagedPolicyParameterTwo3.    To apply your existing IAM managed policy to your new IAM role, create a stack or update an existing stack based on your modified AWS CloudFormation template.Add a new IAM managed policy to a new IAM role1.    In your AWS CloudFormation template, create a new policy using the AWS::IAM::ManagedPolicy resource. See the following JSON and YAML examples.JSON:{    "SampleManagedPolicy": {        "Type": "AWS::IAM::ManagedPolicy",        "Properties": {            "PolicyDocument": {                "Version": "2012-10-17",                "Statement": [                    {                        "Sid": "AllowAllUsersToListAccounts",                        "Effect": "Allow",                        "Action": [                            "iam:ListAccountAliases",                            "iam:ListUsers",                            "iam:GetAccountSummary"                        ],                        "Resource": "*"                    }                ]            }        }    }}YAML:SampleManagedPolicy:    Type: AWS::IAM::ManagedPolicy    Properties:      PolicyDocument:        Version: '2012-10-17'        Statement:          -            Sid: AllowAllUsersToListAccounts            Effect: Allow            Action:              - iam:ListAccountAliases              - iam:ListUsers              - iam:GetAccountSummary            Resource: "*"2.    Use the !Ref logical ID syntax to attach the IAM managed policy resource to the AWS::IAM::Role resource.For example, set Ref to the resource logical ID that you created in step 1 (SampleManagedPolicy). See the following JSON and YAML examples.JSON:{    "RootRole": {        "Type": "AWS::IAM::Role",        "Properties": {            "AssumeRolePolicyDocument": {                "Version": "2012-10-17",                "Statement": [                    {                        "Effect": "Allow",                        "Principal": {                            "Service": [                                "ec2.amazonaws.com"                            ]                        },                        "Action": [                            "sts:AssumeRole"                        ]                    }                ]            },            "Path": "/",            "ManagedPolicyArns": [                {                    "Ref": "SampleManagedPolicy"                }            ]        }    }}YAML:RootRole:    Type: 'AWS::IAM::Role'    Properties:      AssumeRolePolicyDocument:        Version: '2012-10-17'        Statement:          - Effect: Allow            Principal:              Service:                - ec2.amazonaws.com            Action:              - 'sts:AssumeRole'      Path: /      ManagedPolicyArns:        - !Ref SampleManagedPolicy3.    To apply your new IAM managed policy to your new IAM role, create a stack or update an existing stack based on your modified AWS CloudFormation template.Add a new IAM managed policy to an existing IAM role1.    In your AWS CloudFormation template, create a parameter that you can use to pass in the name of your existing roles. See the following JSON and YAML examples.JSON:{    "Parameters": {        "awsExampleRolesParameter": {            "Type": "CommaDelimitedList",            "Description": "Names of existing Roles you want to add to the newly created Managed Policy"        }    }}YAML:Parameters:  awsExampleRolesParameter:    Type: CommaDelimitedList    Description: Names of existing Roles you want to add to the newly created Managed Policy2.    In the Resources section of your template, for the resource of type AWS::IAM::ManagedPolicy, set Ref to the parameter that you created in step 1 (awsExampleRolesParameter). See the following JSON and YAML examples.JSON:{    "Resources": {        "SampleManagedPolicy": {            "Type": "AWS::IAM::ManagedPolicy",            "Properties": {                "PolicyDocument": {                    "Version": "2012-10-17",                    "Statement": [                        {                            "Sid": "AllowAllUsersToListAccounts",                            "Effect": "Allow",                            "Action": [                                "iam:ListAccountAliases",                                "iam:ListUsers",                                "iam:GetAccountSummary"                            ],                            "Resource": "*"                        }                    ]                },                "Roles": {                    "Ref": "awsExampleRolesParameter"                }            }        }    }}YAML:Resources:  SampleManagedPolicy:    Type: 'AWS::IAM::ManagedPolicy'    Properties:      PolicyDocument:        Version: '2012-10-17'        Statement:          - Sid: AllowAllUsersToListAccounts            Effect: Allow            Action:              - 'iam:ListAccountAliases'              - 'iam:ListUsers'              - 'iam:GetAccountSummary'            Resource: '*'      Roles: !Ref awsExampleRolesParameter3.    To apply your new IAM managed policy to your existing IAM role, create a stack or update an existing stack based on your modified AWS CloudFormation template.Follow"


Question: How can I record my Linux terminal session in text mode and share or replay it later?

I want to record my Amazon Elastic Compute Cloud (Amazon EC2) Linux terminal session or SSH session in text mode and then share it or replay it later. How can I do this?

Answer: "I want to record my Amazon Elastic Compute Cloud (Amazon EC2) Linux terminal session or SSH session in text mode and then share it or replay it later. How can I do this?Short descriptionYou can record your SSH or terminal session to a file using the script command. If you need help, you can run the man script command to access the user manual.ResolutionRecord the session1.    Open an SSH terminal. Replace the example IP address in the following command with your IP address or hostname.$ ssh 198.51.100.02.    Start a script session. In the following command, -a enables append mode and -t sets a timing file.$ script -a -t timingfile.txt typescript.txt3.    Run any commands that you want to record. The commands and any output are recorded.4.    When done, exit the script session by typing exit or pressing Ctrl-D.5.    Files named typescript.txt and timingfile.txt contain the recorded data. You can share these files as needed.Replay the session1.    Copy the typescript.txt and timingfile.txt files to a Linux (or Unix) server or replay them from the same server.2.    Use the following command to show the complete output of the typescript.txt file immediately without a delay:$ cat typescript.txt3.    Use the following command to replay the file with normal timing:$ scriptreplay --timing=timingfile.txt typescript.txt4.    Use the following command to replay the file with a timing divisor to enable faster replay:$ scriptreplay --timing=timingfile.txt typescript.txt -d 10Note: The Linux executables script and scriptreplay come pre-installed by default, so they don't need to be installed separately. If you do need to install them, use the following command that corresponds to your operating system:Ubuntu 16 and Ubuntu 18:sudo apt install bsdutilsRHEL 6:sudo yum install util-linux-ngRHEL 7, Amazon Linux 1, and Amazon Linux 2:sudo yum install util-linuxRelated informationHow to record SSH sessions established through a bastion hostFollow"


Question: How do I create a Lambda layer using a simulated Lambda environment with Docker?

I want to create an AWS Lambda layer that's compatible with target environment runtimes. How can I do that?

Answer: "I want to create an AWS Lambda layer that's compatible with target environment runtimes. How can I do that?Short descriptionWhen you develop Lambda functions, you choose between supported Lambda runtime versions. To create Lambda functions and layers that are compatible with specific runtimes, you can do either of the following:Use an AWS Serverless Application Model (AWS SAM) to automatically create the new resources.-or-Use the AWS SAM's underlying amazon/aws-sam-cli-build-image Docker images to create them.For more information, see Creating and sharing Lambda layers.ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.The following instructions use Python as the coding language and a Linux-based computer platform as an example. For other coding languages and platforms, adapt the steps accordingly.Install Docker on your computerIf you haven't already done so, install Docker for Linux, Windows, or macOS on your computer.Create the appropriate directory structure and specify your dependencies in the pip requirements file (requirements.txt)For more information on creating directory structures, see Including library dependencies in a layer. For more information on specifying dependencies, see Requirements files in the pip user guide.Example directory structure for a Lambda layer that's compatible with Python versions 3.6 and 3.8├── requirements.txt└── python/    └── lib/        ├── python3.6/        │   └── site-packages/        └── python3.8/            └── site-packages/Note: The total unzipped size of the function and all layers can't exceed the unzipped deployment package size limit of 250 MB. For more information about Python version support in Lambda, see Building Lambda functions with Python.Install the library dependencies to their appropriate subfolders based on the runtimes that you specified in your directory structureRun the following AWS CLI command once for each runtime that you specified in the directory structure:Important: Replace 3.6 with 3.7 or 3.8, depending on the compatible libraries that you want to install.docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.6" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.6/site-packages/; exit"Create a .zip file archive for your layerRun the following command to zip the python folder and create the layer package:Note: You can replace mypythonlibs with any name that you prefer.zip -r mypythonlibs.zip python > /dev/nullThe .zip file is deflated in the /opt folder of the Lambda environment. The /opt folder is accessible from your Lambda function code.Create or update your layerRun the following command to either create or update your Lambda layer:Important: Make sure that you include the compatible runtimes that you specified earlier. Replace mypythonlibs with the package name that you used in the previous step. Replace My python libs with any description that you prefer.aws lambda publish-layer-version --layer-name mypythonlibs --description "My python libs" --zip-file fileb://mypythonlibs.zip --compatible-runtimes "python3.6" "python3.8"Review the output and copy the layer's Amazon Resource Name (ARN). You need the layer's ARN to complete the next step.Update your Lambda function configuration to use the layerRun the following command to update your Lambda function configuration to use the layer:Important: Replace arn:aws:lambda:us-east-2:123456789012:layer:mypythonlibs:1 with the layer ARN.aws lambda update-function-configuration --layers arn:aws:lambda:us-east-2:123456789012:layer:mypythonlibs:1 --function-name my-function(Optional) To use the Docker images for your Lambda package instead of an AWS SAM fileInstead of using an AWS SAM file, you can also use the amazon/aws-sam-cli-build-image Docker images to create your Lambda package. Using the Docker images allows you to create the package without creating a layer.1.    Run the following command to get the required versions of your dependencies:Important: Replace 3.6 with 3.7 or 3.8, depending on the compatible libraries that you want to install.docker run -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.6" /bin/sh -c "pip install -r requirements.txt -t libs; exit"2.    Create a Lambda deployment package that includes your function code and the correct libraries.Related informationWhy am I getting "unknown service", "parameter validation failed", or "object has no attribute" errors with my Python (Boto 3) Lambda function?Building layersUse any programming language and share common componentsLambda (AWS CLI Command Reference)Follow"


Question: Knowledge Center


Question: How do I update my telephone number that's associated with my AWS account?

I want to update my telephone number because my contact information has changed.

Answer: "I want to update my telephone number because my contact information has changed.ResolutionTo change the AWS account telephone number and alternate contacts, sign in to your AWS account as the root user.You can also use an AWS Identity and Access Management (IAM) user to update the account telephone number and alternate contacts. The IAM user must have administrator access permission and access to the AWS Billing console.Make sure that the telephone number has the correct international dialing code standard with no additional spaces or special characters. Also, remove any leading zeros after the country code. For example, the United Kingdom country code is +44. Enter the telephone number 07911 123456 as +447911123456 for the United Kingdom.To update your phone number, complete the following steps:Log in to the AWS Billing and Cost Management console for the AWS account that you want to update.Next to Contact Information, choose Edit.Update the phone number.Important: Make sure that the phone number has the correct country code and no extra spaces.Choose Update.Related informationHow do I update the mailing address associated with my AWS account?How do I change the email address that's associated with my AWS account?Managing an AWS accountFollow"


Question: How do I migrate my network traffic from a Direct Connect virtual interface associated with a virtual private gateway to a Direct Connect gateway?

I want to migrate network traffic from my existing AWS Direct Connect virtual interface that's associated with a virtual private gateway to a Direct Connect gateway. I want to complete the migration with minimum downtime. How can I do this?

Answer: "I want to migrate network traffic from my existing AWS Direct Connect virtual interface that's associated with a virtual private gateway to a Direct Connect gateway. I want to complete the migration with minimum downtime. How can I do this?Short descriptionTo migrate traffic from a Direct Connect virtual interface that's associated with a virtual private gateway to a new Direct Connect gateway, you must create a new virtual interface. When you create the new virtual interface, be sure to associate it with your new Direct Connect gateway. Then, use Border Gateway Protocol (BGP) attributes on your network device (such as AS path prepending) to migrate traffic to the Direct Connect gateway with minimal downtime.ResolutionNote: Before proceeding, check your vendor documentation for instructions on how to configure BGP on your specific network device.Perform the following steps during a scheduled maintenance window to minimize downtime:Create a new Direct Connect gateway.Create a new private virtual interface.Important: During creation, be sure to associate the virtual interface with the Direct Connect gateway you created in the previous step.Associate the virtual private gateway that's already attached to your virtual private cloud (VPC) with the Direct Connect gateway that you just created.(Optional)To minimize downtime, pre-stage the configuration on your network device for the new virtual interface associated with the new Direct Connect gateway.Use BGP to configure a longer AS_Path routing policy on the same prefixes that you’re configuring the new virtual interface to advertise. This step configures outbound traffic from AWS to prefer the existing virtual interface route (with a shorter AS_Path).Note: To be sure that traffic from your network device still egresses from the existing virtual interface, use the Local Preference BGP attribute on your network device.During your maintenance window, bring down the BGP session for the existing virtual interface on your network device.Wait for the network traffic from AWS to propagate through the new virtual interface associated with the Direct Connect gateway.Check your network device to confirm that it's receiving traffic from the new virtual interface.Related informationAWS Direct Connect virtual interfacesRouting policies and BGP communitiesFollow"


Question: Why did an EC2 instance that I terminated automatically re-launch?

"I terminated an Amazon Elastic Compute Cloud (Amazon EC2) instance. Then, another instance of the same type automatically launched on my account. Why did this happen, and how can I stop new instances from re-launching?"

Answer: "I terminated an Amazon Elastic Compute Cloud (Amazon EC2) instance. Then, another instance of the same type automatically launched on my account. Why did this happen, and how can I stop new instances from re-launching?Short descriptionSome services allow you to automatically launch instances to replace terminated instances. This can improve the fault tolerance of an application.For example:You can configure your Amazon EC2 Auto Scaling groups to launch an instance that replaces an instance that's undergoing maintenance.AWS Elastic Beanstalk environments usually include Auto Scaling groups by default.You can configure the EC2 Fleet with the request type Maintain. In this scenario, EC2 Fleet places an asynchronous request for your desired capacity, and maintains capacity by automatically replenishing any interrupted Spot Instances.You can configure the Spot Fleet request type as Maintain. In this scenario, Spot Fleet places an asynchronous request for your desired capacity, and maintains capacity by automatically replenishing any interrupted Spot Instances.To prevent services from launching replacement instances, do the following:Delete the Auto Scaling group.Terminate the Elastic Beanstalk environment.Delete the EC2 Fleet.Cancel the Spot Fleet request.ResolutionNote: The following actions permanently delete the selected AWS resources. Create a backup of your EC2 instance before proceeding with this resolution. The backup allows you to revert to your previous working state, if necessary.Delete the Auto Scaling group1.    Note the instance ID of the instance that re-launched.2.    Confirm which Auto Scaling group is causing the re-launch:Open the  Amazon EC2 console. Amazon EC2 resources are Region-specific. Be sure that you're in the correct Region. If not, choose the correct Region from the navigation bar at the top of the screen.Choose Auto Scaling groups in the left navigation pane and open it in a new window.Choose the Auto Scaling group, and then select the Activity History tab.Verify the instance ID of the re-launched instance.Note: If the instance ID doesn't match or you don't have any Auto Scaling groups in this Region, skip to the following Terminate the Elastic Beanstalk Environment section.3.    Choose Actions, Delete.4.    Select Yes, delete.5.    Select Instances from the left navigation pane. The instance is terminated and a new instance doesn't launch.Terminate the Elastic Beanstalk environment1.    Open the AWS Elastic Beanstalk console. Amazon EC2 resources are Region-specific. Be sure that you're in the correct Region, if not, choose the correct Region from the navigation bar at the top of the screen.2.    Select Environments, and then choose the name of your environment from the list.3.    Verify that the name of the re-launched EC2 instance is the same as the Elastic Beanstalk environment.Note: If the instance ID doesn't match, skip to the following Delete the EC2 Fleet section.4.    Choose Environment actions, and then choose Terminate environment.5.    Confirm the environment termination. It might take a few minutes for Elastic Beanstalk to terminate the AWS resources running in the environment.Delete the EC2 FleetEC2 Fleet is available only through the API or AWS CLI. Here we will use AWS CLI to delete the fleet.1.    Make sure you have the AWS CLI installed and configured on your machine.2.    Before proceeding, make sure you're in the correct Region, if not, set the correct Region by running following command:aws configure3.    Confirm if you have any existing EC2 Fleets by running following command:aws ec2 describe-fleets4.    To verify the EC2 instance ID to relaunch, run the following command to check information about specific EC2 Fleets. In the following command, replace fleet-73fbd2ce-aa30-494c-8788-1cee4EXAMPLE with your EC2 fleet ID.aws ec2 describe-fleet-instances --fleet-id fleet-73fbd2ce-aa30-494c-8788-1cee4EXAMPLENote: If the instance ID doesn't match, skip to the following Cancel the Spot Fleet section.5.    Run the following command to delete the EC2 Fleet and terminate currently running instances under the Fleet. In the following command,  replace fleet-73fbd2ce-aa30-494c-8788-1cee4EXAMPLE with your EC2 Fleet ID.aws ec2 delete-fleets --fleet-ids fleet-73fbd2ce-aa30-494c-8788-1cee4EXAMPLE --terminate-instanceCancel the Spot Fleet1.    Open the Amazon EC2 console. Amazon EC2 resources are Region-specific. Be sure that you're in the correct Region, if not, choose the correct Region from the navigation bar at the top of the screen.2.    Select Spot Requests.3.    Confirm which Spot Fleet request is causing the re-launch. Select your Spot Fleet request. To see the configuration details, select Description.4.    To verify the Spot Instances for the Spot Fleet, select Instances.5.    Choose your Spot Fleet request.6.    Select Actions, Cancel spot request.7.    In Cancel spot request, verify that you want to cancel the Spot Fleet. Make sure Terminate instances is selected if you want to terminate the current instances and instances re-launched by this Spot Fleet. When you are ready, choose Confirm.ORYou can use the following cancel-spot-fleet-requests command to cancel the specified Spot Fleet request and terminate the instances. In the following command, replace sfr-73fbd2ce-aa30-494c-8788-1cee4EXAMPLE with your Spot Fleet ID.aws ec2 cancel-spot-fleet-requests --spot-fleet-request-ids sfr-73fbd2ce-aa30-494c-8788-1cee4EXAMPLE \--terminate-instancesRelated informationWhat Is Amazon EC2 Auto Scaling?What Is AWS Elastic Beanstalk?Follow"


Question: Why is my Amazon OpenSearch Service domain stuck in the "Processing" state?

My Amazon OpenSearch Service cluster is stuck in the "Processing" state.

Answer: "My Amazon OpenSearch Service cluster is stuck in the "Processing" state.Short descriptionYour OpenSearch Service cluster enters the Processing state when it's in the middle of a configuration change. The cluster can get stuck in the "Processing" state if either situation occurs:A new set of data nodes fails to launch.Shard migration to the new set of data nodes is unsuccessful.A validation check has failed with errors.When you initiate a configuration change, the domain state changes to Processing while OpenSearch Service creates a new environment. In the new environment, OpenSearch Service launches a new set of applicable nodes, such as data, master, or UltraWarm. After the migration completes, the older nodes are terminated.Note: If your OpenSearch Service domain is stuck in the Deleting state after an attempt to delete it from the "Processing" state, then contact AWS Support.ResolutionA new set of data nodes failed to launchWhen you make simultaneous configuration changes to your cluster before the first change completes, then your cluster can get stuck. Make sure to check for any ongoing blue/green deployments in your cluster. To verify whether there are any ongoing blue/green deployments, check the total number of nodes in Amazon CloudWatch. If you observe a higher node count than expected, then a blue or green deployment is likely in progress.Use the following API call to retrieve more information about additional nodes and the shard migration process:GET /_cluster/health?pretty and GET /_cat/recovery?prettyIf you're using an Amazon Virtual Private Cloud (VPC) domain, then check to make sure that you have enough free IP addresses in your subnet. If there aren't enough IP addresses specified in your subnet, then the launch of new nodes fails. As a result, your cluster gets stuck in the Processing state. For more information, see Reserving IP addresses in a VPC subnet.If you encrypted an OpenSearch Service domain, then make sure that your AWS KMS key exists in your AWS account before you change your configuration. If you accidentally deleted the AWS KMS key, then the cluster might get stuck in the Processing state.Your cluster can also get stuck for the following reasons:There's an overloaded master node with too many pending tasks or high CPU and JVM memory pressure levels. Use the cat pending tasks API to check for any pending tasks. You can also check the MasterCPUUtilization and MasterJVMMemoryPressure metrics in Amazon CloudWatch.The prerequisites for Amazon Cognito authentication for OpenSearch Dashboards weren't met. If you configured Amazon Cognito for OpenSearch Dashboards authentication, then make sure that you met the authentication prerequisites. For example, OpenSearch Service must have the user pool, Amazon Cognito identity pool, and AWS Identity Access Management (IAM) role set with correct permissions. The default name for this role is CognitoAccessForAmazonOpenSearch (with the AmazonESCognitoAccess policy attached).Note: If you created a custom IAM role, then make sure that your role has the same permissions as CognitoAccessForAmazonOpenSearch.Shard migration to the new set of data nodes is unsuccessfulA shard migration (from the old set to the new set of data nodes) might be unsuccessful for the following reasons:Your OpenSearch Service cluster is currently in red health status. If your cluster is in red health status, then troubleshoot your red cluster status so that your cluster is in a healthy state.Note: It's a best practice to configure your cluster when it's in a healthy state.Nodes are out of service because of a heavy processing load caused by high JVM memory pressure and CPU usage. To resolve this issue, reduce your network traffic to the cluster or stop the network traffic entirely, to return the cluster to a healthy state. Otherwise, your blue or green deployment process might time out, requiring manual intervention.Because of internal hardware failures, the shards on old data nodes can get stuck during a migration.Note: Depending on your hardware issue, your cluster also might not recover automatically. If your cluster doesn't recover automatically, then OpenSearch Service runs self-healing scripts to return the nodes to a healthy state. The loss of a node's root volume might prevent OpenSearch Service from responding, and an Auto Scaling group then automatically replaces the faulty nodes. If the attached EBS volume for a node goes down, then manual intervention is required to replace the EBS volume. To identify which shards are still operating from an older set of nodes, use the following API commands: cat allocation API, cat nodes API, or cat shards API.There's a stuck shard relocation caused by insufficient free storage in the new set of nodes. This issue occurs when there is new data coming into the cluster during a blue or green deployment process.Note: A blue or green deployment isn't triggered if OpenSearch Service detects less space than is required for a successful data migration.A stuck shard relocation caused by shards that are pinned to an older set of nodes. To make sure that shards aren't pinned to any nodes before a configuration change is made, check the index setting. Or, check to see if your cluster has a write block that is caused by high JVM memory pressure or low disk space.To identify which index shards are stuck and the corresponding index settings, use the following commands:curl -X GET "ENDPOINT/_cluster/allocation/explain?pretty"curl -X GET "ENDPOINT/INDEX_NAME/_settings?pretty"In your index settings, check to see if either of these settings appear:{    "index.routing.allocation.require._name": "NODE_NAME" (OR)    "index.blocks.write": true}If you see "index.routing.allocation.require._name": "NODE_NAME" in your index settings, then remove the following setting:curl -X PUT "ENDPOINT/INDEX_NAME/_settings?pretty" H 'Content-Type: application/json' -d'{    "index.routing.allocation.require._name": null}For more information, see Index-level shard allocation filtering on the Elastic website.If you observe "index.blocks.write": true in your index settings, then your cluster has a write block. The write block is likely caused by high JVM memory pressure or low disk space. Make sure to address these issues before implementing any other troubleshooting tips. For more information about troubleshooting this exception, see ClusterBlockException.Note: If your cluster is stuck in the "Processing" state for more than 24 hours, then your cluster needs manual intervention. Also, if you haven't made any configuration changes but the node count is higher than expected, then a software patch might be in progress.A validation check has failed with errorsA domain can get stuck in the Processing state when a service software update or version upgrade validation check fails with errors. To resolve this issue, see Troubleshooting validation errors.Follow"


Question: How can I troubleshoot issues when creating a load balancer using the AWS Load Balancer Controller?

"I can't create a Network Load Balancer or an Application Load Balancer using the AWS Load Balancer Controller. Or, the load balancer isn't created after I created the Service or Ingress objects."

Answer: "I can't create a Network Load Balancer or an Application Load Balancer using the AWS Load Balancer Controller. Or, the load balancer isn't created after I created the Service or Ingress objects.Short descriptionTo troubleshoot load balancer creation issues, do the following:Make sure that all prerequisites are met.Check the annotations of the Ingress (Application Load Balancer) or Service (Network Load Balancer) object.Review the AWS Load Balancer Controller pod's logs for additional information.If the cluster runs on AWS Fargate, then verify that a Fargate profile is created for the namespace where the Ingress or Service object resides.Check for unaddressed dependencies.The AWS Load Balancer Controller manages Elastic Load Balancing for an EKS cluster. The controller provisions the following resources:An Application Load Balancer when you create a Kubernetes ingress.A Network Load Balancer when you create a Kubernetes service of type LoadBalancer. Previously, the Kubernetes Network Load Balancer was used for instance targets, and the AWS Load Balancer Controller was used for IP targets. With the AWS Load Balancer Controller version 2.3.0 or later, you can create a Network Load Balancer using either target type. For more information, see Target type.ResolutionMake sure that all prerequisites are metFor a list of Application Load Balancer prerequisites, see Application load balancing on Amazon EKS. For a list of Network Load Balancer prerequisites, see Network load balancing on Amazon EKS.1.    Verify that the AWS Load Balancer Controller is successfully provisioned. It's a best practice to use version 2.4.4 or later.2.    Check the number of subnets. Application Load Balancer needs at least two subnets in different Availability Zones. Network Load Balancer needs at least one subnet. The subnets must have at least eight available IP addresses. For more information, see View your VPCs.3.    You must use the following tag in certain scenarios:Key: "kubernetes.io/cluster/cluster-name"Value: "shared" or "owned"If you're using an Application Load BalancerYou must tag exactly one security group in the following scenarios:You're using multiple security groups attached to worker node.You're using the AWS Load Balancer Controller version v2.1.1 or earlier.If you're using a Network Load BalancerIf you're using the AWS Load Balancer Controller version v2.1.1 or earlier, then subnets must be tagged.For information on adding tags from the Amazon EC2 console, see Work with tags using the console. For information on adding tags using the AWS Command Line Interface (AWS CLI), see Work with tags using the command line.Note: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.4.FSPUnless subnet IDs are explicitly specified as annotations in the Service object or Ingress object, make sure that the subnets have the following tags. Without these tags, Subnet Auto Discovery doesn't work.Private subnets tag:Key: "kubernetes.io/role/internal-elb"Value: "1"Public subnets tag:Key: "kubernetes.io/role/elb"Value: "1"Check the annotations of the Ingress or Service objectVerify annotations on the Service object or annotations on the Ingress object. The annotations needed to configure a load balancer are as follows:Note: Other annotations use default values. For a list of all available annotations supported by the AWS Load Balancer Controller for Application Load Balancing, see Ingress annotations on GitHub. For a list of all available annotations supported by the AWS Load Balancer Controller for Network Load Balancing, see Service annotations on GitHub.Application Load Balancerkubernetes.io/ingress.class: alb: IngressClassParams is a Custom Resource Definition (CRD) that's specific to the AWS Load Balancer Controller. This CRD can be used along with IngressClass's parameter field. You can use IngressClassParams to enforce settings for a set of Ingresses.Network Load BalancerWith IP targets: service.beta.kubernetes.io/aws-load-balancer-type: "external" service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"With instance targets: service.beta.kubernetes.io/aws-load-balancer-type: "external" service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"Run one of the following commands to view the Service or the Ingress object. In the following example, replace SERVICE-NAME/INGRESS-NAME and NAMESPACE with the correct values for your use case.kubectl describe service <SERVICE-NAME> -n <NAMESPACE> kubectl describe ingress <INGRESS-NAME> -n <NAMESPACE>Run one of the following commands to edit the Service or Ingress object. In the following examples, replace SERVICE-NAME/INGRESS-NAME and NAMESPACE with the correct values for your use case.kubectl edit service <SERVICE-NAME> -n <NAMESPACE>kubectl edit ingress <INGRESS-NAME> -n <NAMESPACE>Review the AWS Load Balancer Controller pod's logs for additional informationRun the following command to review the AWS Load Balancer Controller logs:kubectl logs -n kube-system deployment.apps/aws-load-balancer-controllerIf none of the controller pods show logs, then make sure that the controller pods are running:kubectl get deployment -n kube-system aws-load-balancer-controllerIf the cluster runs on Fargate, then verify that a Fargate profile is created for the namespace where the Ingress or Service object residesThe IP target type is required when target pods are running on Fargate. Run the following commands to verify that a Fargate profile is created for the namespace where the Ingress or Service object resides. In the following example, replace CLUSTER-NAME with the name of your cluster.eksctl get fargateprofile --cluster <CLUSTER-NAME> -o yamlTo create a Fargate profile, run the following command. In the following example, replace CLUSTER-NAME, REGION, FARGATE-PROFILE-NAME, and NAMESPACE with the correct values for your use case.eksctl create fargateprofile --cluster <CLUSTER-NAME> --region <REGION> --name <FARGATE-PROFILE-NAME> --namespace <NAMESPACE>Check for unaddressed dependenciesAmazon EKS adds the following rules to the node's security group:An inbound rule for client trafficAn inbound rule for each load balancer subnet in the VPC for each Network Load Balancer that you create (for health checks).Deploying a service of type LoadBalancer might fail if Amazon EKS attempts to create rules that exceed the quota for the maximum number of rules allowed for a security group.Review the documentation to make sure that all dependencies are met. For Application Load Balancer, see Application load balancing on Amazon EKS. For Network Load Balancer, see Network load balancing on Amazon EKS. For example, If you're using an Application Load Balancer, then the Service object must specify the NodePort or LoadBalancer to use instance traffic mode.Follow"