Question: How do I troubleshoot 403 Access Denied errors from Amazon S3?

"My users are trying to access objects in my Amazon Simple Storage Service (Amazon S3) bucket, but Amazon S3 is returning the 403 Access Denied error. How can I troubleshoot this error?"

Answer: "My users are trying to access objects in my Amazon Simple Storage Service (Amazon S3) bucket, but Amazon S3 is returning the 403 Access Denied error. How can I troubleshoot this error?ResolutionUse the AWS Systems Manager automation documentsUse the AWSSupport-TroubleshootS3PublicRead automation document on AWS Systems Manager. This automation document helps you diagnose issues reading objects from a public S3 bucket that you specify. You can also use the AWSSupport-TroubleshootS3AccessSameAccount automation document on AWS Systems Manager to help you diagnose access denied from your S3 bucket. The AWSSupport-TroubleshootS3AccessSameAccount doesn't evaluate permissions for cross-account resources.Check bucket and object ownershipFor AccessDenied errors from GetObject or HeadObject requests, check whether the object is also owned by the bucket owner. Also, verify whether the bucket owner has read or full control access control list (ACL) permissions.Confirm the account that owns the objectsBy default, an S3 object is owned by the AWS account that uploaded it. This is true even when the bucket is owned by another account. If other accounts can upload objects to your bucket, then verify the account that owns the objects that your users can't access.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.1.    Run the list-buckets AWS Command Line Interface (AWS CLI) command to get the Amazon S3 canonical ID for your account by querying the Owner ID.aws s3api list-buckets --query "Owner.ID"2.    Run the list-objects command to get the Amazon S3 canonical ID of the account that owns the object that users can't access. Replace DOC-EXAMPLE-BUCKET with the name of your bucket and exampleprefix with your prefix value.aws s3api list-objects --bucket DOC-EXAMPLE-BUCKET --prefix exampleprefixTip: Use the list-objects command to check several objects.3.    If the canonical IDs don't match, then you don't own the object. The object owner can grant you full control of the object by running the put-object-acl command. Replace DOC-EXAMPLE-BUCKET with the name of the bucket that contains the objects. Replace exampleobject.jpg with your key name.aws s3api put-object-acl --bucket DOC-EXAMPLE-BUCKET --key exampleobject.jpg --acl bucket-owner-full-control4.    After the object owner changes the object's ACL to bucket-owner-full-control, the bucket owner can access the object. However, the ACL change alone doesn't change ownership of the object. To change the object owner to the bucket's account, run the cp command from the bucket's account to copy the object over itself.Copy all new objects to a bucket in another account1.    Set a bucket policy that requires objects to be uploaded with the bucket-owner-full-control ACL.2.    Activate and set S3 object ownership to bucket owner preferred in the AWS Management Console.The object's owner is then automatically updated to the bucket owner when the object is uploaded with the bucket-owner-full-control ACL.Create an AWS Identity and Access Management (IAM) role with permissions to your bucketFor on-going cross-account permissions, create an IAM role in your account with permissions to your bucket. Then, grant another AWS account the permission to assume that IAM role. For more information, see Tutorial: Delegate access across AWS accounts using IAM roles.Check the bucket policy or IAM user policiesReview the bucket policy or associated IAM user policies for any statements that might be denying access. Verify that the requests to your bucket meet any conditions in the bucket policy or IAM policies. Check for any incorrect deny statements, missing actions, or incorrect spacing in a policy.Deny statement conditionsCheck deny statements for conditions that block access based on the following:Multi-factor authentication (MFA)encryption keysspecific IP addressspecific VPCs or VPC endpointsspecific IAM users or rolesNote: If you require MFA and users send requests through the AWS CLI, then make sure that the users configure the AWS CLI to use MFA.For example, in the following bucket policy, Statement1 allows public access to download objects (s3:GetObject) from DOC-EXAMPLE-BUCKET. However, Statement2 explicitly denies everyone access to download objects from DOC-EXAMPLE-BUCKET unless the request is from the VPC endpoint vpce-1a2b3c4d. In this case, the deny statement takes precedence. This means that users who try to download objects from outside of vpce-1a2b3c4d are denied access.{  "Id": "Policy1234567890123",  "Version": "2012-10-17",  "Statement": [    {      "Sid": "Statement1",      "Action": [        "s3:GetObject"      ],      "Effect": "Allow",      "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",      "Principal": "*"    },    {      "Sid": "Statement2",      "Action": [        "s3:GetObject"      ],      "Effect": "Deny",      "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",      "Condition": {        "StringNotEquals": {          "aws:SourceVpce": "vpce-1a2b3c4d"        }      },      "Principal": "*"    }  ]}Bucket policies or IAM policiesCheck that the bucket policy or IAM policies allow the Amazon S3 actions that your users need. For example, the following bucket policy doesn’t include permission to the s3:PutObjectAcl action. If the IAM user tries to modify the access control list (ACL) of an object, then the user gets an Access Denied error.{  "Id": "Policy1234567890123",  "Version": "2012-10-17",  "Statement": [    {      "Sid": "Stmt1234567890123",      "Action": [        "s3:PutObject"      ],      "Effect": "Allow",      "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",      "Principal": {        "AWS": [          "arn:aws:iam::111122223333:user/Dave"        ]      }    }  ]}Other policy errorsCheck that there aren’t any extra spaces or incorrect ARNs in the bucket policy or IAM user policies.For example, if an IAM policy has an extra space in the Amazon Resource Name (ARN) as follows: arn:aws:s3::: DOC-EXAMPLE-BUCKET/*. In this case, the ARN is then incorrectly evaluated as arn:aws:s3:::%20DOC-EXAMPLE-BUCKET/ and gives the IAM user an access denied error.Confirm that IAM permissions boundaries allow access to Amazon S3Review the IAM permissions boundaries that are set on the IAM identities that are trying to access the bucket. Confirm that the IAM permissions boundaries allow access to Amazon S3.Check the bucket's Amazon S3 Block Public Access settingsIf you're getting Access Denied errors on public read requests that are allowed, check the bucket's Amazon S3 block public access settings.Review the S3 Block Public Access settings at both the account and bucket level. These settings can override permissions that allow public read access. Amazon S3 Block Public Access can apply to individual buckets or AWS accounts.Review user credentialsReview the credentials that your users have configured to access Amazon S3. AWS SDKs and the AWS CLI must be configured to use the credentials of the IAM user or role with access to your bucket.For the AWS CLI, run the configure command to check the configured credentials:aws configure listIf users access your bucket through an Amazon Elastic Compute Cloud (Amazon EC2) instance, then verify that the instance is using the correct role. Connect to the instance, then run the get-caller-identity command:aws sts get-caller-identityReview temporary security credentialsIf users receive Access Denied errors from temporary security credentials granted using AWS Security Token Service (AWS STS), then review the associated session policy. When an administrator creates temporary security credentials using the AssumeRole API call, or the assume-role command, they can pass session-specific policies.To find the session policies associated with the Access Denied errors from Amazon S3, look for AssumeRole events within the AWS CloudTrail event history. Make sure to look for AssumeRole events in the same timeframe as the failed requests to access Amazon S3. Then, review the requestParameters field in the relevant CloudTrail logs for any policy or policyArns parameters. Confirm that the associated policy or policy ARN grants the necessary Amazon S3 permissions.For example, the following snippet of a CloudTrail log shows that the temporary credentials include an inline session policy that grants s3:GetObject permissions to DOC-EXAMPLE-BUCKET:"requestParameters": {    "roleArn": "arn:aws:iam::123412341234:role/S3AdminAccess",    "roleSessionName": "s3rolesession",    "policy": "{    "Version": "2012-10-17",    "Statement": [{            "Effect": "Allow",            "Action": [                "s3:GetObject"            ],            "Resource": ["arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"]        }    }]}"}Confirm that the Amazon VPC endpoint policy includes the correct permissions to access your S3 buckets and objectsIf users access your bucket with an EC2 instance routed through a VPC endpoint, then check the VPC endpoint policy.For example, the following VPC endpoint policy allows access only to DOC-EXAMPLE-BUCKET. Users who send requests through this VPC endpoint can’t access any other bucket.{  "Id": "Policy1234567890123",  "Version": "2012-10-17",  "Statement": [    {      "Sid": "Stmt1234567890123",      "Action": [        "s3:GetObject",        "s3:PutObject",        "s3:ListBucket"      ],      "Effect": "Allow",      "Resource": [        "arn:aws:s3:::DOC-EXAMPLE-BUCKET",        "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"      ],      "Principal": "*"    }  ]}Review your Amazon S3 access point's IAM policyIf you use an Amazon S3 access point to manage access to your bucket, then review the access point's IAM policy.Permissions granted in an access point policy are only effective if the underlying bucket policy also allows the same access. Confirm that the bucket policy and access point policy grant the correct permissions.Confirm that the object isn't missing object or contains special charactersCheck whether the requested object exists in the bucket. Otherwise, the request doesn't find the object and Amazon S3 assumes that the object doesn't exist. You receive an Access Denied error (instead of 404 Not Found errors) if you don't have proper s3:ListBucket permissions.An object that has a special character (such as a space) requires special handling to retrieve the object.Run the head-object AWS CLI command to check if an object exists in the bucket. Replace DOC-EXAMPLE-BUCKET with the name of the bucket that you want to check.aws s3api head-object --bucket DOC-EXAMPLE-BUCKET --key exampleobject.jpgIf the object exists in the bucket, then the Access Denied error isn't masking a 404 Not Found error. Check other configuration requirements to resolve the Access Denied error.If the object isn’t in the bucket, then the Access Denied error is masking a 404 Not Found error. Resolve the issue related to the missing object.Check the AWS Key Management Service (KMS) encryption configurationNote the following about AWS KMS (SSE-KMS) encryption:If an IAM user can’t access an object that the user has full permissions to, then check if the object is encrypted by SSE-KMS. You can use the Amazon S3 console to view the object’s properties, which include the object’s server-side encryption information.If the object is SSE-KMS encrypted, then make sure that the AWS KMS key policy grants the IAM user the minimum required permissions for using the key. For example, if the IAM user is using the key only for downloading an S3 object, then the IAM user must have kms:Decrypt permissions. For more information, see Allows access to the AWS account and activates IAM policies.If the IAM identity and key are in the same account, then kms:Decrypt permissions should be granted using the key policy. The key policy must reference the same IAM identity as the IAM policy.If the IAM user belongs to a different account than the AWS KMS key, then these permissions must also be granted on the IAM policy. For example, to download the SSE-KMS encrypted objects, the kms:Decrypt permissions must be specified in both the key policy and IAM policy. For more information about cross-account access between the IAM user and an AWS KMS key, see Allowing users in other accounts to use an AWS KMS key.Confirm that the request-payer parameter is specified by users (if you're using Requester Pays)If your bucket has Requester Pays activated, then users from other accounts must specify the request-payer parameter when they send requests to your bucket. To check whether Requester Pays is turned on, use the Amazon S3 console to view your bucket’s properties.The following example AWS CLI command includes the correct parameter to access a cross-account bucket with Requester Pays:aws s3 cp exampleobject.jpg s3://DOC-EXAMPLE-BUCKET/exampleobject.jpg --request-payer requesterCheck your AWS Organizations service control policyIf you're using AWS Organizations, then check the service control policies to make sure that access to Amazon S3 is allowed. Service control policies specify the maximum permissions for the affected accounts. For example, the following policy explicitly denies access to Amazon S3 and results in an Access Denied error:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Deny",      "Action": "s3:*",      "Resource": "*"    }  ]}For more information on the features of AWS Organizations, see Activating all features in your organization.Related informationTroubleshooting Amazon S3Getting S3 request IDs for AWS SupportFollow"


Question: How do I troubleshoot Lambda triggers that poll from MSK and self-managed Kafka clusters?

"My AWS Lambda function is designed to process records from my Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster or self-managed Kafka cluster. However, the Lambda trigger is displaying an error message."

Answer: "My AWS Lambda function is designed to process records from my Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster or self-managed Kafka cluster. However, the Lambda trigger is displaying an error message.Short descriptionAn event source mapping (ESM) is an AWS Lambda resource that reads from an event source and invokes a Lambda function. To invoke a Lambda function, a Lambda-Kafka ESM must be able to perform the following actions:Communicate with the cluster.Poll records from the topic.Communicate with the Lambda Invoke API.Communicate with the AWS Security Token Service (AWS STS) API.If an ESM's networking, authentication, or authorization settings prevent communication with the cluster, then setup fails before it can invoke a function. The trigger then displays an error message that helps troubleshoot the root cause.ResolutionUnderstand ESMsWhen a Lambda function is configured with an Amazon MSK trigger or a self-managed Kafka trigger, an ESM resource is automatically created. An ESM is separate from the Lambda function, and it continuously polls records from the topic in the Kafka cluster. The ESM bundles those records into a payload. Then, it calls the Lambda Invoke API to deliver the payload to your Lambda function for processing.Important: Lambda-Kafka ESMs don’t inherit the VPC network settings of the Lambda function. This is true for both MSK triggers and self-managed Kafka triggers. An MSK ESM uses the subnet and security group settings that are configured on the target MSK cluster. A self-managed Kafka trigger has WAN access by default, but can be configured with network access to a VPC in the same account and AWS Region. Because the network configuration is separated, a Lambda function can run code within a network that doesn’t have a route to the Kafka cluster.Understand the ESM setup processBefore the ESM can invoke the associated Lambda function, the ESM automatically completes the following steps:1.    The ESM calls AWS STS APIs to get a security token.2.    If SourceAccessConfiguration contains a secret, then grab that secret from AWS Secrets Manager API.3.    For self-managed Kakfa ESM: Lambda resolves the IP address of the cluster endpoints from the hostname that's configured in the ESM, under selfManagedEventSourceEndPoints.      For MSK ESM: Get the MSK cluster's subnet and security group configuration.4.    For self-managed Kakfa ESM: Establish a network connection to the broker endpoint.      For MSK ESM: Create a hyperplane elastic network interface with the MSK cluster's security group in each of the MSK cluster's subnets.5.     Authentication:If TLS auth is activated, then check the SSL certificate that's presented by the broker endpoint.Log in to the broker.6.    Authorization:Make sure that the topic exists in the cluster. Ask the cluster brokers if the topic that's configured in the ESM's Topics parameter exists in the cluster.Create a consumer group in the cluster, using the ESM's UUID as the consumer group ID.7.    Poll records from the topic.8.    Bundle the records into a payload that's smaller than 6 MB. This is the limit for Lambda invocation payloads.9.    The ESM invokes the associated Lambda function with the payload of records. To do this, make a synchronous call to the Lambda Invoke API.Troubleshoot network security issuesWhen the ESM sends a request to the broker endpoints and doesn't receive a response, the ESM considers the request as timed out. When a timeout to the broker endpoint occurs, the trigger displays the following error message:"PROBLEM: Connection error. Please check your event source connection configuration. If your event source lives in a VPC, try setting up a new Lambda function or EC2 instance with the same VPC, Subnet, and Security Group settings. Connect the new device to the Kafka cluster and consume messages to ensure that the issue is not related to VPC or Endpoint configuration. If the new device is able to consume messages, please contact Lambda customer support for further investigation."To troubleshoot the issue, follow the steps that are noted in the preceding error message. Also, note the networking configurations in the following sections to be sure that your ESM is properly configured.Note: Timed out requests from the ESM might also occur in situations where the cluster is out of system resources to handle the request. Or, timed out requests might occur when the wrong security settings are configured on the ESM or cluster. If you receive this error and there aren't any issues with the network configuration, then check the cluster broker's access logs for additional information.Networking configuration that a self-managed Kafka ESM usesA self-managed Kafka ESM's network configuration is similar to a Lambda function. By default, the ESM has access to the WAN but isn’t configured for access within a VPC. It can be manually configured with specific subnets and security groups to access a Kafka cluster. However, it can only access a cluster that’s reachable from a VPC in the account that contains the Lambda function. As a result, you can create a self-managed Kafka ESM for a Kafka cluster that’s in the following locations:An on-premises data centerAnother cloud providerThe MSK brokers of a Kafka cluster that’s located in the VPC of a different accountNote: It’s possible to create a self-managed Kafka trigger that consumes from an MSK cluster in another account. However, there are some downsides. Unlike an MSK trigger, AWS Identity and Access Management (IAM) authentication isn’t available for self-managed Kafka triggers. Also, connecting to the MSK cluster over a VPC-peered connection requires specific VPC workarounds. For more information, see How Goldman Sachs builds cross-account connectivity to their Amazon MSK clusters with AWS PrivateLink.Networking configuration of a Lambda-MSK ESMTo communicate with the MSK cluster, an MSK ESM creates a hyperplane elastic network interface inside each subnet that’s used by the cluster. This is similar to how a Lambda function operates within a VPC.An MSK ESM doesn’t use the Lambda function's VPC settings. Instead, the ESM automatically uses the subnet and security group settings that are configured on the target MSK cluster. The MSK ESM then creates a network interface inside each of the subnets that are used by the MSK cluster. These network intefaces use the same security group that’s used by the MSK cluster. The security groups and ingress or egress rules that are used by the MSK ESM can be found with the following CLI commands:1.    Use the AWS CLI MSK command describe-cluster to list the security groups and subnets used by the MSK cluster.2.    Use the describe-security-groups command on the security groups that are listed in the output of describe-cluster.Grant access to trafficThe MSK cluster's security group must include a rule that grants ingress traffic from itself and egress traffic to itself. The traffic must also be granted over one of the following open authentication ports that’s used by the broker:9092 for plaintext9094 for TLS9096 for SASL443 for all configurationsTroubleshoot issues that can occur during initialization, polling, and invocation"PROBLEM: Connection error. Your VPC must be able to connect to Lambda and STS, as well as Secrets Manager if authentication is required. You can provide access by configuring PrivateLink or a NAT Gateway."The preceding error occurs for any of the following reasons:The ESM is configured in a VPC, and calls to the STS API fail or timeout.The ESM is configured in a VPC, and the Secrets Manager API connection attempts fail or timeout.The trigger can access your Kafka cluster, but it times out when invoking your function over the Lambda API.These issues might be due to incorrect VPC settings that prevent your ESM from reaching other services like AWS STS and AWS Secrets Manager. Follow the steps in Setting up AWS Lambda with an Apache Kafka cluster within a VPC to properly configure your VPC settings.If calls to the STS API fail or timeout, then your VPC settings prevent your ESM from reaching the Regional Lambda endpoint on port 443. To resolve this issue, see Setting up AWS Lambda with an Apache Kafka cluster within a VPC.If SourceAccessConfiguration contains a secret, then be sure to retrieve that secret from Secrets Manager."PROBLEM: Certificate and/or private key must be in PEM format."The preceding error occurs if you have a secret that's not in a format that can be deciphered by the ESM.To troubleshoot this, check the format of your secret. Note that Secrets Manager only supports X.509 certificate files in .pem format. For more information, see Provided certificate or private key is not valid (Amazon MSK) or Provided certificate or private key is not valid (Kafka)."PROBLEM: The provided Kafka broker endpoints cannot be resolved."The preceding error occurs when your ESM is unable to translate the hostname into an IP address.To resolve this error, make sure that the ESM is able to reach a DNS server that can translate the hostname. If the endpoint's hostname is within a private network, then configure the ESM to use a VPC with DNS settings that can resolve the hostname."PROBLEM: Server failed to authenticate Lambda or Lambda failed to authenticate server."The preceding error occurs when the server that your ESM is connected to isn’t the server that you configured in the ESM settings.To troubleshoot this, verify that you configured your ESM’s settings for the server that you’re connecting to."PROBLEM: SASL authentication failed."The preceding error occurs when your server login attempt fails.AWS Lambda functions that are triggered from an Amazon MSK topic can access user names and passwords that are secured by AWS Secrets Manager using SASL/SCRAM. You receive an error when your user name and password aren’t recognized as valid.To resolve this error, log in to the broker and check the access logs.Note:"PROBLEM: Cluster failed to authorize Lambda."The preceding error occurs when the ESM logs in to the broker, but the ESM user doesn't have permission to poll records from the topic. To troubleshoot this issue, see Cluster failed to authorize Lambda (Amazon MSK) or Cluster failed to authorize Lambda (Kafka).Related informationAuthentication and authorization errorsFollow"


Question: How do I transfer data between Amazon MSK clusters in different accounts with MirrorMaker 2 that's running on MSK Connect?

I want to use MirrorMaker 2.0 (MM2) that runs on MSK Connect to transfer data between Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters in different accounts.

Answer: "I want to use MirrorMaker 2.0 (MM2) that runs on MSK Connect to transfer data between Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters in different accounts.ResolutionSet up VPC peeringBecause the Amazon MSK clusters are in different Virtual Private Clouds (VPCs), you must create a VPC peering connection. Create this VPC peering connection with a VPC in another AWS account in the same AWS Region or a different Region. For more information, see Create a VPC peering connection.The security groups that are associated with the source Amazon MSK cluster must allow all traffic from the security groups of the target cluster. The target cluster's security groups must also allow all traffic from the MSK cluster's security groups. For more information, see Update your security groups to reference peer security groups.Note: To reference a security group in another account, include the account number in the Source or Destination field. Example: 123456789012/sg-1a2b3c4dCreate a plugin with MM2 plugin informationMSK Connect custom plugins accept a file or folder with a .jar or .zip ending.Complete the following steps:1.    Create a dummy folder or file, and then compress it:mkdir mm2   zip mm2.zip mm2 2.    Upload the .zip object to your Amazon Simple Storage Service (Amazon S3) bucket in the target account:aws s3 cp mm2.zip s3://mytestbucket/Because Apache Kafka and MSK Connect have MirrorMaker libraries built in, you don’t need to add additional .jar files for this functionality. MSK Connect has a prerequisite that a custom plugin must be present at connector creation. Therefore, you must create an empty one for reference.3.    In the target account, use the .zip file to create a custom plugin. Use mm2-connect-plugin as the name for the custom plugin.Create an MSK Connect ConnectorComplete the following steps to create a connector in the target account:1.    Open the Amazon MSK console.2.    In the navigation pane, under MSK Connect, choose Connectors. 3.    Choose Create connector.4.    In the list of custom plugins, select the one that's next to the custom plugin that you created, and then choose Next.5.    Enter a name for the connector, and optionally, a description.6.    From the list of clusters, choose the target cluster.7.    Copy the following configuration, and then paste it into the connector configuration field. Modify the example for your use case.connector.class=org.apache.kafka.connect.mirror.MirrorSourceConnector  tasks.max=1    clusters=primary,replica  source.cluster.alias=primary  target.cluster.alias=replica    topics=example.*  replication.factor=2  topic.creation.default.replication.factor=2  topic.creation.default.partitions=2  consumer.group.id=mm2-connector    refresh.groups.interval.seconds=20  refresh.topics.interval.seconds=20    sync.topic.configs.interval.seconds=20  sync.topic.acls.interval.seconds=20    producer.enable.idempotence=true    transforms=renameTopic  transforms.renameTopic.type=org.apache.kafka.connect.transforms.RegexRouter  transforms.renameTopic.regex=primary.(.*)  transforms.renameTopic.replacement=$1    key.converter=org.apache.kafka.connect.converters.ByteArrayConverter  value.converter=org.apache.kafka.connect.converters.ByteArrayConverter    # Source cluster options  source.cluster.bootstrap.servers=<Source_MSK_Bootstrap_Server_PLAINTEXT>  source.cluster.security.protocol=PLAINTEXT    # Destination cluster options  target.cluster.bootstrap.servers=<Target_MSK_Bootstrap_Server_PLAINTEXT>  target.cluster.security.protocol=PLAINTEXT8.    Set the capacity for your connector.9.    Under Worker configuration, choose Use the MSK default configuration.10.    Under Access permissions, choose the AWS Identity and Access Management (IAM) role that provides the required permissions to MSK Connect. Then, choose Next.11.    On the Security page, under Encryption - in transit, choose Plaintext traffic. Then, choose Next.12.    Optionally, on the Logs page, set the log delivery. Then, choose Next.13.    Under Review and create, choose Create connector.Note: With this configuration, to replicate each topic from the source cluster, MM2 creates two topics in the target cluster. For example, if you have the topic exampleTopic1 on the source cluster, MM2 creates the topics primary.exampleTopic1 and exampleTopic1 on the target cluster. Messages are routed to the exampleTopic1 topic.Create a client instanceYou must create a client instance to create topics and produce or consume data from topics.1.    Launch an Amazon Elastic Compute Cloud (Amazon EC2) instance according to your requirements in either the source or target account. Then, connect to the instance.2.     Run the following command to install Java on the client machine:sudo yum -y install java-113.    Run the following command to download Apache Kafka:wget https://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgz    tar -xzf kafka_2.12-2.8.1.tgz4.    Create the topic exampletopic1 in the Amazon MSK cluster in the source account:<path-to-your-kafka-installation>/bin/kafka-topics.sh --create --bootstrap-server <Source MSK cluster BootstrapServerString> --replication-factor 3 --partitions 1 --topic exampletopic15.    Produce data in the cluster on the source account:<path-to-your-kafka-installation>/bin/kafka-console-producer.sh --broker-list <Source MSK cluster BootstrapServerString> --topic exampletopic1>message 1  >message 26.    List the topics in the cluster on the target account:<path-to-your-kafka-installation>/bin/kafka-topics.sh --bootstrap-server <Target MSK cluster BootstrapServerString> --listThe output must look similar to the following one:__amazon_msk_canary  __amazon_msk_connect_configs_mm2-*****  __amazon_msk_connect_offsets_mm2-*****  __amazon_msk_connect_status_mm2-*****  __consumer_offsets  exampleTopic1  primary.exampleTopic17.    Consume data from the target cluster:<path-to-your-kafka-installation>/bin/kafka-console-consumer.sh --bootstrap-server <Target MSK cluster BootstrapServerString> --topic exampletopic1 --from-beginningmessage 1  message 2Related informationMigrating clusters using Apache Kafka's MirrorMakerFollow"


Question: How can I troubleshoot Lambda function cold start issues?

My AWS Lambda function is experiencing a high latency cold start duration.

Answer: "My AWS Lambda function is experiencing a high latency cold start duration.ResolutionThe initial setup of a Lambda function request for the environment and code are referred to as a cold start time or startup latency. To minimize the cold start time and latency of your Lambda function, follow these instructions for your use case.Lambda function code and configuration best practicesIncrease the memory allocated to the Lambda function. The cold start range depends on the size of your function, the amount of memory that you allocated, and the complexity of your code. Adding more memory proportionally increases the amount of CPU, increasing the overall computational power available. For more information, see Memory and computing power.Minimize the size of your deployment package. The smaller your deployment package, the faster your function starts up. Minimize the number of dependencies and external libraries that your function imports, and keep your deployment package size under 50 MB. For more information, see Lambda deployment packages.Optimize your Lambda function code to minimize the time that it takes to initialize. Reduce the number of dependencies and external libraries that your function imports. Reduce the amount of code run during initialization.Avoid complex computation at startup. If your function requires complex computation at startup, such as loading large datasets, you can perform this in the background. Run the computation in a background thread during the Init phase. Then, cache the results for subsequent invocations. Caching the results helps reduce the time required for complex computation at startup.Reuse Amazon Relational Database Service (Amazon RDS) database connections. If your function connects to an Amazon RDS database, you can create an Amazon RDS Proxy database proxy for your function. A database proxy manages a pool of database connections. Reusing Amazon RDS database connections reduces the time to establish a connection for each time your function's invoked.Configure provisioned concurrency. Functions using provisioned concurrency don't show cold start behavior because the execution environment is prepared before invocation. You can specify the number of function instances to keep warm and manage traffic.Minimize the complexity of your dependencies. Use simple frameworks that load quickly on execution environment startup.To reduce the time that it takes Lambda to unpack deployment packages authored in Java, put your dependency .jar files in a separate /lib directory. Separating the .jar files is faster than putting all your function’s code in a single jar with a large number of .class files. For more information, see Deploy Java Lambda functions with .zip or JAR file archives.Use monitoring to discover issues and observability to discover why. Monitor the cold start performance of your functions, and use Lambda Insights in Amazon CloudWatch to troubleshoot performance issues.For more information, see Best practices for working with Lambda functions.Long INIT duration in provisioned concurrencyIn on-demand Lambda functions, the static initializer is run after a request is received but before the handler is invoked. This results in latency for the requester and contributes to the overall cold start duration.Determine whether your function concurrency exceeds the configured level of provisioned concurrency. You can check the number of times your function is invoked using the ProvisionedConcurrencySpilloverInvocations CloudWatch metric. A non-zero value indicates that all provisioned concurrency is in use and some invocation occurred with a cold start.Check your invocation frequency (requests per second). Functions with provisioned concurrency have a maximum rate of 10 requests per second per provisioned concurrency (Lambda API requests). For example, a function configured with 100 provisioned concurrencies can handle 1000 requests per second. If the invoke rate exceeds 1000 requests per second, some cold starts might occur. For more information, see Lambda: Cold starts with provisioned concurrency.For more information, see How do I troubleshoot Lambda function provisioned concurrency issues?Amazon API Gateway requests integrated with LambdaIf you're using Lambda with API Gateway and you see a high IntegrationLatency metric, review your Lambda function's CloudWatch Logs. High latency must be addressed when an API endpoint that’s integrated with a Lambda function takes too long to send responses to a client. Cold starts in Lambda functions aren't recorded in the function's duration metric, so your API's integration latency might be longer than the function's duration. To see the duration of your function with a cold start, use AWS X-Ray.For more information, see How do I troubleshoot high latency in my API Gateway requests that are integrated with Lambda?Related informationHow do I reduce initialization and invocation duration latency for my Java Lambda function?Improving startup performance with Lambda SnapStart for Java 11 RuntimeOperating Lambda: Performance optimization – Part 1Follow"


Question: How do I troubleshoot HTTP 403 Forbidden errors when using a Lambda authorizer with an API Gateway REST API?

Calls to my Amazon API Gateway REST API are getting 403 Forbidden errors after I created an AWS Lambda authorizer. How do I troubleshoot these errors?

Answer: "Calls to my Amazon API Gateway REST API are getting 403 Forbidden errors after I created an AWS Lambda authorizer. How do I troubleshoot these errors?Short descriptionThis error can occur if:The Lambda authorizer function returns an AWS Identity and Access Management (IAM) policy document that explicitly denies access to the caller.The API has an attached resource policy that explicitly denies access to the caller.If the call to your API has a token or identity sources that are missing, null, or not validated, then you get a 401 Unauthorized error. For more information, see Why am I getting API Gateway 401 Unauthorized errors after creating a Lambda authorizer?Note: This article addresses 403 errors related to Lambda authorizers that are configured for a REST API only. For information on troubleshooting other types of 403 errors, see How do I troubleshoot HTTP 403 Forbidden errors from API Gateway?ResolutionConfirm the cause of the errorNote:If you haven't already done so, turn on CloudWatch Logs for your API Gateway REST API.If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.If you made any changes to the authorizer configuration, the API must be redeployed.1.    Review the error message in the response from API Gateway. Look for an error message similar to one of the following.Example error message for Lambda authorizer functions that return an IAM policy document with an explicit deny{    "message": "User is not authorized to access this resource with an explicit deny"}Example error message for Lambda authorizer functions that return an IAM policy document with an implicit deny{    "message": "User is not authorized to access this resource"}Example error message for REST APIs that have an attached resource policy that implicitly denies access to the caller{    "message": "User: anonymous is not authorized to perform: execute-api:Invoke on resource: <api-resource-arn>"}Example error message for REST APIs that have an attached resource policy that explicitly denies access to the caller{    "message": "User: anonymous is not authorized to perform: execute-api:Invoke on resource: <api-resource-arn> with an explicit deny"}Note: For more information on resulting behavior when access to an API Gateway API is controlled by an IAM policy, see Policy evaluation outcome tables.2.    View the API Gateway execution logs in CloudWatch to review the authorization workflow. Note the Lambda authorizer's output and the outcome of the API Gateway's resource policy evaluation. You get a log error message similar to one of the following.Example log error message for when a required token is missing or doesn't match the token validationExtended Request Id: MY92nHDwwwIdGxzR=Unauthorized request: <request-id>Note: The Extended Request Id is randomly generated. The Extended Request Id value in your logs is different.Example log error message for when a Lambda authorizer returns a policy that denies accessSending request to https://lambda.<region>.amazonaws.com/2015-03-31/functions/<lambda-authorizer-arn>/invocationsAuthorizer result body before parsing:{  "principalId": "user",  "policyDocument": {    "Version": "2012-10-17",    "Statement": [      {        "Action": "execute-api:Invoke",        "Effect": "Deny",        "Resource": "<resource-arn>"      }    ]  }}Using valid authorizer policy for principal: <principal>Successfully completed authorizer executionThe client is not authorized to perform this operation.Note: The policy returned is dependent on your Lambda authorizer. If the resource-arn of the returned policy does not include the requesting resource, request will be implicitly denied.Example log error message for when the API Gateway resource policy denies the requestExtended Request Id: MY-BIVb4GEdGeZB=ExplicitDenyException User: anonymous is not authorized to perform: execute-api:Invoke on resource: <api-resource-arn> with an explicit deny: <request-id>Resolve "not authorized to access this resource" errors from the Lambda authorizerYou might get the not authorized to access this resource errors intermittently because of policy caching. To confirm that Authorization Caching is turned on, review your Lambda authorizer's configuration in the API Gateway console. Then, do one of the following:For a one-time test, run the AWS CLI command flush-stage-authorizers-cache. With the authorizer's cache entries flushed, call your API again.Turn off policy caching, and then call your API again.Note: When you turn off policy caching for a request parameter-based authorizer, API Gateway doesn't validate calls to your API before invoking the Lambda authorizer function.Change the authorizer's cache key by updating the header name specified in Token Source (for token-based authorizers) or Identity Sources (for request parameter-based authorizers). Redeploy your API to commit the changes. Then, call your API again using the newly configured token header or identity sources.If you find the errors consistently, determine why your authorizer explicitly denies access to the caller by reviewing your Lambda authorizer function's code. If you have that determined that the issue is caused by caching, you can update the code so that it allows access to the caller. For instructions, see Why is my API Gateway proxy resource with a Lambda authorizer that has caching activated returning HTTP 403 errors?Resolve "not authorized to perform: execute-api:Invoke" errorsDetermine if your APIs resource policy isn't valid, or if it explicitly denies access to your calls by reviewing your API's resource policy. You can view your API's execution logs to get the response outcome for the resource policy.For more information, see Access policy language overview for Amazon API Gateway and Lambda authorizer and resource policy.Related informationUse API Gateway Lambda authorizersUpdates to a REST API that require redeploymentControlling and managing access to a REST API in API GatewayFollow"


Question: How do I troubleshoot explicit deny error messages when requesting API calls using IAM roles or users?

I received an explicit deny error message when requesting an API call using an AWS Identity and Access Management (IAM) role or user. How can I troubleshoot and resolve explicit deny error messages?

Answer: "I received an explicit deny error message when requesting an API call using an AWS Identity and Access Management (IAM) role or user. How can I troubleshoot and resolve explicit deny error messages?Short descriptionIn order for an IAM entity (role or user) to make a successful API call, the entity must meet the following conditions:The role or user has the correct permissions to request an API call.The permission isn't denied by any statement in all policies that are applicable to the request context.If your IAM entity doesn't meet these conditions, then the API call fails and throws an (AccessDenied) error similar to this:IAM user or role experiencing the issue: arn:XXXXXXXX:iam::XXXXXXXX:role/TestReadOnlyError: An error occurred (AccessDenied) when calling the RunInstances operation: User: arn:aws:iam::XXXXXXXX:user/tester is not authorized to perform: ec2:RunInstances on resource: role TestReadOnly with an explicit denyNote: The troubleshooting steps in this article deal specifically with explicit deny errors and not implicit deny errors. For more information on implicit deny errors, see The difference between implicit and explicit denies.ResolutionExplicit deny errors occur because of issues in one or more of the following policies:Identity-based policiesResource-based policiesPermissions boundaryService control policiesSession policyIdentity-based policiesThe identity-based policy controls the allowed/denied action of an entity. Use these troubleshooting steps to identify issues with identity-based policies.Note: It's a best practice to use Deny with StringNotLike in conditions to prevent accidental privileged access.1.    Check that there is no deny statement in your identity-based policy. This example contains a deny statement:{  "Effect": "Deny",  "Action": "iam:DeleteRole"  "Resource": "*"}2.    Check if Multi-Factor Authentication (MFA) is enforced on your policy. If your IAM entity authenticates without using another authentication factor when MFA is enforced, then the permission is denied. For example, if your entity authenticates using the AWS CLI without MFA, then your API call is denied. See this example of MFA enforcing:{  "Sid": "DenyAllExceptListedIfNoMFA",  "Effect": "Deny",  "NotAction": [    "iam:CreateVirtualMFADevice",    "iam:EnableMFADevice",    "iam:GetUser",    "iam:ListMFADevices",    "iam:ListVirtualMFADevices",    "iam:ResyncMFADevice",    "sts:GetSessionToken"  ],  "Resource": "*",  "Condition": {    "BoolIfExists": {"aws:MultiFactorAuthPresent": "false"}  }}This policy explicitly denies all API calls, except the ones that are mentioned in the NotAction policy element.3.    Make sure that your policy meets all of the required conditions. If your policy has multiple condition operators or multiple keys, then the conditions are evaluated using AND logic. Each RequestTag key must be used in separate statements to get the same AND logic. This is an example of a common issue that causes the API call to fail:{  "Sid": "AllowRunInstancesWithRestrictions2",  "Effect": "Deny",  "Action": [    "ec2:CreateVolume",    "ec2:RunInstances"  ],  "Resource": [    "arn:aws:ec2:*:*:volume/*",    "arn:aws:ec2:*:*:instance/*"  ],  "Condition": {    "ForAllValues:StringNotLike": {      "aws:TagKeys": "Production"    },    "StringEquals": {      "ec2:InstanceType": "t2.micro"    }  }}To avoid an explicit deny access error for these API calls, make sure that the preceding condition is fulfilled.Note: The aws:TagKeys condition is case sensitive.Resource-based policiesThe resource-based policy allows or denies access to a resource. Unlike IAM identity-based policies which are unified, resource-based policies are designed by services. The following troubleshooting steps use Amazon Simple Storage Service (Amazon S3) resource-based policies and a VPC endpoint policy as examples.S3 bucket policy evaluationAmazon S3 Bucket policy evaluation works as follows:Note: This assumes that the bucket ACL is set as default.To access a bucket in the same account, an IAM entity needs permissions in the IAM identity-based the OR bucket policy.To access a bucket in a different account, an IAM entity needs permissions in the bucket policy AND the IAM identity-based to gain access.1.    Check for deny statements in the resource-based policy. This example shows a deny statement in the bucket policy:{  "Effect": "deny",  "Principal": {    "AWS": "arn:aws:iam::111111111111:role/ROLENAME"  },  "Action": "s3:ListBucket",  "Resource": "arn:aws:s3:::MyExampleBucket"}2.    Check that the ARNs described in your policy are correct.3.    The bucket policy denies access if the aws:userid of the current user does not equal what is defined in the policy. See this example:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Deny",      "Principal": "*",      "Action": "s3:*",      "Resource": [        "arn:aws:s3:::MyExampleBucket",        "arn:aws:s3:::MyExampleBucket/*"      ],      "Condition": {        "StringNotLike": {          "aws:userId": [            "AROAEXAMPLEID:*",            "AIDAEXAMPLEID",            "111111111111"          ]        }      }    }  ]}VPC endpointA VPC endpoint policy is an IAM resource policy that is attached to an endpoint. This policy doesn't override or replace the IAM user policies or service-specific policies (such as S3 bucket policies).There are two ways to control access to your Amazon S3 data when using an interface endpoint to connect to Amazon S3, :You can control the AWS principals (AWS accounts, IAM users, and IAM roles) that can use the VPC endpoint to access the endpoint service.You can control the VPCs or VPC endpoints that have access to your buckets by using Amazon S3 bucket policies.This example below is an Amazon S3 bucket policy. The policy restricts access to a specific bucket, called examplebucket, from the VPC endpoint with the ID vpce-11111. The policy denies all access to the bucket if the specified endpoint isn't used. The aws:SourceVpce condition is used to specify the endpoint.{   "Version": "2012-10-17",   "Id": "Policy123456789”,   "Statement": [     {       "Sid": "AccessSpecificVPCEOnly",       "Principal": "*",       "Action": "s3:*",       "Effect": "Deny",       "Resource": ["arn:aws:s3:::examplebucket",                    "arn:aws:s3:::examplebucket/*"],       "Condition": {         "StringNotEqualsIfExists": {           "aws:SourceVpce": "vpce-11111”         }       }     }   ]}Always check that the VPC endpoint isn't explicitly denying access to the resource.Permissions boundaryThe permissions boundary is a managed policy that sets the maximum permissions that an identity-based policy can grant to an IAM entity. This managed policy can restrict permissions to entities, which might result in explicit deny error messages.This example shows an action that is allowed in the IAM policy but is explicitly denied in the permissions boundary. See the permissions boundary below:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Deny",      "Action": "ec2:*"      "Resource": "*"    }  ]}The user has the following permissions:{  "Version": "2012-10-17",    "Statement": {    "Effect": "Allow",    "Action": "ec2:RunInstances",        "Resource": "*"  }}Although the user has the RunInstances permission, they receive an explicit deny message when they request it. To resolve this error, make sure that your permissions boundary and IAM are both explicitly allowing this action.Service control policiesA service control policy (SCP) allows you to manage permissions in your organization. The following example shows a deny statement in the SCP. In this example, the SCP is attached to a member account or to a particular Organization Unit (OU). It explicitly denies access to the RunInstances action:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Deny",      "Action": "ec2:RunInstances"      "Resource": "*"    }  ]}To resolve explicit deny errors, take one of these actions:Detach the SCP from the account.Modify the deny statement by adding a condition that excludes some use case. For example, this SCP in this example does NOT deny ec2:RunInstances if the IAM principal uses the role CloudOps:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Deny",      "Action": "ec2:RunInstances"           "Resource": "*",      "Condition": {        "ArnNotLike": {          "aws:PrincipalARN": "arn:aws:iam::*:role/CloudOps"        }      }    }  ]}Session policiesSession policies are advanced policies that you pass as a parameter when you programmatically create a temporary session for a role or user. You can create a role session, and pass session policies by using the AssumeRole, AssumeRoleWithSAML, or AssumeRoleWithWebIdentity API operations.For example this policy generates the explicit deny error when the user tries to make a RunInstances API call. Always check for deny statements in the session policy:{  "Version": "2012-10-17",    "Statement": {    "Effect": "Deny",    "Action": "ec2:RunInstances",        "Resource": "*"  }}Related informationAccess management for AWS resourcesHow to use service control policies to set permission guardrails across accounts in your AWS OrganizationPermissions boundaries for IAM entitiesHow to restrict Amazon S3 bucket access to a specific IAM roleControlling access from VPC endpoints with bucket policiesFollow"


Question: How can I troubleshoot high latency on my Application Load Balancer?

I'm experiencing high latency and timeouts when I try to access web applications running over targets registered behind an Application Load Balancer. How should I fix these issues?

Answer: "I'm experiencing high latency and timeouts when I try to access web applications running over targets registered behind an Application Load Balancer. How should I fix these issues?Short descriptionPossible causes of high latency on an Application Load Balancer include:Network connectivity issuesHigh memory (RAM) utilization on backend instancesHigh CPU utilization on backend instancesIncorrect web server configuration on backend instancesProblems with web application dependencies running on backend instances, such as external databases or Amazon Simple Storage Service (Amazon S3) bucketsResolution1.    Check for network connectivity issues using the troubleshooting steps in Troubleshoot your Application Load Balancers.2.    Use the curl tool to measure the first byte response and check if slow DNS resolution is contributing to the latency.curl -kso /dev/null https://www.example.com -w "==============\n\n | dnslookup: %{time_namelookup}\n | connect: %{time_connect}\n | appconnect: %{time_appconnect}\n | pretransfer: %{time_pretransfer}\n | starttransfer: %{time_starttransfer}\n | total: %{time_total}\n | size: %{size_download}\n | HTTPCode=%{http_code}\n\n"Example output: | dnslookup: 0.005330 | connect: 0.006682 | appconnect: 0.026540 | pretransfer: 0.026636 | starttransfer: 0.076980 | total: 0.077111 | size: 12130 | HTTPCode=200Perform these tests through the Application Load Balancer. Then, perform the tests while bypassing the Application Load Balancer to targets. This approach helps to isolate the component that's inducing latency. For more information on curl features, see How to use curl features.3.    Check the Averagestatistic of the Amazon CloudWatch TargetResponseTime metric for your Application Load Balancer. If the value is high, there's likely a problem with the backend instances or application dependency servers.4.    Determine which backend instances are experiencing high latency by checking the access log entries for your Application Load Balancer. Check target_processing_time to find backend instances with latency issues. Also, review the request_processing_time and response_processing_time fields to verify any issues with the Application Load Balancer.5.    Check the CloudWatch CPUUtilization metric of your backend instances. Look for high CPU utilization or spikes in CPU utilization. For high CPU utilization, consider upgrading your instances to a larger instance type.6.    Check for memory issues by reviewing the Apache processes on your backend instances.Example command:watch -n 1 "echo -n 'Apache Processes: ' && ps -C apache2 --no-headers | wc -l && free -m"Example output:Every 1.0s: echo –n 'Apache Processes: ' && ps –C apache2 –no-headers | wc -1 && free –mApache Processes: 27          total     used     free     shared     buffers     cachedMem:      8204      7445     758      0          385         4567-/+ buffers/cache:  2402     5801Swap:     16383     189      161947.    Check the MaxClient setting for the web servers on your backend instances. This setting defines how many simultaneous requests the instance can serve. For instances with appropriate memory and CPU utilization experiencing high latency, consider increasing the MaxClient value.Compare the number of processes generated by Apache (httpd) with the MaxClient setting. If the number of Apache processes frequently reaches the MaxClient value, consider increasing the value.[root@ip-192.0.2.0 conf]# ps aux | grep httpd | wc -l 15<IfModule prefork.c>StartServers         10MinSpareServers      5MaxSpareServers      10ServerLimit          15MaxClients           15MaxRequestsPerChild  4000</IfModule>8.    Check for dependencies of your backend instances that might be causing latency issues. Dependencies might include shared databases or external resources (such as Amazon S3 buckets). Dependencies might also include external resource connections, such as network address translation (NAT) instances, remote web services, or proxy servers.9.    Use the following Linux tools to identify performance bottlenecks on the server.uptime – Shows load averages to help determine the number of tasks (processes) waiting to run. On Linux systems, this number includes processes waiting to run on the CPU, as well as processes blocked in uninterruptible I/O (usually disk I/O). This data provides a high-level look at resource load (or demand) that must be interpreted using other tools. When Linux load averages increase, there's a higher demand for resources. To determine which resources are in higher demand, you must use other metrics. For example, for CPUs you can use mpstat -P ALL 1 to measure per-CPU utilization, or top or pidstat 1 to measure per-process CPU utilization.mpstat -P ALL 1 – Shows CPU time breakdowns per CPU, which you can use to check for an imbalance. A single hot CPU might be evidence of a single-threaded application.pidstat 1 – Shows per-process CPU utilization and prints a rolling summary that's useful for watching patterns over time.dmesg | tail – Shows the last 10 system messages, if there are any. Look for errors that might cause performance issues.iostat -xz 1 – Shows the workload applied for block devices (disks) and the resulting performance.free -m – Shows the amount of free memory. Check that these numbers aren’t near-zero in size, which can lead to higher disk I/O (confirm using iostat), and decreased performance.sar -n DEV 1 – Shows network interface throughput (rxkB/s and txkB/s) as a measure of workload. Check if any limits have been reached.sar -n TCP,ETCP 1 – Shows key TCP metrics, including: active/s (number of locally-initiated TCP connections per second), passive/s (number of remotely-initiated TCP connections per second), and retrans/s (number of TCP retransmits per second).iftop – Shows the connections between your server and a remote IP address that are consuming the most bandwidth. n iftop is available in a package with the same name on Red Hat and Debian-based distributions. However, with Red Hat-based distributions, you might instead find n iftop in a third-party repository.Follow"


Question: How do I delete a Route 53 hosted zone created by service discovery?

"I'm trying to delete an Amazon Route 53 (Route 53) hosted zone created by service discovery in the AWS console. However, I'm receiving an error. How do I delete a Route 53 hosted zone created by service discovery?"

Answer: "I'm trying to delete an Amazon Route 53 (Route 53) hosted zone created by service discovery in the AWS console. However, I'm receiving an error. How do I delete a Route 53 hosted zone created by service discovery?Short descriptionHosted zones that were created by service discovery can't be deleted from the Route 53 console. Hosted zones created by service discovery can be deleted only by using either the AWS Cloud Map console or a servicediscovery API call. Remember:Before you can delete a namespace, you must delete all services that were created in the namespace.Before you can delete a service, you must deregister all instances registered that were using the service.ResolutionFind the namespace in AWS Cloud MapOpen the AWS Cloud Map console.Choose the Region where the namespace was created from the Region selector in the navigation bar.In the navigation pane, choose Namespaces.Select the namespace associated with the hosted zone that you want to delete.Remove all associated servicesSelect a service name from the Services section.Select each instance in the Service instances list, and then choose Deregister.After you've deregistered all instances from the service, choose Delete at the top right of the service page to delete the service.Repeat these steps for each associated service until all services are deleted.Delete the namespaceAfter all services are deleted, choose Delete at the top right of the namespace page to delete the namespace.Choose Delete in the confirmation window.When you delete the namespace, the associated Route 53 hosted zone is automatically deleted.Follow"


Question: How do I link an Amazon EMR notebook to a Git repository?

I want to link my Amazon EMR notebook with a Git repository.

Answer: "I want to link my Amazon EMR notebook with a Git repository.ResolutionAssociating Git repositories with Amazon EMR notebooks allows you to save your notebooks in a version-controlled environment. You can associate up to three repositories with a notebook.To create a new EMR notebook and then associate it with an existing Git repository, do the following:1.    Create a private subnet in a virtual private cloud (VPC).2.    Create a NAT gateway.3.    Update the route table to point to the NAT gateway.4.    Launch an Amazon EMR cluster in the private subnet. In the Software configuration section, be sure that you select a configuration that includes Apache Spark, Apache Hadoop, and Apache Livy.5.    When you're waiting for the EMR cluster to reach the WAITING state, add the Git repository. For Git credentials, choose Create a new secret. Be sure that the Username is the alias of the Git account and not the email address. For more information, see Working with aliases.6.    Create a security group with the following outbound rules:Rule 1Type: Custom TCP ruleProtocol: TCPPort Range: 18888Destination: ElasticMapReduceEditors-LivyRule 2Type: HTTPSProtocol: TCPPort Range: 443Destination: 0.0.0.0/0This allows the notebook to reach the internet using the cluster. For more information, see Custom EC2 security group for EMR notebooks when associating notebooks with Git repositories.7.    Add an inbound rule to the ElasticMapReduceEditors-Livy security group:Type: Custom TCP ruleProtocol: TCPPort Range: 18888Destination: Enter the name of the security group that you created in the previous step.8.    Modify the service role for EMR notebooks (EMR_Notebooks_DefaultRole) to allow the secretsmanager:GetSecretValue action.9.    Create an EMR notebook with the following security group settings:In the Security groups section, select Choose security groups.For Security groups for master instance, choose ElasticMapReduceEditors-Livy.For Security groups for notebook instance, choose the security group that you created in step 6.The Git repository status changes to Linked. You can now use the Git repository in the notebook.Related informationAssociating Git-based repositories with EMR notebooksEMR notebooksFollow"


Question: What are the differences between working with custom images and BYOL images in WorkSpaces?

I need to know the differences between working with custom images and BYOL images in Amazon WorkSpaces for Windows. I also need to know the BYOL setup requirements.

Answer: "I need to know the differences between working with custom images and BYOL images in Amazon WorkSpaces for Windows. I also need to know the BYOL setup requirements.ResolutionStandard Amazon WorkSpaces for Windows are based on Windows Server 2016 or 2019 with Desktop Experience. At the kernel level, a Windows Server operating system and a Windows 10 operating system are similar. Yet, you might need a desktop operating system to perform certain tasks. This is when you want to consider bringing your own Windows desktop license, also known as BYOL. BYOL WorkSpaces are available only for Windows.BYOL WorkSpaces image managementBYOL WorkSpaces require dedicated resources for the underlying infrastructure to meet licensing requirements. The images created from template WorkSpaces aren't much different than custom images created with Amazon-provided resources and BYOL images.BYOL images do have a few limitations:You can copy BYOL image to another AWS Region as long as the destination Region is turned on for BYOL.You can share BYOL images with AWS accounts that are turned on for BYOL and are part of your organization with AWS Organizations.You can't launch a BYOL WorkSpace to a directory that doesn’t have BYOL turned on.Setting up BYOL environmentsA BYOL WorkSpace environment must meet several requirements to be successful. The following is a short list of those BYOL requirements:Infrastructure requirements:Your Microsoft licensing agreement must allow Windows to run in a virtual hosted environment.You must commit to running a minimum of 100 WorkSpaces that are non-GPU-enabled, 4 always-on GPU-enabled, or 20 auto-stop GPU-enabled.Your network must not overlap with the management network IP ranges.You need to allow documented management interface port traffic.AWS service requirements:You must have your AWS account turned on for BYOL.You must have a directory service associated with WorkSpaces and BYOL.VM import and operating system configuration:You must have a virtual machine (VM) that runs a supported 64-bit version of Windows and is included in supported operating system versions.You must import your VM to create an AMI for your BYOL WorkSpace.Your VM must meet operating system configuration requirements and only includes software that’s a part of a typical Windows installation.Your key management server must be accessible from WorkSpaces over the customer network.In comparison, for standard WorkSpaces, the requirements general list is shorter:Your network must not overlap with the management network IP ranges.You must allow documented management interface port traffic.You must have directory service associated with WorkSpaces.Related informationCopy a custom WorkSpaces imageShare or unshare a custom WorkSpaces imageBring Your Own Windows desktop licensesManagement interface portsFollow"


Question: How can I pass a connectionId or custom token to VPC Link integration as a header for Amazon API Gateway WebSocket APIs?

I want to pass a connectionId or custom token to VPC Link integration as a header for my Amazon API Gateway WebSocket API. How can I do this?

Answer: "I want to pass a connectionId or custom token to VPC Link integration as a header for my Amazon API Gateway WebSocket API. How can I do this?Short descriptionFor WebSocket APIs, the connectionId is required to send a callback response from the backend. By default, WebSocket APIs don't pass the ConnectionId to VPC link integration.ResolutionUsing the Amazon API Gateway console and AWS CLINote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.1.    Follow the instructions to create a REST API VPC link and associate it with your Network Load Balancer.2.    Follow the instructions to set up a WebSocket API integration and create a VPC link private integration endpoint.3.    To retrieve the integration ID, run the AWS CLI command get-integrations similar to the following:aws apigatewayv2 get-integrations --api-id <api-id>4.    Create and save a JSON file named integration.json in the following format:{    "ApiId": "<api-id>",    "IntegrationId": "<integration id retrieved from previous step>",    "RequestParameters": {        "integration.request.header.ConnectionId": "context.connectionId", //passing context variable connectionId as ConnectionId header to backend        "integration.request.header.<header-key>": "route.request.body.<parameter>", // passing a request body parameter as header to backend        "integration.request.querystring.<querysting-key>": "'static value'" //passing static value as querystring to backend    }}Note: Before proceeding to step 5, remove the comments noted with the forward slashes "//".5.    To update the integration, run the AWS CLI command update-integration similar to the following:aws apigatewayv2 update-integration --cli-input-json file://integration.json6.    To apply these changes, follow the instructions todeploy the REST API.Using AWS CloudFormationIf you manage Amazon API Gateway using CloudFormation, you can use the AWS::ApiGatewayV2::Integration resource to configure request parameters similar to the following:Integration:    Type: AWS::ApiGatewayV2::Integration    Properties:      ApiId: !Ref websocket      ConnectionId: !Ref <VPC-Link-connection-id>      ConnectionType: VPC_LINK      IntegrationMethod: <HTTP method to backend >      IntegrationType: <HTTP_PROXY/HTTP>      IntegrationUri: "<endpoint-url>"      RequestParameters:        "integration.request.header.connectionId": "context.connectionId" #passing context variable 'connectionId' as ConnectionId header to backend        "integration.request.header.<header-key>": "route.request.body.<parameter>"  #passing a request body parameter as header to backend        "integration.request.querystring.<querystring-key>": "'static value'" #passing static value as querystring to backendYou can use data mapping to map data from a route request to the VPC link backend integration. For more information, see Map route request data to integration request parameters.Related informationSetting up WebSocket API integrationsFollow"


Question: How can I advertise VPC routes over a Direct Connect connection to a on-premises network over BGP?

I want to advertise Virtual Private Cloud (Amazon VPC) routes over an AWS Direct Connect VIF BGP session to an on-premises network.

Answer: "I want to advertise Virtual Private Cloud (Amazon VPC) routes over an AWS Direct Connect VIF BGP session to an on-premises network.Short descriptionThe routes AWS advertised to an on-premises network over a Direct Connect Border Gateway Protocol (BGP) session depends on these connection types:Direct Connect private VIF connected to a virtual private gateway (VGW)Direct Connect private VIF connected to a Direct Connect gateway associated with a VGWDirect Connect transit VIF connected to a Direct Connect gateway associated with a transit gatewayResolutionThe Direct Connect on-premises network advertises the routes manually through BGP or through redistribution into BGP. The routes AWS advertises back to on-premises change depending on the type of gateways.Direct Connect private VIF connecting to a VGWThe VGW associated VPC's IPv4/IPv6 CIDR are advertised automatically to an on-premises BGP peer. For example, a VPC with CIDR 10.55.0.0/16 VGW is associated directly with a private VIF. The prefix 10.55.0.0/16 is advertised to on-premises automatically. If there are additional CIDRs associated with the VPC, then those prefixes are advertised over to the BGP peer.Direct Connect Private VIF connected to a Direct Connect gateway associated with VGWYou can have up to 10 VGWs associated with a Direct Connect gateway. All VPC CIDR prefixes are advertised to the on-premises BGP Peer. The allowed prefixes list filters BGP advertisements from AWS towards the on-premises BGP peer.The allowed prefix list allows the same CIDRs or a smaller subnet of the CIDRs to advertise to the Direct Connect gateway.In the following example, VPC-A CIDR 10.77.0.0/16, VPC-B CIDR 10.66.0.0/16 and VPC-C 192.168.0.0/16 are attached to a Direct Connect gatewayIf the allowed prefixes list is set to allow only 10.0.0.0/8, the prefixes 10.77.0.0/16 and 10.66.0.0/16 are received on the on-premises BGP peer. This is because the prefixes are subnets of the allowed prefix list, but 192.168.0.0/16 isn't received on the on-premises BGP peer.If the allowed prefixes list is set to allow 10.0.0.0/8 and 192.168.5.0/24, then the prefixes 10.77.0.0/16 and 10.66.0.0/16 are received on the on-premises BGP peer. This is because the prefixes are subnets of the allowed prefix list, but 192.168.0.0/16 isn't received on the on-premises BGP peer because that IP range doesn't match the allow list.Direct Connect transit VIF connecting to a Direct Connect gateway associated with a transit gatewayYou can connect one Direct Connect gateway to up to three transit gateways. Hundreds of VPCs can send traffic across the transit gateway and through the Direct Connect connection. The on-premises network must have the routes for all individual VPCs or use a summarized route. Routes advertised from the transit gateway towards on-premises with Direct Connect is defined in the allowed prefixes.All prefixes are advertised to the on-premises BGP peer. The allowed prefixes list advertises from the transit gateway to the on-premises Direct Connect peer. You can advertise a route for any IP address like 8.8.8.8/32 even though it's not a VPC CIDR connected to the transit gateway.The allowed prefix list for the transit gateway has a 20 prefix limit. In the following example, VPC-A CIDR 10.77.0.0/16, VPC-B CIDR 10.66.0.0/16 and VPC-C 192.168.0.0/16 are attached to a transit gateway that connects to a Direct Connect gateway. If the allowed prefixes list is set to allow 10.0.0.0/8 and 192.168.5.0/24, you won't receive the three VPC CIDR prefixes on the on-premises network. Instead, you receive the prefixes 10.0.0.0/8 and 192.168.5.0/24 advertised over the BGP.If the allowed prefixes list is set to allow 10.0.0.0/8 and 192.168.0.0/16, you receive the prefixes 10.0.0.0/8 and 192.168.0.0/16 advertised over the BGP.If the allowed prefixes list is set to allow only 0.0.0.0/0, you only receive the default route 0.0.0.0/0 advertised over the BGP.Changes to allowed prefixes on a VGW or a transit gateway association with a Direct Connect gateway are updated for the routes and don't bring down the BGP session.Note: Changes made to the allowed prefixes list can take several minutes to propagate.Related informationAllowed prefixes interactionsDirect Connect quotasFollow"


Question: How do I allow AWS accounts in my organization to publish messages to an Amazon SNS topic in my account?

I want an Amazon Simple Notification Service (Amazon SNS) topic to accept messages from any AWS account in my organization in AWS Organizations. How do I set that up?

Answer: "I want an Amazon Simple Notification Service (Amazon SNS) topic to accept messages from any AWS account in my organization in AWS Organizations. How do I set that up?Short descriptionConfigure the Amazon SNS topic's access policy to allow any account in your organization to publish messages to the topic. In the access policy, include the global condition key, aws:PrincipalOrgID, and specify your organization's ID.Resolution1.    Find your organization's ID in the Organizations console. For more information, see Viewing the details of an organization from the management account.2.    Create a topic in the Amazon SNS console. Note the Amazon Resource Name (ARN) of your new topic.3.    In the Amazon SNS console, edit the topic by doing the following:In the navigation pane, choose Topics.Choose the topic that you created. Then, choose Edit.On the Edit  page, expand Access policy -optional.Paste the following example policy into the JSON editor, and then choose Save changes:Important: Replace snsTopicArn with the topic's ARN. Then, replace myOrgId with your organization's ID.{  "Version": "2012-10-17",  "Id": "__default_policy_ID",  "Statement": [    {      "Sid": "allow-publish-from-organization-accounts",      "Effect": "Allow",      "Principal": {        "AWS": "*"      },      "Action": [        "sns:Publish"      ],      "Resource": "snsTopicArn",      "Condition": {        "StringEquals": {          "aws:PrincipalOrgID": "myOrgId"        }      }    }  ]}Tip: To allow accounts in your organization to perform more Amazon SNS API actions (such as GetTopicAttributes), add actions under "Action" in the policy.4.    Subscribe your email address to the SNS topic for testing. When you create the subscription, make sure that you specify your topic's ARN.5.    In your email, find the subscription confirmation message from AWS Notifications and confirm the subscription.6.    Publish a message to the SNS topic in your account using any AWS account in your organization. In the publish request, make sure that you specify the topic's ARN.The published message appears in your email.Related informationAWS global condition context keysExample cases for Amazon SNS access controlOverview of managing access in Amazon SNSAWS services that you can use with AWS OrganizationsFollow"


Question: How do I turn on MFA delete for my Amazon S3 bucket?

I want to turn on multi-factor authentication (MFA) delete for my Amazon Simple Storage Service (Amazon S3) bucket to protect my objects from unintended deletions.

Answer: "I want to turn on multi-factor authentication (MFA) delete for my Amazon Simple Storage Service (Amazon S3) bucket to protect my objects from unintended deletions.Short descriptionWith Amazon S3 MFA delete, you can prevent accidental object deletions. If you don't turn on MFA delete, then anyone with either of the following permissions can permanently delete an Amazon S3 object:Root user passwordCredentials of a sufficiently privileged AWS Identity and Access Management (IAM) user or role After you turn on MFA delete, only the root user can permanently delete object versions or change the versioning configuration on your S3 bucket. The root user must be authenticated with an MFA device to perform this action.Note: MFA delete is supported only on buckets where versioning is turned on. The bucket owner, AWS account that created the bucket, and all authorized IAM users can turn on versioning. However, only the bucket owner of the root account can turn on MFA delete. To turn on MFA delete for your bucket, complete the following steps:Generate an access key and secret key for the root user.Activate an MFA delete device for the root user.Configure the AWS Command Line Interface (AWS CLI) with root user credentials.Use the PutBucketVersioning API to turn on the MFA delete feature.Confirm that MFA delete is working. Remember to delete your root access keys.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.ResolutionGenerate an access key and secret key for the root userFollow the instructions in Creating access keys for the root user.After you create these keys, you get a warning that explains that you have only this one opportunity to view or download the keys. You can't retrieve these keys later. Therefore, make sure to save them for configuring the AWS CLI.Activate an MFA device for the root userIf you don't have an MFA device that's activated for the root user, then follow the instructions in Enable a virtual MFA device for your AWS account root user (console).If you already have an MFA device that's activated for the root user, then note the ARN.Configure the AWS CLI with the root credentialsRun the AWS CLI configure command.When prompted for the AWS Access Key ID (Example: AKIAEXAMPLEABCQWE), paste the access key ID of the root user that you downloaded in step 1.When prompted for the AWS Secret Access Key of the root user, paste the secret access key ID. You can find this information in the file that contains the root user’s credentials.(Optional) When promoted for the Default region name, you can skip and press Enter.(Optional) When prompted for the Default output format, you can skip and press Enter.Important: If you configured named profiles on the AWS CLI, then you must create another profile for the root user's credentials.To configure a named profile, run the following command:aws configure --profile root_userUse the PutBucketVersioning API to turn on the S3 MFA delete featureTo turn on MFA delete, run the put-bucket-versioning command:aws s3api put-bucket-versioning --bucket mybucketname --versioning-configuration MFADelete=Enabled,Status=Enabled --mfa "arn:aws:iam::(accountnumber):mfa/root-account-mfa-device (pass)"Example:aws s3api put-bucket-versioning --bucket mybucketname --versioning-configuration MFADelete=Enabled,Status=Enabled --mfa "arn:aws:iam::1XXXXXXX6789:mfa/root-account-mfa-device 123789"In this example, 123789 is the six digit example code that's generated with the MFA device.If you used a named profile for the root user, then run the following command:aws s3api put-bucket-versioning --bucket mybucketname --versioning-configuration MFADelete=Enabled,Status=Enabled --mfa "arn:aws:iam::1XXXXXXX6789:mfa/root-account-mfa-device 123789" --profile root_userIf the command is successful, then you don't get an output. If you get an error, then make sure that you installed the latest version of the AWS CLI. Also, confirm that you're using the root user and that the MFA code and ARN are valid.Confirm that MFA delete is workingCheck the Amazon S3 console to make sure that Versioning is turned on for the bucket.To check that MFA delete is turned on, use the GetBucketVersioning API:aws s3api get-bucket-versioning --bucket mybucketname  {      "Status": "Enabled",      "MFADelete": "Enabled"  }After you turn on MFA delete for the bucket, you must include the request header in all future object deletes to permanently delete an object version. The header's value is the chain of your authentication device's serial number, a space, and the authentication code displayed on it. For more information, see Deleting an object from an MFA delete-enabled bucket. You can use the --mfa option in the delete-object command to include the header value.If you try to delete an object version in your bucket without MFA, then you get the following error. You also get this error when you use an IAM user to try to delete the object version:aws s3api delete-object --bucket mybucketname --key myobjectkey --version-id 3HL4kqCxf3vjVBH40Nrjkd  An error occurred (AccessDenied) when calling the DeleteObject operation: Mfa Authentication must be used for this requestTo use the root user to delete an object version in a bucket with MFA delete turned on, run the following command:aws s3api delete-object --bucket mybucketnme --key myobjectkey --version-id 3HLkqCxf3vjVBH40Nrjkd --mfa "arn:aws:iam::(accountnumber):mfa/root-account-mfa-device (pass)"  {      "VersionId": "3HLkqCxf3vjVBH40Nrjkd"  }Note: IAM users or roles within your AWS account that have s3:DeleteObject permission can still make a successful delete-object request to objects within your bucket without specifying a version ID. In a bucket with versioning, this request creates a delete marker. The current version of the object persists as a previous version. The only way to permanently delete an object in these buckets is to specify the version ID of the object version in the delete-object request. With MFA delete turned on, only the root user can make a version-aware delete to objects in the bucket. Also, the root user must authenticate the request with the root user MFA device.As an security best practice, after you turn on MFA delete, do the following:Invalidate and remove the root user credentials that are stored on your AWS CLI. AWS CLI credentials are stored in the configuration folder for your operating system. For more information, see Configuration and credential file settings.Use the AWS Management Console to delete the root user access keys. For instructions, see Deleting access keys for the root user.Related informationConfiguring MFA deleteFollow"


Question: How can I configure an EventBridge rule for GuardDuty to send custom SNS notifications if specific AWS service event types trigger?

"I created an Amazon EventBridge rule to trigger on service event types for Amazon GuardDuty, but the responses are in JSON format. How can I receive an email response with a custom notification?"

Answer: "I created an Amazon EventBridge rule to trigger on service event types for Amazon GuardDuty, but the responses are in JSON format. How can I receive an email response with a custom notification?Short descriptionUse a custom event pattern with the EventBridge rule to match a specific finding type. Then, route the response to an Amazon Simple Notification Service (Amazon SNS) topic.ResolutionThis example uses an Amazon GuardDuty event type UnauthorizedAccess:EC2/MaliciousIPCaller.Custom.Note: You can replace the service name and event type for your specific AWS service.1.    If you haven't already created an Amazon SNS topic, follow the instructions for Getting started with Amazon SNS.Note: The Amazon SNS topic must be in the same Region as your Amazon GuardDuty service.2.    Open the EventBridge console.3.    Select Create rule.4.    Enter a Name for your rule. You can optionally enter a Description.5.    Select the bus that the event applies to.6.    In Rule type, select Rule with an event pattern. Then, select Next.7.    Under Event pattern, choose AWS services for the Event source. Then, choose GuardDuty for the AWS service.8.    For Event type, choose GuardDuty Finding.9.    In the Event pattern preview section, select Edit pattern.10.    Copy the following code, paste it in Event pattern preview section, and then choose Save.{  "source": [    "aws.guardduty"  ],  "detail": {    "type": [      "UnauthorizedAccess:EC2/MaliciousIPCaller.Custom"     ]  }}11.    Select Next.12.    For Target types, select AWS service.13.    For Select a target, choose SNS topic. Then, select your topic from the drop-down menu.14.    Select Next.        (Optional) Add tags to your rule, and then select Next.15.    Review the rule's details, and then select Create rule.16.    Select Create at the bottom of the page.17.    If an event type is triggered, then you receive an SNS notification on the SNS endpoint.Related informationCreating Amazon EventBridge rules that react to eventsTutorial: use input transformer to customize what EventBridge passes to the event targetHow can I troubleshoot GuardDuty custom Amazon SNS notifications that are not being delivered?Follow"


Question: How do I install PECL 7 modules on Elastic Beanstalk environments running on PHP 8.1 with Amazon Linux 2 stacks?

"I want to install Redis, Memcached, or other modules for PECL 7 on an AWS Elastic Beanstalk environment running on PHP 8.1 using Amazon Linux 2 stacks."

Answer: "I want to install Redis, Memcached, or other modules for PECL 7 on an AWS Elastic Beanstalk environment running on PHP 8.1 using Amazon Linux 2 stacks.Short descriptionTo install PECL 7 modules, you must create a configuration file (.ebextension) that includes commands to:Install dependenciesInstall the PECL moduleCreate or modify configuration filesNote: For Amazon Linux 1 stacks, see How do I install PECL 7 modules on Elastic Beanstalk environments running on PHP with Amazon Linux 1 stacks?Resolution1.    Set up the .ebextensions directory in the root of the source bundle of your application. For example:|-- .ebextensions|        |-- redis/memcached.config|        |-- other .config files|-- other application filesFor more information, see Advanced environment customization with configuration files (.ebextensions).Note: All configuration files are stored in the .ebextensions folder. The Elastic Beanstalk environment uses only files that end with the .config extension.2.    In the .ebextensions directory, create a configuration file that includes the following commands for your module.For Redis modules:commands:  01_install_redis:    command: /usr/bin/yes 'no' | /usr/bin/pecl install redis    test: '! /usr/bin/pecl info redis'  02_remove_extension:    command: /bin/sed -i -e '/extension="redis.so"/d' /etc/php.ini  03_create_conf:    command: /bin/echo 'extension="redis.so"' > /etc/php.d/41-redis.iniThe Redis configuration file does the following:Installs the Redis module and tests for installation success (01_install_redis)Removes the Redis extension from the /etc/php.ini file (02_remove_extension)Creates a configuration file called 41-redis.ini at /etc/php.d/ (03_create_conf)For Memcached modules:packages:  yum:    libmemcached-devel: []commands:  01_install_memcached:    command: /usr/bin/yes 'no'| /usr/bin/pecl install memcached    test: '! /usr/bin/pecl info memcached'  02_remove_extension:    command: /bin/sed -i -e '/extension="memcached.so"/d' /etc/php.ini  03_create_conf:    command: /bin/echo 'extension="memcached.so"' > /etc/php.d/41-memcached.iniThe Memcached configuration file does the following:Installs the Memcached module and tests for installation success (01_install_memcached)Removes the Redis extension from the /etc/php.ini file (02_remove_extension)Creates a configuration file called 41-memcached.ini at /etc/php.d/ (03_create_conf)Note: Always test a new .ebextension before adding to a production environment.Follow"


Question: How do I troubleshoot scaling issues with my Amazon ECS capacity provider?

"I have set up a capacity provider for my Amazon Elastic Container Service (Amazon ECS) Cluster. However, the capacity provider doesn’t scale out when the cluster runs out of resources or scale in when the capacity is less occupied."

Answer: "I have set up a capacity provider for my Amazon Elastic Container Service (Amazon ECS) Cluster. However, the capacity provider doesn’t scale out when the cluster runs out of resources or scale in when the capacity is less occupied.Short descriptionThe capacity provider for your Amazon ECS cluster doesn't automatically scale in or scale out due to one or more of the following reasons:The Amazon ECS service isn't associated with the capacity provider.The scaling policies related to the capacity provider aren't attached to the Auto Scaling group.The target tracking scaling policies aren't configured correctly.The target capacity percentage isn't configured in the capacity provider correctly.The task placement strategy isn't defined according to the workload.The ECS Service is failing with some errors and blocking the capacity provider from scaling.You're using managed scaling for the capacity provider, and the Auto Scaling group has custom scaling policies attached to it.The Auto Scaling group has launched the container instance, but is unable to join the cluster.Your container instances are protected from scaling in.The capacity provider is stuck in failed state.The Auto Scaling group is stuck in a loop of scaling out and scaling in.ResolutionThe Amazon ECS service isn't associated with the capacity providerTo check if the ECS service is associated with the capacity provider, run the AWS Command Line Interface (AWS CLI) command describe-services.aws ecs describe-services --cluster example-cluster --services example-service --region example-region --query services[].capacityProviderStrategyIf your ECS service is associated with the capacity provider, then the output must look similar to the following:[  [    {      "capacityProvider": "example-capacity-provider",      "weight": 1,      "base": 1    }  ]]Be sure that the capacityProviderStrategy field is not null in the output. You can view the configuration of the service by reviewing AWS CloudTrail events for CreateService and UpdateService API calls.To resolve this issue, update the ECS service using the AWS CLI commands update-service, run-task, or put-cluster-capacity-providers. You can also update the service using the Amazon ECS console.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.The scaling policies related to the capacity provider aren't attached to the Auto Scaling groupWhen a capacity provider is created and associated with an Auto Scaling group, the Auto Scaling group creates a scaling policy that uses target tracking to modify the desired capacity for accommodating cluster loads.To troubleshoot this issue, review CloudTrail events for UpdateAutoScalingGroup, CreateCapacityProvider, UpdateCapacityProvider, and PutScalingPolicy APIs.Verify that the Auto Scaling group is created as a cluster attachment by running the following command:aws ecs describe-clusters --clusters example-cluster --include ATTACHMENTS --region example-region --query clusters[].attachments[]The output of the command must look similar to the following:[  {    "id": "100a23456-5f0b-4abc-b998-d6789d111a",    "type": "asp",    "status": "CREATED",    "details": [      {        "name": "capacityProviderName",        "value": "example-capacityProvider"      },      {        "name": "scalingPlanName",        "value": "ECSManagedAutoScalingPlan-bb60c8fa-3ed7-4808-b39c-abcdef2345"      }    ]  }]If you're using a managed scaling policy, then check whether the policy is attached to the Auto Scaling group by doing the following:Open the Amazon ECS console.In the navigation pane, choose Clusters.Open the cluster that you want to check.Choose the Capacity Providers tab.For the capacity provider that you want to check, choose the ASG.You are directed to the Auto Scaling groups page in the Amazon EC2 console.Choose the Automatic Scaling tab.You can view the scaling policies.Check whether the scaling policy that you're using is included.Also, be sure to include the prefix AutoScaling-ECSManagedAutoScalingPlan to the name of the Auto Scaling group scaling policy. Otherwise, the Auto Scaling group uses a scaling policy that's different from the one managed by the capacity provider. Note that capacity providers can be used along with other types of scaling policies. For more information, see Service auto scaling.The target tracking scaling policies aren't configured correctlyA target tracking scaling policy tracks a target value for the metric you define. Amazon ECS service auto scaling creates and manages the Amazon CloudWatch alarms that trigger the scaling policy and calculates the scaling adjustment based on the metric and target value. If the target tracking policy isn't configured correctly, tasks might not automatically scale as required.Suppose that the target tracking auto scaling policy is tracking the CPUUtilization metric in CloudWatch, and you specify a target tracking percentage of 60. In this case, the capacity provider works on a best effort basis to keep the aggregate CPU utilization at 60%. This results in a scale out event when the CPU utilization is greater than 60% and a scale in event when the utilization is less than 60%.To resolve this issue, choose the right metric and set the correct scale-in and scale-out values in the target tracking policy based on your workload. For more information, see Target tracking scaling policies.The target capacity percentage isn't configured in the capacity provider correctlyThe target capacity value is used as the target value for the CloudWatch metric that's used in the Amazon ECS-managed target tracking scaling policy. This target capacity value is matched on a best effort basis. The allowed values for this value are integers between 1 and 100. For example, if you set the target capacity to 100%, all instances are utilized and any instances that are not running tasks are scaled in. However, this behavior is not guaranteed at all times. If you need spare capacity, set the target capacity to a value that's slightly lower than 100% based on your requirement.To update the capacity provider with the correct target capacity percentage, follow the instructions in Updating an Auto Scaling group capacity provider using the classic console.The task placement strategy isn't defined according to the workloadTask placement strategies can be specified when you create a service or run a task. You can also update the task placement strategies for existing services. For example, if your workload is memory-intensive and you didn't configure the task placement strategy accordingly, the tasks don't scale in or out based on your memory usage. Be sure to check the task placement strategy types and define these strategies according to your workload.The ECS Service is failing with some errors and blocking the capacity provider from scalingIf your ECS service fails with any errors, then the capacity provider is blocked from scaling in and scaling out. To troubleshoot why the ECS service failed, check the service event messages in the Amazon ECS console.You're using managed scaling for the capacity provider, and the Auto Scaling group has custom scaling policies attached to itWhen your cluster doesn't automatically scale, you might get the following error:"StatusCode": "ActiveWithProblems""StatusMessage": "Scaling plan has been created but failed to be applied to all resources. Problems were encountered for 1 resource. See scaling plan resources for the failure details."This error occurs when both the following conditions are true:You're using the AWS managed scaling for the capacity provider.The Auto Scaling group has custom scaling policies that are not created by Amazon ECS attached.To resolve this error, see Avoiding the ActiveWithProblems error. When you enable managed scaling, Amazon ECS manages the scale-in and scale-out actions of the Auto Scaling group with Auto Scaling scaling plans. It's best practice to always create a new Auto Scaling group and attach this group to the capacity provider.The Auto Scaling group has launched the container instance, but is unable to join the clusterTo check the possible causes for this issue, see Cluster auto scaling considerations.Review CloudTrail events for DescribeContainerInstances API calls to check whether a container instance is associated with the capacity provider.Be sure that the Maximum value for the Auto Scaling group is set to a value greater than zero.Your container instances are protected from scaling inIf you enabled managed termination protection when you configured the capacity provider, Amazon ECS prevents the Amazon EC2 instances in an Auto Scaling group that contain tasks from being terminated during a scale-in action.To make sure that the Auto Scaling group can terminate old instances when you change the desired capacity, do the following:Turn off managed termination protection for the capacity provider.Turn off instance scale-in protection for the Auto Scaling group.For more information, see How do I resolve the error "The managed termination protection setting for the capacity provider is invalid" in Amazon ECS?The capacity provider is stuck in failed stateIt's a best practice to create a new Auto Scaling group to use with your capacity provider instead of using an existing group. If you use an existing Auto Scaling group, you might have issues using the capacity provider. This is because the Amazon EC2 instances in the running state that are associated with the existing group and registered to an Amazon ECS cluster might not be properly registered with the capacity provider.To see the status of the capacity provider, run the AWS CLI command describe-capacity-providers.Also, review CloudTrail events, and check for errors related to the CreateCapacityProvider API.The Auto Scaling group is stuck in a loop of scaling out and scaling inWhen the metric value that's specified in the scaling policy for your ECS service spikes, the Auto Scaling group scales out and launches instances as required. However, if the value of the metric drops after the sudden spike, the Auto Scaling group tries to scale in the instances. If the metric value fluctuates several times within a short time frame, then the Auto Scaling group might get stuck in a loop of scaling out and scaling in. To avoid this issue, be sure to define the threshold value of the metric in the scaling policy according to your workload.Related informationDeep dive on Amazon ECS cluster auto scalingHow do I resolve capacity provider errors in Amazon ECS?AWS Fargate capacity providersAuto Scaling group capacity providersFollow"