Question: Why is my Amazon EC2 instance exceeding its network limits when average utilization is low?

Why is my Amazon Elastic Compute Cloud (Amazon EC2) instance exceeding its network limits when average utilization is low?

Answer: "Why is my Amazon Elastic Compute Cloud (Amazon EC2) instance exceeding its network limits when average utilization is low?Short descriptionYou can query network performance metrics in real time on instances that support enhanced networking through the Elastic Network Adapter (ENA). These metrics provide a cumulative number of packets queued or dropped on each network interface since the last device reset. The following are some of the ENA metrics:bw_in_allowance_exceeded: The number of packets queued or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance.bw_out_allowance_exceeded: The number of packets queued or dropped because the outbound aggregate bandwidth exceeded the maximum for the instance.pps_allowance_exceeded: The number of packets queued or dropped because the bidirectional Packets Per Second (PPS) exceeded the maximum for the instance.Network performance specifications vary according to the instance type. To view specifications, see Current generation instances. In some cases, you might see queuing or drops even though your average bandwidth or PPS as seen in Amazon CloudWatch is low. For example, the NetworkIn, NetworkOut, NetworkPacketsIn, or NetworkPacketsOut metrics in CloudWatch might show amounts that don't suggest a limit being reached.ResolutionMicrobursts are the most common cause of the preceding symptoms. Microbursts are short spikes in demand followed by periods of low or no activity. These bursts typically last only for seconds, milliseconds, or even microseconds. In the case of microbursts, the CloudWatch metrics listed in the previous section aren't granular enough to reflect them.How averages are calculatedThe EC2 metrics in CloudWatch listed in the previous section are sampled every 1 minute. These metrics capture the total bytes or packets transferred in that period. These samples are then aggregated and published to CloudWatch in 5-minute periods. Each statistic in the period returns a different sample value:Sum is the sum of all sample values.SampleCount is the number of aggregated samples (in this case, 5).Minimum is the sample value with the lowest byte / packet count.Average is the average sample value, calculated by dividing Sum by SampleCount.Maximum is the sample value with the highest byte / packet count.Average throughput or PPS can be calculated in two ways:Divide Sum by Period (for example, 300 seconds), for a simple 5-minute average.Divide Maximum by 60 seconds, for an average in the minute with the highest activity.How microbursts are reflected in CloudWatch metricsThe following is an example of microburst, and how it is reflected in CloudWatch:The instance has a network bandwidth performance of 10 Gbps (1.25 GB/s).In a given sample (60 seconds), an outbound data transfer of 20 GB uses up all available bandwidth, causing bw_out_allowance_exceeded to increment. Transfer completes in about 20 seconds, and no further data is sent afterwards.Instance remains idle for the remaining 4 samples (240 seconds).In this example, the average throughput in a 5-minute period is much lower than the one during the microburst:SUM(NetworkOut) / PERIOD = ((20 GB * 1 sample) + (0 GB * 4 samples)) / 300 seconds = ~0.066 GB/s * 8 bits = ~0.533 GbpsEven if you calculate the throughput based on the highest sample, the average still doesn't reflect the throughput amount:MAXIMUM(NetworkOut) / 60 = 20 GB / 60 seconds = ~0.333 GB/s * 8 bits = ~2.667 GbpsMonitoring microburstsTo measure throughput and PPS at a more granular level, use operating system (OS) tools to monitor network statistics. Monitor your network statistics more frequently during periods of shaping or high activity.The following are examples of OS tools:Linuxsarnloadiftopiptraf-ngWindowsPerformance MonitorThe CloudWatch agent can be used on both Linux and Windows to publish these OS-level network metrics to CloudWatch as custom metrics. These metrics can be published at intervals as low as 1 second. Note that high-resolution metrics (those with a period lower than 60 seconds) lead to higher charges. For more information about CloudWatch pricing, see Amazon CloudWatch pricing.It's a best practice to monitor the network performance metrics provided by ENA. The driver version must be greater than or equal to 2.2.10 (Linux) and 2.2.2 (Windows) for you to be able to view the metrics. For more information, see the relevant pages for Linux and Windows.CloudWatch Agent can also publish ENA metrics. For instructions on publishing ENA metrics for Linux, see Collect network performance metrics. For Windows, ENA metrics are available in Performance Monitor, and the CloudWatch Agent can be configured to publish any metrics available there.Preventing microburstsTo avoid microbursts, traffic needs to be paced at the senders, so that it doesn't exceed a maximum throughput or packet rate. This makes microbursts difficult to avoid. Pacing traffic at the senders usually require application-level changes. Depending on how this change is implemented, OS support for traffic pacing might need to be available and turned on at the senders. That might not always be possible, or practical. Microbursting can also happen because of too many connections sending packets in a short period. When this happens, pacing individual senders doesn't help.Due to these reasons, it's a best practice to monitor ENA metrics. If limits are reached often, or if there's evidence that traffic shaping is impacting your applications, do the following:Scale up: Move to a larger instance size. Larger instances generally have higher allowances. Network optimized instances (those with an "n", such as C5n), have higher allowances than their respective non network-optimized counterparts.Scale out: Spread traffic across multiple instances to reduce traffic and contention at individual instances.On Linux, in addition to the preceding options, there are potential mitigation options for advanced users. You can implement these options alone or in combination. It's a best practice to benchmark mitigations in a testing environment to verify that they reduce or eliminate traffic shaping without adversely effecting your workload.SO_MAX_PACING_RATE: This socket option can be passed by an application to the setsockopt system call to specify a maximum pacing rate (bytes per second). The Linux kernel then paces traffic from that socket so that it doesn't exceed the limit. This option requires the following:Application code-level changes.Support from the kernel.The use of Fair Queue (FQ) queuing discipline or the kernel's support for pacing at the TCP layer (applicable to TCP only).Queuing disciplines (qdiscs): qdiscs are responsible for packet scheduling and optional shaping. Some qdiscs, such as fq, help smooth out traffic bursts from individual flows. For more information, see the Traffic Control (TC) manual page.Shallow Transmission (Tx) queues: In some scenarios, shallow Tx queues help reduce PPS shaping. It can be achieved in two ways:Byte Queue Limits (BQL): BQL dynamically limits the number of in-flight bytes on Tx queues. BQL is turned on by default on ENA driver versions shipped with the Linux kernel (those ending with a "K"). For ENA driver versions from GitHub (those ending with a "g"), BQL is available as of v2.6.1g, and is turned off by default. You can turn on BQL using the enable_bql ENA module parameter.Reduced Tx queue length: Reduce Tx queue length from its default of 1,024 packets to a lower amount (minimum 256). You can change this length using the ethtool.Related informationAmazon EC2 instance network bandwidthMonitor network performance for your EC2 instance (Linux)Monitor network performance for your EC2 instance (Windows)Follow"


Question: Why was my EMR cluster terminated?

My Amazon EMR cluster terminated unexpectedly.

Answer:
"My Amazon EMR cluster terminated unexpectedly.ResolutionReview Amazon EMR provisioning logs stored in Amazon S3Amazon EMR cluster logs are stored in an Amazon Simple Storage Service (Amazon S3) bucket that's specified at cluster launch. The logs are stored at s3://example-log-location/example-cluster-ID/node/example-EC2-instance-ID/.Note: Replace example-log-location, example-cluster-ID, and example-EC2-instance-ID with your system's naming.The following is a list of common errors:SHUTDOWN_STEP_FAILED (USER_ERROR)NO_SLAVES_LEFT (SYSTEM_ERROR)The master failed: Error occurred: ??<title>502 Bad Gateway</title>????
502 Bad Gateway
??
nginx/1.16.1??????KMS_ISSUE (USER_ERROR)Terminated with errors, The master node was terminated by user.Note: The preceding are the most common termination errors. EMR clusters might be terminated due to errors other than those listed. For more information, see Resource errors.SHUTDOWN_STEP_FAILED (USER_ERROR)When you submit a step job in your EMR cluster, you can specify the step failure behavior in the ActionOnFailure parameter. The EMR cluster terminates if you select TERMINATE_CLUSTER or TERMINATE_JOB_FLOW for the ActionOnFailure parameter. For more information, see StepConfig.The following is an example error message from AWS CloudTrail:{  "severity": "ERROR",  "actionOnFailure": "TERMINATE_JOB_FLOW",  "stepId": "s-2I0GXXXXXXXX",  "name": "Example Step",  "clusterId": "j-2YJXXXXXXX",  "state": "FAILED",  "message": "Step s-2I0GXXXXXXXX (Example Step) in Amazon EMR cluster j-2YJXXXXXXX failed at 202X-1X-0X 0X:XX UTC."}To avoid this error, use the CONTINUE or CANCEL_AND_WAIT option in the ActionOnFailure parameter when submitting the step job.NO_SLAVES_LEFT (SYSTEM_ERROR)This error occurs when:Termination protection is turned off in the EMR cluster.All core nodes exceed disk storage capacity as specified by a maximum utilization threshold in the yarn-site configuration classification. The default maximum utilization threshold is 90%.The CORE instance is a Spot Instance, and the Spot Instance is TERMINATED_BY_SPOT_DUE_TO_NO_CAPACITY.For information on Spot Instance termination, see Why did Amazon EC2 terminate my Spot Instance?For more information on the NO_SLAVE_LEFT error, see, see Cluster terminated with NO_SLAVE_LEFT and core nodes FAILED_BY_MASTER.The following is an example error message from the instance-controller:202X-0X-0X 1X:5X:5X,968 INFO Poller: InstanceJointStatusMap contains X entries (DD:5 R:3):i-0e336xxxxxxxxxxxx 25d21h R  25d21h ig-22 ip-1x-2xx-xx-1xx.local.xxx.com  I:   52s Y:U    98s c: 0 am:    0 H:R  1.1%Yarn unhealthy Reason : 1/4 local-dirs usable space is below configured utilization percentage/no more usable space [ /mnt/yarn : used space above threshold of 90.0% ] ; 1/1 log-dirs usable space is below configured utilization percentage/no more usable space [ /var/log/hadoop-yarn/containers : used space above threshold of 90.0% ]To resolve this error:Keep termination protection ON for your clusters. For more information, see Termination protection and unhealthy YARN nodes.Use Amazon EMR scaling policies (automatic scaling and managed scaling) to scale core nodes based on your requirements. For more information, see Scaling cluster resources.Add more Amazon Elastic Block Storage (Amazon EBS) capacity to your cluster. For more information, see How can I resolve "Exit status: -100. Diagnostics: Container released on a *lost* node" errors in Amazon EMR?Create an alarm for the MRUnhealthyNodes Amazon CloudWatch metric. You can set up a notification for this alarm to warn you of unhealthy nodes before the 45-minute timeout is reached. For more information, see Create a CloudWatch alarm based on a static threshold.502 Bad GatewayThe 502 Bad Gateway error occurs when Amazon EMR internal systems can't reach the primary node for a period of time. Amazon EMR is terminated if termination protection is turned off. Check the latest instance-controller logs and instance state logs when the instance-controller service is down. The instance-controller standard output shows that the service is terminated because there is insufficient memory. This indicates that the primary node of the cluster is low on memory.The following is an example error message from the instance state log:# dump instance controller stdouttail -n 100 /emr/instance-controller/log/instance-controller.outOpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fb46c7c8000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)## There is insufficient memory for the Java Runtime Environment to continue.# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.# An error report file with more information is saved as:# /tmp/hs_err_pid16110.log# whats memory usage look likefree -m              total        used        free      shared  buff/cache   availableMem:          15661       15346         147           0         167          69Swap:             0           0           0To avoid the preceding error, launch an EMR cluster with a higher instance type to leverage more memory for your cluster's requirements. Also, clean up disk space to avoid memory outages in long running clusters. For more information, see How do I troubleshoot primary node failure with error "502 Bad Gateway" or "504 Gateway Time-out" in Amazon EMR?KMS_ISSUE (USER_ERROR)When using an Amazon EMR security configuration to encrypt an Amazon EBS root device and storage volumes, the role must have proper permissions. If the necessary permissions are missing, then you receive the KMS_ISSUE error.The following is an example error message from AWS CloudTrail:The EMR Service Role must have the kms:GenerateDataKey* and kms:ReEncrypt* permission for the KMS key configuration when you enabled EBS encryption by default. You can retrieve that KMS key's ID by using the ec2:GetEbsDefaultKmsKeyId API.To avoid the preceding error, make sure that security configurations that are used to encrypt the Amazon EBS root device and storage volumes have the necessary permissions. For these configurations, be sure that the Amazon EMR service role (EMR_DefaultRole_V2) has permissions to use the specified AWS Key Management Service (AWS KMS) key.Terminated with errors, The master node was terminated by userWhen the EMR cluster primary node stops for any reason, the cluster terminates with the The master node was terminated by user error.The following is an example error message from AWS CloudTrail:eventTime": "2023-01-18T08:07:02Z",    "eventSource": "ec2.amazonaws.com",    "eventName": "StopInstances",    "awsRegion": "us-east-1",    "sourceIPAddress": "52.xx.xx.xx",    "userAgent": "AWS Internal",    "requestParameters": {        "instancesSet": {            "items": [                {                    "instanceId": "i-xxf6c5xxxxxxxxxxx"                }            ]        },        "force": false},Because stopping the EMR primary or all core nodes leads to cluster termination, avoid stopping or rebooting cluster nodes.Follow"


Question: Why do I get the error "Unable to validate the following destination configurations" when I create an Amazon S3 Event Notification?

I want to troubleshoot the following error when I create an Amazon Simple Storage Service (Amazon S3) Event Notification: "Unable to validate the following destination configurations when creating an Amazon S3 Event Notification"

Answer: "I want to troubleshoot the following error when I create an Amazon Simple Storage Service (Amazon S3) Event Notification: "Unable to validate the following destination configurations when creating an Amazon S3 Event Notification"Short descriptionYou get this error when any of the following conditions aren't met:When you create an Amazon S3 Event Notification, the destination that you specify must have a resource-based policy. This policy must allow S3 to publish notifications to the destination.The Region for the destination must be same as that of the S3 bucket.If the destination is encrypted with an AWS Key Management Service (AWS KMS) key, the key policy must allow access to S3.When you update an event notification configuration, Amazon S3 verifies that all the event destinations exist. S3 also checks that the event destinations have the required resource-based policies to allow S3 to do the following depending on the event destination type:Publish eventsSend messagesInvoke functionsS3 performs these checks on new event destinations as well as any existing event destinations that aren't being removed during the update. When any of these checks fail, you might get the error "Unable to validate the following destination configurations".ResolutionBe sure that all the event destinations existYou get this error when an event destination that's on the bucket doesn't exist.If you configured an existing event notification to send events to a destination that no longer exists, you must remove these events. You must do so before updating the event notification configuration. You might find that the notifications currently exist on the bucket and the destination no longer exists. This might be because the destination existed when the event notification configuration was last updated, but was deleted after.It's a best practice to delete all events with non-existent destinations using the same PutBucketNotificationConfiguration API. You can run this operation using the Amazon S3 console, AWS Command Line Interface (AWS CLI), or AWS SDK.Confirm that the event destinations have valid resource-based policiesBe sure to grant the necessary permissions to Amazon S3 to call the relevant API to publish messages to the following:An Amazon Simple Notification Service (Amazon SNS) topicAn Amazon Simple Queue Service (Amazon SQS) queueAn AWS Lambda functionIf you're adding a new event to the bucket, make sure that the new event has the valid resource-based policy:For more information on permissions related to an Amazon SNS topic or Amazon SQS queue, see Granting permissions to publish messages to an SNS topic or an SQS queue.For more information on permissions related to AWS Lambda function, see Step 7. Configure Amazon S3 to publish events.For information on updating the Lambda function to make sure it has valid policies, see Tutorial: Using an Amazon S3 trigger to invoke a Lambda function.Note: If you're using the S3 console to turn on Event Notifications, S3 automatically attempts to update the resource-based policy for you before adding the event.If you have the valid resource-based policy for the event destination and are still getting the same error, then check all the existing event destinations. Make sure that they have the valid resource-based policies as well. You get the error if any resource-based policies for the event destinations were modified after the event notification configuration was last updated on the bucket. The error persists until all the permissions are valid or the events with non-valid destinations are removed.Make sure that the AWS KMS Key policy is valid for KMS-encrypted topics and queuesIf you activate AWS KMS encryption for an SQS or SNS destination, update the customer managed key policy to allow S3 to use the key. For more information, see AWS KMS key policy.AWS managed keys, such as aws/sns and aws/sqs keys aren't supported. If you're using one of these keys on your event destination, you must update the destination to use a customer managed key. Then, update the key policy.Related informationIAM policy for a destination SNS topicIAM policy for a destination SQS queueFollow"


Question: How can I import a third-party issued TLS/SSL certificate to ACM?

I want to import a third-party issued TLS/SSL certificate into AWS Certificate Manager (ACM).


Question: How do I use an NGINX proxy to access OpenSearch Dashboards from outside a VPC that's using Amazon Cognito authentication?

My Amazon OpenSearch Service cluster is in a virtual private cloud (VPC). I want to use an NGINX proxy to access OpenSearch Dashboards from outside of the VPC with Amazon Cognito authentication. How can I do this?

Answer: "My Amazon OpenSearch Service cluster is in a virtual private cloud (VPC). I want to use an NGINX proxy to access OpenSearch Dashboards from outside of the VPC with Amazon Cognito authentication. How can I do this?Short descriptionUse NGINX to configure an Amazon Elastic Compute Cloud (Amazon EC2) instance as a proxy server. The proxy server then forwards browser requests to Amazon Cognito and OpenSearch Dashboards.Note: This solution works for native Amazon Cognito users only.You can also use an SSH tunnel or Client VPN to access OpenSearch Dashboards from outside a VPC with Amazon Cognito authentication. For more information, see How can I access OpenSearch Dashboards from outside of a VPC using Amazon Cognito authentication?ResolutionImportant: Your OpenSearch Service domain is more secure when you restrict access to users in the VPC. Before you continue, be sure that this procedure doesn't violate your organization's security requirements.1.    Create an Amazon Cognito user pool.2.    Configure a hosted user pool domain.3.    In the Amazon Cognito console navigation pane, choose Users and groups.4.    Choose Create user and then complete the fields. Be sure to enter an email address, and then select the Mark email as verified check box.5.    Choose the Groups tab and then choose Create group. For Precedence, enter 0. For more information, see Creating a new group in the AWS Management Console.6.    Open the Amazon Cognito console again.7.    Choose Manage Identity Pools, and then choose Create new identity pool.8.    Enter a name for your identity pool, select the check box to Enable access to unauthenticated identities, and then choose Create Pool.9.    When you're prompted for access to your AWS resources, choose Allow to create the two default roles associated with your identity pool. Create one default role for unauthenticated users, and then create the other default role for authenticated users.10.    Configure your OpenSearch Service domain to use Amazon Cognito authentication for OpenSearch Dashboards: For Cognito User Pool, choose the user pool that you created. For Cognito Identity Pool, choose the identity pool that you created. For more information, see Get started with OpenSearch Service: Use Amazon Cognito for OpenSearch Dashboards access control.11.    Configure your access policy so that it is a resource-based policy like the following:{     "Version": "2012-10-17",     "Statement": [{          "Effect": "Allow",          "Principal": {               "AWS": "arn:aws:iam::account-id:role/Cognito_identity-nameAuth_Role"          },          "Action": "es:*",          "Resource": "arn:aws:es:region:aws-account-id:domain/domain-name/*"     }]}You must update these values:account-id: your AWS account IDidentity-name: the name of your Amazon Cognito identity pooldomain-name: the name of your OpenSearch Service domainregion: the Region where your OpenSearch Service domain resides, such as us-east-1Note: The settings in this example apply to Amazon Linux 2 AMI. If you're using a different AMI, you might need to adjust your specific settings.12.    Launch an EC2 instance into a public subnet of the same VPC of your OpenSearch Service domain. Make sure that the instance uses the same security group as your domain.13.    (Optional) If you're not using a test environment, then allocate an Elastic IP address to associate with the instance that you created. For more information about allocating Elastic IP addresses, see Work with Elastic IP addresses.14.    (Optional) If you're not using a test environment, then configure your DNS to resolve requests to the Elastic IP address. For more information about how to resolve requests with Amazon Route 53, see Configuring Route 53 to route traffic to an EC2 instance.15.    Connect to your instance, and then install NGINX.For instances launched with an Amazon Linux 2 AMI, use the following command:$ sudo amazon-linux-extras install nginx116.    Get an SSL certificate from a certificate authority (CA) to configure SSL for NGINX.Note: If you're using a test environment, generate a self-signed certificate instead. Note that it's a best practice to use only SSL certificates signed by a third-party certificate authority in your production environment.17.    (Optional) If you're using a test environment with a self-signed certificate, then use the OpenSSL x509 command to generate a private key:$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/cert.key -out /etc/nginx/cert.crtThis command syntax generates cert.key, a private key for the self-signed SSL certificate.18.    Navigate to the /etc/nginx/conf.d directory, and then create a file called default.conf. Modify the file with the following values: /etc/nginx/cert.crt: the path to your SSL certificate /etc/nginx/cert.key: the path to the private key that you generated for the SSL certificate my_domain_host: your OpenSearch Service endpoint my_cognito_host: your Amazon Cognito user pool domain (that you configured in Step 2).You can use the sed command to assign my_cognito_host and my_domain_host as variables, instead of replacing them directly in the default.conf file. Also, make sure to use HTTPS, or you might encounter an error.In this example, be aware of the following:If your Amazon OpenSearch Service domain is running OpenSearch Service version 1.0+, then use the _dashboards endpoint.If your Amazon OpenSearch Service domain is running Elasticsearch versions 5.x-7.x, then use the _plugin/kibana endpoint.Important: The resolver parameter changes according to your VPC settings. The DNS resolver is located at your primary CIDR block's base IP plus two. For example, if you create a VPC with CIDR block 10.0.0.0/24, then your DNS resolver is located at 10.0.0.2.server {  listen 443;  server_name $host;  rewrite ^/$ https://$host/_dashboards redirect;  resolver 10.0.0.2 ipv6=off valid=5s;  set $domain_endpoint my_domain_host;  set $cognito_host my_cognito_host;  ssl_certificate           /etc/nginx/cert.crt;  ssl_certificate_key       /etc/nginx/cert.key;  ssl on;  ssl_session_cache  builtin:1000  shared:SSL:10m;  ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;  ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;  ssl_prefer_server_ciphers on;  location ^~ /_dashboards {    # Forward requests to Dashboards    proxy_pass https://$domain_endpoint;    # Handle redirects to Cognito    proxy_redirect https://$cognito_host https://$host;    # Handle redirects to Dashboards    proxy_redirect https://$domain_endpoint https://$host;    # Update cookie domain and path    proxy_cookie_domain $domain_endpoint $host;    proxy_cookie_path ~*^/$ /_dashboards/;    # Response buffer settings    proxy_buffer_size 128k;    proxy_buffers 4 256k;    proxy_busy_buffers_size 256k;  }   location ~ \/(log|sign|fav|forgot|change|saml|oauth2|confirm) {    # Forward requests to Cognito    proxy_pass https://$cognito_host;    # Handle redirects to Dashboards    proxy_redirect https://$domain_endpoint https://$host;    # Handle redirects to Cognito    proxy_redirect https://$cognito_host https://$host;    proxy_cookie_domain $cognito_host $host;  }}19.    (Optional) Use the sed command to assign the my_domain_host and my_cognito_host variables:sudo sed -i 's/my_domain_host/vpc-cognito-private-xxxxxxxxxx.us-east-1.es.amazonaws.com/' /etc/nginx/conf.d/default.confsudo sed -i 's/my_cognito_host/dean-kumo-xxxxxxx.auth.us-east-1.amazoncognito.com/' /etc/nginx/conf.d/default.confNote: You don't have to replace my_domain_host and my_cognito_host in the /etc/nginx/conf.d/default.conf file.20.    Use this command to restart NGINX.$ sudo systemctl restart nginx.service21.    Use your browser to access the NGINX IP or the DNS name. You're redirected to the Amazon Cognito log-in page.22.    To log in to Dashboards, enter your user name and temporary password. Then, change your password when prompted, and log in again.Related informationUsing a proxy to access OpenSearch Service from DashboardsHow do I troubleshoot Amazon Cognito authentication issues with OpenSearch Dashboards?I get a "User: anonymous is not authorized" error when I try to access my Amazon OpenSearch Service clusterFollow"


Question: How can I turn on audit logging for my Amazon Aurora MySQL-Compatible DB cluster and publish the logs to CloudWatch?

"To meet compliance requirements, I want to turn on audit logging on my Amazon Aurora MySQL-Compatible Edition DB cluster to audit database activity. Then, I want to publish the DB logs to Amazon CloudWatch so that I can perform real-time data analysis."

Answer: "To meet compliance requirements, I want to turn on audit logging on my Amazon Aurora MySQL-Compatible Edition DB cluster to audit database activity. Then, I want to publish the DB logs to Amazon CloudWatch so that I can perform real-time data analysis.Short descriptionUse Advanced Auditing with Amazon Aurora to record and audit database events. Database events can include connections, disconnections, tables queried, or types of queries issued (DML, DDL, or DCL) on an Aurora MySQL-Compatible DB cluster. For more information about the type of information included in the log files, see Audit log details.First, activate the Advanced Auditing parameters on the associated custom DB cluster parameter group. Then, you can publish the Advanced Auditing logs to CloudWatch.Note: If you use Amazon Relational Database Service (Amazon RDS) for MySQL or MariaDB, seeHow can I turn on audit logging for an Amazon RDS for MySQL instance or MariaDB instance and publish the logs to CloudWatch?ResolutionAdvanced Auditing supports the following database capacity types:Aurora ProvisionedAurora Provisioned with Aurora parallel query supportAurora ServerlessNote: If you're using Amazon Aurora Serverless v1, complete the following steps to turn on the audit logging parameters. However, you don't need to configure logs to publish to CloudWatch, as Amazon Aurora Serverless v1 clusters automatically upload these types of logs. To configure log uploads for v1 clusters, modify the value of the log types in the DB cluster parameter group.Turning on Advanced Auditing parameters on the cluster parameter groupCreate a custom DB cluster parameter group.Modify the parameters for Advanced Auditing.Modify the cluster to associate the new custom DB parameter group with your Aurora MySQL-Compatible DB cluster.For details about the Advanced Auditing parameters, see Turning on Advanced Auditing. These parameters are dynamic, so you don't need to reboot your DB cluster. When you change the default parameter group to a custom parameter group, manually reboot the DB instance to apply the new group.Publishing the Advanced Auditing logs to CloudWatchOpen the Amazon RDS console.Choose Databases from the navigation pane.Select the Aurora MySQL-Compatible DB cluster that you want to export log data to CloudWatch.Choose Modify.From the Log exports section, select Audit log.Choose Continue.Review the Summary of modifications, and then choose Modify cluster.Or, you can publish Advanced Auditing logs to CloudWatch Logs by setting the value for the cluster-level DB parameter server_audit_logs_upload to 1. The default value for the parameter is 0. You can also use the AWS Command Line Interface (AWS CLI) to turn on CloudWatch log exports by running a command similar to the following:aws rds modify-db-cluster --db-cluster-identifier <mydbcluster> --cloudwatch-logs-export-configuration '{"EnableLogTypes":["audit"]}'After turning on audit logging and modifying your instance to export logs, events recorded in audit logs are sent to CloudWatch. Then, you can monitor the log events in CloudWatch.Note: Audit data don't appear in the logs unless you also define one or more types of events to audit using the server_audit_events parameter.Related informationAuditing an Amazon Aurora clusterUsing Advanced Auditing with an Amazon Aurora MySQL-Compatible DB clusterPublishing Amazon Aurora MySQL logs to Amazon CloudWatch LogsFollow"


Question: How can I increase the PHP upload size limit in my Lightsail Bitnami WordPress instance?

I want to upload large files onto my Amazon Lightsail Bitnami WordPress instance. How can I increase the default file upload limit on my instance?

Answer: "I want to upload large files onto my Amazon Lightsail Bitnami WordPress instance. How can I increase the default file upload limit on my instance?Short descriptionUpload_max_filesize is the maximum size of an uploaded file. This is the limit for a single file. Post_max_size, on the other hand, is the limit of the entire body of the request. It's a best practice that you set the post_max_size to be greater than or equal to upload_max_filesize.By default, WordPress limits uploads to 40 MB for both post_max_size and upload_max_filesize.To verify the maximum upload limit on your Lightsail instance, run the following commands:# grep -i "upload_max_filesize" /opt/bitnami/php/etc/php.iniupload_max_filesize = 40M# grep -i "post_max_size" /opt/bitnami/php/etc/php.inipost_max_size = 40MResolutionTo modify the maximum upload size, do the following:1.    Open the /opt/bitnami/php/etc/php.ini file to increase upload_max_filesize and post_max_size. Use a text editor, such as the vi editor to access the /opt/bitnami/php/etc/php.ini file.# sudo vi /opt/bitnami/php/etc/php.ini; Maximum size of POST data that PHP will accept.  post_max_size = 64M; Maximum allowed size for uploaded files.  upload_max_filesize = 64M2.    Press the i button to edit the values. After modifying the values, save the file by pressing the ESCAPE button. Then, enter the command :wq!3.    Verify that the LimitRequestBody parameter is specified in the Apache configuration files using the following command:# grep "LimitRequestBody" /opt/bitnami/apache2/conf/httpd.confIf the parameter isn't specified, then there is no size restriction in the Apache web server configuration limits. However, if the parameter is specified and the value is less than the desired upload size, increase the parameter by doing the following:Open /opt/bitnami/apache2/conf/httpd.conf in a text editor such as the vi editor:# sudo vi /opt/bitnami/apache2/conf/httpd.confSet the LimitRequestBody parameter to a new value in bytes, as shown in the following example:LimitRequestBody 65536Press the ESCAPE button and then enter the command :wq! to save the file.4.    Check the configuration details before restarting the services:sudo apachectl configtestSyntax OK5.    Restart the servers for the changes to take effect.sudo /opt/bitnami/ctlscript.sh restartFollow"


Question: How do I troubleshoot high JVM memory pressure on my OpenSearch Service cluster?

"My Amazon OpenSearch Service cluster has high JVM memory pressure, and I don't know how to resolve this."

Answer: "My Amazon OpenSearch Service cluster has high JVM memory pressure, and I don't know how to resolve this.Short descriptionBy default, OpenSearch Service uses 50% of an instance's RAM for JVM heaps up to 32 GiB in size. The JVM memory pressure specifies the percentage of the Java heap in a cluster node. The following guidelines indicate what the JVM memory pressure percentages mean:If JVM memory pressure reaches 75%, then OpenSearch Service initiates the Concurrent Mark Sweep (CMS) garbage collector for x86 Amazon Elastic Compute Cloud (Amazon EC2) instance types. ARM-based Graviton Amazon EC2 instance types use the Garbage-First (G1) garbage collector that uses additional short pauses and heap defragmentation. The garbage collection is a CPU-intensive process. If memory usage continues to grow, then you might encounter ClusterBlockException, JVM OutOfMemoryError, or other cluster performance issues. For more information, see Recovering from a continuous heavy processing load.If JVM memory pressure exceeds 92% for 30 minutes, then OpenSearch Service blocks all write operations.If JVM memory pressure reaches 100%, then OpenSearch Service JVM is configured to exit and eventually restarts on OutOfMemory (OOM).The following reasons can cause high JVM memory pressure:Spikes in the numbers of requests to the cluster.Aggregations, wildcards, and selecting wide time ranges in the queries.Unbalanced shard allocations across nodes or too many shards in a cluster.Field data or index mapping explosions.Instance types that can't handle incoming loads.ResolutionReduce traffic to the cluster to resolve high JVM memory pressure issues. To reduce traffic to the cluster, follow these best practices:Clear the field data cache with the POST /index_name/_cache/clear?fielddata=true API operation.Note: Clearing the cache can disrupt queries that are in progress.Don't aggregate text fields or change the mapping type to "keyword".Scale the domain so that the maximum heap size per node is 32 GB.Turn on slow logs (OpenSearch website) to determine faulty requests.Note: Verify that the JVM memory pressure is below 90%. For more information about slow Elasticsearch queries, see Advanced tuning: finding and fixing slow Elasticsearch queries on the Elasticsearch website.Choose the correct number of shards to optimize search or indexing. For more information about indexing and shard count, see How do I rebalance the uneven shard distribution in my Amazon OpenSearch Service cluster?Delete old or unused indices to reduce the number of shards.For advanced users, you can update the parent field data cache allocation, or request circuit breaker settings according to your use case. For more information about JVM circuit breakers, see JVM OutOfMemoryError.For more information about how to troubleshoot high JVM memory pressure, see Why did my OpenSearch Service node crash?Related informationTroubleshooting Amazon OpenSearch ServiceHow can I scale up or scale out an Amazon OpenSearch Service domain?Get started with Amazon Elasticsearch Service: How many shards do I need?Follow"


Question: "Why is my Amazon Aurora DB cluster clone, snapshot restore, or point in time restore taking so long?"

"I'm performing a cluster clone, snapshot restore, or a point in time operation on my Amazon Aurora cluster."

Answer: "I'm performing a cluster clone, snapshot restore, or a point in time operation on my Amazon Aurora cluster.Short descriptionAmazon Aurora’s continuous backup and restore techniques are optimized to avoid variation in restore times. They also help the cluster’s storage volume to reach full performance as soon as the cluster becomes available. Long restore times are generally caused by long-running transactions in the source database at the time that the backup is taken.ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Amazon Aurora backups your cluster volume’s changes automatically and continuously. The backups are retained for the length of your backup retention period. This continuous backup allows you to restore your data to a new cluster, to any point in time within the retention period specified. This avoids the need for a lengthy binlog roll-forward process. Because you create a new cluster, there is no impact to performance or interruption to your original database.When you initiate a clone, snapshot, or point in time restore, Amazon Relational Database Service (Amazon RDS) calls the following APIs on your behalf:Either RestoreDBClusterFromSnapshot or RestoreDBClusterToPointInTime. These APIs create a new cluster and restore volume from Amazon Simple Storage Service (Amazon S3). This can take up to a few hours to complete. When you restore data to an Aurora cluster, all data is brought in parallel from Amazon S3 to the six copies on your three Availability Zones.Cluster storage volume cloning is a variation of RestoreDBClusterToPointInTime. It uses the copy-on-write protocol, and usually completes in a few minutes.When this step completes, the cluster changes into the Available state. You can check your cluster state by refreshing the console or checking with the AWS CLI.The instance creation process starts only when the cluster is Available. This happens in two stages: setting up the instance configuration and database crash recovery.You can check if the API has finished setting up the instance by looking for the MySQL error log file. You can do this even if the instance is in the Creating status. If the error log file is available to download, then the instance is set up and the engine is now performing crash recovery. The error log file is also the best resource to check on the progress of your database crash recovery, along with Amazon CloudWatch metrics.Note: If you're using the AWS CLI or API to perform a restore operation, then you must invoke the CreateDBInstance call because it's not automatic.Check for long-running write operations on the source databaseIt’s a best practice to confirm that there aren’t long-running write operations on the source database at the time of the snapshot, point-in-time, or clone. Any long-running DCL, DDL, or DML (open write transactions) might lengthen the time it takes for the restored database to become available.For example, you activate the binary log for an Aurora cluster, and this increases the time it takes to perform a recovery. This is because InnoDB automatically checks the logs and performs a roll-forward of the database to the present. It then rolls back any uncommitted transactions that are present at the time of the recovery. For more information on InnoDB crash recovery, see Innodb recovery.When the instance finishes the creation and recovery processes, the cluster and the instance are then ready to accept incoming connections.Note: Aurora doesn't require the binary log. It's a best practice to deactivate it unless it's required. For cross-Region replication, you can evaluate the Aurora global databases instead. Aurora global databases also don't require binary logs.Related informationAmazon Aurora storage and reliabilityRestoring from a DB cluster snapshotFollow"


Question: How can I configure Amazon ECS Service Auto Scaling on Fargate?

I want to configure Amazon Elastic Container Service (Amazon ECS) Service Auto Scaling on AWS Fargate.

Answer: "I want to configure Amazon Elastic Container Service (Amazon ECS) Service Auto Scaling on AWS Fargate.Short descriptionYou can increase or decrease your desired task count by integrating Amazon ECS on Fargate with Amazon CloudWatch alarms and Application Auto Scaling. Then, you can use CloudWatch metrics to configure your CloudWatch alarms.When your CloudWatch alarms trigger an Auto Scaling policy, Application Auto Scaling decides the new desired count based on the configured scaling policy. Then, Application Auto Scaling makes the UpdateService API call to Amazon ECS with the new desired count value. The Amazon ECS service scheduler launches or shuts down tasks to meet the new desired count. Your scaling activity remains in the InProgress state until the desired count and the running count are same.Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.ResolutionNote: The IAM user that accesses Service Auto Scaling settings must have the appropriate permissions for the services that support dynamic scaling. For more information, see IAM permissions required for service auto scaling.Configure Service Auto Scaling for your Fargate serviceWhen you create or update a service in the Amazon ECS console, choose the following from the Set Auto Scaling page:1.    Choose Configure Service Auto Scaling to adjust your service’s desired count.2.    For Minimum number of tasks, enter the lowest number of tasks that you want Service Auto Scaling to use.3.    For Desired number of tasks, enter the number of tasks that you want Service Auto Scaling to use.Note: The desired number of tasks must be in the range of your minimum and maximum task count.4.    For Maximum number of tasks, enter the highest number of tasks that you want Service Auto Scaling to use.Note: The minimum and maximum number of tasks are hard limits for your service.5.    For IAM role for Service Auto Scaling, choose ecsAutoscaleRole.6.    In the Automatic task scaling policies section, choose Auto Scaling Policy.7.    Complete the remaining steps in the setup wizard to create or update your service.Choose a scaling policyChoose either a target tracking policy or step scaling policy based on your requirements and the following considerations:For target tracking policies:You must define a target value (threshold) for your specified metric. Application Auto Scaling creates and manages the CloudWatch alarms that trigger the scaling policy.You can use the ECSServiceAverageCPUUtilization, ECSServiceAverageMemoryUtilization, and ALBRequestCountPerTarget Amazon ECS service metrics for target tracking.A target tracking policy calculates the scaling adjustment (that is, desired task count) based on the metric and target value that you define. You don't need to configure the scaling action like you do with a step scaling policy. This is because a target tracking policy adds or removes capacity as required to keep the metric at, or close to, the specified target value.You must choose a scale-out and scale-in cooldown period.For step scaling policies:You can create or use existing CloudWatch alarms for any metric for step scaling.You must choose scaling actions or step adjustments, such as ScalingAdjustment, MetricIntervalUpperBound, and MetricIntervalLowerBound.You can specify the scaling adjustment type as a percentage of the current capacity of your scalable target, or by using absolute numbers.You can increase or decrease your desired task count by creating scaling policies to handle scale-in and scale-out activity.Note: For more information, see Step 5: Configuring your service to use Service Auto Scaling.Configure Service Auto Scaling using the AWS CLI1.    Register your Fargate service as a scalable target with Application Auto Scaling:aws application-autoscaling register-scalable-target \--service-namespace ecs --scalable-dimension ecs:service:DesiredCount \--resource-id service/your-cluster/your-service-name \--min-capacity 1 --max-capacity 10 --region us-east-12.    Create a target tracking or step scaling policy for the scalable target (that is, your Fargate service).Target tracking policy:Create a target tracking policy:aws application-autoscaling put-scaling-policy \--service-namespace ecs --scalable-dimension ecs:service:DesiredCount \--resource-id service/<cluster>/<service-name> \--policy-name Test-target-tracking-scaling-policy --policy-type TargetTrackingScaling \--target-tracking-scaling-policy-configuration '{ "TargetValue": 75.0, "PredefinedMetricSpecification": {"PredefinedMetricType": "ECSServiceAverageCPUUtilization" }, "ScaleOutCooldown": 60,"ScaleInCooldown": 60}'Note: When you create your target tracking policy, Application Auto Scaling creates CloudWatch alarms for scaling in and scaling out based on the target value (threshold) that you choose.Step scaling policy:1.    Create a step scaling policy with the required step adjustments:aws application-autoscaling put-scaling-policy \--service-namespace ecs --scalable-dimension ecs:service:DesiredCount \--resource-id service/your-cluster/your-service-name \--policy-name Test-Step-scaling-policy-ScaleOut --policy-type StepScaling \--step-scaling-policy-configuration '{"AdjustmentType": "ExactCapacity","StepAdjustments": [ { "MetricIntervalLowerBound": 20, "ScalingAdjustment": 10}, { "MetricIntervalLowerBound": 0, "MetricIntervalUpperBound": 20,"ScalingAdjustment": 5}],"Cooldown": 60, "MetricAggregationType": "Average"}'2.    Associate your scaling policy from step 1 as an alarm action to your new or existing CloudWatch alarm:aws cloudwatch put-metric-alarm \--alarm-name Test-ScaleOut --metric-name MemoryUtilization \--namespace AWS/ECS --statistic Average --period 60 --threshold 60 \--comparison-operator GreaterThanOrEqualToThreshold --evaluation-periods 1 --datapoints-to-alarm 1 \--dimensions Name=ClusterName,Value=your-cluster Name=ServiceName,Value=your-service-name --unit Percent \--alarm-actions "arn:aws:autoscaling:us-east-1:xxxxxxx:scalingPolicy:xxxxxx/Test-Step-scaling-policy-ScaleOut"3.    Create another step scaling policy and CloudWatch alarm for the scale-in activity.Note: When the scale-out activity of your service is in progress, any scale-in activity that's triggered by CloudWatch is blocked until the scale-out activity is completed. The InProgress scale-out activity times out when Amazon ECS doesn't meet the desired count set by Auto Scaling. The time out can have various causes, such as image or networking issues. If a scale-in activity is triggered by CloudWatch metrics during the scale-out cooldown period but a scale-out activity is completed, then a scale-in activity is run.Related informationTroubleshooting service auto scalingService auto scalingDescribeScalingActivitiesAWS CLI command reference for application-autoscalingFollow"


Question: What are the XXXX-VendedLog-Bytes or XXXX-VendedLog-YY-Byte charges on my AWS bill?

I'm seeing charges for XXXX-VendedLog-Bytes or XXXX-VendedLog-YY-Bytes on my AWS bill. I want to see more details regarding the charges and reduce them in the future.

Answer: "I'm seeing charges for XXXX-VendedLog-Bytes or XXXX-VendedLog-YY-Bytes on my AWS bill. I want to see more details regarding the charges and reduce them in the future.Short descriptionNote: In this article, XXXX represents the AWS Region code for the Region where logs have been configured. For example, USE1 is the Region code for us-east-1. YY represents the AWS service that relates to the charges, such as VPC_FLOW or ROUTE_53.XXXX-VendedLog-Bytes or XXXX-VendedLog-YY-Bytes indicate that the logs are configured to transfer data to Amazon Simple Storage Service (Amazon S3).Several AWS services can be configured to publish logs to Amazon S3. These logs are called vended logs. The following are examples of logs that can post to Amazon S3:Amazon Virtual Private Cloud (Amazon VPC) Flow LogsAWS Global Accelerator flow logsAmazon Route 53 Resolver query logsAWS WAF access logsFor more information on how charges are calculated for vended logs, see Amazon CloudWatch pricing.Use AWS usage reports to manually calculate the cost of transferring your data to Amazon S3. Then, compare the costs with the charges in your AWS bill to verify them. Finally, implement the cost savings strategies to reduce future charges.ResolutionUse Cost Explorer to visualize chargesOpen the AWS Billing console.In the navigation pane, choose Cost Explorer.From the navigation pane, choose ReportsChoose Create new report.For Select a report type, select Cost and usage. Then, choose Create Report.From the Report parameters pane, set your filters to find the relevant charges.For Time, choose the relevant dates.For Granularity, choose Daily.For Dimension, choose Usage type.Under Filters, for Service, select CloudWatch. Then, choose Apply.For Usage Type, enter Vended in the search field. Then, select the vended logs that you want to examine. Choose Apply filter.Two graphs display the daily volume of logs and the corresponding costs. You can compare how the log volume changes in relation to the workloads in your environment.Generate a report of CloudWatch chargesOpen the AWS Billing console.In the navigation pane, choose Cost Usage Reports.Choose AWS Usage Report.For Services, choose AmazonCloudWatch.For Usage Types, choose All Usage Types.For Operation, choose All Operations.For Time Period, choose the time period that you're investigating.For Report Granularity, it's a best practice to choose Days, because large usage reports can be truncated.Choose Download report (CSV).(Optional) Analyze Cost and Usage ReportsYou can use a spreadsheet application, such as Microsoft Excel, to analyze the AWS Cost and Usage Reports (AWS CUR) that was generated. This report presents similar data to the graphs in the Use Cost Explorer to visualize charges section of this article. You can filter the report by log types, Regions, or timeframes to have more granular information than the Cost Explorer graph.For example, compare the UsageType field and the vended logs that you are examining in the XXXX-VendedLog-YY-Bytes field. The UsageValue column reflects the volume of log data in bytes. To correlate UsageValue in the AWS CUR report with the Usage (GB) value in the Cost Explorer graph, convert bytes to gibibytes.Reduce future costsVended log charges are based on the volume of logs sent to Amazon CloudWatch Logs, Amazon S3, or Amazon Kinesis Data Firehose. To reduce future fees, review configuration options or strategies that reduce the quantity of logs. The following is an example of suggestions to consider if you use VPC Flow Logs:Turn on VPC Flow Logs only when troubleshooting.Limit scope based on elastic network interfaces (ENIs).Turn on flow logs for only select ENIs or subnets.Turn on flow logs for only REJECT/ACCEPT traffic.Related informationAnalyze CloudWatch cost and usage data with Cost ExplorerFollow"


Question: Why do accounts in my organization show as "suspended" in the AWS Organizations console?

"Some member accounts in my organization show as "suspended" in the AWS Organizations console. Why is this happening, and how can I fix this?"

Answer: "Some member accounts in my organization show as "suspended" in the AWS Organizations console. Why is this happening, and how can I fix this?ResolutionIf a member account of an organization is closed and remains closed for 90 days, then the account is visible in your organization with the "suspended" state for 90 days after the account closure. After 90 days, the closed member account is permanently removed from your organization and is no longer visible in your organization. The AWS Organizations console no longer displays details about the former member account.Note: The account resources in AWS China (Beijing) and AWS China (Ningxia) Regions are subject to the policies of operating partners (Sinnet in the Beijing Region and NWCD in the Ningxia Region). Account closure procedures in China might take longer than in other AWS Regions.All member accounts, open or closed, count against the account quotas for your organization. To increase the account quota, contact AWS Support. For more information, see How do I manage my AWS Service Quotas?Related informationClosing an AWS accountFollow"


Question: How do I turn on logging for my CloudFront distribution?

"I want to log the requests that come to my Amazon CloudFront distributions. How do I turn on logging, and how do I choose between standard logging and real-time logging?"

Answer: "I want to log the requests that come to my Amazon CloudFront distributions. How do I turn on logging, and how do I choose between standard logging and real-time logging?Short descriptionCloudFront standard logs, or access logs, detail every request that's made to a CloudFront distribution. These logs are delivered to an Amazon Simple Storage Service (Amazon S3) bucket. CloudFront doesn’t charge for standard logs. However, you can accrue Amazon S3 charges for storing and accessing files on Amazon S3.CloudFront real-time logs provide information about requests made to a distribution as they occur. You configure the percentage of requests and specific fields that you want to receive in the log records. These logs are delivered to Amazon Kinesis Data Streams. Real-time logs are charged based on the number of log lines that are generated. For more information about CloudFront pricing, see CloudFront Pricing. For more information about Kinesis Data Streams costs, see Amazon Kinesis Data Streams Pricing.ResolutionTurn on CloudFront standard logsBefore turning on standard access logs for your CloudFront distribution, note:Don’t choose an Amazon S3 bucket with S3 Object Ownership set to Bucket owner enforced. This setting removes network access control lists (ACL) for the bucket and the objects in it, which means that CloudFront can't deliver log files to the bucket.CloudFront doesn't deliver standards logs to Amazon S3 buckets in the following AWS Regions: Africa (Cape Town), Asia Pacific (Hong Kong), Asia Pacific (Jakarta), Europe (Milan), and Middle East (Bahrain).Your AWS account must have the s3:GetBucketAcl and s3:PutBucketAcl permissions for the Amazon S3 bucket where you want to deliver the logs.The network ACL for the Amazon S3 bucket must grant your AWS account FULL_CONTROL. If you're the bucket owner, then your account has this permission by default. If you're not the bucket owner, then the bucket owner must update the ACL for the bucket.If the Amazon S3 bucket uses server-side encryption with AWS KMS keys (SSE-KMS) using a customer managed key, then you must have the Required key policy for SSE-KMS buckets.To turn on standard logging for a CloudFront distribution, follow these steps:Access the CloudFront console.Choose the distribution you want to update.On the General tab, under Settings, choose Edit.For Standard logging, select On.Choose the S3 bucket where you want CloudFront to deliver the log files. You can specify an optional prefix for the file names.(Optional) For Cookie logging, select On or Off. If turned on, CloudFront logs the cookies in all requests regardless of which cookies you choose to forward to the origin.Choose Save changes.Turn on CloudFront real-time logsAccess the CloudFront console.From the left-hand navigation, select Logs.Choose the Real-time configurations tab.Choose Create configuration.For Sampling rate, enter the percentage of requests for which you want to receive real-time log records.For Fields, choose the specific fields that you want to receive in the log records. In the Choose options dropdown list, select any fields that you want to include in the configuration.Choose one or more Kinesis data streams to receive real-time logs.Note: CloudFront real-time logs are delivered to the data stream of your choice in Amazon Kinesis Data Streams. To read and analyze your real-time logs, you can build your own Kinesis data stream consumer. Or, use Amazon Kinesis Data Firehose to send the log data to Amazon S3, Amazon Redshift, Amazon OpenSearch Service, or a third-party log processing service.For IAM role, choose Create new service role for the console to create an IAM role for you. To use this option, you must have permission to create IAM roles.-or-Use an existing IAM role.(Optional) In the Distribution section, choose a CloudFront distribution and cache behavior to attach to the real-time log configuration.Choose Create configuration.Follow"


Question: How can I set up host-based routing using an Application Load Balancer?

I have multiple domains that are served by the same Application Load Balancer. I'm using different target groups for each domain. I want to forward the traffic for each domain to their respective target groups. How can I set up host-based routing using an Application Load Balancer?

Answer: "I have multiple domains that are served by the same Application Load Balancer. I'm using different target groups for each domain. I want to forward the traffic for each domain to their respective target groups. How can I set up host-based routing using an Application Load Balancer?ResolutionOpen the Amazon Elastic Compute Cloud (Amazon EC2) console.On the navigation pane, choose Load Balancers under Load Balancing.Select your load balancer, and then choose Listeners.Choose View/edit rules for the load balancer listener that you want to use.Choose the Add rule icon (the plus sign).Choose Insert Rule.Choose Add condition.Choose Host header.Enter the hostname for your domain (for example, example1.com).Choose the checkmark icon to save the condition.Choose Add action to add the desired actions for your condition. To save the actions, choose the checkmark icon.Choose Save.Repeat steps 5-12 to add additional rules to route different domains to different target groups.Notes:To use the same set of target groups to serve multiple domains, add multiple hostnames for your domains.Rules are prioritized based on their position in the rule list. If necessary, you can reorder rules to modify the rule priorities.If a request's host header doesn't match the host headers in any of the rules, the actions configured in the listener's default rule are performed.Related informationListener rules for your Application Load BalancerFollow"


Question: What do I do if I get an "It looks like you aren't authorized" error when trying to use Amazon Lightsail?

"When I try to use the Amazon Lightsail console, I get an error message:"There might be a problem. It looks like you aren't authorized. Try refreshing or signing out of this account and using an account that has access to Lightsail.""

Answer: "When I try to use the Amazon Lightsail console, I get an error message:"There might be a problem. It looks like you aren't authorized. Try refreshing or signing out of this account and using an account that has access to Lightsail."ResolutionThis error appears when the Lightsail service isn't active on your AWS account. Check the following:Your AWS account is fully activated. New AWS accounts can take up to 24 hours to activate. Wait for 10 to 15 minutes, and then try to access Lightsail again.You completed the sign-up process for your AWS account. For more information, see How do I create and activate a new AWS account? Then, enter all necessary account information, verify your phone number, and then add a valid payment method.Your payment method is valid. Your default payment method must be valid. If necessary, update the default payment method, and then try to access Lightsail again.If you still can't access Lightsail, contact AWS Support.Related informationManaging your paymentsFollow"


Question: How do I create an LVM logical volume on a partition created on an EBS volume?

How can I use Logical Volume Manager (LVM) to create a logical volume (LV) on a partition created on my Amazon Elastic Block Store (Amazon EBS) volume?

Answer: "How can I use Logical Volume Manager (LVM) to create a logical volume (LV) on a partition created on my Amazon Elastic Block Store (Amazon EBS) volume?Short descriptionLVM allows you to allocate disk space and strip, re-mirror, and resize logical volumes. Using LVM, you can allocate an EBS volume or a set of EBS volumes to one or more physical volumes.To use LVM on your EBS volume and extend the partitions, follow these steps:Create physical volumes (PV) from your EBS volume.Create a volume group (VG), and then add the physical volumes into the volume group.Create a logical volume (LV), and then mount the directory on the LVM.Create and mount a file system.Resize the logical volume.ResolutionNote: If you already created LVM on your volume and mounted it for use, then follow the instructions beginning at Extend the logical volume.Create physical volumes on the partition of your EBS volumeThe underlying physical storage unit of an LVM logical volume is a block device, such as a partition of an EBS volume or an entire EBS volume.Note: Nitro-based instances expose volumes as NVMe devices. The block device names follow the pattern /dev/nvme1n1, /dev/nvme2n1, /dev/nvme3n1, and so on. If you're using a Nitro-based instance, then replace the device names in the following steps with the appropriate device name. For more information on device naming, see Device names on Linux instances.1.    Open the Amazon Elastic Compute Cloud (Amazon EC2) console.2.    Create your EBS volume and then attach the volume to your instance.3.    Use the gdisk command to create a partition. For the variable Hex code or GUID, enter 8e00. The following example creates the partition /dev/xvdh1 on /dev/xvdh.$ sudo gdisk /dev/xvdhCommand (? for help): nPartition number (1-1218, default 1): 1First sector (34-20971486, default = 2048) or {+-}size{KMGTP}:Last sector (2048-20971486, default = 20971486) or {+-}size{KMGTP}:Current type is 'Linux filesystem'Hex code or GUID (L to show codes, Enter = 8300): 8e00        Changed type of partition to 'Linux LVM'...OK; writing new GUID partition table (GPT) to /dev/xvdh.The operation has completed successfully.Use the lsblk command to confirm creation of the partition:$ lsblkNAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINTxvda    202:0    0   8G  0 disk└─xvda1 202:1    0   8G  0 part /xvdh    202:80   0  10G  0 disk└─xvdh1 202:81   0  10G  0 part4.    Use the pvcreate command to create a physical volume from the partition. The following example, creates a physical volume from /dev/xvdh1:$ sudo pvcreate /dev/xvdh1  Physical volume "/dev/xvdh1" successfully created.Create volume groups and add the physical volumes into the volume groupUse the vgcreate command to create a volume group to combine the new physical volumes. The following example uses one physical volume to create volume group examplegroup1:$ sudo vgcreate examplegroup1 /dev/xvdh1      Volume group "examplegroup1" successfully createdUse the vgs or vgdisplay to view the volume group details:$ sudo vgs  VG             #PV #LV #SN Attr   VSize  VFree  examplegroup1   1   0   0 wz--n- <10.00g <10.00gCreate a logical volume (LV) and a mount directory1.    Use the lvcreate command to create logical volumes (partitions) from your volume group. The following example creates one 9GB logical volume, lvexample1 from the examplegroup1 volume group:$ sudo lvcreate -n lvexample1 -L 9G examplegroup1  Logical volume "lvexample1" createdUse the lvs or lvdisplay command to view the logical volume details:$ sudo lvs  LV          VG   Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert                                   lvexample1 examplegroup1 -wi-a----- 9.00g2.    Use the mkdir command to create a mount directory. The following example creates the directory /mnt1:$ sudo mkdir /mnt1Create and mount a file system1.    Use the following commands to create a file system and mount the partitions for use.Run the mkfs -t command to create the file system. Note: Replace xfs with your file system type, if different. For example, use ext2, ext3, or ext4.$ sudo mkfs -t xfs /dev/examplegroup1/lvexample1Run the lsblk -f command to verify creation of the new file system. Note: The file system type that you created in the previous step appears under FSTYPE.$ lsblk -fNAME                         FSTYPE      LABEL UUID                                   MOUNTPOINTxvda                                                                            ├─xvda1                                                                         └─xvda2                      xfs               66e5e079-770e-4359-a9da-5205c3d8d5af   /xvdh                                                                            └─xvdh1                      LVM2_member       0UnOic-e2ng-XxH5-z0UW-7aTh-RxQK-KMrDqo  └─examplegroup1-lvexample1 xfs              5db36052-81d5-4762-8502-6986ff3964e7Run the mount command to mount the file system on the mount directory that you created in the previous step:$ sudo mount /dev/examplegroup1/lvexample1 /mnt12.    Edit the mount options in the /etc/fstab file so that the new mount persists after reboot./dev/examplegroup1/lvexample1 /mnt1   xfs     defaults,nofail   0   0Note: If your file system type is not xfs, then replace xfs with the type in the /etc/fstab file.Extend the logical volumeThere are two options for extending logical volumes:Option 1: Increase the size of the existing EBS volume.Option 2: Add additional EBS volumes to your volume group.Option 1: Increase the size of the existing EBS volumeNote: When increasing the size of your existing volume, size changes usually take effect within seconds when the volume enters the optimizing state. The volume's performance is affected while in the optimizing state, but doesn't fall below the source configuration specification. Depending on your volume type, performance changes might take from a few minutes to a few hours. For more information, see Monitor the progress of volume modifications.1.    Modify the size of the existing EBS volume.2.    Install the growpart utility:$ sudo yum install cloud-utils-growpartNote: Use the following command to install the growpart utility on Debian or Ubuntu-based systems:$ sudo apt install -y cloud-guest-utils3.    Run the growpart command to extend the partition, and then run the pvresize command to resize the PV. In the following example, partition /dev/xvdh1 is extended:$ sudo growpart /dev/xvdh 1                      CHANGED: disk=/dev/xvdh partition=1: start=2048 old: size=20971519,end=16777182 new: size=41940958,end=41943006$ sudo pvresize /dev/xvdh1                         Physical volume "/dev/xvdh1" changed  1 physical volume(s) resized or updated / 0 physical volume(s) not resizedUse the pvs or pvdisplay to view the physical volume details:$ sudo pvs  PV         VG            Fmt  Attr PSize   PFree  /dev/xvdh1 examplegroup1 lvm2 a--  <20.00g <13.00gUse the vgs or vgdisplay to view the volume group details:$ sudo vgs  VG            #PV #LV #SN Attr   VSize   VFree  examplegroup1   1   1   0 wz--n- <20.00g <13.00g4.    Run the lvextend command to extend the logical volume:$ sudo lvextend -L 19G /dev/examplegroup1/lvexample1Use the lvs or lvdisplay command to view the logical volume details:$ sudo lvs  LV        VG            Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert  lvexample1 examplegroup1 -wi-a----- 19.00g5.    Extend the file system:Note: Make sure that you enter the correct file system type in the /etc/fstab entry.Ext2, Ext3, and Ext4 file systems:$ sudo resize2fs /dev/examplegroup1/lvexample1XFS file systems:$ sudo yum install x    rogs$ sudo xfs_growfs /dev/examplegroup1/lvexample1Option 2: Add additional EBS volumes to the volume group1.    Create another EBS volume of 10 GB, and then attach the volume to the instance. Create a partition on the device /dev/xvdi following step 3 of Create physical volumes on the partition of your EBS volume. Then run the pvcreate command. In the following example, the volume's block device name is /dev/xvdi1.$ sudo pvcreate /dev/xvdi1Physical volume "/dev/xvdi1" successfully created.2.    Use the vgextend command to extend the volume group and add the new volume. The following example extends the volume group examplegroup1 to include the volume /dev/xvdi1:$ sudo vgextend examplegroup1 /dev/xvdi1  Volume group "examplegroup2" successfully extendedTo confirm the extension, run the vgs or vgdisplay command. The following example shows that there are now two PVs in the examplegroup1 volume group:$ sudo vgs  VG                       #PV #LV #SN Attr   VSize   VFree  examplegroup1            2   1   0 wz--n-  29.99g   20.99g3.    Run the lvextend command to extend the logical volume:$ sudo lvextend -L 29G /dev/examplegroup1/lvexample14.    Resize the file system:Note: Depending on your use case, follow the steps for XFS or Ext2, Ext3, and Ext4 file systems.Ext2, Ext3, and Ext4 file systems:$ sudo resize2fs /dev/examplegroup1/lvexample1XFS file systems:$ sudo xfs_growfs /dev/examplegroup1/lvexample1Related informationHow do I create an LVM logical volume on an entire EBS volume?Follow"


Question: How do I resolve "Page AC_LightningAdapter does not exist" errors and "You can't access Lightning Experience" errors when using the Amazon Connect CTI Adapter?

"I receive one of the following errors whenever I try to open the Amazon Connect Contact Control Panel (CCP) in Salesforce: "Page AC_LightningAdapter does not exist" or "You can't access Lighting Experience." Why is the Amazon Connect CTI Adapter returning these errors, and how do I resolve the issue?"

Answer: "I receive one of the following errors whenever I try to open the Amazon Connect Contact Control Panel (CCP) in Salesforce: "Page AC_LightningAdapter does not exist" or "You can't access Lighting Experience." Why is the Amazon Connect CTI Adapter returning these errors, and how do I resolve the issue?Short descriptionIf the CTI Adapter URL that's configured in your Salesforce call center is incorrect, then the Salesforce service console doesn't display the Amazon Connect CCP. Instead, Salesforce displays one of the following errors:"Page AC_LightningAdapter does not exist""You can't access Lightning Experience"To resolve these errors, copy the full URL of the Lightning Adapter Visualforce page (CTI Adapter URL) into your Salesforce call center.ResolutionVerify your Lighting Adapter URL1.    Open the Salesforce service console.2.    Choose Setup.3.    In the Quick Find text box, enter visual. Then, choose Visualforce Pages.4.    In the Name column, select the AC_LightningAdapter link.5.    Choose Preview. A new browser tab opens that contains your Lighting Adapter Visualforce page's URL.6.    Copy the URL.Example Lightning Adapter Visualforce page URLhttps://yourdomain-dev-ed--amazonconnect.visualforce.com/apex/AC_LightningAdapterAdd the Lighting Adapter URL to your Salesforce call center1.    In the Salesforce service console, choose Setup.2.    In the Quick Find text box, enter call. Then, choose Call Centers.Note: If this is your first time opening the Call Centers page, then the Say Hello to Salesforce Call Center page opens. Choose Continue if this happens.3.    Choose AC Lightning Adapter.4.    Choose Edit.5.    In the General Information section, for CTI Adapter URL, enter the Lighting Adapter URL that you copied previously.6.    Choose Save.Test the setupRefresh the Salesforce service console page in your browser. If the CTI Adapter URL is configured correctly, then the CCP appears in the Salesforce service console.Related informationAmazon Connect Salesforce CTI Adapter Installation GuideFollow"


Question: How do I resolve the "java.lang.OutOfMemoryError: Java heap space" error in an AWS Glue Spark job?

My AWS Glue job fails with "Command failed with exit code 1". The Amazon CloudWatch Logs shows the "java.lang.OutOfMemoryError: Java heap space" error.

Answer: "My AWS Glue job fails with "Command failed with exit code 1". The Amazon CloudWatch Logs shows the "java.lang.OutOfMemoryError: Java heap space" error.Short descriptionThe "java.lang.OutOfMemoryError: Java heap space" error indicates that that a driver or executor is running out of JVM memory. To determine whether a driver or an executor causes the OOM, see Debugging OOM exceptions and job abnormalities.Note: The following resolution is for driver OOM exceptions only.Driver OOM exceptions are caused by the following:The AWS Glue Spark job reads a large number of small files from Amazon Simple Storage Service (Amazon S3)Driver intensive operations like collect(), Broadcast joins and Shared variableResolutionResolve driver OOM exceptions caused by a large number of small filesTo resolve driver OOM exceptions that are caused by a large number of small files with DynamicFrames, use one or more of the following methods.Activate useS3ListImplementationWhen it lists files, AWS Glue creates a file index in the driver memory lists. When you set useS3ListImplementation to True, AWS Glue doesn't cache the list of files in the memory all at once. Instead, AWS Glue caches the list in batches. This means that the driver is less likely to run out of memory.See the following example of how to activate useS3ListImplementation with from_catalog:datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "database", table_name = "table", additional_options = {'useS3ListImplementation': True}, transformation_ctx = "datasource0")See the following example that activates useS3ListImplementation with from_options:datasource0 = glueContext.create_dynamic_frame.from_options(connection_type="s3", connection_options = {"paths": ["s3://input_path"], "useS3ListImplementation":True,"recurse":True}, format="json")The useS3ListImplementation feature is an implementation of the Amazon S3 ListKeys operation. This splits large results sets into multiple responses. It's a best practice to use useS3ListImplementation with job bookmarks.GroupingA Spark application processes every small file using a different Spark task. This can lead to OOM because the driver stores and keeps track of the location and task information. When you activate the grouping feature, tasks process a group of multiple files instead of individual files. Grouping is automatically turned on when you use dynamic frames, and when the Amazon S3 dataset has more than 50,000 files. For more information, see Reading input files in larger groups.Filtering with Push Down PredicatesReduce the number of Amazon S3 files and Amazon S3 partitions that the AWS Glue job reads by using push down predicates. This prunes the unnecessary partitions from the AWS Glue table before the underlying data is read. For more information, see Pre-filtering using pushdown predicates.Driver OOM exceptions caused by driver heavy operationsResolve driver OOM exceptions that are caused by driver heavy operations by using one of the following methods.Be mindful of driver intensive operationscollect() is a Spark operation that collects the results from workers, and then returns them back to the driver as a single object. The results can be very large, and that overwhelms the driver. By default, the Spark configuration spark.driver.maxResultSize is set to 1 GB, and helps to protect the driver from being overwhelmed.So, limit these actions, and instead use actions like take(), takeSample() or isEmpty() where possible.Also, be aware that broadcast joins in Spark can cause OOM errors if the relation (table) is bigger than driver’s available memory. Before a relation is broadcasted to executors, it's materialized at the driver node. If multiple tables are being broadcasted, or the relation is too large, then the driver might run into memory shortage. Use the Spark configuration spark.sql.autoBroadcastJoinThreshold and Spark Join Hints to control this.Regularly destroy shared variablesBe sure to use shared variables carefully. Destroy shared variables when you no longer need them because they can cause Spark driver OOM exceptions. There are two types of shared variables, broadcast variables and accumulators.Broadcast variables are read-only data that are sent to executors only once. They are a good solution for storing immutable reference data like small dictionary or small tables shared among all executors.Accumulators provide a writable copy across Spark executors, and can be used to implement distributed counters (as in MapReduce) or sums.Additional troubleshootingAnalyze your dataset and select the right worker type for your job. Consider scaling up to G.1X or G.2X.Use CloudWatch Logs and CloudWatch metrics to analyze driver memory. Set up CloudWatch alarms to alert you when specific thresholds are breached in your job.Turn on Spark UI for your AWS Glue job and view the Spark Event logs to understand why the OOM is occurring. For more information, see Monitoring jobs using the Apache Spark web UI.Related informationOptimize memory management in AWS GlueReading input files in larger groupsBest practices for successfully managing memory for Apache Spark applications on Amazon EMRMonitoring jobs using the Apache Spark web UIFollow"


Question: Why can't I connect to my ElastiCache for Redis cluster?

I can't connect to my Amazon ElastiCache for Redis cluster. How can I troubleshoot this?

Answer: "I can't connect to my Amazon ElastiCache for Redis cluster. How can I troubleshoot this?Short descriptionConnectivity issues can have multiple root causes. The most common issues are:The cluster isn't ready.The cluster is unhealthy.The network configuration is incorrect.The client configuration is incorrect.ResolutionVerify that the cluster is readyIf you recently created the cluster, verify that the cluster creation completed and that the cluster is ready to accept connections.Check the status of the cluster using the ElastiCache console, the AWS Command Line Interface (AWS CLI), or the ElastiCache API. Review the Status column for the following:If the Status column shows Available, the cluster is ready.If the Status column shows Creating, then cluster creation is still on-going. Wait a few minutes until it updates to Available.If the Status column shows Modifying, the cluster's configuration is updating. Wait a few minutes until the modifications finish and the status changes to Available.Verify that the cluster is healthyIn a healthy cluster, each individual node should be in the Available state. To verify the cluster's health, do the following:Check the status of each node in your cluster using the ElastiCache console, the AWS CLI, or the ElastiCache API.Amazon ElastiCache also provides an event log that you can check to see recent events in the cluster.Check the CloudWatch metrics for an overview of the cluster's health. For more information see, Monitoring best practices with Amazon ElastiCache for Redis using Amazon CloudWatch.In cases of hardware, instance, or software failures, Amazon ElastiCache tries to recover and restore the service to working order without user action. To make sure that the service can perform the recovery with minimal disruption, make sure that you properly configure the cluster and Redis client. For more information on failure scenarios, see Minimizing downtime in ElastiCache for Redis with Multi-AZ.Verify network-level connectivity between the cluster and the client resourceTo minimize latency, access ElastiCache from Amazon Elastic Compute Cloud (Amazon EC2) instances. Accessing Amazon ElastiCache from other resources within the same Amazon Virtual Private Cloud (Amazon VPC) also helps minimize latency. However, it's possible to connect from outside of the VPC, or even outside of AWS.For more information on connecting to ElastiCache, see the following:Access Patterns for accessing an ElastiCache cluster in an Amazon VPCAccessing your cluster or replication groupTo automate the connection process, use the VPC Network Access Analyzer service to troubleshoot connectivity issues between AWS resources.Verify that security groups and network ACLs allow connectionsPerform this step on the ElastiCache cluster and on the resource that's initiating the connection. Examples of client resources are:An Amazon EC2 instance.An AWS Lambda function.An Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS) container, and so on.In ElastiCache, make sure that the security groups are configured correctly. For other resources, verify security groups and network ACLs.Confirm the security group on the ElastiCache cluster1.    Select the cluster name from the Redis clusters menu, and then select the Network and security tab.2.    Verify that at least one of the associated security groups allows inbound connections from the client resource to the cluster on the cluster's port.3.    To confirm the port number, check any of the endpoints of the cluster. The endpoints are in the domain_name:port format.Note: The cluster's port is 6379/TCP by default. You can override the port number during cluster creation.4.    On the client resource, verify that the security groups allow outbound connections to the cluster's port and to the CIDR blocks of the cluster's subnets.5.    Verify that the network ACLs allow outgoing and incoming connections between the client and the cluster. The default Network ACLs usually allow all connections.Note: You can use the VPC Network Access Analyzer service to troubleshoot security group and network ACL configurations.Identify the correct endpoint for connectionsThe recommended connection endpoints differ between cluster configurations. For more information on finding the correct endpoints and possible configurations, see Finding connection endpoints.Verify that DNS resolution works on the client sideDNS issues are commonly identified by the Name or service not known and NXDOMAIN error messages.$ nslookup nonexistent.1234id.clustercfg.euw1.cache.amazonaws.comServer:         172.31.0.2Address:        172.31.0.2#53** server can't find nonexistent.1234id.clustercfg.euw1.cache.amazonaws.com: NXDOMAIN$ redis-cli -h nonexistent.1234id.clustercfg.euw1.cache.amazonaws.comCould not connect to Redis at nonexistent.1234id.clustercfg.euw1.cache.amazonaws.com:6379: Name or service not knownIf you see the preceding errors, check the DNS attributes of the VPC client resource.It's a best practice to use the Amazon DNS server, if possible. For more information, see Amazon DNS server.Verify TCP connectivity between the client and RedisUse the curl or telnet command to establish a TCP connection and verify connectivity:$ curl -v telnet://test.1234id.clustercfg.euw1.cache.amazonaws.com:6379*  Trying 172.31.1.242:6379...* Connected to test.1234id.clustercfg.euw1.cache.amazonaws.com (172.31.1.242) port 6379 (#0)In the preceding example, the Connected keyword shows that the TCP connection works.If Connected doesn't appear in the command results, check the following:On the ElastiCache clusterSecurity Groups must allow connections on the cluster's port. Verify the port value on the cluster configuration page (default is TCP/6379). For more information, see Modifying an ElastiCache cluster.The cluster and all of its shards and nodes must be in the available state. For more information, see Viewing a cluster's details.On the client resourceSecurity groups must allow outgoing connections to the cluster's IP and port.Routing tables must have the appropriate routes so that the cluster is reachable.The resource can be in the same VPC. Or, if the resource is in another VPC or outside of AWS, make sure that it has the appropriate connection configured. This might be VPN or VPC-Peering, DirectLink, and so on. For more information, see Accessing your cluster or replication group.Note: Amazon ElastiCache is designed to be accessed from the same VPC to ensure low latency. Connections outside of the VPC introduce extra latency. This extra latency is especially common with connections using the public Internet either directly or through tunneling. Because Redis is very latency-sensitive, the extra latency might cause connectivity and time-out issues.The VPC Reachability Analyzer is a tool to help determine what is blocking access.Troubleshoot connecting to clusters with in-transit encryptionIn-transit encryption works by sending Redis traffic over TLS. The client must have TLS support for the connection to work.$ redis-cli -h encrypted.1234id.clustercfg.euw1.cache.amazonaws.comIf you have redis-cli installed and it has TLS support, then add the --tls argument to the command:$ redis-cli -h encrypted.1234id.clustercfg.euw1.cache.amazonaws.com --tlsencrypted.1234id.clustercfg.euw1.cache.amazonaws.com:6379>If redis-cli was compiled without TLS support, the following error displays:$ redis-cli -h encrypted.1234id.clustercfg.euw1.cache.amazonaws.com --tlsUnrecognized option or bad number of args for: '--tls'To troubleshoot the preceding error, do one of the following:It's a best practice to compile redis-cli with TLS support if you will use redis-cli later. For steps for Amazon Linux 2 and Amazon Linux, see Download and install redis-cli in Step 4: Connect to the cluster's node.-or-Use an alternate command, such as openssl. The openssl command is available on most systems and is useful if a redis-cli with TLS support isn't available. The following is an example of the openssl command:$ openssl s_client -connect encrypted.1234id.clustercfg.euw1.cache.amazonaws.com:6379CONNECTED(00000003)----- omitted --------INFO# Serverredis_version:6.2.6----- omitted -----For more information, see the Connecting to an Encryption/Authentication enabled cluster section in Step 4: Connect to the cluster's node.Troubleshoot connecting to clusters with authenticationredis-cliAll clusters with authentication require TLS. The redis-cli command requires both the --tls and --askpass (or -a) arguments.If the --askpass argument isn't provided, then you receive the following output:$ redis-cli -h auth-cluster.1234id.clustercfg.euw1.cache.amazonaws.com --tlsauth-cluster.1234id.clustercfg.euw1.cache.amazonaws.com:6379> INFO # or any other Redis commandNOAUTH Authentication required.If the command contains an incorrect password, then you receive the following output:$ redis-cli -h auth-cluster.1234id.clustercfg.euw1.cache.amazonaws.com --tls --askpassPlease input password: *************Warning: AUTH failedThe following is an example of a Redis command with the correct password that worked as expected:$ redis-cli -h auth-cluster.1234id.clustercfg.euw1.cache.amazonaws.com --tls --askpassPlease input password: ******************auth-cluster.1234id.clustercfg.euw1.cache.amazonaws.com:6379> INFO# Serverredis_version:6.2.6----- omitted -----opensslYou can test connectivity using the openssl command. Use this command for debugging purposes only:$ openssl s_client -connect master.auth-cluster.3i1yig.euw1.cache.amazonaws.com:6379CONNECTED(00000003)----- omitted --------AUTH topsecretpassword+OKINFO# Serverredis_version:6.2.6----- omitted -----For more information, see the Connecting to an Encryption/Authentication enabled cluster section in Step 4: Connect to the cluster's node.For additional details on troubleshooting ElastiCache connectivity, see Troubleshooting.Follow"


Question: I'm receiving "Connection refused" or "Connection timed out" errors when trying to connect to my EC2 instance with SSH. How do I resolve this?

I'm receiving "Connection refused" or "Connection timed out" errors when trying to connect to my Amazon Elastic Compute Cloud (Amazon EC2) instance using SSH.

Answer: "I'm receiving "Connection refused" or "Connection timed out" errors when trying to connect to my Amazon Elastic Compute Cloud (Amazon EC2) instance using SSH.Short descriptionError message: "ssh: connect to host ec2-X-X-X-X.compute-1.amazonaws.com port 22: Connection timed out". This error message comes from the SSH client. The error indicates that the server didn't respond to the client and the client program gave up (timed out). The following are common causes for this error:The security group or network ACL doesn't allow access.There's a firewall on the instance's operating system.There's a firewall between the client and the server.The host doesn't exist.Error message: "ssh: connect to host ec2-X-X-X-X.compute-1.amazonaws.com port 22: Connection refused". This message comes remotely from a host. The following are common causes for this error:The host reached the instance but there was no service listening on the SSH port.A firewall blocked and was set to reject the package instead of dropping it.ResolutionFor the “Connection timed out” error, verify the followingThe instance's IP address or hostname is correct.The instance is passing its health checks.The security group of the instance allows incoming traffic on TCP port 22.The network ACLs of instance subnet allows incoming traffic on TCP port 22 and allow ephemeral port for the outgoing traffic.The route table of the instance’s subnet is configured properly to provide connectivity between EC2 instance and the SSH client.There isn't a firewall blocking the connection between SSH client and the EC2 instance.SSH isn't blocked by TCP Wrappers in the instance.Note: The last two verification steps require OS-level access of the instance.For the “Connection refused” error, verify the followingThere's no firewall on the instance rejecting the SSH connection.The SSH daemon (sshd) is running and listening on port 22.Note: Both verification steps require OS-level access of the instance.If the instance passes both health checks, use one of the following four listed methods with your configurationMethod 1: Use the EC2 Serial Console for Linux.Method 2: Use AWS Systems Manager Session Manager.Method 3: Run the AWSSupport-TroubleshootSSH automation runbook.Method 4: Use a user data script.Method 1: Use the EC2 Serial Console for LinuxIf configured, you can use the EC2 Serial Console for Linux to troubleshoot OS-level issues on supported Nitro-based instance types. The serial console allows troubleshooting of boot issues, network configuration, and SSH configuration issues. The serial console is accessible using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).Before using the serial console, grant access to it at the account level. Then, create AWS Identity and Access Management (IAM) policies granting access to your IAM users.Note: Each instance using the serial console must include at least one password-based Linux user with sudo access.For more information on configuring the EC2 Serial Console for Linux, see Configure access to the EC2 Serial Console. If there isn’t a Linux account with a login password configured, you must run ssm-user to reset the password for an account with sudo access. For more information on running ssm-user commands, see the section Managing ssm-user sudo account permissions on Linux and macOS.After configuration, connect to the EC2 instance through the EC2 serial console using a password-configured Linux user.Next, run the following commands. These commands verify that the SSH connections aren't being blocked by the OS firewall or TCP wrapper. The commands also verify that the sshd service is running and listening on port 22.1.    If you have iptables rules configured, then run following command to add a rule in iptables accepting all SSH connections on default port 22:$ sudo iptables -I INPUT 1 -p tcp --dport 22 -j ACCEPTBecause it’s a best practice to use security groups instead of an OS-based firewall, the firewall can be deactivated altogether. To deactivate the OS-based firewall, use one of the following set of commands, depending on your operating system:Important: The following commands flush all main iptables rules. They also add a rule allowing incoming SSH connections. Additionally, they change the default policy of the main chain to ACCEPT so that flushing the iptables rule doesn’t affect instance network connectivity. Because these commands flush all main iptables, they also flush any existing rules.Distributions that use UFW (Ubuntu, Debian)$ sudo iptables -F$ sudo iptables -P INPUT ACCEPT$ sudo ufw disableDistributions that use firewalld (Red Hat, CentOS)$ sudo iptables -F $ sudo iptables -P INPUT ACCEPT$ sudo systemctl disable firewalldNote: After you regain access to your instance, review your firewall configuration (UFW, firewalld, iptables).2.    Verify that SSH is running and verify that the SSH TCP port (22) is in listening state:$ sudo systemctl restart sshd$ sudo ss -tpln | grep -iE '22|ssh'LISTEN 0 128 *:22 *:* users:(("sshd",pid=1901,fd=3))LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1901,fd=4))Note: If your system doesn't have the ss command, you can use the legacy netstat command with the same syntax shown in the preceding example.3.    Make sure that the TCP wrapper isn’t blocking an SSH connection:$ if [[ $( cat /etc/hosts.[ad]* | grep -vE '^#' | awk 'NF' | wc -l) -ne 0 ]];\then sudo sed -i '1i sshd2 sshd : ALL: allow' /etc/hosts.allow; fi4.    Next, connect to the instance using SSH.5.    Disconnect the EC2 Serial Console session if it's no longer required.Method 2: Use AWS Systems Manager Session ManagerNote: To use this method, the instance must be an SSM managed instance and its SSM agent ping status must be Online. For more information on Session Manager and a complete list of prerequisites, see Setting up Session Manager.To confirm that SSH connections aren't being blocked by the firewall or TCP wrapper and that the sshd service is running and listening on port 22:1.    Open the AWS Systems Manager.2.    Start a session for the instance using Session Manager.3.    Follow steps 1 - 4 from Method 1: Use EC2 Serial console for Linux.4.    Close the Session Manager session if it's no longer required.Method 3: Run the AWSSupport-TroubleshootSSH runbookThe AWSSupport-TroubleshootSSH automation runbook installs the Amazon EC2Rescue tool for Linux on the instance. This tool checks for and attempts to fix issues that prevent a remote connection Linux host though SSH.To run the AWSSupport-TroubleshootSSH runbook:1.    Open the AWSSupport-TroubleshootSSH page.2.    Select Run this Automation (console).To learn more, see How can I use the AWSSupport-TroubleshootSSH Automation workflow to troubleshoot SSH connection issues?Method 4: Use a user data scriptIf none of the methods described are suitable in your environment, then use an EC2 user-data script. The EC2 user-data script turns off the OS-level firewall and the TCP wrapper, and then restarts the sshd service.Important:This procedure requires a stop and start of the EC2 instance. If the instance has any data stored on instance store volumes, that data is deleted after stopping the instance.If the instance is part of an Amazon EC2 Auto Scaling group, then terminating the instance might also stop instances within the Auto Scaling group.If the instance is launched by services that use AWS Auto Scaling, then terminating the instance might also stop instances within the Auto Scaling group.Instance termination depends on the instance scale-in protection settings for the Auto Scaling group. If the instance is part of an Auto Scaling group, temporarily remove the instance from the Auto Scaling group before starting the resolution steps.Stopping and starting the instance changes the public IP address of the instance. It's a best practice to use an Elastic IP address instead of a public IP address when routing external traffic to the instance.Follow these steps to configure user-data for the instance:1.    Open the Amazon EC2 console.2.    Choose Instances from the navigation pane, and then select the instance that you plan to connect to.3.    Stop the instance.4.    Choose Actions, Instance Settings, Edit User Data.5.    Copy the following user data script into the Edit User Data dialog box, and then choose Save.Content-Type: multipart/mixed; boundary="//"MIME-Version: 1.0 --//Content-Type: text/cloud-config; charset="us-ascii"MIME-Version: 1.0Content-Transfer-Encoding: 7bitContent-Disposition: attachment; filename="cloud-config.txt" #cloud-configcloud_final_modules:- [scripts-user, always] --//Content-Type:    text/x-shellscript; charset="us-ascii"MIME-Version: 1.0Content-Transfer-Encoding: 7bitContent-Disposition: attachment; filename="userdata.txt" #!/bin/bashiptables -P INPUT ACCEPTiptables -Fsystemctl restart sshd.service || service sshd restartif [[ $( cat /etc/hosts.[ad]* | grep -vE '^#' | awk 'NF' | wc -l) -ne 0 ]];\then sudo sed -i '1i sshd2 sshd : ALL: allow' /etc/hosts.allow; fi--//6.    Connect to the instance using SSH.7.    The preceding user data script is set to run on every reboot of the instance. After regaining access to the instance, remove the user data script.Note: The preceding command flushes all main iptables rules. After regaining access to the instance, review the firewall configuration for accuracy (for example, UFW, firewalld, iptables).To remove user data:1.    Complete steps 1–4 in the Method 4: Use a user data script section.2.    Delete the user data script in the Edit User Data dialog box.Related informationError connecting to your instance: Connection timed outHow do I troubleshoot Amazon EC2 instance connection timeout errors from the internet?Why can't I connect to my Amazon EC2 Linux instance using SSH?Why is my EC2 Linux instance unreachable and failing one or both of its status checks?Follow"