Question: My EC2 Linux instance failed the instance status check due to operating system issues. How do I troubleshoot this?

My Amazon Elastic Compute Cloud (Amazon EC2) Linux instance failed the instance status check due to operating system issues. Now it doesn't boot successfully. How can I resolve this?

Answer: "My Amazon Elastic Compute Cloud (Amazon EC2) Linux instance failed the instance status check due to operating system issues. Now it doesn't boot successfully. How can I resolve this?Short descriptionYour EC2 Linux instance might fail the instance status check for the following reasons:You updated the kernel and the new kernel didn’t boot.The file system entries in /etc/fstab are incorrect or the file system is corrupted.There are incorrect network configurations on the instance.ResolutionThere are 3 methods for troubleshooting OS issues.Important:Methods 2 and 3 require a stop and start of the instance. Be aware of the following:If your instance is instance store-backed or has instance store volumes containing data, the data is lost when you stop the instance. For more information, see Determine the root device type of your instance.If your instance is part of an Amazon EC2 Auto Scaling group, stopping the instance might terminate the instance. If you launched the instance with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk, then your instance might be part of an AWS Auto Scaling group. Instance termination in this scenario depends on the instance scale-in protection settings for your Auto Scaling group. If your instance is part of an Auto Scaling group, then temporarily remove the instance from the Auto Scaling group before starting the resolution steps.Stopping and starting the instance changes the public IP address of your instance. It's a best practice to use an Elastic IP address instead of a public IP address when routing external traffic to your instance. If you are using Amazon Route 53, you might have to update the Route 53 DNS records when the public IP changes.Method 1: Use the EC2 Serial ConsoleIf you enabled EC2 Serial Console for Linux, then you can use it to troubleshoot supported Nitro-based instance types. The serial console helps you troubleshoot boot issues, network configuration, and SSH configuration issues. The serial console connects to your instance without the need for a working network connection. You can access the serial console using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).Before using the serial console, grant access to it at the account level. Then, create AWS Identity and Access Management (IAM) policies granting access to your IAM users. Also, every instance using the serial console must include at least one password-based user. If your instance is unreachable and you haven’t configured access to the serial console, follow the instructions in Method 2. For information on configuring the EC2 Serial Console for Linux, see Configure access to the EC2 Serial Console.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.Method 2: Run the EC2Rescue for Linux toolEC2Rescue for Linux automates diagnosing and troubleshooting operating system issues on unreachable instances. For more information, see How do I use EC2Rescue for Linux to troubleshoot operating system-level issues?Method 3: Manually correct errors using a rescue instance1.    Launch a new EC2 instance in your virtual private cloud (VPC) using the same Amazon Machine Image (AMI). Launch the new instance in the same Availability Zone as the impaired instance. The new instance becomes your rescue instance.Or, use an existing instance that you can access, if it uses the same AMI and is in the same Availability Zone as your impaired instance.2.    Stop the impaired instance.3.    Detach the Amazon Elastic Block Store (Amazon EBS) root volume (/dev/xvda or /dev/sda1) from your impaired instance. Note the device name (/dev/xvda or /dev/sda1) of your root volume.4.    Attach the volume as a secondary device ( /dev/sdf) to the rescue instance.5.    Connect to your rescue instance using SSH.6.    Create a mount point directory (/rescue) for the new volume attached to the rescue instance:$ sudo mkdir /rescue7.    Mount the volume at the directory you created in step 6:$ sudo mount /dev/xvdf1 /rescueNote: The device (/dev/xvdf1) might be attached to the rescue instance with a different  device name. Use the lsblk command to view your available disk devices along with their mount points to determine the correct device names.8.    If you haven't already done so, retrieve the system log of the instance to verify what error is occurring. The next steps depend on the error message listed in the system log. The following is a list of common errors that can cause instance status check failure. For additional errors, see Troubleshooting system log errors for Linux-based instances.Kernel PanicIf a Kernel Panic error message is in the system log, then the kernel might not have the vmlinuz or initramfs files. The vmlinuz and initramfs files are necessary to boot successfully.1.    Run the following commands:cd /rescue/bootls -l2.    Check the output to verify that there are vmlinuz and initramfs files corresponding to the kernel version you intend to boot.The following output example is for an Amazon Linux 2 instance with kernel version, 4.14.165-131.185.amzn2.x86_64. The /boot directory has the files initramfs-4.14.165-131.185.amzn2.x86_64.img and vmlinuz-4.14.165-131.185.amzn2.x86_64, so it will boot successfully.uname -r4.14.165-131.185.amzn2.x86_64cd /boot; ls -ltotal 39960-rw-r--r-- 1 root root      119960 Jan 15 14:34 config-4.14.165-131.185.amzn2.x86_64drwxr-xr-x 3 root root     17 Feb 12 04:06 efidrwx------ 5 root root       79 Feb 12 04:08 grub2-rw------- 1 root root 31336757 Feb 12 04:08 initramfs-4.14.165-131.185.amzn2.x86_64.img-rw-r--r-- 1 root root    669087 Feb 12 04:08 initrd-plymouth.img-rw-r--r-- 1 root root    235041 Jan 15 14:34 symvers-4.14.165-131.185.amzn2.x86_64.gz-rw------- 1 root root   2823838 Jan 15 14:34 System.map-4.14.165-131.185.amzn2.x86_64-rwxr-xr-x 1 root root   5718992 Jan 15 14:34 vmlinuz-4.14.165-131.185.amzn2.x86_643.    If the initramfs and or the vmlinuz files aren't present, try booting the instance using a previous kernel that has both of these files. For instructions on how to boot your instance using a previous kernel, see How do I revert to a known stable kernel after an update prevents my Amazon EC2 instance from rebooting successfully?4.    Run the unmount command to unmount the secondary device from your rescue instance:$ sudo umount /rescueIf the unmount operation isn't successful, then you might have to stop or reboot the rescue instance to enable a clean unmount.5.    Detach the secondary volume (/dev/sdf) from the rescue instance, and then attach it to the original instance as /dev/xvda (root volume).6.    Start the instance, and then verify if the instance is responsive.For additional information on resolving kernel panic errors, see I'm receiving a "Kernel panic" error after I've upgraded the kernel or tried to reboot my EC2 Linux instance. How can I fix this?Failed to mount or Dependency failedIf you see errors such as "Failed to mount" or "Dependency failed" in your system log, the /etc/fstab file might have incorrect mount point entries.1.    Verify the mount point entries in the /etc/fstab are correct. For information on correcting the /etc/fstab file entries, see the Auto mount failures because of incorrect entries in the /etc/fstab file section of Why is my EC2 instance not booting and going into emergency mode?2.    It's a best practice to run the fsck or xfs_repair tool to correct any file system errors. If there are inconsistencies in the file system, the fsck or xfs_repair tool corrects them.Note: Create a backup of your file system before running the fsck or xfs_repair tool.Run the unmount command to unmount your mount point before running the fsck or xfs_repair tool:$ sudo umount /rescueRun the fsk or xfs_repair tool, depending on your file system.For ext4 file systems:$ sudo fsck /dev/sdffsck from util-linux 2.30.2e2fsck 1.42.9 (28-Dec-2013)/dev/sdf: clean, 11/6553600 files,459544/26214400 blocksFor XFS file systems:$ sudo xfs_repair /dev/sdfxfs_repair /dev/xvdfPhase 1 - find and verify superblock...Phase 2 - using internal log        - zero log...        - scan filesystem freespace and inode maps...        - found root inode chunkPhase 3 - for each AG...        - scan and clear agi unlinked lists...        - process known inodes and perform inode discovery...        - agno = 0        - agno = 1        - agno = 2        - agno = 3        - process newly discovered inodes...Phase 4 - check for duplicate blocks...        - setting up duplicate extent list...        - check for inodes claiming duplicate blocks...        - agno = 0        - agno = 1        - agno = 2        - agno = 3Phase 5 - rebuild AG headers and trees...        - reset superblock...Phase 6 - check inode connectivity...        - resetting contents of realtime bitmap and summary inodes        - traversing filesystem ...        - traversal finished ...        - moving disconnected inodes to lost+found ...Phase 7 - verify and correct link counts...done3.    Detach the secondary volume (/dev/sdf) from the rescue instance, and then attach it to the original instance as /dev/xvda (root volume).4.    Start the instance and then verify if the instance is responsive.Bringing up interface eth0: failedIf you see the error "Bringing up interface eth0: failed", verify that the ifcfg-eth0 file has the correct network entries. The network configuration file corresponding to the primary interface, eth0, is located at /etc/sysconfig/network-scripts/ifcfg-eth0. If the device name of your primary interface isn't eth0, then there is a file that begins with ifcfg and is followed by the name of your device in the directory /etc/sysconfig/network-scripts on the instance.1.    Run the cat command to view the network configuration file for the primary interface, eth0.The following are the correct entries for the network configuration file located in /etc/sysconfig/network-scripts/ifcfg-eth0.Note: Replace eth0 in the following command with the name of your primary interface, if different.$ sudo cat /etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0BOOTPROTO=dhcpONBOOT=yesTYPE=EthernetUSERCTL=yesPEERDNS=yesDHCPV6C=yesDHCPV6C_OPTIONS=-nwPERSISTENT_DHCLIENT=yesRES_OPTIONS="timeout:2 attempts:5"DHCP_ARP_CHECK=no2.    Verify that ONBOOT is set to yes, as shown in the preceding example. If ONBOOT isn't set to yes, then eth0 (or your primary network interface) isn't configured to come up at boot.To change the ONBOOT value:Open the file in an editor. In the example given here, the vi editor is used.$ sudo vi /etc/sysconfig/network-scripts/ifcfg-eth0Press I to insert.Scroll the cursor to the ONBOOT entry, and then change the value to yes.Save and exit the file by pressing :wq!.3.    Run the unmount command to unmount the secondary device from your rescue instance:$ sudo umount /rescueIf the unmount operation isn't successful, then you might have to stop or reboot the rescue instance to enable a clean unmount.4.    Detach the secondary volume (/dev/sdf) from the rescue instance, and then attach it to the original instance as /dev/xvda (root volume).5.    Start the instance and then verify if the instance is responsiveRelated informationWhy is my EC2 Linux instance unreachable and failing one or both of its status checks?Troubleshoot instances with failed status checksWhy is my Linux instance not booting after I changed its type to a Nitro-based instance type?Follow"


Question: How can I get maximum I/O performance from my EBS volumes hosted on EC2 Nitro-based instances?

I'm running my workload on Amazon Elastic Compute Cloud (Amazon EC2) Nitro-based instances. How can I make sure I'm getting the maximum I/O performance from Amazon Elastic Block Store (Amazon EBS) volumes hosted on my instances?

Answer: "I'm running my workload on Amazon Elastic Compute Cloud (Amazon EC2) Nitro-based instances. How can I make sure I'm getting the maximum I/O performance from Amazon Elastic Block Store (Amazon EBS) volumes hosted on my instances?Resolution1.    Verify that your EBS volume isn't hitting IOPS limits. Latency increases if your volume hits its IOPS limit. Increased latency negatively impacts performance. For information on optimizing volume performance, see  How do I optimize the performance of my Amazon EBS Provisioned IOPS volumes?If you're using a GP2 volume check that your volume hasn't exhausted the burst credits.2.    To benefit from the performance made possible by NVMe storage, you must run one of the following operating systems. Or, make sure that the kernel version supports an I/O scheduler with multi-queue capability. The most commonly used multi-queue I/O schedulers are kyber, mq-deadline, and bfq (budget fair queue).Amazon Linux AMI or higher and kernel 4.12 or newerCentOS – 7.0 or higher and kernel 3.10 or newerRed Hat - 7.0 or higher and kernel 3.10 or newerUbuntu 19.10 with kernel 5.0 or Ubuntu 18.04.03 with kernel 5.0 onwards multi-queue is enabled by defaultUbuntu – 16.04 or 16.10 multi-queue schedulers aren't kernel compiled and need separate module loadingSUSE 12 or SUSE 11 with SP3 or higherWindows Server 2008 R2, 2012 R2, and 2016 or higherNote: For operating systems such as Oracle, Linux, Debian, and so on, make sure that you use a kernel version that includes or supports a multi-queue I/O scheduler. The CentOS operating system and its kernel version supports a multi-queue I/O scheduler.If you're using an operating system older than those listed, you might see a slight decline in the I/O performance. This is because the I/O scheduler on CentOS 6 is a simple scheduler that doesn't support multiple queues. Nitro-based instances have multi-queue processing at the host level. This creates incompatibility between the scheduler at the operating system and host levels and might cause performance degradation.I/O (read or write) requests submitted to the EBS volume travel through several layers before the volume intercepts them. These layers include user space, the virtual file system, the page cache, the block layer, the scheduler, and the device driver. Tests and benchmarking results using blktrace, blkparse, and btt tools show a slight delay in the I/O scheduler (I2D) layer when using older kernel versions with non multi-queue schedulers on Nitro-based instances.The I/O scheduler plays an important role in merging, processing, and queuing I/O’s. In Nitro-based systems, EBS volumes are exposed as NVMe block devices. NVME natively supports multiple hardware submission and completion queues. CentOS 7 has the Multi-Queue Block I/O Queueing Mechanism (blk-mq) that allows device drivers to map I/O requests to multiple hardware or software queues. This improves the I/O performance on Nitro-based instances because of the kernel's multi-queuing capabilities. Earlier kernel versions don't have this capability. Because of this, it's a best practice to use a modern operating system with the latest kernel for maximum performance on Nitro-based systems.I/O scheduler on CentOS 6$cat /sys/block/xvdf/queue/scheduler noop anticipatory deadline [cfq]$cat config-2.6.32-754.30.2.el6.x86_64 | grep -i blk_mqNote: The config file in the CentOS 6 kernel doesn't return the blk_mq as it is using noop scheduler**.**I/O scheduler on CentOS 7, kernel 3.10 +$cat /sys/block/nvme1n1/queue/scheduler [none] mq-deadline kyberNote: Kyber and mq-deadline are multi-queue schedulers present only in higher kernel versions (3.10+ for CentOS 7).To update the scheduler at the operating system level during the run time of the EC2 instance, use the following command:#sudo echo 'kyber'> /sys/block/xvdf/queue/schedulerTo change the I/O scheduler permanently, modify the grub configuration and update the elevator parameter in the following format:Note: The following steps are for CentOS and RHEL.1.    Run the following command:#sudo vim /etc/default/grubGRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet elevator=kyber"2.    Run the following command:#sudo grub2-mkconfig -o /boot/grub2/grub.cfgThe I/O scheduler remains set even if the EC2 instance is rebooted.Follow"


Question: How can I rotate a Secrets Manager secret in a private VPC?

"I tried to rotate an AWS Secrets Manager secret for an AWS service in an Amazon Virtual Private Cloud (Amazon VPC). However, the operation failed and Amazon CloudWatch Logs show that the AWS Lambda task timed out."

Answer: "I tried to rotate an AWS Secrets Manager secret for an AWS service in an Amazon Virtual Private Cloud (Amazon VPC). However, the operation failed and Amazon CloudWatch Logs show that the AWS Lambda task timed out.Short descriptionSecrets Manager can't rotate secrets for AWS services running in Amazon VPC private subnets because these subnets don't have internet access.ResolutionFollow these instructions to configure an Amazon VPC interface endpoint to access your Secrets Manager Lambda function and private Amazon Relational Database Service (Amazon RDS) instance. In the following example, a private Aurora RDS instance in Amazon VPC named vpc-0abb11f5a28a8abe7 is used.Important:Before you begin, be sure that you've installed and configured the AWS Command Line Interface (AWS CLI).If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.Create SGs for the Secrets Manager VPC endpoint, RDS instance, and Lambda rotation functionFollow these instructions for creating security groups (SGs) using the AWS CLI.1.    SG for Secrets Manager Amazon VPC endpoint:Note: Replace vpc-id vpc-0abb11f5a28a8abe7 with your VPC ID.$ aws ec2 create-security-group --vpc-id vpc-0abb11f5a28a8abe7 --group-name SMVPCEndpointSG --description "secretsmanager VPCEndpoint SG"{    "GroupId": "sg-vpc-endpoint"}2.    Security group for Lambda rotation function:$ aws ec2 create-security-group --vpc-id vpc-0abb11f5a28a8abe7 --group-name LambdaFunctionSG --description "Lambda Rotation Function SG"{    "GroupId": "sg-lambda-function"}3.    (Optional) Create an SG for the RDS instance:Note: This step is required if your RDS instance uses only the default security group.$ aws ec2 create-security-group --vpc-id vpc-0abb11f5a28a8abe7 --group-name RDSInstanceSG --description "RDS Instance SG"{    "GroupId": "sg-rds-instance"}Add rules to Amazon VPC endpoint and RDS instance SGs1.    Get the CIDR range for your VPC:$ aws ec2 describe-vpcs --vpc-ids vpc-0a05c93c7ef7a8a1c --query 'Vpcs[].CidrBlock' --output text10.0.0.0/162.    Configure the security group rules for the Amazon VPC endpoint to allow inbound traffic on port 443 from your VPC: $ aws ec2 authorize-security-group-ingress --group-id sg-vpc-endpoint --protocol tcp --port 443 --cidr 10.0.0.0/163.    Configure the RDS instance SG to allow inbound connections from the Lambda function security group:Note:Replace your-rds-security-group with your SG (either an existing SG or the optional RDS instance SG).Replace your-db-port with the port that your database is configured to use.$ aws ec2 authorize-security-group-ingress --group-id your-rds-security-group --protocol tcp --port your-db-port --source-group sg-lambda-functionAttach SGs to AWS resources1.    If you created the optional RDS instance SG, modify the RDS instance configuration:Note: Replace your-existing-rds-security-groups with the group or groups attached to the RDS instance.$ aws rds modify-db-instance --db-instance-identifier your-rds-instance --vpc-security-group-ids sg-rds-instance your-existing-rds-security-groups2.    Follow the instructions to update the Lambda function configuration:$ aws lambda update-function-configuration --function-name your-lambda-function \--vpc-config SubnetIds=subnet-076c28105d486f3bd,subnet-0af00c796ccdc725f,SecurityGroupIds=sg-lambda-functionCreate an Amazon VPC interface endpoint for the Secrets Manager service and associate it with an SGFollow the instructions for creating an interface endpoint:Note: Replace your-region with your AWS Region and the subnet IDs used for your RDS instance.$ aws ec2 create-vpc-endpoint --vpc-id vpc-0abb11f5a28a8abe7 --vpc-endpoint-type Interface \--service-name com.amazonaws.your-region.secretsmanager --subnet-ids subnet-076c28105d486f3bd subnet-0af00c796ccdc725f \--security-group-ids sg-vpc-endpointImportant: Your Amazon VPC must have the DNS hostnames and DNS resolution attributes activated. For more information, see Viewing and updating DNS support for your VPC.Verify that Secrets Manager can rotate the secret1.    Follow the instructions for rotating the Secrets Manager secret:Note: Replace your-secret with your Secrets Manager secret. $ aws secretsmanager rotate-secret --secret-id your-secretSecrets Manager retries the previous rotation.Note: Because previous attempts to rotate the password were unsuccessful, you might receive an output similar to the following:An error occurred (InvalidRequestException) when calling the RotateSecret operation: A previous rotation isn't complete. That rotation will be reattempted.2.    Monitor the function in the AWS Lambda console. If the rotation is successful, then the Amazon CloudWatch log streams contain an entry similar to the following:[INFO] 2019-10-22T07:59:32.627Z 96179023-5b67-4e98-a057-885f68bc69f2 finishSecret: Successfully set AWSCURRENT stage to version 175b5e38-341f-4cd0-8c58-2b1e49769642 for secret arn:aws:secretsmanager:your-region:your-account:secret:your-secret3.    Retrieve the Secrets Manager secret to confirm that it rotated successfully:Note: Replace your-secret-arn with your Secrets Manager secret ARN.aws secretsmanager get-secret-value --secret-id your-secret-arnNote: The Secrets Manager rotation function runs asynchronously in the background. The rotation function can take several minutes to complete.Related informationHow can I rotate an AWS Secrets Manager secret for a DB connection that requires an SSL connection?Follow"


Question: Why do I see write latency spikes every five minutes after I upgrade my Amazon RDS for PostgreSQL instance to version 11 or higher?

I've upgraded my Amazon Relational Database Service (Amazon RDS) instance running PostgreSQL to version 11 or higher. Why do I see Write Latency spike in Amazon CloudWatch every five minutes?

Answer: "I've upgraded my Amazon Relational Database Service (Amazon RDS) instance running PostgreSQL to version 11 or higher. Why do I see Write Latency spike in Amazon CloudWatch every five minutes?ResolutionWith an idle Amazon Relational Database Service (Amazon RDS) for PostgreSQL instance, you might notice a spike in the Amazon CloudWatch metric Write Latency every five minutes. This spike correlates with a spike in the Enhanced Monitoring metric Write Total to approximately 64 MB after you upgrade to PostgreSQL version 11 or later. The value of the wal_segment_size parameter becomes 64 MB after the upgrade. These spikes might not be noticeable with version 10 or earlier because the value of wal_segment_size is 16 MB. For more information, see the update for September 7, 2018 in Amazon RDS - Document history.The archive_timeout configuration in RDS for PostgreSQL is set to 5 minutes. This setting means that the archival process copies the Write-Ahead Logging (WAL) segments to be archived to Amazon Simple Storage Service (Amazon S3) every five minutes. In an idle system, this copy process is usually the only I/O operation, and so this operation might be visibly dominant in the CloudWatch graphs. However, you might not see this pattern in a busy system. For example, suppose that you run following workload for 30 minutes on a db.t3.small instance to simulate a busy system on an RDS for PostgreSQL instance with a 20 GB Amazon Elastic Block Store (Amazon EBS) volume:#pgbench --host=$HOST --username=$USER --port=$PORT --protocol=simple  --progress=2  --client=1 --jobs=1 $DB -T 1800#pgbench --initialize --fillfactor=90 --scale=100  --port=$PORT --host=$HOST --username=$USER $DBWith this workload, you don't see spikes in the CloudWatch Write Latency metric.But suppose that you have the following use cases:You have an I/O request that takes 10 milliseconds to complete, and nine other I/O requests that take 1 millisecond each to complete. The average latency of these requests is calculated as follows:Average latency = (10 + (9 * 1)) / 10 = 1.9 millisecondsYou have an I/O request that takes 10 milliseconds to complete, and you have no other I/O requests. The average latency in this case is calculated as follows:Average latency = 10 / 1 = 10 millisecondsBoth use cases include the same I/O request that takes 10 milliseconds to complete. However, when you calculate the average latency, the slow request stands out in the second use case where you have fewer I/O requests running along with the slow request. If an idle system has one slow I/O request due to the high block size, the latency is calculated only from this request. In a busy system with multiple I/O requests, most with smaller block sizes, the average latency is calculated from all these requests.In these cases, you might see a Write Latency spike every five minutes in an idle RDS for PostgreSQL system. If your database instance runs a busy workload, then you might not see these spikes.Example: Suppose that you have two t2.small Amazon Elastic Compute Cloud (Amazon EC2) instances each with 8 gp2 volumes.Run the following command on crontab to create a 64 MB file with a block size of 64 MB every five minutes in the first Amazon EC2 instance:*/5 * * * * dd if=/dev/zero of=/tmp/big_file.txt bs=64MB count=1 oflag=dsync ; rm -rf /tmp/big_file.txtNote: Be sure to replace big_file.txt in the command with the name of your file.Also, run the following command on crontab to create 100 files each with a block size of 8 KB every one minute in the second Amazon EC2 instance:* * * * * for i in {1..100} ; do dd if=/dev/zero of=/tmp/small_file.txt bs=8k count=100 oflag=dsync ; rm -rf /tmp/small_file.txt ; doneNote: Be sure to replace small_file.txt in the command with the name of your file.You might notice that the second EC2 instance shows higher Writer Latency spikes in CloudWatch than the first EC2 instance.Related informationBest practices for working with PostgreSQLFollow"


Question: How do I enable debug logs in my Athena JDBC/ODBC driver?

I want to enable debug logs in my Amazon Athena Java Database Connectivity (JDBC)/Open Database Connectivity (ODBC) driver.

Answer: "I want to enable debug logs in my Amazon Athena Java Database Connectivity (JDBC)/Open Database Connectivity (ODBC) driver.ResolutionYou can configure logging in the Athena JDBC/ODBC driver to help you troubleshoot issues and errors.Important: Enable logging only for the required amount of time to capture an issue. Logging decreases performance and might take up a large quantity of disk space.JDBC driverFor information on activating logging in the JDBC driver, see Simba documentation for Configuring logging. To capture the debug logs, configure the following properties in your JDBC driver:For LogLevel, select 5For UseAwsLogger, select 1.Note: With this property, the JDBC driver logs information from AWS API calls as well.For LogPath, enter a location on your host machine where you want the driver logs to be captured.You can also activate logging by configuring similar options in your business intelligence tool.To be sure that the new settings take effect, restart your JDBC application, and then reconnect to the server. The Simba Athena JDBC Driver produces the following log files in the location specified in the LogPath property:AthenaJDBC_driver.log file with logs on driver activity that's not specific to the connection.AthenaJDBC_connection_[1234].log file for each connection made to the database. The number 1234 in the file name identifies the log file. This file logs driver activity that's specific to the connection.If the LogPath value isn't valid, then the driver sends the logged information to the standard output stream (System.out).ODBC driverFor information on activating logging for your ODBC driver, see Simba documentation for Simba Athena ODBC driver with SQL connector.To download the ODBC driver, see Connecting to Amazon Athena with ODBC.To activate logging in the ODBC driver for your Windows machine, do the following:From the Start menu, choose ODBC Data Sources.In the ODBC Data Source Administrator, choose the System DSN tab.Choose Add.Form the list of drivers, select Simba Athena ODBC Driver.Choose Finish.A new wizard opens where you can add information, such as AWS Region, schema, workgroup, and Amazon Simple Storage Service (Amazon S3) output location, and AWS Key Management Service (AWS KMS) key.Choose Logging Options.For Log Level, select INFO.For Log Path, select a path on your Windows machine.Choose OK.The ODBC driver logs required for troubleshooting are generated in the log path that you selected.To activate logging in the ODBC driver for your non-Windows machine, do the following:Linux: Edit the /opt/simba/athenaodbc/lib/64/simba.athenaodbc.ini file, and then configure the following properties:For LogLevel, select 5.For LogPath, enter a path to your host.MAC: Edit the /Library/simba/athenaodbc/lib/simba.athenaodbc.ini file, and then configure the following properties:For LogLevel, select 5.For LogPath, enter a path to your host.Save the simba.athenaodbc.ini configuration file and restart your ODBC application to make sure that the new settings are effective. The Simba Athena ODBC Driver produces the following log files at the location you specify using the LogPath key:The simbaathenaodbcdriver.log file with logs on driver activity that's not specific to the connection.The simbaathenaodbcdriver_connection_1234.log file for each connection made to the database. The number 1234 in the file name identifies the log file. This file logs driver activity that is specific to the connection.Related informationConnecting to Amazon Athena with ODBC and JDBC DriversFollow"


Question: How do I set up an Active/Passive Direct Connect connection to AWS?

I want to set up an Active/Passive AWS Direct Connect connection to AWS using private virtual interfaces (VIF).

Answer: "I want to set up an Active/Passive AWS Direct Connect connection to AWS using private virtual interfaces (VIF).ResolutionNote: It's a best practice to use dual Direct Connect connections through different data centers or providers to transport production workloads to and from AWS.Configure the following:Two routers on an on-premises location to terminate the primary and secondary Direct Connect connections to avoid a single point of device failure.A private VIF on each of the Direct Connect routers that terminate to the same Amazon Virtual Private Cloud (Amazon VPC).High availability routing protocols (such as HSRP, VRRP, and GLBP) on two routers. This allows local servers to use multiple routers that act as a single virtual router. This configuration helps maintain connectivity even if the primary router fails. You can also run an internal routing protocol such as Border Gateway Protocol (BGP).In an Active/Passive Direct Connect setup (failover), an active connection handles traffic and a passive connection is on standby. If the active connection becomes unavailable, then all traffic must be routed through the passive connection. Use Autonomous System (AS) prepending for the routes on one of your links to set the link as passive.Note: Check your vendor documentation for commands that are specific to your network device.For more information on configuring redundant Direct Connect connections on AWS, see Configure redundant connections.Influencing outbound traffic from an on-premises location to AWSThe BGP local preference attribute can be used on the on-premises routers to prefer an exit point from the local AS. If there are multiple exit points from the AS, then the local preference attribute is used to select the exit point for a specific route. The link with the highest local preference attribute is selected.Influencing inbound traffic to an on-premises location from AWSFor an Active/Passive configuration of Direct Connect connections:Apply the local preference BGP community tag. Set a higher preference to the advertised prefixes for the primary or active connection. Then, set a medium or lower preference for the passive connection.AS Path prepend using a shorter AS path on the active connection and a longer AS path on the passive connection.Advertise the most specific route using BGP on the active connection.Note: AS Path prepending can't be used to configure Active/Passive connections in the following scenarios:Connection A (virtual interface VIF-A) is in Region 1.Connection B (virtual interface VIF-B) is in Region 2.Both virtual interfaces connect to a VPC in Region 1 using a Direct Connect gateway.Both virtual interfaces advertise the same prefixes with the same BGP attributes (such as AS Path and MED) on both connections from the on-premises location.For these scenarios, outbound traffic from the VPC to the on-premises location prefers Connection A due to being in the same Region as the VPC. To setup Active/Passive Direct Connect connections in this scenario, apply a local preference BGP community tag. Also, you can advertise a more specific route on an active connection.Related informationHow can I use BGP communities to influence the preferred routing path on Direct Connect links from AWS to my network?How can I use BGP communities to control the routes advertised and received over the AWS public virtual interface with Direct Connect?Follow"


Question: What’s the difference between a hosted virtual interface (VIF) and a hosted connection?

I'm not sure if I should use a hosted virtual interface (VIF) or a hosted connection with AWS Direct Connect.

Answer: "I'm not sure if I should use a hosted virtual interface (VIF) or a hosted connection with AWS Direct Connect.ResolutionWhen using Direct Connect, you can create a VIF in one of three ways:On the AWS account where the parent Direct Connect connection exists ("standard VIF")On another AWS account using the same parent Direct Connect connection, or a VIF purchased from an Amazon Partner Network (APN) partner who provisions it for you ("hosted VIF")A physical ethernet connection that an AWS Direct Connect partner provisions for youHosted VIFs can connect to public resources or an Amazon Virtual Private Cloud (Amazon VPC) in the same way as standard VIFs. However, the account that owns the VIF is different from the connection owner. Bandwidth is shared across all VIFs on the parent connection. For instructions on creating a hosted VIF, see Create a hosted virtual interface.Hosted connections allow an APN partner to create a Direct Connect connection for you. Complete bandwidth is allocated for the Direct Connect hosted connection. This allows creation of one VIF rather than having multiple VIFs on the same Direct Connect connection. The hosted Direct Connect connection values are 50 Mbps, 100 Mbps, 200 Mbps, 300 Mbps, 400 Mbps, 500 Mbps, 1 Gbps, 2 Gbps, 5 Gbps, and 10 Gbps. AWS Direct Connect partners that meet specific requirements can create a 1 Gbps, 2 Gbps, 5 Gbps or 10 Gbps hosted connection. Contact an AWS Partner Network (APN) to create the Direct Connect connection. Then, complete the steps to accept a hosted connection.Note: You can create only one VIF per hosted Direct Connect connection. You can't modify VLAN information to create a new VIF from the AWS Management Console. Each hosted connection is assigned a VLAN on creation by the partner. Each VIF on a given connection requires a unique VLAN tag that can be provisioned only by the partner. To get additional private or public VIFs, request another hosted connection from your APN partner. To create transit virtual interfaces, you must have 1 Gbps or larger of available hosted connection bandwidth.Related informationWhat is AWS Direct Connect?Follow"


Question: How do I configure notifications for Amazon RDS or Amazon Redshift maintenance windows?

"When does the maintenance window start and end, and how can I set up notifications for Amazon Relational Database Service (Amazon RDS) and Amazon Redshift maintenance windows? I want to be notified about maintenance windows, so I can minimize the impact to my resources."

Answer: "When does the maintenance window start and end, and how can I set up notifications for Amazon Relational Database Service (Amazon RDS) and Amazon Redshift maintenance windows? I want to be notified about maintenance windows, so I can minimize the impact to my resources.ResolutionYou can set up event notifications for Amazon RDS and Amazon Redshift clusters and instances so that you receive notifications regarding on-going or completed maintenance. You can receive these notifications by either email or SMS using Amazon Simple Notification Service (Amazon SNS).To see which Amazon RDS DB instances are scheduled to receive hardware maintenance during your maintenance window, review the DB instances that are listed in the Affected resources tab. For more information, see the maintenance notification email sent to your account.While the maintenance is in progress, Single-AZ deployments are unavailable for a few minutes during your maintenance window. If you use a Multi-AZ deployment and the primary DB instance is in an affected Availability Zone at the time of maintenance, the database is unavailable until failover is complete (usually about 60 seconds). This failover occurs in your maintenance window. If the primary DB instance is not in an affected Availability Zone at the time of maintenance, then the secondary is replaced during your maintenance window without failover, and without impacting availability. To identify the Availability Zone of your primary or secondary DB instance, see describe-db-instances.Amazon RDSTo configure event notifications for Amazon RDS:Create an Amazon SNS topic and subscribe to a topic using your endpoints, such as email accounts or mobile numbers.Open the Amazon RDS console, and then from the navigate page choose Event subscriptions.Choose Create event subscription.Enter the Name, select the ARN for the SNS topic, and select Instances as the Source type.Choose Select specific instances, and select your instance.Choose Select specific event categories, and select Maintenance.Choose Create.After you create an event subscription, you'll receive notifications with the following event codes:RDS-EVENT-0026 for maintenance in progress.RDS-EVENT-0027 for maintenance that is complete.Amazon RedshiftTo configure event notifications for Amazon Redshift:Open the Amazon Redshift console, and then from the navigation pane choose Events.Choose Event Subscriptions, and then choose Create event subscription.In the Subscription details section, enter an Event subscription name for this event.In the Subscription type section, under Source type, choose Cluster.For Cluster, choose Select cluster, and then add your Cluster IDs.For Severity, choose error or info, error.In the Subscription actions section, select Create a new SNS topic or Choose an existing SNS topic.Choose Create event subscription.After you create an event subscription, you receive notifications with the following event codes:REDSHIFT-EVENT-2003 for Amazon Redshift maintenance in progress.REDSHIFT-EVENT-2004 for Amazon Redshift maintenance that is complete.Note: Amazon Redshift scheduled maintenance doesn't trigger an event. To view upcoming maintenance windows, see Maintenance windows.Related informationUsing Amazon RDS event notificationAmazon Redshift event notificationsMaintaining a DB instanceHow do I minimize downtime during required RDS maintenance?Follow"


Question: How do I troubleshoot 504 errors returned while using an Application Load Balancer?

"I found HTTP 504 errors in Application Load Balancer access logs, Amazon CloudWatch metrics, or when connecting to my service through an Application Load Balancer. How do I fix this?"

Answer: "I found HTTP 504 errors in Application Load Balancer access logs, Amazon CloudWatch metrics, or when connecting to my service through an Application Load Balancer. How do I fix this?Short descriptionA HTTP 504 error is a HTTP status code that indicates a gateway or proxy has timed out.Application Load Balancer HTTP 504 errors can occur if:The load balancer failed to establish a connection to the target before the connection timeout expired (10 seconds).The load balancer established a connection to the target but the target didn't respond before the idle timeout period elapsed.The network ACL for the subnet didn't allow traffic from the targets to the load balancer nodes on the ephemeral ports (1024-65535).The target returns a Content-Length header value that is larger than the entity body. The load balancer timed out waiting for the missing bytes.The target is an AWS Lambda function and the service didn't respond before the connection timeout expired.ResolutionCheck your load balancers idle timeout and modify if necessaryLoad balancer HTTP 504 errors can occur if the backend instance didn't respond to the request within the configured idle timeout period. By default, the idle timeout for Application Load Balancer is 60 seconds.If CloudWatch metrics are enabled, check CloudWatch metrics for your Application Load Balancer. The HTTPCode_ELB_5XX metric indicates the 504 error originated from the load balancer. If there aren't any HTTPCode_ELB_504_Count metric datapoints, the 504 errors are being returned by your application servers, not the load balancer.Check the maximum and average values for the CloudWatch metric TargetResponseTime. The timeout value may indicate the time elapsed after the load balancer request was received from the target.To resolve this:Modify the idle timeout for your load balancer so that the HTTP request completes within the idle timeout period.-or-Modify your application to respond to the HTTP request faster. Make sure that the application doesn't take longer to respond than the configured idle timeout.(Optional) Add the following custom filters on the backend web servers application logs to help determine the cause of the slow response times:Apache web servera- Apache : %D in log formatNginx web serverb- Nginx: $request_time and $upstream_response_timein log formatIIS web serverc- IIS: “time-taken” in log formatApache Tomcat web serverd- Apache Tomcat Access logs: %D in log formatMake sure that your load balancer allows traffic with registered targetsVerify that the network security groups associated with the load balancer and the backend targets allow traffic from each other in both directions on the traffic and health check ports. Make sure that the network ACL for the subnet allows traffic from the targets to the load balancer nodes on the ephemeral ports (1024-65535).Note: It's a best practice to use the following security group rules for your Application Load Balancer.If you review the CloudWatch metric TargetConnectionErrorCount with sum statistic, you are likely to see positive datapoints. For example, the number of connections that aren't successfully established between the load balancer and target.For more information, see Configure the idle timeout using the console.Make sure that your Lambda function responds before the connection timeout expiresIf your target is a Lambda function, check the performance metric duration with max statistics to verify the amount of time that event processes. For more information, see Using performance metrics.Related informationHTTP 504: Gateway timeoutHow can I troubleshoot high latency on my Application Load Balancer?How do I troubleshoot issues configuring authentication in my Application Load Balancer?How do I troubleshoot 504 errors returned while using a Classic Load Balancer?Follow"


Question: How do I set up the RabbitMQ Federation plugin on my Amazon MQ broker?

I want to distribute messages from an on-premises broker to an Amazon MQ for RabbitMQ broker using the RabbitMQ Federation plugin. How do I set up my Amazon MQ for RabbitMQ broker to work with the RabbitMQ Federation plugin?

Answer: "I want to distribute messages from an on-premises broker to an Amazon MQ for RabbitMQ broker using the RabbitMQ Federation plugin. How do I set up my Amazon MQ for RabbitMQ broker to work with the RabbitMQ Federation plugin?Short descriptionTo set up the RabbitMQ Federation plugin on an Amazon MQ for RabbitMQ broker, you must configure the following on the downstream broker:One or more federation-upstream parameters that define the federation connections to remote queues or exchanges.A RabbitMQ policy that allows specific remote queues or exchanges to become federated.An exchange with a new queue bound to it to receive messages.ResolutionNote: The following procedure requires two brokers: An upstream broker (on premises or cloud based) and a downstream broker (the Amazon MQ for RabbitMQ broker).Create one or more federation-upstream parameters that define the federation connections to remote queues or exchanges1.    Open the Amazon MQ console.2.    In the left navigation pane, choose Brokers.3.    In the Name column, choose the name of your downstream broker.4.    In the Connections pane, select the URL that's listed below RabbitMQ web console. The RabbitMQ web console login page for your downstream broker opens.5.    Enter your Username and Password for your broker. Then, choose Login to log in to the RabbitMQ web console.6.    In the RabbitMQ web console, choose the Admin tab. Then, in the right navigation pane, choose Federation Upstreams.7.    Choose Add a new upstream.8.    For Name, enter a name for your upstream parameter.9.    For URI, delete the default amqp:// text and enter the upstream broker's URI in the following format:amqps://$UPSTREAM_BROKER_USER:$UPSTREAM_BROKER_PASSWORD@$UPSTREAM_BROKER_ENDPOINT:567110.    Choose Add upstream.Create a RabbitMQ policy that allows specific remote queues or exchanges to become federated1.    Open the RabbitMQ web console of the downstream broker. For instructions, see steps 1-5 of the To create a federation-upstream parameter using the RabbitMQ web console section of this article.2.    Choose the Admin tab. Then, in the right navigation pane, choose Policies.3.    Choose Add / update a policy.4.    For Name, enter a name for your policy.5.    For Pattern, enter a regular expression that matches the queues or exchange names that you want to allow to become federated.6.    For Apply to, select Exchanges from the dropdown list.Note: Applying the policy to Exchanges allows the policy to be applied to exchanges in the downstream broker only.7.    For Priority, enter the numeral 1.Note: The Priority setting is used to determine which policy should be applied to an exchange or queue if multiple policies have the same name. Specifying the Priority setting as 1 prevents any default Amazon MQ policies that have the same name from overriding the policy.8.    For Definition, enter federation-upstream-set as the key in the first text box, before the = sign. Then, enter all as the value in the second text box, to the right of the = sign.9.    Choose Add / update policy.Create an exchange with a new queue bound to it to receive messages1.    Open the RabbitMQ web console of the downstream broker. For instructions, see steps 1-5 of the To create a federation-upstream parameter using the RabbitMQ web console section of this article.2.    Choose the Queues tab.3.    Under Add a new queue, for Name, enter a name for the new queue that is receiving messages.4.    Choose Add queue.5.    Choose the Exchanges tab.6.    Under Add a new exchange, for Name, enter the regular expression that matches exchange name that you used when creating the RabbitMQ policy.7.    Choose Add exchange.Note: RabbitMQ also creates an exchange on the upstream broker that has the same name automatically.8.    Choose the Exchanges tab.9.    Under All exchanges, choose the name of the new exchange that you created.10.    Under Bindings, for Add binding from this exchange, select To queue from the dropdown list. Then, enter the name of the queue that you created.11.    For Routing key, enter a value of your choice. The messages sent to the exchange on the upstream broker using this value are also sent to the queue bound to this exchange.12.    Choose Bind.To test the setupSend a test message to the upstream broker's source exchange1.    Open the RabbitMQ web console of the upstream broker.Note: For more information about how to access the RabbitMQ web console, see Management plugin in the RabbitMQ documentation.2.    Choose the Exchanges tab. Then, select the source exchange's name.3.    In the left navigation pane, choose Publish message. The Publish message section opens.4.    In the Routing key text box, enter the routing key value that you created.5.    In the Payload text box, enter a test message.Note: You can leave the Headers and Properties text boxes blank.6.    Choose the Publish message button. The test message is sent to the source exchange. If your federation parameter is configured correctly, then the message is also sent to the downstream broker's destination queue.Verify that the test message is received by the downstream broker's destination queue1.    Open the RabbitMQ web console of the downstream broker.2.    Choose the Queues tab. Then, choose the destination queue's name.3.    In the left navigation pane, choose Get messages. The Get messages section opens.4.    Choose the Get messages button. If the federation parameter is configured correctly, the test message that you sent to the source exchange appears as the Payload value.Related informationHow do I set up the RabbitMQ Shovel plugin on my Amazon MQ broker?Follow"


Question: How can I increase the maximum query string length in Amazon Athena?

"When I run an Amazon Athena query with a long string, I get an error similar to the following: "'queryString' failed to satisfy constraint: Member must have length less than or equal to 262144 (Service: AmazonAthena; Status Code: 400; Error Code: InvalidRequestException;". I want to increase the maximum string length."

Answer: "When I run an Amazon Athena query with a long string, I get an error similar to the following: "'queryString' failed to satisfy constraint: Member must have length less than or equal to 262144 (Service: AmazonAthena; Status Code: 400; Error Code: InvalidRequestException;". I want to increase the maximum string length.ResolutionMaximum query string length (262,144 bytes) is not an adjustable quota. This means that AWS Support can't increase the quota for you. Instead, split long queries into multiple, smaller queries. Here are some examples of how you can do that:Run multiple DDL statements. For example, assume that you're running an ALTER TABLE ADD PARTITION statement to add thousands of partitions to a table. If the query string is greater than 262,144 bytes, run multiple ALTER TABLE ADD PARTITION statements instead of one long one.Use views. Each view allows a query string length of 262,144 bytes.Use CREATE TABLE AS SELECT (CTAS) to break a long query into multiple queries and pass the results of one query to another query.You can also shorten table, column, and database names to decrease string length.Related informationService quotasFollow"


Question: How do I resolve permission errors with auto remediation for the AWS Config rule s3-bucket-logging-enabled?

"I configured the AWS Config rule s3-bucket-logging-enabled to auto remediate noncompliant Amazon Simple Storage Service (Amazon S3) buckets. However, the remediation execution failed, and the AWS Config console displays the Action status error "Action execution failed (details." I opened the Details page, but it doesn't contain enough information to troubleshoot the issue."

Answer: "I configured the AWS Config rule s3-bucket-logging-enabled to auto remediate noncompliant Amazon Simple Storage Service (Amazon S3) buckets. However, the remediation execution failed, and the AWS Config console displays the Action status error "Action execution failed (details." I opened the Details page, but it doesn't contain enough information to troubleshoot the issue.Short descriptionThe AWS Config rule s3-bucket-logging-enabled uses the AWS Systems Manager Automation document AWS-ConfigureS3BucketLogging to remediate noncompliant resources. The Systems Manager service must be allowed in the Automation role trust policy using AWS Identity and Access Management (IAM), which is passed as the AutomationAssumeRole parameter. Also, the Automation role must have PutBucketLogging permissions, and the target Amazon S3 bucket must be configured to store logs.ResolutionFor a more detailed error message, run the AWS Command Line Interface (AWS CLI) command describe-remediation-execution-status. Then, follow these instructions to troubleshoot the error message. For more information, see How can I troubleshoot failed remediation executions in AWS Config?**Important:**Before you begin, be sure that you have installed and configured the AWS CLI. If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI."Step fails when it is Execute/Cancelling action. An error occurred (MalformedXML) when calling the PutBucketLogging operation: The XML you provided was not well-formed or did not validate against our published schema. Please refer to Automation Service Troubleshooting Guide for more diagnosis details".To resolve this error message, see Why did the AWS Config auto remediation action for the SSM document AWS-ConfigureS3BucketLogging fail with the error "(MalformedXML)" when calling the PutBucketLogging API?"Step fails when it is Execute/Cancelling action. An error occurred (AccessDenied) when calling the PutBucketLogging operation: Access Denied. Please refer to Automation Service Troubleshooting Guide for more diagnosis details."This error occurs because the AutomationAssumeRole role doesn't have permissions to call the PutBucketLogging API on the noncompliant S3 buckets. You can use the following example policy to allow the role to call the PutBucketLogging API:{    "Version": "2012-10-17",    "Statement": [            {            "Effect": "Allow",            "Action": "s3:PutBucketLogging",            "Resource": [                            "arn:aws:s3:::<BUCKET_NAME_1>",                            "arn:aws:s3:::<BUCKET_NAME_2>",                            "arn:aws:s3:::<BUCKET_NAME_3>"                         ]             }           ]}Note: If you require that remediation happens on all the buckets in an AWS Region, limit the role's permission to a specific Region using the condition key aws:RequestedRegion."Invalid execution parameters sent to Systems Automation. The defined assume role is unable to be assumed."This error occurs because the IAM AutomationAssumeRole role can't be assumed by the Systems Manager Automation service. Use the following example policy to allow Systems Manager to assume the IAM role:{  "Version": "2012-10-17",  "Statement": [  {      "Effect": "Allow",      "Principal": {      "Service": "ssm.amazonaws.com"  },   "Action": "sts:AssumeRole"  } ]}Related informationRemediating noncompliant AWS resources by AWS Config rulesAmazon S3 bucket compliance using AWS Config Auto Remediation featureFollow"


Question: What are the best practices to configure the Amazon VPC CNI plugin to use an IP address in VPC subnets with Amazon EKS?

I want to configure the Amazon Virtual Private Cloud (VPC) Container Network Interface (CNI) plugin to use an IP address in VPC subnets with Amazon Elastic Kubernetes Service (Amazon EKS). What are the best practices?

Answer: "I want to configure the Amazon Virtual Private Cloud (VPC) Container Network Interface (CNI) plugin to use an IP address in VPC subnets with Amazon Elastic Kubernetes Service (Amazon EKS). What are the best practices?Short descriptionOne of the primary components of the Amazon VPC CNI is the L-IPAM daemon. You configure this daemon to allocate IP addresses to nodes.If a new pod is scheduled on a node, then the kubelet invokes the CNI binary. The CNI binary calls the L-IPAMD to get an IP address for the new pod. The pod then tracks the elastic network interfaces and IP addresses that are attached to the instance.You can use certain configuration variables to control how many network interfaces and IP addresses are maintained. For more information, see WARM_ENI_TARGET, WARM_IP_TARGET and MINIMUM_IP_TARGET and WARM_PREFIX_TARGET, WARM_IP_TARGET and MINIMUM_IP_TARGET (from the GitHub website).ResolutionThe following are basic best practices for using configuration variables to control how many network interfaces and IP addresses are maintained.WARM_ENI_TARGETUse the WARM_ENI_TARGET variable to determine how many elastic network interfaces the L-IPAMD keeps available so that pods are immediately assigned an IP address when scheduled on a node.To prevent depleting available IP addresses in the subnet, make sure to check the worker node instance type and the maximum number of network interfaces and private IPv4 addresses per interface. For example, if you set WARM_ENI_TARGET=3 for an m5.xlarge node, then three elastic network interfaces are always attached to the node. The node then assigns 45 IP addresses, 15 per elastic network interface. Because the 45 IP addresses are reserved for this node, those addresses can't be used for pods that are scheduled on other worker nodes.If you expect your application to scale greatly, then you can use WARM_ENI_TARGET to quickly accommodate newly scheduled pods.WARM_IP_TARGETUse the WARM_IP_TARGET variable to make sure that you always have a defined number of IP addresses available in the L-IPAMD's warm pool.For clusters with low productivity, use WARM_IP_TARGET so that only the required number of IP addresses are assigned to the network interface. This prevents the IP addresses of the elastic network interfaces from being blocked.MINIMUM_IP_TARGETUse the MINIMUM_IP_TARGET to make sure that a minimum number of IP addresses are assigned to a node when it initially appears. This variable is generally used with the WARM_IP_TARGET variable.If you know the minimum number of pods that you'll run per node, then use MINIMUM_IP_TARGET so that the required number of IP addresses are assigned. If the IP addresses are readily available, then the pods can receive them as the pods are scheduled.Set this variable with WARM_IP_TARGET to make sure that there are available IP addresses on the node for future pods.WARM_PREFIX_TARGETUse the WARM_PREFIX_TARGET variable to make sure that you always have a defined number of prefixes (/28 CIDR blocks) added to the instance's network interface. You can use WARM_PREFIX_TARGET only for CNI version 1.9.0 or later, and you must activate the Amazon VPC CNI IP address prefix assignment capability.If you use the IP address prefix assignment, then make sure that the WARM_PREFIX_TARGET variable is set to a value greater than or equal to 1. If it's set to 0, then you receive the following error:Error: Setting WARM_PREFIX_TARGET = 0 is not supported while WARM_IP_TARGET/MINIMUM_IP_TARGET is not set. Please configure either one of the WARM_{PREFIX/IP}_TARGET or MINIMUM_IP_TARGET env variablesFor smaller subnets, use WARM_IP_TARGET with WARM_PREFIX_TARGET to avoid allocating too many prefixes. Allocating too many prefixes can deplete available IP addresses.To learn more about how these configuration variables affect IP address utilization, see CNI configuration variables (from the GitHub website).Follow"


Question: How do I troubleshoot errors when creating an Amazon S3 bucket?

"I want to create an Amazon Simple Storage Service (Amazon S3) bucket in my account. However, I'm getting some errors."

Answer: "I want to create an Amazon Simple Storage Service (Amazon S3) bucket in my account. However, I'm getting some errors.ResolutionYou might get one of the following errors when you try to create an Amazon S3 bucket:"A conflicting conditional operation is currently in progress against this resource""TooManyBuckets: You have attempted to create more buckets than allowed""Access Denied""BucketAlreadyExists""Networking Error"Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you're using the most recent AWS CLI version.A conflicting conditional operation is currently in progress against this resourceThis error occurs for the following reasons:You try to create a bucket with the same name as a bucket that was recently deleted.You try to create a bucket with the same name at the same time as someone else or another AWS Identity and Access Management (IAM) identity.Because bucket names are required to be globally unique, you can't have two buckets with the same name. After bucket deletion, wait a few hours for the changes to take effect across all AWS Regions. In most cases, you can create a bucket within a few hours after the previous bucket was deleted. However, in a few cases, it might take 48 to 72 hours for the changes to take effect. Therefore, it's a best practice to wait at least 48 hours to create the S3 bucket again.TooManyBuckets: You have attempted to create more buckets than allowedThis error occurs when you try to create multiple buckets and are exceeding the bucket limit for your AWS account. By default, Amazon S3 has a soft limit of 100 buckets. If you require additional buckets, then you can request a service quota increase to increase your account bucket limit to a maximum of 1,000 buckets.Access DeniedTo create an S3 bucket, you must have s3:CreateBucket permission for the IAM entity that's trying to create the bucket. Also, check the service control policies (SCPs) to make sure that they don't explicitly deny the s3:CreateBucket permission.BucketAlreadyExistsAmazon S3 bucket names must be globally unique. If you get the error "Bucket name already exists" or "BucketAlreadyExists", then use a different bucket name when you create the s3 bucket. These error messages indicate that another AWS account already owns an s3 bucket with the same name.For example, you get the error "Bucket name is already owned by you" or "BucketAlreadyOwnedByYou". These error messages indicate that your account is already using the specified bucket name. Review the existing buckets on the Amazon S3 console. Or, run the head-bucket AWS CLI command to confirm whether you already have a bucket with the same name.Networking ErrorTo resolve this error, try the following approaches:Check your browser settings. You might get this error because of a proxy server that's involved or extensions that you installed or turned on, such as AdBlocker or uBlock. Work with your networking team to troubleshoot the issue.Clear the cache and cookies on your browser. Try a different browser to find out if the issue persists across browsers. Or, try using incognito mode in your web browser to access the Amazon S3 console.If the issue still persists, then run the AWS CLI create-bucket command to create the bucket:$ aws s3api create-bucket --bucket my-bucket --region eu-west-1Follow"


Question: How do I determine throttling in my CloudWatch logs?

"I receive a "RequestLimitExceeded" or "ThrottlingException" error when working with Amazon CloudWatch logs, and my API call is throttled."

Answer: "I receive a "RequestLimitExceeded" or "ThrottlingException" error when working with Amazon CloudWatch logs, and my API call is throttled.Short descriptionWhen you work with CloudWatch logs, you might exceed the API rate limit. When this happens, you receive a RequestLimitExceeded or ThrottlingException error, and your API call is throttled. You must identify where and when throttling is happening so you can resolve these errors and make informed rate limit increase requests.ResolutionIntermittent throttling on CloudWatch logs when accessing logsYou can use the FilterLogEvents or GetLogEvents API calls to list your log events or log streams. These API calls have a hard limit, and they don't qualify for a limit increase. This means that if you use the FilterLogEvents API to search for log events from a specified log group, the default quaAPI has a default quota. This quote is 5 transactions per second (TPS) per account or AWS Region. If you reach this limit, then you receive the RateExceeded error.Use these best practices to avoid throttling errors in this use case:Use a subscription filter to immediately retrieve log data from CloudWatch logs in real time. For more information, see Using CloudWatch Logs subscription filters and Real-time processing of log data with subscriptions.Use CloudWatch Logs Insights to quickly get log data from CloudWatch logs. Use queries to filter your logs to view specific log groups.Export log data to Amazon Simple Storage Service (Amazon S3) for batch use cases.Note: Log data can take up to 12 hours to become available for export from CloudWatch Logs. Therefore, it's not a best practice to use this method for real-time analysis and processing.ThrottlingException errors when using an application/script to fetch CloudWatch log dataTo collect CloudWatch logs, you can develop a collector script. This script attempts a DescribeLogStream or GetLogEvents API call to pull data from different log streams or different time frames in the same log group. API calls such as FilterLogEvents, GetLogEvents and DescribeLogStreams are designed for human interaction and not for automation. This means that you receive an error and the API call is throttled.Use these best practices to avoid throttling in this use case:Use exponential backoff and retries when you make an API call. For more information, see Exponential backoff and jitter and Error retries and exponential backoff in AWS.Distribute your API calls over time. Schedule actions with some randomization so that they are spread over a period of time.Add sleep intervals between consecutive API calls. Add some delay between API calls that are sent from the same script or application. If you send API calls in rapid succession, then this is more likely to cause rate errors.In some cases, you might use a SIEM solution such as Splunk to fetch logs from CloudWatch. SIEM solutions are used to gather data from multiple systems and analyze this data to detect unusual behavior. You might experience API throttling when you use the Splunk plugin. In order to avoid this issue, create a CloudWatch logs subscription filter with Amazon Kinesis Data Firehose and deliver the log data to Splunk. For more information, see the Splunk documentation for Configure Kinesis inputs for the Splunk Add-on for AWS.Throttling errors when integrating PutLogEvents API calls with Lambda functionThe PutLogEvents API call is used to upload logs to a specified log stream in batches of 1 MB. This API has a rate limit of 800 TPS, per account, per Region. This applies except for the following Regions where the quota is 1500 TPS per account per Region: US East (N. Virginia), US West (Oregon), and Europe (Ireland). You can request a quota increase.For more information on this, and to request a quota increase, see CloudWatch Logs quotas.Manage your CloudWatch Logs service quotasAWS defines quotas for services to protect performance and to be sure of availability. CloudWatch has quotas for metrics, alarms, API request, and alarm email notifications. Use these steps to visualize your service quotas and set alarms if you reach the threshold:1.    Open the Service Quotas console.2.    In the navigation pane, choose AWS services.3.    From the AWS services list, search for Amazon CloudWatch logs.4.    The Service quotas list shows you several attributes or options: the service quota name, the applied value (if its available), AWS default quota, and whether the quota value is adjustable.5.    To view more information about a service quota, such as the description, choose the quota name.6.    After you choose the quota name, you can see more information about this quota. For example, if you choose GetLogEvents throttle limit in transactions per second, then you see the following items:DescriptionQuota codeQuota ARNUtilization: %Applied quota valueAWS default quota valueAdjustable: Y/N7.    In the Amazon CloudWatch alarms section, choose Create alarm, and enter an Alarm threshold and Alarm name.Related informationHow do I avoid throttling when I call PutMetricData in the CloudWatch API?Follow"


Question: How does the ACM managed renewal process work with email-validated certificates?

I have an email-validated certificate with AWS Certificate Manager (ACM). How does ACM managed renewal work with email-validated certificates?

Answer: "I have an email-validated certificate with AWS Certificate Manager (ACM). How does ACM managed renewal work with email-validated certificates?ResolutionACM provides managed renewal for your Amazon-issued SSL/TLS certificates. An email-validated certificate is eligible for renewal if it's associated with AWS services integrated with AWS Certificate Manager.ACM certificates are valid for 13 months (395 days). To be renewed, email-validated certificates require an action by the domain owner. ACM begins sending renewal notices 45 days before expiration, using the domain's WHOIS mailbox addresses and to five common administrator addresses. The notifications contain a link that the domain owner can choose for renewal. After all listed domains are validated, ACM issues a renewed certificate with the same ARN. For more information, see Renewal for domains validated by email.You can use the AWS Certificate Manager console or the ACM API to request that ACM send you a domain validation email for your certificate renewal. For instructions, see Request a domain validation email message.You can also use automation with ACM certificates that are email-validated instead of manually validating certificates. For instructions, see Automating email validation.Note: These scenarios don't apply if you use DNS to validate domain ownership.Related informationI validated my domain names using the AWS Certificate Manager (ACM) managed renewal process, but the status is still pending validationFollow"


Question: How are credits and promotions calculated in an organization in AWS Organizations?

"I'm using AWS Organizations and joining or creating an organization, and my AWS account has active credits on it. How do my credits apply to the consolidated bill?"

Answer: "I'm using AWS Organizations and joining or creating an organization, and my AWS account has active credits on it. How do my credits apply to the consolidated bill?ResolutionHere a few things to keep in mind if you are joining or creating an organization with active credits on your account:AWS applies credits under an account to the organization's consolidated bill, beginning the first full billing cycle after an account joins the organization.Credits on a standalone account are applied to usage costs incurred by the account before joining the organization. With credit sharing on, the remaining credit balance, if any, is applied across all applicable usage incurred by member accounts from the start of the next billing cycle.Credits redeemed on a standalone account at any time during the month of joining the organization are applied to the usage generated by the standalone account until joining the organization.To change this behavior, sign in to the payer account, and then disable credit sharing for the organization. For more information, see Sharing AWS credits.When you turn credit sharing off, be aware of the following that affects the rest of the current billing cycle:Bills are calculated based on the credit sharing preference on the last day of the month. If you turn credit sharing off before the last day of the month, then the bill for that billing cycle doesn't include the credits of member accounts.If the account was a member account in a different organization, then the credits on the member account apply to the bill of the former payer account until the end of the billing cycle. Any charges incurred by the member account after joining the new organization are billed to the current payer account.Related informationAWS creditsConsolidated billing for AWS OrganizationsFollow"


Question: How can I build a dynamic routing based VPN using a Palo-Alto firewall and AWS VPN?

I want to build a dynamic AWS Virtual Private Network (AWS VPN) between AWS and a Palo-Alto firewall.

Answer: "I want to build a dynamic AWS Virtual Private Network (AWS VPN) between AWS and a Palo-Alto firewall.ResolutionPrerequisitesYou must have a virtual private cloud (VPC) with an IP-CIDR that doesn't overlap with the on-premise network. This VPC must be associated with a virtual private gateway (VGW) or attached to a transit gateway (TGW).AWS configuration1.    Create a customer gateway (CGW). When you create the CGW, you can either provide your autonomous system number (AS number) or choose the default option. When you choose the default, AWS provides an AS number for your CGW.2.    Create a Site-to-Site VPN. For Gateway, choose VGW or TGW, and for Routing options, choose Dynamic.3.    Download the configuration file from the AWS Management Console.The configuration file provides the following:AWS public IP and pre-shared keyIP address and MTU configuration for your Palo-Alto tunnel interfaceBorder Gateway Protocol (BGP) configuration and BGP IP to configure in the Palo-Alto firewallPalo-Alto configurationPalo-Alto provides next generation firewalls that support route-based VPN, by default. So when you create a VPN between Palo-Alto and AWS, you don't require proxy IDs.Note: The following encryption, DH-group, and authentication settings remain the same for both IKE-crypto and IPsec-crypto. The lifetime of the phase 1 and phase 2 settings are 8 hours and 1 hour, by default.Encryption: AES-256-GCMDH-Group: 20Authentication: SHA-3841.    Create the IKE-Crypto profile using the algorithm above.2.    Create the IPsec-Crypto profile using the algorithm above.3.    Build the tunnel interface. For IPv4, provide the tunnel interface IP. You can find this in section 3 of the configuration file that you downloaded from the AWS Management Console. Under Advanced, set an MTU OF 1427.4.    Create the IKE-Gateway, using the following configurations:For Version, choose IKEv2 only, and for Authentication, choose pre-shared key.In the advanced section, make sure that NAT Traversal is turned on.Choose the IKE-Crypto profile that you created in step 1.Turn on liveness check at an interval of 5 seconds.5.    From the Network tab, choose IPsec Tunnels, and then create the IPsec tunnel. Choose the Tunnel interface and IKE-Gateway that you created in the previous step.6.    Commit changes. After this completes, take SSH access of the firewall, and then run the following commands to start VPN negotiation:test vpn ike-sa gateway <IKE-Gateway-Name>        test vpn ipsec-sa tunnel <IPsec-Tunnel-Name>On the GUI interface, under IPsec Tunnels, the status is now green.Configure BGP routing as Palo-AltoNote: AWS VPN doesn't support graceful restart and Bidirectional Forwarding Detection (BFD).First, create the redistribution profile. Then, configure BGP using these settings:1.    On the General tab, choose the check box to turn on BGP.2.    Add a router ID, and enter the Palo-Alto AS number.3.    On the Peer-group tab, choose Create a new peer group.4.    For Peer AS, enter the AWS AS number. For Peer address, enter the AWS BGP IP. You can find both of these numbers in section 4 of the configuration file that you previously downloaded from the AWS Management Console.4.    In Connection options, for keep-alive interval, choose 10 seconds. For Hold time, choose 30 seconds.5.    Choose the R****edist rules tab, and then create a redist rule. For Name, choose the redistribution profile that you created previously. Then, choose Commit changes.Next, verify that BGP is established.1.    Choose the Network tab, and then choose Virtual router.2.    Choose More run time stats, and then choose BGP. Under Peer, verify that the status is established.Related informationHow do I download AWS Site-to-Site VPN example configuration files?Follow"


Question: How do I resolve drift detection errors in CloudFormation with my AWS managed rule "cloudformation-stack-drift-detection-check" for AWS Config?

I want to resolve drift detection errors in AWS CloudFormation with my AWS managed rule cloudformation-stack-drift-detection-check for AWS Config.

Answer: "I want to resolve drift detection errors in AWS CloudFormation with my AWS managed rule cloudformation-stack-drift-detection-check for AWS Config.Short descriptionYou can receive the following errors even if the drift status of your stack shows as IN_SYNC in the CloudFormation console:"AWS CloudFormation failed to detect drift, defaulting to NON_COMPLIANT. Re-evaluate the rule and try again. If the problem persists contact AWS CloudFormation support.""Your stack drift detection operation for the specific stack has failed. Check your existing AWS CloudFormation role permissions and add the missing permissions."Depending upon the error that you receive, choose either of the following options:If you receive the NON-COMPLIANT error, complete the steps in the Update your AWS Config rules section.If you receive the permissions error, complete the steps in the Set the correct permissions section.Note: The AWS managed rule cloudformation-stack-drift-detection-check checks CloudFormation stacks for drift detection using the DetectStackDrift API. This rule has both a periodic and configuration change trigger type. You get an error message if drift detection fails.ResolutionUpdate your AWS Config rulesYou can receive a NON_COMPLIANT error message in the AWS Config console for the following reasons:The resources in your CloudFormation stack don't support drift detection. You receive an error as an annotation against the NON_COMPLIANT resource for your AWS Config rule. For example, you receive a "Drift detection is not supported for ResourceType" error in the AWS CloudTrail log.Your AWS Config rule depends on the availability of DetectStackDrift. You receive a throttling or "Rate Exceeded" error because AWS Config defaults the rule to NON_COMPLIANT when throttling occurs.To resolve the error resulting from no drift detection support:Ignore the CloudFormation stack in NON_COMPLIANT status. The error is a false positive in this scenario. For more information, see Evaluating Your Resources.-or-Create a custom AWS Config rule that includes a provision to exclude your stacks from evaluations. For AWS Config rule examples, see AWS Config Rules Repository and RDKlib.To resolve the error resulting from the availability of DetectStackDrift:1.    Evaluate your AWS Config rule again.Note: For example, you can attach config.amazonaws.com trusted permissions and ReadOnlyAccess(AWS Managed) policy permissions for the role's Amazon Resource Name (ARN), which is provided in the rule required parameter cloudformationRoleArn. For more information, see Considerations when detecting drift.2.    Perform drift detection on your CloudFormation stacks.3.    Run the evaluation through your AWS Config rule.Set the correct permissionsYou receive a permissions error when the AWS Identity and Access Management (IAM) role for the required cloudformationRoleArn parameter doesn't have the DetectStackDrift API, DetectStackResourceDrift API, or other required service permissions.To resolve this error, edit the trust policy for your IAM role. You must attach config.amazonaws.com and ReadOnlyAccess(AWS Managed) permissions for cloudformationRoleArn.Related informationCloudFormation Workshop: Drift detection labFollow"


Question: Why is my connection redirected to the reader instance when I try to connect to my Amazon Aurora writer endpoint?

I am using an Amazon Aurora cluster endpoint/writer endpoint in my application server but my application connects to the reader instance instead.

Answer: "I am using an Amazon Aurora cluster endpoint/writer endpoint in my application server but my application connects to the reader instance instead.Short descriptionWhen you try to connect to your Aurora cluster endpoint or writer endpoint, your application might connect to the reader instance instead. This happens when the endpoint and its mapped IP addresses are cached on the client application end.Aurora cluster endpoints always point to the Aurora writer instance. When a failover happens, the Aurora cluster endpoint points to the new writer instance. If you are using the cluster endpoint, then during failover your read/write connections are automatically redirected to an Aurora replica. This replica instance is promoted to primary.So, during failover, the underlying IP address of your Aurora instance can change, and the cached value might no longer be in service.A client trying to connect to a database using a DNS name must resolve that DNS name to an IP address by querying a DNS server. The client then caches the responses. Per protocol, DNS responses specify the Time to Live (TTL), which governs how long the client should cache the record. Aurora DNS zones use a short TTL of five seconds. But many systems implement client caches with different settings, which can make the TTL longer.If a client tries to connect to the cluster when the DNS record changes haven't been propagated, then the client receives an old address. This causes the client to connect to the previous primary instance, which is now the reader instance.So, caching the DNS data for an extended time can cause connection failures.The client no longer gets TCP traffic from the database after failover initiates. Instead, it’s up to the client to time out. This hard fencing of the original primary database on any failover means that the client sees similar behavior during planned and unplanned failovers.ResolutionCheck if you are connecting to the writer instance or an Aurora replica.To determine if your client is connecting to the writer instance or to an Aurora replica, use the @@innodb_read_only variable:mysql> select @@innodb_read_only;A value of 0 means that you are connected to the writer instance.Run this query to determine which server you're connected to, and if that server is a writer or reader:mysql> select concat("You are connected to '",server_id,"', which is a ",if(SESSION_ID='MASTER_SESSION_ID',"Writer","Reader")) as CONNECTION_STATUS from information_schema.replica_host_status where SERVER_ID in (select @@aurora_server_id);+-----------------------------------------------------------------+| CONNECTION_STATUS                                               |+-----------------------------------------------------------------+| You are connected to 'aurora-test-instance1', which is a Writer |+-----------------------------------------------------------------+1 row in set (0.08 sec)Troubleshoot multiple reader instances in a clusterAurora reader endpoints are DNS CNAME entries. If a cluster has multiple reader instances, then when you resolve the reader endpoint, you get an instance IP that's chosen in round robin fashion. This is because the reader endpoint contains all Aurora replicas, and it provides DNS-based, round-robin load balancing for new connections.Make sure that you keep resolving the endpoint without caching DNS to get a different instance IP on each resolution. If you resolve the endpoint only once and keep the connection in your pool, then every query on that connection goes to the same instance. If you cache DNS, you receive the same instance IP each time you resolve the endpoint.Follow best practicesMake sure that your network and client configurations don't further increase the DNS cache TTL. If you use any form of connection pooling or other multiplexing, you might need to flush or reduce the time-to-live for any cached DNS information. If your client application is caching the DNS data of your DB instances, then set a TTL value of less than 30 seconds.Use Amazon Relational Database Service (Amazon RDS) Proxy to manage connections. For more information, see Using Amazon RDS Proxy.Review the best practices for Using smart drivers.Use a TCP-based load balancer such as Elastic Load Balancing or HA/Proxy.Related informationTypes of Aurora endpointsDNS cachingWhy did I get a read only error after an Amazon Aurora DB cluster failed over?Follow"


Question: How do I allow or block requests from a specific country or geolocation using AWS WAF?

Answer: "How do I allow or block requests from a specific country or geolocation using AWS WAF?Short descriptionUse the Geographic match rule statement to block access to your site from specific countries or to allow access only from specific countries.To allow some web requests based on country of origin, add a geographic match rule statement for the countries that you want to allow. Then, add a second geographic match rule statement for the countries that you want to block.Note: If you use CloudFront geographic restriction to block a country from accessing your content, then any request from that country is blocked and isn't forwarded to AWS WAF. If you want to allow or block requests based on geography with other AWS WAF criteria, use an AWS WAF geographic match rule statement instead.ResolutionTo allow or block requests from a specific country or geolocation using AWS WAF, do the following:1.    Open the AWS WAF console.2.    In the navigation pane, under AWS WAF, choose Web ACLs.3.    For Region, select the AWS Region where you created your web ACL.       Note: Select Global if your web ACL is set up for Amazon CloudFront.4.    Select your web ACL.5.    Choose Rules, and then choose Add Rules, Add my own rules and rule groups.6.    For Rule Builder, enter a name for your rule.       Note: The name must have one to 128 valid characters, such as A to Z, a to z, 0 to 9, - (hyphen), and _ (underscore).7.    For If a request, choose matches the statement.8.    For Choose an inspection option, choose Originates from a country.9.    For Choose country codes, choose the country that you want requests to be inspected for.10.  (Optional) Choose Source IP address or IP address in header for use in determining the country of origin.       Warning: When a request routes through a CDN or other proxy network, the source IP address identifies the proxy. Then, the original IP address is sent in aheader. Use caution when using IP address in header, as headers can be handled inconsistently by proxies, and they can be modified to bypass inspection.11.   For Action, choose either Allow to allow requests or Block to block requests from the country chosen in step 9.Note: When choosing to block or allow requests, consider the default action set for the web ACL. If the default action is Block, then requests that come from all countries are blocked, except for countries that are explicitly allowed in this step. This is the simplest configuration to manage the geographical requests that are allowed, but this configuration doesn't provide an opportunity to inspect the requests' content.If the default action is Allow, then you can create a negate (NOT) rule to specify the countries that must not be blocked. The action on this rule must be block. This creates a rule that doesn't block requests from the countries that you want to allow, but also doesn't explicitly allow them. Requests must wait to pass other user-defined rules, such as inspections for malicious content, before being allowed by the default action. This is a more robust configuration.12.   Choose Add Rule.13.   (Optional) For Set Rule Priority, select your rule, and then move its priority. The rules are processed in the order that they appear.For more information, see Processing order of rules and rule groups in a web ACL.14.   Choose Save.Follow"