Question: How do I resolve the invalid parameter error that I receive in Amazon SNS in response to API calls?

I receive an invalid parameter error message in Amazon Simple Notification Service (Amazon SNS) in response to API calls.

Answer: "I receive an invalid parameter error message in Amazon Simple Notification Service (Amazon SNS) in response to API calls.Short descriptionIf you receive the following error:InvalidParameter - Error message: Invalid parameter: Attributes Reason: Platform credentials are invalidComplete the steps in the Resolve platform credential errors section.If you receive the following error:"message": "Invalid parameter: The MessageGroupId parameter is required for FIFO topics" even if the customer has provided MessageGroupIdComplete the steps in the Resolve FIFO errors section.If you receive the following error:Couldn't set text messaging attributes.Error code: InvalidParameter - Error message: Invalid parameter:Complete the steps in the Resolve text messaging (SMS) errors section.If you receive the following error:Couldn't delete subscription.Error code: InvalidParameter - Error message: Invalid parameter: SubscriptionArn Reason: An ARN must have at least 6 elements, not 1Complete the steps in the Resolve subscription delete errors section.ResolutionResolve platform credential errorsImportant: Before completing the following steps, check the provider certificate type in your Apple Developer account. Then, download the certificate.1.    Open the Amazon SNS console.2.    On the navigation pane, choose Push notifications.3.    Choose Create platform application.4.    For Application name, enter a name for your application.5.    For Push notification platform, select Apple iOS/VoIP/Mac.6.    In the Apple Credentials section, select the Used for development in sandbox check box if you're using a certificate of type Sandbox.Important: You must not select this check box if you're using a certificate of type Sandbox & Production. Otherwise, you receive an error.7.    For Push certificate type, select your type of certificate.8.    Choose Choose file, and then select the certificate that you want to load credentials from.9.    Choose Create platform application.Resolve FIFO errorsVerify that you're using AWS SDK for JavaScript in the Browser or AWS SDK for JavaScript in Node.js version 2.777.0 or above.You can receive Amazon SNS FIFO errors related to an invalid MessageGroupId or invalid MessageDeduplicationId even though the IDs are provided in the API request. Amazon SNS FIFO attributes are available for newer SDK versions only. For more information, see Changelog for AWS SDK for JavaScript.Resolve text messaging (SMS) errors1.    Open the Amazon SNS console.2.    On the navigation pane, choose Text messaging (SMS).3.    For Text messaging preferences, choose Edit.4.    For Account spend limit - Optional Parameter, enter the numeric value 1. This default value is the monthly spend limit in USD. You can request an account spend limit increase if you want the account monthly spend limit to be greater than $1.Note: By default, every Amazon SNS account has a spend quota of $1 in every AWS Region. You receive an error in the Amazon SNS console when the value is greater than 1.5.    Choose Save changes.Resolve subscription delete errorsYou receive this error when you try to remove a subscription that's in PendingConfirmation status. To remove a subscription, the Amazon SNS Unsubscribe API is called, which requires SubscriptionARN. If you use the console to delete the subscription, then the Amazon SNS console automatically selects the SubscriptionARN from the selected subscription. For a subscription in PendingSubscription status, the console doesn't show SubscriptionId. As a result, the console uses the PendingConfirmation as a SubscriptionId. Then, the API calls fails and returns an error.For a subscription in Pending confirmation status:1.    Open the Amazon SNS console.2.    On the navigation pane, choose Topics, and then choose your topic.3.    Select the topic with the subscription in Pending confirmation status, and then select the subscription in Pending confirmation status.4.    Choose Request confirmation.A subscribe notification is sent to the endpoint to confirm the subscription.5.    Copy the subscribe URL from the notification. This URL looks similar to the following:https://sns.us-east-1.amazonaws.com/confirmation.html?TopicArn=arn:aws:sns:your-aws-region:your-account-number:your-topic-name&Token=your-token&Endpoint=your-endpoint6.    On the navigation menu in Amazon SNS, choose Subscriptions, and then select your subscription.7.    Choose Confirm subscription.8.    Paste the subscribe URL from step 5 into the dialog box, and then choose Confirm subscription9.    Select your subscription, and then choose Delete.For a subscription with Deleted status:Check the endpoint where the unsubscribe link was chosen. After the subscription is unsubscribed, the endpoint receives another notification similar to the following:Your subscription to the topic below has been deactivated:arn:aws:sns:Your-AWS-Region:Your-Account-Number:Your-Topic-NameIf this was in error or you wish to resubscribe, click or visit the link below:ResubscribePlease click on the resubscribe link present in the notification. This will confirm the subscription. On success, please select the specific subscription and click on Delete button.-or-Wait three days for the subscription to be removed. By default, Amazon SNS automatically removes subscriptions with PendingConfirmation in three days.Follow"


Question: How can I improve the CPU performance of my Amazon EC2 Linux instances?

I want to improve the performance of my Amazon Elastic Compute Cloud (Amazon EC2) Linux instances. What are some ways I can do that?

Answer: "I want to improve the performance of my Amazon Elastic Compute Cloud (Amazon EC2) Linux instances. What are some ways I can do that?ResolutionTo improve the performance of your Amazon EC2 Linux instances, try the following recommendations:Hardware virtual machine (HVM) Amazon Machine Images (AMIs)We recommend using HVM AMIs for improved performance. HVM AMIs also offer newer instance classes (for example, M5, M4, and R4) and Amazon EC2 features such as enhanced networking. For more information, see Linux AMI virtualization types.Enhanced networkingTo improve performance, you can use enhanced networking on supported instance types at no additional charge. Enhanced networking uses single root I/O virtualization (SR-IOV), which is a method of device virtualization that provides higher I/O performance and lower CPU utilization when compared to traditional virtualized network interfaces. For supported instance types and instructions, see Enhanced networking on Linux and Enhanced networking on Windows.To enable enhanced networking, the instances must use an HVM AMI and be launched in an Amazon Virtual Private Cloud (Amazon VPC).Note: It's a best practice to use the updated version of the Elastic Network Adapter (ENA) or the Intel 82599 Virtual Function (VF) interface driver.Non-volatile memory express (NVMe) volumesFor storage, using NVMe instance store volumes can assist with performance. Depending on your kernel version and instance type, improved performance with NVMe volumes can vary with workload. For more information, see Amazon EBS and NVMe and SSD instance store volumes.Note: To use the Kyber I/O scheduler for certain workloads, be sure that your Amazon EC2 Linux instance is running kernel 4.12 or newer.HugePagesHugePages can improve performance for workloads that deal with large amounts of memory access. For more information, see the HugePages documentation on the kernel.org website. For best practices regarding High-Performance Computing (HPC) workloads, see High-Performance Computing Lens.Latest kernel version and instance typesUsing the latest kernel version and instance types is highly recommended for performance. If you’re using M3, C3, or other older instance types, consider migrating to M4 or similar instance types, as well as using the latest kernel version available for the operating system. For more information, see Amazon EC2 Instance Types.Jumbo framesAvoid small packets whenever possible. If your workload supports it, use larger packets with jumbo frames. For more information, see Network maximum transmission unit (MTU) for your EC2 instance.Data Plane Development Kit (DPDK)You might see performance benefits from using DPDK-based versions of software to move networking outside the kernel and into userspace. Using DPDK can require a software update that includes DPDK support. For more information, see the DPDK website.Process Context Identifier (PCID)If Kernel Page-Table Isolation (KPTI) is enabled on your instance's operating system, then also enabling PCID can improve CPU performance. You must verify that both the kernel and instance type support PCID.Scaling instancesTo improve performance, consider increasing the size of your instances, or increasing the number of instances.TimersThe tsc timer is generally the best performing timer available to most instances. If you’re using a xen timer, you might see improved performance by moving to the tsc timer. If you’re using an older operating system that’s using the jiffies timer, consider moving to an operating system that preferably supports tsc, or supports xen at minimum.Note: Older instance types, such as M1 or M2, provide an emulated tsc timer. Consider moving to a newer instance type for better tsc timer performance.Related informationBest practices for Amazon EC2Follow"


Question: How do I resolve the "Cannot find module" or "Cannot find Package" errors when I run Lambda code in Node.js?

I receive the "Cannot find module" or "Cannot find Package" error when I run AWS Lambda code in Node.js.

Answer: "I receive the "Cannot find module" or "Cannot find Package" error when I run AWS Lambda code in Node.js.Short descriptionIf your Lambda environment can't find the specified library in the Node.js code, then you receive one of these errors. These errors occur because Lambda isn't prepackaged with all Node.js libraries.If a module is missing with CommonJS, then you receive the following error: Runtime.ImportModuleError: Error: Cannot find module.If a module is missing with JavaScript ES6, then you receive the following error: Error [ERR_MODULE_NOT_FOUND]: Cannot find package.To resolve these errors, create a deployment package or Lambda layer that includes the libraries that you want to use in your Node.js code.For Node.js runtimes 16 and earlier, Lambda doesn't support layered JavaScript ES module dependencies. You must include the dependencies in the deployment.Lambda supports JavaScript ES module dependencies for Node.js 18.Use the following steps to create a Lambda layer. Create a Lambda layer instead of a deployment package so that you can reuse the layer across multiple Lambda functions. Each Lambda runtime adds specific /opt directory folders to the PATH variable. If the layer uses the same folder structure, then your Lambda function's code can access the layer content without specifying the path.Important: The library that you import for Node.js must be inside the nodejs/node_modules folder structure.ResolutionIt's a best practice to create a Lambda layer on the same operating system that your Lambda runtime is based on. For example, all versions of Node.js are based on the Amazon Linux 2 Amazon Machine Images (AMI).Create a Lambda layer for a Node.js library:1.    Create an Amazon Elastic Compute Cloud (Amazon EC2) instance from an Amazon Linux 2 AMI.2.    Create an AWS Identity and Access Management (IAM) role with permissions to call the publish-layer-version API. Then, attach the IAM role to the EC2 instance.Note: Your EC2 instance now has permissions to upload Lambda layers for the publish-layer-version API call.3.    Connect to your EC2 instance, and then install Node.js:$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash$ . ~/.nvm/nvm.sh$ nvm install node4.    Create a nodejs folder:$ mkdir nodejs5.    Install the aws-xray-sdk library into the nodejs folder:$ cd nodejs$ npm init  ***answer all queries regarding initialization that create a package.json file***$ npm install --save aws-xray-sdkNote: Replace the aws-xray-sdk example library with the Node.js library that you want to import.6.    Create a zip archive for the aws-xray-sdk library:$ zip -r layer.zip nodejsNote: The aws-xray-sdk library is in the required folder format for a Node.js layer: nodejs/node_modules.7.    Publish the Lambda layer:$ aws lambda publish-layer-version --layer-name xray --zip-file fileb://layer.zip --compatible-runtimes nodejs12.x --region us-east-1Note: To add modules ECMAScript 6 (ES6) or CommonJS in your code read, see JavaScript ES6/CommonJS syntax.Related informationHow do I troubleshoot "permission denied" or "unable to import module" errors when uploading a Lambda deployment package?Follow"


Question: How do I troubleshoot memory and CPU issues in Elastic Beanstalk?

"I want to know why my AWS Elastic Beanstalk CPU and memory usage is too high, and I want to troubleshoot issues."

Answer: "I want to know why my AWS Elastic Beanstalk CPU and memory usage is too high, and I want to troubleshoot issues.Short descriptionTo gain insight into your Elastic Beanstalk CPU and memory usage and troubleshoot issues, do the following:Configure the environment to collect metrics on memory and CPU usageAnalyze log files to troubleshoot memory and CPU usage issuesResolutionConfigure your environment to collect metricsTo gain insight into memory and CPU usage, do the following:Check the health of your Amazon Elastic Compute Cloud (Amazon EC2) instances with the Elastic Beanstalk health agent.Monitor your instances using Amazon CloudWatch.Turn on enhanced health reporting and monitoring, and then review the Environment health page for Amazon EC2 instance metrics.Turn on CloudWatch memory usage metrics in Amazon Linux 2 or Windows environments.Configure advanced health rules for your Elastic Beanstalk environment (such as CPU load averages).Analyze your log files to troubleshoot your memory and CPU issueAfter gathering data from your metrics, do the following:Review the log files from the instances in your Elastic Beanstalk environment.Analyze key log files (such as /var/log/messages), and then search for messages about high CPU or out-of-memory (OOM) events.Use your log file results to figure out why the process is consuming a high amount of CPU or memory.Note: For example, you might find a large number of requests in Apache access logs or NGINX access logs.If the issue is ongoing, then do the following:Connect to your Linux instance using SSH.Use the Linux top or free command to view details on CPU and memory consumption.Configure the ATOP monitoring tool. Review CPU and memory utilization stats.CPU-based command: atop -CMemory-based command: atop -mIf you don't find any errors, then the traffic load to your environment might have increased. To handle increased traffic, try the following:Scale up by increasing the instance type.Scale out to decrease the load per instance by increasing the size of the environment's Auto Scaling group.Follow"


Question: How do I troubleshoot issues related to PostGIS extension when upgrading my RDS for PostgreSQL instance?

The major version upgrade for my Amazon Relational Database Service (Amazon RDS) for PostgreSQL instance has failed due to issues with PostGIS extension.

Answer: "The major version upgrade for my Amazon Relational Database Service (Amazon RDS) for PostgreSQL instance has failed due to issues with PostGIS extension.Short descriptionThe major version upgrade for your Amazon RDS for PostgreSQL instance might fail for multiple reasons. The most common reason might be that you didn't update the PostGIS extension or dependent extensions, such as address_standardizer, address_standardizer_data_us, postgis_tiger_geocoder, postgis_topology, or postgis_raster, to the current supported version before performing the major version upgrade.To check if your RDS PostgreSQL instance major version upgrade failed due to PostGIS or dependent extensions, do the following:1.    Check the events for the instance. You might see an event similar to the following:Database instance is in a state that cannot be upgraded: PreUpgrade checks failed: The instance could not be upgraded because one or more databases have settings or usages that are not compatible with the target engine version. Please check the precheck log file for more details2.    Check the precheck log files for the database instance that was created during the upgrade process. During the major version upgrade, RDS for PostgreSQL performs certain precheck procedures on the instance to identify issues that might cause the upgrade process to fail. You can find information about any issues encountered during the precheck in the pg_upgrade_precheck.log file. Amazon RDS appends a timestamp to the file name.If the precheck log file includes a message similar to the following, then the upgrade process failed due to PostGIS or dependent extensions.------------------------------------------------------------------Upgrade could not be run on Sun May 22 14:20:45 2022------------------------------------------------------------------The instance could not be upgraded from 9.6.22.R1 to 12.7.R1 because of following reasons.Please take appropriate action on databases that have usages incompatible with requested major engine version upgrade and try again.- Following usages in database 'test_db' need to be corrected before upgrade:-- The instance could not be upgraded because the PostGIS extension and its dependent extensions (address_standardizer,address_standardizer_data_us, postgis_tiger_geocoder, postgis_topology, postgis_raster) installation in one or more databases isnot compatible with your desired upgrade path. Please upgrade postgis and its dependent extensions to version supported in requested version.----------------------- END OF LOG  ----------------------To check the installed version of PostGIS and dependent extensions, run the following commands:postgres=> select * FROM pg_available_extensions where name like '%postgis%';postgres=> select * FROM pg_available_extensions where name like '%address_standardizer%';postgres=> select probin from pg_proc where proname = 'postgis_raster_lib_version';The value in the installed_version column shows the currently installed version of the extension. If this column has no value, then it means that the extension isn't installed.ResolutionTo troubleshoot the upgrade failure in your RDS for PostgreSQL instance, use one of the following methods based on your use case.Remove the PostGIS extensionDrop the extension and recreate it after the upgrade is successful under the following conditions:You don't need the PostGIS or dependent extensions in the instance.An update path doesn't exist for the extension.Note: Dropping the extension also removes the dependent columns.1.    To drop the extension, run the following command:postgres=> DROP EXTENSION example_extension CASCADE;2.    After dropping the extensions in all the databases where they are installed, upgrade your RDS for PostgreSQL instance.3.    After upgrading the instance, create the extension, if needed, by running the following command:postgres=> CREATE EXTENSION extension_name;Update the PostGIS extensionDo the following:Perform the upgrade in multiple steps.Update the extension version in each step before upgrading the RDS for PostgreSQL instance.Check for the next highest supported update path for the extension installed in your RDS for PostgreSQL instance. Run the following command:postgres=> SELECT * FROM pg_extension_update_paths('extension_name') WHERE source='current_version' AND target NOT LIKE '%next%' AND source<target AND path LIKE '%--%';Be sure to replace the following:extension_name with the name of the installed extensioncurrent_version with the currently installed version of the extensionIf the command doesn't return an update path, but the installed_version and default_version values of the extensions are same, then the extension is already updated. In this case, you can perform the engine version upgrade.For example:postgres=> SELECT * FROM pg_extension_update_paths('postgis') WHERE source='2.3.7' AND target NOT LIKE '%next%' AND source<target AND path LIKE '%--%';Example output:source | target |     path------+--------+--------------2.3.7  | 2.5.2  | 2.3.7--2.5.2 (1 row)If the values for the installed_version and default_version are different, upgrade the installed PostGIS and dependent extensions to the highest supported target version.To upgrade the extension, run the following command:postgres=> ALTER EXTENSION PostgreSQL-extension UPDATE TO 'new-version';Note: Be sure to update the PostGIS extension along with the dependent extensions in the instance. These extensions are installed at the database level. Therefore, be sure to update the extensions in all the databases where they were installed.For updating PostGIS and postgis_raster extension when you're on PostGIS version 2.5.0 or later, run the following command:postgres=> SELECT postgis_extensions_upgrade();In PostGIS version 3.0 and later, the raster functionality is broken out as a separate extension. If you don't use raster, then run the following command after updating the extension:postgres=> DROP EXTENSION postgis_raster;After updating the extensions, you can upgrade the RDS for PostgreSQL instance to the engine version in which these extension versions are supported.After upgrading your instance, you can repeat the steps of updating the extension version followed by upgrading the RDS instance engine version until you install the target engine version.Related informationHow to perform a major version upgradeUsing PostgreSQL extensions with Amazon RDS for PostgreSQLFollow"


Question: How can I resolve node label and YARN ResourceManager failures in Amazon EMR?

"I enabled node labels on an Amazon EMR cluster. Then, YARN ResourceManager failed."

Answer: "I enabled node labels on an Amazon EMR cluster. Then, YARN ResourceManager failed.Short descriptionThis issue affects Amazon EMR release versions 5.19.0 - 5.21.0. In these versions, Amazon EMR stores node label files in HDFS:DEFAULT_DIR_NAME = "node-labels"MIRROR_FILENAME = "nodelabel.mirror"EDITLOG_FILENAME = "nodelabel.editlog"Amazon EMR stores these files at the following location in yarn-site.xml on all nodes: yarn.node-labels.fs-store.root-dir: '/apps/yarn/nodelabels'. The issue happens when these files become corrupted when you lose all nodes that contain the file's blocks during a resize operation. ResourceManager then restarts, gets stuck in a restart loop, and then CommonNodeLabelsManager throws an exception.To find the exception, search for "org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager" in /var/log/hadoop-yarn/yarn-yarn-resourcemanager-*.log.To resolve this error, delete the node label files. Then, restart ResourceManager to recreate the files.Resolution1.    Check file system health and locate the blocks:hdfs fsck /apps/yarn/nodelabels/ -locations -blocks -files2.    Remove the files:hdfs dfs -rm -skipTrash /apps/yarn/nodelabels/*3.    Restart ResourceManager:sudo stop hadoop-yarn-resourcemanagersudo start hadoop-yarn-resourcemanager4.    When ResourceManager restarts, it recreates the node label files. This resolves the restart loop. However, you can't submit YARN applications yet. Before you can submit YARN applications, manually add node label entries:yarn rmadmin -addToClusterNodeLabels "CORE(exclusive=false)"5.    List the labels to confirm that ResourceManager recreated them:yarn cluster --list-node-labelsRelated informationUnderstand node types: master, core, and task nodesFollow"


Question: How can I receive a notification when a deadlock event occurs on my Amazon RDS SQL Server DB instance?

I have an Amazon Relational Database Service (Amazon RDS) DB instance that runs SQL Server. I want to be notified as soon as a deadlock event occurs on my RDS DB instance. How can I do this?

Answer: "I have an Amazon Relational Database Service (Amazon RDS) DB instance that runs SQL Server. I want to be notified as soon as a deadlock event occurs on my RDS DB instance. How can I do this?Short DescriptionSQL Server uses a lock monitor thread to monitor deadlock events. If a deadlock event is detected, then SQL Server uses a resource/cost-based mechanism to declare one of the transactions a deadlock victim. The victim transaction is rolled back and error code 1205 is posted in the error logs.To resolve this issue, enable deadlock trace flags (1222, 1204) to capture deadlock events in the SQL error logs. Next, publish SQL Server error logs to Amazon CloudWatch and create CloudWatch alarms and Amazon Simple Notification Service (Amazon SNS) notifications that send an alert whenever a deadlock event occurs.ResolutionEnable trace flags through RDS Parameter groupsNote: If you have an existing custom parameter group, then skip to step 5.Open the Amazon RDS console, and then choose Parameter groups from the navigation pane.Choose Create parameter group.For Parameter group family, choose the SQL Server version and edition you are using. For example, SQL Server 2016 Enterprise Edition uses sqlserver-ee-13.0.Enter a Group name and Description, and then choose Create.On the Parameter groups page, choose the group that you created in the previous step.Choose Edit parameters, and check the box beside 1204 and 1222.Edit the Values for both 1204 and 1222 to 1.Choose Preview changes. On the next page, choose Save changes.From the navigation pane, choose Databases.In the DB identifier section, choose your RDS DB instance.Choose Modify.From the Database options section, for DB parameter group, choose the parameter group that you created.Note: Creating a new parameter group and attaching it to an RDS DB instance requires that you reboot the RDS DB instance for the changes to take effect.Push the SQL Server error log to Amazon CloudWatchFrom the Error logs section, choose the box for Error log. This makes sure that the SQL Server error logs are published to CloudWatch Logs.Choose Continue.From the Scheduling of modifications section, choose Apply immediately, and then choose Modify DB Instance.Note: Your RDS DB instance is now in the modifying status. Wait until it has returned to the available status. If you used an existing parameter group, skip step 4 as you do not need to reboot your RDS DB instance. If you have created a new parameter group, proceed with step 4.From the Databases page, choose your RDS DB instance, and then choose Actions. Reboot the RDS DB instance for the changes to take effect.After the RDS DB instance comes online, you can verify the deadlock trace status by running DBCC TRACESTATUS (1222, 1204) on your RDS DB instance. If the Global trace command returns 1 in the output windows, then the deadlock trace is enabled.Simulate a deadlock transactionSimulate a deadlock transaction on your RDS DB instance to validate if the deadlock victim queries are logged in the SQL Server error logs. You can check the deadlock events in SQL Server error logs by running the following query on your RDS DB instance:sp_readerrorlog 0, 1, 'deadlock'Create a filter pattern and Amazon CloudWatch alarmOpen the CloudWatch console.From the Logs section, choose Log groups.Choose the SQL Server error logs of your RDS DB instance. The logs are listed in the following format:(/aws/rds/instance/<Your-RDS-Instance-Name>/error)Choose Create metric filter.From the Define Logs Metric Filter page, in the Filter Pattern section, enter Deadlock.Choose Assign metric.Enter Deadlock in both the Filter Name and Metric Name fields.Choose Create Filter. After the deadlock filter is created, choose Create alarm.From the Specify metric and conditions page, under Metric name, enter Deadlock.Under Statistic, enter Minimum.Under Period, from the dropdown menu, choose the time period for the alarm, for example, 1 minute.From the Conditions section, choose the following settings:Threshold type: StaticWhenever Deadlock is: Greater > thresholdThan: Enter 013. Choose Next.14. From the Notification section, under Alarm state trigger, choose In alarm.15. Select an Amazon SNS topic, or choose Create new topic using the email address you want to receive alerts, and then choose Next.16. From the Name and description section, enter an Alarm name and Alarm description, and choose Next.17. From the Preview and create page, review your alarm configuration, and then choose Create alarm.Use the same process to create CloudWatch alarms for other SQL Server fatal errors and login failed errors. This lets you identify and receive notifications about many SQL Server errors before the database system has a major issue.Related InformationMySQL Database Log FilesFollow"


Question: How can I troubleshoot stage failures in Spark jobs on Amazon EMR?

I want to troubleshoot stage failures in Apache Spark applications on Amazon EMR.

Answer: "I want to troubleshoot stage failures in Apache Spark applications on Amazon EMR.Short descriptionIn Spark, stage failures happen when there's a problem with processing a Spark task. These failures can be caused by hardware issues, incorrect Spark configurations, or code problems. When a stage failure occurs, the Spark driver logs report an exception similar to the following:org.apache.spark.SparkException: Job aborted due to stage failure: Task XXX in stage YYY failed 4 times, most recent failure: Lost task XXX in stage YYY (TID ZZZ, ip-xxx-xx-x-xxx.compute.internal, executor NNN): ExecutorLostFailure (executor NNN exited caused by one of the running tasks) Reason: ...ResolutionFind the reason codeFor Spark jobs submitted with --deploy-mode client, the reason code is in the exception that's displayed in the terminal.For Spark jobs submitted with --deploy-mode cluster, run the following command on the master node to find stage failures in the YARN application logs. Replace application_id with the ID of your Spark application (for example, application_1572839353552_0008).yarn logs -applicationId application_id | grep  "Job aborted due to stage failure" -A 10You can also get this information from YARN ResourceManager in the application master container.Resolve the root causeAfter you find the exception, use one of the following articles to resolve the root cause:How do I resolve "Container killed on request. Exit code is 137" errors in Spark on Amazon EMR?How do I resolve "no space left on device" stage failures in Spark on Amazon EMR?How do I resolve ExecutorLostFailure "Slave lost" errors in Spark on Amazon EMR?Follow"


Question: How can I verify signatures generated by AWS KMS asymmetric keys?

"I want to use Key Management Service (AWS KMS) to sign a file. Then, I want to share the file, its signature, and the public key for verification that the signature is valid. I don't want to provide API access for users to access my AWS KMS key."

Answer: "I want to use Key Management Service (AWS KMS) to sign a file. Then, I want to share the file, its signature, and the public key for verification that the signature is valid. I don't want to provide API access for users to access my AWS KMS key.ResolutionThe following example uses AWS KMS with an ECC_NIST_P256 (secp256r1) asymmetric key pair. When AWS KMS generates a signature file using this key pair, it's created according to NIST FIPS 168-4. An ECDSA digital signature containing (r, s) values is generated as specified in ANS X9.62. Because of the open standard, you can verify this signature using OpenSSL.To get the signature format for RSA key pairs, follow the instructions to create and manage your AWS KMS keys.Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Sign a local file using AWS KMSAfter creating the AWS KMS key pair in your account, refer to the key pair using the AWS CLI to sign a file. The response received from the AWS KMS API is encoded in Base64. The following example uses the --query parameter to get the signature value from the response and places it in the sign.b64 file.[ec2-user@ip-172-31-23-22 ~]$ aws kms sign --key-id arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab --message fileb://message.txt --signing-algorithm ECDSA_SHA_256 --query 'Signature' --output text > sign.b64Note: You can submit messages up to 4096 bytes. To sign a larger message, generate a hash digest of the message. Then, provide the hash digest in the message parameter. To indicate whether the message is a full message or a digest, use the MessageType parameter. Be sure to take note of the signing algorithm, as this is needed for verifying the signature later.Because the signature is in Base64 format, you can convert it to binary format using the Linux base64 encoding command similar to the following:[ec2-user@ip-172-31-23-22 ~]$ base64 -d sign.b64 > sign.binTo decode Base64 files for Windows OS, run the following command:certutil -decode C:\Temp\sign.b64 C:\Temp\sign.binVerify the AWS KMS signature using OpenSSLYou can now share the sign.b64 signature file. To verify the signature file, you must have the public key. To get the public key, run the AWS CLI command get-public-key similar to the following:[ec2-user@ip-172-31-23-22 ~]$ aws kms get-public-key --key-id arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab --output text --query 'PublicKey' > KMSPublic.b64To convert the base64 file to DER encoding with another file named KMSPublic.key, run the following command:[ec2-user@ip-172-31-23-22 ~]$ base64 -d KMSPublic.b64 > KMSPublic.keyYou now have the public key and signature in binary format with the message.txt file. To verify the signature, run the OpenSSL command dgst similar to the following:[ec2-user@ip-172-31-23-22 ~]$ openssl dgst -sha256 -verify KMSPublic.key -keyform DER -signature sign.bin message.txtVerified OKThis example output of "Verified OK" indicates that the verification was successful.If you didn't receive a verification response, make sure that:The OpenSSL signature algorithm is the same one that was used to sign the file.Your files aren't Base64 encoded.Related informationCryptographic primitivesAWS KMS conceptsFollow"


Question: How do I resolve the error "Exception in thread "main" java.lang.RuntimeException: Read throughput should not be less than 1" when moving data from DynamoDB to S3?

"I created an AWS Data Pipeline to export data from Amazon DynamoDB to Amazon Simple Storage Service (Amazon S3). When I try to export the data, I get the "Exception in thread "main" java.lang.RuntimeException: Read throughput should not be less than 1" error."

Answer: "I created an AWS Data Pipeline to export data from Amazon DynamoDB to Amazon Simple Storage Service (Amazon S3). When I try to export the data, I get the "Exception in thread "main" java.lang.RuntimeException: Read throughput should not be less than 1" error.ResolutionIf you're creating a new pipeline to export data from DynamoDB table, then create a pipeline using the Export DynamoDB table to S3 Data Pipeline template. For more information on creating the pipeline, see Part two: Export data from DynamoDB.If you're using an existing pipeline to export data from DynamoDB to S3, then you must use the latest EMR-DynamoDB connector for the pipeline. You might get the error when you export data with an older version of the EMR-DynamoDB connector.Be sure that the Amazon EMR version for the DynamoDB table is 5.24.0 or later. DynamoDB tables configured for On-Demand Capacity are supported only when using Amazon EMR release version 5.24.0 or later. For more information, see Export data from DynamoDB - Before you begin.To update your existing pipeline to use the latest EMR-DynamoDB connector and Amazon EMR version 5.24.0 or later, do the following:1.    Open the AWS Data Pipeline console.2.    Select the pipeline, and then choose Actions.3.    Choose Edit.4.    Choose EmrActivity in the left pane.5.    Expand the Activities pane, and then find the EmrActivity object.6.    In the Step field, enter the following JAR name:s3://dynamodb-dpl-#{myDDBRegion}/emr-ddb-storage-handler/4.11.0/emr-dynamodb-tools-4.11.0-SNAPSHOT-jar-with-dependencies.jar7.    Choose EmrCluster in the left pane.8.    Expand the Resources pane.9.    For Release Label, enter emr-5.24.0 or any of the later EMR release versions.10.    Choose Save.Related informationHow can I use Data Pipeline to back up a DynamoDB table to an S3 bucket that's in a different account?Follow"


Question: Why is the Deleted Documents metric so high in my Amazon OpenSearch Service cluster?

"I've deleted documents in my Amazon OpenSearch Service cluster, but I don't see any disk space reclaimed."

Answer: "I've deleted documents in my Amazon OpenSearch Service cluster, but I don't see any disk space reclaimed.Short descriptionIn OpenSearch Service, the DeletedDocuments metric is a counter that shows the number of documents that are marked for deletion. The DeletedDocuments metric shows an increase after the delete requests are processed and decreases after the index segments are merged within your cluster.OpenSearch Service automatically runs the merge API operation using the merge policy setting. During a merge, the smaller segments are merged into larger segments to maintain the index size. Documents marked for deletion are also expunged to free up additional disk space.To reclaim disk space immediately, you can delete an index instead of deleting individual documents. Or, you can also use the force merge API along with the only_expunge_deletes parameter to clear up the deleted documents within an index.To maintain the index metadata while reclaiming disk space in your cluster, consider the following approaches:Check the number of deleted documents.Confirm the size of your documents.Expunge the deleted documents.Add storage space to your OpenSearch Service domain.ResolutionCheck the number of deleted documentsTo check the number of deleted documents in your OpenSearch Service cluster, run the cluster stats API. The value obtained from the cluster stats API call appears in the DeletedDocuments metric for your cluster. The output returns a summation of deleted documents for all the indices present in your cluster. You can check this count using the "docs.deleted" field in the response output.For example, if your cluster has three indices (index1, index2, and index3), then run the index stats API call:GET index1/_stats..."docs": {     "count": 100,     "deleted": 1}... GET index2/_stats..."docs": {     "count": 100,     "deleted": 5}...GET index3/_stats..."docs": {     "count": 100,     "deleted": 8}...The cluster stats API call then adds the "docs.deleted” field for all indices that are present in your cluster:..."docs": {     "count": 1227677521,     "deleted": 14}...If you delete index2, then the cluster stats API call calculates the values only for index1 and index3:GET _cluster/stats..."docs": {     "count": 1227677521,     "deleted": 9}The segments are now merged and the index metadata for index2 is erased. As a result, the DeletedDocuments metric value decreased to 9.Confirm the size of your documentsTo check the document sizes and count for an index, use the cat indices API. Make sure that the new document is the same size as the existing document in your OpenSearch Service cluster. Using the same document size makes sure that deleted documents don't take up additional disk space.Expunge the deleted documentsTo manually reclaim disk space, run the force merge API along with the only_expunge_deletes parameter set as “true”:POST /<index-name>/_forcemerge?only_expunge_deletes=trueNote: This operation expunges only the segments containing documents marked for deletion.As a result, the force merge decreases the amount of disk space in use. After the new segments are created, old segments are removed and new segments no longer contain the documents marked for deletion. For more information about deleted documents, see Lucene's handling of deleted documents on the Elasticsearch website.However, be aware of the following when performing the force merge operation:Perform force merge on your cluster only when there is enough free storage space. This action is a resource-intensive operation.The force merge operation triggers an I/O intensive process and blocks all new requests to your cluster until the merge is complete.Only call the force merge operation against read-only indices, when no additional data is being written to the index. Calling force merge against a read/write index can cause very large segments to be produced (>5 GB per segment). When this happens, the automatic merge policy doesn't consider these very large segments for future merges until the segments contain mostly deleted documents. As a result, disk usage increases and search performance worsens.You can also use the delete by query API or delete API to manually delete any documents in your cluster.Reclaim disk space immediatelyTo reclaim disk space immediately, use the delete index API. Deleting an index doesn't create any delete markers. Instead, the delete index API clears the index metadata, and disk space is immediately reclaimed. The reclaimed disk space is reflected in the DeletedDocuments metric.Note: It's a best practice to delete old indices that aren't in use. If you delete an active index, make sure to block the automatic creation of indices. For more information, see Automatically create data streams and indices on the Elasticsearch website.Related informationHow do I troubleshoot low storage space in my Amazon OpenSearch Service domain?Monitoring OpenSearch cluster metrics with Amazon CloudWatchFollow"


Question: Why is my Kinesis data stream throttling?

Why is my Amazon Kinesis data stream throttling?

Answer: "Why is my Amazon Kinesis data stream throttling?Short DescriptionEven though your metrics are within Kinesis data stream quotas, your stream can throttle for the following reasons:WriteThroughputExceeded and Rate Exceeded errors.Enhanced Kinesis stream monitoring is disabled.Hidden micro spikes in the CloudWatch metrics.CloudWatch metrics report only on successful operations and exclude any failed operations.ResolutionYour Amazon Kinesis data stream can throttle for the following reasons:WriteThroughputExceeded and Rate Exceeded errorsYour stream can produce WriteThroughputExceeded and Rate Exceeded throttling errors, which are caused by the following:The number of records written to the Kinesis data stream exceeds the stream quotas.The size of the records (including the partition keys) exceeds 1 MB.The total throughput in bytes exceeds the Kinesis stream limits.The producer is making too many rapid requests to write to the stream, usually indicated with an error stipulating "Slow down" or "Rate exceeded".Enhanced Kinesis stream monitoring is disabledSome shards in your Kinesis data stream might receive more records than others. This can lead to throttling errors in the stream, resulting in overworked shards, also known as hot shards. Hot shards indicate that the partition key being used in PUT operations isn't equally distributing the records across the shards in a Kinesis stream.Hot shards can be hidden in the metrics if the disable-enhanced-monitoring feature is on. This is because the stream level metrics are an aggregated value across all the shards present. When enhanced monitoring of streams is disabled, you can't examine any streams on an individual basis. To examine streams on a per shard basis, use the enable-enhanced-monitoring command.Hidden micro spikes in the CloudWatch metricsKinesis stream quotas are determined per shard and per second. When values aggregate over the 60 second time limit in CloudWatch, a micro spike that breaches the Kinesis time quotas can occur. Also, the overall number of records within a minute can seem low because metrics are aggregated for 60 seconds. However, a record at any particular second within the minute actually represents a higher number. The traffic might show that it is below stream quotas, but the shard throttled within that particular second appears as throttling on the stream.CloudWatch metrics only report on successful operations and exclude failed operationsKinesis metrics record only successful operations on the stream. Therefore, operations that are throttled might not be ingested into the Kinesis data stream. This can result in a breach of stream limits and throttling without any metric indications.When there are failed records that aren't able to enter the Kinesis data stream, the stream throttles. If there is a retry mechanic in the producer, failed records are tried again. This can also cause a delay in processing.To check if there are too many records being sent to the Kinesis data stream, calculate the total number of records sent. You can do this by adding the total number of incoming records to the number of throttled records. This gives you an overview of the number of incoming records during throttling events.Related InformationKinesis Data Streams QuotasFollow"


Question: How can I use a Lambda-backed API as the custom identity provider for my Transfer Family server without using CloudFormation?

"I want to use an AWS Lambda-backed Amazon API Gateway API as the custom identity provider for my AWS Transfer Family server. However, I don't want to use one of the AWS CloudFormation stack templates for my configuration. Or, I want to customize my configuration. How can I set up a custom identity provider this way?"

Answer: "I want to use an AWS Lambda-backed Amazon API Gateway API as the custom identity provider for my AWS Transfer Family server. However, I don't want to use one of the AWS CloudFormation stack templates for my configuration. Or, I want to customize my configuration. How can I set up a custom identity provider this way?Short descriptionYou can choose to manually configure a Lambda-backed API Gateway API as the custom identity provider for your Transfer Family server, instead of using one of the CloudFormation stack templates. To do this, you can directly configure a Lambda function and an API Gateway API with your server.For example, to set up the same configuration as the basic stack template, follow these steps:1.    Create an AWS Identity and Access Management (IAM) role for the Lambda execution role.2.    Create the Lambda function.3.    (Optional) Create an IAM role for the API Gateway logging role.4.    Set up an API Gateway API as the identity provider.5.    Create IAM roles for the Transfer Family server and server users.6.    Create the Transfer Family server.7.    Set up your credential store.ResolutionImportant: This configuration is one example of how to set up your custom identity provider without using a CloudFormation stack template. You must modify the configuration steps based on your specific needs for the custom identity provider.Create an IAM role for the Lambda execution role1.    Use the IAM console to create the execution role. Note: For this configuration, you need only the basic execution permissions. However, be sure to customize the permissions for your use case.2.    Modify the role's trust policy to add the following statement:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Service": "lambda.amazonaws.com"      },      "Action": "sts:AssumeRole"    }  ]}Create the Lambda functionFollow the steps to create a Lambda function using the console. Be sure to configure the following:1.    For the Execution role, select the Lambda execution role that you created.2.    For the Function code, you can use the default Lambda function for authentication. Or, you can use a custom function.(Optional) Create an IAM role for the API Gateway logging roleYou can use Amazon CloudWatch Logs to help you debug errors with your REST API. To create a logging role, follow these steps:1.    Create an IAM role for API Gateway and attach the AmazonAPIGatewayPushToCloudWatchLogs managed policy to the role.2.    Modify the role's trust policy to add the following statement:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Service": "apigateway.amazonaws.com"      },      "Action": "sts:AssumeRole"    }  ]}Set up an API Gateway API as the identity provider1.    Open the API Gateway console.2.    From the navigation pane, choose APIs.3.    Choose Create API.4.    For Choose an API type, navigate to REST API, and then choose Build. Then, enter the following: For Choose the protocol, select REST. For Create new API, select New API. For API name, enter a name for your API. For Endpoint Type, select Regional.Note: Don't build your API from the REST API Private option.5.    Choose Create API.6.    From the navigation pane, choose Models. Then, choose Create.7.    For Model name, enter UserConfigResponseModel.8.    For Content type, enter application/json.9.    For Model schema, enter the following:{"$schema":"http://json-schema.org/draft-04/schema#","title":"UserUserConfig","type":"object","properties":{"Role":{"type":"string"},"Policy":{"type":"string"},"HomeDirectory":{"type":"string"},"PublicKeys":{"type":"array","items":{"type":"string"}}}}10.    ChooseCreate model.11.    From the navigation pane, choose Resources.12.    Expand Actions, and then choose Create Resource.13.    Create a resource for your Transfer Family servers. For Resource Name, enter Servers. For Resource Path, enter servers.14.    Create a resource for the server ID. For Resource Name, enter ServerID. For Resource Path, enter {serverId}.15.    Create a resource for your server users. For Resource Name, enter Users. For Resource Path, enter users.16.    Create a resource for the user names. For Resource Name, enter Username. For Resource Path, enter {username}.17.    Create a resource for user configurations. For Resource Name, enter GetUserConfig. For Resource Path, enter config.18.    Expand Actions, and then choose Create Method.19.    From the dropdown list, select GET. Then, choose the check icon next to GET to create the method.20.    For GET - Setup, enter the following: For Integration type, select Lambda Function. For Lambda Region, select the AWS Region that your Lambda function is in. For Lambda Function, select the function that you created.21.    Choose Save.22.    Choose Method Request. Then, enter the following: For Authorization, select AWS_IAM. For HTTP Request Headers, choose Add header. Then, enter Password as a header name. For URL Query String Parameters, choose Add query string. Then, enter protocol as a query string name. Choose Add query string again, and then enter sourceIp as another query string name.23.    From the navigation bar of the GET method, choose Method Execution to return to the GET - Method Execution page.24.    Choose Integration Request. Then, expand Mapping Templates.25.    For Request body passthrough, select When no template matches the request Content-Type header.26.    For Content-Type, enter application/json. Choose the check icon to confirm.27.    For the template, enter the following:{ "username": "$input.params('username')", "password": "$util.escapeJavaScript($input.params('Password')).replaceAll("\\'","'")", "serverId": "$input.params('serverId')", "protocol": "$input.params('protocol')","sourceIp": "$input.params('sourceIp')" }28.    ChooseSave.29.    From the navigation bar of the GET method, choose Method Execution to return to the GET - Method Execution page.30.    Choose Integration Response. Confirm that there are no mapping templates and that the response body is passed through to the method response.31.    From the navigation bar of the GET method, choose Method Execution to return to the GET - Method Execution page.32.    Choose Method Response. Then, expand 200.33.    For Response Body for 200, edit the Models value, and then select the UserConfigResponseModel that you created in steps 7 through 10. Choose the check icon to confirm.34.    From the Resources navigation pane, choose GET, choose Actions, and then choose Deploy API. Enter the following: For Deployment stage, select [New Stage]. For Stage name, enter a name for the stage.35.    Choose Deploy.36.    From the navigation pane, choose Stages. Then, choose Create and enter the following: For Stage name, enter prod. For Deployment, select the deployment that you created as a new stage.37.    Choose Create.Create IAM roles for the Transfer Family server and server users1.    Create an IAM role for the Transfer Family server that grants permissions to invoke the API that you created, similar to the following:{  "Version": "2012-10-17",  "Statement": [      {          "Action": [              "execute-api:Invoke"          ],          "Resource": "arn:aws:execute-api:{Region}:{AWS-Account-ID}:{API-Gateway-id}/{stage}/GET/*",          "Effect": "Allow"      },      {          "Action": [              "apigateway:GET"          ],          "Resource": "*",          "Effect": "Allow"      }  ]}2.    Modify the role's trust policy to add the following statement:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Service": "transfer.amazonaws.com"      },      "Action": "sts:AssumeRole"    }  ]}3.     Create an IAM role for the Transfer Family server users that grants access to the Amazon Simple Storage Service (Amazon S3) bucket. Then, modify the role's trust policyto add the statement from step 2.4.    Create an IAM role for the Transfer Family server that sends logs to CloudWatch. Then, modify the role's trust policy to add the statement from step 2.Create the Transfer Family serverFollow the steps to create a Transfer Family server. Be sure to configure the following:1.    For Identity provider type, select Custom. Then, for Custom provider, enter the API that you created.2.    For Invocation role, select the IAM role that grants the server permissions to invoke the API that you created.3.    For Logging role, select the IAM role for the server that sends logs to CloudWatch.Set up your credential storeYou must set up your credential store for the user configuration. For more information on setting up a credential store, see Default Lambda function and Enable password authentication for AWS Transfer for SFTP using AWS Secrets Manager.At minimum, you must set up the following fields for users to be able to perform actions on the Amazon S3 bucket:User namePasswordHomeDirectoryRoleNote: The role is the Amazon Resource Name (ARN) of the IAM role that you created to grant server users access to the Amazon S3 bucket.Testing and troubleshooting your setupYou can test your setup in the following ways:Use the API Gateway console to test.Use the Transfer Family console or the AWS Command Line Interface (AWS CLI) to test the identity provider.Try logging in to the Transfer Family server from a client.If you run into issues with your setup, you can troubleshoot in the following ways:Check the response from the test identity provider. If the status code isn't 200, then there might be issues with the API setup.Confirm that the IAM roles that you created have the correct permissions and trust relationships.This configuration turns on logging, and you can review Lambda execution logs, API Gateway logs, and Transfer Family logs to identify issues.If the custom identity provider setup that you need matches one of the available CloudFormation stacks, then you can try deploying the CloudFormation template and compare with your setup to identify issues.Follow"


Question: Why do I get errors when I try to read JSON data in Amazon Athena?

"When I try to read JSON data in Amazon Athena, I receive NULL or incorrect data errors."

Answer: "When I try to read JSON data in Amazon Athena, I receive NULL or incorrect data errors.ResolutionCheck the following common issues:Use the correct JSON SerDeAthena processes JSON data using one of two JSON SerDes:The native Apache Hive/HCatalog JsonSerDe (org.apache.hive.hcatalog.data.JsonSerDe)The OpenX SerDe (org.openx.data.jsonserde.JsonSerDe)If you're not sure which SerDe that you used, try both of the SerDe versions. If you use the OpenX SerDe, you can ignore malformed records to identify the lines that are causing the errors, as shown in the following example. When ignore.malformed.json is set to true, malformed records return as NULL.CREATE EXTERNAL TABLE json (    id int,    name string)ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'WITH SERDEPROPERTIES ( 'ignore.malformed.json' = 'true')LOCATION 's3://awsexamplebucket/';Query the new table to identify the files with malformed records. For example:SELECT "$PATH", * FROM your_table where your_column is NULLUse one line per recordThe following JSON records are formatted correctly:{ "id" : 50, "name":"John" }{ "id" : 51, "name":"Jane" }{ "id" : 53, "name":"Jill" }The following JSON records are formatted incorrectly:{  "id" : 50,  "name":"John"},{  "id" : 51,  "name":"Jane"}{  "id" : 53,  "name":"Jill"}These records are also formatted incorrectly:{ "id" : 50, "name":"John" } { "id" : 51, "name":"Jane" } { "id" : 53, "name":"Jill" }Use the correct data type in each columnThe second line in the following example contains an incorrect data type for "age". The column value should be "11" instead of "eleven". This causes the following error message: HIVE_BAD_DATA: Error parsing field value 'eleven' for field 1: For input string: "eleven".{"name":"Patrick","age":35,"address":"North Street"}{"name":"Carlos","age":"eleven","address":"Flowers Street"}{"name":"Fabiana","age":22,"address":"Main Street"}Use the correct extension for compressed JSON filesWhen you use a compressed JSON file, the file must end in ".json" followed by the extension of the compression format, such as ".gz". For example, this is a correctly formatted extension for a gzip file: "myfile.json.gz".Use case-insensitive columns or set the case.insensitive property to falseAthena is case-insensitive by default. If you have column names that differ only by case (for example, “Column” and “column”), Athena generates an error ("HIVE_CURSOR_ERROR: Row is not a valid JSON Object - JSONException: Duplicate key") and your data is not visible in Athena. The easiest way to avoid this problem is to generate your data with case-insensitive columns.If you use the OpenX SerDe, you can use case-sensitive key names. To do this, set the case.insensitive SerDe property to false and add mapping for the uppercase key. For example, to use upper and lowercase columns like this:{"Username": "bob1234", "username": "bob" }Use these SerDe properties:CREATE external TABLE casesensitive_json (user_name String,username String)ROW FORMAT serde 'org.openx.data.jsonserde.JsonSerDe'WITH SERDEPROPERTIES ( 'mapping.user_name' = 'Username','case.insensitive'='false')LOCATION 's3://awsexamplebucket/';Be sure that all rows in the JSON SerDE table are in JSON formatTo find if there are invalid JSON rows or file names in the Athena table, do the following:1.    Create a table with a delimiter that's not present in the input files. Run a command similar to the following:CREATE EXTERNAL TABLE IF NOT EXISTS json_validator (jsonrow string) ROW FORMAT DELIMITEDFIELDS TERMINATED BY '%'location 's3://awsexamplebucket/';2.    Run a query similar to the following to return the file name, row details, and Amazon S3 path for the invalid JSON rows.WITH testdataset AS (SELECT "$path" s3path,jsonrow,try(json_parse(jsonrow)) isjson FROM json_validator)SELECT * FROM testdataset WHERE ISJSON IS NULL;Related informationBest practices for reading JSON dataTroubleshooting JSON-related errorsFollow"


Question: How do I resolve intermittent connection issues when using a NAT instance?

"I'm using a NAT instance to connect instances to the internet in an Amazon Virtual Private Cloud (Amazon VPC) subnet. But, the instances have intermittent connection issues."

Answer: "I'm using a NAT instance to connect instances to the internet in an Amazon Virtual Private Cloud (Amazon VPC) subnet. But, the instances have intermittent connection issues.Short descriptionThe intermittent connection issues might be related to the following:Port exhaustion on the sourceNetwork access control list (network ACL) rulesNetwork issuesResolutionReduce port exhaustion on the sourceVerify that the instances in the private subnet reached their operating system-level connection limits. To get the number of active connections, run the netstat command.Linux:netstat -ano | grep ESTABLISHED | wc --lnetstat -ano | grep TIME_WAIT | wc --lWindows:netstat -ano | find /i "estab" /cnetstat -ano | find /i "TIME_WAIT" /cNote: If you configured the TCP parameter to reuse ports during the TIME_WAIT state, then remove TIME_WAIT from the preceding commands.If the command returns a value that's near the allowed local port range (source port for client connections), then you might experience port exhaustion. To reduce port exhaustion, try one of the following solutions.Increase the operating system's ephemeral port range by running the following command:For Windows, run as administrator:netsh int ipv4 set dynamicport tcp start=1025 num=61000For Linux, run as root:$echo 1024 65535 > /proc/sys/net/ipv4/ip_local_port_rangeNote: Depending on the type of distribution, set the preceding parameter permanently during reboots.-or-Resolve any application-level issues that drain the available connections.Verify that the network ACL rules allow traffic for the ephemeral port rangeThe network ACL of the public and private subnets must allow traffic for the ephemeral port range (1024-65535).For example, to allow your Amazon Elastic Compute Cloud (Amazon EC2) instances to access an HTTPS website, the network ACL associated with the private subnet must have the following rules:Inbound rules:SourceProtocolPort RangeAllow / DenyPUBLIC_IPTCP1024-65535ALLOWOutbound rules:DestinationProtocolPort RangeAllow / DenyPUBLIC_IPTCP443ALLOWThe network ACL that's associated with the NAT instance subnet must have the following rules:Inbound rules:SourceProtocolPort RangeAllow / DenyVPC CIDRTCP443ALLOWPUBLIC_IPTCP1024-65535ALLOWOutbound rules:DestinationProtocolPort RangeAllow / DenyPUBLIC_IPTCP443ALLOWVPC CIDRTCP1024-65535ALLOWTraffic is dropped if the following occurs:The network ACL allows only a subset of the ephemeral port range.The instances in the private subnet or NAT instance use a source port that's outside of the ephemeral port range.Check for network issuesIf the network performance is degraded on the source, NAT, network medium, or destination, then you can experience connection issues. To resolve this issue, see How do I troubleshoot network performance issues between Amazon EC2 Linux instances in a VPC and an on-premises host over the internet gateway?You can also experience network issues when the network exceeds its maximums at the instance level. For supported instance types, you can use the network performance metrics from the Elastic Network Adaptor (ENA) to monitor when traffic exceeds the maximums. To learn more, see Monitor network performance for your EC2 instance.Related informationEphemeral portsFollow"


Question: How do I troubleshoot an "Invalid Refresh Token" error from my Amazon Cognito user pool API?

I need information about how to troubleshoot the "Invalid Refresh Token" error that's returned by the Amazon Cognito user pool API.

Answer: "I need information about how to troubleshoot the "Invalid Refresh Token" error that's returned by the Amazon Cognito user pool API.Short descriptionRefresh tokens issued by an Amazon Cognito user pool are used to retrieve new access and ID tokens. A request for a new access and ID token using the refresh token might fail with an "Invalid Refresh Token" error for the following reasons:You're using a different app client ID than the one that issued the refresh token.Your device tracking is turned on and the device isn't confirmed with the ConfirmDevice API.Your device tracking is turned on and you aren't passing a device key as AuthParameters in the AdminInitiateAuth or InitiateAuth API.Your device tracking is turned on and you obtained a refresh token using the USER_PASSWORD_AUTH authentication flow.ResolutionYou're using a different app client ID than the one that issued the refresh tokenWhen a user authenticates using the AdminInitiateAuth API, the InitiateAuth API, or the hosted UI, the app client ID is used as the ClientId parameter. You must use the same app client ID to refresh the tokens using the AdminInitiateAuth or InitiateAuth API.If you use a different app client ID, then the API to refresh the tokens fail with the "Invalid Refresh Token" error.Your device tracking is turned on and the device isn't confirmed with the ConfirmDevice APIWhen device tracking is turned on for a user pool, you must call the ConfirmDevice API to confirm the device and begin device tracking. When you confirm the device, you can then use the refresh token to get new access and ID tokens.For more information, see How do I use remembered devices in my Amazon Cognito user pool?If you don't confirm the device, and you call the AdminInitiateAuth or InitiateAuth API for new tokens, then you receive the "Invalid Refresh Token" error.Your device tracking is turned on and you aren't passing a device key as AuthParameters in the AdminInitiateAuth or InitiateAuth APIWhen you turn on device tracking for a user pool, a unique device key is returned after successfully authenticating. When you try to get new tokens using the refresh token, you must pass the device key as AuthParameters in the AdminInitiateAuth or InitiateAuth API.Note: Replace example_refresh_token, example_secret_hash, and example_device_key with your own values.{  "AuthFlow": "REFRESH_TOKEN_AUTH",  "AuthParameters": {    "REFRESH_TOKEN": "example_refresh_token",    "SECRET_HASH": "example_secret_hash",      "DEVICE_KEY": "example_device_key"  }}The SECRET_HASH is one of the required AuthParameters when your app client is configured with a client secret.If the device key isn't passed as one of the AuthParameters, you receive the "Invalid Refresh Token" error.Your device tracking is turned on and you obtained a refresh token using the USER_PASSWORD_AUTH authentication flowThe user pool device tracking feature can only be used with the USER_SRP_AUTH authentication flow. The initial authentication using the AdminInitiateAuth API, the InitiateAuth API, or the hosted UI must use the USER_SRP_AUTH flow.When you use the USER_PASSWORD_AUTH authentication flow, you receive the access, ID, and refresh tokens. But, when you attempt to get new tokens using the refresh token, you receive the "Invalid Refresh Token" error.Follow"


Question: How do I resolve problems when connecting to my Amazon RDS DB instance?

I can't connect to my Amazon Relational Database Service (Amazon RDS) DB instance.

Answer: "I can't connect to my Amazon Relational Database Service (Amazon RDS) DB instance.Short descriptionThere are a number of root causes for the inability to connect to an Amazon RDS DB instance. Here are a few of the more common reasons:The RDS DB instance is in a state other than available, so it can't accept connections.Your source to connect to the DB instance doesn't have authorization access in your security group, network access control lists (ACLs), or local firewalls.The wrong DNS name or endpoint is used to connect to the DB instance.The Multi-AZ DB instance failed over, and the secondary DB instance uses a subnet or route table that doesn't allow inbound connections.The user authentication is incorrect because of one of the following reasons:You're using an incorrect user name or password at the database level to access the instance from the DB client.You don't have the required database permissions to access the instance.The client is running on a version that's incompatible with the database version.Tip: You can use the following troubleshooting steps to identify the source of the connectivity issue. Or you can use the AWSSupport-TroubleshootConnectivityToRDS AWS Systems Manager Automation document to diagnose the issue for you. This automation document can diagnose network ACLs based on the primary IP address of the Amazon Elastic Compute Cloud (Amazon EC2) instance. However, ephemeral ports aren't verified. The automation doc also checks security groups based on the primary IP address of the EC2 instance, but that automation doesn't check specific ports. For more information, see Run an automation.ResolutionBe sure that your DB instance is in the available stateIf you recently launched or rebooted your DB instance, then confirm that the DB instance is in the available state in the Amazon RDS console. Depending on the size of your DB instance, it can take up to 20 minutes for the DB instance to become available for network connections.If your DB instance is in the failed state, see Why is my Amazon RDS DB instance in a failed state?Be sure that your DB instance allows connectionsBe sure that traffic from the source connecting to your DB instance isn't gated by one or more of the following:Any Amazon Virtual Private Cloud (Amazon VPC) security groups associated with the DB instance. If necessary, add rules to the security group associated with the VPC that allow traffic related to the source in and out of the DB instance. You can specify an IP address, a range of IP addresses, or another VPC security group. For general information about VPC and DB instances, see Scenarios for accessing a DB instance in a VPC.Any DB security group associated with the DB instance. If the DB instance isn't in a VPC, then the instance might be using a DB security group to gate traffic. Update your DB security group to allow traffic from the IP address range or Amazon EC2 security group that you use to connect.Connections outside a VPC. Be sure that the DB instance is publicly accessible and associated with a public subnet (for example, the route table allows access from an internet gateway). For more information, see Scenarios for accessing a DB instance in a VPC.If your DB instance is in a private subnet, be sure to use VPC peering or AWS Site-to-Site VPN to connect to your instance securely. With Site-to-Site VPN, you configure a customer gateway that allows you to connect your VPC to your remote network. You can use VPC peering by creating a peering connection between your source VPC and your instance's VPC to access the instance from outside its VPC. You can also use an Amazon EC2 instance as a bastion (jump) host.Network ACLs. Network ACLs act as a firewall for resources in a specific subnet in a VPC. If you use ACLs in your VPC, then be sure that they have rules that allow inbound and outbound traffic to and from the DB instance.Network or local firewalls. Ask your network administrator if your network allows traffic to and from the ports that the DB instance uses for inbound and outbound communication.Note: Amazon RDS doesn't accept internet control message protocol (ICMP) traffic, including ping.Troubleshoot potential DNS name or endpoint issuesWhen connecting to your DB instance, you use a DNS name (endpoint) provided by the Amazon RDS console. Be sure to use the correct endpoint. Also, provide the endpoint in the correct format to the client that you use to connect to the DB instance. For more information on DB engine connections and how to use an endpoint in various client applications, see Getting started with Amazon RDS.For example, use nslookup to the DB instance endpoint from an Amazon EC2 instance within the VPC:nslookup myexampledb.xxxx.us-east-1.rds.amazonaws.com Server: xx.xx.xx.xx Address: xx.xx.xx.xx#53See the following example of a non-authoritative answer:Name: myexampledb.xxxx.us-east-1.rds.amazonaws.com Address: 172.31.xx.xTroubleshoot database level issuesBe sure that you're using the correct user name and password to access the instance from your DB client.Be sure that the user has the database permissions to connect to the DB instance.Check for any resource throttling in Amazon RDS, such as CPU or memory contention. This contention might lead to issues with establishing newer connections to the instance.Be sure that the instance hasn't reached the max_connections limit.Check the route tables associated with your instanceWhen you create a Multi-AZ deployment, you launch multiple replica DB instances in different Availability Zones to improve the fault tolerance of your application. Be sure that the subnets associated with each DB instance are associated with the same or similar route tables. If your primary DB instance fails over to a standby replica that's associated with a different route table, then traffic might not be routed correctly. Even if that traffic was previously routed without an issue, it might no longer be routed correctly.For more information about configuring route tables, see Configure route tables. For additional information about Multi-AZ deployments, see Multi-AZ deployments for high availability.Note: If you can connect to your DB instance but you get authentication errors, see How do I reset the master user password for my RDS DB instance?Verify the connectivityVerify your connection by running one of these commands:telnet <RDS endpoint> <port number>nc -zv <RDS endpoint> <port number>If either the telnet or nc commands succeed, then a network connection was established. This means that the issue is likely caused by the user authentication to the database, such as user name and password.Related informationCan't connect to Amazon RDS DB instanceHow can I troubleshoot connectivity to an Amazon RDS DB instance that uses a public or private subnet of a VPC?Security in Amazon RDSUsing SSL/TLS to encrypt a connection to a DB instanceFollow"


Question: How do I upload SSL certificates for my Classic Load Balancer to prevent clients from receiving “untrusted certificate” errors?

A client SSL/TLS connection to my Classic Load Balancer fails with an error message similar to the following:"The security certificate presented by this website was not issued by a trusted certificate authority.""example.com uses an invalid security certificate. The certificate is not trusted because the issuer certificate is unknown.""example.com uses an invalid security certificate. The certificate is not trusted because it is self signed."I also encounter errors when attempting to upload SSL/TLS certificates to my Classic Load Balancer.


Question: How do I send feedback to AWS?

I have some suggestions to help improve an AWS document or service.

Answer: "I have some suggestions to help improve an AWS document or service. ResolutionTo submit feedback about an AWS console or specific AWS service, open the service's console, and then choose Feedback. Or, use AWS re:Post. To submit general feedback about AWS re:Post, choose Feedback on any re:Post page. To submit feedback on an AWS Knowledge Center article in re:Post, comment on the article.To submit general feedback about AWS Documentation, choose Feedback on the AWS Documentation homepage. To submit feedback on a specific page, choose Provide feedback. You can also choose either Yes or No for Did this page help you? After submitting your vote, you have the option to enter detailed feedback.Related informationHow do I get help with my AWS account and resources?Follow"


Question: I received GuardDuty UnauthorizedAccess brute force finding type alerts for my Amazon EC2 instance. What should I do?

Amazon GuardDuty detected alerts for the UnauthorizedAccess:EC2/RDPBruteForce or UnauthorizedAccess:EC2/SSHBruteForce finding types for my Amazon Elastic Compute Cloud (Amazon EC2) instance.

Answer: "Amazon GuardDuty detected alerts for the UnauthorizedAccess:EC2/RDPBruteForce or UnauthorizedAccess:EC2/SSHBruteForce finding types for my Amazon Elastic Compute Cloud (Amazon EC2) instance.Short descriptionBrute force attacks can indicate unauthorized access to your AWS resources. For more information, see Finding types.ResolutionFollow these instructions to check the GuardDuty finding type description, finding IDs, and detector IDs for more details about the brute force attack.Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Check the GuardDuty finding type descriptionFollow the instructions to view and analyze your GuardDuty findings.In the finding details pane, note the title of the finding type similar to the following:"198.51.100.0 is performing RDP brute force attacks against i-99999999. Brute force attacks are used to gain unauthorized access to your instance by guessing the RDP password."In this example, the description indicates which Amazon EC2 instance is impacted, the direction of the brute force attack, and the IP address.Check the GuardDuty finding IDs and detector IDs1.    Open the GuardDuty console.2.    In the navigation pane, choose Findings.3.    In Finding type, choose the UnauthorizedAccess finding type.4.    In the finding type details pane, choose the Finding ID.5.    In Findings JSON, note the GuardDuty finding and detector IDs.6.    Run this AWS CLI command:Note: Replace your-detector-id and your-findings-id with your GuardDuty detector and finding IDs.aws guardduty get-findings --detector-id your-detector-id --finding-ids your-findings-id --query 'Findings[].Service.Action.NetworkConnectionAction.ConnectionDirection'You receive an output similar to the following:[    "INBOUND"]7.    Run this AWS CLI command:aws guardduty get-findings --detector-id your-detector-id --finding-ids your-findings-id --query 'Findings[].Service.Action.NetworkConnectionAction.RemoteIpDetails.IpAddressV4'You receive an output similar to the following:[    "198.51.100.0"]In this example, the Amazon EC2 instance security group allows SSH/RDP traffic and is open to the public.To mitigate the issue, you can restrict SSH/RDP traffic for only a set of IP addresses authorized to access the Amazon EC2 instance.To restrict SSH traffic, see Adding a rule for inbound SSH traffic to a Linux instance.To restrict RDP traffic, see Adding a rule for inbound RDP traffic to a Windows instance.Related informationHow to use Amazon GuardDuty and AWS Web Application Firewall to automatically block suspicious hostsHow do I use GuardDuty to identify SSH brute force attacks on Linux instances?How do I set up a trusted IP address list for GuardDuty?Follow"


Question: Why are my Amazon ECS container logs not delivered to Amazon CloudWatch Logs?

I can't find my Amazon Elastic Container Service (Amazon ECS) task's container logs in Amazon CloudWatch Logs.

Answer: "I can't find my Amazon Elastic Container Service (Amazon ECS) task's container logs in Amazon CloudWatch Logs.Short descriptionYour Amazon ECS container logs aren't delivered to CloudWatch Logs due to one or more of the following reasons:The awslogs log driver is not configured correctly in your Amazon ECS task definitions.The AWS Identity and Access Management (IAM) role doesn't have the required permissions.There are issues with your networking configuration.The log level for the container is not configured correctly.ResolutionThe awslogs log driver is not configured properlyTo send the container log information to CloudWatch Logs, you must configure the containers in your tasks correctly. Be sure that you configured the logConfiguration parameter correctly when you define the parameters for our container. If your Amazon ECS task definition involves multiple containers, then you must configure this parameter for each container.Be sure that you added the awslogs log driver to the logConfiguration parameter in your task definition.For Amazon Elastic Compute Cloud (Amazon EC2) launch type, perform these additional checks:Be sure that your Amazon ECS container instance includes at least version 1.9.0 of the container agent. For information about checking your agent version and updating to the latest version, see Updating the Amazon ECS container agent.If you aren't using the Amazon ECS-optimized AMI for your container instances, then be sure that the awslogs log driver is specified on the container instance when you start the agent using the following environment variable. For more information, see Installing the Amazon ECS container agent.ECS_AVAILABLE_LOGGING_DRIVERS='["json-file","awslogs"]'The IAM role doesn't have the required permissionsBe sure the IAM role for your Amazon ECS container instance has logs:CreateLogStream and logs:PutLogEvents permissions.For Fargate launch type, use the Amazon ECS task execution IAM role with logs:CreateLogStream and logs:PutLogEvents permissions. For more information, see Amazon ECS task execution IAM role.For Amazon EC2 launch type, check the following:If you are not using Amazon ECS task execution IAM role in your task definition, be sure that logs:CreateLogStream and logs:PutLogEvents permissions are granted on the IAM role used by your container instance.If you're using the Amazon ECS task execution IAM role, be sure that the value of the container agent parameter ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE is set to true.Note: When you use the Amazon ECS-optimized AMI with version 1.16.0 -1 or later of the ecs-init package, the default value of this parameter is true. On Windows, the default value of this parameter false. For more information, see Amazon ECS container agent configuration.There are issues with your network configurationIf your Amazon Virtual Private Cloud (Amazon VPC) doesn't have an internet gateway, and your tasks use the awslogs log driver to send log information to CloudWatch Logs, then be sure that you created an interface Amazon VPC endpoint for CloudWatch Logs. For more information, see Using CloudWatch Logs with interface VPC endpoints.The log level for the container is not configured correctlyThe awslogs log driver simply passes the container logs that are the STDOUT and STDERR I/O streams from Docker to CloudWatch Logs. Therefore, verify that your application sends the logs to STDOUT and STDERR I/O streams. Be sure to set the correct log level for your application during the container build. Depending on your application, you might set the log level through an environment variable or a configuration file.Related informationMonitoring your container instancesHow do I troubleshoot missing container logs for Amazon ECS or Amazon EKS?Follow"


Question: Why is my Amazon RDS for MySQL DB instance using more storage than expected?

"My Amazon Relational Database Service (Amazon RDS) for MySQL DB instance is using more space than expected. Why is this happening, and how can I optimize disk storage?"

Answer: "My Amazon Relational Database Service (Amazon RDS) for MySQL DB instance is using more space than expected. Why is this happening, and how can I optimize disk storage?ResolutionYou can use the FreeStorageSpace Amazon CloudWatch metric to monitor the available storage space for an RDS DB instance. But FreeStorageSpace doesn't describe how the DB instance is consuming storage.Use the following strategies to reclaim storage space:Run OPTIMIZE TABLETables consume some space that isn't actively in use, but Amazon RDS allocates that space to the tables anyway. If innodb_file_per_table is turned on (it's turned on by default), then you can reclaim that space using OPTIMIZE TABLE. OPTIMIZE TABLE works for InnoDB, MyISAM, and ARCHIVE tables. While Amazon RDS accepts the OPTIMIZE TABLE command, it actually runs the ALTER TABLE...FORCE command instead. When this occurs, you receive a warning message similar to the following: "Table does not support optimize, doing recreate + analyze instead." For more information, see the MySQL documentation for OPTIMIZE TABLE.To check for fragmentation, run a query like this:SELECTtable_name,data_length,max_data_length,index_length,data_freeFROMinformation_schema.tables WHERE table_schema='schema_name';The data_free column highlights the amount of free space allocated to a table that isn't actively in use. You might be able to reclaim this space by using OPTIMIZE TABLE. OPTIMIZE TABLE works  if the table is created in a separate tablespace, according to the Amazon RDS default innodb_file_per_table configuration setting. For more information, see the MySQL documentation for File-per-table tablespaces.Reduce application table storageTo see how much storage is used by application tables on your DB instance, run a query like this:SELECT table_schema,SUM(data_length + index_length + data_free)/1024/1024 AS total_mb,SUM(data_length)/1024/1024 AS data_mb,SUM(index_length)/1024/1024 AS index_mb,SUM(data_free)/1024/1024 AS free_mb,COUNT(*) AS tables,CURDATE() AS today FROM information_schema.tablesGROUP BY table_schemaORDER BY 2 DESC;To locate the largest application table on your DB instance, run a query like this:SELECT table_schema,table_name,(data_length + index_length + data_free)/1024/1024 AS total_mb,(data_length)/1024/1024 AS data_mb,(index_length)/1024/1024 AS index_mb,(data_free)/1024/1024 AS free_mb,CURDATE() AS todayFROM information_schema.tablesORDER BY 3 DESC;Note: The total storage used by an individual database and table can't be calculated if the database includes tables with variable-length columns that are longer than 768 bytes. For example, this includes BLOB, TEXT, VARCHAR, or VARBINARY.Reduce binary log storageAdding a read replica causes the source instance's binary log to use additional storage. To find out how much storage the binary log on the source instance is using, check the BinLogDiskUsage CloudWatch metric. A large amount of growth might indicate that one or more Read Replicas are not synchronized.Reduce or turn off general log and slow query log storageWhen you turn off the general log and slow query log parameters, your instance starts to store these logs. It also stores the backups of these logs. To rotate these files and control disk usage, see mysql.rds_rotate_general_log and mysql.rds_rotate_slow_log.Note: Turn off the general and slow query logs when you're not actively using them to troubleshoot.Manage or reduce InnoDB system tablespace sizeThe system tablespace contains the InnoDB data dictionary and undo space, and it starts out at 10 MB. After space is allocated, the file will always be at least that size, though long-running transactions can consume more available storage.By default, Amazon RDS sets innodb_file_per_table to 1. This means that the data for each tablespace is stored in its own .ibd file. To recover space that is marked as reusable for related tables, use OPTIMIZE TABLE to resize per table tablespace files, or drop a table.If innodb_file_per_table is set to 0, all tables are also allocated to the system tablespace. Dropping tables or indexes, or deleting or truncating data from tables allocated in the system tablespace marks the space previously occupied as reusable. But, innodb_file_per_table doesn't free up any space to the file system.Because it's not possible to shrink the system tablespace in-place, export your current database's data. Then import the data into a new instance. To reduce downtime, configure your new MySQL instance as a replica of the source Amazon RDS instance. When the replica is synchronized with the source Amazon RDS instance, switch to the new instance.Note: Restoring from a snapshot or creating a Read Replica won't help you recover space from the system tablespace. This is because both methods use a snapshot of the source instance storage volume that contains the system tablespace.Related informationAmazon RDS DB instance running out of storageModifying an Amazon RDS DB instanceHow can I troubleshoot the error "MySQL HA_ERR_RECORD_FILE_FULL" when using Amazon RDS for MySQL?Follow"


Question: How do I troubleshoot namespaces in a terminated state in my Amazon EKS cluster?

"I tried to delete a namespace in my Amazon Elastic Kubernetes Service (Amazon EKS) cluster. However, the namespace is stuck in the "Terminating" status."

Answer: "I tried to delete a namespace in my Amazon Elastic Kubernetes Service (Amazon EKS) cluster. However, the namespace is stuck in the "Terminating" status.Short descriptionTo delete a namespace, Kubernetes must first delete all the resources in the namespace. Then, it must check registered API services for the status. A namespace gets stuck in Terminating status for the following reasons:The namespace contains resources that Kubernetes can't delete.An API service has a False status.Resolution1.    Save a JSON file like in the following example:kubectl get namespace TERMINATING_NAMESPACE -o json > tempfile.jsonNote: Replace TERMINATING_NAMESPACE with the name of your stuck namespace.2.    Remove the finalizers array block from the spec section of the JSON file:"spec": {        "finalizers": [            "kubernetes"        ]    }After you remove the finalizers array block, the spec section of the JSON file looks like this:"spec" : {    }3.    To apply the changes, run the following command:kubectl replace --raw "/api/v1/namespaces/TERMINATING_NAMESPACE/finalize" -f ./tempfile.jsonNote: Replace TERMINATING_NAMESPACE with the name of your stuck namespace.4.    Verify that the terminating namespace is removed:kubectl get namespacesRepeat these steps for any remaining namespaces that are stuck in the Terminating status.Related informationWhat is Amazon EKS?Namespaces (on the Kubernetes site)Follow"


Question: Why is my Amazon RDS DB instance in a failed state?

"My Amazon Relational Database Service (Amazon RDS) instance is in a failed state. What does this mean, and how do I fix it?"

Answer: "My Amazon Relational Database Service (Amazon RDS) instance is in a failed state. What does this mean, and how do I fix it?ResolutionThe failed state for RDS DB instances usually indicates that the hardware associated with your RDS DB instance failed, and that AWS wasn’t able to recover it.If you previously configured a backup solution for your RDS DB instance, restore it from one of the following backups:An automatic or manual backup.Point-in-time recovery (PITR).Note: PITR is available only for RDS DB instances with automatic backups. To retain the same endpoint name for your RDS DB instance, perform a PITR of your instance using a different name before deleting the instance that is in the failed state.After recovering your instance by using your preferred backup solution, delete the failed instance.After the failed DB instance is deleted, you can optionally rename the newly restored DB instance using the old DB instance's endpoint name, which means you don't need to change the configuration of your application to connect to the new DB instance.When you restore a DB instance from a snapshot, the I/O latency might be high to start. As you access the data on the instance, the latency decreases.To help mitigate failures, consider running your RDS DB instances in a Multi-AZ deployment, and keep the best practices for Amazon RDS in mind when designing your architecture.Related informationTroubleshooting for Amazon RDSFollow"