Question: How can I troubleshoot 410 GoneException errors with Amazon API Gateway WebSocket APIs?

I get a "410 GoneException" error when trying to post a message to my Amazon API Gateway WebSocket API similar to the following:..."errorMessage": "GoneException: 410"...How do I troubleshoot this?

Answer: "I get a "410 GoneException" error when trying to post a message to my Amazon API Gateway WebSocket API similar to the following:..."errorMessage": "GoneException: 410"...How do I troubleshoot this?Short descriptionYou might receive the "410 GoneException" error if:A message was posted to the WebSocket API before the connection was established.The connection has been terminated, or the connection doesn't exist anymore.The client disconnected and then tried to reconnect using the same connectionId.Note: For WebSocket APIs, the connectionId is required to send a callback response from the backend.ResolutionTo troubleshoot WebSocket API errors, turn on Amazon CloudWatch Logs.Note: For Log level, choose INFO to generate execution logs for all requests.Execution logs contain information that you can use to identify and troubleshoot most API errors. For example:The requests that your API receivesThe integration backend responses from your APIThe response provided by AWS Lambda authorizersThe requestId for AWS integration endpointsFor more information about each type of logging, see CloudWatch log formats for API Gateway.Before posting messages to WebSocket APIs, make sure that your connection is established by calling the getConnection operation using the AWS SDK ApiGatewayManagementApi.Avoid making the postToConnection request from the Lambda function integrated with your $connect route. For example, if you want to post a message after the client connection is established, do the following:Create a child Lambda function that's asynchronously invoked by the parent function.On initiation of a new connection, the $connect route is invoked and the parent function invokes the child function passing the connectionId.The Lambda child function receives the connectionId and calls the getConnection operation to verify that the connection is valid. Then, the message is sent using the postToConnection method.Related informationAbout WebSocket APIs in API GatewayHow can I pass a connectionId or custom token to VPC Link integration as a header for Amazon API Gateway WebSocket APIs?Follow"


Question: How do I delete or terminate Amazon RDS resources?

I want to reduce or eliminate the Amazon Relational Database Service (Amazon RDS) resources that are associated with my AWS account.

Answer: "I want to reduce or eliminate the Amazon Relational Database Service (Amazon RDS) resources that are associated with my AWS account.ResolutionThere are three main resources in Amazon RDS that incur billing charges: DB instances, DB clusters, and snapshots. To get a list of these resources that are active on your account, run the following commands in the AWS Command Line Interface (AWS CLI):aws rds describe-db-instancesaws rds describe-db-clustersaws rds describe-db-snapshotsaws rds describe-db-cluster-snapshotsTo stop all Amazon RDS related charges for an account, do the following in all AWS Regions:Delete all RDS DB instances.Delete Amazon Aurora clusters, if you have any.Delete all snapshots.For RDS DB instances and Aurora DB clusters, make sure that the delete protection feature is turned off. Otherwise, you can't delete these resources. For more information, see How can I troubleshoot an error that I received when deleting an Amazon Aurora cluster?Automated Backups can also incur charges. Unlike manual snapshots that must be deleted manually, automated backups persist only for the specified retention period. This retention period applies when you delete the DB instance. To delete your backups manually, see Deleting retained automated backups. Note that after backups are deleted, you can't recover their data.If you purchased reserved DB instances, then those instances are billed according to contract that you agreed to when you purchased the instance.If you delete the Amazon RDS resources mentioned earlier but you still see charges on your monthly bill, review your AWS Billing console dashboard.Related informationAmazon RDS pricingFollow"


Question: How can I hide or encrypt my database credentials in AWS Data Pipeline?

I want to hide or encrypt my database credentials in AWS Data Pipeline.

Answer: "I want to hide or encrypt my database credentials in AWS Data Pipeline.Short descriptionData pipeline encrypts the database credentials on transfer. However, when you access the pipeline during export, the pipeline JSON definition password is visible.You can mask your database credentials in the pipeline by doing one of the following:Create and use tags to restrict AWS Identity and Access Management (IAM) users from viewing your password.Create a custom Amazon Machine Image (AMI) by exporting the database credentials as environment variables.Use AWS Secrets Manager to store your database password in the pipeline.ResolutionTaggingYou can grant your AWS Identity Access Management (AWS IAM) users full or restricted access to pipelines based on tags.To restrict IAM users from viewing the password, create and attach an IAM policy that allows the use of read-only Data Pipeline actions, such as Describe, GetPipelineDefinition, and QueryObjects. For more information, see Example policies for AWS Data Pipeline.Custom AMIYou can secure your database credentials in the pipeline by doing the following:1.    Export your database user name and password as environment variables on your Amazon Elastic Compute Cloud (Amazon EC2) instance: Run the following command to export the database credentials as environment variables on your Amazon EC2 instance:sudo vi /etc/profileexport USERNAME=awsuserexport PASSWORD=2.    Create a custom AMI from the Amazon EC2 instance.3.    Pass this variable in a PostgreSQL connection method to the database instance as a ShellCommandActivity: Define the following ShellCommandActivity using the Data Pipeline Architect:psql "host=redshiftendpoint port=5439 dbname=Dbname user=$USERNAME password=$PASSWORD" -c "select 1;"For more information, see Define an activity using the AWS Data Pipeline Architect.AWS Secrets ManagerThe most secure way to encrypt your database credentials in the pipeline is to do the following:Use the AWS Secrets Manager to store your password.Install PostgreSQL to connect to Redshift database instance or SQL Workbench to connect to other database instances.You can use AWS Secrets Manager to store the password as an environment variable. You can then pass this variable in a PostgreSQL connection method to the database instance as a ShellCommandActivity. For more information, see Connect to your cluster by using the psql tool.1.    Create and store your Redshift database credentials in AWS Secrets Manager.2.    Attach the following policy to the DatapipelineDefaultResource role to grant List/Read access to the secret stored in AWS Secret Manager:{"Version": "2012-10-17",    "Statement": [        {            "Sid": "SecretPolicytoretrievesecretvalue",            "Effect": "Allow",            "Action": [                "secretsmanager:GetSecretValue",                "secretsmanager:DescribeSecret",                "secretsmanager:ListSecretVersionIds"            ],            "Resource": "arn:aws:secretsmanager:us-east-1:1111222233334444:secret:mysecret/secret-10aaPx"        }    ]}Replace the following in the above policy:us-east-1 with the Region where your secret is present1111222233334444 with your AWS account IDmysecret with the name of your virtual foldersecret-10aaPx with the name of your secret3.    Define the following ShellCommandActivity using the Data Pipeline Architect:sudo yum -y install aws-cli jq postgresql95-server.x86_64 postgresql95-contrib.x86_64 postgresql95.x86_64 postgresql95-libs.x86_64 && export USERNAME=`aws secretsmanager get-secret-value --secret-id mysecret/secret --region regionname | jq -r '.SecretString' | cut -f4 -d \"\\\"\"` && export AWS_REGION=region_name && export PASSWORD=`aws secretsmanager get-secret-value --secret-id mysecret/secret --region region_name | jq -r '.SecretString' | cut -f8 -d \"\\\"\"` && psql "host=redshiftendpointname port=5439 dbname=sampledbname user=$USERNAME password=$PASSWORD" -c "select 1";Related informationSecurity in AWS Data PipelineFollow"


Question: How do I troubleshoot issues with monitoring live conversations in Amazon Connect?

"I set up live monitoring for voice and/or chat in Amazon Connect, but it’s not working. How do I troubleshoot the issue?"

Answer: "I set up live monitoring for voice and/or chat in Amazon Connect, but it’s not working. How do I troubleshoot the issue?ResolutionTo resolve issues with monitoring live conversations in Amazon Connect, first review this article to identify the specific issue that you’re experiencing. Then, follow the troubleshooting steps listed for that issue.If you can’t select the icon to initiate live monitoring in the Amazon Connect Contact Control Panel (CCP)Confirm that the manager’s or agent trainee’s security profile includes the required permissions to monitor live conversations. For more information, see Assign permissions to monitor live conversations.If you receive a “Failed to monitor agent” error message when trying to initiate live monitoringConfirm that the following is true:Another call isn't being routed to the manager or trainee at the same time.The manager or trainee doesn't have any open contacts in their CCP.Your contact flow includes a Set recording and analytics behavior contact block.The Set recording and analytics behavior block is configured to allow Call recording for both Agent and Customer. For more information, see Properties in Contact block: Set recording an analytics behavior.Your contact flow is Published, not Saved as a draft only.Note: If you choose Save when creating a contact flow, Amazon Connect saves a draft of the flow. To activate the flow, you must choose Publish.The channel (Voice, Chat, or Task) that you're trying to monitor is activated in your routing profile. For more information, see Create a routing profile.The conversation that you're trying to monitor is still active.Note: To confirm if a conversation is still active, check your Real-time metrics report.No more than five people are trying to monitor the conversation at the same time.Note: Amazon Connect allows a maximum of five people to listen in on the same conversation at the same time. This limit can't be increased. To confirm how many people are trying to listen in on the same conversation, check your Real-time metrics report.If you receive a “softphone” error when trying to initiate live monitoringFollow the instructions in How do I troubleshoot Amazon Connect CCP softphone errors?If there are audio quality issuesFollow the instructions in How to determine whether a workstation is the source of problems.Follow"


Question: What happens during an EC2 scheduled maintenance event and what preventative actions can I take?

My Amazon Elastic Compute Cloud (Amazon EC2) instance has a maintenance event scheduled and I want to better understand how it affects me.

Answer: "My Amazon Elastic Compute Cloud (Amazon EC2) instance has a maintenance event scheduled and I want to better understand how it affects me.Short descriptionAmazon EC2 maintenance is often a live-update to minimize customer impact. Because a live-update isn't always possible, a scheduled maintenance event can sometimes occur.A scheduled maintenance event might require your instance to undergo one of the following:Stop or retireRebootLose network connectivity or powerThe instance actions that your maintenance event requires depends on the instance configuration and the type of maintenance that is being performed.ResolutionDepending on your situation, you can perform one of the following actions before the specified maintenance window:Reschedule the maintenance eventIf your instance must undergo a reboot, then reschedule the event so that your instance reboots at a date and time that suits you. For more information, see Reschedule a scheduled event.Stop and start your instanceIf your instance has an Amazon Elastic Block Store (Amazon EBS) volume as its root device, then complete maintenance by stopping and starting your instance. A stop and start migrates your instance to a new underlying host.The stop and start changes the status of the scheduled maintenance event to Completed. The actions, when completed, can take up to an hour to be reflected in the status change. For more information, see View scheduled events.Important: Stopping and restarting the instance changes the public IP address of your instance. It's a best practice to use an Elastic IP address instead of a public IP address when routing external traffic to your instance.Automatically recover an instanceYou can create an Amazon CloudWatch alarm that monitors an Amazon EC2 instance. You can configure the alarm to automatically recover the instance if the instance is impaired due to an underlying hardware failure. For more information, see Recover your instance.Create an instance store-backed AMIIf the instance is based on an instance store or an ephemeral root volume, then you might face issues if your instance is scheduled to terminate rather than reboot. You must create an instance store-backed AMI, then migrate your data and relaunch the instance.Let the maintenance event continue as scheduledIf the maintenance schedule is network connectivity, power maintenance, or a required reboot, then you can let the maintenance event continue as scheduled without intervention.Related informationAmazon EC2 maintenance help pageFollow"


Question: When should I restart and resume my AWS DMS task that is in a Stopped or Failed status?

I have an AWS Database Migration Service (AWS DMS) task that is in the Stopped or Failed state. When should I resume or restart my AWS DMS task to continue replication?

Answer: "I have an AWS Database Migration Service (AWS DMS) task that is in the Stopped or Failed state. When should I resume or restart my AWS DMS task to continue replication?Short descriptionWhen your AWS DMS task is in a Stopped or Failed status, you have two options to allow you to continue replication:Resume - When you resume a task, AWS DMS continues replication from the last point before the task stopped or failed.Restart - When you restart a task, AWS DMS begins replication from the start, and uses the table preparation mode that you chose when you created the task. For example, table preparation modes include Drop table on target, Truncate, and Do nothing. For more information, see Full-load task settings.The behavior of the resume and restart actions varies based on which of the three AWS DMS migration methods that you're using. These methods include full load, full load and change data capture (CDC), and CDC only migration. For more information, see Creating a task.ResolutionMigrate existing data (full load)For full load tasks, if you start the task manually during task creation, then use the restart action in the AWS DMS console to start replication. This reloads all of the tables in the migration.You can also use the reload table data option to reload specific tables that failed during migration. This means that tables that are already loaded don't need to load again, and any tables that didn't finish loading are loaded again.If you use the resume option while migrating multiple tables, then tables that didn't fully complete starts over. Tables that completed migration aren't affected. If you're migrating a single table, it's a best practice to restart the task rather than using the resume option.Migrate existing data and replication ongoing changes (full load and CDC)When you use a full load and CDC task, AWS DMS migrates table data, and then applies data changes that occur on the source. If you restart the task, this loads all the tables again, and starts capturing source changes from the restart time. If your task is configured with the Do nothing preparation mode, then manually empty target tables before restarting the task.If you resume the task, then only changes that were captured after the last stop point are applied to the database. If the migration task stops during the CDC phase, then AWS DMS maintains the checkpoint information for future use. You can view the task checkpoint in the Overview details tab of the AWS DMS console.Replicate data changes only (CDC only)If you're using a CDC only task, you can start capturing source data changes from either the current time or from the CDC start point. If you restart the task when a CDC start point is defined, then the reload operation loads all changes from this point in time. If you restart a task without a CDC start point, then CDC changes from when the task was stopped until it was restarted are lost.The resume operation continues replicating changes from the last stop point, regardless of the CDC start point configuration. If you restart a task with a Truncate target table preparation mode, then AWS DMS leaves existing target tables and their metadata in place. But, it deletes all existing data from these tables before it restarts migration.Common scenarios for resuming or restarting an AWS DMS taskHere are some common examples of when you can use the resume or restart operations on your AWS DMS task.Restart the AWS DMS task:If the source database management system (DBMS) doesn't contain the recovery log files or transaction log files to resume the CDC process, restart the task. Restarting the task loads all table data and continues capturing changes.If an AWS DMS task is in an Error status, this means that one or more of the tables in the task couldn't be migrated. In an Error status, the task continues to load other tables from the selection rule, but a task with a Failed status stops with fatal errors. After you resolve the errors, reload the tables, or restart the task to resolve the error status. For more information, see Why is my AWS DMS task in an error status?If a full load and CDC or a CDC only task is stopped, then the data changes can spill over from memory to disk. Depending on the volume of change data swapped into disk, a task resume might take a long time to continue replication. This is because AWS DMS takes a longer time to read these changes from disk. So, if it's feasible, restart the task to avoid this wait time.If you change between using Oracle LogMiner and AWS DMS Binary Reader, then make sure to restart the CDC task.Note: After modifying the CDC method, if you restart a CDC-only task that is configured with CDC recovery checkpoint, you might see an error similar to:"[SOURCE_CAPTURE ]D: Invalid context provided for the Binary Reader based CDC. Restart task is required."To resolve this error, start the task based on timestamp in the CDC start point.Resume the AWS DMS task:If you move a task to a new replication instance, resume the task to continue replicating changes from the point it was last stopped.If you want to upgrade your source or target databases, then stop any AWS DMS tasks that are running on these databases. Resume your tasks after your upgrades are complete.If you plan to upgrade your source or target databases, then stop any AWS DMS tasks that are running on these databases. Resume the tasks after your upgrades are complete. But, to perform a PostgreSQL engine version upgrade, you can't have any replication slots on the instance. So, drop any replication slots before you upgrade your engine, and then restart the task to recreate the replication slot.Related informationSources for data migrationWorking with an AWS DMS replication instanceRebooting a replication instanceFollow"


Question: How do I troubleshoot SigV4 Signature Mismatch errors with Amazon S3 presigned URLs?

"I'm using an Amazon Simple Storage Service (Amazon S3) presigned URL to make requests to my Amazon S3 bucket, and I'm getting the SignatureDoesNotMatch error."

Answer: "I'm using an Amazon Simple Storage Service (Amazon S3) presigned URL to make requests to my Amazon S3 bucket, and I'm getting the SignatureDoesNotMatch error.Short descriptionYou get this error when the signature value that's calculated from your HTTP request to AWS doesn't match the value that Amazon S3 calculates. For more information, see List of error codes.When you generate a preresigned URL, the client calculates a unique signature for the request to authenticate the request. Then, Amazon S3 calculates a signature based on the parameters that are sent in the HTTP request and compares both signatures. If the signatures don't match, then you receive the SignatureDoesNotMatch error.To troubleshoot this error, do the following:Validate the HTTP method: Confirm that the HTTP requests that you made to S3 for the GET, PUT, and DELETE requests match the HTTP method that the request was generated for.Check the secret access key: Make sure that you use the correct secret access key to generate the presigned URL. If the secret access key was modified or is incorrect, then use the correct access key to regenerate the presigned URL.Verify the bucket name and key name in the URL: Make sure that the bucket name and object key name are correct and match those of the object that you're trying to access.Validate the headers that are used in HTTP request: Verify that the headers were calculated in the signature. Make sure that the HTTP headers that were used to generate the signature match those that were sent to S3 in the HTTP request.Be sure that the AWS Region is correct: Validate that the signature is generated for the Region where the bucket currently exists.ResolutionIncorrect HTTP methodWhen you generate a presigned URL, you assign an HTTP action to the URL. You get this error when the action that the client sent in the HTTP request doesn't match the HTTP action that's used in the URL. For example, if the request is signed for GET, but the HTTP action that's used in the request is PUT, then you get the error. For the URL to work, verify that you generate and use the URL for the intended HTTP method.Incorrect secret access keyYou get this error when the access key or secret access key that you use to generate the presigned URL is in one of the following states:IncorrectNot validTurned offCheck that you didn't add any mismatched characters or incorrect spacing to the access key.Be sure that the access key and secret key that you provided during signature generation match the ones that were provided by AWS.Incorrect key name or bucket nameYou get this error when you use a presigned URL that you generated for a bucket or key to access a different bucket or key.Verify that the bucket name and object name are correct and match the ones that were included in the signature generation of the URL. Note that a mismatched case also causes the signatures to be mismatched.Unsigned for headers or incorrect header valuesBe sure that you didn't pass the headers that weren't signed for in the signature generation of your presigned URL. If you intend to send headers with your presigned S3 URL, then they must be accounted for within the generation of the URL's signature.Also, make sure that the value of the header that you passed matches the one that was generated during signature calculation. Any mismatched header values, including case mismatches, result in the SignatureDoesNotMatch error.Signature Region and bucket Region mismatchBefore sending the presigned URL to S3, validate that the Region where the URL was generated matches the Region where the bucket currently exists.Use the GetBucketLocation API request to check the Region of an S3 bucket. You can also run the following AWS Command Line Interface (AWS CLI) command:$ aws s3api get-bucket-location --bucket example-bucketNote: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.The output looks similar to the following one:{      "LocationConstraint": "us-west-2"  }Related informationUsing presigned URLsWhen I try to access an S3 object, I get the error "Request has expired." Why?Follow"


Question: How do I view full email headers on popular email clients to use for troubleshooting?

I want to see the full email headers of emails exchanged between Amazon Simple Email Service (Amazon SES) and my receiving mail server. How do I view full email headers?

Answer: "I want to see the full email headers of emails exchanged between Amazon Simple Email Service (Amazon SES) and my receiving mail server. How do I view full email headers?Short descriptionEmails exchanged between Amazon SES and a receiving mail server contain email headers. These headers provide detailed information about an email, including:Sender and recipient detailsAuthentication resultsThe network path taken by the email between the sender and receiverHeader fields that are most commonly shown in an email client include:From: Sender’s name and emailTo: Recipient’s name and emailDate: Sent date/time of the emailSubject: Subject line that was entered by the senderHowever, sometimes details found only in the full email headers are required in order to perform advanced auditing or troubleshooting tasks. For example:Message ID valueSender/recipient detailsNetwork pathAuthentication results (SPF/DKIM/DMARC)Full email headers look similar to the following example:Delivered-To: hello@examplerecipient.comReceived: by 2002:a17:907:104f:0:0:0:0 with SMTP id oy15csp1032604ejb;        Fri, 30 Apr 2021 20:39:47 -0700 (PDT)X-Google-Smtp-Source: ABdhPJzh1mAfg8ADsMqkmtRwbDilz3ii9Lsgjk2P2Cn0t3F1dn93MTB1gEQY0WeqB8cUwgpkmj4bX-Received: by 2002:a37:744:: with SMTP id 65mr8623779qkh.393.1619840386902;        Fri, 30 Apr 2021 20:39:46 -0700 (PDT)ARC-Seal: i=1; a=rsa-sha256; t=1619840386; cv=none;        d=examplerecipientdomain.com; s=arc-20160816;        b=PMSOJmkrw1P59egAibf1euOjQM0JQ2n8pk00+uvlk7CV9zuCZEFglG3w+LVMlw0wAZ         XtQ6T5bO4YyeqrDCqFcleBnnvfTuFRaup4+J62CWiVI7R/k+4q0PPdik8TIdDMxATc/g         qQKgr5HZGhu1zm8n5oY1B0yg4JpjOGVIxOjc9dB/15ix2tQrPVxoHQPMHYVofPtY5BeN         G0LpjCXCrmqwCCsGS2mgJ0eEg9vFV9UUIdtTBr+6CPV+V45wIgOYkGMLuHhzalXKZXbh         w7fm+nne7x9LclK9xvtHW1CajBZeD+bNMD3exqVYNbf2lJ36V1mDJ0uIvq1x5u2i14pL         7KAQ==ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=examplerecipientdomain.com; s=arc-20160816;        h=feedback-id:date:message-id:content-transfer-encoding:mime-version         :subject:to:from:dkim-signature:dkim-signature;        bh=VOs4y6gu8JLAPLN/Bq4M9b94CryYOzlLzn92GcncOkw=;        b=ztAEAfTEQ6yLnV/Q4gUYIOUFAhy8zW2LRziT1YosBxayEsTQhdDR3nThK1P75aui7w         RPg73Of+uwqjAoQXf7ZlHWKfrlCre+8CresaKAzFw8FLpI1xac4dgcoFwH9whJ4jxs1c         N9AWmeRjEhspATIUxYkcJ8+OsUG/2TDPQu3k93cRO8NhCCD2sz6WiN/Cl64Rk6CBwsse         ipZl6qZRo2dNravnsN2jXxUdABSUrTsKKSI/4Q8Gw2preRKRrpWBqyPFYswizDewQXuI         f8twJZVfSdHoA8XwrKcVAfUxcpCryGYaeo9sHlheXbBf3I2kNWVHnvM8fyPs4VJAbudH         ayoQ==ARC-Authentication-Results: i=1; mx.examplerecipient.com;       dkim=pass header.i=@examplesender.com header.s=qk6bvwgtoedeefsxxzx7zyh756x7pud2 header.b=eRCWqyY+;       dkim=pass header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=VUA6FdBW;       spf=neutral (examplerecipientdomain.com: 54.240.48.95 is neither permitted nor denied by best guess record for domain of 010001792602c3fc-SaMpLe-a4fc-4135-bc5a-b9fcaba8d509-000000@custommailfromdomain.examplesender.com) smtp.mailfrom=010001792602c3fc-SaMpLe-a4fc-4135-bc5a-b9fcaba8d509-000000@custommailfromdomain.examplesender.comReturn-Path: <010001792602c3fc-SaMpLe-a4fc-4135-bc5a-b9fcaba8d509-000000@custommailfromdomain.examplesender.com>Received: from a48-95.smtp-out.amazonses.com (a48-95.smtp-out.amazonses.com. [54.240.48.95])        by mx.examplerecipient.com with ESMTPS id c21si2902247qkj.313.2021.04.30.20.39.46        for <hello@examplerecipient.com>        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128);        Fri, 30 Apr 2021 20:39:46 -0700 (PDT)Received-SPF: neutral (examplerecipientdomain.com: 54.240.48.95 is neither permitted nor denied by best guess record for domain of 010001792602c3fc-SaMpLe-a4fc-4135-bc5a-b9fcaba8d509-000000@custommailfromdomain.examplesender.com) client-ip=54.240.48.95;Authentication-Results: mx.examplerecipient.com;       dkim=pass header.i=@examplesender.com header.s=qk6bvwgtoedeefsxxzx7zyh756x7pud2 header.b=eRCWqyY+;       dkim=pass header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=VUA6FdBW;       spf=neutral (examplerecipientdomain.com: 54.240.48.95 is neither permitted nor denied by best guess record for domain of 010001792602c3fc-SaMpLe-a4fc-4135-bc5a-b9fcaba8d509-000000@custommailfromdomain.examplesender.com) smtp.mailfrom=010001792602c3fc-SaMpLe-a4fc-4135-bc5a-b9fcaba8d509-000000@custommailfromdomain.examplesender.comDKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qk6bvwgtoedeefsxxzx7zyh756x7pud2; d=examplesender.com; t=1619840386; h=From:To:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID:Date; bh=VOs4y6gu8JLAPLN/Bq4M9b94CryYOzlLzn92GcncOkw=; b=eRCWqyY+fg2dyuk0+vz+RATrOd6MxysLa6SS13u014Y5liKanL+SoWNhCeWBU1z2 vcVUnS7VNSF04+Cn1f0KssOyec9y/dlFN50upm59kyCc5bYQrDnSuRdDjDYMNd6tHfP OXbsl9SF2qR83+6zNOOY8LzbL7oCe8cXPoK4bL9s=DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1619840386; h=From:To:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID:Date:Feedback-ID; bh=VOs4y6gu8JLAPLN/Bq4M9b94CryYOzlLzn92GcncOkw=; b=VUA6FdBWX2t78RPPo5T+VBhLY7ykrrk6Cvf3ye9OLHiKFtA+ZhZ7wUKoC85MaO4L +Ng+q7StDqYv2L7FKvoL2aUUMFAZcYOgzoXneO8enu8/5nRecowZdRfe0gMM3f5aL2R AJQYwmwHbCj7hNmHN/0tKhVRynIEdEnkKoe2Qook=From: hello@examplesender.comTo: hello@examplerecipient.comSubject: SESMIME-Version: 1.0Content-Type: text/plain; charset=UTF-8Content-Transfer-Encoding: 7bitMessage-ID: <010001792602c3fc-SaMpLe-a4fc-4135-bc5a-b9fcaba8d509-000000@email.amazonses.com>Date: Sat, 1 May 2021 03:39:46 +0000Feedback-ID: 1.us-east-1.27emrSSn5f4pRLShWT3CyWg+hDbDkwgJZdVmQ6/qxx8=:AmazonSESX-SES-Outgoing: 2021.05.01-54.240.48.95Hi there, this message is sent using SES.ResolutionTo view the full email headers of an email, follow the steps for the email client that you use.Microsoft Outlook 2016, 2013, or 2010Double-click the email that you want to view headers for so that it opens in a full window.Choose File, and then choose Properties.View the message header in the Internet headers field at the bottom of the Properties window.Apple MailOpen the email that you want to view headers for.From the menu, choose View, Message, Raw Source.The full message appears with email headers in a pop-up window.Mozilla ThunderbirdChoose the email that you want to view headers for from the reading pane to open it in a new tab or window.From the menu, choose View, Message Source.The full message appears with email headers in a pop-up window.GmailOpen the email that you want to view headers for.Choose the More menu, and then choose Show original.A new tab opens which includes the full message with email headers.Hotmail (Outlook.com)From your outlook.com inbox, without opening the email, open the context (right-click) menu for the email, and then choose View, View message source.The full message appears with email headers in a pop-up window.Yahoo! MailFrom your yahoo.com inbox, without opening the email, open the context (right-click) menu for the email, and then choose View raw message.A new tab opens which includes the full message with email headers.Follow"


Question: How can I monitor my Lightsail resource usage from the command line?

My website or application hosted in Amazon Lightsail is down or running very slowly. How can I monitor my Lightsail resource usage from the command line so that I can troubleshoot these issues?

Answer: "My website or application hosted in Amazon Lightsail is down or running very slowly. How can I monitor my Lightsail resource usage from the command line so that I can troubleshoot these issues?Short descriptionThere are several factors, including high resource utilization, that affect services running on Lightsail instances. Using common tools you can monitor resources such as CPU, memory, disk I/O, and network usage on Lightsail instances in real time from the command line.Analyzing and monitoring Lightsail instance resource usage from the command line provides real-time system insight. This also allows for a granular view of which processes are using the most CPU, memory, disk I/O, or network.The following are resources you can use to analyze and monitor your Lightsail instance's resources from the command line:The atop toolThe htop toolThe iostat, vmstat, and mpstat commandsResolutionThe atop toolThe atop tool reports the activity of all processes, even if those processes have finished during the specified interval. This tool logs all relevant system information continuously. Because of this continuous logging, if issues reoccur, you have historical data to analyze. The atop tool is included in official repositories for most Linux distributions. Install the atop tool by running the following commands:Red Hat-based systems$ sudo yum install atopDebian-based systems$ sudo apt install atopThe atop tool logs all activity in 600-second intervals by default. To change the atop tool's configuration so that activities are logged in 60-second intervals, run the following command:Red Hat-based systems$sudo sed 's/600/60/' /etc/atop/atop.daily -iDebian-based systems$ sudo sed 's/600/60/' /etc/default/atop -iAfter running the preceding command, atop logs all activities within 60-second internals. The data is stored in log files in /var/log/atop. These files are named in the following format " atop_ccyymmdd". For example, "atop_20210902" is the recording for September 2, 2021.To access the log file, run the command atop -r atologfilepath. The command and log file are shown in the following example:atop -r /var/log/atop/atop_20210902 ATOP - ip-172-20-139-91                2021/09/02  17:03:44                ----------------                 3h33m7s elapsedPRC |  sys    6.51s  |  user   7.85s  |  #proc    103  |  #tslpi    81 |  #tslpu     0  |  #zombie    0  |  #exit      0  |CPU |  sys     0%  |  user      3%  |  irq       0%  |  idle    197% |  wait      0%  |  ipc notavail  |  curscal   ?%  |cpu |  sys     0%  |  user      1%  |  irq       0%  |  idle     98% |  cpu000 w  0%  |  ipc notavail  |  curscal   ?%  |cpu |  sys     0%  |  user      1%  |  irq       0%  |  idle     98% |  cpu001 w  0%  |  ipc notavail  |  curscal   ?%  |In the preceding output example, the first recorded snapshot was at 2021/09/02 17:03:44. To move forward to the next snapshot, press the ‘ t’ key on the keyboard. To move back to the previous snapshot, press the "T" (uppercase "t") key.The following are commonly used options that you can launch atop with:Sort by memory$ atop -mSort by disk details$ atop -dSort by network details$ atop -nThe htop toolThe htop tool is an interactive and real-time process monitoring application for Linux that shows your usage per CPU or core. This tool also provides a meaningful text graph of your memory and swap usage.Run the following commands to install htop:Red Hat-based systems$ sudo yum install htopDebian and Ubuntu-based systems$ sudo apt install htopUse the following command to run htop on your system:$ htopThe htop tool lists all running processes on the system and includes information about how much CPU and memory each process is using. The command used to start the process is also included.The following are definitions for each column in the output:PID: A process ID number for the process.BUSER: The process owner.PR: The process’s priority. The lower the number, the higher the priority.NI: The nice value of the process, which affects its priority.VIRT: How much virtual memory the process is using.RES: How much physical RAM the process is using, measured in kilobytes.SHR: How much shared memory the process is using.S: The current status of the process (zombied, sleeping, running, uninterruptedly sleeping, or traced).CPU: The percentage of the processor time used by the process.MEM: The percentage of physical RAM used by the process.TIME+: How much processor time the process has used.COMMAND: The name of the command that started the process.The iostat, vmstat, and mpstat commandsThe iostat, vmstat, and mpstat commands are part of the sysstat bundle. Use the following commands to install this set of tools in your Lightsail instance:Red Hat-based systems, such as Amazon Linux, CentOS, or RHEL$ sudo yum install sysstatDebian and Ubuntu$ sudo apt-get install sysstatUse the following commands to start and enable sysstat for continuous data collection:$ sudo systemctl start sysstat$ sudo systemctl enable sysstat'iostat' commandThe iostat command collects and displays system input and output storage device statistics. This tool is often used to trace storage device performance issues including devices and disks. The following are common iostat options:iostat: Get report and statistics.iostat -x: Show more detailed statistics information.iostat -c: Show only the CPU statistic.iostat -d: Display only the device report.iostat -xd: Show extended I/O statistics for the device only.The following is example iostat command output:$ iostatLinux 4.14.246-187.474.amzn2.x86_64     10/22/2021     _x86_64_    (1 CPU)avg-cpu:  %user   %nice %system %iowait  %steal   %idle           5.34    0.00    1.36    1.06    1.67   90.58Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtnxvda             41.64      1496.90       890.16     502148     298612'vmstat' commandThe vmstat (virtual memory statistics) command collects and displays information about system memory, processes, interrupts, paging, and block I/O. The vmstat command allows you to specify a sampling interval that permits observing system activity in near-real time. The following is an example of the vmstat command and output:$ vmstat 2 5procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----r b swpd free buff cache si so bi bo in cs us sy id wa st3 0 0 1409320 2088 485404 0 0 568 724 232 393 4 2 93 1 10 0 0 1409332 2088 485404 0 0 0 67 146 261 0 0 100 0 00 0 0 1408916 2088 485404 0 0 0 0 157 276 0 0 100 0 00 0 0 1408980 2088 485408 0 0 0 0 161 259 0 0 100 0 00 0 0 1408984 2088 485408 0 0 0 0 105 190 0 0 100 0 0The following are commonly used options you can use with the vmstat command:The -a option displays the active and inactive memory of the system:$ vmstat -aprocs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----r b swpd free inact active si so bi bo in cs us sy id wa st2 0 0 1402016 311232 218864 0 0 60 77 74 129 0 0 99 0 0The -d option reports disk statistics:$ vmstat -ddisk- ------------reads------------ ------------writes----------- -----IO------total merged sectors ms total merged sectors ms cur secnvme0n1 7728 25 433681 4544 3133 93 555950 4196 0 1'mpstat' commandThe mpstat command collects and displays performance statistics for all logical processors in the system. Running the mpstat command with no options generates a single report that contains the performance statistics for all logical processors since boot time.$ mpstat Linux 4.18.0-305.el8.x86_64 (ip-xxx-xx-x-xx.ec2.internal)     10/27/2021     _x86_64_    (2 CPU)04:23:26 PM  CPU    %usr   %nice    %sys  %iowait    %irq   %soft  %steal  %guest  %gnice   %idle04:23:26 PM  all    8.42    0.13    2.43    1.49     0.18    0.05    0.22    0.00    0.00   87.08The first line in the preceding example is a set of column labels. The second line is the value for each column:%usr: % CPU usage at the user level.%nice: % CPU usage for user processes labeled "nice".%sys: % CPU usage at the system (Linux kernel) level.%iowait: % CPU usage idling waiting on a disk read/write.%irq: % CPU usage handling hardware interrupts.%soft: % CPU usage handing software interrupts.%steal: % CPU usage being forced to wait for a hypervisor handling other virtual processors.%guest: % CPU usage spent running a virtual processor.idle: %CPU usage on idle time (no processes, and not waiting on a disk read/write).Using the preceding command line tools, you can identify the processes with high resource usage. And, you can use this information to choose instances with sufficient resources for your workload.Follow"


Question: Why can't I join an organization in AWS Organizations?

"I can't join a particular organization in AWS Organizations. Why is that, and what can I do to fix it?"

Answer: "I can't join a particular organization in AWS Organizations. Why is that, and what can I do to fix it?ResolutionHere are some reasons why you might not be able to join an organization:You're already a member account of a different organization. Remove your account from its current organization, and then try again.You're already the management account of a different organization. If you try to link your account to a management account, but you've already created an organization with your account as the management account, then you receive an error. To join an organization as a member account, you must convert your account from a management account to a standalone account. For more information, see remove all member accounts from your organization and delete the organization.Your account isn't verified and activated. Be sure that you fully complete the account sign-up and activation process. Some services can take up to 48 hours to fully activate after the account sign-up process is completed. For more information, see How do I create and activate a new AWS account?You didn't accept the invitation to the organization, or the organization didn't send you one. Be sure to accept the invitation in the Organizations console. You might need to wait a few minutes before the Organizations console or Billing and Cost Management console reflect that you're a member of the organization. If you didn't receive an invitation, ask the administrator of the management account to send you a new invitation.Your account has a different seller of record or AWS partition than the management account of the organization that you're trying to join. See Why do I receive the error "You can only join an organization whose Seller of Record is same as your account" when I'm trying to join an organization?Related informationWhat is AWS Organizations?Managing the AWS accounts in your organizationFollow"


Question: How do I use Amazon ECS service discovery with CloudFormation?

I want to use Amazon Elastic Container Service (Amazon ECS) service discovery with AWS CloudFormation.

Answer: "I want to use Amazon Elastic Container Service (Amazon ECS) service discovery with AWS CloudFormation.Short descriptionThe Amazon ECS console allows you to create service discovery resources during service creation. However, the AWS::ECS::Service resource type for CloudFormation doesn't allow you to create service discovery resources during service creation.Note: The Amazon ECS console uses the integration APIs CreatePrivateDnsNamespace, CreateService (AWS Cloud Map), and CreateService (Amazon ECS). In CloudFormation, you can use the resource types AWS::ServiceDiscovery::PrivateDnsNamespace, AWS::ServiceDiscovery::Service, and AWS::ECS::Service to implement an integration in a single CloudFormation template.Resolution1.    In the Resources section of your CloudFormation template, create a private service discovery namespace, such as example.com. Create the namespace in an existing Amazon Virtual Private Cloud (Amazon VPC). See the following JSON and YAML examples.JSON:{  "PrivateNamespace": {    "Type": "AWS::ServiceDiscovery::PrivateDnsNamespace",    "Properties": {      "Name": "example.com",      "Vpc": "vpc-xxxxxxx"    }  }}YAML:PrivateNamespace:    Type: AWS::ServiceDiscovery::PrivateDnsNamespace    Properties:        Name: example.com        Vpc: vpc-xxxxxxxNote: Replace example.com with a name for your namespace. Replace vpc-xxxxxxx with the ID of a VPC from your account.2.    Create a service discovery service using the namespace ID of the namespace that you created in Step 1.You can get the value of the namespace ID using the Ref intrinsic function in your CloudFormation template. See the following JSON and YAML examples.JSON:{  "DiscoveryService": {    "Type": "AWS::ServiceDiscovery::Service",    "Properties": {      "Description": "Discovery Service for the Demo Application",      "DnsConfig": {        "RoutingPolicy": "MULTIVALUE",        "DnsRecords": [          {            "TTL": 60,            "Type": "A"          },          {            "TTL": 60,            "Type": "SRV"          }        ]      },      "HealthCheckCustomConfig": {        "FailureThreshold": 1      },      "Name": "example-service-name",      "NamespaceId": {        "Ref": "PrivateNamespace"      }    }  }}YAML:DiscoveryService:  Type: AWS::ServiceDiscovery::Service  Properties:     Description: Discovery Service for the Demo Application    DnsConfig:      RoutingPolicy: MULTIVALUE      DnsRecords:        - TTL: 60          Type: A        - TTL: 60          Type: SRV    HealthCheckCustomConfig:       FailureThreshold: 1    Name: example-service-name    NamespaceId: !Ref PrivateNamespaceNote: Replace example-service-name with the name of your service.The DnsConfig property specifies the DNS records that are automatically created in Amazon Route 53. These DNS records are created when your Amazon ECS task is registered with the service discovery service. DNS records of type A and SRV are supported. For more information, see Configuring your service to use Service Discovery.Tip: It's a best practice to use the HealthCheckCustomConfig property to allow Amazon ECS to report task health to service discovery. Amazon ECS uses information from container checks, health checks, and task state to update the health of service discovery with AWS Cloud Map.3.    Create an Amazon ECS service that integrates with the service discovery namespace and service that you created in the previous steps. Use the ServiceRegistries property in the AWS:ECS::Service resource to specify your integration. See the following JSON and YAML examples.JSON:{  "EcsService": {    "Type": "AWS::ECS::Service",    "Properties": {      "Cluster": "awsExampleCluster",      "DesiredCount": 4,      "LaunchType": "FARGATE",      "NetworkConfiguration": {        "AwsvpcConfiguration": {          "SecurityGroups": [            "sg-xxxxxxx"          ],          "Subnets": [            "subnet-xxxxxxx"          ]        }      },      "ServiceName": "awsExampleService",      "TaskDefinition": "awsExampleTaskDefinition",      "ServiceRegistries": [        {          "RegistryArn": {            "Fn::GetAtt": [              "DiscoveryService",              "Arn"            ]          },          "Port": 80        }      ]    }  }}YAML:EcsService:  Type: AWS::ECS::Service  Properties:    Cluster: awsExampleCluster    DesiredCount: 4    LaunchType: FARGATE    NetworkConfiguration:      AwsvpcConfiguration:        SecurityGroups:          - sg-xxxxxxx        Subnets:          - subnet-xxxxxxx    ServiceName: awsExampleService    TaskDefinition: awsExampleTaskDefinition    ServiceRegistries:      - RegistryArn: !GetAtt DiscoveryService.Arn        Port: 80Note: Replace awsExampleCluster with your cluster. Replace awsExampleService with your service. Replace awsExampleTaskDefinition with your task definition. Replace sg-xxxxxxx with your security group ID. Replace subnet-xxxxxxx with your subnet ID.For RegistryArn, use the Amazon Resource Name (ARN) of the service discovery service that you want to integrate with your Amazon ECS service. Use the intrinsic function Fn::GetAtt to get this value from your CloudFormation template.If you use the awsvpc mode, set the value to either Port or a combination of ContainerPort and ContainerName (as specified in the task definition). If you use host or bridge mode, set the value to ContainerPort or ContainerName (as specified in the task definition).4.    Open the CloudFormation console, and then choose Create stack to deploy the resources based on your updated template.5.    To verify that your tasks can be resolved from your VPC, run the following commands:$ dig awsExampleService.awsExampleNamespace. +short$ dig srv awsExampleService.awsExampleNamespace. +short$ curl awsExampleService.awsExampleNamespace. -INote: The value of awsExampleService is the name for your AWS::ECS::Service resource. The value of awsExampleNamespace is the name for your AWS::ServiceDiscovery::PrivateDnsNamespace resource.The commands return output similar to the following:For $ dig awsExampleService.awsExampleNamespace. +short:172.31.182.0172.31.160.124172.31.137.81172.31.149.244For $ dig srv awsExampleService.awsExampleNamespace. +short:1 1 80 ffe95d27ea8d4f7aba0dfed87297fc5a.awsExampleService.awsExampleNamespace.1 1 80 44a17fa781974a93bb563bc1826a8697.awsExampleService.awsExampleNamespace.1 1 80 d640ecb3d283421bb2d1318caf4b0d66.awsExampleService.awsExampleNamespace.1 1 80 65aff6fff33144b2ad79d283ab52cfe9.awsExampleService.awsExampleNamespace.For $ curl awsExampleService.awsExampleNamespace. -I:HTTP/1.1 200 OKServer: nginx/1.15.12Date: Wed, 15 May 2019 02:25:19 GMTContent-Type: text/htmlContent-Length: 612Last-Modified: Tue, 16 Apr 2019 13:08:19 GMTConnection: keep-aliveETag: "5cb5d3c3-264"Accept-Ranges: bytesFollow"


Question: How can I create an EventBridge event rule to notify me that my AWS root user account was used?

How can I receive notifications when my AWS root user account is used?

Answer: "How can I receive notifications when my AWS root user account is used?ResolutionLaunch an AWS CloudFormation stack to create an Amazon Simple Notification Service (Amazon SNS) topic. Then, create an Amazon EventBridge event rule to monitor userIdentity root logins from the AWS Management Console.Important: Before you begin, make sure that your AWS CloudTrail Management read/write events are set to All or Write-only for EventBridge events to trigger the log-in event notification. For more information, see Read and write events.1.    Copy and paste this YAML template into your favorite editor tool and then save it.# Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.# Permission is hereby granted, free of charge, to any person obtaining a copy of this# software and associated documentation files (the "Software"), to deal in the Software# without restriction, including without limitation the rights to use, copy, modify,# merge, publish, distribute, sublicense, and/or sell copies of the Software, and to# permit persons to whom the Software is furnished to do so.## THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.AWSTemplateFormatVersion: '2010-09-09'Description: ROOT-AWS-Console-Sign-In-via-CloudTrailMetadata:  AWS::CloudFormation::Interface:    ParameterGroups:    - Label:        default: Amazon SNS parameters      Parameters:      - Email AddressParameters:  EmailAddress:    Type: String    AllowedPattern: "^[\\x20-\\x45]?[\\w-\\+]+(\\.[\\w]+)*@[\\w-]+(\\.[\\w]+)*(\\.[a-z]{2,})$"    ConstraintDescription: Email address required.    Description: Enter an email address you want to subscribe to the Amazon SNS topic      that will send notifications if your account's AWS root user logs in.Resources:  RootActivitySNSTopic:    Type: AWS::SNS::Topic    Properties:      DisplayName: ROOT-AWS-Console-Sign-In-via-CloudTrail      Subscription:      - Endpoint:          Ref: EmailAddress        Protocol: email      TopicName: ROOT-AWS-Console-Sign-In-via-CloudTrail  EventsRule:    Type: AWS::Events::Rule    Properties:      Description: Events rule for monitoring root AWS Console Sign In activity      EventPattern:        detail-type:        - AWS Console Sign In via CloudTrail        detail:          userIdentity:            type:            - Root      Name:        Fn::Sub: "${AWS::StackName}-RootActivityRule"      State: ENABLED      Targets:      - Arn:          Ref: RootActivitySNSTopic        Id: RootActivitySNSTopic    DependsOn:    - RootActivitySNSTopic  RootPolicyDocument:    Type: AWS::SNS::TopicPolicy    Properties:      PolicyDocument:        Id: RootPolicyDocument        Version: '2012-10-17'        Statement:        - Sid: RootPolicyDocument          Effect: Allow          Principal:            Service: events.amazonaws.com          Action: sns:Publish          Resource:          - Ref: RootActivitySNSTopic      Topics:      - Ref: RootActivitySNSTopicOutputs:  EventsRule:    Value:      Ref: EventsRule    Export:      Name:        Fn::Sub: "${AWS::StackName}-RootAPIMonitorEventsRule"    Description: Event Rule ID.2.    Open the CloudFormation console in the US East (N. Virginia) Region, and then choose Create Stack.Note: The CloudFormation stack must be created in the US East (N. Virginia) Region.3.    Choose Create stack, and then choose With new resources (standard).4.    Choose Upload a template file, Next, and then Choose file.5.    Choose the template that you saved in step 1, and then choose Next.6.    In Stack name, enter a name that is meaningful to you, such as Root-AWS-Console-Sign-In-CloudTrail.7.    In EmailAddress, enter your email address, and then choose Next.Note: AWS sends the confirmation email to this email address.8.    In Options, choose Next, and then choose Create.9.    Check your email inbox for the AWS confirmation email, and then choose Confirm subscription to confirm the SNS subscription request. You'll receive a Subscription confirmed! message.10.    To test notifications, sign out of the AWS Management Console. Then, sign in to the AWS Management Console with your AWS root user account.11.    Check your email inbox for an AWS notification message. Note the CloudTrail records userIdentity, sourceIPAddress, and MFAUsed containing details for the log-in event.To stop receiving notifications, delete the CloudFormation stack that you created in step 2.Related informationCreating a stack on the CloudFormation consoleHow to receive notifications when your AWS account’s root access keys are usedMonitor and notify on AWS account root user activityCreating CloudWatch alarms with a CloudFormation templateFollow"


Question: How do I expose the Kubernetes services running on my Amazon EKS cluster?

I want to expose the Kubernetes services running on my Amazon Elastic Kubernetes Service (Amazon EKS) cluster.

Answer: "I want to expose the Kubernetes services running on my Amazon Elastic Kubernetes Service (Amazon EKS) cluster.Short descriptionTo expose the Kubernetes services running on your cluster, first create a sample application. Then, apply the ClusterIP, NodePort, and LoadBalancer Kubernetes ServiceTypes to your sample application.Keep in mind the following details:ClusterIP exposes the service on a cluster's internal IP address.NodePort exposes the service on each node’s IP address at a static port.LoadBalancer exposes the service externally using a load balancer.Note: Amazon EKS supports the Network Load Balancer and the Classic Load Balancer for pods running on Amazon Elastic Compute Cloud (Amazon EC2) instance worker nodes. Amazon EKS provides this support by using a LoadBalancer. You can load balance network traffic to a Network Load Balancer (instance or IP targets) or a Classic Load Balancer (instance target only).ResolutionCreate a sample application1.    Define and apply a deployment file. The following example creates a ReplicaSet that spins up two nginx pods, and then creates filed called nginx-deployment.yaml.cat <<EOF > nginx-deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata:  name: nginx-deployment  labels:    app: nginxspec:  replicas: 2  selector:    matchLabels:      app: nginx  template:    metadata:      labels:        app: nginx    spec:      containers:      - name: nginx        image: nginx:1.14.2        ports:        - containerPort: 80EOF2.    Create the deployment:kubectl apply -f nginx-deployment.yaml3.    Verify that your pods are running and have their own internal IP addresses:kubectl get pods -l 'app=nginx' -o wide | awk {'print $1" " $3 " " $6'} | column -tOutput:NAME                               STATUS   IPnginx-deployment-574b87c764-hcxdg  Running  192.168.20.8nginx-deployment-574b87c764-xsn9s  Running  192.168.53.240Create a ClusterIP service1.    Create a file called clusterip.yaml, and then set type to ClusterIP. For example:cat <<EOF > clusterip.yamlapiVersion: v1kind: Servicemetadata:  name: nginx-service-cluster-ipspec:  type: ClusterIP  selector:    app: nginx  ports:    - protocol: TCP      port: 80      targetPort: 80EOF2.    Create the ClusterIP object in Kubernetes using either a declarative or imperative command.To create the object and apply the clusterip.yaml file, run the following declarative command:kubectl create -f clusterip.yamlOutput:service/nginx-service-cluster-ip created-or-To expose a deployment of ClusterIP type, run the following imperative command:kubectl expose deployment nginx-deployment  --type=ClusterIP  --name=nginx-service-cluster-ipOutput:service "nginx-service-cluster-ip" exposedNote: The expose command creates a service without creating a YAML file. However, kubectl translates your imperative command into a declarative Kubernetes Deployment object.3.    Access the application and get the ClusterIP number:kubectl get service nginx-service-cluster-ipOutput:NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGEnginx-service-cluster-ip   ClusterIP   10.100.12.153   <none>        80/TCP    23s4.    Delete the ClusterIP service:kubectl delete service nginx-service-cluster-ipOutput:service "nginx-service-cluster-ip" deletedCreate a NodePort service1.    To create a NodePort service, create a file called nodeport.yaml, and then set type to NodePort. For example:cat <<EOF > nodeport.yamlapiVersion: v1kind: Servicemetadata:  name: nginx-service-nodeportspec:  type: NodePort  selector:    app: nginx  ports:    - protocol: TCP      port: 80      targetPort: 80EOF2.    Create the NodePort object in Kubernetes using either a declarative or imperative command.To create the object and apply the nodeport.yaml file, run the following declarative command:kubectl create -f nodeport.yaml-or-To expose a deployment of NodePort type, run the following imperative command:kubectl expose deployment nginx-deployment  --type=NodePort  --name=nginx-service-nodeportOutput:service/nginx-service-nodeport exposed3.    Get information about nginx-service:kubectl get service/nginx-service-nodeportOutput:NAME                     TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGEnginx-service-nodeport   NodePort   10.100.106.151   <none>        80:30994/TCP   27sImportant: The ServiceType is a NodePort and ClusterIP that are created automatically for the service. The output from the preceding command shows that the NodePort service is exposed externally on the port (30994) of the available worker node's EC2 instance. Before you access NodeIP:NodePort from outside the cluster, you must set the security group of the nodes to allow incoming traffic. You can allow incoming traffic through the port (30994) that's listed in the output of the preceding kubectl get service command.4.    If the node is in a public subnet and is reachable from the internet, check the node’s public IP address:kubectl get nodes -o wide |  awk {'print $1" " $2 " " $7'} | column -tOutput:NAME                                      STATUS  EXTERNAL-IPip-10-0-3-226.eu-west-1.compute.internal  Ready   1.1.1.1ip-10-1-3-107.eu-west-1.compute.internal  Ready   2.2.2.2-or-If the node is in a private subnet and is reachable only inside or through a VPC, then check the node’s private IP address:kubectl get nodes -o wide |  awk {'print $1" " $2 " " $6'} | column -tOutput:NAME                                      STATUS  INTERNAL-IPip-10-0-3-226.eu-west-1.compute.internal  Ready   10.0.3.226ip-10-1-3-107.eu-west-1.compute.internal  Ready   10.1.3.1075.     Delete the NodePort service:kubectl delete service nginx-service-nodeportOutput:service "nginx-service-nodeport" deletedCreate a LoadBalancer service1.    To create a LoadBalancer service, create a file called loadbalancer.yaml, and then set type to LoadBalancer. For example:cat <<EOF > loadbalancer.yamlapiVersion: v1kind: Servicemetadata:  name: nginx-service-loadbalancerspec:  type: LoadBalancer  selector:    app: nginx  ports:    - protocol: TCP      port: 80      targetPort: 80EOF2.    Apply the loadbalancer.yaml file:kubectl create -f loadbalancer.yamlOutput:service/nginx-service-loadbalancer created-or-Expose a deployment of LoadBalancer type:kubectl expose deployment nginx-deployment  --type=LoadBalancer  --name=nginx-service-loadbalancerOutput:service "nginx-service-loadbalancer" exposed3.    Get information about nginx-service:kubectl get service/nginx-service-loadbalancer |  awk {'print $1" " $2 " " $4 " " $5'} | column -tOutput:NAME                        TYPE          EXTERNAL-IP                        PORT(S)nginx-service-loadbalancer  LoadBalancer  *****.eu-west-1.elb.amazonaws.com  80:30039/TCP4.    Verify that you can access the load balancer externally:curl -silent *****.eu-west-1.elb.amazonaws.com:80 | grep titleYou should receive the following output between HTML title tags: "Welcome to nginx!"5.    Delete the LoadBalancer service:kubectl delete service nginx-service-loadbalancerOutput:service "nginx-service-loadbalancer" deletedNote: By default, the preceding LoadBalancer service creates a Classic Load Balancer.6.    To create a Network Load Balancer with an instance type target, add the following annotation to the service manifest:service.beta.kubernetes.io/aws-load-balancer-type: nlb-or-To create a Network Load Balancer with IP targets, deploy the AWS Load Balancer Controller, and then create a load balancer that uses IP targets.Follow"


Question: How do I check the resource utilization for my SageMaker notebook instance?

"I started an Amazon SageMaker notebook instance to train models, or to load large datasets, and the notebook instance appears to be frozen. How do I view my SageMaker instance resource use?"

Answer: "I started an Amazon SageMaker notebook instance to train models, or to load large datasets, and the notebook instance appears to be frozen. How do I view my SageMaker instance resource use?ResolutionWhen using the SageMaker notebook instance resources to prototype, train models, or when working with large datasets, SageMaker’s browser or notebook instances might appear unresponsive. If your browser or notebook instances appear unresponsive, you can view your SageMaker resource utilization to view current resource use.You can view your SageMaker resource utilization using one of the following approaches:Running Linux-based commandsReviewing Amazon CloudWatch metricsViewing SageMaker resource utilization with Linux commandsSageMaker notebook instances are based on Amazon Linux. You can run Linux commands from the SageMaker terminal to view SageMaker resource utilization.To run SageMaker Linux commands to view your resource utilization, do the following:Open the SageMaker console.On the navigation pane, choose Notebook Instances.Open Jupyter or JupyterLab next to the SageMaker notebook instance of your choice.Open terminal.Run the following commands to view your SageMaker resource utilization:topThe preceding command displays available system memory (RAM) and processor load.ps -axThe preceding command displays tasks running and processor load.df -hThe preceding command displays disk space utilization and availability.free -mThe preceding command displays system memory (RAM) utilization and availability.Viewing SageMaker resource utilization using CloudWatchYou can use CloudWatch to view your SageMaker resource utilization by using a lifecycle configuration script. For example, the publish-instance-metrics script publishes the system-level metrics from the notebook instance into CloudWatch.To configure your SageMaker notebook instance to view all metrics from CloudWatch:Open the SageMaker console.On the navigation pane, choose Notebook Instances.Choose Open Jupyter or Jupyterlab next to the SageMaker notebook instance of your choice.Open terminal.Input the following command to open the amazon-cloudwatch-agent-config-wizard:sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizardFollow the steps in the wizard. When prompted, do the following:Choose On-premises hostChoose no for StatsD DaemonChoose no for CollectDWhen the wizard completes, it automatically creates a config.json file. This file is used in the next step.Start the CloudWatch agent on your server with the following command:sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:///opt/aws/amazon-cloudwatch-agent/bin/config.json -sFrom your CloudWatch console, choose Metrics, then choose CWAgent.The CWAgent displays your current SageMaker metrics.For more information about example AWS lifecycle configuration scripts for SageMaker notebooks, see amazon-sagemaker-notebook-instance-lifecycle-config-samples.Related informationMonitor Amazon SageMaker with Amazon CloudWatchMetrics collected by the CloudWatch agentMonitor Amazon SageMakerTerminals - Jupyter Project documentation for TerminalsFollow"


Question: I accidentally disabled network connectivity to my Amazon EC2 Windows instance. How do I fix it?

"I accidentally disabled network connectivity in Windows. Now, I’ve lost connection to my Amazon Elastic Compute Cloud (Amazon EC2) Windows instance. How can I fix this?"

Answer: "I accidentally disabled network connectivity in Windows. Now, I’ve lost connection to my Amazon Elastic Compute Cloud (Amazon EC2) Windows instance. How can I fix this?ResolutionMethod 1: Use the EC2 Serial ConsoleIf you enabled  EC2 Serial Console for Windows, you can use it to troubleshoot supported Nitro-based instance types. The serial console helps you troubleshoot boot issues, network configuration, and SSH configuration issues. The serial console connects to your instance without the need for a working network connection. You can access the serial console using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).Before using the serial console, grant access to it at the account level. Then create AWS Identity and Access Management (IAM) policies granting access to your IAM users. Also, every instance using the serial console must include at least one password-based user. If your instance is unreachable and you haven’t configured access to the serial console, then follow the instructions in Method 2: Create a temporary elastic network interface. For information on configuring the EC2 Serial Console for Linux, see Configure access to the EC2 Serial Console.Method 2: Create a temporary elastic network interfaceYou can create a temporary elastic network interface, and attach the network interface to the Amazon EC2 Windows instance. Then, you can temporarily connect to the instance and fix the issue.Open the Amazon EC2 console, and then choose Instances from the navigation pane.Select your instance. From the Description tab, make a note of the Subnet ID.Create a new network interface in the same subnet as the instance.Important: Be sure to select a security group that allows incoming Remote Desktop Protocol (RDP) traffic from your IP address.Attach the new network interface to the instance.Note: The network interface might take a few minutes to come online. If you connect to the instance using RDP, associate an Elastic IP address with the network interface.Using the new network interface, connect to the instance using RDP.Change the network connection settings in Windows to use DHCP. Or, specify the correct private IP address settings. For instructions, see Configure a secondary private IPv4 address for your Windows instance.Detach the temporary network interface.Note: If you associated an Elastic IP address with the network interface but no longer need the Elastic IP address, release the Elastic IP address.Related informationTroubleshoot EC2 Windows instancesFollow"


Question: How can I reduce the latency of HLS delivery in MediaLive?

I'm noticing a delay in the HLS delivery of my AWS Elemental MediaLive channel. How can I fix this?

Answer: "I'm noticing a delay in the HLS delivery of my AWS Elemental MediaLive channel. How can I fix this?ResolutionDelays in HLS delivery can be due to various factors, such as encoding and decoding, network conditions, and player buffers. For more information on measuring latency and optimizing your configuration, see How to Compete with Broadcast Latency Using Current Adaptive Bitrate Technologies.One way to potentially reduce latency is to shorten the segment length of the HLS group:Important: Shortening segment length can impact video quality and increase the chance of player-buffering events. Before you proceed, consider the impact for your use case.Open the MediaLive console.From the navigation pane, choose Channels. Then, choose the channel that you want to modify.Choose Actions, and then choose Edit.Choose the HLS output group that you want to modify.Choose Manifest and Segments.For Segment Length, change the value to 1.Then, choose one output within the HLS group.Choose GOP Structure.For GOP Size Units, select SECONDS.For GOP Size, enter 1 to match the segment length of one second.Repeat steps 7 through 10 for all outputs in the HLS group.After you update the outputs, choose Update Channel.Follow"


Question: Why is my AWS Glue crawler not adding new partitions to the table?

My AWS Glue crawler doesn't add new partitions to the table.

Answer: "My AWS Glue crawler doesn't add new partitions to the table.Short descriptionWhen the crawler scans the source data files under a new partition, the crawler compares the following attributes of the source files with those of the existing table:File formatCompression typeSchemaStructure of Amazon Simple Storage Service (Amazon S3) partitionsIf any of these attributes of the partition differ from attributes of the table, then the partition is skipped and not added to the metadata. A difference in the name, sequence, or number of partitions in the Amazon S3 path is considered as a change in the partition schema or structure.ResolutionTroubleshoot the issueCheck the crawler logs to identify the issue:Open the AWS Glue console.In the navigation pane, choose Crawlers.Select the crawler, and then choose the Logs link to view the logs on the CloudWatch console.Review the logs to check if the crawler skipped the new partition.For example, suppose that the log includes entries look similar to the following:Folder partition keys do not match table partition keys, skipped folder: doc-example-bucket/doc-example-path/doc-example-table/year=2021/month=01/sday=05/This entry suggests that the partition structure for the Amazon S3 location doesn't match the partition keys defined for the table. This might happen when the partition structure isn't consistent across the table source location.If the AWS Glue crawler creates multiple tables, then the log entries look similar to the following:INFO : Created table doc-example-table in database doxtest_dbIf you see similar logs, then compare the schema and partition structure of the location of these tables with those of the original table.Resolve the issueBased on the information from the CloudWatch logs, consider one or more of the following solution options:If the issue is caused by inconsistent partition structure, then make the structure consistent by renaming the S3 path manually or programmatically.If the partition is skipped due to mismatch in file format, compression format, or schema, and the data isn't required to be included in the intended table, then consider the following:Use an exclude pattern to skip any unwanted files.Move the unwanted file to a different location.If your data has different schemas in some input files and similar schemas in other input files, then combine compatible schemas when you create the crawler. On the Configure the crawler's output page, under Grouping behavior for S3 data (optional), select Create a single schema for each S3 path. When this setting is turned on and the data is compatible, then the crawler ignores the similarity of specific schemas when evaluating S3 objects in the specified include path. For more information, see How to create a single schema for each Amazon S3 include path.If the crawler is creating multiple tables, then see How can I prevent the AWS Glue crawler from creating multiple tables?Follow"


Question: Why am I getting a 200 response with "InternalError" or "SlowDown" for copy requests to my Amazon S3 bucket?

"I'm trying to perform a copy operation on my Amazon Simple Storage Service (Amazon S3) bucket. However, Amazon S3 returns a 200 response with "InternalError" or "SlowDown," similar to the following:"AmazonS3Exception: We encountered an internal error. Please try again. (Service: Amazon S3; Status Code: 200; Error Code: InternalError; Request ID: 0EXAMPLE9AAEB265)"How can I avoid this error?"

Answer: "I'm trying to perform a copy operation on my Amazon Simple Storage Service (Amazon S3) bucket. However, Amazon S3 returns a 200 response with "InternalError" or "SlowDown," similar to the following:"AmazonS3Exception: We encountered an internal error. Please try again. (Service: Amazon S3; Status Code: 200; Error Code: InternalError; Request ID: 0EXAMPLE9AAEB265)"How can I avoid this error?ResolutionIf you get a 200 response with "InternalError" or "SlowDown" embedded in the response, then an internal error occurred after the copy operation started. You might see 200 responses with these errors when you run CopyObject, UploadPartCopy, or CompleteMultipartUpload API operations.Note: Because a 200 response can contain an error, consider designing your application to parse the contents of the response body to look for any errors. Then, the application can handle the success or error within the response appropriately.A 200 response with "InternalError" or "SlowDown" is similar to a 5xx error. To troubleshoot or avoid a 200 response with "InternalError or "SlowDown", consider the following:Enable a retry mechanism in the application that's sending requests to Amazon S3.Configure your application to gradually increase request rates.Related informationHow do I troubleshoot HTTP 5xx errors from Amazon S3?Follow"


Question: How do I troubleshoot load balancers created by the Kubernetes service controller in Amazon EKS?

I can't create a Kubernetes service that's backed by a load balancer on Amazon Elastic Kubernetes Service (Amazon EKS).

Answer: "I can't create a Kubernetes service that's backed by a load balancer on Amazon Elastic Kubernetes Service (Amazon EKS).Short descriptionAmazon EKS uses one of two controllers to manage a load balancer: AWS Load Balancer Controller or the Kubernetes service controller. The following troubleshooting steps apply only to load balancers that are managed by the Kubernetes service controller. For more information, see The Service Controller on the Kubenetes AWS Cloud Provider website and Type LoadBalancer on the Kubernetes website.To troubleshoot your service load balancer, verify that you have the following configurations:The correct tags for your Amazon Virtual Private Cloud (Amazon VPC) subnetsThe required AWS Identity and Access Management (IAM) permissions for your cluster's IAM roleA valid Kubernetes service definitionLoad balancers that stay within your account limitEnough free IP addresses on your subnetsA correctly configured load balancer to avoid connection timeout issuesHealthy load balancer targetsIf you still have an issue after verifying all the preceding items, then follow the steps in the Try additional troubleshooting steps section.ResolutionThe following steps apply to the Classic Load Balancer and the Network Load Balancer. For the Application Load Balancer, see Application load balancing on Amazon EKS.Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Use the correct tags for your Amazon VPC subnets1.    Open the Amazon VPC console.2.    On the navigation pane, choose Subnets.3.    Choose the Tags tab for each subnet, and then confirm that a tag exists. For example:Key: kubernetes.io/cluster/yourEKSClusterName  Value: sharedNote: In this case, the Value can be shared or owned.By default, the controller automatically discovers the subnets. If there are multiple subnets per Availability Zone, then the controller prioritizes subnets in the following order:Subnets with the correct role tag: kubernetes.io/role/elb for public subnets and kubernetes.io/role/internal-elb for private subnetsSubnets with the cluster tag: kubernetes.io/cluster/CLUSTER_NAMEThe subnet that’s first in lexicographic order4.    For your public subnets, confirm that the following tag exists:Key: kubernetes.io/role/elb  Value: 1Note: To see if a subnet is public, check the route table that's associated with the subnet. A public subnet has a route to an internet gateway (igw-xxxxxxxxxxx). A private subnet has a route to the internet through a NAT gateway or NAT instance, or no route to the internet at all.Important: You must have the tag in step 4 to create an internet-facing load balancer service. An internet-facing load balancer routes requests from clients to targets over the internet.5.    For your private subnets, confirm that the following tag exists:Key: kubernetes.io/role/internal-elb  Value: 1Important: You must have the tag in step 5 to create an internal load balancer service. An internal load balancer uses private IP addresses to route requests to targets.Set the required AWS Identity and Access Management (IAM) permissions for your cluster's IAM role1.    Open the Amazon EKS console.2.    On the navigation pane, choose Clusters.3.    Select your cluster, and then note your Cluster IAM Role ARN.4.    Open the IAM console.5.    On the navigation pane, choose Roles.6.    Select the role that matches the Cluster IAM Role ARN that you identified in step 3.7.    Confirm that the AWS managed policy AmazonEKSClusterPolicy is attached to your role.Note: The Amazon EKS control plane assumes the preceding IAM role to create a load balancer for your service.Use a valid Kubernetes service definitionIn the YAML file for your Kubernetes service, verify that spec.type is set to LoadBalancer.See the following example of a Kubernetes service that's backed by a load balancer:apiVersion: v1  kind: Service  metadata:    annotations:      # This annotation is only required if you are creating an internal facing ELB. Remove this annotation to create public facing ELB.      service.beta.kubernetes.io/aws-load-balancer-internal: "true"    name: nginx-elb    labels:      app: nginx  spec:    type: LoadBalancer    ports:    - name: "http"      port: 80      targetPort: 80    selector:      app: nginxNote: To customize your service with a different annotation, see Internal load balancer and TLS support on AWS on the Kubernetes website.By default, Kubernetes built-in service controller creates a Classic Load Balancer. For Network Load Balancer provision, the service must have the annotation service.beta.kubernetes.io/aws-load-balancer-type: "nlb". For more details, see Network Load Balancer support on AWS on the Kubernetes website.Verify that your load balancers are within your account limitBy default, an AWS account has a maximum of 20 load balancers per AWS Region.To check how many load balancers that you have, open the Amazon Elastic Compute Cloud (Amazon EC2) console. Then, choose Load Balancers from the navigation pane.If you reach the maximum number of load balancers, then you can apply for an increase with Service Quotas.Verify that there are enough free IP addresses on your subnetsTo create a load balancer, each subnet of that load balancer must have a minimum of eight free IP addresses. This is required for both the Classic Load Balancer and Network Load Balancer.Connection timeout issuesIf your load balancer is experiencing intermittent or constant timeout issues, then there might be an issue with the load balancer configuration. This can happen when the backend service can’t handle the incoming traffic, or too many requests are sent to the load balancer. For more information, see How do I troubleshoot Elastic Load Balancing "Connection timed out" errors?Verify load balancer health checksIf you don't correctly configure the endpoint, or the backend service isn't responding to health check requests, then Load balancer health check issues occur.To verify that the load balancer is pointing to the correct endpoint, run the AWS CLI command describe-target-health:aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:us-west-2:1234567890:targetgroup/my-targets/6d0ecf831eec9f09To check the health check configuration of the target group, run the AWS CLI command describe-target-groups:aws elbv2 describe-target-groups --target-group-arns arn:aws:elasticloadbalancing:us-west-2:1234567890:targetgroup/my-targets/6d0ecf831eec9f09For more information, see Health checks for your target groups.Try additional troubleshooting stepsTo check the Kubernetes service for an error message that can help you troubleshoot the issue, run the following describe-service command:$ kubectl describe service my-elb-serviceIf the service is created successfully, then you receive an output that's similar to the following example:...  ...  Events:    Type    Reason                Age   From                Message    ----    ------                ----  ----                -------    Normal  EnsuringLoadBalancer  47s   service-controller  Ensuring load balancer    Normal  EnsuredLoadBalancer   44s   service-controller  Ensured load balancerIf the service isn't created successfully, then you receive an error message.To get more information about error messages, check the following resources:Use the Amazon EKS control plane logs.Find out more about the Kubernetes LoadBalancer service on the Kubernetes website.Review the AWS cloud provider on the Kubernetes GitHub website.Related informationWhy does a subnet in use by load balancers in my VPC have insufficient IP addresses?Amazon EKS troubleshootingTroubleshoot your Application Load BalancersTroubleshoot your Classic Load BalancerFollow"


Question: Troubleshooting AWS DMS error "Last Error Replication task out of memory. Stop Reason FATAL_ERROR Error Level FATAL"

"When using AWS Database Migration Services (AWS DMS), I receive the error "Last Error Replication task out of memory. Stop Reason FATAL_ERROR Error Level FATAL"."

Answer: "When using AWS Database Migration Services (AWS DMS), I receive the error "Last Error Replication task out of memory. Stop Reason FATAL_ERROR Error Level FATAL".Short descriptionWhen using AWS DMS, you receive the following error: "Last Error Replication task out of memory. Stop Reason FATAL_ERROR Error Level FATAL". To find the root cause of the error, review the AWS DMS task logs. For the preceding error, the logs provide the following information: “Task process for 'XXXXXXX' failed because it ran out of memory”.To resolve this error, complete one or more of the following steps:Change the task settings or memory-related parameters.Scale up the replication instance class based on fluctuations in Amazon CloudWatch metrics for replication instances, such as FreeMemory and SwapUsage.Split a single task into multiple tasks based on the size of data that's migrated and the amount of memory that's required for the task.ResolutionNote: You must stop the task prior to making modifications. After you’ve made the modifications, you must resume the task. In-flight tables are reloaded from scratch if the task stops during the full load phase.Change the task settings or memory-related parametersCheck if you can scale the task settings or memory-related parameters that require higher memory capacity. The following are some of the most common task settings and parameters to check:LOB settingsValidation parameters, such as ThreadCount and PartitionSizeParallel thread parameters, such as ParallelLoadThreads, ParallelLoadBufferSize, ParallelLoadQueuesPerThread, ParallelApplyThreads, ParallelApplyBufferSize, and ParallelApplyQueuesPerThread.Batch apply parameters, such as BatchApplyTimeoutMin, BatchApplyTimeoutMax, BatchApplyMemoryLimit, and BatchSplitSize.Other memory-related task settings, such as MinTransactionSize, MemoryLimitTotal, MemoryKeepTime, and StatementCacheSize.For more details on the preceding task settings and parameters, see How does AWS DMS use memory for migration?Scale up replication instance class based on fluctuations in Amazon CloudWatch metricsCheck the replication instance’s FreeMemory and SwapUsage metrics. If FreeMemory decreases or SwapUsage either increases or fluctuates, then consider moving to a larger replication instance.Also, consider using memory-optimized instances. Memory-optimized instances are suited for memory-intensive workloads, such as ongoing migrations and replications of high-throughput transactions. For more information on replication instance size and types, see Choosing the right AWS DMS replication instance for your migration.Split a single task into multiple tasks based on the size of data that's migrated and the amount of memory that's required for the taskIf the replication instance has multiple tasks, then you can use the DMS MemoryUsage metric to observe the amount of memory that the task consumes. To determine why the task is holding memory in the CDC phase, compare CDCChangesMemorySource and CDCChangesMemoryTarget, and then troubleshoot the respective endpoint.When multiple tasks are running on the replication instance, do one or more of the following actions:Reduce the number and type of tasks that are running on the replication instance.Move the failed task to a different replication instance, and then try again.Increase instance capacity.For tasks that have multiple tables loading in parallel or many tables and schemas being migrated, do one or more of the following actions:Reduce the number of tables that are loading in parallel.Reduce the number of total tables and schemas being migrated.Use a different task on a different replication instance to offload the migration of some of the tables and schemas.Increase instance capacity.Related informationTarget metadata task settingsChange processing tuning settingsAWS Database Migration Service metricsFollow"


Question: How do I resize an Amazon Redshift cluster?

I want to resize an Amazon Redshift cluster. How does that impact performance and billing?

Answer: "I want to resize an Amazon Redshift cluster. How does that impact performance and billing?ResolutionThere are four ways to resize an Amazon Redshift cluster:Elastic resize: If elastic resize is available as an option, use elastic resize to change the node type, number of nodes, or both. Note that when you only change the number of nodes, the queries are temporarily paused and connections are kept open. An elastic resize takes between 10-15 minutes. During a resize operation, the cluster is read-only.Classic resize: Use classic resize to change the node type, number of nodes, or both. Choose this option when you're resizing to a configuration that isn't available through elastic resize. A resize operation takes two hours or more, or can take up to several days depending on your data size. During the resize operation, the source cluster is read-only.Snapshot, restore, and resize: To be sure that the cluster is available during a classic resize operation, copy the existing cluster. Then, resize the new cluster. If data is written to the source cluster after a snapshot is taken, the data must be manually copied over. The manual data copy to the newly created target cluster must take place after the migration completes.Fast Classic Resize: Fast classic resize is as quick as elastic resize and functions similar to classic resize. In this resize operation there are two main stages. In stage 1 (critical path), data is migrated from the source to a target cluster and the cluster is in read only mode. In stage 2 (off critical path) redistributing of the data, done in the previous data distribution style, is completed in the background. The duration of this stage depends on the volume to distribute and cluster workload.For more information, see Overview of managing clusters in Amazon Redshift.Prerequisites for resizeTo verify if your cluster is eligible for elastic resize, run the following AWS CLI or AWS CloudShell command:aws redshift describe-node-configuration-options --cluster-identifier <cluster-id> --action-type resize-clusterNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.If the cluster is eligible for elastic resize, the output is similar to following in AWS CLI:{    "NodeConfigurationOptionList": [        {            "NodeType": "dc2.large",            "NumberOfNodes": 2,            "EstimatedDiskUtilizationPercent": 0.01      },        {            "NodeType": "ra3.16xlarge",            "NumberOfNodes": 2,            "EstimatedDiskUtilizationPercent": 0.01        }]        }If the cluster isn't eligible for elastic resize, the output is similar to following in AWS CLI:{    "NodeConfigurationOptionList": []}Performance benchmarkBefore a resize, you can perform a benchmark test on existing cluster workloads and target cluster workloads for making resize decisions.Resize operation speedIf elastic resize is used to resize a cluster with the same node type, the operation doesn't create a new cluster. As a result, the operation completes quickly. The time required to complete a classic resize or a snapshot and restore operation might vary, depending on the following factors:The workload on the source cluster.The number and size of the tables being transferred from source to target cluster.How evenly data is distributed across the compute nodes and slices.The node configuration in the source and target clusters.Note: If you perform a classic resize on a cluster with a large volume of data, and the nodes aren't RA3, data migration might be slow. It can take several days to migrate a cluster with multiple terabytes (TB) of data. Data transfer for RA3 nodes completes more quickly.Optimize operation speedTo reduce the time required for a classic resize or a snapshot and restore operation:Migrate to RA3 node type as resize is quicker.Run the table inspector script from AWS Labs to identify skewed tables. To fix skewed tables, choose an appropriate distribution key. For more information, see Amazon Redshift Engineering’s advanced table design playbook: Distribution styles and distribution keys.Remove unused tables. To identify unused tables, run the unscanned table summary script from AWS Labs. Note: The unscanned table summary only shows recent history (between 2-5 days). Use the System object persistence utility to capture usage data over a longer period.Identify tables with missing statistics from AWS Labs and run the ANALYZE command on those tables.For more information about optimizing your resize performance, see Top 10 performance tuning techniques for Amazon Redshift.To check the status of your resize operation using the Amazon Redshift console, choose the Status tab on the cluster details page. The Status tab shows the average rate of transfer, the elapsed time, and the remaining time.TroubleshootingDuring a resize operation, your table will increase or decrease in size. This behavior is expected. For more information, see Why does a table in my Amazon Redshift cluster consume more disk storage space than expected?If your cluster has a status of NONE in the AWS CLI, then the target cluster is still being provisioned. When your target cluster is being provisioned, it hasn't copied over yet. After your target cluster is provisioned, the status changes to IN_PROGRESS.If your AWS CloudFormation StackSets has failed to resize with an error of "An internal error has occurred. Please try your query again at a later time". Check if the cluster is eligible for elastic resize. CloudFormation stack uses elastic resize where Classic:false is set by default.If you receive an error message prompting you to "Please choose a larger target cluster," then your data does not fit into the target cluster. Resize your Amazon Redshift cluster with more nodes or a different node type.To cancel a resize operation before it completes, choose cancel resize from the cluster list in the Amazon Redshift console. For more information, see Snapshot, restore, and resize.Billing for resized clustersDuring the resize operation, you're billed for the clusters that are available to you. For example, during the resize operation, you're billed for the source configuration. After the resize is complete, you're no longer billed for the source configuration. Billing starts for the target configuration as soon as the cluster status changes to Available.When you resize smaller node types (large, xlarge) to larger node types (8xlarge), your cluster requires more storage per node. The more storage you have per node, the more metadata that's written when you run a COMMIT. Therefore, the base cost for a single COMMIT operation is higher for larger nodes. If you run several small COMMIT operations concurrently, you might see a decrease in performance. For improved performance, group multiple changes into a single COMMIT operation.If you purchased Reserved Instances, then your billing depends on the resized cluster configuration, reserved node types, and reserved node count. For more information, see How reserved nodes work.Related informationResizing clusters in Amazon RedshiftTroubleshooting connection issues in Amazon RedshiftBuilding high-quality benchmark tests for Amazon Redshift using SQLWorkbench and psqlFollow"


Question: How do I troubleshoot insufficient capacity error when launching my Amazon SageMaker resources?

I'm getting the Insufficient Capacity error when I'm trying to launch one of the following Amazon SageMaker resources:Training jobBatch transform jobEndpointNotebook instanceSageMaker Studio App

Answer: "I'm getting the Insufficient Capacity error when I'm trying to launch one of the following Amazon SageMaker resources:Training jobBatch transform jobEndpointNotebook instanceSageMaker Studio AppResolutionThis error occurs when AWS doesn't have enough available on-demand capacity for an Amazon Elastic Compute Cloud (Amazon EC2) instance type within a specific Region or Availability Zone to complete your request. Note that capacity isn't static and varies depending on the time of day and workloads in a particular Region or Availability Zone. This error doesn't result from resource limits that are applied to your account. Because capacity issues are transient, be sure to retry your request when you get these errors.If you need the instance immediately, do the following:Switch to a different instance type. Switch to a different instance size in the same family, or use a different instance family based on your workload. A different instance type might have more capacity.Launch the resource in a different Region. Be sure to verify the SageMaker instance types that are available in each Region.If your request can be delayed, try at a later time to check whether your desired instance is available.Based on the SageMaker resource that you're trying to launch, you can also try the following approaches:If you're launching the SageMaker Studio App, it's a best practice to launch the SageMaker Studio domain with subnets that cover wider Availability Zones. Configuring the resource with subnets that span out in multiple Availability Zones minimizes the possibility of experiencing capacity issues.If you're launching a notebook instance or training job, consider launching the instance with the same instance type while specifying different subnets in different Availability Zones.Related informationInsufficient instance capacityHow do I troubleshoot InsufficientInstanceCapacity errors when starting or launching an EC2 instance?Follow"