Question: Why is Kinesis Data Firehose creating so many small files in S3?

"I'm trying to push data from Amazon Kinesis Data Firehose to Amazon Simple Storage Service (Amazon S3). However, I noticed that Kinesis Data Firehose is creating many small files in my Amazon S3 bucket. Why is this happening?"

Answer: "I'm trying to push data from Amazon Kinesis Data Firehose to Amazon Simple Storage Service (Amazon S3). However, I noticed that Kinesis Data Firehose is creating many small files in my Amazon S3 bucket. Why is this happening?Short descriptionKinesis Data Firehose delivers smaller records than specified (in the BufferingHints API) for the following reasons:Compression is enabled.Kinesis Data Firehose delivery stream has scaled.Amazon Kinesis Data Streams is listed as the data source.ResolutionCompression is enabledIf compression is enabled on your Kinesis Data Firehose delivery stream, both of the BufferingHints parameters are applied before the compression. Check the SizeInMBs and IntervalInSeconds parameters to confirm.After each batch of records is buffered, the parameters are applied. When the data records are buffered and compressed, smaller files are created in Amazon S3.Kinesis Data Firehose delivery stream has scaledIf a limit increase was requested or Kinesis Data Firehose has automatically scaled, then the Data Firehose delivery stream can scale. By default, Kinesis Data Firehose automatically scales delivery streams up to a certain limit. Amazon Kinesis' automatic scaling behavior reduces the likelihood of throttling without requiring a limit increase.When Kinesis Data Firehose's delivery stream scales, it can cause an effect on the buffering hints of Data Firehose.Note: The BufferSize is set when you configure your Kinesis Data Firehose.There is also a proportional number of parallel buffering within the Kinesis Data Firehose delivery stream, where data is delivered simultaneously from all these buffers. For example, Kinesis Data Firehose can buffer the data and create a single file based on the buffer size limit. If Kinesis Data Firehose scales to double the current throughput limit, then two separate channels will create the files within the same time interval. If Kinesis Data Firehose scales up to four times, there will be four different channels creating four files in S3 during the same time interval.Note: The number of channels created internally will depend on Kinesis Data Firehose. In the example above, four channels were created.Check to make sure that the Kinesis Data Firehose delivery stream hasn't scaled beyond the default limit. To view the current limit of your Kinesis Data Firehose delivery stream, check the following Amazon CloudWatch metrics:BytesPerSecondLimitRecordsPerSecondLimitPutRequestsPerSecondLimitIf the values of these metrics differ from the default quota limits, then it indicates that Kinesis Data Firehose' delivery stream has scaled.Kinesis Data Stream is listed as the data sourceWhen a Kinesis Data Stream is listed as a data source of Kinesis Data Firehose, then Kinesis Data Firehose scales internally. By default, Kinesis Data Firehose tries to meet the volume capacity of the Kinesis Data Stream. This scaling causes a change in the buffering size and can lead to the delivery of smaller sized records.Note: Buffering hint options are treated as hints. As a result, Kinesis Data Firehose might choose to use different values to optimize the buffering.Follow"


Question: How can I rotate an AWS Secrets Manager secret for a DB connection that requires SSL?

"My AWS Secrets Manager rotation function can't connect to an Amazon Relational Database Service (Amazon RDS) instance with Secure Socket Layer (SSL). I receive an error similar to the following:": setSecret: Unable to log into database with previous, current, or pending secret of secret""

Answer: "My AWS Secrets Manager rotation function can't connect to an Amazon Relational Database Service (Amazon RDS) instance with Secure Socket Layer (SSL). I receive an error similar to the following:": setSecret: Unable to log into database with previous, current, or pending secret of secret"ResolutionRotation functions for Amazon RDS (except Amazon RDS for Oracle) and Amazon DocumentDB automatically use SSL/TLS to connect to your database if it's available. If SSL or TLS isn't available, then the rotation functions use an unencrypted connection.If you set up secret rotation before December 20, 2021, then your rotation function might be based on an older template that doesn't support SSL/TLS. To support connections that use SSL/TLS, you must recreate your rotation function. For more information, see My database requires an SSL/TLS connection but the AWS Lambda rotation function isn't using SSL/TLS.Verify connection to the database for your DB user and retrieve the secret1.    To connect to your Amazon RDS instance, see Connecting to an Amazon RDS DB instance.2.    To connect your Amazon DocumentDB, see Developing with Amazon DocumentDB.3.    To retrieve the secret, see Retrieve secrets from AWS Secrets Manager in code.4.    To automatically rotate the secret, see Automatically rotate an Amazon RDS, Amazon DocumentDB, or Amazon Redshift secret.Related informationCustomize a Lambda rotation function for Secrets ManagerHow do I create a rotation function with an AWS Secrets Manager secret for an unsupported database?AWS Secrets Manager now automatically enables SSL connections when rotating database secretsFollow"


Question: What is the maintenance window for Amazon RDS and Amazon Redshift?

What is the maintenance window for Amazon Relational Database Service (Amazon RDS) and Amazon Redshift? What is the difference between cluster maintenance and instance maintenance?

Answer: "What is the maintenance window for Amazon Relational Database Service (Amazon RDS) and Amazon Redshift? What is the difference between cluster maintenance and instance maintenance?ResolutionThe maintenance window for Amazon RDS and Amazon Redshift occurs weekly, and DB instances can receive upgrades to the operating system (OS) or to the DB engine. AWS requires at least a 30-minute window in your instance's weekly schedule to confirm that all instances have the latest patches and upgrades. During the maintenance window, tasks are performed on clusters and instances. For the security and stability of your data, maintenance can cause instances to be unavailable.The maintenance window defines when the deployment or operation begins, but maintenance itself can take longer to complete. As a result, the time used for some operations can exceed the duration of the maintenance window.Amazon RDSAmazon RDS maintenance can affect all instances or just specific instances. To minimize the impact that Amazon RDS maintenance can have on your workload, see How do I minimize downtime during required Amazon RDS maintenance?Amazon AuroraFor all versions after Amazon Aurora 1.8, DB updates are performed cluster-wide, and maintenance is deployed during the cluster's maintenance window. DB engine updates to Aurora clusters can be performed across all instances or on a one-by-one basis, depending on the type of update. For more information, see Maintaining an Amazon Aurora DB cluster.Amazon RedshiftAmazon Redshift maintenance affects all instances that require updates for OS- or engine-specific updates. To minimize the impact that Amazon Redshift maintenance can have on your workload, schedule around maintenance windows.Related informationWhat happens to Amazon RDS and Amazon Redshift queries that are running during a maintenance window?How do I configure notifications for Amazon RDS or Amazon Redshift maintenance windows?How long is the Amazon RDS maintenance window?Follow"


Question: Why is my certificate renewal still pending after I validated my domain names using the ACM managed renewal process?

"I validated my domain using the AWS Certificate Manager (ACM) managed renewal process, but the status is still "Pending validation." How can I resolve this?"

Answer: "I validated my domain using the AWS Certificate Manager (ACM) managed renewal process, but the status is still "Pending validation." How can I resolve this?Short descriptionThe renewal process differs for email-validated certificates and DNS-validated certificates.ACM tries to automatically renew your ACM certificates 60 days before the certificate expires for DNS-validated certificates. To confirm that a domain is validated, expand the certificate's details in the ACM console. Or, use the describe-certificate command in the AWS Command Line Interface (AWS CLI). If ACM can't automatically validate one or more domain names in the certificate, then the renewal status is "Pending validation."This can happen because:Not all the domains listed in the ACM certificate are validated.The automatic validation failed.The managed renewal process is asynchronous.The original certificate expired.Note: For email-validated certificates renewals, ACM begins sending renewal notices 45 days before expirations that require action by the domain owner.ResolutionUse the following instructions to troubleshoot the ACM renewal status "Pending validation."Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.Not all the domains listed in the ACM certificate are validatedIf you validate domains manually, then each domain included in the ACM certificate must be validated.If you use email validation, then a set of validation emails is sent for each domain. You must complete the steps included in these emails to validate the domains. For more information, see Email validation.The automatic validation failedIf ACM can't automatically validate a domain, see Handling failures in managed certificate renewal.The managed renewal process is asynchronousIt can take up to a few hours for ACM to obtain the new certificate. During this time, the status in the ACM console remains Pending validation.If the update is delayed, then the domain's validation status in the ACM console is Success and the certificate's renewal status is Pending validation.The original certificate expiredIf the original email-validated ACM certificate expires, then the certificate status changes from Issued to Pending validation. You must validate the domain within 72 hours, or the renewal status changes from Pending validation to Failed. If the renewal fails, you must request another public certificate for the domains.Related informationManaged renewal for ACM certificatesCheck a certificate's renewal statusWhy did my publicly trusted ACM certificate fail managed renewal?Follow"


Question: Why is my Amazon ECS task stopped?

My Amazon Elastic Container Service (Amazon ECS) task stopped. How do I troubleshoot issues stopping my Amazon ECS task?

Answer: "My Amazon Elastic Container Service (Amazon ECS) task stopped. How do I troubleshoot issues stopping my Amazon ECS task?Short descriptionYour Amazon ECS tasks might stop due to a variety of reasons. The most common reasons are:Essential container exitedFailed Elastic Load Balancing (ELB) health checksFailed container health checksUnhealthy container instanceUnderlying infrastructure maintenanceService scaling event triggeredResourceInitializationErrorCannotPullContainerErrorTask stopped by userUnderstanding the correlation between a stopped task and stopped reason can help reduce the effort needed to troubleshoot.ResolutionYou can view the details of a stopped task using the DescribeTasks API. However, the details for the stopped task appear only for one hour in the returned results. To view stopped task details longer, you can use this AWS CloudFormation template to store Amazon CloudWatch Logs from an EventBridge event that is triggered when a task is stopped.Stopped reasonsEssential container in task exitedAll tasks must have at least one essential container. If the essential parameter of a container is marked as true and that container fails or stops for any reason, then all other containers that are part of that task are stopped. To understand why a task exited with this reason, identify the exit code using DescribeTasks API and navigate to Common exit codes section of this article.Task failed ELB health checksWhen a task fails due to ELB health checks, confirm that your container security group allows traffic originating from ELB. Consider the following:Define a minimum health check grace period. This instructs the service scheduler to ignore Elastic Load Balancing health checks for a predefined time period after a task has been instantiated.By default, a target starts to receive its full share of requests as soon as it's registered with a target group and passes an initial health check. Using slow start mode gives targets time to warm up before the load balancer sends them a full share of requests.Monitor the CPU and memory metrics of the service. For example, high CPU can make your application unresponsive and result in a 502 error.Check your application logs for application errors.Check if the ping port and the health check path are configured correctly.Curl the health-check path from within Amazon Elastic Compute Cloud (Amazon EC2) and confirm the response code.Failed container health checksHealth checks can be defined in the TaskDefinition API or Dockerfile.You can view the health status of both individual containers and the task with the DescribeTasks API operation.Be sure that the health check command exit status indicates that the container is healthy. Check your container logs for application errors using the log driver settings specified in the task definition. Following are the possible values:0: success – The container is healthy and ready for use.1: unhealthy – The container isn't working correctly.2: reserved– Don't use this exit code.(instance i-xx) (port x) is unhealthy in (reason Health checks failed)This indicates the container status is unhealthy. To troubleshoot this issue:Verify the security group attached to the container instance is permitting traffic.Confirm there is a successful response from the back end without delay.Set the response time value correctly.Check the access logs of your load balancer for more information.Service ABCService: ECS is performing maintenance on the underlying infrastructure hosting the taskThis indicates that the task was stopped due to a task maintenance issue. For more information, see AWS Fargate task maintenance.A service makes sure that the scheduling strategy specified is followed and tasks are rescheduled when they are stopped or failed. If the container instance is part of an Auto Scaling group. A new container instance must be launched and tasks placed. For more information, see Verifying a scaling activity for an Auto Scaling group.ECS service scaling event triggeredThis is a standard service message. Amazon ECS leverages the Application Auto Scaling service to provide this functionality. The ECS service has the ability to increase or decrease the desired count of task automatically. Consider the following actions:Review Cloudwatch Alarms.Review if any deployments were scheduled.ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failedTo troubleshoot this error, see How do I troubleshoot the error "unable to pull secrets or registry auth" in Amazon ECS?CannotPullContainerErrorThis error indicates that the task execution role being used doesn't have permission to communicate to Amazon ECS. To troubleshoot this issue:Verify that the Task Execution Role has the needed permissions. Amazon ECS provides the managed policy named AmazonECSTaskExecutionRolePolicy which contains the permissions for most use cases.Verify the ECR Service Endpoint is accessible to: ecr.region.amazonaws.com and dkr.ecr.region.amazonaws.comFor private images needing authentication, ensure the repositoryCredentials and credentialsParameter are defined with the correct information. For more information, see Private registry authentication for tasks.Task stopped by userThis indicates the task received a StopTask. You can identify who initiated the call by viewing StopTask in CloudTrail for userIdentity information.Common exit codes0 – Entrypoint, success, or CMD is completing its execution and thus the container is stopped.1 – Refers to application error. For more information, review application logs.137 – Occurs when the Task was force exit ( SIGKILL) for the container:Failing to respond to a SIGTERM within a default 30-second period after which the SIGKILL value is sent and containers are forcibly stopped. The default 30-second period can be configured on the ECS container agent with ECS_CONTAINER_STOP_TIMEOUT parameter.This could also occur in an Out-of-Memory (OOM) situation. Review your CloudWatch metrics to verify if OOM occurred.139 – Occurs when a segmentation fault is experienced. Likely, the application tried to access a memory region which it not available, or there is an unset or invalid environment variable.255 – Occurs when the ENTRYPOINT CMD command in your container failed due to an error. Review your CloudWatch Logs to confirm this.Common error messagesNo Container Instances were found in your clusterReview the container instances section for your cluster. If needed, you can launch a container instance.InvalidParameterExceptionBe sure any parameters defined in TaskDefinition are present and the ARN is correct. Verify that the task role and task execution role has sufficient permissions.You've reached the limit of the number of tasks you can run concurrentlyFor more information on limits, see the ECS Service Quotas.For all other quota increase requests, create a case in the AWS Support console, and then choose Service limit increase.Follow"


Question: How can I schedule Amazon RDS for MySQL DB instance binlog files to be uploaded to Amazon S3?

I need to back up Amazon Relational Database Service (Amazon RDS) for MySQL DB instance binlog files to Amazon Simple Storage Service (Amazon S3). How can I schedule downloading the files and syncing them to an S3 bucket?

Answer: "I need to back up Amazon Relational Database Service (Amazon RDS) for MySQL DB instance binlog files to Amazon Simple Storage Service (Amazon S3). How can I schedule downloading the files and syncing them to an S3 bucket?Short descriptionBy default, you can't copy binlog files from an Amazon RDS for MySQL DB instance to a S3 bucket that you choose. Instead, download the binlog files to an Amazon Elastic Compute Cloud (Amazon EC2) instance. Then, sync the files to an S3 bucket by using a cron job.ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.1.    Enable binlog file retention on the RDS DB instance so that you can access MySQL binary logs. This example sets the retention period to 24 hours:mysql> call mysql.rds_set_configuration('binlog retention hours', 24);2.    Run the following command on your Amazon RDS for MySQL DB instance. Confirm that the binlog retention hours parameter is correctly set to 24 (or 1 day):mysql> call mysql.rds_show_configuration;Expected output:+------------------------+-------+-----------------------------------------------------------------------------------------------------------+| name                   | value | description                                                                                |+------------------------+-------+-----------------------------------------------------------------------------------------------------------+| binlog retention hours | 24    | binlog retention hours specifies the duration in hours before binary logs are automatically deleted.      || source delay           | 0     | source delay specifies replication delay in seconds between current instance and its master.              || target delay           | 0     | target delay specifies replication delay in seconds between current instance and its future read-replica. |+------------------------+-------+-----------------------------------------------------------------------------------------------------------+3.    Install MySQL on the EC2 instance for the mysql and mysqlbinlog commands:$ sudo yum update$ sudo yum install mysql574.    Confirm that you installed the utilities correctly by running the following command:$ rpm -qa|grep mysqlExpected output:mysql57-server-5.7.21-2.6.amzn1.x86_64mysql57-common-5.7.21-2.6.amzn1.x86_64mysql57-errmsg-5.7.21-2.6.amzn1.x86_64mysql57-5.7.21-2.6.amzn1.x86_645.    Create an S3 bucket and an AWS Identity and Access Management (IAM) user account.6.    Grant the IAM user permissions to access the S3 bucket.7.    Install the AWS Command Line Interface, and then configure the AWS CLI, if you haven't already.8.    Download the rds-binlog-to-s3 script from the GitHub website. This script downloads binlog files, and then uploads the files to an S3 bucket.9.    Modify the script permissions:$ chmod 744 rds-binlog-to-s3.sh10.    Run the rds-binlog-to-s3 script on the EC2 instance using the AWS CLI. Modify the mysqlbinlog command in the script to include any mysqlbinlog options you want to add:$ rds-binlog-to-s3.sh11.    Set a cron job to sync the binlog files to the S3 bucket:$ crontab -e    0 0  * * * /home/ec2-user/scheduler/RDS-binlog-to-s3.shRelated informationMySQL database log filesFollow"


Question: "How can I access Amazon S3 Requester Pays buckets from AWS Glue, Amazon EMR, or Amazon Athena?"

"I want to access an Amazon Simple Storage Service (Amazon S3) Requester Pays bucket from AWS Glue, Amazon EMR, or Amazon Athena."

Answer: "I want to access an Amazon Simple Storage Service (Amazon S3) Requester Pays bucket from AWS Glue, Amazon EMR, or Amazon Athena.Short descriptionTo access S3 buckets that have Requester Pays turned on, all requests to the bucket must have the Requester Pays header.ResolutionAWS GlueAWS Glue requests to Amazon S3 don't include the Requester Pays header by default. Without this header, an API call to a Requester Pays bucket fails with an AccessDenied exception. To add the Requester Pays header to an ETL script, use hadoopConfiguration().set() to turn on fs.s3.useRequesterPaysHeader on the GlueContext variable or the Apache Spark session variable.GlueContext:glueContext._jsc.hadoopConfiguration().set("fs.s3.useRequesterPaysHeader","true")Spark session:spark._jsc.hadoopConfiguration().set("fs.s3.useRequesterPaysHeader","true")The following is an example of how to use the header in an ETL script. Replace the following values:database_name: the name of your database your_table_name: the name of your table s3://awsdoc-example-bucket/path-to-source-location/: the path to the source bucket s3://awsdoc-example-bucket/path-to-target-location/: the path to the destination bucketimport sysfrom awsglue.transforms import *from awsglue.utils import getResolvedOptionsfrom pyspark.context import SparkContextfrom awsglue.context import GlueContextfrom awsglue.job import Jobfrom awsglue.dynamicframe import DynamicFrame## @params: [JOB_NAME]args = getResolvedOptions(sys.argv, ['JOB_NAME'])sc = SparkContext()glueContext = GlueContext(sc)spark = glueContext.spark_sessionjob = Job(glueContext)job.init(args['JOB_NAME'], args)spark._jsc.hadoopConfiguration().set("fs.s3.useRequesterPaysHeader","true")# glueContext._jsc.hadoopConfiguration().set("fs.s3.useRequesterPaysHeader","true")##AWS Glue DynamicFrame read and writedatasource0 = glueContext.create_dynamic_frame.from_catalog(database = "your_database_name", table_name = "your_table_name", transformation_ctx = "datasource0")datasource0.show()datasink = glueContext.write_dynamic_frame.from_options(frame = datasource0, connection_type = "s3", connection_options = {"path":"s3://awsdoc-example-bucket/path-to-source-location/"}, format = "csv")##Spark DataFrame read and writedf = spark.read.csv("s3://awsdoc-example-bucket/path-to-source-location/")df.show()df.write.csv("s3://awsdoc-example-bucket/path-to-target-location/")job.commit()Amazon EMRSet the following property in /usr/share/aws/emr/emrfs/conf/emrfs-site.xml:<property>   <name>fs.s3.useRequesterPaysHeader</name>   <value>true</value></property>AthenaTo allow workgroup members to query Requester Pays buckets, choose Enable queries on Requester Pays buckets in Amazon S3 when you create the workgroup. For more information, see Create a workgroup.Related informationDownloading objects in Requester Pays bucketsHow do I troubleshoot 403 Access Denied errors from Amazon S3?Follow"


Question: How can I load data from the Slack app to Amazon S3 by using Amazon AppFlow?

"I want to extract my data from the Slack app, and load it to Amazon Simple Storage Service (Amazon S3). How can I use Amazon Appflow to do this do this?"

Answer: "I want to extract my data from the Slack app, and load it to Amazon Simple Storage Service (Amazon S3). How can I use Amazon Appflow to do this do this?Short descriptionTo load your Slack data to Amazon S3, first create an app on Slack. Then, extract data from that app to Amazon S3 using Amazon Appflow. You can also use the method in this article to load data from an existing Slack app to Amazon S3.To do this, you need to:Note your client ID, client secret, and Slack instance name.Set a redirect URL.Set the required user scopes in Slack.ResolutionLog in to Slack, and create a workspace.Create a channel, and give access to your users by adding their email ID.Create an app in the workspace that you created by adding a name, and then choosing the Create App button.After you create an app, note the following information on the confirmation page. You need this information to create an Amazon AppFlow connector to Slack. See this example:App ID: A0****DDate of app creation: October 20, 2021Client ID: 2****97.2**9Client secret: 5****95Signing secret: d0****f4Verification code: U******qOn the same page, under Install your app section, choose permission scope.Under User Token Scopes, add an OAuth Scope for your user token.Under Redirect URLs, add a URL similar to this:For us-eas-1 Region use: https://console.aws.amazon.com/appflow/oauthFor all other Regions, use: https://region.console.aws.amazon.com/appflow/oauthNote: Be sure to replace "region" in this URL with the Region you are using.Open the Amazon AppFlow console.Choose Create flow, and then under Source name, choose Slack.Under Choose Slack connection, choose Create new connection.Under Connect to Slack, Enter the details that you noted in step 4.For Destination name, choose Amazon S3, and for Destination bucket, choose the S3 bucket that you want to use.Map all fields, and then choose Run flow.After your flow is finished running, check the output file in the S3 bucket that you specified as a destination.Related informationWhat is Amazon AppFlow?Supported source and destination applicationsFollow"


Question: I'm seeing inbound traffic in the VPC flow logs for my public NAT gateway. Is my public NAT gateway accepting inbound traffic from the internet?

"My Virtual Private Cloud (VPC) flow logs show Action = ACCEPT for inbound traffic coming from public IP addresses. However, my understanding of network address translation (NAT) gateways was that they don't accept traffic from the internet. Is my NAT gateway accepting inbound traffic from the internet?"

Answer: "My Virtual Private Cloud (VPC) flow logs show Action = ACCEPT for inbound traffic coming from public IP addresses. However, my understanding of network address translation (NAT) gateways was that they don't accept traffic from the internet. Is my NAT gateway accepting inbound traffic from the internet?ResolutionNAT gateways managed by AWS don't accept traffic initiated from the internet. However, there are two reasons why information in your VPC flow logs might appear to indicate that inbound traffic is being accepted from the internet.Reason #1: Inbound internet traffic is permitted by your security group or network access control lists (ACL)VPC flow logs show inbound internet traffic as accepted if the traffic is permitted by your security group or network ACLs. If network ACLs attached to a NAT gateway don’t explicitly deny traffic from the internet, then the traffic to the NAT gateway appears accepted. However, the traffic isn't actually accepted by the NAT gateway and is dropped.To confirm this, do the following:Open the Amazon CloudWatch console.In the navigation pane, choose Insights.From the dropdown, select the log group that contains the NAT gateway's elastic network interface and the private instance's elastic network interface.Run the query below.filter (dstAddr like 'xxx.xxx' and srcAddr like 'public IP')| stats sum(bytes) as bytesTransferred by srcAddr, dstAddr| limit 10Note: You can use just the first two octets in the search filter to analyze all network interfaces in the VPC. In the example above, replace xxx.xxx with the first two octets of your VPC classless inter-domain routing (CIDR). Also, replace public IP with the public IP that you're seeing in the VPC flow log entry.Query results show traffic on the NAT gateway private IP from the public IP, but not traffic on other private IPs in the VPC. These results confirm that the incoming traffic was unsolicited. However, if you do see traffic on the private instance's IP, then follow the steps under Reason #2.Reason #2: Traffic to the public IP was initiated from a private instanceIf there are private instances using NAT gateway for internet access, then your VPC flow log includes the response traffic from the public IP address. To confirm that traffic to the public IP was initiated from your private instance, run the query below:Note: Before running the query, do the following:Select the time frame that corresponds with the time frame when you observed traffic in the VPC flow logs.If you have multiple log groups in your VPC, then select the appropriate one.filter (dstAddr like 'public IP' and srcAddr like 'xxx.xxx') or (srcAddr like 'public IP' and dstAddr like 'xxx.xxx')| limit 10Be sure to replace xxx.xxx with the first two octets of your VPC CIDR. Replace public IP with the public IP you're seeing in the VPC flow log entry. Increase the limit if more than 10 resources in your VPC have initiated traffic to the public IP.The query results show bi-directional traffic between the private instance and public IP addresses. To determine whether the private instance or external public IP address is the initiator, see the following example:2022-09-28T12:05:18.000+10:00 eni-023466675b6xxxxxx 10.0.101.222 8.8.8.8 53218 53 6(17) 4 221 1664330718 1664330746 ACCEPT OK2022-09-28T12:05:18.000+10:00 eni-023466675b6xxxxxx 8.8.8.8 10.0.101.222 53 53218 6(17) 4 216 1664330718 1664330746 ACCEPT OKIn this TCP/UDP traffic (protocol ID = 6 or 17) example, the private IP address10.0.101.222 with the source port53218 (ephemeral port) is the initiator. The IP address8.8.8.8 with destination port53 is the receiver and responder.Note: It's a best practice to turn on the TCP flag field for your VPC flow log.For example, the following entries have TCP flag field in the last column:2022-09-28T12:05:52.000+10:00 eni-023466675b6xxxxxx 10.0.1.231 8.8.8.8 50691 53 6(17) 3 4 221 1664330752 1664330776 ACCEPT OK 22022-09-28T12:05:21.000+10:00 eni-023466675b6xxxxxx 8.8.8.8 10.0.1.231 53 50691 6(17) 19 4 216 1664330721 1664330742 ACCEPT OK 18The private IP address 10.0.101.222 is the initiator with TCP flag 2, which represents a TCP SYN packet.In the following ICMP protocol example, there isn't enough information to determine which side is the initiator because there is no port information or TCP flag:2022-09-27T17:57:39.000+10:00 eni-023466675b6xxxxxx 10.0.1.231 8.8.8.8 0 0 1 17 1428 1664265459 1664265483 ACCEPT OK2022-09-27T17:57:39.000+10:00 eni-023466675b6xxxxxx 8.8.8.8 10.0.1.231 0 0 1 0 17 1428 1664265459 1664265483 ACCEPT OKThe format of your flow log entry depends on how it was created. For more information on log line formats, see View flog logs.Related informationLogging IP traffic using VPC Flow LogsSample queriesFollow"


Question: How do I resolve an "Unauthorized" error when I send emails in Amazon SES?

"I'm trying to send an email using Amazon Simple Email Service (Amazon SES), but I keep receiving an "Unauthorized" error."

Answer: "I'm trying to send an email using Amazon Simple Email Service (Amazon SES), but I keep receiving an "Unauthorized" error.Short descriptionThe "Unauthorized" error can occur due to the following reasons:You're using the incorrect AWS Identity and Access Management (IAM) user or role to send emails.The IAM identity that you're using doesn't have ses:SendEmail or ses:SendRawEmail permissions.The IAM policy or authorization policy denies your IAM identity permission for the ses:SendEmail or ses:SendRawEmail actions.The IAM permissions boundaries don't allow access to Amazon SES.AWS Organizations service control policies (SCPs) don't allow Amazon SES access.For cross-account sending: The authorization policy for the sending identity doesn't allow the IAM identity to send emails.To troubleshoot authorization errors, follow these steps:Verify that you're using the correct IAM identity to send emails.Make sure that the IAM identity has ses:SendEmail and ses:SendRawEmail permissions to send email.Check if there are any deny statements in the IAM policy or authorization policy that are blocking access.Check whether an action that's allowed in your IAM policy isn't allowed in the permissions boundary.Include all required actions in the permissions boundary using the IAM console.If you use AWS Organizations, then verify that you don't have any SCPs that explicitly deny Amazon SES actions.For cross-account sending, check that the authorization policy for the sending identity grants the required permissions to the IAM identity.ResolutionCheck that the IAM identity has ses:SendEmail and ses:SendRawEmail permissionsMake sure that your IAM identity has the correct permissions to send emails.Follow these steps:Open the IAM console.Select the IAM user or role that's used to send emails.Select the IAM identity name that you're using to send emails.In the Permissions tab of your IAM identity, expand each policy to view its JSON policy document.Search for policies that are related to Amazon SES access. Then, confirm that you have permissions for the ses:SendEmail or ses:SendRawEmail actions.The following example IAM policy allows the IAM identity to send emails:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "ses:SendEmail",        "ses:SendRawEmail"      ],      "Resource": "*"    }  ]}Note: Review SendEmail and SendRawEmail for information on what sending actions you can take with each one. If an IAM policy doesn't exist, then create an IAM policy to grant you access to send emails.Check whether there are any deny statements that block accessCheck the IAM policies for any deny statements that might be denying access to send emails.If there are deny statements, then check for conditions that block access based on the following:ses:Recipientsses:FromAddressses:FromDisplayNameses:FeedbackAddressaws:CurrentTimeaws:EpochTimeaws:SecureTransportaws:SourceIpaws:UserAgentConfirm that IAM permissions boundaries allow access to Amazon SESReview the IAM permissions boundaries that are set on the IAM identity that's trying to access Amazon SES. Confirm that the IAM permissions boundaries allow access to Amazon SES. For more information, see Delegating responsibility to others using permissions boundaries.Check if there are any AWS Organizations SCPs that don't allow Amazon SES accessIf you're using AWS Organizations, then check the SPCs for any statements that explicitly deny the ses:SendEmail and ses:SendRawEmail or any other Amazon SES action. Delete the service control policies that explicitly deny Amazon SES actions in accordance with your organization's security policies.For example, the following policy denies access to all Amazon SES actions:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Deny",      "Action": “ses:*”,      "Resource": "*"    }  ]}Check that the sending authorization policy in Amazon SES grants permissions to the delegate senderWhen you authorize other users to send emails from the identities that you own, check that the authorization policy has the correct permissions. Also, check for any explicit deny statements. Make sure that the sender uses the same Amazon SES endpoint in the AWS Region that you verified the identity in. You must remove yourself (identity owner) and the delegate sender from the sandbox to send emails to unverified addresses. To view, edit, or remove a policy, see Managing your sending authorization policies.Follow"


Question: Why doesn't updating a secret in Secrets Manager automatically update the secret in AWS CloudFormation?

"I updated my secret in AWS Secrets Manager, but the dynamic reference didn't get updated during my AWS CloudFormation stack update."

Answer: "I updated my secret in AWS Secrets Manager, but the dynamic reference didn't get updated during my AWS CloudFormation stack update.Short descriptionWhen you update a secret in Secrets Manager, CloudFormation doesn't automatically detect that the value of the secret changed. During stack updates, CloudFormation doesn't retrieve the value of the dynamic reference unless a change is made to the dynamic reference itself.After updating the secret in Secrets Manager, you must update the CloudFormation stack and also specify the VersionId in the dynamic reference string. Specifying the VersionId is a best practice to be sure that CloudFormation retrieves the updated version of the secret.ResolutionNote: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.The following example scenario demonstrates how to update your Secrets Manager dynamic reference in CloudFormation.1.    Dynamically reference the Secrets Manager secret in your CloudFormation template.For example, the following CloudFormation template dynamically references a Secrets Manager secret with the secret-id mysecret and secret-string {"MyKey":"MyValue"}:Resources:    SG:        Type: AWS::EC2::SecurityGroup        Properties:            GroupDescription: '{{resolve:secretsmanager:mysecret:SecretString:MyKey}}'            SecurityGroupIngress:                  - IpProtocol: tcp                    FromPort: 8080                    ToPort: 8080                    CidrIp: 0.0.0.0/0Note: The preceding dynamic reference resolves to MyValue for the GroupDescription property.2.    Update the secret in Secrets Manager by running the update-secret command through the AWS Command Line Interface (AWS CLI), and specify the secret-string. For the example scenario, the secret-string is {"MyKey":"NewValue"}.Note: The VersionId doesn't appear on the AWS Management Console. If you updated the secret through the AWS Management Console, then run the get-secret-value command through the AWS CLI to retrieve the VersionId.3.    Note the VersionId from the output.Example output:$ aws secretsmanager update-secret --secret-id mysecret --secret-string {\"MyKey\":\"NewValue\"}{    "ARN": "arn:aws:secretsmanager:us-east-1:xxxxxxxxxxxx:secret:mysecret-ABCDeF",    "Name": "mysecret",    "VersionId": "ab01234c-5d67-89ef-01gh-2ijk345l6m78"}Note: The VersionId "ab01234c-5d67-89ef-01gh-2ijk345l6m78" from preceding output isn't automatically applied to the dynamic reference in the CloudFormation stack.4.    Update the stack and specify the VersionId in the dynamic reference string. For example:Resources:    SG:        Type: AWS::EC2::SecurityGroup        Properties:            GroupDescription: '{{resolve:secretsmanager:mysecret:SecretString:MyKey::ab01234c-5d67-89ef-01gh-2ijk345l6m78}}'            SecurityGroupIngress:                  - IpProtocol: tcp                    FromPort: 8080                    ToPort: 8080                    CidrIp: 0.0.0.0/0Note: In the preceding example template, the :: before the VersionId "ab01234c-5d67-89ef-01gh-2ijk345l6m78" is the correct syntax. This section of the dynamic reference string is left blank because the version-stage doesn't need to be specified.Follow"


Question: How can I troubleshoot the error "MySQL HA_ERR_RECORD_FILE_FULL" when using Amazon RDS for MySQL?

I received the following error when trying to write to an Amazon Relational Database Service (Amazon RDS) MySQL DB instance:"Error_code: 1114; handler error HA_ERR_RECORD_FILE_FULL"How can I resolve this error?

Answer: "I received the following error when trying to write to an Amazon Relational Database Service (Amazon RDS) MySQL DB instance:"Error_code: 1114; handler error HA_ERR_RECORD_FILE_FULL"How can I resolve this error?Short descriptionThis error most often occurs on read replicas that cause the replication to fail, or when the table that the data is written to is full. This error can also have one of the following causes:The table that is sending the error message is a MEMORY engine table, and the table reached its maximum size.The table is a MyISAM engine table, and the table reached the maximum pointer size.The table uses the InnoDB engine, and the InnoDB tablespace file is full.The Amazon RDS DB instance has run out of disk space.The table has reached the file size limit.For more information about the storage engines that Amazon RDS supports, see supported storage engines for MySQL on Amazon RDS.ResolutionMEMORY engine table reached its maximum sizeConfirm that the MEMORY engine table has reached its maximum size by running a command similar to the following and checking the engine value:mysql> show table status from database_name like 'table_name'\GIf the value of the data_length variable in the output is greater than the value of the max_data_length variable, then the table has reached its maximum size. This is the cause of the error message.You can resolve this error by increasing the size of the max_heap_table_size parameter in the custom parameter group associated with your DB instance. For more information, see creating a DB parameter group. Be sure not to exceed the DB instance class memory limit that is based on the DB instance class that you use. For more information about the memory available for your DB instance, see Amazon RDS instance types.MyISAM engine table reached the maximum pointer sizeConfirm that the MyISAM engine table has reached the maximum pointer size by running a command similar to the following and by checking the engine value:mysql> show table status from database_name like 'table_name'\GYou can resolve this error by using the ALTER TABLE command to alter the existing table's maximum size:mysql> ALTER TABLE tbl_name MAX_ROWS=1000000000 AVG_ROW_LENGTH=nnn;Optionally, you can change the default size limit for all MyISAM tables. Then, set the myisam_data_pointer_size parameter in your custom DB parameter group to a higher value.InnoDB tablespace file is fullThe maximum tablespace size for an InnoDB table is four billion pages (16 TB).You can partition tables into multiple tablespace files for tables larger than 1 TB. You can use different types of partitioning, such as RANGE, LIST, and HASH, to partition your table based on your use case. For example, you can truncate the old data based on the year you created it, or you can create separate partitions for each year. For more information, see MySQL file size limits and the MySQL documentation for partitioning.Important: Thoroughly test the impact that partitioning might have on your application before implementing the changes in a production environment.You can convert an existing table to a partitioned table by using an alter table statement similar to the following:ALTER TABLE table_name PARTITION BY HASH(id)PARTITIONS 8;Note: Converting an existing table to a partitioned table doesn't recover space allocated for the InnoDB tablespace. To recover the InnoDB space, see how do I resolve problems with my Amazon RDS MySQL DB instance that is using more storage than expected?The Amazon RDS DB instance is out of disk spaceIf the Amazon RDS DB instance is in the STORAGE_FULL state, you receive the HA_ERR_RECORD_FILE_FULL error. To resolve this error, add more storage space to your DB instance. For more information, see Amazon RDS DB instance running out of storage.Optionally, you can monitor the available storage space for the DB instance using the Amazon CloudWatch FreeStorageSpace metric. For more information, see the overview of monitoring Amazon RDS. You can also subscribe to the low storage Amazon RDS event notification so that you're notified when your DB instance consumes more than 90% of its allocated storage. For more information, see Amazon RDS event categories and event messages.The table has reached the file size limitNote: Some existing DB instances have a lower limit. For example, MySQL DB instances created before April 2014 have a file and table size limit of 2 TB. This 2 TB file size limit also applies to DB instances or read replicas created from DB snapshots taken before April 2014, regardless of when the DB instance was created.If you have a DB instance that has a lower size limit, you can take a MySQL dump of your data by using mysqldump. Then, import the data into a new DB instance that has a higher limit.Related informationHow do I resolve problems that occur when Amazon RDS DB instances run out of storage?Follow"


Question: Can I verify my domain for Amazon SES in multiple accounts or AWS Regions?

Can I verify my domain for Amazon Simple Email Service (Amazon SES) in multiple AWS accounts or AWS Regions?

Answer: "Can I verify my domain for Amazon Simple Email Service (Amazon SES) in multiple AWS accounts or AWS Regions?Short descriptionAmazon SES now uses DomainKeys Identified Mail (DKIM) to verify and manage your domain identity. If you set up your domain through DKIM, then you can verify your identity across multiple accounts or Regions through DKIM-based verification. For domains that are verified by DKIM and managed by Amazon Route 53, Amazon SES automatically updates the DNS server with the necessary records. For more information, see Creating and verifying identities in Amazon SES.If your domain is verified by DKIM but isn't managed by Route 53, then you must configure three DKIM CNAME records to your DNS provider.Domains that are verified with TXT records must manually add accounts or Regions to the record in order to verify them. If you created your domain with a TXT record and use Route 53 as your DNS service, then follow these steps.ResolutionFollow these steps to verify your domain in multiple accounts or Regions through your domain's TXT record:1.    Verify your domain in the first account or Region.2.    Open the Route 53 console.3.    From the navigation pane, choose Hosted zones.4.    Choose the TXT record that you created when you verified your domain in the first account or Region.5.    In the Value box, go to the end of the existing attribute value, and then press Enter.6.    Add the attribute value for the additional account or Region.7.    Save the record set. Note that some DNS providers don't allow you to assign multiple values to the same TXT record. In this case, first verify the domain once with _amazonses in the TXT record's attribute name. Then, verify the domain again with _amazonses removed from the attribute name. In the following example, the domain is example.com:_amazonses.example.com    TXT    "TokenValue"example.com               TXT    "TokenValue"Warning: For this use case where you must add _amazonses to the attribute name, you can verify the same domain only two times.8.    After you update the record set, validate the record. On a Linux operating system or on macOS, you can run a dig command:$dig TXT _amazonses.example.com +shortOn a Windows operating system, you can run the nslookup command:$nslookup -type=TXT  _amazonses.example.comThe expected output for either the dig command or the nslookup command is the value of the TXT record. Along with your existing TXT records, the output should include the recently added TXT record, similar to the following:"amSXqexampleyqEPQy+jkMfF63uB9y6MG1FA/Du3x8Q=""0biRnexample5QDj43bNpT5Z+gQfdpfKqaP2tm8x4NE="9.    After the correct record is added and is resolving, Amazon SES shows the domain as verified. If the domain isn't verified within 72 hours, you can retry the verification process.Related informationDomain and email address verification problemsFollow"


Question: How do I upload WordPress files larger than 2 MB to my Elastic Beanstalk environment?

"When I try to upload WordPress files larger than 2 MB to my AWS Elastic Beanstalk environment, I receive the following error message: "The uploaded file exceeds the upload_max_filesize directive in php.ini.""

Answer: "When I try to upload WordPress files larger than 2 MB to my AWS Elastic Beanstalk environment, I receive the following error message: "The uploaded file exceeds the upload_max_filesize directive in php.ini."Short descriptionThe default PHP configuration on an Amazon Elastic Compute Cloud (Amazon EC2) instance doesn't allow you to upload WordPress files larger than 2 MB. To upload files larger than 2 MB, change the default value of the upload_max_filesize parameter in the /etc/php.ini file.Important: M and MB are equivalent expressions for "megabyte." For example, 2M is equal to 2 MB. However, use only M in your configuration file, as MB isn't valid in a configuration file.ResolutionChoose the resolution that applies to your solution stack.Update the upload file size for an Elastic Beanstalk environment running a PHP solution stackTo increase the file size across all instances, complete the following steps to update the underlying Amazon Machine Image (AMI).1.    In a code editor, create a new file named change_upload_size.config, and then add the following code to the file:files:    "/etc/php.d/99uploadsize.ini":        mode: "000644"        owner: root        group: root        content: |            upload_max_filesize = 100M            post_max_size = 100Mcommands:    remove_old_ini:        command: "rm -f /etc/php.d/99uploadsize.ini.bak"Note: In the preceding example, the value of the post_max_size parameter is updated from the default 32M to 100M.Tip: To modify php.ini parameters such as memory_limit, see the php.ini directives for post_max_size. The maximum memory_limit value in Elastic Beanstalk is 256M.2.    In the root of your application source bundle, create a new directory called .ebextensions, and then move the change_upload_size.config file to the .ebextensions  directory.If the .ebextensions directory is already inside your application source bundle, then keep the change_upload_size.config file in the .ebextensions directory with the other .config files. The directory structure should be similar to the following example:├── wp-site.zip/     ├── .ebextensions/         ├── change_upload_size.config    ├── wp-content/    ├── wp-includes/    ├── wp-admin/    ├── LICENSE    ├── README.md    ├── index.php    ├── license.txt    ├── readme.html    ├── ...    ├── ...    ├── ...    ├── ...    ├── wp-settings.php    ├── wp-signup.php    ├── wp-trackback.php    └── xmlrpc.phpImportant: For Amazon Linux 2 platforms based on Elastic Beanstalk, change the file name 99uploadsize.ini to z99uploadsize.ini everywhere in the .ebextension to allow the new PHP settings to override the defaults.3.    Open the Elastic Beanstalk console, and then create a zip archive of your application source bundle.4.    To upload your code to an existing Elastic Beanstalk environment, choose your environment, and then choose Upload and Deploy.Note: To deploy your code in a new Elastic Beanstalk environment, see Creating an Elastic Beanstalk environment.If the error message doesn't go away after you upload the .ebextensions file, then use a YAML validator to check for white space errors in your code. Also, consider using the remove_old_ini command to remove unnecessary backup files. For more information, see Extending php.ini.Change the upload file size for a single Amazon EC2 instance running a LAMP stack1.    To open the /etc/php.ini file with the vi editor, run the following command:$ sudo vi /etc/php.iniNote: The /etc/php.ini file name or path can vary depending on your version of PHP or the AMI that you're using.2.    In the vi editor, change the value of the upload_max_filesize parameter to a larger size than 2 MB, and then save the file.Tip: To modify php.ini parameters such as memory_limit, see the php.ini directives for post_max_size. For more information, see the post_max_size entry on the PHP website.3.    To restart the Apache server gracefully, run the appropriate command for your environment.For Amazon Linux:$ sudo service httpd gracefulFor Amazon Linux 2:$ sudo service httpd reloadNote: The /etc/php.ini file name or path can vary depending on your version of PHP or the AMI that you're using. The command to restart the Apache server can also vary if you're using a distribution other than Amazon Linux AMI. WordPress limits the memory of its PHP processes by default.Related informationAdvanced environment customization with configuration files (.ebextensions)Deploying a high-availability WordPress website with an external Amazon RDS database to Elastic BeanstalkTutorial: Hosting a WordPress blog with Amazon LinuxDescription of core php.ini directivesFollow"


Question: How do I troubleshoot Amazon EC2 instance connection timeout errors from the internet?

"I can't connect to an Amazon Elastic Compute Cloud (Amazon EC2) instance in my Amazon Virtual Private Cloud (Amazon VPC) from the internet. When I try to connect, the connection hangs, and then I receive an error similar to "Network error: Connection timed out". How can I fix this?"

Answer: "I can't connect to an Amazon Elastic Compute Cloud (Amazon EC2) instance in my Amazon Virtual Private Cloud (Amazon VPC) from the internet. When I try to connect, the connection hangs, and then I receive an error similar to "Network error: Connection timed out". How can I fix this?ResolutionNote: For "Permission Denied" or "Connection Refused" errors, see I'm receiving "Connection refused" or "Connection timed out" errors when trying to connect to my EC2 instance with SSH. How do I resolve this?Verify that your security settings allow appropriate accessReview the following security settings for Amazon EC2 instances in your VPC:Verify that your instance has an associated public IP address or Elastic IP address. Be sure to use this IP address when connecting to the instance.Note: For more troubleshooting guidance, see How can I fix the connection to my Amazon EC2 instance or elastic network interface that has an attached Elastic IP address?Add a rule to your security groups to allow access to your instance from your IP address using SSH.Verify that your instance is passing system and instance status checks.Verify that your network ACLs allow access to your instanceVerify that network ACLs allow access to your instance over SSH from your IP address as follows. For an example configuration, see Example: Control access to instances in a subnet.First, find the Subnet ID for your instance:Open the Amazon EC2 console.In the navigation pane, under Instances, choose Instances.Select your instance.Choose Details.Note the Subnet ID.Then, review the inbound and outbound rules for the network ACL:Open the Amazon VPC console.In the navigation pane, under Virtual Private Cloud, choose Subnets.In the content pane, select the Subnet ID that you previously noted.Choose the Network ACL tab.Important: If you have more than one subnet associated with your instance, then complete steps 5 and 6 for each subnet.Check if the inbound rules differ from the default network ACL configuration. If the rules differ, then add a rule to allow inbound traffic for SSH to and from your IP address. For an example configuration for SSH, see Custom network ACL.Check if the outbound rules differ from the default network ACL configuration. If the rules differ, then add a rule to allow outbound traffic for SSH to and from your IP address.Verify that your VPC route table allows traffic to and from the internetFirst, find the Subnet ID for your instance:Open the Amazon EC2 console.In the navigation pane, under Instances, choose Instances.Select your instance.Choose Details.Note the Subnet ID.Then, verify that your VPC route table allows traffic to and from the internet:Open the Amazon VPC console.In the navigation pane, under Virtual Private Cloud, choose Subnets.In the content pane, select the Subnet ID that you previously noted.Choose the Routes table tab.Verify that you have a default route (a route whose destination is 0.0.0.0/0) pointing to your internet gateway. If there's no default route to your internet gateway, then choose Internet Gateways under Virtual Private Cloud from the navigation pane.Select your VPC's internet gateway.Note: You can find the VPC ID on the Details page of the instance.In the Details view, note the ID value of the internet gateway.Add a new route with a Destination of 0.0.0.0/0 and a Target of your internet gateway ID. Be sure to save your new route table configuration.Note: If connecting from a bastion host, the VPC route table can't have a default route (0.0.0.0/0) to an internet gateway because it's a private subnet. For more information, see How do I troubleshoot accessing my EC2 instance using an SSH connection through a bastion host?Check for conflicts with your local firewalls and routing tablesIf you continue to experience connection problems, check for conflicts with your local firewall rules or local routing tables.Related informationWhy can't my Amazon EC2 instance connect to the internet using an internet gateway?How do I troubleshoot problems connecting to my Amazon EC2 Linux instance using SSH?Why can't my EC2 instance in a private subnet connect to the internet using a NAT gateway?Why can't I connect to an Amazon EC2 instance within my Amazon VPC from the internet?Get started with Amazon VPCFollow"


Question: How do I set up IPAM with accounts outside my Organizations?

I recently integrated with another AWS Organizations. I want to integrate the IP Address Manager (IPAM) of my organization with AWS accounts in the other Organizations.

Answer: "I recently integrated with another AWS Organizations. I want to integrate the IP Address Manager (IPAM) of my organization with AWS accounts in the other Organizations.ResolutionNote: Review the considerations and limitations with integrating the Amazon Virtual Private Cloud (Amazon VPC) IPAM of your organization with accounts that are external to your AWS Organizations.PrerequisiteNote: If you already integrated your IPAM with your organization, then proceed to the next section.Create an IPAM in the delegated admin account of organization A. This account becomes the administrative account for IPAM service functionality for accounts in both organizations. For more information on how to delegate an IPAM account, see Integrate IPAM with accounts in an AWS Organizations.Identify the AWS accounts for integrationAt a minimum, you must have two AWS Organizations for integration. Each has its own IPAM administrator account.The Organizations owner account for organization A.The IPAM admin account for organization A.The Organizations owner account for organization B.The IPAM admin account for organization B.Integrate your IPAM with the accounts in the new organizationThis example integrates the IPAM for organization A with the accounts in organization B. After integration, organization A's IPAM admin account provides IPAM service functionality for accounts in both organizations.Register the organization B IPAM admin accountComplete the following steps in organization B's management account:Open the IPAM console.Register the organization B admin account as the IPAM delegated administrator. For more information, see Integrate IPAM with accounts in an AWS Organizations.Set up the resource discovery in organization BComplete the following steps in organization B's IPAM admin account. For more information on resource discoveries, see Work with resource discoveries.Open the IPAM console.Create a resource discovery.Use the Resource Access Manager to share the resource discovery with organization A's IPAM admin account.Note: The resource discovery must be in the same AWS Region as the IPAM in organization A. In most cases, the resource discovery must also have the same operating Regions as the IPAM. However, to have more control over the monitoring of accounts in organization B, select a subset of the IPAM’s operating Regions.Accept the resource share in organization AComplete the remaining steps in organization A's IPAM admin account.Open the IPAM console.Accept the resource share invitation in organization A’s IPAM admin account.Associate the IPAM with the resource discovery.Note: Repeat the preceding steps to integrate accounts from another AWS Organizations into your IPAM setup.Manage IPAM resourcesOrganization A’s IPAM account can now monitor and manage IPAM resources that accounts in organization B owns. For more information, see Managing IP address space in IPAM.You can now share IPAM pools directly with member accounts in organization B. For more information, see Share an IPAM pool using AWS RAM.Stop monitoring accounts in organization BTo stop the IPAM in organization A from monitoring accounts in organization B, take any of the following actions:Disassociate the resource discovery from the IPAM in organization A's admin account.Update a resource share in AWS RAM in organization B's IPAM admin account to stop sharing the resource discovery.Delete the resource directory in organization B's IPAM admin account.Follow"


Question: "What is the AWS Free Tier, and how do I use it?"

"I want to test AWS using resources covered under the AWS Free Tier. I want to know how the AWS Free Tier works, and what I should keep in mind when using it."

Answer: "I want to test AWS using resources covered under the AWS Free Tier. I want to know how the AWS Free Tier works, and what I should keep in mind when using it.Short descriptionThe AWS Free Tier is automatically activated on each new AWS account. With the AWS Free Tier, you can try out some AWS services free of charge up to a specific maximum amount of usage each month.Important: Not all AWS services are free. Some services launched under the Free Tier have usage limits. If you exceed the usage limits, then you're charged at standard rates.Check the AWS Free Tier offerings to be sure that the services you intend to use are covered and the applicable usage limits.ResolutionKeep the following in mind when using the AWS Free Tier:Free Tier includes three different types of offeringsTrials: These are short-term trial offers that start from the date that you activate a service. You pay standard rates after the trial period expires. For more information on how different services are charged, see AWS Pricing.12-Months free: These offers provide limited usage for 12 months after your initial sign-up date. You pay standard rates after your 12 months free usage term expires or if your application use exceeds the free tier limits.Always free: These offers are available to all AWS customers and don't expire at the end of your 12-month AWS Free Tier term.Not all free usage is part of the AWS Free TierSome services offer free trials outside of AWS Free Tier. Other services offer some level of service for free, whether or not your account is covered under the AWS Free Tier.Check the service's pricing documentation to find out if that service has a free trial option.Your access to services is not restricted by the AWS Free TierYou are responsible for the services that you launch. You might incur charges for any services and usage that are not covered by the Free Tier. For information on how much your resources will cost after the Free Tier limits are exceeded, see How do I estimate the cost of my planned AWS resource configurations?Avoid unnecessary chargesTo avoid unnecessary charges, see How do I make sure I don't incur charges when I'm using the AWS Free Tier?Related informationAWS Free Tier FAQsEligibility for the AWS Free TierI unintentionally incurred charges while using the AWS Free Tier. How do I make sure that I'm not billed again?Follow"


Question: How do I move my provisioned Amazon Redshift cluster into a different subnet?

I'm trying to move my Amazon Redshift cluster into a different subnet. How can I do that?

Answer: "I'm trying to move my Amazon Redshift cluster into a different subnet. How can I do that?Short descriptionWhen a new Amazon Redshift cluster is launched, an Amazon Virtual Private Cloud (Amazon VPC), security group, and cluster subnet group are provided. During the launch, Amazon Redshift picks up random subnets from the associated cluster subnet group.You have two options for moving your existing provisioned Amazon Redshift cluster into a different subnet: You can either use an existing subnet group, or you can create a new subnet group that contains the subnet where you will move the provisioned Amazon Redshift cluster. After you have a subnet group with the preferred subnet, you must restore your cluster from a snapshot. Be sure that there is only one subnet in the subnet group. Otherwise, Amazon Redshift might not launch the cluster in that specific subnet.ResolutionTo move your Amazon Redshift cluster into a different subnet group, perform these steps:1.    Create a manual snapshot of the cluster.2.    Restore the cluster using the snapshot that you just created.3.    Under Additional configurations, turn off the Use defaults settings.4.    Choose Network and security.5.    Select the desired subnet group.6.    Choose Restore cluster from snapshot.For more information, see How do I move my Amazon Redshift cluster from one VPC to another VPC?To confirm the subnet and subnet group that your Amazon Redshift cluster is using, perform these steps:1.    Open the Amazon Redshift console.2.    Choose the Clusters tab.3.    Select your Amazon Redshift cluster.4.    Choose the Properties tab.5.    Navigate to Network and security to view network details. The Network and security information shows you the subnet group that's being used.6.    Under the Properties tab, navigate to Node IP addresses. This section has information that can help you verify the subnet that's being used. The subnet listed must be from your subnet group.Subnet best practicesWhen you move your Amazon Redshift cluster into a different subnet, consider the following best practices:Configure your cluster subnet group so that the subnets reside in a homogenous network. In a homogenous network configuration, all the subnets that belong to that group share similar networking rules, performing as a uniform network. A homogenous network configuration also minimizes any connectivity issues that can occur when moving your Amazon Redshift cluster into a different subnet.Be sure that there are enough IP addresses present in each of your subnets within the subnet group. Otherwise, Amazon Redshift provisions your cluster into two or more subnets within the same Availability Zone (AZ). If there are no available subnets within the same Availability Zone, or there aren't enough IP addresses present, your cluster launch might fail. To resolve a launch failure, delete any unused elastic network interfaces. Or, you can move your AWS resources into a different subnet by choosing a different subnet group.Follow"


Question: How do I use EC2Rescue for Linux to troubleshoot operating system-level issues?

"I can't connect to my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance or I'm experiencing boot issues. To correct these problems, I need to fix common issues such as OpenSSH file permissions or gather system (OS) logs for analysis and troubleshooting. How can I use EC2Rescue for Linux to do this?"

Answer: "I can't connect to my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance or I'm experiencing boot issues. To correct these problems, I need to fix common issues such as OpenSSH file permissions or gather system (OS) logs for analysis and troubleshooting. How can I use EC2Rescue for Linux to do this?Short descriptionEC2Rescue for Linux is a tool that helps diagnose and troubleshoot problems on Amazon EC2 Linux instances. EC2Rescue for Linux is run on your Amazon EC2 Linux instance to correct operating system-level issues. EC2Rescue for Linux also collects advanced logs, system utilization reports, and configuration files for further analysis.Common scenarios addressed by EC2Rescue for Linux:Collect system utilization reports such as vmstat, iostat, mpstat, and so on.Collect logs and details such as syslog, dmesg, application error logs, and SSM logs.Detect system problems such as asymmetric routing or duplicate root device labels.Automatically remediate system problems such as correcting OpenSSH file permissions or disabling known problematic kernel parameters.System RequirementsEC2Rescue for Linux requires an Amazon EC2 Linux instance that meets the following  prerequisites:Supported operating systemsAmazon Linux 2Amazon Linux 2016.09+SLES 12+RHEL 7+Ubuntu 16.04+Software requirementsPython 2.7.9+ or 3.2+Note: If you’ve enabled EC2 Serial Console for Linux, then you can use it to troubleshoot supported Nitro-based instance types. The serial console helps you troubleshoot boot issues, network configuration, and SSH configuration issues. The serial console connects to your instance without the need for a working network connection. You can access the serial console using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).Before using the serial console, grant access to the console at the account level. Then create AWS Identity and Access Management (IAM) policies granting access to your IAM users. Also, every instance using the serial console must include at least one password-based user. If your instance is unreachable and you haven’t configured access to the serial console, follow the instructions in the Resolution section. For information on configuring the EC2 Serial Console for Linux, see Configure access to the EC2 Serial Console.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.ResolutionTo troubleshoot an unreachable Amazon EC2 Linux instance using EC2Rescue for Linux, do the following:1.    Launch a new Amazon EC2 instance in your virtual private cloud (VPC) using the same Amazon Machine Image (AMI) and in the same Availability Zone as the impaired instance. The new instance becomes your "rescue" instance. Or, you can use an existing instance that you can access, if it uses the same AMI and is in the same Availability Zone as your impaired instance.2.    Detach the Amazon Elastic Block Store (Amazon EBS) root volume (/dev/xvda or /dev/sda1) from your impaired instance. Make a note of the device name to be sure it is the same when you re-attach it later3.    Attach the EBS volume as a secondary device (/dev/sdf) to the rescue instance.4.    Connect to your rescue instance using SSH.5.    Become root, identify the correct device name using lsblk, then save this for use throughout the process:$ sudo -i# lsblk# rescuedev=/dev/xvdf1Note: The device (/dev/xvdf1) might be attached to the rescue instance with a different device name. Use the lsblk command to view your available disk devices along with their mount points to determine the correct device names.6.    Select an appropriate temporary mountpoint to use, and ensure it exists, use /mnt unless this is already in use:.# rescuemnt=/mnt# mkdir -p $rescuemnt7.    Mount the root file system from the attached volume:# mount $rescuedev $rescuemntNote: If the volume mount fails, check dmesg | tail. If the logs suggest conflicting UUID, use the option -o nouuid.8.    Mount special file systems and change the root directory (chroot) to the newly mounted file system:# for i in proc sys dev run; do mount --bind /$i $rescuemnt/$i ; done# chroot $rescuemnt9.     Download and install the EC2Rescue Tool for Linux on an offline Linux root volume:# curl -O https://s3.amazonaws.com/ec2rescuelinux/ec2rl.tgz# tar -xf ec2rl.tgz10.    Verify the installation by listing the help file:# cd ec2rl-<version_number># ./ec2rl help11.    Run EC2Rescue for Linux with no options to run all modules:# ./ec2rl run12.    View the results in /var/tmp/ec2rl:# cat /var/tmp/ec2rl/*/Main.log | more13.    Enable remediation for the supported modules based on the results:# ./ec2rl run --remediate14.    After remediation is complete, exit from chroot and unmount the secondary device:# exit# umount $rescuemnt/{proc,sys,dev,run,}Note: If the unmount operation isn't successful, you might have to stop or reboot the rescue instance to enable a clean unmount.15.    Detach the secondary volume (/dev/sdf) from the rescue EC2 instance, and then attach it to the original instance as /dev/xvda or /dev/sda1 (root volume). Ensure this is the same as seen in step 2.16.    Start the EC2 instance and then verify that the instance is responsive.Note: You can also use an AWS Systems Manager Automation document to troubleshoot connection issues. For more information, see Walkthrough: Run the EC2Rescue tool on unreachable instances. The AWSSupport-ExecuteEC2Rescue document is designed to automate steps normally required to use EC2Rescue for Linux. These steps are a combination of Systems Manager actions, AWS CloudFormation actions, and AWS Lambda functions.Additional troubleshootingFor general instructions on recovering a Linux instance, see Instance recovery when a host computer fails. For Windows instances, see Troubleshoot an unreachable instance.If your instance's root device is an Amazon EBS-backed volume, then try stopping and then starting the instance. For more information, see Stop and start your instance.For instance-store backed instances, if you created a custom AMI of the instance, you might be able to restore your instance using the AMI as a backup. For instructions on creating a new instance from an AMI you own, see Launch an instance using the old launch instance wizard.In some cases, your EBS volume might have I/O access disabled, which can render your instance inaccessible. For instructions on how to identify and troubleshoot this, see Work with the Auto-Enabled IO volume attribute.If you've lost the SSH key pair, you can reset it using Systems Manager Automation and AWSSupport-ResetAccess document.Related informationRecover your impaired instances using EC2Rescue and Amazon EC2 Systems Manager AutomationFollow"


Question: How do I monitor AWS VPN tunnels using Amazon CloudWatch alarms?

"I want to configure Amazon CloudWatch alarms to monitor my AWS VPN tunnels, and then receive notifications when my AWS VPN connection is  down."

Answer: "I want to configure Amazon CloudWatch alarms to monitor my AWS VPN tunnels, and then receive notifications when my AWS VPN connection is  down.Short descriptionMonitor the status of an AWS VPN through its metric data, which is automatically sent to CloudWatch. The VPN tunnel state is reported as a Boolean value in the CloudWatch metric TunnelState. A value of 0 indicates that the tunnel is down and 1 indicates that the tunnel is up. Set up a CloudWatch alarm based on this CloudWatch metric to notify you when one or both VPN tunnels are down.ResolutionSet up your AWS VPN and confirm that both tunnels are available. Then, configure a CloudWatch alarm to monitor the connection.Set up a topic for your alarm in Amazon Simple Notification Service (Amazon SNS). If you don't have an existing topic, then create an Amazon SNS topic. Then, subscribe to the topic.Create a CloudWatch alarm to monitor the VPN tunnel state.(Optional) Test your alarm by making the tunnels of your AWS VPN unavailable to invoke a notification from your SNS topic.Related informationMonitoring VPN tunnels using Amazon CloudWatchHow do I check the current status of my VPN tunnel?Follow"


Question: How do I troubleshoot my Amazon MWAA environment that's stuck in the "Creating" state?

"I tried to create an Amazon Managed Workflows for Apache Airflow (Amazon MWAA) environment, but it's stuck in the "Creating" state."

Answer: "I tried to create an Amazon Managed Workflows for Apache Airflow (Amazon MWAA) environment, but it's stuck in the "Creating" state.ResolutionRun a troubleshooting script to verify that the prerequisites for the Amazon MWAA environment, such as the required AWS Identity and Access Management (IAM) role permissions and Amazon Virtual Private Cloud (Amazon VPC) setup are met. For more information, see the Verify environment script in AWS Support Tools on GitHub.If your Amazon MWAA environment is stuck in the "Creating" state for a shorter duration, then the issue might be due to the missing IAM permissions for other AWS services, such as the following: Amazon Simple Storage Service (Amazon S3), Amazon CloudWatch, Amazon Simple Queue Service (Amazon SQS) and Amazon Elastic Container Registry (Amazon ECR), and AWS Key Management Service (AWS KMS). Be sure that your execution role and service-linked role has the required permissions. If you're using a customer managed key, be sure to update the customer managed key policy as well. For troubleshooting steps, see I tried to create an environment but it shows the status as "Create failed".If your environment is stuck for more than 30 minutes in the "Creating" state, then the issue might be related to the networking configuration. The root cause of the issue and the appropriate resolution depend on your networking setup.Your network configuration lacks the route to AWS services or the internetTo resolve this issue, based on the type of routing you choose, verify that the network configuration meets the respective prerequisites for the environment:Public routing: Be sure that your Amazon VPC infrastructure has two public and two private subnets. Public subnets get public IP addresses and have the default route to internet gateway. Private subnets get only private IP addresses and have no route to the internet gateway. Instead, they have a route to the NAT gateway. For more information, see Public routing over the internet. Typically, the network flow with public routing looks similar to the following:Private subnet - Default route to NAT gateway - NAT gateway associated with the public subnet - public subnet - default route to the internet gateway - internetPrivate routing: Your Amazon VPC without internet access needs additional VPC service endpoints to use the Apache Airflow on MWAA. These Amazon VPC endpoints include Amazon S3, monitoring, ecr.dkr, ecr.api, logs, sqs, kms, airflow.api, airflow.env, and airflow.ops. For more information, see Creating the required VPC service endpoints in an Amazon VPC with private routing and Private routing without internet access. Be sure that the VPC endpoints have private DNS enabled. Verify that the endpoints are associated with the environment's subnets and security group. Also, be sure that the VPC endpoint policy for each endpoint is configured to allow full access to the endpoint.The security group or network access control list (ACL) restricts the network trafficTo resolve this issue, verify that the security group specifies a self-referencing inbound rule to itself or the port range HTTPS 443 and TCP 5432. The security group must specify an outbound rule for all traffic. The network ACL must have an inbound or outbound rule that allows all traffic. For an example, see Example ACLs.Downloading the container image from Amazon ECR has failedIf you use an Amazon VPC without internet access, then be sure that you created an Amazon S3 gateway endpoint and granted the minimum required permissions to Amazon ECR to access Amazon S3 in that Region.For troubleshooting issues related to the Amazon VPC network with public/private routing, see I tried to create an environment and it's stuck in the "Creating" state.Related informationAbout networking on Amazon MWAAFollow"


Question: Why did I get a read only error after an Amazon Aurora DB cluster failed over?

I'm using an Amazon Aurora DB cluster and I am receiving the following error after a failover:"The MySQL server is running with the --read-only option so it cannot execute this statement"How can I resolve this error?

Answer: "I'm using an Amazon Aurora DB cluster and I am receiving the following error after a failover:"The MySQL server is running with the --read-only option so it cannot execute this statement"How can I resolve this error?Short descriptionWhen an Amazon Aurora DB cluster experiences a Multi-AZ failover, the cluster endpoints update automatically to reflect and point to the newly appointed roles of the Writer and Reader. The old Writer is rebooted, and then set into a read-only mode while an existing replica is promoted to become a Writer.If you receive a read-only error message, this means that you are trying to perform a data definition language (DDL), data manipulation language (DML), or data control language (DCL) operation through an existing node that has the role of a Reader. In Aurora MySQL, you can confirm this by checking the innodb_read_only variable, as follows:mysql> show variables where variable_name='innodb_read_only';+------------------+-------+| Variable_name    | Value |+------------------+-------+| innodb_read_only | ON    |+------------------+-------+1 row in set (0.01 sec)ResolutionUtilize the Cluster Writer endpointBecause the role of a DB instance in an Aurora cluster can change, it is a best practice to use the cluster Writer endpoint to make sure that you are always pointing to the latest Writer. If you use the DB instance endpoint or a direct IP Address, you might not be aware that a failover has occurred, and will get a read-only message if you re-connect to the same host. This will stop you from performing any DDL/DML changes, as intended.Don't excessively cache DNSIf you are not using a smart driver, then you depend on the DNS record updates and propagation after a failover event occurs. Aurora DNS zones use a short time-to-live (TTL) of 5 seconds, so it is important that your network and client configurations don't further increase this. DNS caching can occur at multiple layers of an architecture, such as the operating system (OS), the network layer and the application container. It is important that you understand how each of these layers is configured. If there is unintended DNS caching beyond the TTL of 5 seconds, it is possible that you will re-connect to the old writer after a failover.Java Virtual Machines (JVM) can excessively cache DNS, indefinitely. When the JVM resolves a hostname to an IP address, it caches the IP address for a specified period of time (TTL). On some configurations, the JVM default TTL is set to never refresh DNS entries until the JVM is restarted. This can lead to read-only errors after a failover. In this case, it is important to manually set a small TTL so that it will periodically refresh.Use a Smart DriverThe Amazon Aurora DB cluster endpoints propagate DNS record updates automatically, but the process doesn't happen instantly. This can cause delays in responding to an event that occurred on the database and the event might be handled by the application. A Smart Driver uses the DB cluster topography through the INFORMATION_SCHEMA.REPLICA_HOST_STATUS metadata table, which is in near-real-time. This helps to route connections to the appropriate role, and helps load-balance across the existing replicas. MariaDB Connector/J is an example of a third party Smart Driver that has native support for Aurora MySQL.Note: Even Smart Drivers might be affected by excessive DNS Caching.Test which instance you are connected toAs mentioned in the best practices of the Aurora connection management handbook, when not using a smart driver, you should test and understand which instance that you are logged into after establishing a new connection. This can help you make sure that you are connected to the correct instance. You can test whether you are connected to the writer instance or an Aurora Reader using the @@innodb_read_only variable. This example shows a value of 0 which means you are connected to the writer.mysql> select @@innodb_read_only; +--------------------+ | @@innodb_read_only | +--------------------+ | 0                  | +--------------------+ 1 row in set (0.00 sec)Related informationAmazon Aurora connection managementSetting the JVM TTL for DNS name lookupsDBAs connection management handbookFollow"


Question: How do I troubleshoot and prevent Amazon SQS queue access issues when using a Deny queue policy?

"I've lost access to my Amazon Simple Queue Service (Amazon SQS) queue because of a Deny queue policy. How can I regain access to my Amazon SQS queue, and how can I prevent losing access again?"

Answer: "I've lost access to my Amazon Simple Queue Service (Amazon SQS) queue because of a Deny queue policy. How can I regain access to my Amazon SQS queue, and how can I prevent losing access again?Short descriptionIf you lose access to an Amazon SQS queue because of a Deny queue policy, try to access the queue using AWS account root user credentials. Or, try to access the queue using entities excluded from the policy, if applicable.This article also presents best practices to follow so that you can prevent Amazon SQS queue access loss when using Deny queue policies.The following example Deny policy denies all AWS Identity and Access Management (IAM) entities access to all Amazon SQS queue actions:{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "deny-sqs-actions",      "Effect": "Deny",      "Principal": "*",      "Action": "SQS:*",      "Resource": "queueName"    }  ]}ResolutionTroubleshoot Amazon SQS queue access issuesIf you lose access to your Amazon SQS queue while using a Deny policy in the Amazon SQS Access Policy, try the following steps:Use your root user credentials to access the queue. Account owner root user credentials allow full access to all resources in the account. This includes removing a Deny All resource policy associated to the Amazon SQS queue.If the Deny policy is restricted to specific entities, then try to gain access with the entities that are excluded from the policy.Note: Verify that the excluded entities have the permissions required to access the queue.If you can't use root user credentials, or if you're not sure which entities are excluded from the Deny policy, then contact AWS Support. Create a support case and include the following information:Confirm that you attempted to access the Amazon SQS queue using root user credentials. Include the reason why you can't use root user credentials, or why using root user credentials didn't resolve the issue.Confirm that you are the queue owner.Provide a detailed reason as to why queue access is needed.Avoid losing access to your Amazon SQS queueTo prevent losing access to your Amazon SQS queue, follow these best practices when using a Deny policy in the Queue Access Policy.Don't explicitly deny the root user account, as you can lose access to the queue entirely.Use an Allow policy alongside the Deny policy to be sure that there is an entity that can still gain access to the queue.Note: A Deny policy can override an Allow Policy unless the Allow policy doesn't match the condition of the Deny policy. For more information, see Authorization.Example policy including Allow and Deny statements:{   "Version": "2012-10-17",   "Id": "Queue1_Policy_UUID",   "Statement": [{      "Sid":"Queue1_Allow_Access",      "Effect": "Allow",      "Principal": {"AWS": "arn:aws:iam::111122223333:user/User1"},      "Action": “sqs:*",      "Resource": "queueName"   }, {      "Sid":"Queue1_Deny_Access",      "Effect": "Deny",      “NotPrincipal": {"AWS": "arn:aws:iam::111122223333:user/User1"},      "Action": "sqs:*",      "Resource": "queueName"   }]}This example policy achieves the following:Allows a specific IAM user (User1) access to all Amazon SQS actions on the specified Amazon SQS queue.Denies queue access to all principals except the specified IAM user (User1)."NotPrincipal" in the Deny policy excludes the specified principal. However, the Allow statement is also needed to grant access permissions to the excluded principal.Related informationOverview of managing access in Amazon SQSFollow"


Question: How can I use AWS Glue workflows to automatically start a job when a crawler run completes?

I want to use AWS Glue workflows to automatically start a job when a crawler run completes.

Answer: "I want to use AWS Glue workflows to automatically start a job when a crawler run completes.Short descriptionTo start a job when a crawler run completes, create an AWS Glue workflow and two triggers: one for the crawler and one for the job. This method requires you to start the crawler from the Workflows page on the AWS Glue console.Note: You can also use an AWS Lambda function and an Amazon EventBridge rule to automate job runs. When you choose this option, the Lambda function is always on. The function monitors the crawler regardless of where or when you start the function. For more information, see How can I use a Lambda function to automatically start an AWS Glue job when a crawler run completes?ResolutionBefore completing the following steps, be sure that you have:An AWS Glue extract, transform, and load (ETL) jobAn AWS Glue crawlerAn AWS Identity and Access Management (IAM) role for AWS Glue that has the AWSGlueServiceRole policy attached to itCreate the workflowOpen the AWS Glue console.In the navigation pane, choose Workflows, and then choose Add workflow.Enter a name for the workflow, and then choose Add workflow. The new workflow appears in the list on the Workflows page.Create the trigger for the crawlerOn the Workflows page, select your new workflow, and then choose the Graph tab.Choose Add trigger, and then choose the Add new tab. For Trigger type, choose On demand.Choose Add. The trigger appears on the graph.On the graph, choose Add node.On the Crawlers tab, select your crawler, and then choose Add.Create the trigger for the AWS Glue jobOn the Action menu above the graph, choose Add trigger.Choose the Add new tab, and then select the following options: For Trigger type, choose Event. For Trigger logic, choose Start after ALL watched event.Choose Add. The trigger appears on the graph.On the graph, to the left of the job trigger that you just created, choose Add node.On the Crawlers tab, select your crawler, and then choose Add. The trigger appears on the graph.On the graph, to the right of the job trigger that you just created, choose Add node.On the Jobs tab, select the job that you want to start when the crawler run completes, and then choose Add.Test the workflowOn the Actions menu, next to the Add workflow button, choose Run. The Last run status column changes to Running.Check the Graph tab to see the status of the workflow. Or, open your corresponding crawler or job to confirm that it's running.Related informationCreating and building out a workflow manually in AWS GlueFollow"


Question: "How can I check, scale, delete, or drain my worker nodes in Amazon EKS?"

"I launched my Amazon Elastic Kubernetes Service (Amazon EKS) worker nodes using eksctl or the AWS Management Console. Now I want to check, scale, drain, or delete my worker nodes."

Answer: "I launched my Amazon Elastic Kubernetes Service (Amazon EKS) worker nodes using eksctl or the AWS Management Console. Now I want to check, scale, drain, or delete my worker nodes.Short descriptionComplete the steps in the appropriate section based on your needs:Check your worker nodesScale your worker nodesDrain your worker nodesDelete your worker nodesResolutionCheck your worker nodesTo list the worker nodes registered to the Amazon EKS control plane, run the following command:kubectl get nodes -o wideThe output returns the name, Kubernetes version, operating system, and IP address of the worker nodes.To get additional information on a single worker node, run the following command:kubectl describe node/node_nameNote: Replace node_name with your value. For example: ip-XX-XX-XX-XX.us-east-1.compute.internalThe output shows more information about the worker node, including labels, taints, system information, and status.Scale your worker nodesNote: If your node groups appear in the Amazon EKS console, then use a managed node group. Otherwise, use an unmanaged node group.(Option 1) To scale your managed or unmanaged worker nodes using eksctl, run the following command:eksctl scale nodegroup --cluster=clusterName --nodes=desiredCount --name=nodegroupNameNote: Replace clusterName, desiredCount, and nodegroupName with your values.-or-(Option 2) To scale your managed worker nodes without eksctl, complete the steps in the "To edit a node group configuration" section of Updating a managed node group.-or-(Option 3) To scale your unmanaged worker nodes using AWS CloudFormation, complete the following steps:1.    Use a CloudFormation template to launch your worker nodes for Windows or Linux.2.    Modify the NodeAutoScalingGroupDesiredCapacity, NodeAutoScalingGroupMinSize, or NodeAutoScalingGroupMaxSize parameters in your CloudFormation stack.Drain your worker nodesImportant: The drain action isolates the worker node and tells Kubernetes to stop scheduling any new pods on the node. Pods running on the target node are evicted from draining nodes, which means the pods will be stopped. Consider the effect this can have on your production environment.You can drain either an entire node group or a single worker node. Choose the appropriate option.(Option 1) Drain the entire node group:If you're using eksctl to launch your worker nodes, then run the following command:eksctl drain nodegroup --cluster=clusterName --name=nodegroupNameNote: Replace clusterName and nodegroupName with your values.To uncordon the node group, run the following command:eksctl drain nodegroup --cluster=clusterName --name=nodegroupName --undoNote: Replace clusterName and nodegroupName with your values.If you're not using eksctl to launch your worker nodes, then identify and drain all the nodes of a particular Kubernetes version. For example:#!/bin/bashK8S_VERSION=1.18.8-eks-7c9bdanodes=$(kubectl get nodes -o jsonpath="{.items[?(@.status.nodeInfo.kubeletVersion==\"v$K8S_VERSION\")].metadata.name}")for node in ${nodes[@]}do    echo "Draining $node"    kubectl drain $node --ignore-daemonsets --delete-local-datadoneTo identify and uncordon all the nodes of a particular Kubernetes version, use the following code:#!/bin/bashK8S_VERSION=1.18.8-eks-7c9bdanodes=$(kubectl get nodes -o jsonpath="{.items[?(@.status.nodeInfo.kubeletVersion==\"v$K8S_VERSION\")].metadata.name}")for node in ${nodes[@]}do    echo "Uncordon $node"    kubectl uncordon $nodedoneNote: To get the version of your worker node, run the following command:$ kubectl get nodesNAME                                      STATUS   ROLES    AGE     VERSIONip-XXX-XXX-XX-XXX.ec2.internal            Ready    <none>   6d4h    v1.18.8-eks-7c9bdaip-XXX-XXX-XX-XXX.ec2.internal            Ready    <none>   6d4h    v1.18.8-eks-7c9bdaNote: The version number appears in the VERSION column.(Option 2) Drain a single worker node:If you're not using eksctl to launch your worker nodes or you want to drain only a specific node, then gracefully isolate your worker node:kubectl drain node_name --ignore-daemonsetsNote: Replace node_name with your value.To undo the isolation, run the following command:kubectl uncordon node_nameNote: Replace node_name with your value.To migrate your existing applications to a new worker node group, see Migrating to a new node group.Delete your worker nodesImportant: The delete action is unrecoverable. Consider the impact this can have on your production environment.If you're using eksctl, then run the following command:eksctl delete nodegroup --cluster=clusterName --name=nodegroupNameIf you have a managed node group, then complete the steps in Deleting a managed node group.If you have an unmanaged node group and you launched your worker nodes with a CloudFormation template, then delete the CloudFormation stack. You must delete the stack that you created for your node group for Windows or Linux.If you have an unmanaged node group and didn't use a CloudFormation template to launch your worker nodes, then delete the Auto Scaling group for your worker nodes. Or, terminate the instance directly if you didn't use an Auto Scaling group.Follow"