Question: How do I troubleshoot slow logs in Amazon OpenSearch Service?

"I enabled search slow logs for my Amazon OpenSearch Service domain. However, I receive an error, or the slow logs don't appear in my Amazon CloudWatch log group. How do I resolve this?"

Answer: "I enabled search slow logs for my Amazon OpenSearch Service domain. However, I receive an error, or the slow logs don't appear in my Amazon CloudWatch log group. How do I resolve this?ResolutionI receive an error when I try to set up slow logsIf your AWS account exceeds ten resource policies for your Region, you receive the following error message in Amazon CloudWatch Logs:"Unable to create the Resource Access Policy - You have reached the maximum limit for number of Resource Access Policies for CloudWatch Logs. Please select an existing policy and edit it or delete an older policy and try again."To resolve this error message, create a resource policy that includes multiple log groups.For example:{    "Version": "2012-10-17",    "Statement": [{        "Effect": "Allow",        "Principal": {            "Service": "es.amazonaws.com"        },        "Action": [            "logs:PutLogEvents",            "logs:CreateLogStream"        ],        "Resource": [            "ARN-Log-Group-1",            "ARN-Log-Group-2",            "ARN-Log-Group-3",            "ARN-Log-Group-4"        ]    }]}Note: The AWS Identity and Access Management (IAM) policy limit can't be increased.I don't see any slow logs being deliveredIf you don't see your slow logs being delivered to CloudWatch, check your IAM policy or OpenSearch Service thresholds.Because OpenSearch Service requires permission to write to CloudWatch Logs, you must have the proper IAM policy to log your queries. To update your IAM policy, navigate to Search Slow Logs, and then choose Select Setup.For example:{    "Version": "2012-10-17",    "Statement": [{        "Effect": "Allow",        "Principal": {            "Service": "es.amazonaws.com"        },        "Action": [            "logs:PutLogEvents",            "logs:CreateLogStream"        ],        "Resource": "arn:aws:logs:us-east-1:588671893395:log-group:/aws/aes/domains/myes/search-logs:*"    }]}Also, make sure to set an appropriate timing threshold for your domain. For example, if all your requests complete before the set threshold, then your logs won't be delivered to your log group.You can also set individual INDEX level thresholds for each debug level (TRACE, DEBUG, INFO, and WARN).For example, you can set the threshold for WARN debug levels to ten seconds for the YOURINDEXNAME index in OpenSearch Dashboards:PUT /YOURINDEXNAME/_settings{"index.search.slowlog.threshold.query.warn": "10s"}Note: You can set TRACE to "0" milliseconds to log all queries for your domain. However, because logging all queries is resource-intensive, your  domain performance might be impacted.Then, check your threshold using the following command:GET/YOURINDEXNAME/_settings?prettyOpenSearch Service logs any queries that exceed the defined threshold.Best practicesAvoid making multiple configuration changes (such as enabling or disabling logs that are published to CloudWatch) at the same time. Too many configuration changes at one time trigger multiple blue/green deployments. Multiple blue/green deployments can cause the OpenSearch Service domain to get stuck in a processing state. For more information about blue/green deployment, see Making configuration changes in OpenSearch Service.Set your threshold for both the query phase and fetch phase to identify slow search queries.Test with a low threshold value, and slowly increase the threshold to log only the queries that are affecting performance or requiring optimization.Choose the appropriate number of shards for your cluster to optimize cluster performance. For more information about shard maintenance, see Amazon OpenSearch Service best practices.For slow logs, enable logging at the TRACE, DEBUG, INFO, and WARN debug levels. Because each debug level logs different categories of information, it's a best practice to enable logging according to the request status.Related informationAnalyzing Amazon OpenSearch Service slow logs using CloudWatch Logs streaming and OpenSearch DashboardsHow do I troubleshoot CloudWatch Logs so that it streams to my Amazon OpenSearch Service domain?Viewing Amazon OpenSearch Service error logsFollow"


Question: How do I resolve the error "The association iip-assoc-xxxxxxxx is not the active association" on my EC2 instance?

I'm receiving the following error message on my Amazon Elastic Compute Cloud (Amazon EC2) instance while updating the instance profile:"The association iip-assoc-xxxxxxxx is not the active association"How do I resolve this error?

Answer: "I'm receiving the following error message on my Amazon Elastic Compute Cloud (Amazon EC2) instance while updating the instance profile:"The association iip-assoc-xxxxxxxx is not the active association"How do I resolve this error?Short descriptionThis error usually occurs when you attempt to update the instance profile while a previous disassociation is still unfulfilled by the API. You can use the AWS Command Line Interface (AWS CLI) to identify if an unfulfilled disassociation is causing the error, and to correct the issue.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.Resolution1.    Run the following command to identify the instance profile associations for the instance:aws ec2 describe-iam-instance-profile-associations --filters "Name=instance-id,Values=i-xxxxxxxxxxxxxxxxx"The command output has multiple associations, each with a unique association ID (AssociationID) and status (State). Some of the associations are in the associating state and some are in the disassociating state, as shown in the following example output:{"IamInstanceProfileAssociations": [  {    "AssociationId": "iip-assoc-xxxxxxxxxxxxxxxx",    "InstanceId": "i-xxxxxxxxxxxxxxxx",    "IamInstanceProfile": {      "Arn": "arn:aws:iam::xxxxxxxxxx:instance-profile/xxxxxxx",      "Id": "xxxxxxxxxxxxxxxxxx"     },    "State": "disassociating"  }, {    "AssociationId": "iip-assoc-xxxxxxxxxxxxxxxx",    "InstanceId": "i-xxxxxxxxxxxxxxxx",    "IamInstanceProfile": {      "Arn": "arn:aws:iam::xxxxxxxxxxxx:instance-profile/xxxxxxxxx",      "Id": "xxxxxxxxxxxxxxxx"     },    "State": "associating"  } ]}2.    Run the following command to disassociate all the association IDs, including those in associating and disassociating states. Replace iip-assoc-xxxxxxxxxxxxxxxxxx with the appropriate association-id.aws ec2 disassociate-iam-instance-profile --association-id iip-assoc-xxxxxxxxxxxxxxxxxx3.    After disassociating all association IDs, try updating the instance profile again.Note: If the error persists after following the resolution steps, stop and start the instance. Then, run the disassociate-iam-instance-profile command again. Be aware that data stored in instance store volumes is lost when you stop the instance. Before stopping the instance, review the list of the effects of stopping an instance.Related informationUsing instance profilesHow do I attach or replace an instance profile on an Amazon EC2 instance?Follow"


Question: How can I resolve a custom domain name from resources in my VPC without modifying the DHCP options set?

I want to resolve a custom domain name from resources in my Amazon Virtual Private Cloud (Amazon VPC) without modifying the DHCP options set. How can I do this?

Answer: "I want to resolve a custom domain name from resources in my Amazon Virtual Private Cloud (Amazon VPC) without modifying the DHCP options set. How can I do this?ResolutionTo configure a private hosted zone to specify how Amazon Route 53 responds to DNS queries for domains within your VPCs, do the following:Create a private hosted zone for your domain name and attach it to the required VPC.Create records in the private hosted zone to specify how Route 53 responds to DNS queries from resources in the attached VPC.(Optional) Associate a Route 53 private hosted zone with a VPC on a different AWS account.Follow"


Question: "Why is my Windows WorkSpace stuck in the starting, rebooting, or stopping status?"

"My Amazon WorkSpaces Windows WorkSpace is stuck in the starting, rebooting, or stopping status."

Answer: "My Amazon WorkSpaces Windows WorkSpace is stuck in the starting, rebooting, or stopping status.Short descriptionThe WorkSpace status depends on health checks of the WorkSpace's components, such as PCOIP Standard Agent, SkylightWorkSpacesConfigService, or Web Access agent. Sometimes the WorkSpace's operating system (OS) takes a long time to boot. Or, the WorkSpace's components fail to report the actual status of the WorkSpace. These issues might cause the status to become stuck in the starting, rebooting, or stopping state.ResolutionIf your Windows WorkSpace is stuck in the starting, rebooting, or stopping status, then review the following common causes and troubleshooting tips:Pending updatesIf the Workspaces console or client application sends a reboot command to the WorkSpace, then the WorkSpace might be busy applying updates. A reboot action also initiates any pending WorkSpaces component updates.Windows updates or other operating system (OS) activity can delay the boot process, and WorkSpaces health checks might take longer to reflect the actual status. Wait a few minutes, and then try again to connect to the WorkSpace.WorkSpaces services in stopped state or blocked by antivirus softwareTo verify that the required WorkSpaces services are started at the OS level, follow these steps:1.    Connect to the WorkSpace using Remote Desktop Connection (RDP).2.    Open Administrative Tools.3.    Open the context (right-click) menu for Services, and then choose Open. Check if the Status is Stopped for any of the following services:STXHD Hosted Application ServicePCoIP Standard Agent for WindowsSkyLightWorkspaceConfigService4.    If the status is Stopped, then open the context (right-click) menu for the service, and choose Start. The status now Running.5.    Verify that the Startup Type for all three services is set to Automatic. To change the Startup Type, open the context (right-click) menu for the service, and then choose Properties. For Startup type, choose Automatic, and then choose Ok.If the status of the three services is Running, then antivirus software might be blocking the services. To fix this, set up an allow list for the locations where the service components are installed. For more information, see Required configuration and service components for WorkSpaces.Network adapters turned onVerify that both network adapters on the WorkSpace are turned on:1.    Connect to the WorkSpace using RDP.2.    Open the Start menu, and then navigate to Control Panel > Network and Internet > Network and Sharing Center.3.    Verify that there are two networks listed in the View your active networks section. If there's only one network listed, then choose Change adapter settings. Confirm that Ethernet 3 is turned on. If necessary, open the context (right-click) menu for Ethernet 3, and then choose Enable.Restart the WorkSpaceIf you still have issues, then restart the WorkSpace from the WorkSpaces console. Wait a few minutes for the WorkSpace to come online, and then try again to connect to the WorkSpace. The status of the WorkSpace changes within 90 minutes.Related informationWhy can't I create a new WorkSpace?Follow"


Question: Why is CloudFront serving outdated content from Amazon S3?

"I'm using Amazon CloudFront to serve objects stored in Amazon Simple Storage Service (Amazon S3). I updated my objects in Amazon S3, but my CloudFront distribution is still serving the previous versions of those files."

Answer: "I'm using Amazon CloudFront to serve objects stored in Amazon Simple Storage Service (Amazon S3). I updated my objects in Amazon S3, but my CloudFront distribution is still serving the previous versions of those files.Short descriptionBy default, CloudFront caches a response from Amazon S3 for 24 hours (Default TTL of 86,400 seconds). If your request lands at an edge location that served the Amazon S3 response within 24 hours, then CloudFront uses the cached response. This happens even if you updated the content in Amazon S3.Use one of the following ways to push the updated Amazon S3 content from CloudFront:Invalidate files.Update existing content with a CloudFront distribution.ResolutionInvalidate the Amazon S3 objectsYou can invalidate an Amazon S3 object to remove it from the CloudFront distribution's cache. After the object is removed from the cache, the next request retrieves the object directly from Amazon S3.Before you run this process, consider the following:You can't invalidate specific versions of an object that uses cookies or headers to vary the response. CloudFront invalidates all versions of the object in this case.Each AWS account is allowed 1,000 free invalidation paths per month. For more information, see Amazon CloudFront pricing.When you create an invalidation, be sure that the object paths meet the following requirements:The object paths must be for individual objects or the paths must end with the wildcard character (*). For example, you can't run an invalidation on a path similar to /images/*.jpeg because the path isn't for an individual object, and it doesn't end in a wildcard.The specified path must exactly match the capitalization of the object's path. Invalidation requests are case-sensitive.To remove specific versions of an object based on a query string, include QueryString in the invalidation path.Object invalidations typically take from 10 to 100 seconds to complete. You can check the status of an invalidation by viewing your distribution from the CloudFront console.Use object versioningIf you update content frequently, it's a best practice that you use object versioning to clear the CloudFront distribution's cache. For frequent cache refreshes, using object versioning might cost less than using invalidations.Use one of these ways to add versioning to your objects:Store new versions of the object at the origin with the version number in the key name. For example, if you update /image_v1.png, then you store a new version of the object as /image_v2.png.Update the object at the origin but cache based on a query string with the object version. For example, the query string updates from /image.png?ver=1 to /image.png?ver=2. You can use a cache policy to specify which query strings are included in the cache key and origin requests.Note: You can still request the previous version (/image.png?ver=1) while it's available in the CloudFront cache.Consider the following advantages and disadvantages for each method of object versioning:Storing new versions of the object at the origin (Amazon S3) allows you to revert changes to previous versions that are still available under the previous names. However, storing multiple versions of an object can increase your storage costs.Updating the object at the origin but caching based on the query string can reduce your storage costs. However, to prepare for any rollbacks, it's a best practice to keep previous object versions offline.Note: Specifying versioned file names or directory names is not related to Amazon S3 Object Versioning. Using the Amazon S3 Versioning feature does not update the content automatically. You must specify file paths carefully, as you can't cancel an invalidation request after you have started one.Related informationManaging how long content stays in the cache (expiration)Query string forwarding and cachingFollow"


Question: Why are the metrics on the DynamoDB console different from the CloudWatch metrics?

The graphs on the Metrics tab in the Amazon DynamoDB console are different from the graphs in the Amazon CloudWatch console.

Answer: "The graphs on the Metrics tab in the Amazon DynamoDB console are different from the graphs in the Amazon CloudWatch console.ResolutionThe metrics in the CloudWatch console are raw and provide more statistics options than the metrics in the DynamoDB console. The metrics in the DynamoDB console are average values over one-minute intervals. For example, ConsumedWriteCapacityUnits is the sum of the consumed units over one minute, divided by the number of seconds in a minute (60).To set the graphs to look the same in both CloudWatch and DynamoDB, be sure that the period and the time range are the same:Open the DynamoDB console.In the navigation pane, choose Tables.Choose your table, and then choose the Metrics tab.Choose View all CloudWatch metrics to open the CloudWatch console.Choose the category that the metric is in, such as Table metrics.Check the boxes next to the table name for the metrics that you want to see.Choose the Graphed metrics tab.In the Statistic dropdown list, choose Sum.In the Period dropdown list, choose 1 Minute.If there are provisioned and consumed metrics on the CloudWatch graph, then use the arrows in the Y Axis column to move the provisioned values to the right Y axis and the consumed values to the left Y axis.Use a math expression to divide the metrics by 60 (for example, m2/60).Missing metricsIf CloudWatch doesn't list a metric for DynamoDB, then it's probably because DynamoDB doesn't have recent data for that metric. CloudWatch lists only the metrics that were active in the past two weeks. This prevents you from seeing too many older metrics when you call ListMetrics.Related informationDynamoDB metrics and dimensionsUsing metric mathMonitoring with Amazon CloudWatchGraphing a metricFollow"


Question: Why do I get zero records when I query my Amazon Athena table?

"I ran a CREATE TABLE statement in Amazon Athena with expected columns and their data types. When I run the query SELECT * FROM table-name, the output is "Zero records returned.""

Answer: "I ran a CREATE TABLE statement in Amazon Athena with expected columns and their data types. When I run the query SELECT * FROM table-name, the output is "Zero records returned."ResolutionHere are some common reasons why the query might return zero records.File selected in crawler settingsIf you're using a crawler, be sure that the crawler is pointing to the Amazon Simple Storage Service (Amazon S3) bucket rather than to a file.Incorrect LOCATION pathVerify the Amazon S3 LOCATION path for the input data. If the input LOCATION path is incorrect, then Athena returns zero records.Double slash in LOCATION pathAthena doesn't support table location paths that include a double slash (//). For example, the following LOCATION path returns empty results:s3://doc-example-bucket/myprefix//input//To resolve this issue, copy the files to a location that doesn't have double slashes. Here is an example AWS Command Line Interface (AWS CLI) command to do so:aws s3 cp s3://doc-example-bucket/myprefix//input// s3://doc-example-bucket/myprefix/input/ --recursiveNote: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.Data for multiple tables stored in the same S3 prefixGlue crawlers create separate tables for data that's stored in the same S3 prefix. However, when you query those tables in Athena, you get zero records.For example, your Athena query returns zero records if your table location is similar to the following:s3://doc-example-bucket/table1.csvs3://doc-example-bucket/table2.csvTo resolve this issue, create individual S3 prefixes for each table similar to the following:s3://doc-example-bucket/table1/table1.csvs3://doc-example-bucket/table2/table2.csvThen, run a query similar to the following to update the location for your table table1:ALTER TABLE table1 SET LOCATION 's3://doc-example-bucket/table1';Partitions not yet loadedAthena creates metadata only when a table is created. The data is parsed only when you run the query. If your table has defined partitions, the partitions might not yet be loaded into the AWS Glue Data Catalog or the internal Athena data catalog. Use MSCK REPAIR TABLE or ALTER TABLE ADD PARTITION to load the partition information into the catalog.MSCK REPAIR TABLE: If the partitions are stored in a format that Athena supports, run MSCK REPAIR TABLE to load a partition's metadata into the catalog. For example, if you have a table that is partitioned on Year, then Athena expects to find the data at Amazon S3 paths similar to the following:s3://doc-example-bucket/athena/inputdata/year=2020/data.csvs3://doc-example-bucket/athena/inputdata/year=2019/data.csvs3://doc-example-bucket/athena/inputdata/year=2018/data.csvIf the data is located at the Amazon S3 paths that Athena expects, then repair the table by running a command similar to the following:CREATE EXTERNAL TABLE Employee (    Id INT,    Name STRING,    Address STRING) PARTITIONED BY (year INT)ROW FORMAT DELIMITED FIELDS TERMINATED BY ','LOCATION 's3://doc-example-bucket/athena/inputdata/';After the table is created, load the partition information:MSCK REPAIR TABLE Employee;After the data is loaded, run the following query again:SELECT * FROM Employee;ALTER TABLE ADD PARTITION: If the partitions aren't stored in a format that Athena supports, or are located at different Amazon S3 paths, run ALTER TABLE ADD PARTITION for each partition. For example, suppose that your data is located at the following Amazon S3 paths:s3://doc-example-bucket/athena/inputdata/2020/data.csvs3://doc-example-bucket/athena/inputdata/2019/data.csvs3://doc-example-bucket/athena/inputdata/2018/data.csvGiven these paths, run a command similar to the following:ALTER TABLE Employee ADD    PARTITION (year=2020) LOCATION 's3://doc-example-bucket/athena/inputdata/2020/'    PARTITION (year=2019) LOCATION 's3://doc-example-bucket/athena/inputdata/2019/'    PARTITION (year=2018) LOCATION 's3://doc-example-bucket/athena/inputdata/2018/'After the data is loaded, run the following query again:SELECT * FROM Employee;Hive hidden filesVerify that your file names don't start with an underscore (_) or a dot (.).Example:s3://doc-example-bucket/athena/inputdata/_file1s3://doc-example-bucket/athena/inputdata/.file2If the files in your S3 path have names that start with an underscore or a dot, then Athena considers these files as placeholders. Athena ignores these files when processing a query. For more information, see Athena cannot read hidden files. If all the files in your S3 path have names that start with an underscore or a dot, then you get zero records.Note: If your S3 path includes placeholders along with files whose names start with different characters, then Athena ignores only the placeholders and queries the other files. Therefore, you might get one or more records.Related informationCreating tables in AthenaUsing AWS Glue crawlersFollow"


Question: How can I resolve the AWS STS AssumeRoleWithWebIdentity API call error "InvalidIdentityToken"?

The AWS Security Token Service (AWS STS) API call AssumeRoleWithWebIdentity failed with an "InvalidIdentityToken" error.

Answer: "The AWS Security Token Service (AWS STS) API call AssumeRoleWithWebIdentity failed with an "InvalidIdentityToken" error.Short descriptionIf your AssumeRoleWithWebIdentity API call fails, then you might receive an error that's similar to the following message:"An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation. Couldn't retrieve verification key from your identity provider."This error might occur in the following scenarios:The .well_known URL and jwks_uri of the identity provider (IdP) are inaccessible from the public internet.A custom firewall is blocking the requests.There's latency of more than 5 seconds in API requests from the IdP to reach the AWS STS endpoint.STS is making too many requests to your .well_known URL or the jwks_uri of the IdP.Note: Because this issue fails on the client side, AWS CloudTrail event history doesn't log this error.ResolutionVerify public access for .well_known and jwks_uriVerify that the .well_known URL and jwks_uri of the IdP are publicly accessible. This can be checked using your browser, Windows command, or Linux command. To do this, complete one of the following actions:To check access, navigate to the following links in your browser:https://BASE_SERVER_URL/.well-known/openid-configurationhttps://BASE_SERVER_URL/.well-known/jwks.json-or-Run the following commands:Windows:wget https://BASE_SERVER_URL/.well-known/openid-configurationwget https://BASE_SERVER_URL/.well-known/jwks.jsonLinux:curl https://BASE_SERVER_URL/.well-known/openid-configurationcurl https://BASE_SERVER_URL/.well-known/jwks.jsonNote: To confirm if you can access the links, check for the 200 status code in the request response.Check firewall settingsIf the .well_known URL and jwks_uri of the IdP aren't accessible, then check the firewall settings. Make sure that the domains aren't on a deny list.Depending on the current configuration of the firewall, the domains might need to be added to an allow list.If the firewall settings aren't accessible, then use the browser with a device from a different network, such as a phone. To check access from the browser, use the instructions in step 1. If the web request succeeds, then the firewall is blocking the request.If the server that's making the AssumeRoleWithWebIdentity API call is an Amazon Elastic Compute Cloud (Amazon EC2) instance, then check the configuration settings. For instructions, see Why can't I connect to a website that is hosted on my EC2 instance?Check operation latencyCheck the latency for the total operation. This includes the following attributes:Request/Response time from STSRequest/Response time from IdPMinimize STS latencyUse AWS Regional endpoints instead of global endpoints for the STS service. This verifies that the requests are routed to the geographically closest server to minimize latency. For more information, see Writing code to use AWS STS Regions.Note: For AWS SDKs, the Region parameter routes the request's destination endpoint to where the call is made within the sts_regional_endpoint configuration.Evaluate IdP latencyThe IdP makes requests to the STS endpoint. To check if the request to the STS endpoint takes too long, analyze the IdP's outgoing packets within the IdP logs.Note: If the request from the IdP to the STS endpoint takes more than 5 seconds, then the request might time out and fail. You can contact your identity provider to request an increase for geographical availability to reduce latency for this API call.(Optional) Use exponential backoff and increase retriesThe AssumeRoleWithWebIdentity API depends on retrieving information from the identity provider (IdP). To avoid throttling errors, most IdPs have API limits, and API calls might not get the required keys back from the IdP. To help successfully assume a role if the API has intermittent issues reaching your IdP, take the following steps:Use exponential backoff.Increase your retries, and set a maximum number of retries. Also, implement a maximum delay interval. For more information, see Error retries and exponential backoff in AWS.Reduce STS requests to .well_known and jwks_uriIf your JSON Web Key Set (JWKS) sets either Pragma: no-cache or Cache-Control: no-cache response headers, then STS doesn't cache your JWKS. For keys that are referenced in an ID_TOKEN but aren't in the cache, STS performs a callback. In this case, STS might make too many requests to your .well_known URL and jwks_uri.Therefore, to reduce callbacks from STS, verify that your JWKS doesn't set either of these response headers. This allows STS to cache your JWKS.Related informationWelcome to the AWS Security Token Service API ReferenceHow can I resolve API throttling or "Rate exceeded" errors for IAM and AWS STS?Follow"


Question: How can I troubleshoot Lambda function invoke issues with Amazon EFS integration?

I invoked an AWS Lambda function with Amazon Elastic File System (Amazon EFS) integration and received an error message.

Answer: "I invoked an AWS Lambda function with Amazon Elastic File System (Amazon EFS) integration and received an error message.Short descriptionThe following are pre-requisites for mounting Amazon EFS access points with Lambda:The Lambda function's execution role must have the following elasticfilesystem permissions:elasticfilesystem:ClientMountelasticfilesystem:ClientWrite (not required for read-only connections)Your AWS Identify and Access Management (IAM) user must have the following permissions:elasticfilesystem:DescribeMountTargetsThe EFS File System security group must allow NFS (port 2049) inbound traffic from the Lambda security group or IP address range.The Lambda security group must allow NFS (port 2049) outbound traffic to the EFS security group or IP address range.The Lambda function and Amazon EFS access points must be in the same AWS Region and Availability Zone.For more information, see How do I create the correct EFS access point configuration to mount my file system using a Lambda function?ResolutionFollow these troubleshooting steps for the error message with your Lambda function.EFSMountFailureException:The Lambda function couldn't mount the configured EFS file system due to a permission or configuration issue. Check the Lambda functions permissions. Then, confirm that the EFS file system and access point exist and are ready for use. For more information, see Function could not mount the EFS file system.EFSMountConnectivityException:The Lambda function couldn't make a network connection to the configured EFS file system with the NFS protocol (TCP port 2049). Check the security group and routing configuration for the Amazon Virtual Private Cloud (Amazon VPC) subnets. For more information, see Function could not connect to the EFS file system.EFSMountTimeoutException:The Lambda function was able to make a network connection to the configured EFS file system, but the mount operation timed out. Retry invoking the Lambda function. If the Lambda function times out again, then limit the functions reserved concurrency to reduce the load volume on the EFS file system. For more information, see Function could not mount the EFS file system due to timeout.PermissionError: Permission denied: '/mnt/xyz/abc':Lambda doesn't have access to the specified Amazon EFS access point. To troubleshoot Amazon EFS access points, see What are common EFS access point configurations?For more information, see Troubleshoot invocation issues in Lambda.Related informationWorking with Amazon EFS access pointsFollow"


Question: How do I synchronize SQL Server Agent jobs between the primary and secondary hosts in my RDS for SQL Server Multi-AZ instance?

I want to synchronize SQL Server Agent jobs between the primary and secondary hosts in my Amazon Relational Database (Amazon RDS) for Microsoft SQL Server Multi-AZ instance. How do I do this?

Answer: "I want to synchronize SQL Server Agent jobs between the primary and secondary hosts in my Amazon Relational Database (Amazon RDS) for Microsoft SQL Server Multi-AZ instance. How do I do this?Short descriptionAmazon RDS for SQL Server uses always on/mirroring for the Multi-AZ setup behind the scenes. SQL Server Agent jobs are stored in the msdb system database. This system database isn't replicated as part of your Multi-AZ deployment. So, the SQL Server Agent jobs aren't synchronized automatically. You must recreate the jobs on the new primary after failover. However, the jobs are present on the old primary where they were originally created. If you fail back the instance to the previous primary (where the jobs were initially created) you still see the jobs. To create the jobs in Multi-AZ, create the jobs in the primary (active) instance first. Then, fail over the RDS instance and create the same jobs on the new primary instance.To avoid manually creating jobs on the new primary, turn on SQL Agent Job replication. When job replication is turned on in your Multi-AZ environment, the SQL Server Agent jobs are copied from the primary host to the secondary host automatically. You don't have to create the jobs manually on the new primary replica because they synchronized through the agent replication feature. The jobs are available in both the replicas.For more information, see Multi-AZ deployments for Amazon RDS for Microsoft SQL Server.ResolutionTurn on the SQL agent replication featureRun the following procedure with the admin account on the primary instance to turn on SQL Server Agent job replication.Important note: Make sure that you run this procedure on the instance that has all the agent jobs available. If the instance without the available agent jobs becomes the primary and you turn on this feature, then any jobs on the secondary instance are deleted. Also note that all existing and newly created jobs are replicated as part of this feature.EXECUTE msdb.dbo.rds_set_system_database_sync_objects@object_types = 'SQLAgentJob';Validate that the SQL agent replication feature is turned onUse the following query to verify that the replication feature is turned on:SELECT * from msdb.dbo.rds_fn_get_system_database_sync_objects();The preceding query returns SQLagentjob for object_class if the replication feature is turned on. If the query returns nothing, then the feature isn't turned on.Verify when modified and new jobs last synchronized with the secondaryUse the following query to check the last_sync_time of the replication:SELECT * from msdb.dbo.rds_fn_server_object_last_sync_time();In the query results, if the sync time is past the job updated or creation time, then the job is synced with the secondary.Note: If you don't know the time of the job creation or update, run the following query to retrieve the timestamp, and then run the preceding query:select name as 'JobName',JobStatus = CASE    When enabled =1 Then 'Active'    Else 'Inactive'    END,date_created As 'JobCreatedOn' ,date_modified as 'LastModified'from msdb..sysjobsNote: It takes few minutes for the jobs to synchronize between the replicas.If you want to perform DB failover to confirm that the jobs are replicated, wait for the last_sync_time to update before proceeding with the Multi-AZ failover.Status of the jobs on secondaryA SQL Server Agent XP is in the Disabled state with or without using the replication feature on the secondary replica. So, the jobs don't run on the secondary server.Supported and unsupported job categories for agent job replicationJobs in the following categories are replicated:[Uncategorized (Local)][Uncategorized (Multi-Server)][Uncategorized]Data CollectorDatabase Engine Tuning AdvisorDatabase MaintenanceFull-TextNote: Only jobs that use step type as T-SQL are replicated.The following are categories that don't support replication:Jobs with step types such as SQL Server Integration Services (SSIS), SQL Server Reporting Services (SSRS), replication, and PowerShell.Jobs that use Database Mail and server-level objects.Turn off SQL Server Agent job replicationTo turn off SQL Server Agent job replication, run the following command:EXECUTE msdb.dbo.rds_set_system_database_sync_objects @object_types = '';After you turn off replication, modifications to existing and newly created jobs no longer sync with the other replica.Follow"


Question: Why is the traffic for my web content getting routed to the wrong CloudFront edge location?

"I'm using Amazon CloudFront to distribute my web content. However, the traffic to my website is routed to the wrong edge location. How can I fix this?"

Answer: "I'm using Amazon CloudFront to distribute my web content. However, the traffic to my website is routed to the wrong edge location. How can I fix this?Short descriptionCloudFront routes traffic based on the distribution's price class, associated geolocation databases, and EDNS0-Client-Subnet support. Depending on the combination of these factors, your website's viewers might be routed to an unexpected edge location. This can increase the overall latency for retrieving an object from a CloudFront edge location.To troubleshoot routing to an unexpected edge location, check the following:The price class supports the edge location that you expect.The DNS resolver supports Anycast routing.The DNS resolver supports EDNS0-Client-Subnet.ResolutionThe price class supports the edge location that you expectCheck the edge locations that are included in the price class of your CloudFront distribution. You can update the price class of your distribution if you want to include other edge locations.The DNS resolver supports Anycast routingIf the DNS resolver supports Anycast routing, then there are multiple edge locations that the DNS resolver uses. This means that a requester's edge location is based on optimal latency, which might result in an unexpected location for the resolver's IP address.To check if the DNS resolver supports Anycast, run one of these commands multiple times:Note: In these example commands, be sure to replace example.com with the DNS resolver domain name that you're using.On Linux or macOS, run a dig command, similar to the following:dig +nocl TXT o-o.myaddr.l.example.comOn Windows, run an nslookup command, similar to the following:nslookup -type=txt o-o.myaddr.l.example.comIf the output includes the same IP address each time you run the command, then the DNS resolver doesn't support Anycast. If the output includes a different IP address each time you run the command, then the DNS resolver supports Anycast. This might explain an unexpected edge location.The DNS resolver supports EDNS0-Client-SubnetTo determine how you can avoid incorrect routing, first check if the DNS resolver supports EDNS0-Client-Subnet by running one of these commands:Note: In these example commands, be sure to replace example.com with the DNS resolver domain name that you're using.On Linux or macOS, run a dig command, similar to the following:dig +nocl TXT o-o.myaddr.l.example.comOn Windows, run an nslookup command, similar to the following:nslookup -type=txt o-o.myaddr.l.example.comNote: Check the TTL value, and be sure to run the command when the TTL expires. Otherwise, you might get a cached response from the recursive resolver.If the DNS resolver doesn't support EDNS0-Client-Subnet, then the output is similar to the following:$ dig +nocl TXT o-o.myaddr.l.example.com  +short"192.0.2.1"In the previous example, 192.0.2.1 is the IP address of the closest DNS server that's using Anycast. This DNS resolver doesn't support EDNS0-Client-Subnet. To avoid incorrect routing, you can do one of the following:Change to a DNS resolver to a recursive DNS resolve that's located geographically closer to your website's clients.Change to a DNS resolver that does support EDNS0-Client-Subnet.If the DNS resolver supports EDNS0-Client-Subnet, then the output contains a truncated client subnet (/24 or /32) to the CloudFront authoritative name server, similar to the following:$ dig +nocl TXT o-o.myaddr.l.example.com @8.8.8.8 +short"192.0.2.1""edns0-client-subnet 198.51.100.0/24"In the previous example, 192.0.2.1 is the closest DNS resolver IP address. Additionally, the client-subnet range is 198.51.100.0/24, which is used to respond to DNS queries. To avoid incorrect routing when the DNS resolver does support EDNS0-Client-Subnet, confirm that a public geolocation database is associated with the client-subnet range that's sending the query to the DNS resolver. If the DNS resolver is forwarding the truncated version of the client IP addresses to CloudFront name servers, then CloudFront checks a database that's based on several public geolocation databases. The IP addresses must be correctly mapped in the geolocation database so that requests are routed correctly.If the DNS resolver supports EDNS0-Client-Subnet, you can verify the edge location that traffic is routed to by first resolving your CloudFront CNAME by running a DNS lookup command like dig:$ dig dftex7example.cloudfront.net. +short13.224.77.10913.224.77.6213.224.77.6513.224.77.75Then, run a reverse DNS lookup on the IP addresses that are returned by the previous command:$ dig -x 13.224.77.62 +shortserver-13-224-77-62.man50.r.cloudfront.net.In the previous example, the traffic is routed to the Manchester edge location.Tip: For an additional test, you can use a public DNS resolver that supports EDNS0-client-subnet, such as 8.8.8.8 or 8.8.4.4. Send queries with edge location IP addresses to the public DNS resolver. Then, check the results of the DNS queries to see whether CloudFront has the correct information about your EDNS0-client-subnet.Follow"


Question: How can I see who's been accessing my Amazon S3 buckets and objects?

I'm looking for a way to track who's accessing my Amazon Simple Storage Service (Amazon S3) buckets and objects. How can I do that?

Answer: I'm looking for a way to track who's accessing my Amazon Simple Storage Service (Amazon S3) buckets and objects. How can I do that?ResolutionYou can track who's accessing your bucket and objects in the following ways:Use Amazon S3 server access logging to see information about requests to your buckets and objects. You can use Amazon Athena to analyze your server access logs.Use AWS CloudTrail to track API calls to your Amazon S3 resources. You can also use Athena to query your CloudTrail logs.Related informationMonitoring Amazon S3Follow


Question: How do I restore the backup of my Amazon DynamoDB table to a different Region?

I want to restore the backup of my Amazon DynamoDB table to a different Region.

Answer: "I want to restore the backup of my Amazon DynamoDB table to a different Region.ResolutionTo restore your DynamoDB table to a different Region, you can use either of the following approaches.Restore a DynamoDB table to a different Region using DynamoDBOpen the DynamoDB console.In the navigation pane, choose Backups.In the list displayed, choose the backup from which you want to restore the table.Choose Restore.For Name of restored table, enter the new table name.For Secondary indexes, select your desired option.For Destination AWS Region, select Cross-Region.For Select the destination AWS Region, choose the Region of your choice.For Encryption key management, select your desired option.Choose Restore.Restore a DynamoDB table to a different Region using AWS GlueYou can use an AWS Glue job to restore a DynamoDB table to another Region. AWS Glue provides more flexibility with the restoration process. You might choose this approach if you don't want to restore all attributes or fields to the target table in the new Region. This approach works only for a table that's exported to Amazon Simple Storage Service (Amazon S3).1.    After exporting the DynamoDB table to Amazon S3 using the Export to S3 feature, create an AWS Glue job. Be sure to specify the following information in the Script tab:datasource0 = glueContext.create_dynamic_frame.from_options(    connection_type="dynamodb",    connection_options={        "dynamodb.export": "ddb",        "dynamodb.tableArn": "arn:aws:dynamodb:source-region:account-number:table/TableName",        "dynamodb.unnestDDBJson": True,        "dynamodb.s3.bucket": "example-bucket",        "dynamodb.s3.prefix": "dynamodb",        "dynamodb.s3.bucketOwner": "account-number",    })Note: Be sure to use the transform node ApplyMapping, and specify the fields that must be present in the target table. This setting automatically generates the PySpark code based on the input provided.Example:applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("resource_id", "string", "resource_id", "string")], transformation_ctx = "applymapping1")2.    Specify a sink operation to directly write to the destination table in the target Region.Example:glueContext.write_dynamic_frame_from_options (frame = MappedFrame, connection_type = "dynamodb", connection_options = { "dynamodb.region": "example-region", "dynamodb.output.tableName": "example_table", "dynamodb.throughput.write.percent": "1.0" })3.    Run the job from the AWS Glue console to load data from the current Region to the target Region.Related informationRestoring a DynamoDB table from a backupPoint-in-time recovery for DynamoDBWorking with jobs on the AWS Glue consoleFollow"


Question: How can I associate an Application Load Balancer with my Lightsail instance?

I want to use the features of an Amazon Application Load Balancer on my Amazon Lightsail instance. How do I do that?

Answer: "I want to use the features of an Amazon Application Load Balancer on my Amazon Lightsail instance. How do I do that?Short descriptionYou can use a Lightsail load balancer to add redundancy to your web application or to handle more web traffic. You can also use a Lightsail load balancer to install an Amazon provided SSL certificate for your website hosted in a Lightsail instanceHowever, the Lightsail load balancer has limitations when compared to the Amazon Application Load Balancer. Some of the areas Application Load Balancer provides increased flexibility include the following:Flexibility in load balancer health checks.End-to-end encryption of data in transit between the load balancer and the instance.Use of firewall services such as AWS WAF, and so on.If you want to use these features for your website in Lightsail, you can associate an Application Load Balancer with the Lightsail instance. To set up Application Load Balancer, do the following:Configure Amazon VPC peering in Lightsail to allow the instance to connect to AWS resources such as Application Load Balancer.(Optional) Generate an AWS Certificate Manager (ACM) SSL certificate if you want to associate it with your website.Configure the Load Balancer with the target in the target group set to the Lightsail instance's private IP address.Point your domain to the new load balancer in DNS.ResolutionConfigure VPC peering in LightsailFor instructions on configuring VPC peering, see Set up Amazon VPC peering to work with AWS resources outside of Lightsail.You must to activate VPC peering for the Region where your Lightsail instance is located. To do this, you must have a default Amazon VPC in that Region and the necessary AWS Identity and Access Management (IAM) permissions. For more information, see What are the minimum IAM permissions needed to set up communication between Lightsail and other AWS services using VPC peering?To check whether you have a default VPC, see View your default VPC and default subnets.If you don’t have a default Amazon VPC, then you can create one. To learn more, see Create a default VPC.(Optional) Generate an ACM certificateTo associate an ACM certificate with your domain and access websites using HTTPS, see Requesting a public certificate using the console.It's a best practice to give two names to the certificate. For example, example.com and *.example.com. By doing this, the same certificate can be used for the main domain and for subdomains, such as www.example.com or xyz.example.com. Keep in mind that this doesn't cover a Wildcard SSL certificate for two levels for the subdomain, for example abc.xyz.example.com.It's also a best practice to use DNS domain validation instead of email validation. DNS validation has multiple benefits over email validation.If you have issues validating the ownership of the domain using either DNS or email, see the following:Why is my AWS Certificate Manager (ACM) certificate DNS validation status still pending validation?Why am I not receiving validation emails when using ACM to issue or renew a certificate?Configure the target group for the load balancer and register the targetFor instructions, see Configure a target group.When configuring your target, keep the following in mind:Make sure to choose IP as the target type.Keep the protocol as HTTP and port as 80 if you don't have an SSL certificate installed inside your Lightsail instance. Make sure that there aren't any HTTPs redirections configured inside your instance. Otherwise, this might cause an infinite redirection loop error.Keep the protocol as HTTPS and the port as 443. If you want to encrypt the data in transit from the Application Load Balancer to your Lightsail instance. Make sure that you have an SSL certificate installed in your instance for this.Choose the default VPC in the VPC sectionIn the Register target section, choose Other Private IP addresses under Network and specify the private IP address of your Lightsail instance. For information on obtaining the private IP address of your Lightsail instance, see Private and public IPv4 addresses for instances.Configure the load balancerFor instructions, see Configure a load balancer and a listener.When configuring the load balancer, keep the following in mind:Make sure that you choose the default VPC and at least two Availability Zones. You can choose any Availability Zones.Choose a security group or create a new one. Make sure that the security group has port 80 open. Also, open port 443 if you're attaching an ACM certificate with the load balancer.Add a new HTTPS listener if you want to access your website with HTTPS using an ACM certificate.Point both the HTTP and the HTTPS listener to the target group created in the previous step.Update the DNS entries of the domain to point to the ALB DNS nameIt's a best practice to use Amazon name servers and Amazon Route 53 for the domains that use Application Load Balancer with the website. This is because AWS provides the DNS name for the load balancer, not the IP address. Most name servers don't support adding a hostname for apex domain, such as example.com. They only support this for subdomains such as www.example.com or blog.example.com. However, Route 53 provides the alias feature that allows you to directly point the apex domain, example.com, to the load balancer DNS name.Note: Even if you use Lightsail DNS for your domain, you must switch the DNS to Route 53. This is because it's not possible to point the apex domain to the Application Load balancer DNS name in Lightsail DNS.To update the name servers of the domain to Amazon, if it's not already using Amazon, see Making Route 53 the DNS service for a domain that's in use.To get the DNS name for a load balancer, see Getting the DNS name for an ELB load balancer.To update your Route 53 hosted zone to point the domain to the load balancer DNS name, see Routing traffic to an ELB load balancer.Note: If there are already DNS records for the domain in Route 53 that are pointing to an EC2 instance IP address, edit those records instead of adding new records.Final checkAccess the domain in your browser and confirm that the website is loading correctly. Now that you have connected an Application Load Balancer with your Lightsail instance, you can use its different features that aren't present with Lightsail Load Balancer. You can also set up firewall services such as AWS WAF.Follow"


Question: Why does failover take longer than expected on my AWS Direct Connect connection after configuring BFD?

"I enabled Bidirectional Forwarding Detection (BFD) for my AWS Direct Connect connection for fast failure detection and failover with graceful restart enabled. However, I have failover delays or connection issues."

Answer: "I enabled Bidirectional Forwarding Detection (BFD) for my AWS Direct Connect connection for fast failure detection and failover with graceful restart enabled. However, I have failover delays or connection issues.ResolutionIt's a best practice not to configure graceful restart and BFD at the same time to avoid failover or connection issues. For fast failover, configure BFD without graceful restart enabled.For more information, see BGP quotas.Related informationDirect Connect network requirementsFollow"


Question: How do I troubleshoot Amazon Pinpoint campaign message failures?

"My Amazon Pinpoint campaign isn't processing or targeting any endpoints, and it isn't sending any messages. How do I troubleshoot campaign failures in Amazon Pinpoint?"

Answer: "My Amazon Pinpoint campaign isn't processing or targeting any endpoints, and it isn't sending any messages. How do I troubleshoot campaign failures in Amazon Pinpoint?ResolutionFor endpoints that aren't processed or targeted1.    Verify if the campaign has the Use recipients local time setting activated. For instructions, see Sending the campaign at a specific date and time in the Amazon Pinpoint User Guide.2.    If the setting is activated, then make sure that each endpoint definition includes a valid Demographic.Timezone attribute.Note: If an endpoint doesn't have a valid Demographic.Timezone attribute defined and the campaign's Use recipients local time setting is activated, the endpoint isn't processed.For endpoints that are processed and targeted, but not receiving messagesAmazon Pinpoint campaigns can fail to deliver messages to specific endpoints for many reasons. To troubleshoot message failures to specific endpoints, first configure Amazon Pinpoint to send information about events to Amazon Kinesis. Then, recreate the message failures and review the failed events in Amazon Kinesis to identify what's causing the error. After you've identified what's causing the error, remediate the issue.Note: The event_type value in each streaming event indicates the cause of most message failures. If the event_type value doesn't provide a clear indication of what's causing the message failure, then review your campaign's CampaignSendMessageThrottled metric in Amazon CloudWatch. This metric shows the number of campaign messages that weren't sent because your AWS account's ability to send messages was throttled. For more information, see Amazon Pinpoint quotas.For failed emailsMake sure that you identify any streaming events with the following event_type value: _email.rendering_failure. This event type usually indicates that the email template includes a variable that is either not valid or missing.-or-(If you don't have Kinesis streaming activated) Review your campaign's Amazon Simple Email Service (Amazon SES) Rendering Failures metric in Amazon CloudWatch. This metric also usually indicates that the email template includes a variable that is either not valid or missing.If you see streaming events with an _email.rendering_failure event type, or any Amazon SES Rendering Failures metrics in CloudWatch, do the following:1.    Verify that all of the message variables in the template file have a corresponding endpoint attribute in the endpoint definition (segment file).2.    Verify that all of the message variables in the template file are in the correct format in the endpoint definition.3.    Specify the Default attribute values for all of the message variables in the template file. For more information, see Creating email templates in the Amazon Pinpoint User Guide.Note: If you don't specify a default value and a value doesn't exist for a recipient, then Amazon Pinpoint doesn't send the message.For more information, see Adding personalized content to message templates in the Amazon Pinpoint User Guide.Follow"


Question: "Why does my Amazon SageMaker notebook instance get stuck in the Pending state, and then fail?"

"When I create or start an Amazon SageMaker notebook instance, the instance enters the Pending state. The notebook instance appears to be stuck in this state, and then it fails."

Answer: "When I create or start an Amazon SageMaker notebook instance, the instance enters the Pending state. The notebook instance appears to be stuck in this state, and then it fails.Short descriptionThe Pending status means that SageMaker is creating the notebook instance. If any step in the creation process fails, SageMaker attempts to create the notebook again. This is why a notebook might stay in the Pending state longer than expected. If SageMaker still can't create the notebook instance, the status eventually changes to Failed .ResolutionConfirm the failure reasonCheck the  FailureReason response in the DescribeNotebookInstance API. You can also find the failure reason on the SageMaker console:To see a pop-up window that shows a shortened version of the failure reason, pause on Failed in the Status column.To see the full failure reason, choose the name of the notebook instance. The failure reason appears at the top of the Notebook instance settings section.Use the failure reason to troubleshoot the root cause.Common errors"fatal: unable to access 'https://github.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/': Failed to connect to github.com port 443: Connection timed out"This error happens when the networking configuration for the notebook instance doesn't support the domain name or connection for the external Git repository.Important: Notebook instances that are deployed in a Virtual Private Cloud (VPC) don't automatically inherit custom route tables, like subnet route tables for VPC peering connections. If you need a custom route table, create a lifecycle configuration script that adds the route on startup. For more information, see Understanding Amazon SageMaker notebook instance networking configurations and advanced routing options.To validate that the Git connection is active and that you can connect to the repository from a notebook instance: Create a new notebook instance without an associated Git repository. Then, open the Jupyter console and use a terminal session to run the following commands:1.FSPResolve the hostname of the server:dig repo_hostnameIf the answer section of the output is empty, the notebook wasn't able to resolve the hostname. For example, the answer section for github.com displays as:;; ANSWER SECTION:github.com.    16  IN     A   20.248.137.482.FSPIf the answer section of the output contains a response, the domain name resolution works. You can then run the following command to test the connection to the hostname:curl -v your-git-repo-url:4433.FSPIf the connection is refused or times out, verify the VPC security group rules and route tables. If the connection is successful, use git commands to test your credentials:git pull https://your-git-repo-url"Lifecycle Configuration failed"If a lifecycle configuration script runs for longer than five minutes, it fails, and the notebook instance is neither created nor started. For suggestions on how to decrease script runtime, see Customize a notebook instance using a lifecycle configuration script. To troubleshoot issues with the script, check the Amazon CloudWatch logs for the lifecycle configuration:Log group: /aws/sagemaker/NotebookInstancesLog stream: notebook-instance-name/LifecycleConfigOnStart or notebook-instance-name/LifecycleConfigOnCreate"This Notebook Instance type 'ml.m4.xlarge' is temporarily unavailable. We apologize for the inconvenience. Please try again in a few minutes, or try a different instance type."This error happens when Amazon Elastic Compute Cloud (Amazon EC2) doesn't have enough available capacity for the instance type that you selected. Capacity varies based on the demand for that instance type in that Region at that time. Try the request again later to see if capacity levels have changed. Or, choose a different instance type.HTTP 500 internal errorsAn HTTP 500 error indicates that an unexpected error occurred while creating the notebook instance. To rule out transient issues, try creating the notebook instance again.Related informationAssociate Git repositories with SageMaker notebook instancesCommon errorsFollow"


Question: How can I identify and resolve unwanted health checks from Route 53?

My server is receiving unwanted requests from Amazon Route 53 health check servers.

Answer: "My server is receiving unwanted requests from Amazon Route 53 health check servers.Short descriptionWhen you associate health checks with an endpoint, Amazon Route 53 sends health check requests to the endpoint IP address. These health checks validate that the endpoint IP addresses are operating as intended. An issue might occur if an incorrect IP address is specified or if a health check isn't updated or deleted when necessary.ResolutionIdentify the source of unwanted requests1.    Use Route 53 IP address ranges to find the source IP address of the unwanted request. For more information, see ROUTE53_HEALTHCHECKS in IP address ranges of Route 53 servers.2.    Check the application server logs to determine whether the Route 53 health check servers sent the request. When performing health checks, Route 53 health checks set the following HTTP header:"Amazon-Route53-Health-Check-Service (ref <reference ID/ b5996862-d894-4595-88da-7940808e9665>; report http://amzn.to/1vsZADi)"Example Application Load Balancer access log:http 2020-05-12T14:14:25.000265Z app/myapplicationloadbalancer 54.241.32.97:49816 10.0.3.64:80 -1 -1 -1 502 - 241 288 "GET http:// <ALB DNS NAME>:80/ HTTP/1.1" "Amazon-Route53-Health-Check-Service (ref b5996862-d894-4595-88da-7940808e9665; report http://amzn.to/1vsZADi)" - - arn:aws:elasticloadbalancing:us-east-1:<account ID>:targetgroup/mytargetgroupExample Microsoft Internet Information Services (IIS) access log:Amazon+Route+53+Health+Check+Service;+ref:b5996862-d894-4595-88da-7940808e9665;+report+http://amzn.to/1vsZADiExample Apache access log:54.228.16.1 - - [time] "GET / HTTP/1.1" 403 3839 "-" "Amazon Route 53 Health Check Service; ref:47d9bc51-39d6-4cd9-9a7f-4c981c5db165; report http://amzn.to/1vsZADi"Example NGINX access log:NGINX access log entry: 54.232.40.80 - - [time] "GET / HTTP/1.1" 200 3770 "-" "Amazon Route 53 Health Check Service; ref:2e44063d-3b85-47c3-801e-6748cd542386; report http://amzn.to/1vsZADi" "-"Delete or block the source of unwanted requests1.    Copy the health check ID from the application service logs.2.    If the health check is available from your AWS account, then update the health check to monitor the intended IP address or domain name. Or, if it's no longer required, then delete the health check.If the health check isn't available from your AWS account, then block the IP address of the health check. To block the IP address, use firewall rules, security groups, or network access control lists (NACLs).Important: To report suspected Route 53 health check abuse, see Stop unwanted Amazon Route 53 health checks.Related informationHow can I stop Route 53 health check requests that are sent to my application?Follow"


Question: How do I troubleshoot a circuit breaker exception in Amazon OpenSearch Service?

"I'm trying to send data to my Amazon OpenSearch Service cluster. However, I receive a circuit breaking exception error that states that my data is too large. Why is this happening and how do I resolve this?"

Answer: "I'm trying to send data to my Amazon OpenSearch Service cluster. However, I receive a circuit breaking exception error that states that my data is too large. Why is this happening and how do I resolve this?Short descriptionWhen a request reaches OpenSearch Service nodes, circuit breakers estimate the amount of memory needed to load the required data. OpenSearch Service then compares the estimated size with the configured heap size limit. If the estimated size of your data is greater than the available heap size, then the query is terminated. As a result, a CircuitBreakerException is thrown to prevent overloading the node.OpenSearch Service uses the following circuit breakers to prevent JVM OutofMemoryError exceptions:RequestFielddataIn flight requestsAccountingParentNote: It's important to know which of these five circuit breakers raised the exception because each circuit breaker has its own tuning needs. For more information about circuit breaker types, see Circuit breaker settings on the Elasticsearch website.To obtain the current memory usage per node and per breaker, use the following command:GET _nodes/stats/breakerAlso, note that circuit breakers are only a best-effort mechanism. While circuit breakers provide some resiliency against overloading a node, you might still end up receiving an OutOfMemoryError. Circuit breakers can track memory only if it's explicitly reserved, so estimating the exact memory usage upfront isn't always possible. For example, if you have a small amount of memory heap, the relative overhead of untracked memory is larger. For more information about circuit breakers and node resiliency, see Improving node resiliency with the real memory circuit breaker on the Elasticsearch website.To avoid overloading your data nodes, follow the tips provided in the Troubleshooting high JVM memory pressure section.ResolutionCircuit breaker exceptionIf you're using Elasticsearch version 7.x and higher with 16 GB of heap, then you receive the following error when the circuit breaker limit is reached:{    "error": {        "root_cause": [{            "type": "circuit_breaking_exception",            "reason": "[parent] Data too large, data for [<http_request>] would be [16355096754/15.2gb], which is larger than the limit of [16213167308/15gb]",            "bytes_wanted": 16355096754,            "bytes_limit": 16213167308        }],        "type": "circuit_breaking_exception",        "reason": "[parent] Data too large, data for [<http_request>] would be [16355096754/15.2gb], which is larger than the limit of [16213167308/15gb]",        "bytes_wanted": 16355096754,        "bytes_limit": 16213167308    },    "status": 503}This example output indicates that the data to be processed is too large for the parent circuit breaker to handle. The parent circuit breaker (a circuit breaker type) is responsible for the overall memory usage of your cluster. When a parent circuit breaker exception occurs, the total memory used across all circuit breakers has exceeded the set limit. A parent breaker throws an exception when the cluster exceeds 95% of 16 GB, which is 15.2 GB of heap.You can verify this logic by calculating the difference between memory usage and set circuit breaker limit. Use the values from our example output, and subtract "real usage: [15283269136/14.2gb]" from "limit of [16213167308/15gb]". This calculation shows that this request needs around 1.02 GB of new bytes reserved memory to successfully process the request. However, in this example, the cluster had less than 0.8 GB of available free memory heap when the data request came in. As a result, the circuit breaker trips.The circuit breaker exception message can be interpreted like this:data for [<http_request>]: Client sends HTTP requests to a node in your cluster. OpenSearch Service either processes the request locally or passes it onto another node for additional processing.would be [#]: How the heap size looks when the request is processed.limit of [#]: Current circuit breaker limit.real usage: Actual usage of the JVM heap.new bytes reserved: Actual memory needed to process the request.JVM memory pressureA circuit breaking exception is often caused by high JVM memory pressure. JVM memory pressure refers to the percentage of Java heap that is used for all data nodes in your cluster. Check the JVMMemoryPressure metric in Amazon CloudWatch to determine current usage.Note: JVM heap size of a data node is set to half the size of physical memory (RAM), up to 32 GB. For example, if the physical memory (RAM) is 128 GB per node, the heap size will still be 32 GB (the maximum heap size). Otherwise, heap size is calculated as half the size of physical memory.High JVM memory pressure can be caused by following:Increase in the number of requests to the cluster. Check the IndexRate and SearchRate metrics in Amazon CloudWatch to determine your current load.Aggregation, wildcards, and using wide time ranges in your queries.Unbalanced shard allocation across nodes or too many shards in a cluster.Index mapping explosions.Using the fielddata data structure to query data. Fielddata can consume a large amount of heap space, and remains in the heap for the lifetime of a segment. As a result, JVM memory pressure remains high on the cluster when fielddata is used. For more information, see Fielddata on the Elasticsearch website.Troubleshooting high JVM memory pressureTo resolve high JVM memory pressure, try the following tips:Reduce incoming traffic to your cluster, especially if you have a heavy workload.Consider scaling the cluster to obtain more JVM memory to support your workload.If cluster scaling isn't possible, try reducing the number of shards by deleting old or unused indices. Because shard metadata is stored in memory, reducing the number of shards can reduce overall memory usage.Enable slow logs to identify faulty requests.Note: Before enabling configuration changes, verify that JVM memory pressure is below 85%. This way, you can avoid additional overhead to existing resources.Optimize search and indexing requests, and choose the correct number of shards. For more information about indexing and shard count, see Get started with OpenSearch Service: How many shards do I need?Disable and avoid using fielddata. By default, fielddata is set to "false" on a text field unless it's explicitly defined as otherwise in index mappings.Change your index mapping type to a keyword, using reindex API or create or update index template API. You can use the keyword type as an alternative for performing aggregations and sorting on text fields.Avoid aggregating on text fields to prevent increases in field data. When you use more field data, more heap space is consumed. Use the cluster stats API operation to check your field data.Clear the fielddata cache with the following API call:POST /index_name/_cache/clear?fielddata=true (index-level cache)POST */_cache/clear?fielddata=true (cluster-level cache)Warning: If you clear the fielddata cache, any in-progress queries might be disrupted.For more information, see How do I troubleshoot high JVM memory pressure on my OpenSearch Service cluster?Related informationBest practices for Amazon OpenSearch ServiceHow can I improve the indexing performance on my Amazon OpenSearch Service cluster?Follow"


Question: Why can't my Secrets Manager rotation function connect to an Aurora PostgreSQL database using scram-sha-256?

My AWS Secrets Manger rotation function can't connect to an Amazon Aurora PostgreSQL database using the scram-sha-256 algorithm.

Answer: "My AWS Secrets Manger rotation function can't connect to an Amazon Aurora PostgreSQL database using the scram-sha-256 algorithm.Short descriptionIf your database is Aurora PostgreSQL version 13 or later, the rotation function might fail to connect to the database if:The database uses scram-sha-256 to encrypt passwords.The rotation function uses libpq-based client's version 9 or earlier.Important: If you set up automatic secret rotation before December 30, 2021, then your rotation function bundled an older version of libpq that doesn't support scram-sha-256.ResolutionFollow these steps to check for database users for scram-sha-256 encryption and the rotation function libpq version.Determine which database users use scram-sha-256 encryptionTo check for users with scram-sha-256 encrypted passwords, see the AWS blog SCRAM authentication in Amazon Relational Database Service for PostgreSQL 13.Determine what version of libpq your rotation function uses1.    Open the Lambda console.2.    In the navigation pane, choose Functions, and then select the Lambda function name that failed to rotate.3.    Choose the Code tab.4.    Choose Actions, choose Export function, and then choose Download deployment package.5.    Uncompress the zip file into the work directory.6.    Run the following Linux command in the work directory:readelf -a libpq.so.5 | grep RUNPATHIf you see the string PostgreSQL-9.4.x, or any major version less than 10, then the rotation function doesn't support scram-sha-256.Example output for a rotation function that doesn't support scram-sha-256:0x000000000000001d (RUNPATH) Library runpath: [/local/p4clients/pkgbuild-a1b2c/workspace/build/PostgreSQL/PostgreSQL-9.4.x_client_only.123456.0/AL2_x86_64/DEV.STD.PTHREAD/build/private/tmp/brazil-path/build.libfarm/lib:/local/p4clients/pkgbuild-a1b2c/workspace/src/PostgreSQL/build/private/install/lib]    * Example output for a rotation function that supports scram-sha-256:Example output for a rotation function that supports scram-sha-256:0x000000000000001d (RUNPATH) Library runpath: [/local/p4clients/pkgbuild-a1b2c/workspace/build/PostgreSQL/PostgreSQL-10.x_client_only.123456.0/AL2_x86_64/DEV.STD.PTHREAD/build/private/tmp/brazil-path/build.libfarm/lib:/local/p4clients/pkgbuild-a1b2c/workspace/src/PostgreSQL/build/private/install/lib]If your database uses scram-sha-256 and the example output indicate that the rotation function doesn't support scram-sha-256, then you must recreate your rotation function.Related informationTroubleshoot AWS Secrets Manager rotationFollow"


Question: How can I calculate my burst credit billing on my EC2 instance?

I want to calculate my burst credit billing on my Amazon Elastic Compute Cloud (Amazon EC2) instance. How can I do this?

Answer: "I want to calculate my burst credit billing on my Amazon Elastic Compute Cloud (Amazon EC2) instance. How can I do this?ResolutionNote: Burst credit billing applies to EC2 burstable instance types only. For example, T4g, T3a, and T3 instance types, and the previous generation T2 instance types.You can configure burstable instances to run in one of two modes: standard mode or unlimited mode.You can check whether your burstable performance instance is configured as unlimited or standard using the EC2 console or the AWS Command Line Interface (AWS CLI). For more information, see View the credit specification of a burstable performance instance and View the default credit specification.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.Billing in standard modeIn standard mode, you're charged using the standard  On-Demand hourly rate for your instance.Billing in unlimited modeIn unlimited mode, the hourly instance price automatically covers all CPU usage spikes for the shorter of the following:The time that the instance's average CPU utilization is at or below the baseline over a rolling 24-hour period.The instance's lifetime.If the instance runs at a higher CPU utilization for a prolonged period, it can do so for a flat additional rate per vCPU-Hour. For information on vCPU-Hour charges, see T2/T3/T4g Unlimited Mode Pricing.Credits that are spent by an instance in unlimited mode after it depletes its accrued credit balance are called surplus credits. For information on when surplus credit charges occur, see Surplus credits can incur charges.You can use the Amazon CloudWatch metric CPUSurplusCreditsCharged to see surplus credits that are charged for your instance. For more information, see  Additional CloudWatch metrics for burstable performance instances.For examples of how EC2 burstable instances in unlimited mode spend credits and are charged, see Unlimited mode examples.Related informationKey concepts and definitions for burstable performance instancesFollow"


Question: How can I automatically attach a persistent secondary EBS volume to a new EC2 Linux Spot Instance at boot?

I want to use a user data script to automatically launch a persistent secondary Amazon Elastic Block Store (Amazon EBS) volume to my new Amazon Elastic Compute Cloud (Amazon EC2) Linux Spot Instance at boot. How do I do this?

Answer: "I want to use a user data script to automatically launch a persistent secondary Amazon Elastic Block Store (Amazon EBS) volume to my new Amazon Elastic Compute Cloud (Amazon EC2) Linux Spot Instance at boot. How do I do this?Short descriptionTo automatically attach a persistent secondary EBS volume to a new EC2 Linux Spot Instance at boot, add a user data script to an EC2 launch template. Use the template when configuring your Spot Instance Request.PrerequisiteCreate or use an AWS Identity and Access Management (IAM) role that at a minimum has attach-volume access granted for Amazon EC2. This role will be attached to the launch template.ResolutionStep 1: Configure a launch template with an IAM Role and user data script1.    Open the Amazon EC2 console.2.    Select Launch Templates, and then select Create launch template.3.    Choose the instance AMI, type, and size. Or, choose an existing AMI.4.    Associate a key pair with the template.5.    Choose a subnet in the same Availability Zone as the EBS volume.6.    Select Advanced details.7.    Add the IAM role that at a minimum has attach-volume access granted, as shown in the following example:{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "ec2:AttachVolume",                "ec2:DetachVolume"            ],            "Resource": [                "arn:aws:ec2:*:*:instance/*",                "arn:aws:ec2:*:*:volume/vol-xxxxxxxxxxxx"            ]        },        {            "Effect": "Allow",            "Action": "ec2:DescribeVolumes",            "Resource": "arn:aws:ec2:*:*:volume/vol-xxxxxxxxxxxx"        }    ]8.    Add a user data script to the template. The following is an example user data script. Replace the region and volume-id to match your environment.#!/bin/bash      OUTPUT=$(curl http://169.254.169.254/latest/meta-data/instance-id)      aws ec2 attach-volume --volume-id vol-xxxxxxxxxxxx --device /dev/xvdf --instance-id $OUTPUT --region ap-southeast-1Step 2: Configure a Spot Request using the launch template created in Step 11.    Select Spot Instance, and then select Request Spot Instance.2.    Select Launch Templates, and then choose the Launch Template created in Step 1. All the information configured on the template auto-populates.3.    Choose the same Availability Zone as the EBS volume.4.    Select create Spot Request.After the Spot request completes, the persistent secondary EBS volume is attached to the new Spot Instance automatically at boot.Follow"


Question: How can I set up a private network connection between a file gateway and Amazon S3?

I want to set up a private network connection between AWS Storage Gateway's file interface (file gateway) and Amazon Simple Storage Service (Amazon S3). I don't want my gateway to communicate with AWS services over the internet. How can I do that?

Answer: "I want to set up a private network connection between AWS Storage Gateway's file interface (file gateway) and Amazon Simple Storage Service (Amazon S3). I don't want my gateway to communicate with AWS services over the internet. How can I do that?Short descriptionYou can set up a private network connection between a file gateway and Amazon S3 within an Amazon Virtual Private Cloud (Amazon VPC) where the gateway appliance connects with service endpoints over an internal private network. To set up this private connection within a VPC, do the following:Create either a VPC gateway endpoint or an interface endpoint for Amazon S3.Create a file gateway using a VPC endpoint.Amazon S3 File Gateway supports two Amazon S3 endpoints. However, you need to create only one type of endpoint based on your use case.Note: Amazon S3 gateway endpoints can't be used with on-premises gateways. An Amazon S3 gateway endpoint is used with Amazon EC2 instance-based gateways. Amazon S3 interface endpoints can be used with both on-premises and EC2 instance-based gatewaysResolutionCreate a VPC gateway endpoint for Amazon S3Open the Amazon VPC console.In the navigation pane, choose Endpoints.Choose Create Endpoint.For Service category, select AWS services.For Service Name, select the Service Name that ends with s3 and has Type as Gateway.For VPC, select the VPC that you want to use when accessing Storage Gateway.For Configure route tables, select the Route Table ID for your configuration.Choose Create endpoint.When using Gateway VPC endpoints, VPC endpoint policies are used to restrict access and only allow requests to S3 buckets from authorized users. In addition, you can control which buckets are accessible from a particular VPC. This is the best practices model for accessing S3 from a VPC in the same Region. To use a Gateway VPC endpoint from on-premises applications, or to access S3 from a VPC in a different AWS Region, you must have set up a fleet of proxy servers with private IP addresses in your VPC. This results in changes to your on-premises applications so that they direct requests to the proxy servers, and then forward them to S3 through your VPC endpoint.Create a VPC interface endpoint for Amazon S3Open the Amazon VPC console.In the navigation pane, choose Endpoints.Choose Create Endpoint.For Service category, select AWS services.For Service Name, select the service name that ends with s3 and has Type as Interface.For VPC, select the VPC and subnets that you want to use when accessing Storage Gateway.For Security group, select the security group where port 443 is opened.Choose Create endpoint.Create a file gateway using the VPC endpointTo create a file gateway using a VPC endpoint, you must create a VPC endpoint for Storage Gateway, create and configure a file gateway and activate your gateway in a VPC.Note: If you're using on-premises Storage Gateway using a private connectivity with AWS, then you can use interface endpoint for Amazon S3 that works without an Amazon Elastic Compute Cloud (Amazon EC2) proxy.Create file share using the VPC interface endpoint for Amazon S3With Amazon S3 File Gateway, you can create a file share that can be accessed using either the Network File System (NFS) or Server Message Block (SMB) protocol. For more information on creating a file share, see Creating a file share.Test the network connectivityNote: Testing the connectivity helps you to check if the Storage Gateway appliance can connect with the service endpoint over the required TCP port.Connect to the file gateway's Amazon EC2 host instance using SSH.In the SSH session, enter 3 to select 3: Test Network Connectivity.The tests return [ PASSED ] for a successful network connection.Related informationUse cases (AWS PrivateLink and VPC endpoints)Performing maintenance tasks on the local consoleSecure hybrid access to Amazon S3 using AWS PrivateLinkFollow"


Question: How can I share CloudHSM clusters with other AWS accounts?

My organization has multiple AWS accounts. How can I share my AWS CloudHSM clusters with these AWS accounts?

Answer: "My organization has multiple AWS accounts. How can I share my AWS CloudHSM clusters with these AWS accounts?Short descriptionYou can use AWS Resource Access Manager to share subnets for the Amazon Virtual Private Cloud (Amazon VPC) containing your CloudHSM with other AWS accounts.ResolutionUse AWS RAM to access CloudHSM with another AWS account in AWS Organizations. In the following example, Account 1 contains the CloudHSM cluster, and Account 2 contains the CloudHSM client instance.Use AWS RAM to enable sharingWith your Organizations management account, open the AWS RAM console in the same Region as your CloudHSM, and choose Settings.Select the Enable sharing within your AWS Organization check box.With your Organizations management account, open the AWS Organization Console.Choose Settings, and note the Organization ID.Create a resource share with Account 1 for with other accountsOpen the AWS RAM console with Account 1 in the same Region as your CloudHSM.In the navigation pane, in Shared by me, choose Resource shares.Choose Create resource share.In Name, enter a name for the resource share.In Resources, choose the Amazon VPC subnet ID for your CloudHSM.In Principals, uncheck select Allow external accounts.In the Add AWS account number search pane, enter the Organization ID, choose Add, and then choose Create resource share.Note: You can also share Organizational Units (OUs) and AWS accounts.Configure the security group to allow the CloudHSM client to connect to the CloudHSM clusterOpen the CloudHSM console with Account 1 in the same Region as your CloudHSM cluster.In the navigation pane, choose Clusters.In Cluster ID, choose the CloudHSM cluster that you want to share.In Security group, choose the security group.Choose the Inbound tab, and then choose Edit.Choose Add Rule.In Port Range, enter 2223-2225.In Source, enter the private IP address of your client instance, and then choose Save.Note: To get the client instance private IP address, see view the IPv4 addresses using the EC2 console.Create client instances for the subnets shared with Account 2Open the Amazon EC2 console with Account 2, choose Launch Instance, and then select an Amazon Machine Image (AMI).Choose Next: Configure Instance Details.In Network, choose the Amazon VPC that's shared with Account 2.In Subnet, choose the subnet that's shared with Account 2.In Auto-assign Public IP, choose Enable, and then choose Next: Add Storage.Choose Next: Add Tags, and then choose Next: Configure Security Group.In Assign a security group, choose either Create a new security group or Select an existing security group (depending on your instance type).Choose Review and Launch, and then choose Launch.Choose an existing key pair or create a new one (depending on your instance type), and then select the agreement check box.Choose Launch Instances.Related informationSharing your AWS resourcesWorking with security groupsShared subnets permissionsFollow"


Question: Why is the AWS Glue crawler running for a long time?

"The AWS Glue crawler has been running for several hours or longer, and is still not able to identify the schema in my data store."

Answer: "The AWS Glue crawler has been running for several hours or longer, and is still not able to identify the schema in my data store.Short descriptionHere are some common causes of long crawler run times:Frequently adding new data: During the first crawler run, the crawler reads the first megabyte of each file to infer the schema. During subsequent crawler runs, the crawler lists files in the target, including files that were crawled during the first run, and reads the first megabyte of new files. The crawler doesn't read files that were read in the previous crawler run. This means that subsequent crawler runs are often faster. This is due to the incremental crawl feature, if activated. With this option, crawler only reads new data in subsequent crawl runs. However, when you add a lot of files or folders to your data store between crawler runs, the run time increases each time.**Crawling compressed files:**Compressed files take longer to crawl. That's because the crawler must download the file and decompress it before reading the first megabyte or listing the file.Note: For Apache Parquet, Apache Avro, and Apache Orc files, the crawler doesn't crawl the first megabyte. Instead, the crawler reads the metadata stored in each file.ResolutionBefore you start troubleshooting, consider whether or not you need to run a crawler. Unless you need to create a table in the AWS Glue Data Catalog and use the table in an extract, transform, and load (ETL) job or a downstream service, such as Amazon Athena, you don't need to run a crawler. For ETL jobs, you can use from_options to read the data directly from the data store and use the transformations on the DynamicFrame. When you do this, you don't need a crawler in your ETL pipeline. If you determine that a crawler makes sense for your use case, use one or more of the following methods to reduce crawler run times.Use an exclude patternAn exclude pattern tells the crawler to skip certain files or paths. Exclude patterns reduce the number of files that the crawler must list, making the crawler run faster. For example, use an exclude pattern to exclude meta files and files that have already been crawled. For more information, including examples of exclude patterns, see Include and exclude patterns.Use the sample size featureThe AWS Glue crawler supports the sample size feature. With this feature, you can specify the number of files in each leaf folder to be crawled when crawling sample files in a dataset. When this feature is turned on, the crawler randomly selects some files in each leaf folder to crawl instead of crawling all the files in the dataset. If you have previous knowledge about your data formats and know that schemas in your folders do not change, then use the sampling crawler. Turning on this feature significantly reduces the crawler run time.Run multiple crawlersInstead of running one crawler on the entire data store, consider running multiple crawlers. Running multiple crawlers for a short amount of time is better than running one crawler for a long time. For example, assume that you are partitioning your data by year, and that each partition contains a large amount of data. If you run a different crawler on each partition (each year), the crawlers complete faster.Combine smaller files to create larger onesIt takes more time to crawl a large number of small files than a small number of large files. That's because the crawler must list each file and must read the first megabyte of each new file.Related informationHow do I resolve the "Unable to infer schema" exception in AWS Glue?Why does the AWS Glue crawler classify my fixed-width data file as UNKNOWN when I use a built-in classifier to parse the file?Follow"


Question: How do I get SSM agent logs for Fargate tasks that have Amazon ECS Exec activated?

"I want to get AWS Systems Manager Agent (SSM Agent) logs for AWS Fargate tasks that have Amazon Elastic Container Service (Amazon ECS) Exec activated. But, I don't know how."

Answer: "I want to get AWS Systems Manager Agent (SSM Agent) logs for AWS Fargate tasks that have Amazon Elastic Container Service (Amazon ECS) Exec activated. But, I don't know how.Short descriptionBefore you start using Amazon ECS Exec, see Prerequisites for using ECS Exec.To get SSM agent logs for Fargate tasks that have ECS Exec activated, create an Amazon Elastic File System (Amazon EFS) file system. Then, mount the Amazon EFS file system on the Fargate container. Finally, mount the same file system on an Amazon Elastic Compute Cloud (Amazon EC2) instance to get the SSM agent logs.Important: Your Amazon EFS file system, Amazon ECS cluster, and Fargate tasks must all be in the same Amazon Virtual Private Cloud (Amazon VPC).Note: The following resolution is only for Fargate tasks that have ECS Exec activated. Use the resolution steps solely for debugging. Launch it as a standalone task, or keep your desiredCount of tasks to "1" in your Amazon ECS service to avoid overriding logs. You can also use the following resolution for scenarios where you must check non stderr/stdout logs from the containers.ResolutionCreate your Amazon EFS file system and mount it on a Fargate container in a Task or Service.Create your Amazon EFS file system.Note the Amazon EFS ID and security group ID.Edit your file system security group rules to allow inbound connections on port 2049 from the security group that's associated with your Fargate task.Update your Amazon ECS security group to allow outbound connections on port 2049 to your file system's security group.Open the Amazon ECS console, and navigate to your Amazon ECS task definition.In the Volumes section, choose Add volume.For Name, enter a name for your volume.For Volume type, enter "EFS".For File system ID, enter the ID for your file system.In the Containers definition section, navigate to the STORAGE AND LOGGING section, and select the volume that you created for the source volume.For Container path, select /var/log/amazon.Update the Fargate service or task with the task definition that you created.Mount the Amazon EFS file system on an Amazon EC2 instance and get the SSM Agent logs1.    Mount your file system on an EC2 instance.2.    Run the following command to get the log data:sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-01b0bxxxxxxxx.efs.ap-southeast-1.amazonaws.com:/ /efsExample output:# df -hFilesystem                                          Size  Used Avail Use% Mounted onfs-01b0bxxxxxxxx.efs.us-west-2.amazonaws.com:/      8.0E     0  8.0E   0% /efsThe following is an example of logs stored at path /var/log/amazon/ssm/amazon-ssm-agent.log in the Fargate container:[root@ip-172-31-32-32 efs]# cd ssm/[root@ip-172-31-32-32 ssm]# lsamazon-ssm-agent.log  audits[root@ip-172-31-32-32 ssm]# cat amazon-ssm-agent.log | tail -n 102022-10-20 11:50:34 INFO [ssm-agent-worker] [MessageService] [MessageHandler] ended idempotency deletion thread2022-10-20 11:50:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread started2022-10-20 11:50:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread done2022-10-20 11:55:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread started2022-10-20 11:55:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread done2022-10-20 12:00:34 INFO [ssm-agent-worker] [MessageService] [MessageHandler] started idempotency deletion thread2022-10-20 12:00:34 WARN [ssm-agent-worker] [MessageService] [MessageHandler] [Idempotency] encountered error open /var/lib/amazon/ssm/170b15cacf5846ed836bcd7903cbee48-2531612879/idempotency: no such file or directory while listing replies in /var/lib/amazon/ssm/170b15cacf5846ed836bcd7903cbee48-2531612879/idempotency2022-10-20 12:00:34 INFO [ssm-agent-worker] [MessageService] [MessageHandler] ended idempotency deletion thread2022-10-20 12:00:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread started2022-10-20 12:00:37 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] send failed reply thread done[root@ip-172-31-32-32 ssm]#Follow"


Question: How do I make sure that my client I/O isn't disrupted because of security patches?

I want to know some best practices for maintaining high availability in MSK clusters during security patching.

Answer: "I want to know some best practices for maintaining high availability in MSK clusters during security patching.Short descriptionAmazon Managed Streaming for Apache Kafka (Amazon MSK) uses rolling updates to maintain high availability, and support cluster I/O during patching. During this process, brokers are rebooted one by one, and the next broker isn't rebooted until the partitions on the current rebooted broker fully catches up (in-sync). It's normal to see transient disconnect errors on your clients during this update process.To prevent clients from experiencing downtime during security patching, use the following best practices to make the clusters highly available.ResolutionSet up a three-AZ clusterIf an Availability Zone fails, then a three-AZ cluster guards against any downtime.Amazon MSK sets the broker.rack broker property to achieve a rack-aware replication assignment for fault tolerance at the Availability Zone level. This means that when you use a three-AZ cluster with a replication factor (RF) of three, each of the three partition replicas is in a separate Availability Zone.Note: Having a two-AZ cluster with an RF-3 doesn’t allow each of the three partition replicas to be in a separate Availability Zone. Amazon MSK doesn't allow you to create a cluster in a single Availability Zone.Make sure that the replication factor is the Availability Zone countWhen you restart a broker during security patching, the leader becomes unavailable. As a result, one of the follower replicas gets elected as the new leader so that the cluster can continue to service clients.An RF-1 can lead to unavailable partitions during a rolling update because the cluster doesn’t have any replicas to promote as a new leader. An RF-2, with a minimum in-sync replica (minISR) of one, might result in data loss, even when producer acknowledgement (acks) is set to "all." For a write to be successful, a minISR of one requires only the leader to acknowledge the write. If the leader replica's broker goes down immediately after the acknowledgement but before the follower replica catches up, then data loss occurs. For more information about min.insync.replicas, see the Apache Kafka Documentation.Set minimum minISR to at most RF-1Setting minISR to the value of RF can result in producer failures when one broker is out of service because of a rolling update. If the replicas don't send an acknowledgement for the producer to write, then the producer raises an exception. For example, if AZ equals three and RF equals three, then the producer waits for all three partition replicas (including the leader) to acknowledge the messages. When one of the brokers is out of service, only two of the three partitions return the acknowledgements, resulting in producer exceptions.This scenario assumes the producer acks is set to "all." When you set producer acks to "all", the record isn't lost as long as at least one in-sync replica remains alive. For more details about producer acks, see the Apache Kafka Documentation.Include at least one broker from each AZ in the client connection stringThe client uses a single broker's endpoint to bootstrap a connection to the cluster. During the initial client connection, the broker sends metadata with information about the brokers that the client must access.When a broker becomes unavailable, the connection fails. For example, you have only one broker in a client's connection string. During patching, the client fails to establish an initial connection with the cluster when you restart the broker.Or, you have multiple brokers in a client connection string. In this case, the client's connection string allows failover when the broker that's used for establishing the connection goes offline. For more information on how to set up a connection string with multiple brokers, see Getting the bootstrap brokers for an Amazon MSK cluster.Allow retriesWhen you reboot a broker, leader partitions on that broker become unavailable. As a result, Apache Kafka promotes replica partitions on another broker as new leaders. The client now requests a metadata update to locate the new leader partitions. During this change, it's normal for your client to experience transient errors.By default, clients have retries built in to them to handle these types of transient errors. Confirm that your client is configured for retries. For more information on configuring retries, see the Apache Kafka Documentation.Follow"