Question: How do I set up AD FS as a SAML identity provider with an Amazon Cognito user pool?

I want to use Active Directory Federation Services (AD FS) as Security Assertion Markup Language 2.0 (SAML 2.0) identity provider (IdP) with an Amazon Cognito user pool. How do I set all that up?

Answer: "I want to use Active Directory Federation Services (AD FS) as Security Assertion Markup Language 2.0 (SAML 2.0) identity provider (IdP) with an Amazon Cognito user pool. How do I set all that up?Short descriptionAmazon Cognito user pools allow sign-in through a third party (federation), including through a SAML IdP such as AD FS. For more information, see Adding user pool sign-in through a third party and Adding SAML identity providers to a user pool.You can set up an AD FS server and domain controller on an Amazon Elastic Compute Cloud (Amazon EC2) Windows instance, and then integrate your setup with your user pool using Amazon Cognito's hosted web UI.Important: For this resolution, you need a domain name that you own. If you don't own a domain, you can register a new domain with Amazon Route 53 or another Domain Name System (DNS) service.ResolutionCreate an Amazon Cognito user pool with an app clientFor more information, see Tutorial: creating a user pool and setting up the hosted UI with the Amazon Cognito console.Note: When creating a user pool, the standard attribute email is selected by default. For more information about user pool attributes, see Configuring user pool attributes.Set up an EC2 Windows instanceConfigure and launch an EC2 Windows instance, and then set up an AD FS server and domain controller on it. For more information, see How do I set up AD FS on an Amazon EC2 Windows instance to work with federation for an Amazon Cognito user pool?Configure your AD FS server as SAML IdP in Amazon CognitoFor more information, see Creating and managing a SAML identity provider for a user pool (AWS Management Console) and follow the instructions under To configure a SAML 2.0 identity provider in your user pool.When creating the SAML IdP, for Metadata document, either paste the metadata document endpoint URL or upload the .xml metadata file.Map email address from IdP attribute to user pool attributeFor more information, see Specifying identity provider attribute mappings for your user pool and follow the instructions under To specify a SAML provider attribute mapping.When adding a SAML attribute, for SAML Attribute, enter http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress. For User pool attribute, choose Email from the list.Change app client settings in Amazon CognitoIn the Amazon Cognito console management page for your user pool, under App integration, choose App client settings. Then, do the following:Under Enabled identity providers, select the check box for the SAML IdP you configured. For example, ADFS.For Callback URL(s), enter a URL where you want your users to be redirected after logging in.For Sign out URL(s), enter a URL where you want your users to be redirected after logging out.Under Allowed OAuth Flows, select the Authorization code grant and Implicit Grant check boxes.Under Allowed OAuth Scopes, select all check boxes.Choose Save changes. For more information, see App client settings terminology.Test your setup using the Amazon Cognito hosted web UIEnter this URL in your web browser: https://domainNamePrefix.auth.region.amazoncognito.com/login?response_type=token&client_id=appClientId&redirect_uri=https://www.example.com Note: For the URL, use values from your user pool and app client. Find the domain (including domain name prefix and AWS Region) for your user pool in the Amazon Cognito console on the App integration page. Find the app client ID in App client settings. Replace https://www.example.com with the callback URL for your SAML IdP.Choose the name of the SAML IdP that you configured. You're redirected to the AD FS authentication page.Under Sign in with your organizational account, enter the user name and password for your Active Directory user.Choose Sign in. If sign-in is successful, then Amazon Cognito returns user pool tokens and a successful SAML response. For more information on viewing the SAML response, see How to view a SAML response in your browser for troubleshooting. Note: When decoded, the SAML response should include the required attribute NameID.Related informationBuilding ADFS federation for your web app using Amazon Cognito user poolsIntegrating third-party SAML identity providers with Amazon Cognito user poolsSAML user pool IdP authentication flowHow do I set up a third-party SAML identity provider with an Amazon Cognito user pool?Follow"


Question: How do I troubleshoot Network Load Balancer health check failures for Amazon ECS tasks on Fargate?

I want to troubleshoot Network Load Balancer health check failures that I receive when running Amazon Elastic Container Service (Amazon ECS) tasks on AWS Fargate.

Answer: "I want to troubleshoot Network Load Balancer health check failures that I receive when running Amazon Elastic Container Service (Amazon ECS) tasks on AWS Fargate.Short descriptionWhen health checks are configured, your Network Load Balancer periodically sends health check requests to each registered target. For TCP health checks, A health check simply attempts to open a TCP connection on the specified port. Failure to open a connection on the specified port within the configured timeout is considered unhealthy. For a UDP service, target availability is tested using non-UDP health checks on your target group. For HTTP and HTTPS health checks, see How do I troubleshoot Application Load Balancer health check failures for Amazon ECS tasks on Fargate?When your Amazon ECS tasks fails a Network Load Balancer health check, the following errors from your Amazon ECS service event message can appear:Health checks failed error - (service AWS-service) (port 80) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789) due to (reason Health checks failed)Target is in an Availability Zone that is not turned on for the load balancer error - (service AWS-service) (port 80) is unhealthy in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789) due to (reason Target is in an Availability Zone that is not enabled for the load balancer)Example error message from your Amazon ECS task console:Task failed ELB health checks in (target-group arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/aws-targetgroup/123456789)If you receive container health check failures, then see How do I troubleshoot the container health check failures for Amazon ECS tasks?If your Amazon ECS tasks have stopped, then see Checking stopped tasks for errors.Note: Amazon ECS tasks can return an unhealthy status for multiple reasons. If the following resolution doesn't resolve your errors, then see Troubleshooting service load balancers.ResolutionImportant: Review all AWS Command Line Interface (AWS CLI) commands and replace all instances of example strings with your specific values. For example, replace example-task-private-ip with your specific task's private IP address.Note: If you receive errors when running AWS CLI commands, then confirm that you're running a recent version of the AWS CLI.Health checks failedTo troubleshoot your load balancer health check failures on your Amazon ECS Fargate tasks, complete the following steps:Check the connectivity between your load balancer and Amazon ECS taskConfirm that your tasks respond correctly to manual checks within your Amazon Virtual Private Cloud (Amazon VPC)Check the status and configuration of the application in your Amazon ECS containerCheck the connectivity between your load balancer and Amazon ECS taskMake sure that your load balancer is allowed to perform health checks on your Amazon ECS tasks:If your container is mapped to port 80, then confirm that your container security group allows inbound traffic on port 80.Make sure that the Amazon ECS Fargate ENI security group allows traffic on the Amazon VPC CIDR range. This allows the Network Load Balancer nodes to reach the Amazon ECS tasks to perform health checks. For more information, see Target security groups.Confirm that the network access control lists (ACL) associated with the subnets of the elastic network interface for your Fargate task allow ingress traffic. Ingress traffic must be allowed on the health check port. Also, confirm that the network ACL allows egress traffic on the ephemeral ports.Confirm that your tasks respond correctly to manual checks within your Amazon VPCConfirm that your Amazon Elastic Compute Cloud (Amazon EC2) instance tasks within your Amazon VPC respond correctly to manual checks:Note: You can either create a cluster for the Amazon EC2 launch type or launch the new Amazon EC2 instance. If you don't want to launch an Amazon EC2 instance, you can use the ECS exec feature. To do this, launch a standalone task in the same VPC with --enable-execute-command.Use SSH to connect to an Amazon EC2 instance within your Amazon VPC.Test your target group configuration by using one of the following options:(Option 1) For HTTP health checks:$ curl -Iv http://<example-task-private-ip>:<example-port>/<healthcheck_path>Example output:HTTP/1.1 200 OKNote: You can receive successful status codes in the range of 200-399for configurations set for HTTP health checks on the target group.(Option 2) For TCP health checks that don't use SSL with the targets:$ nc -z -v -w10 example-task-private-ip example-portExample output:nc -z -v -w10 10.x.x.x 80Connection to 10.x.x.x port 80 [tcp/http] succeeded!(Option 3) For TCP health checks that require SSL for backend health checks:$ nc -z -v -w10 --ssl example-task-private-ip example-portExample output:nc -z -v -w10 10.x.x.x 443Connection to 10.x.x.x port 443 [tcp/https] succeeded!Check the status and configuration of the application in your ECS containerCheck that the ping port and the health check path for your target group are configured correctly.Monitor the CPU and memory utilization metrics for your Amazon ECS service.If your Amazon ECS task requires a longer health check grace period for registering to the Network Load Balancer, then increase the healthCheckGracePeriodSeconds.  To update the health check grace period, run the following command:$ aws ecs update-service --cluster example-cluster --service example-service --region <example-region> --health-check-grace-period-seconds <example-value-in-seconds>Check your application logs for application errors. For more information, see Viewing awslogs container logs in CloudWatch Logs.Confirm the response code sent by your application on the HealthCheckPath. If your application has access logging configured, then check the response logged using the ELB-HealthChecker/2.0 keyword. If you're using CloudWatch Logs, then use Log Insights and run the following query:fields @timestamp, @message| sort @timestamp desc| filter @message like /ELB-HealthChecker/Target is in an Availability Zone that is not turned on for the load balancerWhen you turn on an Availability Zone (AZ) for your load balancer, load balancing creates a load balancer node in the AZ. If you register targets in an AZ, then you need to turn on the AZ so that the registered targets receive traffic. For more information, see Availability Zones and load balancer nodes.To identify the Availability Zones that your load balancer is configured for, run the following command:$ aws elbv2 describe-load-balancers --load-balancer-arn <example-arn-load-balancer> --region <example-region> --query "LoadBalancers[].AvailabilityZones[].ZoneName"Note: You can't turn off Availability Zones for a Network Load Balancer after you create it, but you can turn on additional Availability Zones.To identify the Availability Zones that your Amazon ECS Fargate task is configured for, run the following command:Note: The following command returns subnets that your service is configured for.$ aws ecs describe-services —cluster <example-cluster-name> —services <example-service-name> --region <example-region> --query "services[].networkConfiguration.awsvpcConfiguration.subnets"To identify the Availability Zones of the preceding subnets, use the preceding subnet IDs in the following command:Note: The following command returns Availability Zones that your service is configured for.$ aws ec2 describe-subnets --subnet-ids <example-subnet-ids> --region <example-region> --query "Subnets[].AvailabilityZone"Note: You can change the subnet configuration of an Amazon ECS service using the AWS CLI update-service commandRelated informationTroubleshooting service load balancersHealth checks for your target groupsFollow"


Question: How do I install and configure the unified CloudWatch agent to push metrics and logs from my EC2 instance to CloudWatch?

I want to use the unified Amazon CloudWatch agent to push metrics and logs from my Amazon Elastic Compute Cloud (Amazon EC2) instance to CloudWatch. How can I do this?

Answer: "I want to use the unified Amazon CloudWatch agent to push metrics and logs from my Amazon Elastic Compute Cloud (Amazon EC2) instance to CloudWatch. How can I do this?Short descriptionRegardless of your platform (Windows or Linux), the steps to install the unified Amazon CloudWatch agent are the same:Create an AWS Identity and Access Management (IAM) role to run the CloudWatch agent. Then, attach that IAM role to the EC2 instance.Download and install the CloudWatch agent.Create an agent configuration file that specifies the metrics/logs that you plan to push to CloudWatch from your EC2 instance.Start the CloudWatch agent using the configuration file that you created in step 3.ResolutionNote: Before you begin, be sure that you established internet connectivity in your EC2 instance. Internet connectivity is necessary to connect your EC2 instance to the required endpoints.Installing the unified CloudWatch agent (Amazon Linux or Amazon Linux 2)1.    Create an IAM role to run the CloudWatch agent on your EC2 instance:      Open the IAM console.In the navigation pane, choose Roles.Choose Create role.For Choose the service that will use this role, choose EC2.Choose Next: Permissions.In the list of policies, select the CloudWatchAgentServerPolicy check box.Choose Next: Tags, and then choose Next: Review.For Role name, enter a name for the role, such as CloudWatchAgentServerRole.(Optional) Provide a role description.Confirm that CloudWatchAgentServerPolicy appears next to Policies.Choose Create role.Attach this newly created IAM role to the EC2 instance.2.    Download and install the unified CloudWatch agent on your EC2 instance:Download:wget https://s3.<region>.amazonaws.com/amazoncloudwatch-agent-<region>/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpmInstall:sudo rpm -U ./amazon-cloudwatch-agent.rpm3.    Create the agent configuration file.Note: For simplicity, you can create the agent configuration file using the wizard. Later, you can manually edit the file to add or remove metrics or logs. For more information, see the following section, Tips for completing the agent configuration file wizard.Run the wizard:sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard4.    Start the CloudWatch agent:sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:configuration-file-path -sNote: Replace configuration-file-path with the configuration file path that you received in step 3.Metrics and logs are now pushed to CloudWatch. By default, the agent pushes metrics to NameSpace CWAgent.Installing the unified CloudWatch agent (Windows)1.    Follow step 1 above to create an IAM role to run the CloudWatch agent on your EC2 instance.2.    Download and install the unified CloudWatch agent on your EC2 instance:Download:https://s3.<region>.amazonaws.com/amazoncloudwatch-agent-<region>/windows/amd64/latest/amazon-cloudwatch-agent.msiMove to the directory containing the package, and then enter the following:msiexec /i amazon-cloudwatch-agent.msi3.    Create the agent configuration file:Note: For simplicity, you can create the agent configuration file using the wizard. Later, you can manually edit the file to add or remove metrics or logs. See "Tips for completing the agent configuration file wizard" below.cd "C:\Program Files\Amazon\AmazonCloudWatchAgent"amazon-cloudwatch-agent-config-wizard.exeBy default, the wizard creates the agent configuration file in the following location: C:\Program Files\Amazon\AmazonCloudWatchAgent\config.json.4.    Start the CloudWatch agent. Enter the following in PowerShell as an administrator:& $Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1 -a fetch-config -m ec2 -c file:<configuration-file-path> -s      Example command:& $Env:ProgramFiles\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1 -a fetch-config -m ec2 -c file:'C:\Program Files\Amazon\AmazonCloudWatchAgent\config.json' -sMetrics and logs are now pushed to CloudWatch. By default, the agent pushes metrics to NameSpace CWAgent.Tips for configuring the agent configuration file wizardDo you want to turn on StatsD daemon? and Do you want to monitor metrics from CollectD? – Only choose Yes for these questions if you have the StatsD and CollectD plugins installed on your EC2 instance.Which default metrics configuration do you want? – For more information on the different detail levels, see CloudWatch agent predefined metric sets. Then, choose the option that's best for your use case.Do you have an existing CloudWatch Logs Agent configuration file to import for migration? – This question is asking whether you already have the awslogs agent installed and running. If you do, the wizard reads the awslogs agent configuration file (/etc/awslogs/awslogs.conf or /var/awslogs/etc/awslogs.conf) to determine the log files you need to monitor.Do you want to store the configuration in the Parameter Store, a capability of AWS Systems Manager? – Choose Yes if you want to store this agent configuration file in the Parameter Store so that you can reuse the file later. Important: The command for starting the agent varies when the agent configuration file is stored in the Parameter Store.When you complete the wizard, the agent configuration file is stored in /opt/aws/amazon-cloudwatch-agent/bin/config.json by default. You can manually edit this file to add or remove metrics or logs.Starting and stopping the CloudWatch agentFor steps on starting and stopping the CloudWatch agent in Linux and Windows, see Stopping and restarting the CloudWatch Agent.Note: When modifying the agent configuration file, start the agent using the fetch-config command to be sure that the agent reflects your changes.Follow"


Question: How can I restrict an IAM user or role to specific attributes in a DynamoDB table?

I want to make sure that an AWS Identity and Access Management (IAM) entity can access only specific attributes in an Amazon DynamoDB table.

Answer: "I want to make sure that an AWS Identity and Access Management (IAM) entity can access only specific attributes in an Amazon DynamoDB table.ResolutionUse IAM policy conditions for fine-grained access control. Fine-grained access control lets you control access to individual items or item attributes for both read and write operations in a DynamoDB table.Use predefined AWS keys and DynamoDB keys to specify conditions in an access policy. Here are some examples of common use cases for DynamoDB condition keys:dynamodb:LeadingKeys: Allows a user to access items that have a specific partition key value. Allows a user to access only items with a partition key value that matches a unique user ID.For example, you can use a substitution variable, such as ${www.amazon.com:user_id}. For more information on substitution variables, see Using web identity federation.Note: You can also use ''dynamodb:LeadingKeys'': ["${aws:PrincipalArn}"] to match with the role's ARN Example: arn:aws:iam::Account-ID:role/myRole and "dynamodb:LeadingKeys": ["${aws:userid}"] to match with role-id:caller-specified-role-name. For information on using aws:userid for other principals, see Request information that you can use for policy variables.dynamodb:Select: Allows a query or scan operation on an index to return a response only if the operation requests the index's projected attributes. Allows only operations that request item count rather than the actual items.dynamodb:Attributes: Allows only requests that request specific attributes. For PutItem, allows requests only when the item has specific attributes in addition to the primary key and sort key attributes.dynamodb:ReturnValues: For UpdateItem, allows only requests that have ReturnValues set to NONE (default) or UPDATED_OLD (returns updated attributes, as they appear before the UpdateItem operation).dynamodb:ReturnConsumedCapacity: Allows only requests that have ReturnConsumedCapacity set to NONE or requests that don't specify ReturnConsumedCapacity.Example: Allow read-only access to certain attributes of a table's itemsThe table in this example has the following properties:Table name: GameScoresPrimary partition key: a string with a UserId attributePrimary sort key: a string with a GameTitle attributeNon-key attributes: PersonalDetails, Wins, Losses, TopScore, TopScoreDateTimeYou allow a third-party application to read the TopScore and TopScoreDateTime attributes for each user. Because UserId and GameTitle are the partition and sort key attributes, you must also allow these attributes in the IAM policy. The example policy must meet the following criteria:For query or scan API operations: Allow requests only when the Select parameter is set to SPECIFIC_ATTRIBUTES or when ProjectionExpression contains UserId, GameTitle, TopScore, or TopScoreDateTime.For GetItem, BatchGetItem, and TransactGetItem API operations: Allow requests only when the ProjectionExpression parameter contains UserId, GameTitle, TopScore, or TopScoreDateTime.Note: If the application is reading a secondary index for this table, then the policy must allow the partition and sort keys for secondary index.The following example IAM policy uses the dynamodb:Attributes condition key:{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "LimitAccessToSpecificAttributes",            "Effect": "Allow",            "Action": [                "dynamodb:GetItem",                "dynamodb:Query",                "dynamodb:BatchGetItem",                "dynamodb:Scan",                "dynamodb:TransactGetItems"            ],            "Resource": [                "arn:aws:dynamodb:eu-west-1:123456789012:table/GameScores"            ],            "Condition": {                "ForAllValues:StringEquals": {                    "dynamodb:Attributes": [                        "TopScoreDateTime",                        "TopScore",                        "UserId",                        "GameTitle"                    ]                },                "StringEquals": {                    "dynamodb:Select": "SPECIFIC_ATTRIBUTES"                }            }        }    ]}The client's requests must specify the allowed attributes. If the requests don't specify attributes, then DynamoDB tries to return all attributes. Because the IAM policy doesn't allow the client to read all attributes, the client gets an AccessDeniedException.Here's an example of a query request that returns an AccessDeniedException because it doesn't use Select or ProjectionExpression to specify allowed attributes:$ aws dynamodb query --table-name GameScores --key-condition-expression "UserId = :useridval" --expression-attribute-values '{":useridval":{"S":"stefano_123"}}'Here's an example of a successful query request that specifies the TopScore, TopScoreDateTime, and GameTitle attributes:$ aws dynamodb query --table-name GameScores --key-condition-expression "UserId = :useridval" --expression-attribute-values '{":useridval":{"S":"stefano_123"}}' --projection-expression "TopScore, TopScoreDateTime, GameTitle"Here's an example of a successful GetItem request:$ aws dynamodb get-item --table-name GameScores --key '{"UserId":{"S":"stefano_123"},"GameTitle":{"S":"Game Zero"}}' --projection-expression "UserId, GameTitle, TopScore, TopScoreDateTime"For another example of how to use condition keys to limit access, see Permissions use case.Related informationFine-grained access control for Amazon DynamoDBAmazon DynamoDB API referenceFollow"


Question: How do I install libraries in my Amazon MWAA environment?

I want to install libraries in my Amazon Managed Workflows for an Apache Airflow (Amazon MWAA) environment.

Answer: "I want to install libraries in my Amazon Managed Workflows for an Apache Airflow (Amazon MWAA) environment.Short descriptionUse the requirements.txt and plugins.zip files to install Python libraries in Amazon MWAA. If you use the requirements.txt file to install packages, then the packages are installed from the Python Package Index (from the PyPI website) by default. If you ship libraries (.whl files) with compiled artifacts, then use the plugins.zip file to install these Python wheels.For Amazon MWAA versions 2.2.2 and 2.4.3, outbound internet access isn't available on the private web server option. Amazon MWAA installs requirements and plugins on the web server in the Amazon MWAA service virtual private cloud (VPC). These requirements include a NAT gateway with outbound internet access. However, for earlier Amazon MWAA versions (2.0.2 and 1.10.12), requirements and plugins aren't installed on the web server by default.For Amazon MWAA versions 2.2.2 and 2.4.3, the Amazon MWAA local runner provides a command line utility to download and package Python dependencies (.whl) and requirements in the plugins.zip file.For all versions of Amazon MWAA, you can use the plugins.zip file to install custom Apache Airflow operations, hooks, sensors, or interfaces. Plugins export environment variables, authentication, and config files such as, .crt and .yaml.ResolutionInstall libraries using .whl on environments with a private web server (Amazon MWAA versions 2.2.2 and 2.4.3)Set up your Amazon MWAA local environment1.    Build the Docker image and set up an Amazon MWAA local environment (from the GitHub website). MWAA repository provides a command line interface (CLI) utility that replicates an Amazon MWAA environment locally.2.    Add the Python library and dependences to a requirements.txt file. 3.    Use the following script to test the requirements.txt file:#aws-mwaa-local-runner % ./mwaa-local-env test-requirementsThe output looks similar to the following one:  Installing requirements.txtCollecting aws-batch (from -r /usr/local/airflow/dags/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/5d/11/3aedc6e150d2df6f3d422d7107ac9eba5b50261cf57ab813bb00d8299a34/aws_batch-0.6.tar.gzCollecting awscli (from aws-batch->-r /usr/local/airflow/dags/requirements.txt (line 1))  Downloading https://files.pythonhosted.org/packages/07/4a/d054884c2ef4eb3c237e1f4007d3ece5c46e286e4258288f0116724af009/awscli-1.19.21-py2.py3-none-any.whl (3.6MB)    100% |████████████████████████████████| 3.6MB 365kB/s .........Installing collected packages: botocore, docutils, pyasn1, rsa, awscli, aws-batch  Running setup.py install for aws-batch ... doneSuccessfully installed aws-batch-0.6 awscli-1.19.21 botocore-1.20.21 docutils-0.15.2 pyasn1-0.4.8 rsa-4.7.2Build the .whl files from the requirements.txt fileRun the following package-requirements local-runner command to build the .whl files from the requirements.txt file:#aws-mwaa-local-runner % ./mwaa-local-env package-requirementsThe command downloads all .whl files into the folder: aws-mwaa-local-runner/plugin.Create a plugins.zip file including .whl files and an Amazon MWAA constraintDownload and copy the constraints.txt into the plugin's directory. Then, run the following command to create the plugins.zip file: #aws-mwaa-local-runner % curl -o plugins/constraints.txt "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt"#aws-mwaa-local-runner % zip -j dags/plugins.zip plugins/constraints.txtCreate a new requirements.txt file that points to the .whl files that are packaged in the plugins.zip file1. Create the new requirements.txt file. In a text tool of your choice, use a format similar to the following one:=========new requirements.txt==========--find-links /usr/local/airflow/plugins--no-index--constraint “/usr/local/airflow/plugins/constraints.txt”aws-batch==0.6====================================2. Upload the plugins.zip files and requirements.txt files to the Amazon Simple Storage Service (Amazon S3) bucket of your MWAA cluster. Then, update the environment.Use Python wheels to install custom librariesA Python wheel is a package file with compiled artifacts. To install this package, place the .whl file in a plugins.zip file. Then, refer to this file in a requirements.txt file. After you add the .whl file to the plugins.zip file, update the environment. The .whl file is shipped to the  Amazon Elastic Container Service (Amazon ECS) Fargate container's location: /usr/local/airflow/plugins/. Install Python wheels1.    Create the plugins.zip file. Run the following command to create a local Amazon MWAA plugin directory on your system:$ mkdir plugins2.    Copy the .whl file into the plugins directory that you created. Run the following command to change the directory to point to your local Airflow plugins directory:$ cd pluginsRun the following command to confirm that the contents have executable permissions:plugins$ chmod -R 755Run the following command to zip the contents within your plugin folder:plugins$ zip -r plugins.zip .3.    Include the path of the .whl file in the requirements.txt file (for example, /usr/local/airflow/plugins/example_wheel.whl).Note: Be sure to turn on versioning for your Amazon S3 bucket.4.    Upload the plugins.zip and requirements.txt files to an Amazon S3 bucket (for example, s3://example-bucket/plugins.zip).5.    Specify the plugins.zip version on the Amazon MWAA console.Install custom operators, hooks, sensors, or interfacesAmazon MWAA supports Apache Airflow’s built-in plugin manager. The plugin manager allows you to use custom Apache Airflow operators, hooks sensors, or interfaces. These custom plugins are placed in the plugins.zip file with either a flat and nested directory structure. The contents of plugins.zip file are written to the backend Amazon ECS Fargate containers at: /usr/local/airflow/plugins/. For more information, see Examples of custom plugins.Create a custom plugin to generate runtime environment variablesCreate a custom plugin that generates runtime environment variables on your Amazon MWAA environment. Then, use these environment variables in your Directed Acyclic Graph (DAG) code. For more information, see Creating a custom plugin that generates runtime environment variables.Export PEM, .crt, and configuration files (.yaml)If you don't need to update certain files continuously when your environment is running, then use the plugins.zip file to distribute the files. You can also create DAGs and use the plugins.zip for files that don't require user access (for example, certificate, PEM, and configuration YAML files).The following example command bundles a ca-certificates.crt file into a plugins.zip file:$ zip plugins.zip ca-certificates.crtAfter you update the environment with this plugins.zip file, the .crt file is synced on the path /usr/local/airflow/plugins/ca-certificates.crt on each of the worker containers. Then, your DAGs access this file. Follow the same process for other files types.Install custom plugins and binariesYou can use Oracle for Amazon MWAA to create a custom plugin, and combine it with other custom plugins and binaries in your plugins.zip file. This includes non-Python packages. For more information, see Creating a custom plugin with Oracle.To add binaries to your containers, or to set and modify environment variables, see Creating a custom plugin with Apache Hive and Hadoop.Troubleshoot package installationUse aws-mwaa-local-runner (from the GitHub website) to test DAGs, custom plugins, and Python dependencies.View the log file. You can view the file from the Apache Airflow Worker or Scheduler log groups.Important: Before you install the packages or plugins.zip file, it's a best practice to use the Amazon MWAA CLI utility to test Python dependencies and the plugins.zip file.Related informationPluginsFollow"


Question: How do I troubleshoot common issues with Amazon EKS node group update failures?

I want to update my Amazon Elastic Kubernetes Service (Amazon EKS) node groups using the newest Amazon Machine Image (AMI) versions.

Answer: "I want to update my Amazon Elastic Kubernetes Service (Amazon EKS) node groups using the newest Amazon Machine Image (AMI) versions.Short descriptionNewer releases of Amazon EKS also include new versions of Amazon AMIs for node group updates. Customers with their workload deployed over multiple node groups face the challenge of updating their nodes to keep up with their release cycle.When you initiate a managed node group update, Amazon EKS automatically updates your nodes for you. If you're using an Amazon EKS-optimized AMI, then Amazon EKS automatically applies the latest security patches and operating system updates to your nodes as part of the latest AMI release version. To implement the update, AWS Auto Scaling launches the nodes in every Availability Zone where the nodes are present in the node group. This service also rebalances the Availability Zone. Because the existing nodes drain only once, the launch step succeeds. The scale-down phase decrements the Auto Scaling group's maximum size and desired size by one to return to the values before the update. See "Scale down phase" in Managed node update behavior for more information.ResolutionDuring this update process, you might see some of the following errors that require their own mitigation steps. Being aware of these issues in advance lets you minimize downtime. See Managed node update behavior for more information on update errors.Update failed due to PodEvictionFailureError message : Reached max retries while trying to evict pods from nodes in node group.This error indicates that the upgrade is blocked by PodEvictionFailure. If the pods don't leave the node within 15 minutes and there's no force flag, then the upgrade phase fails with a PodEvictionFailure error.The following are reasons for PodEvictionFailure error in the upgrade phase:Aggressive PDB (Pod Disruption Budget)Aggressive PDB is defined on the pod when there are multiple PDBs pointing to the same pod.PDB indicates the number of disruptions that can be tolerated at a given time for a class of pods (a budget of faults). Whenever a voluntary disruption causes the pods for a service to drop below the budget, the operation pauses until it can maintain the budget. The node drain event halts temporarily until more pods become available so that the budget isn’t crossed by evicting the pods. For more information, see Disruptions on the Kubernetes website.To confirm a smooth managed node group update, you must either remove the pod disruption budgets temporarily or use the force option to update. This option doesn't respect pod disruption budgets. Instead, this option implements the updates by forcing the nodes to restart.Note: If the app is a Quorum-based application, then the force option can cause the application to become temporarily unavailable.Run the following command to confirm that you have PDBs configured in your cluster:$ kubectl get pdb --all-namespacesOr, if you turned on audit logging in the Amazon EKS console, follow these steps:1.    Under the Clusters tab, choose the desired cluster (for example, rpam-eks-qa2-control-plane) from the list.2.    Under the Logging tab, choose Audit. This redirects you to the Amazon CloudWatch console.3.    In the CloudWatch console, choose Logs. Then, choose Log Insights and run the following query:fields @timestamp, @message| filter @logStream like "kube-apiserver-audit"| filter ispresent(requestURI)| filter objectRef.subresource = "eviction" | display @logStream, requestURI, responseObject.message| stats count(*) as retry by requestURI, responseObject.message4.    Select Custom from the top right to identify the date for the update. If there’s a node group update failure due to aggressive PDB, resposeObject.message describes the reason for the pod eviction failure.5.    If PDB caused the failure, modify the PDB using the following command. Then, upgrade the node group again:$ kubectl edit pdb pdb_name;Deployment tolerating all the taintsAfter every pod is evicted, the node becomes empty because the node was tainted in the earlier steps. However, if the deployment tolerates every taint, then the node is more likely to be non-empty, leading to a pod eviction failure. See  Taints and tolerations on the Kubernetes website for more information.Update failed due to non-valid release versionError Message: Requested release version 1.xx is not valid for kubernetes version 1.xx.To resolve this issue, run the upgrade command again. This command upgrades the node groups to the same version as the control plane's Kubernetes version:eksctl upgrade nodegroup --name=test --cluster=test --kubernetes-version=1.xxNote: Replace 1.xx version with the version supported by Amazon EKS control plane.Update failed as node group has health issuesError message: Nodegroup has health issue other than [ AsgInstanceLaunchFailures, InstanceLimitExceeded, InsufficientFreeAddresses, ClusterUnreachable]This failure occurs if you manually modified the Auto Scaling group of the node group to use a different version of its Amazon Elastic Compute Cloud (Amazon EC2) launch template. Or, you might have deleted the version of the template associated with the node group. The EKS node group uses a default launch template that conflicts with the Auto Scaling group's launch template. This launch template causes EKS to show the node group as degraded.If you haven't deleted the launch template yet, manually change the launch template version of the Auto Scaling group back to the appropriate version. This action reverses the node group to a healthy and active state. You can now reinitiate the update process.Follow"


Question: How do I correlate the query plan with the query report in Amazon Redshift?

I'm trying to correlate the query plan with the query report in my Amazon Redshift cluster. How can I do that?

Answer: "I'm trying to correlate the query plan with the query report in my Amazon Redshift cluster. How can I do that?Short descriptionTo determine the usage required to run a query in Amazon Redshift, use the EXPLAIN command. The EXPLAIN command displays the execution plan for a query statement without actually running the query. The execution plan outlines the query planning and execution steps involved.Then, use the SVL_QUERY_REPORT system view to view query information at a cluster slice level. You can use the slice-level information for detecting uneven data distribution across the cluster, which impacts query performance.Note: In the SVL_QUERY_REPORT, the rows column indicates the number of rows that are getting processed per cluster slice. The rows_pre_filter column indicates the total number of rows emitted before filtering the rows marked for deletion.Amazon Redshift processes the query plan and translates the plan into steps, segments, and streams. For more information, see Query planning and execution workflow.ResolutionCreating a table and fetching the explain plan and SVL query report for the query1.   Create two tables with different sort keys and distribution keys.2.    Run the following query where join was not performed on a distribution key:select eventname, sum (pricepaid) from sales, event where sales.eventid = event.eventid group by eventname order by 2 desc;This query distributes the inner table to all compute nodes.3.    Retrieve the query plan:EXPLAIN <query>;                                               QUERY PLAN                                               -------------------------------------------------------------------------------------------------------- XN Merge  (cost=1002815368414.24..1002815368415.67 rows=571 width=27)   Merge Key: sum(sales.pricepaid)   ->  XN Network  (cost=1002815368414.24..1002815368415.67 rows=571 width=27)         Send to leader         ->  XN Sort  (cost=1002815368414.24..1002815368415.67 rows=571 width=27)               Sort Key: sum(sales.pricepaid)               ->  XN HashAggregate  (cost=2815368386.67..2815368388.10 rows=571 width=27)                     ->  XN Hash Join DS_BCAST_INNER  (cost=109.98..2815367496.05 rows=178125 width=27)                           Hash Cond: ("outer".eventid = "inner".eventid)                           ->  XN Seq Scan on sales  (cost=0.00..1724.56 rows=172456 width=14)                           ->  XN Hash  (cost=87.98..87.98 rows=8798 width=21)                                 ->  XN Seq Scan on event  (cost=0.00..87.98 rows=8798 width=21)(12 rows)4.    Run the following query:select * from svl_query_report where query = <query_id> order by segment, step, elapsed_time, rows;Mapping the query plan with the query report1.    Run the following query to obtain the svl_query_report:select query,slice,segment,step,start_time,end_time,elapsed_time,rows,bytes,label from svl_query_report where query = 938787 and segment = 0 order by segment, step, elapsed_time, rows;EXPLAIN <query>;->  XN Hash  (cost=87.98..87.98 rows=8798 width=21)   ->  XN Seq Scan on event  (cost=0.00..87.98 rows=8798 width=21)Here's an example output: query  | slice | segment | step |         start_time         |         end_time          | elapsed_time | rows | bytes  |            label              --------+-------+---------+------+----------------------------+---------------------------+--------------+------+--------+------------------------------ 938787 |     0 |       0 |    0 | 2020-05-22 11:11:48.828309 | 2020-05-22 11:11:48.82987 |         1561 | 4383 | 128626 | scan   tbl=278788 name=event  938787 |     1 |       0 |    0 | 2020-05-22 11:11:48.828309 | 2020-05-22 11:11:48.82987 |         1561 | 4415 | 128918 | scan   tbl=278788 name=event  938787 |     0 |       0 |    1 | 2020-05-22 11:11:48.828309 | 2020-05-22 11:11:48.82987 |         1561 | 4383 |      0 | project                       938787 |     1 |       0 |    1 | 2020-05-22 11:11:48.828309 | 2020-05-22 11:11:48.82987 |         1561 | 4415 |      0 | project                       938787 |     0 |       0 |    2 | 2020-05-22 11:11:48.828309 | 2020-05-22 11:11:48.82987 |         1561 | 4383 | 126660 | bcast                         938787 |     1 |       0 |    2 | 2020-05-22 11:11:48.828309 | 2020-05-22 11:11:48.82987 |         1561 | 4415 | 126920 | bcast                         (6 rows)This output indicates that when the segment value is 0, Amazon Redshift performs a sequential scan operation to scan the event table.2.    Run the following query to obtain the query report of segment 1:select query,slice,segment,step,start_time,end_time,elapsed_time,rows,bytes,label from svl_query_report where query = 938787 and segment = 1 order by segment, step, elapsed_time, rows;Here's an example output:query  | slice | segment | step |       start_time          |          end_time          | elapsed_time | rows | bytes  |     label           --------+-------+---------+------+----------------------------+----------------------------+--------------+------+--------+-------------------------------------------938787 |     1 |       1 |    0 | 2020-05-22 11:11:48.826864 | 2020-05-22 11:11:48.830037 |         3173 |    0 |      0 | scan   tbl=376297 name=Internal Worktable  38787 |     0 |       1 |    0 | 2020-05-22 11:11:48.826864 | 2020-05-22 11:11:48.831142 |         4278 | 8798 | 253580 | scan   tbl=376297 name=Internal Worktable 938787 |     1 |       1 |    1 | 2020-05-22 11:11:48.826864 | 2020-05-22 11:11:48.830037 |         3173 |    0 |      0 | project                                   938787 |     0 |       1 |    1 | 2020-05-22 11:11:48.826864 | 2020-05-22 11:11:48.831142 |         4278 | 8798 |      0 | project                                   938787 |     1 |       1 |    2 | 2020-05-22 11:11:48.826864 | 2020-05-22 11:11:48.830037 |         3173 |    0 |      0 | hash   tbl=439                            938787 |     0 |       1 |    2 | 2020-05-22 11:11:48.826864 | 2020-05-22 11:11:48.831142 |         4278 | 8798 | 359156 | hash   tbl=439                            (6 rows)The query continues to run until the segment value is 1 and a hash table operation is performed on the inner table in the join.3.    Run the following query to get the SVL_QUERY_REPORT for a query with a segment value of 2:EXPLAIN <query>;->  XN Hash Join DS_BCAST_INNER  (cost=109.98..2815367496.05 rows=178125 width=27)                           Hash Cond: ("outer".eventid = "inner".eventid)   ->  XN Seq Scan on sales  (cost=0.00..1724.56 rows=172456 width=14)4.    Run the following query:select query,slice,segment,step,start_time,end_time,elapsed_time,rows,bytes,label from svl_query_report where query = 938787 and segment = 2 order by segment, step, elapsed_time, rows;Here's an example output:  query  | slice | segment | step |         start_time         |          end_time          | elapsed_time | rows  |  bytes  |            label             --------+-------+---------+------+----------------------------+----------------------------+--------------+-------+---------+------------------------------ 938787 |     1 |       2 |    0 | 2020-05-22 11:11:48.839297 | 2020-05-22 11:11:48.865857 |        26560 | 86519 | 1730380 | scan   tbl=278792 name=sales  938787 |     0 |       2 |    0 | 2020-05-22 11:11:48.838371 | 2020-05-22 11:11:48.865857 |        27486 | 85937 | 1718740 | scan   tbl=278792 name=sales  938787 |     1 |       2 |    1 | 2020-05-22 11:11:48.839297 | 2020-05-22 11:11:48.865857 |        26560 | 86519 |       0 | project                       938787 |     0 |       2 |    1 | 2020-05-22 11:11:48.838371 | 2020-05-22 11:11:48.865857 |        27486 | 85937 |       0 | project                       938787 |     1 |       2 |    2 | 2020-05-22 11:11:48.839297 | 2020-05-22 11:11:48.865857 |        26560 | 86519 |       0 | project                       938787 |     0 |       2 |    2 | 2020-05-22 11:11:48.838371 | 2020-05-22 11:11:48.865857 |        27486 | 85937 |       0 | project                       938787 |     1 |       2 |    3 | 2020-05-22 11:11:48.839297 | 2020-05-22 11:11:48.865857 |        26560 | 86519 |       0 | hjoin  tbl=439                938787 |     0 |       2 |    3 | 2020-05-22 11:11:48.838371 | 2020-05-22 11:11:48.865857 |        27486 | 85937 |       0 | hjoin  tbl=439                938787 |     1 |       2 |    4 | 2020-05-22 11:11:48.839297 | 2020-05-22 11:11:48.865857 |        26560 | 86519 |       0 | project                       938787 |     0 |       2 |    4 | 2020-05-22 11:11:48.838371 | 2020-05-22 11:11:48.865857 |        27486 | 85937 |       0 | project                       938787 |     1 |       2 |    5 | 2020-05-22 11:11:48.839297 | 2020-05-22 11:11:48.865857 |        26560 | 86519 |       0 | project                       938787 |     0 |       2 |    5 | 2020-05-22 11:11:48.838371 | 2020-05-22 11:11:48.865857 |        27486 | 85937 |       0 | project                       938787 |     1 |       2 |    6 | 2020-05-22 11:11:48.839297 | 2020-05-22 11:11:48.865857 |        26560 |   576 |   34916 | aggr   tbl=448                938787 |     0 |       2 |    6 | 2020-05-22 11:11:48.838371 | 2020-05-22 11:11:48.865857 |        27486 |   576 |   34916 | aggr   tbl=448                938787 |     1 |       2 |    7 | 2020-05-22 11:11:48.839297 | 2020-05-22 11:11:48.865857 |        26560 |   576 |   17728 | dist                          938787 |     0 |       2 |    7 | 2020-05-22 11:11:48.838371 | 2020-05-22 11:11:48.865857 |        27486 |   576 |   17768 | dist                         (16 rows)   In this example output, the query is run when the segment value is 2, and performs a sequential scan operation to scan the sales table. In the same segment, an aggregate operation is performed to aggregate results and a hash join operation is performed to join tables. The join columns for one of the tables is not a distribution key or a sort key. As a result, the inner table is distributed to all the compute nodes as DS_BCAST_INNER, whichcan be seen in the EXPLAIN plan. 5.    Run the following query to get the SVL_QUERY_REPORT for a query with a segment value of 3:select query,slice,segment,step,start_time,end_time,elapsed_time,rows,bytes,label from svl_query_report where query = 938787 and segment = 3 order by segment, step, elapsed_time, rows;->  XN Sort  (cost=1002815368414.24..1002815368415.67 rows=571 width=27)               Sort Key: sum(sales.pricepaid)   ->  XN HashAggregate  (cost=2815368386.67..2815368388.10 rows=571 width=27)Here's an example output: query  | slice | segment | step |         start_time         |          end_time          | elapsed_time | rows | bytes |                   label                   --------+-------+---------+------+----------------------------+----------------------------+--------------+------+-------+------------------------------------------- 938787 |     1 |       3 |    0 | 2020-05-22 11:11:48.836693 | 2020-05-22 11:11:48.866529 |        29836 |  546 | 28792 | scan   tbl=376298 name=Internal Worktable  938787 |     0 |       3 |    0 | 2020-05-22 11:11:48.836654 | 2020-05-22 11:11:48.866529 |        29875 |  606 | 31824 | scan   tbl=376298 name=Internal Worktable  938787 |     1 |       3 |    1 | 2020-05-22 11:11:48.836693 | 2020-05-22 11:11:48.866529 |        29836 |  273 | 16580 | aggr   tbl=451                             938787 |     0 |       3 |    1 | 2020-05-22 11:11:48.836654 | 2020-05-22 11:11:48.866529 |        29875 |  303 | 18336 | aggr   tbl=451                             938787 |     1 |       3 |    2 | 2020-05-22 11:11:48.836693 | 2020-05-22 11:11:48.866529 |        29836 |  273 |     0 | project                                    938787 |     0 |       3 |    2 | 2020-05-22 11:11:48.836654 | 2020-05-22 11:11:48.866529 |        29875 |  303 |     0 | project                                    938787 |     1 |       3 |    3 | 2020-05-22 11:11:48.836693 | 2020-05-22 11:11:48.866529 |        29836 |  273 | 14396 | sort   tbl=453                             938787 |     0 |       3 |    3 | 2020-05-22 11:11:48.836654 | 2020-05-22 11:11:48.866529 |        29875 |  303 | 15912 | sort   tbl=453                             938787 |     1 |       3 |    4 | 2020-05-22 11:11:48.836693 | 2020-05-22 11:11:48.866529 |        29836 |    0 |     0 | merge                                      938787 |     0 |       3 |    4 | 2020-05-22 11:11:48.836654 | 2020-05-22 11:11:48.866529 |        29875 |    0 |     0 | merge                                      938787 |     1 |       3 |    5 | 2020-05-22 11:11:48.836693 | 2020-05-22 11:11:48.866529 |        29836 |    0 |     0 | aggr   tbl=456                             938787 |     0 |       3 |    5 | 2020-05-22 11:11:48.836654 | 2020-05-22 11:11:48.866529 |        29875 |    0 |     0 | aggr   tbl=456                             938787 |     1 |       3 |    6 | 2020-05-22 11:11:48.836693 | 2020-05-22 11:11:48.866529 |        29836 |    0 |     0 | project                                    938787 |     0 |       3 |    6 | 2020-05-22 11:11:48.836654 | 2020-05-22 11:11:48.866529 |        29875 |    0 |     0 | project                                   (14 rows)The query continues to run until the segment value is 3 and a hash aggregate operation and sort operation are performed. A hash aggregate operation is performed on unsorted grouped aggregate functions. The sort operation is performed to evaluate the ORDER BY clause.6.    Run the following query to get the SVL_QUERY_REPORT for a query with a segment value of 4 and 5: select query,slice,segment,step,start_time,end_time,elapsed_time,rows,bytes,label from svl_query_report where query = 938787 and segment = 4  order by segment, step, elapsed_time, rows; query  | slice | segment | step |         start_time         |          end_time          | elapsed_time | rows | bytes |                 label                 --------+-------+---------+------+----------------------------+----------------------------+--------------+------+-------+---------------------------------------- 938787 |     1 |       4 |    0 | 2020-05-22 11:11:48.915148 | 2020-05-22 11:11:48.915215 |           67 |  273 | 14396 | scan   tbl=453 name=Internal Worktable 938787 |     0 |       4 |    0 | 2020-05-22 11:11:48.915148 | 2020-05-22 11:11:48.915424 |          276 |  303 | 15912 | scan   tbl=453 name=Internal Worktable 938787 |     1 |       4 |    1 | 2020-05-22 11:11:48.915148 | 2020-05-22 11:11:48.915215 |           67 |  273 |  8888 | return                                  938787 |     0 |       4 |    1 | 2020-05-22 11:11:48.915148 | 2020-05-22 11:11:48.915424 |          276 |  303 |  8864 | return                                  938787 |  6411 |       5 |    1 | 2020-05-22 11:11:48.914501 | 2020-05-22 11:11:48.916049 |         1548 |  576 |     0 | project 938787 |  6411 |       5 |    2 | 2020-05-22 11:11:48.914501 | 2020-05-22 11:11:48.916049 |         1548 |  576 | 18360 | return 938787 |  6411 |       5 |    0 | 2020-05-22 11:11:48.914501 | 2020-05-22 11:11:48.916049 |         1548 |    0 |     0 | merge (7 rows)After all the segments are used, the query runs a network operation on segments 4 and 5 to send intermediate results to the leader node. The results are sent to the leader node for additional processing.After the query is run, use the following query to check the execution time of the query in milliseconds:select datediff (ms, exec_start_time, exec_end_time) from stl_wlm_query where query= 938787;date_diff ----------- 101(1 row)Optimizing your queryTo optimize your query while analyzing the query plan, perform the following steps:1.    Identify the steps with the highest cost.2.    Check if there are any high-cost sort operations. Note that performance of a query depends on the data distribution method along with the data being scanned by the query. Be sure to select the proper distribution style for a table to minimize the impact of the redistribution step. Additionally, use a sort key for suitable columns to improve query speed and reduce the number of blocks that need to be scanned. For more information on how to choose distribution and sort keys, see Amazon Redshift Engineering’s advanced table design playbook: distribution styles and distribution keys.The following examples use the STL_ALERT_EVENT_LOG table to identify and correct potential query performance issues:select query, btrim(event) as event, btrim(solution) as solution from stl_alert_event_log where query = 940313; query  |              event               |          solution          --------+----------------------------------+------------------------- 940313 | Missing query planner statistics | Run the ANALYZE command (1 row)In this example output, the query for the ANALYZE command can be used to improve query performance because the statistics for the query are outdated.You can also use the EXPLAIN plan to see if there are any alerts that are being populated for the query:explain select eventname, sum (pricepaid) from sales, event where sales.eventid = event.eventid group by eventname order by 2 desc;                                                QUERY PLAN                                                        ----------------------------------------------------------------------------------------------------------- XN Merge  (cost=1029210993681.72..1029210993682.22 rows=200 width=330)   Merge Key: sum(sales.pricepaid)   ->  XN Network  (cost=1029210993681.72..1029210993682.22 rows=200 width=330)         Send to leader         ->  XN Sort  (cost=1029210993681.72..1029210993682.22 rows=200 width=330)               Sort Key: sum(sales.pricepaid)               ->  XN HashAggregate  (cost=29210993673.57..29210993674.07 rows=200 width=330)                     ->  XN Hash Join DS_BCAST_INNER  (cost=109.98..29210955741.88 rows=7586340 width=330)                           Hash Cond: ("outer".eventid = "inner".eventid)                           ->  XN Seq Scan on sales  (cost=0.00..1724.56 rows=172456 width=16)                           ->  XN Hash  (cost=87.98..87.98 rows=8798 width=322)                                 ->  XN Seq Scan on event  (cost=0.00..87.98 rows=8798 width=322) ----- Tables missing statistics: event, sales ----- ----- Update statistics by running the ANALYZE command on these tables -----3.    Check the join types.Note: A nested loop is the least optimal join because it is mainly used for cross-joins and some inequality joins.The following example shows a cross-join between two tables. A nested loop join is being used and the first cost value is 0.00. This cost value is the relative cost for returning the first row of the cross-join operation. The second value (3901467082.32) provides the relative cost of completing the cross-join operation. Note the cost difference between the first and last row. The nested loops negatively impact your cluster’s performance by overloading the queue with long-running queries:explain select * from sales cross join event;                                             QUERY PLAN                                                       --------------------------------------------------------------------------------------------------------- XN Nested Loop DS_BCAST_INNER  (cost=0.00..3901467082.32 rows=1517267888 width=88)   ->  XN Seq Scan on sales  (cost=0.00..1724.56 rows=172456 width=53)   ->  XN Seq Scan on event  (cost=0.00..87.98 rows=8798 width=35) ----- Nested Loop Join in the query plan - review the join predicates to avoid Cartesian products -----(4 rows)Note: Amazon Redshift selects a join operator based on the distribution style of the table and location of the data required.To optimize the query performance, the sort key and distribution key have been changed to "eventid" for both tables. In the following example, the merge join is being used instead of a hash join:explain select eventname, sum (pricepaid) from sales, event where sales.eventid = event.eventid group by eventname order by 2 desc;                                           QUERY PLAN                                                  ------------------------------------------------------------------------------------------------ XN Merge  (cost=1000000004965.12..1000000004966.55 rows=571 width=27)   Merge Key: sum(sales.pricepaid)   ->  XN Network  (cost=1000000004965.12..1000000004966.55 rows=571 width=27)         Send to leader         ->  XN Sort  (cost=1000000004965.12..1000000004966.55 rows=571 width=27)               Sort Key: sum(sales.pricepaid)               ->  XN HashAggregate  (cost=4937.55..4938.98 rows=571 width=27)                     ->  XN Merge Join DS_DIST_NONE  (cost=0.00..4046.93 rows=178125 width=27)                           Merge Cond: ("outer".eventid = "inner".eventid)                           ->  XN Seq Scan on sales  (cost=0.00..1724.56 rows=172456 width=14)                           ->  XN Seq Scan on event  (cost=0.00..87.98 rows=8798 width=21)(11 rows)4.    Identify any broadcast operators with high-cost operations.Note: For small tables, broadcast operators aren't always considered non-optimal because the redistribution of small tables does not impact query performance as much relatively.5.    Run the following query to check the execution time of the query.select datediff (ms, exec_start_time, exec_end_time) from stl_wlm_query where query= 940715; date_diff ----------- 34 (1 row)A difference in execution time for both queries confirms that the query plan has correctly correlated to the query report.Related informationMapping the query plan to the query summaryReviewing query plan stepsUsing the SVL_QUERY_REPORT viewFollow"


Question: How do I troubleshoot DynamoDB Streams in my Lambda functions?

I want to use or troubleshoot Amazon DynamoDB Streams with my AWS Lambda functions.

Answer: "I want to use or troubleshoot Amazon DynamoDB Streams with my AWS Lambda functions.ResolutionThe following are common questions when using DynamoDB Streams with Lambda function:Why won't my Lambda function scale when the DynamoDB stream is a trigger?When you turn on a DynamoDB stream on a DynamoDB table, Amazon DynamoDB associates one shard for each partition. For example, if your DynamoDB table has 10 partitions and you turn on DynamoDB Streams on this table, then you have 10 shards.If the number of partitions in your table increases, then the number of shards in the stream also increases.Each partition on a DynamoDB table can handle up to 3000 read capacity units (RCUs), 1000 write capacity units (WCUs), and 10 GB data. Exceeding any of these parameters results in the following:The addition of a new partition to the table.The creation of a new shard in the DynamoDB stream.How can I control data processing from the DynamoDB stream?Batch size and batch window help control data processing from the stream.Batch window: Sets how long to wait for records before invoking. The batch window provides control on data processing from the DynamoDB Stream. Note that this behavior depends on the data availability within the stream.Batch size: Sets the maximum records in the batch.The Lambda function isn't invoked until the following conditions are met:The payload size reaches 6MB (synchronous invoke limit).The batch window reaches its maximum value (60 seconds, in this example).The batch size reaches its maximum value.How is Parallelization Factor used to speed up data processing?Parallelization Factor processes large amounts of records quickly by allowing more concurrent executions. You can set Parallelization Factor (default: 1 up to 10) to increase the number of shards that are processed. When you turn on Parallelization Factor, make sure to use random or unique partition keys to achieve the highest throughput.Calculation: Parallelization Factor (concurrent batches per shard) * Shards = Concurrent executionWhat is the BisectBatchonFunctionError setting?If the Lambda function fails, then the batches are split in two when the BisectBatchonFunctionError option is set to true. The split batches are then retried until the problem record is found. Retries are processed based on the maximum retry and record age settings.If the Retry attempts option is set to 0, then retries aren't attempted for the failed record. In this case, DynamoDB Stream discards failed records or sends them to the Dead Letter Queue (DLQ), if configured.Example 1In the following example 'p' represents the problem record and the Retry Attempts are set to 0.Batch record: [1,2,3p,4,5p]Split 1: [3p,4,5p]Split 2: [3p] [4,5p] The retry discards [3p] because it's identified as the problem record. Or, it's sent to the DLQ, if configured.Split 3: [4] is processed. [5p] is discarded or sent to the DLQ, if configured.Example 2In the following example, 'x' represents the problem record. Retry attempts are set to -1.Batch records inserted: [1,2,3x,4,5x][3x,4,5x][3x,4,5x][3x,4,5x][3x,4,5x]Why does IteratorAge in Lambda increase for my DynamoDB stream?The following are common reasons for the IteratorAge in Lambda to increase:There's a bad record in the DynamoDB stream.There's a high volume of write operations (PutItem to BatchWriteItem) to the stream. The Lambda function might not be able to keep up with processing a high write volume. If this occurs, then increase the DynamoDB table provisioned capacity to increase the partition count per 1000 WCUs. Increasing the provisioned capacity increases the number of concurrent Lambda executions. For more information, see the previous section Why won't my Lambda function scale when the DynamoDB Stream is a trigger?There was a drop in the number of DynamoDB partitions, such as migrating to a new account or to a new table.There's throttling or function errors in the Lambda function. AWS Lambda retries records until the entire batch successfully processes or the age of the record expires. Also, the DynamoDB Streams retention period is 24 hours. To avoid data loss, it's a best practice to set up the DLQ. If the DLQ is configured, then AWS Lambda sends failed record batches to the DLQ after retires are completed or the record age expires.To resolve Lambda function errors, check the Amazon CloudWatch Logs for details on the error.There's an increase in the Lambda function duration.You must optimize error handling and Parallelization Factor.For detailed information, see Why is my Lambda IteratorAge metric increasing, and how do I decrease it?Follow"


Question: How can I turn Service Broker on and off for Amazon RDS SQL Server?

How can I turn on and turn off Service Broker for Amazon Relational Database Service (Amazon RDS) SQL Server?

Answer:
"How can I turn on and turn off Service Broker for Amazon Relational Database Service (Amazon RDS) SQL Server?Short descriptionService Broker is supported for Amazon RDS, but Service Broker endpoints aren't supported on Amazon RDS. For more information about Service Broker, see the Microsoft documentation for Service Broker.ResolutionWhen you turn on Service Broker, Amazon RDS requests a database lock. Before turning on Service Broker, close all open connections to the database.1.    Check to see whether Service Broker is turned on:select name,is_broker_enabled from sys.databaseswhere name= [YourDB]2.    If Service Broker isn't turned on, then generate a new Service Broker:ALTER DATABASE [YourDB] SET NEW_BROKER;GO3.    Turn on Service Broker on an existing database:ALTER DATABASE [YourDB] SET ENABLE_BROKER;GOIf you run a Multi-AZ deployment, then run this command:ALTER DATABASE [YourDB] SET PARTNER OFF;GOThen, turn on Service Broker again in a Multi-AZ deployment:ALTER DATABASE [YourDB] SET ENABLE_BROKER WITH ROLLBACK IMMEDIATETo turn off Service Broker on an existing database, run this command:ALTER DATABASE [YourDB] SET DISABLE_BROKER;GOYou can't turn on Service Broker on a database that is already part of an availability group. If you run the above resolution on such a database, then you receive an error similar to the following:ALTER DATABASE [test] SET PARTNER OFF;
GO
Msg 1416, Level 16, State 1, Line 1
Database "test" is not configured for database mirroring.Instead, turn on Service Broker before adding it to the availability group. To do this, convert the RDS database to single AZ, and then turn on Service Broker.Convert your RDS database to Single AZ.Turn on Service Broker:
ALTER DATABASE [DBName ] SET NEW_BROKER;
GO

ALTER DATABASE [DBName ] SET ENABLE_BROKER;
GO
Verify that Service Broker is turned on:select name,is_broker_enabled from sys.databases
 where name= [DBName ]Convert your RDS database back to Multi-AZ.Related informationMicrosoft SQL Server on Amazon RDSMigrating Microsoft SQL Server databases to the AWS CloudFollow"


Question: How can I revoke access to a Client VPN endpoint for a specific client?

I created an AWS Client VPN endpoint with certificate-based authentication for multiple clients. How can I revoke access to the Client VPN endpoint for a specific client?


Question: How can I get my CloudFormation stack to update if it's stuck in the UPDATE_ROLLBACK_FAILED state?

My AWS CloudFormation stack is stuck in the UPDATE_ROLLBACK_FAILED state.

Answer: "My AWS CloudFormation stack is stuck in the UPDATE_ROLLBACK_FAILED state.Short descriptionIf your stack is stuck in the UPDATE_ROLLBACK_FAILED state after a failed update, then the only actions that you can perform on the stack are the ContinueUpdateRollback or DeleteStack operations. This is because CloudFormation requires further input from you to acknowledge that the stack is out of sync with the template that the stack is attempting to roll back to. To retry the rollback and resolve the error, you can use ContinueUpdateRollback.Tip: To resolve the error, you might need to raise limits, change permissions, or modify other settings. For information on common errors that cause update rollback failures, see Update rollback failed.In some cases, retrying the rollback doesn't resolve the error. In these cases, you must skip resources, and also acknowledge that these resources no longer reflect the desired state in the CloudFormation template. To skip the resources that caused the rollback failures, complete the following steps in the Resolution section.ResolutionYou can roll the stack back to a working state with either the CloudFormation console or AWS Command Line Interface (AWS CLI).Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.CloudFormation console1.    Open the CloudFormation console.2.    From the navigation pane, choose Stacks.3.    From the Stack name column, select the stack that's stuck in UPDATE_ROLLBACK_FAILED status.4.    If you don't want to skip resources, choose Stack Actions, and then choose Continue update rollback.-or-If you want to skip FAILED resources during rollback, complete the following:1.    From the Stack name column, select the stack that's stuck in UPDATE_ROLLBACK_FAILED status.2.    Choose Stack Actions, and then choose Continue update rollback.3.    In the Continue update rollback dialog box, expand Advanced troubleshooting.4.    In the Resources to skip - optional section, select the resources that you want to skip.5.    Choose Continue update rollback.AWS CLIIn the AWS CLI, run the continue-update-rollback command. For example:$ aws cloudformation continue-update-rollback --stack-name awsstackname123 --resources-to-skip awsfaultyresource123Note: Replace awsstackname123 with the name of your stack. Replace awsfaultyresource123 with the logical IDs of your resources.Important: CloudFormation sets the status of specified resources to UPDATE_COMPLETE. Then, CloudFormation continues to roll back the stack. After the rollback is complete, the state of the skipped resources is inconsistent with the state of the resources in the stack template. Before performing another stack update, update the stack or resources to be consistent with each other. If you don't do this, subsequent stack updates can fail, and the stack becomes unrecoverable.If you have nested stacks that must be rolled back to a working state, see Using ResourcesToSkip to recover a nested stacks hierarchy.Follow"


Question: How can I grant my Amazon EC2 instance access to an Amazon S3 bucket in another AWS account?

I want to use an Amazon Elastic Compute Cloud (Amazon EC2) instance to access my Amazon Simple Storage Service (Amazon S3) bucket in another account. How can I grant this access without storing credentials on the instance?

Answer: "I want to use an Amazon Elastic Compute Cloud (Amazon EC2) instance to access my Amazon Simple Storage Service (Amazon S3) bucket in another account. How can I grant this access without storing credentials on the instance?ResolutionNote: The steps for granting instance access to an Amazon S3 bucket are similar to granting access to other AWS resources in another account.Follow these steps to grant an Amazon EC2 instance in Account A access to a bucket in another account (Account B).From Account B, create an IAM role1.    Sign in to the AWS Management Console with Account B.2.    Open the AWS Identity and Access Management (IAM) console.3.    In the navigation pane, choose Roles.4.    Choose Create role.5.    For Select type of trusted entity, choose Another AWS account.6.    For Account ID, enter the account ID of Account A.7.    Choose Next: Permissions.8.    Attach a policy to the role that delegates access to Amazon S3. For example, this policy grants access for s3:GetObject on objects stored in the bucket:{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "s3:GetObject",                "s3:ListBucket"            ],            "Resource": [                "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*",                "arn:aws:s3:::DOC-EXAMPLE-BUCKET"            ]        }    ]}Note: You can modify the IAM policy based on the Amazon S3 bucket-level and object-level permissions required for your use case.9.    Choose Next: Add tags.10.    You can add optional tags to the role. Or, you can leave the fields blank, and then choose Next: Review.11.    For Role name, enter a name for the role.12.    Choose Create role.From Account B, get the role's ARN1.    From the IAM console's navigation pane, choose Roles.2.    Choose the IAM role that you created.3.    Note the value that's listed for Role ARN.From Account A, create another role (instance profile) and attach it to the instance1.    Sign in to the AWS Management Console with Account A.2.    Open the IAM console.3.    From the navigation pane, choose Roles.4.    Choose Create role.5.    For Select type of trusted entity, choose AWS service.6.    For Choose the service that will use this role, choose EC2.7.    Choose Next: Permissions.8.    Choose Next: Tags.9.    You can add optional tags to the role. Or, you can leave the fields blank, and then choose Next: Review.10.    For Role name, enter a name for the role.11.    Choose Create role.12.    From the list of roles, choose the role that you just created.13.    Choose Add inline policy, and then choose the JSON view.14.    Enter the following policy. Replace arn:aws:iam::111111111111:role/ROLENAME with the Amazon Resource Name (ARN) of the IAM role that you created in Account B.{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": "sts:AssumeRole","Resource": "arn:aws:iam::111111111111:role/ROLENAME"}]}15.    Choose Review policy.16.    For Name, enter a name for the policy.17.    Choose Create policy.18.    Attach the IAM role (instance profile) to the Amazon EC2 instance that you use to access the Amazon S3 bucket.From the Amazon EC2 instance, create a profile for the role in the CLI config fileNote: If you use SSH and Session Manager to connect to your EC2 instances, then you must perform these steps for both the ec2-user and ssm-user.1.    Connect to the Amazon EC2 instance. For more information, see Connect to your Linux instance or Connecting to your Windows instance.2.    After you connect to the instance, verify if the directory already has a folder named ~/.aws.To find the ~/.aws folder, run the following ls command to list the directory:ls -l ~/.aws3.    If you find the ~/.aws folder, then proceed to the next step. If the directory doesn't have a ~/.aws folder yet, then create the folder by running the mkdir command:mkdir ~/.aws/4.    Within the ~/.aws folder, use a text editor to create a file. Name the file config.5.    In the file, enter the following text. Replace enterprofilename with the name of the role that you attached to the instance. Then, replace arn:aws:iam::111111111111:role/ROLENAME with the ARN of the role that you created in Account B.[profile enterprofilename]role_arn = arn:aws:iam::111111111111:role/ROLENAMEcredential_source = Ec2InstanceMetadata6.    Save the file.Verify that the instance profile can assume the roleTo verify that your instance's role (instance profile) can assume the role in Account B, run the following command while connected to the instance. Replace profilename with the name of the role that you attached to the instance.$aws sts get-caller-identity --profile profilenameThe command returns a response similar to the following:"Account": "11111111111","UserId": "AROAEXAMPLEID:sessionName","Arn": "arn:aws:sts::111111111111:assumed-role/ROLENAME/sessionName"Confirm that the value for "Arn" matches the ARN of the role that you created in Account B.Verify access to the Amazon S3 bucketTo verify that your instance can access the Amazon S3 bucket, run this list command while connected to the instance. Replace profilename with the name of the role that you attached to the instance.aws s3 ls s3://DOC-EXAMPLE-BUCKET --profile profilenameIf your instance can access the bucket successfully, then you receive a response that lists the contents of the bucket, such as this:PRE Hello/2018-08-15 16:16:51 89 index.htmlRelated informationIAM Tutorial: Delegate access across AWS accounts using IAM rolesFollow"


Question: How do I troubleshoot errors with SSL certificates that are generated by API Gateway?

I'm experiencing issues with self-signed and expired SSL certificates installed on my backend system. How to I fix these errors?

Answer: "I'm experiencing issues with self-signed and expired SSL certificates installed on my backend system. How to I fix these errors?Short descriptionWhen Amazon API Gateway performs an SSL handshake with the backend, API Gateway expects the backend to provide certificates that are obtained from trusted issuers. API Gateway expects the certificates to be valid, and not expired. API Gateway also expects the chain of trust to be intact. This means that API Gateway expects the certificate to contain a root certificate authority (CA), intermediate CAs, and the parent certificate details. With this information, API Gateway can complete certificate validation by going through the chain of certificates.ResolutionTest HTTP proxy integrationTo familiarize yourself with HTTP proxy integrations, test bad SSL certificates from the API Gateway console. For more information about bad SSL certificates and for testing, see the Bad SSL website.1.    Create a resource named "/selfsigned" with a GET method. Then, configure an HTTP proxy integration with the URL https://self-signed.badssl.com/.From the API Gateway console, test the API. You receive the following error:Thu Dec 15 16:05:05 UTC 2022 : Sending request to https://self-signed.badssl.com/Thu Dec 15 16:05:05 UTC 2022 : Execution failed due to configuration error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target2.    Create a resource named "/expiredcert" with a GET method. Then, configure an HTTP proxy integration with the URL https://expired.badssl.com/.From the API Gateway console, test the API. You receive the following error:Thu Dec 15 16:06:02 UTC 2022 : Sending request to https://expired.badssl.com/Thu Dec 15 16:06:02 UTC 2022 : Execution failed due to configuration error: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed3.    Create a resource named "/untrustedRootCA" with a GET method. Configure an HTTP proxy integration with the URL https://untrusted-root.badssl.com/.From the API Gateway console, test the API. You receive the following error:Thu Dec 15 16:06:28 UTC 2022 : Sending request to https://untrusted-root.badssl.com/Thu Dec 15 16:06:28 UTC 2022 : Execution failed due to configuration error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested targetVPC link integrationWith VPC link integration, API Gateway performs certificate validation with the next hop that performs TLS termination.When a Network Load Balancer has a TLS listener, the Network Load Balancer performs a TLS termination and creates another connection to the target. The certificate attached to the Network Load Balancer must meet all the requirements. A Network Load Balancer doesn't perform certificate validation during the SSL handshake with the target. The Network Load Balancer accepts expired or self-signed certificates that are installed on the target instances. The Network Load Balancer and the target groups are bound within a VPC and communications are secure. If the Network Load Balancer is using a TCP listener, the TLS handshake happens end-to-end. In these cases, the backend application must comply with the SSL requirements.API Gateway supports Server Name Indication (SNI) during an SSL handshake over a VPC link integration.If the backend Network Load Balancer has a self-signed or private certificate that hasn't been issued by a CA, you receive the following error:Execution failed due to configuration error: PKIX path building failed:sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested targetThe workaround for the execution failed error is to set insecureSkipVerification to true in the integration's tlsConfig object:Note: Replace EXAMPLE-REST-API-ID and EXAMPLE-RESOURCE-ID with your values.aws apigateway update-integration --rest-api-id EXAMPLE-REST-API-ID --resource-id EXAMPLE-RESOURCE-ID --http-method GET --patch-operations "op='replace',path='/tlsConfig/insecureSkipVerification',value=true"Related informationGenerate and configure an SSL certificate for backend authenticationAPI Gateway-supported certificate authorities for HTTP and HTTP proxy integrationsTarget groups for your Network Load BalancersFollow"


Question: What currencies does AWS currently support?

I want to pay my bill in a currency other than US dollars (USD). What currencies does AWS support?

Answer: "I want to pay my bill in a currency other than US dollars (USD). What currencies does AWS support?ResolutionAWS generates invoices in US dollars (USD) by default. For supported payment methods, see What payment methods does AWS accept?Important: Your seller of record information and location might impact your available local currencies. For location specific information see, Amazon Web Services Tax Help.AWS supports the following currencies:AUD – Australian DollarBRL – Brazilian RealCAD – Canadian DollarCHF – Swiss FrancCNY – Chinese Yuan (China UnionPay and Bank Redirect)DKK – Danish KroneEUR – EuroGBP – British PoundHKD – Hong Kong DollarJPY – Japanese YenKRW – South Korean WonMYR – Malaysian RinggitNOK – Norwegian KroneNZD – New Zealand DollarSEK – Swedish KronaSGD – Singapore DollarUSD – United States DollarZAR – South African RandNote:Only credit card payments are supported for NOK. Invoices with terms on NOK aren't supported.Only invoices with terms are supported for SGD. Credit card payments on SGD aren't supported for Amazon Web Services Inc. (AWS Inc.)Amazon Web Services Singapore Private Limited (AWS Singapore) supports only SGD for credit cards.For instructions on how to choose a different currency for your account, see Changing which currency you use to pay your bill.Follow"


Question: Why am I unable to create or attach an encrypted EBS volume using a custom AWS KMS key?

Why am I unable to create or attach an encrypted Amazon Elastic Block Store (Amazon EBS) volume from a snapshot encrypted with an AWS Key Management Service (AWS KMS) customer managed key?

Answer: "Why am I unable to create or attach an encrypted Amazon Elastic Block Store (Amazon EBS) volume from a snapshot encrypted with an AWS Key Management Service (AWS KMS) customer managed key?Short descriptionYou might not be able to create or attach an encrypted EBS volume from an encrypted snapshot if your snapshot and the custom KMS key used to encrypt the snapshot are in the same account and are missing permissions from the key policy. Be sure that you allow the AWS Identity and Access Management (IAM) user or role in the KMS key policy.ResolutionIdentify the missing permissions from the policy using AWS CloudTrail Event history1.    Open the CloudTrail console.2.    Choose Event history, and for Time range, enter a 15-minute window around when the AttachVolume or CreateVolume API calls were made.3.    Choose Filter, select Event source, and then enter kms.amazonaws.com.4.    After the results load, select the download button in the upper-right corner of the console, and then select Download CSV.5.    Open the file that you just downloaded, and then filter the Error code column for AccessDenied. All Event names with the AccessDenied error code are usually the missing permissions.Note: CloudTrail events take up to 15 minutes to appear on the Events history tab. Checking Events history immediately after receiving the CreateVolume or AttachVolume failed status might not show any events.After identifying the missing permissions, use the following solution to fix the issue:Verify that the KMS key policy includes the IAM user or role that is attempting to attach or create the volumeFrom the AWS KMS console default view:1.    Open the AWS KMS console.2.    Choose Customer managed keys, and then select the appropriate key.3.    Under Key policy, scroll down to Key users. Verify that the Key users section lists the IAM user or role that you’re using to create the volume.4.    If the Key users section doesn't list the user or role, choose Add, select the user or role, and then choose Add.From the AWS KMS console policy view:If you've ever edited the AWS KMS key policy manually before, the key policy is only available in policy (JSON) view. Be sure that you list the ARN of the IAM user or role as Principal in the statement allowing required AWS KMS permissions.Example: Key policyThe following is an example of a KMS key policy allowing UserA access to Key1:{    "Version": "2012-10-17",    "Id": "key-consolepolicy-3",    "Statement": [        {            "Sid": "Enable IAM User Permissions",            ...        },        {            "Sid": "Allow access for Key Administrators",            ...        },        {            "Sid": "Allow use of the keys",            "Effect": "Allow",            "Principal": {                "AWS": "arn:aws:iam::111111111111:user/UserA"            },            "Action": [                "kms:Encrypt",                "kms:Decrypt",                "kms:ReEncrypt*",                "kms:GenerateDataKey*",                "kms:DescribeKey"            ],            "Resource": "*"        },        {            "Sid": "Allow attachment of persistent resources",            "Effect": "Allow",            "Principal": {                "AWS": "arn:aws:iam::111111111111:user/UserA"            },            "Action": [                "kms:CreateGrant",                "kms:ListGrants",                "kms:RevokeGrant"            ],            "Resource": "*",            "Condition": {                "Bool": {                    "kms:GrantIsForAWSResource": "true"                }            }        }    ]}Follow"


Question: How can I provide cross-account access to resources in the AWS Glue Data Catalog?

"I want to use services, such as Amazon EMR, Amazon Athena, and AWS Glue, with the AWS Glue Data Catalog in another account."

Answer: "I want to use services, such as Amazon EMR, Amazon Athena, and AWS Glue, with the AWS Glue Data Catalog in another account.ResolutionThe way that you access cross-account resources in the AWS Glue Data Catalog depends on the AWS service that you use to connect. That access method also depends on whether you use AWS Lake Formation to control access to the Data Catalog.If you aren't sure whether you're using Lake Formation and you want to share a table, run the following AWS Command Line Interface (AWS CLI) command:aws glue get-table --database-name DOC-EXAMPLE-DB --name DOC-EXAMPLE-TABLE --query 'Table.IsRegisteredWithLakeFormation'Be sure to replace the following values in this command:DOC-EXAMPLE-DB with the name of the databaseDOC-EXAMPLE-TABLE with the name of the tableIf the command returns true, then you're using Lake Formation.Note: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.The Data Catalog could be in a hybrid environment with some tables using Lake Formation and some others using AWS Glue permissions. To upgrade your Data Catalog to Lake Formation, see Upgrading AWS Glue data permissions to the AWS Lake Formation model.Note: This article covers the solution options for cross-account access within a single AWS Region. Accessing resources in a different Region is beyond the scope of this article. To replicate the Data Catalog from your account to an account in a different AWS Region, see AWS Glue Data Catalog replication utility on GitHub.Accessing the Data Catalog without Lake FormationIf you aren't using Lake Formation, then do the following to grant resource-level permissions to account A from account B's AWS Glue Data Catalog.Note: Account A is the extract, transform, and load (ETL) account, and account B is the account where the AWS Glue Data Catalog resources are located.Accessing resources with an AWS Glue ETL job1.    Attach a resource policy similar to the following in account B. This allows account A access to the databases and tables from account B. You can attach the policy using the AWS Glue console by choosing Catalog settings in the navigation pane.{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "AWS": "arn:aws:iam::1111222233334444:root"      },      "Action": "glue:*",      "Resource": [        "arn:aws:glue:us-east-1:5555666677778888:catalog",        "arn:aws:glue:us-east-1:5555666677778888:database/DOC-EXAMPLE-DB",        "arn:aws:glue:us-east-1:5555666677778888:table/DOC-EXAMPLE-DB/*"      ]    }  ]}Be sure to replace the following values in this policy:1111222233334444 with the account ID for account A5555666677778888 with the account ID for account Bus-east-1 with the Region of your choiceDOC-EXAMPLE-DB with the name of the databaseYou can also limit access to a specific role in account A that's used to run the job. Do this by including the ARN of the role in the policy. For example:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "AWS": "arn:aws:iam::1111222233334444:role/service-role/AWSGlueServiceRole-Glue-Test"      },      "Action": "glue:*",      "Resource": [        "arn:aws:glue:us-east-1:5555666677778888:catalog",        "arn:aws:glue:us-east-1:5555666677778888:database/DOC-EXAMPLE-DB",        "arn:aws:glue:us-east-1:5555666677778888:table/DOC-EXAMPLE-DB/*"      ]    }  ]}Be sure to replace the following values in this policy:1111222233334444 with the account ID for account A5555666677778888 with the account ID for account Bus-east-1 with the Region of your choiceDOC-EXAMPLE-DB with the name of the databaseAWSGlueServiceRole-Glue-Test with the ARN of the role that's used to run the ETL job2.    The AWS Identity and Access Management (IAM) user in account A that runs the ETL job needs access to the databases and tables in account B. In account A, attach an IAM policy to the AWS Glue ETL job's IAM role to access the database and tables in account B:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "glue:GetDatabase",        "glue:GetConnection",        "glue:GetTable",        "glue:GetPartition"      ],      "Resource": [        "arn:aws:glue:us-east-1:5555666677778888:catalog",        "arn:aws:glue:us-east-1:5555666677778888:database/default",        "arn:aws:glue:us-east-1:5555666677778888:database/DOC-EXAMPLE-DB",        "arn:aws:glue:us-east-1:5555666677778888:table/DOC-EXAMPLE-DB/*"      ]    }  ]}Note: If you're using Athena with the AWS Glue Data Catalog, then include the default database in the policy. This makes sure that the GetDatabase and CreateDatabase actions succeed. For more information, see Access policy to the Default database and catalog per AWS Region.Be sure to replace the following values in this policy:1111222233334444 with the account ID for account A5555666677778888 with the account ID for account Bus-east-1 with the Region of your choiceDOC-EXAMPLE-DB with the name of the database3.    After providing the required permission to account A, you can test if account A has access to the Data Catalog in account B. To test this, create an ETL job with the following script:"""Create Spark Session with cross-account AWS Glue Data Catalog"""from pyspark.sql import SparkSessionspark_session = SparkSession.builder.appName("Spark Glue Example") \.config("hive.metastore.client.factory.class", \"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory") \.config("hive.metastore.glue.catalogid", "5555666677778888") \.enableHiveSupport() \.getOrCreate()table_df = spark_session.sql("SELECT * FROM DOC-EXAMPLE-DB.DOC-EXAMPLE-TABLE limit 10")table_df.show()Be sure to replace the following values in the script:5555666677778888 with the account ID for account BDOC-EXAMPLE-DB with the name of the databaseDOC-EXAMPLE-TABLE with the name of the tableAccessing resources with Amazon EMRTo access the Data Catalog in a different account with Amazon EMR, see How can I use Hive and Spark on Amazon EMR to query an AWS Glue Data Catalog that's in a different AWS account?Accessing resources with AthenaTo access the Data Catalog in a different account with Athena, see Cross-account access to AWS Glue data catalogs.Accessing the Data Catalog with Lake FormationIf you are using Lake Formation, then you can use either of the following methods to grant cross-account access to the Data Catalog:Named resourceTag-based access control (TBAC)Important: To prevent new tables in the Data Catalog from having a default Super permission to IAMAllowedPrincipals, do the following:1.    Open the AWS Lake Formation console.2.    In the navigation pane, choose Data Catalog, and then choose Settings.3.    Clear both Use only IAM access control for new databases and Use only IAM access control for new tables in this database.4.    Choose Save.For more information, see Changing the default security settings for your data lake.Granting cross-account permissions using the named resource methodAWS Resource Access Manager (AWS RAM) is used for providing database permissions by using the named resource method.To access a resource from a different account in your organization, activate resource sharing within the organization.To access a resource in a different account that's outside of your organization, see Accessing and viewing shared Data Catalog tables and databases.To grant Lake Formation permissions to account A for the Data Catalog resources in account B, do the following:1.    Attach a resource policy similar to the following in the Data Catalog for account B:{  "Effect": "Allow",  "Action": [    "glue:ShareResource"  ],  "Principal": {    "Service": [      "ram.amazonaws.com"    ]  },  "Resource": [    "arn:aws:glue:us-east-1:5555666677778888:table/*/*",    "arn:aws:glue:us-east-1:5555666677778888:database/*",    "arn:aws:glue:us-east-1:5555666677778888:catalog"  ]}Be sure to replace 5555666677778888 in the policy with the account ID for account B.2.    Use the named resource method to grant Lake Formation permissions to the Data Catalog databases and tables in Account B. For more information, see Granting Data Catalog permissions using the named resource method.Granting cross-account permissions using TBACWith TBAC, you can define policy tags and assign these tags to AWS Glue databases, tables, and columns. These tags can then be used to apply fine-grained access to these Data Catalog resources. For more information, see Data sharing using tag-based access control.To grant Lake Formation permissions using TBAC, do the following:1.    Attach a resource policy similar to the following in Data Catalog for account B:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "AWS": "arn:aws:iam::1111222233334444:root"      },      "Action": "glue:*",      "Resource": [        "arn:aws:glue:us-east-1:5555666677778888:catalog",        "arn:aws:glue:us-east-1:5555666677778888:database/*",        "arn:aws:glue:us-east-1:5555666677778888:table/*"      ],      "Condition": {        "Bool": {          "glue:EvaluatedByLakeFormationTags": true        }      }    }  ]}Be sure to replace the following values in the policy:1111222233334444 with the account ID for account A5555666677778888 with the account ID for account B2.    Create the policy tags.3.    Use the TBAC method to grant Lake Formation permissions on Data Catalog resources. For more information, see Granting Data Catalog permissions using the LF-TBAC method.Related informationGranting cross-account accessSpecifying AWS Glue resource ARNsAbout upgrading to the Lake Formation permissions modelMigration between the Hive metastore and the AWS Glue Data CatalogAWS Glue resource policies for access controlFollow"