Question: How can I resolve access denied issues caused by permissions boundaries?

I received an access denied or unauthorized error when trying to access my AWS service. How can I troubleshoot access denied errors on my AWS account?

Answer: "I received an access denied or unauthorized error when trying to access my AWS service. How can I troubleshoot access denied errors on my AWS account?Short descriptionYou might receive an access denied or unauthorized error because your AWS Identity and Access Management (IAM) policy does not meet specific conditions requirements. First, review any service control policies (SCPs) on your account, and then check that there are no denies present in your resource-based policies. If this doesn't resolve the error, then the issue might be caused by the presence of a permissions boundary.A permissions boundary is a feature that allows you to use a managed policy to set the maximum permissions that an identity-based policy can grant to an IAM entity (user or role). When you set a permissions boundary for an entity, that entity can only perform actions that are allowed by both its identity-based policies and its permissions boundary.Note: The permissions boundary sets the maximum permissions for an entity, but does not grant those permissions.To troubleshoot authorization errors, follow these steps:Check if an action is allowed in your IAM policy but not in the permissions boundaryInclude all required actions in the permissions boundary using the IAM consoleUse the "iam:PermissionsBoundary" condition key in your IAM policyResolutionCheck if an action is allowed in your IAM policy, but not in the permissions boundaryThe following example shows an action that is allowed in an IAM policy, but not in the permissions boundary. In this example, an IAM user has the policy USER_IAM_POLICY attached to it:IAM policy:(USER_IAM_POLICY) “Effect”: “Allow”,            “Action”: [                “ec2:*”,                “s3:*”            ],This policy gives the user full access to Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3) services. The user also has a permissions boundary named USER_PB_POLICY set.Permissions Boundary:(USER_PB_POLICY) “Effect”: “Allow”,            “Action”: [                “cloudwatch:*”,                “s3:*”            ],The permissions boundary sets the maximum permissions that the user can perform. In this example, this permission boundary allows full access to Amazon CloudWatch and Amazon S3 services. But, because Amazon S3 is the only service that is allowed in both the IAM policy and the permissions boundary, the user only has access to S3. If the user tries to access Amazon EC2, they receive an access denied error.To resolve this error, edit the permissions boundary and allow access to Amazon EC2:“Effect”: “Allow”,            “Action”: [                “cloudwatch:*”,                “s3:*”,                “ec2:*”            ],Include all required actions in the permissions boundary using the IAM consoleFollow these steps to edit the permissions boundary to include all actions that a user requires:Open the IAM console.In the navigation pane, choose Roles/Users.Choose the IAM entity you want to edit.In the Permissions boundary section, check your settings. If a permissions boundary is set, this means that there is a permissions boundary in place. The name of the managed policy that is used as a permissions boundary on your IAM entity is listed in this section.Expand the JSON policy, and check if the action you require is whitelisted in the permissions boundary. If your action is not whitelisted, edit the JSON policy to allow all actions that your IAM entity requires.For more information on editing policies, Editing IAM policies.Use the iam:PermissionsBoundary condition key in your IAM policiesAdd the iam:PermissionsBoundary condition key to your IAM policies. This condition key checks that a specific policy is attached as a permissions boundary on an IAM entity.The following example shows an IAM policy named RestrictedRegionpermissionsBoundary:{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "EC2RestrictRegion",            "Effect": “Allow”,            "Action": "ec2:*”             "Resource": "*",            "Condition": {                "StringEquals": {                    "aws:RequestedRegion": [                        "us-east-1"                   ]                }            }        }Create a policy and attach it to a delegated admin who has the responsibility to create users. When we attach the following example policy to the admin, they can only create an IAM user when they attach the RestrictedRegionPermissionsBoundary policy to that user. If the admin tries to create an IAM user without attaching the policy, they receive an access denied error.{            "Sid": "CreateUser",            "Effect": "Allow",            "Action": [                "iam:CreateUser"            ],            "Resource": "arn:aws:iam::111222333444:user/test1*",            "Condition": {                "StringEquals": {                    "iam:PermissionsBoundary": "arn:aws:iam::111222333444:policy/RestrictedRegionPermissionsBoundary"                }            }To set the IAM policy RestrictedRegionPermissionsBoundary as a permissions boundary when creating a new user, follow these steps:Open the IAM console.In the navigation pane, choose Users, and then choose Add Users.Enter the user name that you want to edit, choose AWS access type, and then choose next.Expand the Set permissions boundary section, and choose Use a permissions boundary to control the maximum role permissions.In the search field, enter RestrictedRegionPermissionsBoundary, and then choose the radio button for your policy.Choose Next:Tags.Review your settings and create a user.Related informationPermissions boundaries for IAM entitiesEvaluating effective permissions with boundariesFollow"


Question: Why am I getting the "CROSSSLOT Keys in request don't hash to the same slot" error while doing multi-key operations on a Redis (cluster mode enabled) ElastiCache cluster?

Why am I getting "CROSSSLOT Keys in request don't hash to the same slot" error while doing multi-key operations on an Amazon ElastiCache for Redis (cluster mode enabled) cluster even though the keys are stored on the same node?

Answer: "Why am I getting "CROSSSLOT Keys in request don't hash to the same slot" error while doing multi-key operations on an Amazon ElastiCache for Redis (cluster mode enabled) cluster even though the keys are stored on the same node?Short descriptionThis error occurs because keys must be in the same hash slot and not just the same node. To implement multi-key operations in a sharded Redis (cluster mode enabled) ElastiCache cluster, the keys must be hashed to the same hash slot. You can force keys into the same hash slot by using hashtags.In this example, the following sets, "myset2" and "myset," are in the same node:172.31.62.135:6379> scan 01) "0"2)  1) "myset"    2) "myset2"But a multi-key operation isn't supported:172.31.62.135:6379> SUNION myset myset2(error) CROSSSLOT Keys in request don't hash to the same slotThis is because the keys aren't in the same hash slot. In this example, the two sets are in two different slots, 560 and 7967:172.31.62.135:6379> CLUSTER KEYSLOT myset(integer) 560172.31.62.135:6379> CLUSTER KEYSLOT myset2(integer) 7967ResolutionMethod 1You can use a Redis client library that provides support for Redis (cluster mode enabled) clusters. For more information about Redis clusters, see theredis-py-cluster website.For example, using redis-cli returns the CROSSSLOT error when used to get keys from slots located in different shards:redis-cli -c -h RedisclusterCfgEndpointRedisclusterCfgEndpoint:6379> mget key1 key2(error) CROSSSLOT Keys in request don't hash to the same slotUsing the redis-py-cluster to get keys from slots located in different shards returns the correct output:>>> from rediscluster import RedisCluster>>> startup_nodes = [{"host": "RedisclusterCfgEndpoint", "port": "6379"}]>>> rc = RedisCluster(startup_nodes=startup_nodes, decode_responses=True,skip_full_coverage_check=True)>>> print(rc.mget("key1","key2"))Method 2When creating keys that are used by multi-key operations on a cluster mode enabled cluster, use hashtags to force the keys into the same hash slot. When the key contains a "{...}" pattern, only the substring between the braces, "{" and "}," is hashed to obtain the hash slot.For example, the keys {user1}:myset and {user1}:myset2 are hashed to the same hash slot, because only the string inside the braces "{" and "}", that is, "user1", is used to compute the hash slot.172.31.62.135:6379> CLUSTER KEYSLOT {user1}:myset(integer) 8106172.31.62.135:6379> CLUSTER KEYSLOT {user1}:myset2(integer) 8106172.31.62.135:6379> SUNION {user1}:myset {user1}:myset21) "some data for myset"2) "some data for myset2"Now that both sets are hashed to the same hash slot, you can perform a multi-key operation.Follow"


Question: How can I register a domain in Route 53?

I want to register my domain with Amazon Route 53.

Answer: "I want to register my domain with Amazon Route 53.Short descriptionBefore registering a domain in Route 53, verify that the TLD is supported by Route 53 and review the associated cost implications. For more information, see Registering a new domain.Note: Be aware when registering your domain name that you can't change the name after you register it. If you register an incorrect domain name, you must register another domain name and specify the correct name. Refunds aren't given for a domain name that you registered incorrectly.ResolutionFor a list of TLDs that you can use to register domains with Amazon Route 53, see Domains that you can register with Amazon Route 53.For a step-by-step guide on registering domain names, see How to register a domain name with Amazon Route 53.For information on transferring a domain to Amazon Route 53, see Transferring registration for a domain to Amazon Route 53.Related informationRegistering domainsUpdating domain settingsRenewing registration for a domainRestoring an expired or deleted domainReplacing the hosted zone for a domain that is registered with Route 53Follow"


Question: How can I use an Application Load Balancer to route requests based on the source IP address?

I want to use an Application Load Balancer to perform specific actions on requests based on the source IP address of the request.

Answer: "I want to use an Application Load Balancer to perform specific actions on requests based on the source IP address of the request.ResolutionThere are several use cases for performing specific actions based on the source IP address of a request. For example, you have two versions of an application. One version is a public version that's for global users. The other is an internal version that includes some extended (beta) features. You want the internal version to be available only to employees who are accessing the application from corporate network CIDRs. To accomplish this, and other similar tasks, configure listener rules based on source IP addresses.A rule that's based on source IP address checks the source IP address in the IP header (layer-3). If there's a proxy or firewall that changes the source IP address, then specify the proxy or firewall's IP address in the listener rule.Note: Don't use listener rules to block requests from clients. It's a best practice to use security groups or network access control lists instead. To block a large number of clients, you can use AWS WAF.1.    Create an Application Load Balancer. Or, use an Application Load Balancer that you already created.2.    Open the Amazon Elastic Compute Cloud (Amazon EC2) console.3.    On the navigation pane, under Load Balancing, choose Load Balancers.4.    Select your load balancer.5.    Choose the Listeners tab.6.    Select your listener, and then choose Actions. Then, select Manage rules.7.    Choose the Add rules icon (the plus sign), and then choose Insert rule.8.    Choose Add condition, and then choose Source IP.9.    Specify the IP addresses that you plan to configure a different action for.Note: You can specify either a single IP address or network CIDRs with prefixes. For example, specify 1.1.1.1/32 or 10.8.0.0/21.10.   Choose Add action, and then select the required action. See the following examples of actions:Forward: This forwards the request to a different target group, such as a target group that runs an internal version of an application.Return fixed response: This blocks specific users or provides custom responses to specific users.12.   To save the condition, choose the checkmark icon.13.   To save the rule, choose Save.Related informationListener rules for your Application Load BalancerFollow"


Question: How can I identify Availability Zones using AWS Resource Access Manager?

I want to identify the Availability Zone (AZ) ID using AWS Resource Access Manager (AWS RAM).

Answer: "I want to identify the Availability Zone (AZ) ID using AWS Resource Access Manager (AWS RAM).Short descriptionAWS maps Availability Zones to names for each account to be sure that resources are distributed across the Availability Zones of a Region. For example, the Availability Zone us-east-1a for your AWS account might not have the same location as us-east-1a for another AWS account.ResolutionTo coordinate Availability Zones across accounts for VPC sharing, use the AZ ID. For example, use1-az1 is one of the Availability Zones in the us-east-1 Region.To find the AZ IDs for the Availability Zones in an account:Open the AWS Resource Access Manager console.In the navigation pane, choose Resource Access Manager.Under Your AZ ID, view the AZ IDs for the selected Region.Using the AZ IDs, determine the location of resources in one account relative to the resources in another account. For example, if you share a subnet in an Availability Zone with an AZ ID of use-az2 with another account, the subnet is available to that account in the Availability Zone whose AZ ID is also use-az2. You can find the AZ ID for each subnet in the Amazon Virtual Private Cloud (Amazon VPC) console.For more information, see Availability Zone IDs for your AWS resources.Related informationShare VPCs with AWS RAMVPC sharing: A new approach to multiple accounts and VPC managementHow can I map Availability Zones across my accounts?Follow"


Question: How do I transfer ownership of a domain or an Amazon Route 53 hosted zone to a different AWS account?

I want to transfer ownership of a domain or Amazon Route 53 hosted zone from one AWS account to another. How can I do this?

Answer: "I want to transfer ownership of a domain or Amazon Route 53 hosted zone from one AWS account to another. How can I do this?Short descriptionYou can transfer a domain from one AWS account to another using the TransferDomainToAnotherAwsAccount command.Although it's a best practice to transfer a domain using an API call, you can also transfer a domain by contacting AWS Support.If you don't own both the source and destination accounts that you're transferring the domain to and from, you must do one of the following:Migrate the existing hosted zone to the AWS account that you're transferring the domain to,-or-Create a new hosted zone in an AWS account that you own.If you don't own the account that created the hosted zone that routes traffic for the domain, then you can't control how traffic is routed.ResolutionMigrate a hosted zoneFollow the steps in Migrating a hosted zone to a different AWS account.Transfer a domainNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.1.    Run the following command in the AWS CLI. Replace example.com with your domain name. Replace 111122223333 with your AWS account ID.aws route53domains transfer-domain-to-another-aws-account --domain-name example.com --account-id 111122223333 --region us-east-12.    In the output, note the Password value.3.    To accept the transfer, log in to the AWS account that is the destination account. Then, run this command. Replace example.com with your domain name. Replace YourPassword with the password that you noted in step 2.aws route53domains accept-domain-transfer-from-another-aws-account --domain-name example.com --password YourPassword --region us-east-1If you receive errors accepting the transfer, see the Troubleshoot accepting a domain section of this article.Note: The accept-transfer command must be completed within three days of the transfer domain call. After three days, the transfer is canceled.4.    After accepting the transfer, view the domain by accessing the Route 53 console and then choosing the Registered Domains tab.Troubleshoot accepting a domainIf you encounter errors while accepting a transfer, then the generated password might contain special characters. Use one of two options to solve this:(Option 1) Use a text file to accept the domain transfer1.    Create a .txt file that contains the password that was generated in Step 2 of the Transfer a domain section. Use a simple format.2.    Run the accept-transfer command:aws route53domains accept-domain-transfer-from-another-aws-account --domain-name <domain name> --password file:///tmp/password.txt --region us-east-1Note: Replace  with your domain, and replace password in password.txt with the name of your file. Also, make sure that the Region is correct.(Option 2) Use quotation marks around the password stringsUse single or double quotation marks around your password to prevent the AWS CLI from misinterpreting special characters.The following example uses double quotation marks:aws route53domains accept-domain-transfer-from-another-aws-account --domain-name example.com --password "YourPassword" --region us-east-1The following example uses single quotation marks:aws route53domains accept-domain-transfer-from-another-aws-account --domain-name example.com --password 'YourPassword' --region us-east-1Related informationTransferring a domain to a different AWS accountFollow"


Question: How do I assign a static hostname to an Amazon EC2 instance running Ubuntu Linux?

"I changed the hostname of my Amazon Elastic Compute Cloud (Amazon EC2) instance. However, when I reboot or stop and then restart the instance, the hostname changes back. How do I make the new hostname to persist?"

Answer: "I changed the hostname of my Amazon Elastic Compute Cloud (Amazon EC2) instance. However, when I reboot or stop and then restart the instance, the hostname changes back. How do I make the new hostname to persist?Short descriptionTo be sure that the hostname persists when rebooting or stopping and starting your instance, add the hostname to the appropriate configuration files on your instance.Note: The following steps apply to Ubuntu Linux. For instructions that apply to other distributions, see one of the following:Changing the system hostnameHow do I assign a static hostname to an Amazon EC2 instance running RHEL 5 or 6, CentOS 5 or 6, or Amazon Linux?How do I assign a static hostname to an Amazon EC2 instance running SLES?How do I assign a static hostname to an Amazon EC2 instance running RHEL 7 or CentOS 7?Resolution1.     Use vim to open the /etc/hosts file.sudo vim /etc/hosts2.     Update the /etc/hosts file to include your persistent hostname for localhost, similar to the following:127.0.0.1 localhost persistent-hostnameNote: You might have to create an entry for localhost if the /etc/hosts file on your EC2 instance doesn't have an entry for it.For more information about the hosts file on Ubuntu, see the Ubuntu 18.04 hosts file manpage.3.     If your EC2 instance uses IPv6, add the following configuration data.::1 ip6-localhost ip6-loopback  fe00::0 ip6-localnet  ff00::0 ip6-mcastprefix  ff02::1 ip6-allnodes  ff02::2 ip6-allrouters  ff02::3 ip6-allhosts4.    Save and exit the vim editor.Note: After making this change, press SHIFT+:[colon] to open a new command entry box in the vim editor. Type wq, and then press Enter to save changes and exit vim. Or use SHIFT + ZZ to save and close the file.5.    Run the hostnamectl command and specify the new hostname. Replace the persistent-hostname with the new hostname.sudo hostnamectl set-hostname persistent-hostname6.     After you start or reboot the EC2 instance, run the Linux hostname command without any parameters to verify that the hostname change persisted.hostnameThe command returns the new hostname.Note: If you install any system updates that affect the /etc/hosts file, the hostname file, or the hostname utility, you must run these steps again.Related informationChanging the hostname of your Linux instanceFollow"


Question: Why did I receive a bill after I closed my AWS account?

"I closed my AWS account, but I received another bill. Why did I get another bill from AWS?"

Answer: "I closed my AWS account, but I received another bill. Why did I get another bill from AWS?ResolutionWhen you close your AWS account, you must terminate all your resources or you might continue to incur charges. The on-demand billing for your resources stops when you close your account.However, you might receive a bill after you close your account due to one of the following reasons:You incurred charges in the month before you closed your account: You receive a final bill for the usage incurred between the beginning of the month and the date that you closed your account. For example, if you closed your account on January 15th, then you receive a bill for usage incurred from January 1st through January 15th at the beginning of February.You have active capacity reservations on your account: You might have provisioned Amazon Elastic Compute Cloud (Amazon EC2) Reserved Instances, Amazon Relational Database Service (Amazon RDS) Reserved Instances, Amazon Redshift Reserved Instances, or Amazon ElastiCache Reserved Cache Nodes. You continue to receive a bill for these resources until the reservation period expires. For more information, see Reserved Instances.You signed up for Savings Plans: You continue to receive a bill for your compute usage covered under Savings Plans until the plan term is completed.Important: Within 90 days of closing your account, you can sign in to your account, view past billing, and pay for AWS bills.To pay your unpaid AWS bills, do the following:Open the Billing and Cost Management console.Choose Payments from the navigation pane. You can view your overdue bills in the Payments Due tab.Choose Verify and pay next to your unpaid bills to retry your payments.If the payment doesn’t have Verify and pay next to it, then contact AWS Support and ask them to retry the payment for you.Related informationHow do I close my AWS account?Considerations before you close your accountAvoiding unexpected chargesFollow"


Question: Why am I getting the error "A conflicting conditional operation is currently in progress against this resource" from Amazon S3 when I try to re-create a bucket?

"I deleted my Amazon Simple Storage Service (Amazon S3) bucket. Now I'm trying to create a new bucket with the same name. However, I'm getting the error: "A conflicting conditional operation is currently in progress against this resource. Try again." How can I resolve this?"

Answer: "I deleted my Amazon Simple Storage Service (Amazon S3) bucket. Now I'm trying to create a new bucket with the same name. However, I'm getting the error: "A conflicting conditional operation is currently in progress against this resource. Try again." How can I resolve this?ResolutionAfter you send a request to delete a bucket, Amazon S3 queues the bucket name for deletion. A bucket name must be globally unique because the namespace is shared by all AWS accounts. Because Amazon S3 is a large distributed system, changes such as deleting a bucket take time to become eventually consistent across all AWS Regions.Until the bucket is completely deleted by Amazon S3, you can't use the same bucket name. However, when the bucket is deleted and the name is available, other accounts can use the bucket name. If another account uses the bucket name, you can't use the same name.Note: If you must keep a bucket name, you can empty the bucket instead of deleting it.If your application automatically creates buckets, be sure to choose a bucket-naming logic that's unlikely to cause naming conflicts. Additionally, verify that your application's logic chooses a different bucket name when a bucket name is already taken.Related informationCreating, configuring, and working with Amazon S3 bucketsFollow"


Question: Why does my AWS Glue job fail with the error "Temporary directory not specified" when I insert or extract data from Amazon Redshift?

My AWS Glue job fails with the error "Temporary directory not specified" when I insert or extract data from Amazon Redshift.

Answer: "My AWS Glue job fails with the error "Temporary directory not specified" when I insert or extract data from Amazon Redshift.Short descriptionHere are a few things to remember when your AWS Glue job writes or reads data from Amazon Redshift:Your AWS Glue job writes data into an Amazon Redshift cluster: The job initially writes the data into an Amazon Simple Storage Service (Amazon S3) bucket in CSV format. Then, the job issues a COPY command to Amazon Redshift.Your AWS Glue job reads data from an Amazon Redshift cluster: The job first unloads the data into an Amazon S3 bucket in CSV format using the UNLOAD command. Then, the job loads the data into the DynamicFrame from these temporary bucket files.You might get this error either when either of the following conditions are true:You are unloading the data from Amazon Redshift into the temporary S3 bucket.You are loading the data from the S3 bucket to Amazon Redshift using the COPY or UNLOAD command.ResolutionThe following are some of the common causes and solution options for this error.Define a temporary directoryThe most common reason for this error is the missing temporary S3 bucket that's used by the AWS Glue job as a staging directory. Therefore, be sure to define an S3 bucket as the temporary directory for your job. For more information on how to define a temporary bucket, see Special parameters used by AWS Glue.Verify the IAM role permissionsVerify the IAM role permissions to be sure that you have the right permissions to access the temporary S3 bucket. Also, be sure that you didn't block required permissions for the bucket in the following policies for the AWS Glue IAM role:Bucket policyS3 VPC endpoint policyAWS Organizations policyService control policySome examples of required permissions are ListObjects, GetObject, and PutObject.Verify the name of the temporary directoryBe sure that the name of the S3 bucket that's used as the temporary directory doesn't have a period in it to avoid getting the following exception:Caused by: java.sql.SQLException: [Amazon](500310) Invalid operation: UNLOAD destination is not supported.Verify AWS Key Management Service (AWS KMS) permissionsIf you use customer managed keys from AWS Key Management Service (AWS KMS) to encrypt your data, then be sure to do the include extraunloadoptions in additional_options for your ETL statement in the AWS Glue script. For example:datasource0 = glueContext.create_dynamic_frame.from_catalog(    database = "database-name",     table_name = "table-name",     redshift_tmp_dir = args["TempDir"],    additional_options = {"extraunloadoptions":"ENCRYPTED KMS_KEY_ID 'CMK key ID'"},     transformation_ctx = "datasource0"  )If you are using AWS KMS to encrypt S3 data and facing permission issues related to AWS KMS, then be sure of the following:You have the AWS KMS action permissions similar to the following in the AWS Glue IAM role.You added the AWS Glue IAM role into the AWS KMS key."Action": [    "kms:Encrypt",    "kms:Decrypt",    "kms:ReEncrypt*",    "kms:GenerateDataKey*",    "kms:DescribeKey"    ]Verify the IAM role permissions for AWS Glue Python Shell jobIf you are trying to run the COPY or UNLOAD command from an AWS Glue Python Shell job and can't load the credentials, then be sure of the following:You added the AWS Glue IAM role in Amazon Redshift.The AWS Glue job role includes Amazon Redshift and AWS Glue in its trust relationship policy. For example:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Service": [          "glue.amazonaws.com",          "redshift.amazonaws.com"        ]      },      "Action": "sts:AssumeRole"    }  ]}Check for packet dropThe failure of your queries to reach the Amazon Redshift cluster even after the connection is successful might be due to the maximum transmission unit (MTU) size mismatch between Amazon Redshift and AWS Glue network path. Try configuring Amazon Redshift security groups to allow ICMP "destination unreachable". For more information, see Queries appear to hang and sometimes fail to reach the cluster.Define the temporary directory in the AWS CloudFormation templateIf you created your AWS Glue job using CloudFormation, then be sure that you provided the temporary directory location in the DefaultArguments parameter in your CloudFormation template. For example:"DefaultArguments": { "--TempDir": "s3://doc-example-bucket/doc-example-folder"}Define temporary directory in the DynamicFrameIf you get the error even after defining the temporary directory in your AWS Glue job, then check if you are reading or writing into Amazon Redshift. You can do so using the AWS Glue's DynamicFrame method. Then, confirm the following:You passed your temporary directory into the redshift_tmp_dir property of your DynamicFrame:To create a DynamicFrame using a Data Catalog database and table, see create_dynamic_frame_from_catalog.To create a DynamicFrame with a specified connection and format, see create_dynamic_frame_from_options.To write into a DynamicFrame using information from a Data Catalog database and table, see write_dynamic_frame_from_catalog.To write into a DynamicFrame with a specified connection and format, see write_dynamic_frame_from_options.To write into a DynamicFrame or DynamicFrameCollection with a specified connection and format see write_from_options.To write into a DynamicFrame or DynamicFrameCollection using information from a specified JDBC connection, see write_from_jdbc_conf.You have TempDir specified in the getResolvedOptions function for your ETL job:Use the following commands to retrieve the job name and TempDir parameters:import sysfrom awsglue.utils import getResolvedOptionsargs = getResolvedOptions(sys.argv,['JOB_NAME','TempDir'])TempDir = args['TempDir']For more information, see Accessing parameters using getResolvedOptions.Related informationMoving data to and from Amazon RedshiftFollow"


Question: Why is my Amazon MSK cluster going into the HEALING state?

I want to troubleshoot my Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster that's in HEALING state.

Answer: "I want to troubleshoot my Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster that's in HEALING state.ResolutionYour Amazon MSK cluster goes into the HEALING state when the service is running an internal operation to address an issue (Example: brokers are unresponsive). However, you can use the cluster to produce and consume data. You can't perform Amazon MSK API or AWS Command Line Interface (AWS CLI) update operations on the cluster until it returns to the ACTIVE state.Use the Amazon CloudWatch metrics for Amazon MSK to see why the cluster is in HEALING state:Open the CloudWatch console.In the navigation pane, choose Metrics, and then choose All metrics.In the Browse tab, then choose AWS/Kafka.Under Metrics, Choose Cluster Name.Select the cluster that you want to monitor.If you see spikes in the ActiveControllerCount or OfflinePartitionsCount metric, they indicate that one or more brokers are unhealthy. This might have caused your cluster to go into the HEALING state.For broker-level metrics, choose Broker ID, Cluster Name under Metrics.From the list, select the entries with the cluster name and the metrics CpuUser and CpuSystem. Check if the sum of these two values for all the entries reaches an average of higher than 60% for the cluster. If so, high CPU utilization might have caused the broker to go into the HEALING state. For more information on monitoring CPU usage, see Best practices - Monitor CPU usage.The following are the common reasons for an Amazon MSK cluster to go into the HEALING state:A node or an Amazon Elastic Block Store (Amazon EBS) volume must be replaced because of a hardware failure.A node doesn't meet the Amazon MSK performance SLA for the broker, and the node must be replaced for optimal performance.Note that Amazon MSK is a fully managed service. Therefore, brokers have self-managed workflows that perform corrective actions on themselves, such as replacing nodes during failure situations. When an Amazon EBS volume in a broker becomes unhealthy, Amazon MSK observes the state of the volume for a certain period of time. If the volume becomes healthy during this time, no action is performed. If the volume continues to be unhealthy after this period, then Amazon MSK automatically replaces this volume. The cluster goes into the HEALING state when Amazon MSK performs these actions. However, this doesn't affect the availability of the Amazon MSK cluster as long as you follow the best practices. Even when the broker is in HEALING state, the cluster can handle requests from producers and consumers.Rarely, your cluster might enter into a perpetual HEALING state. This might be caused due to the following reasons:Workload on the cluster is high, and the brokers are being continuously replaced. To avoid this issue, it's a best practice not to use t3.small instances for hosting production clusters. If you're using m5 instances, make sure that you chose right size for your cluster. You can determine the size for your cluster based on your workload and by monitoring your CPU usage. Also, make sure that the number of partitions per broker doesn't exceed the recommended value.The Auto Scaling Group is unable to bring up a new instance. This might happen due to an internal issue or a missing dependency. For example, the AWS Key Management Service (AWS KMS) key that was specified during cluster creation might no longer be accessible.A rare internal event impacted the availability of the underlying Amazon Elastic Compute Cloud (Amazon EC2) instances or caused Amazon EBS latency in an Availability Zone or AWS Region.If your cluster stays in perpetual HEALING state that's not load induced, then contact AWS Support.Related informationCluster statesFollow"


Question: How do I troubleshoot a HTTP 500 or 503 error from Amazon S3?

"When I make a request to Amazon Simple Storage Service (Amazon S3), Amazon S3 returns a 5xx status error. How do I troubleshoot these errors?"

Answer: "When I make a request to Amazon Simple Storage Service (Amazon S3), Amazon S3 returns a 5xx status error. How do I troubleshoot these errors?Short descriptionAmazon S3 can return one of the following 5xx status errors:AmazonS3Exception: Internal Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Internal Error; Request ID: A4DBBEXAMPLE2C4D)AmazonS3Exception: Slow Down (Service: Amazon S3; Status Code: 503; Error Code: 503 Slow Down; Request ID: A4DBBEXAMPLE2C4D)The error code 500 Internal Error indicates that Amazon S3 can't handle the request at that time. The error code 503 Slow Down typically indicates that the number of requests to your S3 bucket is very high. For example, you can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket. However, in some cases, Amazon S3 can return a 503 Slow Down response if your requests exceed the amount of bandwidth available for cross-Region copying.Because Amazon S3 is a distributed service, a very small percentage of 5xx errors is expected during normal use of the service. All requests that return 5xx errors from Amazon S3 can be retried. This means that it's a best practice to have a fault-tolerance mechanism or to implement retry logic for any applications making requests to Amazon S3. By doing so, S3 can recover from these errors.To resolve or avoid 5xx status errors, consider the following approaches:Use a retry mechanism in the application making requests.Configure your application to increase request rates gradually.Distribute objects across multiple prefixes.Monitor the number of 5xx error responses.Note: Amazon S3 doesn't assign additional resources for each new prefix. It automatically scales based on call patterns. As the request rate increases, Amazon S3 optimizes dynamically for the new request rate.ResolutionUse a retry mechanism in the application making requestsBecause of the distributed nature of Amazon S3, requests that return 500 or 503 errors can be retried. It's a best practice to build retry logic into applications that make requests to Amazon S3.All AWS SDKs have a built-in retry mechanism with an algorithm that uses exponential backoff. This algorithm implements increasingly longer wait times between retries for consecutive error responses. Most exponential backoff algorithms use jitter (randomized delay) to prevent successive collisions. For more information, see Error retries and exponential backoff in AWS.Configure your application to gradually increase request ratesTo avoid the 503 Slow Down error, configure your application to start with a lower request rate (transactions per second). Then, increase the application's request rate exponentially. Amazon S3 automatically scales to handle a higher request rate.Distribute objects across multiple prefixesThe request rates described in performance guidelines and design patterns apply per prefix in an S3 bucket. To set up your bucket to handle overall higher request rates and to avoid 503 Slow Down errors, you can distribute objects across multiple prefixes. For example, if you're using your S3 bucket to store images and videos, you can distribute the files into two prefixes similar to the following:mybucket/imagesmybucket/videosIf the request rate on the prefixes increases gradually, Amazon S3 scales up to handle requests for each of the two prefixes. S3 will scale up to handle 3,500 PUT/POST/DELETE or 5,500 GET requests per second. As a result, the overall request rate handled by the bucket doubles.Monitor the number of 5xx status error responsesTo monitor the number of 5xx status error responses that you're getting, you can use one of these options:Turn on Amazon CloudWatch metrics. Amazon S3 CloudWatch request metrics include a metric for 5xx status responses.Turn on Amazon S3 server access logging. Because server access logging captures all requests, you can filter and review all requests that received a 500 Internal Error response. You can also parse logs using Amazon Athena.Additional troubleshootingIf you continue to see high a 5xx status error rates, contact AWS Support. Include the Amazon S3 request ID pairs for the requests that failed with a 5xx status error code.Related informationTroubleshooting Amazon S3Follow"


Question: How do I troubleshoot low bandwidth issues on my VPN connection?

I'm experiencing low bandwidth on my VPN connection. What tests can I run to verify that the issue is not occurring inside my Amazon Virtual Private Cloud (Amazon VPC)?

Answer: "I'm experiencing low bandwidth on my VPN connection. What tests can I run to verify that the issue is not occurring inside my Amazon Virtual Private Cloud (Amazon VPC)?ResolutionLaunch two EC2 instances running Linux for testingBefore beginning performance tests, launch Amazon Elastic Compute Cloud (Amazon EC2) Linux instances in at least two different Availability Zones in the same VPC. You'll use these instances for network performance testing. Verify that the instances support enhanced networking on Linux.Note: When performing network testing between instances that aren't co-located in the same placement group or that don't support jumbo frames, check and set the MTU on your Linux instance.Then, make sure that you can connect to the instances through SSH. Finally, configure the security groups used by your instances to allow communication over the port used by iperf3. The default port for testing TCP performance is 5201.Note: You can use -p to configure iperf3 to use your desired port.Install the iperf3 network benchmark tool on both instancesConnect to your Linux instances using a terminal session, and then install iperf3:To install iperf3 on RHEL-based Linux hosts:$ sudo yum install iperf3To install iperf3 on Debian/Ubuntu hosts:$ sudo apt-get update$ sudo apt-get upgrade$ sudo apt-get install git gcc make$ git clone https://github.com/esnet/iperf3$ cd iperf3$ ./configure$ sudo make$ sudo make install# optionally run "make clean" to free up disk space# by removing artifacts in the build tree.$ sudo make clean$ sudo ldconfigNext, run the following command to configure one instance as a server to listen on the default port:$ sudo iperf3 -s -VRun network tests using iperf3Configure your on-premises host as a client, and then run one or more of the following tests against your instance:The output of the following commands displays the results of 20 parallel streams with increasing window size per TCP connection:sudo iperf3 -c <Private/public IP of instance> -P 20 -w 128K -Vsudo iperf3 -c <Private/public IP of instance> -P 20 -w 512K -Vsudo iperf3 -c <Private/public IP of instance> -P 20 -w 1024K -VThe output of the following commands displays the results of increasing bandwidth capacity and a time frame of 30 seconds per UDP connection:iperf3 -c <Private/public IP of EC2 instance> -u -b 200M -t 30iperf3 -c <Private/public IP of EC2 instance> -u -b 500M -t 30iperf3 -c <Private/public IP of EC2 instance> -u -b 1G -t 30Run the iperf3 tests between the private IP addresses of your EC2 instances and on-premises hosts bi-directionally to benchmark the network throughput on your VPN connection. Then, run these tests between the two public IP addresses of your instances to benchmark throughput over the internet.**Note:**The -w option denotes the window size.This size must be lower than kernel parameter net.core.rmem_max and net.core.wmem_max on both sides.Depending on the system build, rmem_max or wmem_max may be lower than 512KB by default.If lower than 512KB by default, increase rmem_max and wmem_max on both sides before iperf test.Example:Verify current rmrm_max and wmem_max value:$ sudo sysctl net.core.rmem_max net.core.rmem_max = 212992$ sudo sysctl net.core.wmem_max net.core.wmem_max = 212992Increase window size to 2048KB:$ sudo sysctl -w net.core.rmem_max=2097152$ sudo sysctl -w net.core.wmem_max=2097152Related informationHow do I benchmark network throughput between Amazon EC2 Linux instances in the same VPC?Follow"


Question: Can I use EBS Multi-Attach volumes to enable multiple EC2 instances to simultaneously access a standard file system?

I want to access my Amazon Elastic Block Store (Amazon EBS) volume from more than one Amazon Elastic Compute Cloud (Amazon EC2) instance. Can I use Amazon EBS Multi-Attach to enable multiple EC2 instances to simultaneously access a standard file system?

Answer: "I want to access my Amazon Elastic Block Store (Amazon EBS) volume from more than one Amazon Elastic Compute Cloud (Amazon EC2) instance. Can I use Amazon EBS Multi-Attach to enable multiple EC2 instances to simultaneously access a standard file system?ResolutionStandard file systems aren't supported with EBS Multi-Attach. File systems such as XFS, EXT3, EXT4, and NTFS aren't designed to be simultaneously accessed by multiple servers or EC2 instances. Therefore, these file systems don't have built-in mechanisms to manage the coordination and control of writes, reads, locks, caches, mounts, fencing, and so on.Enabling multiple servers to simultaneously access a standard file system can result in data corruption or loss. The operation of standard file systems on EBS Multi-Attach volumes isn't a supported configuration.EBS Multi-Attach allows the attachment of a single io1 Provisioned IOPS volume to up to 16 Nitro-based instances in the same Availability Zone. EBS Multi-Attach volumes can be used as a block-level subcomponent of an overall shared storage solution. Configuration and operation of shared storage systems should be attempted only with a deep understanding of the potential pitfalls and configuration requirements. For more information on using EBS Multi-Attach, refer to Attaching a volume to multiple instances with Amazon EBS Multi-Attach - Considerations and limitations.Related informationWorking with Multi-AttachFollow"


Question: How can I get notifications for AWS Backup jobs that failed?

I want to be notified if my AWS Backup job fails. How can I set up email notifications for an unsuccessful backup job?

Answer: "I want to be notified if my AWS Backup job fails. How can I set up email notifications for an unsuccessful backup job?Short descriptionUse Amazon Simple Notification Service (Amazon SNS) to send email notifications about failed backup jobs. Follow these steps to configure Amazon SNS and your backup vault for notifications:1.    Create an SNS topic to send AWS Backup notifications to.2.    Configure your backup vault to send notifications to the SNS topic.3.    Create an SNS subscription that filters notifications to backup jobs that are unsuccessful.4.    Monitor emails for notifications.To receive notifications for other events, such as restore jobs and recovery points, see Using Amazon SNS to track AWS Backup events.ResolutionCreate an SNS topic to send AWS Backup notifications1.    Open the Amazon SNS console.2.    From the navigation pane, choose Topics.3.    Choose Create topic.4.    For Name, enter a name for the topic.5.    Choose Create topic.6.    Under the Details of the topic that you just created, copy the value for ARN (Amazon Resource Name). You need this value for later steps.7.    Above the Details pane, choose Edit.8.    Expand Access policy.9.    In the JSON editor, append the following permissions into the policy:Important: Replace the value for Resource with the ARN that you copied in step 6.{      "Sid": "My-statement-id",      "Effect": "Allow",      "Principal": {        "Service": "backup.amazonaws.com"      },      "Action": "SNS:Publish",      "Resource": "arn:aws:sns:eu-west-1:111111111111:exampletopic"}10.    Choose Save changes.Configure your backup vault to send notifications to the SNS topic1.    Install and configure the AWS Command Line Interface (AWS CLI).2.    Using the AWS CLI, run the put-backup-vault-notifications command with --backup-vault-events set to BACKUP_JOB_COMPLETED. Replace the following values in the example command:--endpoint-url: the endpoint for the AWS Region where you have the backup vaulteu-west-1: the AWS Region where you have the backup vault--backup-vault-name: the name of your backup vault--sns-topic-arn: the ARN of the SNS topic that you createdaws backup put-backup-vault-notifications --endpoint-url https://backup.eu-west-1.amazonaws.com --backup-vault-name examplevault --sns-topic-arn arn:aws:sns:eu-west-1:111111111111:exampletopic --backup-vault-events BACKUP_JOB_COMPLETEDNote: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.3.    Run the get-backup-vault-notifications command to confirm that notifications are configured:aws backup get-backup-vault-notifications --backup-vault-name examplevaultThe command returns output similar to the following:{    "BackupVaultName": "examplevault",    "BackupVaultArn": "arn:aws:backup:eu-west-1:111111111111:backup-vault:examplevault",    "SNSTopicArn": "arn:aws:sns:eu-west-1:111111111111:exampletopic",    "BackupVaultEvents": [        "BACKUP_JOB_COMPLETED"    ]}Create an SNS subscription that filters notifications to backup jobs that are unsuccessful1.    Open the Amazon SNS console.2.    From the navigation pane, choose Subscriptions.3.    Choose Create subscription.4.    For Topic ARN, select the SNS topic that you created.5.    For Protocol, select Email-JSON.6.    For Endpoint, enter the email address where you want to get email notifications about failed backup jobs.7.    Expand Subscription filter policy.8.    In the JSON editor, enter the following:{  "State": [    {      "anything-but": "COMPLETED"    }  ]}9.    Choose Create subscription.10.    The email address that you entered in step 6 receives a subscription confirmation email. Be sure to confirm the SNS subscription.Monitor emails for notificationsWhen your vault has an unsuccessful backup job, you get an email notification similar to the following:"An AWS Backup job was stopped. Resource ARN : arn:aws:ec2:eu-west-1:111111111111:volume/vol-example56d7w92d4b. BackupJob ID : example4-3dd5-5678-b52d-90bd749355a5"You can test notifications by creating two on-demand backups and then stopping one of the backups. You get an email notification for the stopped backup only.Related informationTroubleshooting AWS BackupFollow"


Question: How can I prevent a Hadoop or Spark job's user cache from consuming too much disk space in Amazon EMR?

The user cache for my Apache Hadoop or Apache Spark job is taking up all the disk space on the partition. The Amazon EMR job is failing or the HDFS NameNode service is in safe mode.

Answer: "The user cache for my Apache Hadoop or Apache Spark job is taking up all the disk space on the partition. The Amazon EMR job is failing or the HDFS NameNode service is in safe mode.Short descriptionOn an Amazon EMR cluster, YARN is configured to allow jobs to write cache data to /mnt/yarn/usercache. When you process a large amount of data or run multiple concurrent jobs, the /mnt file system can fill up. This causes node manager failures on some nodes, which then causes the job to freeze or fail.Use one of the following methods to resolve this problem:Adjust the user cache retention settings for YARN NodeManager. Choose this option if you don't have long-running jobs or streaming jobs.Scale up the Amazon Elastic Block Store (Amazon EBS) volumes. Choose this option if you have long-running jobs or streaming jobs.ResolutionOption 1: Adjust the user cache retention settings for NodeManagerThe following attributes define the cache cleanup settings:yarn.nodemanager.localizer.cache.cleanup.interval-ms: This is the cache cleanup interval. The default value is 600,000 milliseconds. After this interval—and if the cache size exceeds the value set in yarn.nodemanager.localizer.cache.target-size-mb—files that aren't in use by running containers are deleted.yarn.nodemanager.localizer.cache.target-size-mb: This is the maximum disk space allowed for the cache. The default value is 10,240 MB. When the cache disk size exceeds this value, files that aren't in use by running containers are deleted on the interval set in yarn.nodemanager.localizer.cache.cleanup.interval-ms.To set the cleanup interval and maximum disk space size on a running cluster:1.    Open /etc/hadoop/conf/yarn-site.xml on each core and task node, and then reduce the values for yarn.nodemanager.localizer.cache.cleanup.interval and yarn.nodemanager.localizer.cache.target-size-mb. For example:sudo vim /etc/hadoop/conf/yarn-site.xmlyarn.nodemanager.localizer.cache.cleanup.interval-ms 400000yarn.nodemanager.localizer.cache.target-size-mb 51202.    To restart the NodeManager service, run the following commands on each core and task node:sudo stop hadoop-yarn-nodemanagersudo start hadoop-yarn-nodemanagerNote: In Amazon EMR release versions 5.21.0 and later, you can also use a configuration object, similar to the following, to override the cluster configuration or specify additional configuration classifications for a running cluster. For more information, see Reconfigure an instance group in a running cluster.To set the cleanup interval and maximum disk space size on a new cluster, add a configuration object similar to the following when you launch the cluster:[    {      "Classification": "yarn-site",     "Properties": {       "yarn.nodemanager.localizer.cache.cleanup.interval-ms": "400000",       "yarn.nodemanager.localizer.cache.target-size-mb": "5120"      }    }]Remember that the deletion service doesn't complete on running containers. This means that even after you adjust the user cache retention settings, data might still be spilling to the following path and filling up the file system:{'yarn.nodemanager.local-dirs'}/usercache/user/appcache/application_id ,Option 2: Scale up the EBS volumes on the EMR cluster nodesTo scale up storage on a running EMR cluster, see Dynamically scale up storage on Amazon EMR clusters.To scale up storage on a new EMR cluster, specify a larger volume size when you create the EMR cluster. You can also do this when you add nodes to an existing cluster:Amazon EMR release version 5.22.0 and later: The default amount of EBS storage increases based on the size of the Amazon Elastic Compute Cloud (Amazon EC2) instance. For more information about the amount of storage and number of volumes allocated by default for each instance type, see Default Amazon EBS storage for instances.Amazon EMR release versions 5.21 and earlier: The default EBS volume size is 32 GB. Of this amount, 27 GB is reserved for the /mnt partition. HDFS, YARN, the user cache, and all applications use the /mnt partition. Increase the size of your EBS volume as needed (for example, 100-500 GB or more). You can also specify multiple EBS volumes. Multiple EBS volumes will be mounted as /mnt1, /mnt2, and so on.For Spark streaming jobs, you can also perform an unpersist ( RDD.unpersist()) when processing is done and the data is no longer needed. Or, explicitly call System.gc() in Scala ( sc._jvm.System.gc() in Python) to start JVM garbage collection and remove the intermediate shuffle files.Follow"


Question: How can I resolve high CPU utilization on my T2 or T3 EC2 Windows instance if my CPU is being throttled?

My T2 or T3 Amazon Elastic Compute Cloud (Amazon EC2) Windows instance is experiencing high CPU utilization because my CPU is being throttled. How can I fix this?

Answer: "My T2 or T3 Amazon Elastic Compute Cloud (Amazon EC2) Windows instance is experiencing high CPU utilization because my CPU is being throttled. How can I fix this?ResolutionTo resolve CPU throttling, you can either enable T2/T3 Unlimited, or change the instance type.Enable T2/T3 UnlimitedThis solution has no downtime, but might cost more, especially if the instance has to burst often.Open the Amazon EC2 console, and then choose Instances from the navigation pane.Select the instance that is being throttled.For Actions, choose Instance Settings, Change T2/T3 Unlimited.Choose Enable.Note: If you still see high CPU utilization after you enable T2/T3 Unlimited, troubleshoot the process that is causing the issue. For more information, see How do I diagnose high CPU utilization on my EC2 Windows instance when my CPU is not being throttled?Change the instance typeThis solution allows you to select an instance that is better suited to your current needs if the instance has to burst often.Note: Stop your instance before you change your instance type.You can change to one of the following instance types:A T2 or T3 instance type with a higher CPU credit limit.An instance type that doesn't use a CPU credit bucket model.For more information, see How do I get more CPU and memory for my EC2 instance?Related InformationHow can I find out if the CPU on my T2 or T3 EC2 Windows instance is being throttled?Follow"


Question: How do I remove sensitive data from my CloudFront logs?

"By default, Amazon CloudFront standard logs capture sensitive data for some of its fields. Due to privacy concerns, I want to remove this part of the logs."

Answer: "By default, Amazon CloudFront standard logs capture sensitive data for some of its fields. Due to privacy concerns, I want to remove this part of the logs.Short descriptionNote: This article uses the example of Client-IP (c-ip) field.CloudFront logs capture c-ip as one of its fields by default. There are three ways to remove c-ip from your logs.Trigger an AWS Lambda function that removes the field on the log delivery into Amazon Simple Storage Service (Amazon S3).Have an Amazon Elastic Compute Cloud (Amazon EC2) process that runs at certain intervals to remove the field.Use CloudFront real-time logs, and remove the sensitive field before you send the log data to Amazon S3.ResolutionTrigger a Lambda functionOne way to remove the c-ip field is to use Amazon S3 notification events. When CloudFront delivers the log file into the Amazon S3 bucket, configure your bucket to trigger a Lambda function.Create a Lambda function1.    Open the AWS Lambda console.2.    Under Functions, create a new Lambda function that has the following configurations:Uses the object name from the Amazon S3 event.Gets the object from the S3 bucket.3.    Remove the c-ip column, or replace the values with anonymized data.Note: Replace the values to keep the same format in case you have other applications process the logs further.4.    Save and upload the log back to Amazon S3.Create a new event1.    In the logs target bucket, go to Properties.2.    Under Event notifications, create a new event.3.    Select the event type Put, and the destination Lambda function.4.    Select the Lambda function created in step 1, and then choose Save.Important: To avoid a recursive invocation (infinite loop) with your Lambda function, perform the following actions:Have your CloudFront logs delivered to an initial staging prefix. For example, "original".Have the Amazon S3 event triggered on that prefix only.Have the Lambda function deliver the logs into a different prefix. For example, "processed".If you deliver the logs into the same prefix, the Lambda function triggers again and creates a recursive invocation. For more information, see Avoiding recursive invocation with Amazon S3 and AWS Lambda.Note: To keep Amazon S3 costs low, set up an Amazon S3 Lifecycle policy to expire the original logs after a certain time period.Have an Amazon EC2 processUse Amazon EventBridge to create a scheduled rule (cron) that launches an EC2 instance and processes the log files at a scheduled recurrence. For example, one time per day. When the process is done, stop the EC2 instance until the next recurrence to save on costs.1.    Configure EventBridge and Lambda to start an EC2 instance at a given time. For more information, see How do I stop and start Amazon EC2 instances at regular intervals using Lambda?2.    On the EC2 instance, deploy a code that'll download the logs for a certain time period. For example, a full day. Remove the c-ip column to process the logs, or replace the column values with anonymized data. Upload the processed logs back to the S3 bucket.Optional: Merge all the processed logs into a single file to save on Amazon S3 Lifecycle transitions costs. This process is helpful if you intend to store the logs for long time periods.Use Kinesis Data FirehoseUse CloudFront real-time logs to select the fields that you want to save. Later, have Amazon Kinesis Data Firehose send the log data to Amazon S3.When you configure CloudFront real-time logs, a list of fields that are included in each real-time log record is available. Each log record contains up to 40 fields. You have the option to receive all the available fields or only the fields that you must monitor and analyze performance. Deactivate the field c-ip to exclude the field from your logs.Note: Due to the use of Amazon Kinesis Data Streams, this option can get expensive. Consider the other two options (Trigger a Lambda function or Have an Amazon EC2 process) for a more cost-effective solution.Follow"


Question: How do I customize my Elastic Beanstalk environment using .ebextensions?

"How do I customize my AWS Elastic Beanstalk environment using .ebextensions to create files, install packages, and run commands on my Amazon Elastic Compute Cloud (Amazon EC2) instances?"

Answer: "How do I customize my AWS Elastic Beanstalk environment using .ebextensions to create files, install packages, and run commands on my Amazon Elastic Compute Cloud (Amazon EC2) instances?Short descriptionConfigure your Amazon EC2 instances in an Elastic Beanstalk environment by using Elastic Beanstalk configuration files (.ebextensions).Configuration changes made to your Elastic Beanstalk environment won't persist if you use the following configuration methods:Configuring an Elastic Beanstalk resource directly from the console of a specific AWS service.Installing a package, creating a file, or running a command directly from your Amazon EC2 instance.Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.ResolutionSet up your .ebextensions directory1.    In the root of your application bundle, create a hidden directory named .ebextensions.2.    Store your configuration file in the .ebextensions directory.Your application source bundle should look similar to the following example:~/workspace/my-application/|-- .ebextensions|   |-- 01-server-configuration.config|   `-- 02-asg-healthcheck.config|-- index.php`-- styles.cssCustomize your Elastic Beanstalk environmentTo customize your environment, consider the following:Use the option_settings key to modify the environment configuration. You can choose from general options for all environments and platform-specific options.Note: Recommended values are applied when you create or update an environment on the Elastic Beanstalk API by a client. For example, the client could be the AWS Management Console, Elastic Beanstalk Command Line Interface (EB CLI), AWS CLI, or SDKs. Recommended values are directly set at the API level and have the highest precedence. The configuration setting applied at the API level can't be changed using option_settings, as the API has the highest precedence.Precedence rules can result in your option_settings modifications not being applied to the environment configuration. To remove the configurations directly applied during environment creation or make the update on the Elastic Beanstalk API, use the update-environment command with the --options-to-remove flag.If there are no option settings for your desired resource configuration, use the Resources key to customize the resources in your Elastic Beanstalk environment.Note: Resources defined in configuration files are added to the AWS CloudFormation template that's used to launch your environment. All AWS CloudFormation resource types are supported. For more information on logical resource names, see Modifying the resources that Elastic Beanstalk creates for your environment.Use keys to customize software on Linux or Windows servers.For configuration file samples, see the AWS GitHub repository.Apply your custom settings to your application1.    Create an application source bundle that includes your configuration files.Note: Folders starting with a period, such as .ebextensions, can be hidden by file browsers. To keep these folders visible, include the .ebextensions folder in the root of your application bundle when you create your application source bundle.2.    Deploy your updated Elastic Beanstalk application.Follow"


Question: How do I configure the hosted web UI for Amazon Cognito?

"I want to configure the hosted web UI for my Amazon Cognito user pool, but I'm not sure what settings to turn on. How do I set it up?"

Answer: "I want to configure the hosted web UI for my Amazon Cognito user pool, but I'm not sure what settings to turn on. How do I set it up?Short descriptionWhen you create a user pool in Amazon Cognito and then configure a domain for it, Amazon Cognito automatically provisions a hosted web UI to let you add sign-up and sign-in pages to your app.If you're not sure how to set this up or what settings to use—such as the types of OAuth 2.0 flows and scopes to turn on—then follow the steps in this article.ResolutionIf you haven't already done so, create a user pool, and create an app client in the user pool. Then, follow these instructions:Note: These directions use the new Amazon Cognito console.Add a domain name to your user poolIn the Amazon Cognito console, choose User pools, and then choose your user pool.Under App integration, choose Domain name, and then choose Actions.Choose Create Cognito domain to add your own domain prefix to the Amazon Cognito hosted domain. Or, choose Create custom domain to add your own custom domain.Change app client settingsIn the Amazon Cognito console, choose User pools, and then choose your user pool.Under App integration, choose your app client from the App clients and analytics section.Choose Edit from the Hosted UI section.Do the following:For Allowed callback URLs, enter the URL of your web application that will receive the authorization code. Your users are redirected here when they sign in.For Allowed sign-out URLs - optional, enter the URL where you want to redirect your users when they sign out.For Identity providers, choose Cognito user pool from the dropdown list.For OAuth 2.0 grant types, select either Authorization Code grant or Implicit grant OAuth 2.0 authentication flow. Authorization code grant type is used by confidential and public clients to exchange an auth code for an access token. Implicit grant type is only used when there's a specific reason that authorization code grant can't be used. For more information, see Understanding Amazon Cognito user pool OAuth 2.0 grants.For OpenID Connect scopes, select openid and any other OAuth scopes that you want Amazon Cognito to add in the tokens for when your users authenticate. For example, phone and email.For Custom scopes, select any custom scopes that you want to authorize for this app.Choose Save changes.For more information, see Configuring a user pool app client.(Optional) Customize the hosted web UIYou can add a custom logo or customize the CSS for the hosted web UI. For more information, see Customizing the built-in sign-in and sign-up webpages.(Optional) Construct the URL for the hosted web UIIf you want to control which parameters are included in the login URL for the hosted web UI, then construct the URL manually.In the Amazon Cognito console, choose User pools, and then choose your user pool.Under App integration, copy the Domain URL under the Domain section. Then, paste the URL into a text editor for reference.Under App clients and analytics, click your client name.Copy the Client ID to your clipboard. Then, paste the ID into a text editor for reference.Copy one of the Allowed callback URLs to your clipboard. Then, paste the URL into a text editor for reference.Construct the URL for the hosted web UI by pasting together the information that you just copied into this format:domainUrl/login?response_type=code&client_id=appClientId&redirect_uri=callbackUrlFor example: https://my-user-pool.auth.us-east-1.amazoncognito.com/login?response_type=code&client_id=a1b2c3d4e5f6g7h8i9j0k1l2m3&redirect_uri=https://my-website.com-or-Choose View Hosted UI in the App client section to access the default URL for the login endpoint. Then, replace parts of the URL as detailed earlier.If you turned on Authorization code grant earlier for OAuth 2.0 grant types, then using this URL prompts Amazon Cognito to return an authorization code when your users sign in. If you turned on Implicit grant for OAuth 2.0 grant types earlier and you want Amazon Cognito to return an access token instead when your users sign in, then replace response_type=code with response_type=token in the URL.Launch the hosted web UINote: If you constructed the URL for the hosted web UI manually, enter that URL in your web browser instead.In the Amazon Cognito console, choose User pools, and then choose your user pool.Under App integration, click your Client name from the App clients and analytics section.Under Hosted UI, choose View Hosted UI. The sign-in page of the hosted web UI opens in a new browser tab or window.Related informationGetting started with user poolsFollow"


Question: Why can't I launch EC2 instances from my copied AMI?

I copied my Amazon Machine Image (AMI) to a different account or Region. I'm not able to launch Amazon Elastic Compute Cloud (Amazon EC2) instances from the copied AMI. How do I fix this?

Answer: "I copied my Amazon Machine Image (AMI) to a different account or Region. I'm not able to launch Amazon Elastic Compute Cloud (Amazon EC2) instances from the copied AMI. How do I fix this?Short descriptionYou might not be able to launch instances from a copied AMI with an encrypted Amazon Elastic Block Store (Amazon EBS) for the following reasons:The AWS Key Management Service (KMS) customer managed key's (KMS key) key policy is missing the proper principals to allow the requesting account's access.The AWS Identity and Access Management (IAM) entity in the requesting account doesn't have the necessary KMS permissions for the volume's cross-account KMS key.ResolutionEnable cross-account access to existing KMS custom keys on the copied AMIFor detailed instructions, see How to enable cross-account access to existing custom keys in Share custom encryption keys more securely between accounts by using AWS Key Management Service.Set permissions for EC2 instances to access the KMS key1.    Open the AWS KMS console.Note: Make sure you're in the correct Region.2.    Choose Customer managed keys, and then select the appropriate key.3.    Under Key policy, scroll down to Key users. Verify that the Key users section lists all internal and external accounts and users that need access to the key.4.    If any accounts or users are missing from the Key users section, select Policy view.Note: If you've ever edited the AWS KMS key policy manually, the key policy is only available in policy (JSON) view.5.    Verify that the Allow use of the key statement in the key policy is correct. The statement must include the ARN of all accounts and users who need access to the key.The following is an example of the Allow use of the key statement in the default key policy. The Allow use of the key statement in the following example includes the following ARNs:The external AWS account containing the copied AMI.The parent account of the AMI.A user within the external account.For an overview and example of the entire default key policy, see Default key policy.{             "Sid": "Allow use of the key",            "Effect": "Allow",            "Principal": {                "AWS": [                    "arn:aws:iam::111122223333:root",                    "arn:aws:iam::444455556666:root",                    "arn:aws:iam::111122223333:user/UserA"                ]            },            "Action": [                "kms:Encrypt",                "kms:Decrypt",                "kms:ReEncrypt*",                "kms:GenerateDataKey*",                "kms:DescribeKey"            ],            "Resource": "*"        },        {            "Sid": "Allow attachment of persistent resources",            "Effect": "Allow",            "Principal": {                "AWS": [                    "arn:aws:iam::111122223333:root",                    "arn:aws:iam::444455556666:root",                    "arn:aws:iam::111122223333:user/UserA"                ]            },            "Action": [                "kms:CreateGrant",                "kms:ListGrants",                "kms:RevokeGrant"            ],            "Resource": "*",            "Condition": {                "Bool": {                    "kms:GrantIsForAWSResource": "true"                }            }        }    ]}6.    If you haven't already created the IAM policy, proceed to the next section to create and assign the policy.Create the IAM policy and attach it to your IAM user or group1.    Sign in to the IAM console with your user that has administrator permissions.2.    Choose Policies.3.    Choose Create policy.4.    Choose the JSON tab. Copy the following sample JSON policy, and then paste it into the JSON text box. Replace arn:aws:kms:REGION:MAINACCOUNTNUMBER:key/1a345678-1234-1234-1234-EXAMPLE with the ARN of your KMS key.{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "AllowUseOfTheKey",            "Effect": "Allow",            "Action": [                "kms:Encrypt",                "kms:Decrypt",                "kms:ReEncrypt*",                "kms:GenerateDataKey*",                "kms:DescribeKey"            ],            "Resource": [                "arn:aws:kms:REGION:MAINACCOUNTNUMBER:key/1a345678-1234-1234-1234-EXAMPLE"            ]        },        {            "Sid": "AllowAttachmentOfPersistentResources",            "Effect": "Allow",            "Action": [                "kms:CreateGrant",                "kms:ListGrants",                "kms:RevokeGrant"            ],            "Resource": [                "arn:aws:kms:REGION:MAINACCOUNTNUMBER:key/1a345678-1234-1234-1234-EXAMPLE"            ],            "Condition": {                "Bool": {                    "kms:GrantIsForAWSResource": true                }            }        }    ]}5.    Choose Review policy. The Policy Validator reports any syntax errors.6.    On the Review page, enter KmsKeyUsagePolicy for the policy name. Review the policy Summary to see the permissions granted by your policy, and then choose Create policy to save the policy. The new policy appears in the list of managed policies and is ready to attach to your IAM user or group.7.    In the navigation pane of the IAM console, choose Policies.8.    At the top of the policy list, in the search box, start typing KmsKeyUsagePolicy until you see your policy. Then check the box next to KmsKeyUsagePolicy in the list.9.    Choose Policy actions, and then choose Attach.10.    For Filter, choose Users.11.    In the search box, start typing username until your user is visible on the list. Then check the box next to that user in the list.12.    Choose Attach Policy.Related informationCopying an AMIEditing keysTutorial: Create and attach your first customer managed policyValidating IAM policiesFollow"


Question: How do I identify errors related to Data API in Amazon Redshift?

How do I identify the reason why a Data API query in Amazon Redshift failed?

Answer: "How do I identify the reason why a Data API query in Amazon Redshift failed?ResolutionAmazon Redshift Data API is asynchronous, meaning you can run long-running queries without having to wait for it to complete. When a Data API query fails, the status of the query isn't displayed immediately. To determine the reasons for failure, use the DescribeStatement action for single or multiple queries. To run the DescribeStatement, you must have the statement ID.Single queryTo run a single query against the cluster, use the ExecuteStatement action to return a statement ID:Note: The following example command uses the AWS Secrets Manager authentication method. The command runs an SQL statement against a cluster and returns an identifier to fetch the results.aws redshift-data execute-statement    --region us-east-1    --secret arn:aws:secretsmanager:us-east-1:123456789012:secret:myuser-secret-hKgPWn    --cluster-identifier redshift-cluster-1    --sql "select * from test_table;"    --database devNote: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.The output looks similar to the following:{    "ClusterIdentifier": "redshift-cluster-1",    "CreatedAt": "2022-09-16T12:22:31.894000+05:30",    "Database": "dev",    "Id": "458c568d-717b-4f36-90bd-e642bfb06cbf",    "SecretArn": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myuser-secret-hKgPWn"}The preceding SQL statement returns an ExecuteStatementOutput, which includes the statement Id. You can check the status of the query using DescribeStatement and entering the statement ID:aws redshift-data describe-statement --id 458c568d-717b-4f36-90bd-e642bfb06cbfThe output for DescribeStatement provides the following additional details:RedshiftPidQuery durationNumber of rows inSize of the result setRedshiftQueryIDThe output looks similar to the following:{    "ClusterIdentifier": "redshift-cluster-1",    "CreatedAt": "2022-09-16T12:22:31.894000+05:30",    "Duration": -1,    "Error": "ERROR: relation \"test_table\" does not exist",    "HasResultSet": false,    "Id": "458c568d-717b-4f36-90bd-e642bfb06cbf",    "QueryString": "select * from test_table;",    "RedshiftPid": 1074727629,    "RedshiftQueryId": -1,    "ResultRows": -1,<    "ResultSize": -1,    "SecretArn": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myuser-secret-hKgPWn",    "Status": "FAILED",    "UpdatedAt": "2022-09-16T12:22:32.365000+05:30"}The "Error": section in the preceding response displays the exact error. Which in the preceding example is "ERROR: relation "test_table" does not exist".Multiple queriesTo run multiple queries against the cluster use the BatchExecuteStatement action to return a statement ID:aws redshift-data batch-execute-statement    --region us-east-1    --secret-arn arn:aws:secretsmanager:us-east-1:123456789012:secret:myuser-secret-hKgPWn     --cluster-identifier redshift-cluster-1        --database dev    --sqls "select * from test_table;" "select * from another_table;"The output looks similar to the following:{    "ClusterIdentifier": "redshift-cluster-1",    "CreatedAt": "2022-09-16T12:37:16.707000+05:30",    "Database": "dev",    "Id": "08b4b917-9faf-498a-964f-e82a5959d1cb",    "SecretArn": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myuser-secret-hKgPWn"}To get the status of the queries, use the DescribeStatement with the statement ID from the preceding response:aws redshift-data describe-statement --id 08b4b917-9faf-498a-964f-e82a5959d1cbThe output looks similar to the following:{    "ClusterIdentifier ": "redshift-cluster-1 ",    "CreatedAt ": "2022-09-16T12:37:16.707000+05:30 ",    "Duration ": 0,    "Error ": "Query #1 failed with ERROR: relation \ "test_table\"does not exist ",    "HasResultSet ": false,    "Id ": "08b4b917-9faf-498a-964f-e82a5959d1cb ",    "RedshiftPid ": 1074705048,    "RedshiftQueryId ": 0,    "ResultRows ":-1,    "ResultSize ": -1,    "SecretArn ": "arn:aws:secretsmanager:us-east-1:123456789012:secret:myuser-secret-hKgPWn ",    "Status ": "FAILED ",    "SubStatements ": [        {            "CreatedAt ": "2022-09-16T12:37:16.905000+05:30 ",            "Duration ": -1,            "Error ": "ERROR: relation \ "test_table\" does not exist ",            "HasResultSet ": false,            "Id ": "08b4b917-9faf-498a-964f-e82a5959d1cb:1",            "QueryString ": "select * from test_table; ",            "RedshiftQueryId ": -1,            "ResultRows ": -1,            "ResultSize ": -1,            "Status ": "FAILED ",            "UpdatedAt ": "2022-09-16T12:37:17.263000+05:30 "        },        {            "CreatedAt ": "2022-09-16T12:37:16.905000+05:30",            "Duration ": -1,            "Error ": "Connection or an prior query failed. ",            "HasResultSet ": false,            "Id ": "08b4b917-9faf-498a-964f-e82a5959d1cb:2 ",            "QueryString ": "select * from another_table;",            "RedshiftQueryId ": 0,            "ResultRows ": -1,            "ResultSize": -1,            "Status ": "ABORTED ",            "UpdatedAt ": "2022-09-16T12:37:17.263000+05:30 "        }    ],    "UpdatedAt ": "2022-09-16T12:37:17.288000+05:30 "}The preceding output displays the status of all sub-statements for a multi-statement query. The "Error": section in the preceding response displays the exact error for each sub-statement.To troubleshoot problems with the Data API, see Troubleshooting issues for Amazon Redshift Data API.Monitoring Data API eventsData API events can be monitored using Amazon EventBridge. This information can be sent to an AWS Lambda function that's integrated with Amazon Simple Notification Service (Amazon SNS) to send notifications. For more information, see Building an event-driven application with AWS Lambda and the Amazon Redshift Data API.Follow"


Question: How can I grant my Amazon EC2 instance access to an Amazon S3 bucket?

I'm unable to access an Amazon Simple Storage Service (Amazon S3) bucket from my Amazon Elastic Compute Cloud (Amazon EC2) instance. How can I activate read/write access to S3 buckets from an EC2 instance?

Answer: "I'm unable to access an Amazon Simple Storage Service (Amazon S3) bucket from my Amazon Elastic Compute Cloud (Amazon EC2) instance. How can I activate read/write access to S3 buckets from an EC2 instance?Short descriptionTo connect to your S3 buckets from your EC2 instances, you must do the following:1.    Create an AWS Identity and Access Management (IAM) profile role that grants access to Amazon S3.2.    Attach the IAM instance profile to the instance.3.    Validate permissions on your S3 bucket.4.    Validate network connectivity from the EC2 instance to Amazon S3.5.    Validate access to S3 buckets.ResolutionCreate an IAM instance profile that grants access to Amazon S31.    Open the IAM console.2.    Choose Roles, and then choose Create role.3.    Select AWS Service, and then choose EC2 under Use Case.Note: Creating an IAM role from the console with EC2 selected as the trusted entity automatically creates an IAM instance profile with the same name as the role name. However, if the role is created using the AWS Command Line Interface (AWS CLI) or from the API, then an instance profile isn't automatically created. For more information, refer to I created an IAM role, but the role doesn't appear in the dropdown list when I launch an instance. What do I do?4.    Select Next: Permissions.5.    Create a custom policy that provides the minimum required permissions to access your S3 bucket. For instructions on creating custom policies, see Writing IAM policies: how to grant access to an Amazon S3 bucket and Identity and access management in Amazon S3.Note: Creating a policy with the minimum required permissions is a security best practice. However, to allow EC2 access to all your Amazon S3 buckets, use the AmazonS3ReadOnlyAccess or AmazonS3FullAccess managed IAM policy.6.    Select Next: Tags, and then select Next: Review.7.    Enter a Role name, and then select Create role.Attach the IAM instance profile to the EC2 instance1.    Open the Amazon EC2 console.2.    Choose Instances.3.    Select the instance that you want to attach the IAM role to.4.    Choose the Actions tab, choose Security, and then choose Modify IAM role.5.    Select the IAM role that you just created, and then choose Save. The IAM role is assigned to your EC2 instance.Validate permissions on your S3 bucket1.    Open the Amazon S3 console.2.    Select the S3 bucket that you want to verify the policy for.3.    Choose Permissions.4.    Choose Bucket Policy.5.    Search for statements with Effect: Deny.6.    In your bucket policy, edit or remove any Effect: Deny statements that are denying the IAM instance profile access to your bucket. For instructions on editing policies, see Editing IAM policies.Validate network connectivity from the EC2 instance to Amazon S3For your EC2 instance to connect to S3 endpoints, the instance must be one of the following:EC2 instance with a public IP address and a route table entry with the default route pointing to an Internet GatewayPrivate EC2 instance with a default route through a NAT gatewayPrivate EC2 instance with connectivity to Amazon S3 using a gateway VPC endpointTo troubleshoot connectivity between a private EC2 instance and an S3 bucket, see Why can’t I connect to an S3 bucket using a gateway VPC endpoint?Validate access to S3 buckets1.    Install the AWS CLI on your EC2 instance.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.2.    Verify access to your S3 buckets by running the following command. Replace DOC-EXAMPLE-BUCKET with the name of your S3 bucket.aws s3 ls s3://DOC-EXAMPLE-BUCKETNote: S3 objects encrypted with an AWS Key Management Service (AWS KMS) key must have kms: Decrypt permissions granted in the following:The IAM role attached to the instance.The KMS key policy.If these permissions aren't granted, then you can't copy or download the S3 objects. For more information, see Do I need to specify the AWS KMS key when I download a KMS-encrypted object from Amazon S3?Related informationWhy can’t I connect to an S3 bucket using a gateway VPC endpoint?Follow"


Question: Why is the core node in my Amazon EMR cluster running out of disk space?

"I'm running Apache Spark jobs on an Amazon EMR cluster, and the core node is almost out of disk space."

Answer: "I'm running Apache Spark jobs on an Amazon EMR cluster, and the core node is almost out of disk space.ResolutionDetermine which core nodes are unhealthyNodes that have at least one Amazon Elastic Block Store (Amazon EBS) volume attached are considered unhealthy if they hit more than 90% disk utilization. To determine which nodes might have reached 90% disk utilization, do the following:1.    Check the Amazon CloudWatch metric MRUnhealthyNodes. This metric indicates the number of unhealthy nodes of an EMR cluster.Note: You can create a CloudWatch Alarm to monitor the MRUnhealthyNodes metric.2.    Connect to the primary node and access the instance controller log at /emr/instance-controller/log/instance-controller.log. In the instance controller log, search for InstanceJointStatusMap to identify which nodes are unhealthy.For more information, see High disk utilization in How do I resolve ExecutorLostFailure "Slave lost" errors in Spark on Amazon EMR?3.    Log in to the core nodes and then run the following command to determine if a mount has high utilization:df -hRemove unnecessary local and temporary Spark application filesWhen you run Spark jobs, Spark applications create local files that consume the rest of the disk space on the core node. If the df -h command shows that /mnt, for example, is using more than 90% disk space, check which directories or files have high utilization.Run the following command on the core node to see the top 10 directories that are using the most disk space:cd /mntsudo du -hsx * | sort -rh | head -10If the /mnt/hdfs directory has high utilization, check the HDFS usage and remove any unnecessary files, such as log files. Reducing the retention period helps in cleaning the log files from HDFS automatically.hdfs dfsadmin -reporthadoop fs -du -s -h /path/to/dirReduce the retention period for Spark event and YARN container logsA common cause of HDFS usage is the /var/log directory. The /var/log directory is where log files such as Spark event logs and YARN container logs are stored. You can change the retention period for these files to save space.The following example command displays the /var/log/spark usage.Note: /var/log/spark is the default Spark event log directory.hadoop fs -du -s -h /var/log/sparkReduce the default retention period for Spark job history filesSpark job history files are located in /var/log/spark/apps by default. When the file system history cleaner runs, Spark deletes job history files older than seven days. To reduce the default retention period, do the following:On a running cluster:1.    Connect to the primary node using SSH.2.    Add or update the following values in /etc/spark/conf/spark-defaults.conf. The following configuration runs the cleaner every 12 hrs. The configuration clear files that are more than 1 day old. You can customize this time period for your individual use case in the spark.history.fs.cleaner.internval and spark.history.fs.cleaner.maxAge parameters.------spark.history.fs.cleaner.enabled truespark.history.fs.cleaner.interval 12hspark.history.fs.cleaner.maxAge 1d------3.    Restart the Spark History Server.During cluster launch:Use the following configuration. You can customize the time period for your individual use case in the spark.history.fs.cleaner.internval and spark.history.fs.cleaner.maxAge parameters.{"Classification": "spark-defaults","Properties": {"spark.history.fs.cleaner.enabled":"true","spark.history.fs.cleaner.interval":"12h","spark.history.fs.cleaner.maxAge":"1d"   }}For more information on these parameters, see Monitoring and instrumentation in the Spark documentation.Reduce the default retention period of YARN container logsSpark application logs, which are the YARN container logs for your Spark jobs, are located in /var/log/hadoop-yarn/apps on the core node. Spark moves these logs to HDFS when the application is finished running. By default, YARN keeps application logs on HDFS for 48 hours. To reduce the retention period:1.    Connect to the primary, core, or task nodes using SSH.2.    Open the /etc/hadoop/conf/yarn-site.xml file on each node in your Amazon EMR cluster (primary, core, and task nodes).3.    Reduce the value of the yarn.log-aggregation.retain-seconds property on all nodes.4.    Restart the ResourceManager daemon. For more information, see Viewing and restarting Amazon EMR and application processes.You can also reduce the retention period by reconfiguring the cluster. For more information, see Reconfigure an instance group in a running cluster.Reduce /mnt/yarn usageIf the /mnt/yarn directory is highly utilized, adjust the user cache retention or scale the EBS volumes on the node. For more information, see How can I prevent a Hadoop or Spark job's user cache from consuming too much disk space in Amazon EMR?Resize the cluster or scale Amazon EMRAdd more core nodes to mitigate HDFS space issues. And, add any of the core or task nodes if directories other than HDFS directories are getting full. For more information, see Scaling cluster resources.You can also extend the EBS volumes in existing nodes or use a dynamic scaling script. For more information, see the following:How do I resolve "no space left on device" stage failures in Spark on Amazon EMR?Dynamically scale up storage on Amazon EMR clustersRelated informationConfigure cluster hardware and networkingHDFS configurationWork with storage and file systemsFollow"


Question: Why is my Amazon EFS file system read-only?

"My Amazon Elastic File System (Amazon EFS) file system is available and mounted correctly on my Amazon Elastic Compute Cloud (Amazon EC2) instance, but I can't write to it. How do I fix this?"

Answer: "My Amazon Elastic File System (Amazon EFS) file system is available and mounted correctly on my Amazon Elastic Compute Cloud (Amazon EC2) instance, but I can't write to it. How do I fix this?Short descriptionThe following are two common issues that prevent you from writing to your file system:The mount option in the /etc/fstab file is set to read-only access.The associated AWS Identity and Access Management (IAM) policy indicates read-only access, or root access disabled.ResolutionNote: This resolution uses the Amazon EFS mount helper. The Amazon EFS mount helper is preinstalled on Amazon Linux. If you're using another distribution, see  Installing the amazon-efs-utils package on other Linux distributions.Verify that mount options are correct in the /etc/fstab file1.    Run the following command to check the current mount options for the file system:$ mount -t nfs4In the following example output, the variable ro  indicates that the file system currently allows read-only access.file-system-id.efs.region.amazonaws.com:/ on /efs type nfs4 (ro,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,noresvport,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.0.2.0,local_lock=none,addr=192.0.0.0)2.    Change the mount parameter to rw (read/write access) in the /etc/fstab file using an editing tool such as vi:Note: Replace file-system-id with the ID of your file system.file-system-id:/ efs-mount-point efs rw,_netdev 0 03.    Run the following command to unmount and remount the file system.$ sudo mount -o remount,rw /efs -t efs && mount -t nfs4file-system-id.efs.region.amazonaws.com:/ on /efs type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,noresvport,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=190.0.2.0,local_lock=none,addr=190.0.2.0)4.    Create or edit a file in the file system to confirm that you can write to the file system.Verify that permissions are set correctly1.    Open the Amazon EFS console.Note: Make sure that you're in the same Region as your Amazon EFS file system.2.    Select the file system you want to check, and then choose View details.3.    On the File system policy tab, select Edit.4.    Uncheck the following options, if selected:Prevent root access by defaultEnforce read-only access by default5.    Select Set policy.6.    Select Save policy.7.    Run the umount command to unmount the file system.$ sudo umount /efs8.    Run the mount command to mount the file system again to apply the changes.Note: Replace file-system-id with the ID of your file system.$ sudo mount -t efs -o iam fs-file-system-id /efs9.    Add the following line to the /etc/fstab to make the new mount persistent after reboot using an editing tool such as vi:Mount with IAM authorization to an instance that has an instance profile:file-system-id:/ efs-mount-point efs _netdev,iam 0 0Mount with IAM authorization to a Linux instance using a credentials file:file-system-id:/ efs-mount-point efs _netdev,iam,awsprofile=namedprofile 0 0Mount using an EFS access point:file-system-id efs-mount-point efs _netdev,accesspoint=access-point-id 0 0For more information, see Using /etc/fstab to mount automatically.10.    Create or edit a file in the file system to confirm that you can write to the file system.Related informationNew for Amazon EFS - IAM authorization and access pointsMounting EFS file systems from another account or VPCFollow"


Question: "I configured Amazon CloudWatch to export log data to Amazon S3, but the log data is either missing or invalid. How do I resolve this issue?"

"I configured Amazon CloudWatch to export log data to Amazon Simple Storage Service (Amazon S3) as described at Exporting log data to Amazon S3 using the AWS Command Line Interface (AWS CLI). But despite completing these steps, I can't locate any useful log file data at the specified Amazon S3 destination. What do I need to do?"

Answer: "I configured Amazon CloudWatch to export log data to Amazon Simple Storage Service (Amazon S3) as described at Exporting log data to Amazon S3 using the AWS Command Line Interface (AWS CLI). But despite completing these steps, I can't locate any useful log file data at the specified Amazon S3 destination. What do I need to do?Short descriptionThis issue occurs because you must specify the time interval for the log data using timestamps expressed as the number of milliseconds that have elapsed since Jan 1, 1970 00:00:00 UTC.ResolutionExport CloudWatch log data to Amazon S3 by specifying the time interval for the log data using starting and ending timestamps that are expressed in milliseconds.For example, to export CloudWatch log data to an Amazon S3 bucket or folder for the previous two-hour period, use the following syntax:aws logs create-export-task --task-name "example-task" --log-group-name "/var/logs/example-logs" --from $(($(date -d "-2 hours" +%s%N)/1000000)) --to $(($(date +%s%N)/1000000)) --destination " log_bucket" --destination-prefix "example-logs"Related informationExporting log data to Amazon S3Linux command to get time in millisecondsExporting log data to Amazon S3 using the AWS CLIExporting log data to Amazon S3 using the CloudWatch consoleFollow"