Question: How do I resolve "OutOfMemoryError" Hive Java heap space exceptions on Amazon EMR that occur when Hive outputs the query results?

I'm running an Apache Hive query on Amazon EMR. Hive throws an OutOfMemoryError exception while outputting the query results.

Answer: "I'm running an Apache Hive query on Amazon EMR. Hive throws an OutOfMemoryError exception while outputting the query results.Short descriptionThe OutOfMemoryError exception usually happens during INSERT OVERWRITE commands when there's not enough heap space on hive-server2, the Hive metastore, or the client side. To resolve this issue, increase the maximum memory allocation for the JVM or increase HADOOP_HEAPSIZE.ResolutionUse one or more of the following solutions to resolve OutOfMemoryError exceptions.Note: These solutions don't cover OutOfMemoryError exceptions that occur during Apache Tez container memory tuning.Increase the maximum memory allocation for the JVMWhen you launch a Hive shell, 1 GB of memory is allocated by default. The maximum memory allocation is defined by the -Xmx parameter. If your process attempts to use more than the maximum value, Hive kills the process and throws the OutOfMemoryError exception. To resolve this issue, increase the -Xmx value in the Hive shell script (in MB), and then run your Hive query again.Identify OutOfMemoryError in logsCheck for the following error in your log location (For example: /mnt/var/log/hive/user/hadoop/hive.log):# java.lang.OutOfMemoryError: Java heap space# -XX:OnOutOfMemoryError="kill -9 %p"#   Executing /bin/sh -c "kill -9 12345"...KilledIf you see this error message, the JVM heap space is running out of memory. Increase HADOOP_HEAPSIZE for the Hive CLI service in /etc/hive/conf/hive-env.sh, as appears in the following example. You can also increase the HADOOP_HEAPSIZE using the Amazon EMR reconfiguration API for hive-env classification. The default value is 1000. Increase it as appropriate for your use case. Then, run the Hive query again.export HADOOP_HEAPSIZE=2048Important: This setting applies to hive-server2, the Hive metastore, and the Hive CLI after these services are restarted. Optionally, you can set separate values for each of these services.Updating memory allocation when creating a clusterAmazon EMR provides the configuration API operation to update the default configuration when you update the cluster using the configuration object. Update the HADOOP_HEAPSIZE value based on your use case:{  "Classification": "hive-env",  "Properties": {},  "Configurations": [    {      "Classification": "export",      "Properties": {        "HADOOP_HEAPSIZE": "2048"      },      "Configurations": []    }  ]}Updating memory allocation on a running clusterYou can increase the value of HADOOP_HEAPSIZE on a running cluster by either connecting to the master mode using SSH or Amazon EMR reconfiguration API. The Amazon EMR reconfiguration API is available only for Amazon EMR releases 5.21 and later. If the memory updates are made by SSH into the cluster, you must restart the hive services for the changes to take effect.Beeline or SQL Workbench/J1.    If you're running the same query from Beeline or SQL Workbench/J, check /mnt/var/log/hive/hive-server2.log and hive-server2.out for heap space garbage collection errors. For example:Transaction isolation: TRANSACTION_REPEATABLE_READ0: jdbc:hive2://emr-analytics_master.abc.aws.> select id, name, x.* from mydb.location a, curated_admin.nxpepnd1_tpn_prvdr_pra_fclt b, curated_admin.test_table c where a.test = b._id and a._id = b._id and b._prod_id = c.prod_f_id;## java.lang.OutOfMemoryError: Java heap space# -XX:OnOutOfMemoryError="kill -9 %p"#   Executing /bin/sh -c "kill -9 27745"...Killed2.    If you find errors similar to this, then increase HADOOP_HEAPSIZE for hive-server2 in /etc/hive/conf/hive-env.sh. This setting also applies to the Hive metastore and the Hive client.export HADOOP_HEAPSIZE=2048Optionally, use conditional statements to specify different heap sizes for hive-server2, the metastore, and the client. For example:export HIVE_CLIENT_HEAPSIZE=1024export HIVE_METASTORE_HEAPSIZE=2048export HIVE_SERVER2_HEAPSIZE=3072 if [ "$SERVICE" = "metastore" ]thenexport HADOOP_HEAPSIZE=$HIVE_METASTORE_HEAPSIZEelif [ "$SERVICE" = "hiveserver2" ]thenexport HADOOP_HEAPSIZE=$HIVE_SERVER2_HEAPSIZEexport HADOOP_OPTS="$HADOOP_OPTS -server -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/hive"elseexport HADOOP_HEAPSIZE=$HIVE_CLIENT_HEAPSIZEfi3.    Run the Hive query again after you update these settings. If you still get the OutOfMemoryError exception and you're running multiple clients at the same time, continue to step 4. If you still get the OutOfMemoryError exception and you aren't running multiple clients at the same time, skip to step 8.4.    Increase the -Xmx parameters for each client, as appropriate for your use case.5.    Choose the garbage collector that's appropriate for your use case by adding -XX:+UseParNewGC (new parallel garbage collector) or -XX:+UseConcMarkSweepGC (concurrent mark sweep garbage collector) in the HADOOP_OPTS lines, as shown in the following example. For more information about choosing a garbage collector, see Garbage collection and Java HotSpot VM Options in the Java documentation.export HADOOP_HEAPSIZE=2048if [ "$SERVICE" = "cli" ]; thenif [ -z "$DEBUG" ]; thenexport HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xmx12288m -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+useParNewGC -XX:-useGCOverheadLimit"elseexport HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xmx12288m -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-useGCOverheadLimit"fifi6.    Hive throws the OutOfMemoryError exception if garbage collection doesn't succeed in a specified amount of time. To remove the time limit, remove -XX:-UseGCOverheadLimit, or replace it with -XX:+UseGCOverheadLimit. Optionally, modify -XX:-UseGCOverheadLimit to specify a new time limit for garbage collection. For more information, see The Parallel Collector in the Java documentation.7.    Run the Hive query again. If Hive throws a heap space error on the terminal during runtime—and if there are no errors in hive.log or hive-server2.log—then your Hive client is probably running out of memory. For example:Transaction isolation: TRANSACTION_REPEATABLE_READ0: jdbc:hive2://emr-analytics_master.abc.aws.> select id, name, x.* from mydb.location a, curated_admin.nxpepnd1_tpn_prvdr_pra_fclt b, curated_admin.test_table c where a.test = b._id and a._id = b._id and b._prod_id = c.prod_f_id;## java.lang.OutOfMemoryError: Java heap space# -XX:OnOutOfMemoryError="kill -9 %p"#   Executing /bin/sh -c "kill -9 27745"...Killed8.    To resolve this error, increase the client memory, and then run the query again.9.    If Hive still throws the OutOfMemoryError exception, then complete the following steps for your client.Beeline:By default, Beeline attempts to buffer the entire output relation before printing it to stdout. This behavior can cause an OutOfMemoryError exception when the output relation is large. To resolve the OutOfMemoryError exception in Beeline, launch Beeline using the following command, and then retry the Hive query:beeline --incremental=trueSQL Workbench/J:In a 32-bit Java Runtime Environment (JRE), the application can use up to 1 GB of memory by default. In a 64-bit JRE, the application can use up to 65% of the available physical memory by default. To verify the amount of memory available to the application, choose Help and then choose About.For macOS: Increase the -Xmx1024m value in the Info.plist file as needed. Info.plist is usually located in the /Applications/SQLWorkbenchJ2.app/Contents directory. For example, to double the amount of memory available to the application, change the value from -Xmx1024m to -Xmx2048m. Then, run the query again.For Windows: Create an INI file and then add the vm.heapsize.preferred parameter to the INI file to increase the amount of memory that is available to the application.If you're using shell or batch scripts, you can increase the available memory when you install SQL Workbench/J. The command in the following example creates 3 GB of available memory during installation:java -Xmx3g -jar sqlworkbench.jarNote: If the OutOfMemoryError exception is on the client side rather than on hive-server2 or the Hive CLI, save the output to Amazon Simple Storage Service (Amazon S3) or HDFS. Don’t use Beeline or SQL Workbench/J to view the query results.Restarting HiveIf you modify Hive properties in hive-site.xml or hive-env.sh, you might need to restart Hive before the updated settings take effect. Restarting hive-server 2 impacts the running queries on the cluster. It's a best practice to restart the processes when no queries are running or during a planned maintenance window.The commands that you use to restart hive processes vary depending on the Amazon EMR release version. For more information, see How do I restart a service in Amazon EMR?For Amazon EMR release versions 5.30 and later or 6.0 and later:1.    Connect to the master node using SSH.2.    Restart the metastore:sudo systemctl stop hive-hcatalog-serversudo systemctl start hive-hcatalog-serversudo systemctl status hive-hcatalog-server3.    Restart hive-server 2:sudo systemctl stop hive-server2sudo systemctl start hive-server2sudo systemctl status hive-server2For Amazon EMR release versions 4.7.0 to 5.29:1.    Connect to the master node using SSH.2.    Restart the metastore:sudo stop hive-hcatalog-serversudo start hive-hcatalog-serversudo status hive-hcatalog-serverNote: Don't attempt to restart the metastore using the sudo restart hive-hcatalog-server command.3.    Restart hive-server2:sudo stop hive-server2sudo start hive-server2sudo status hive-server2For Amazon EMR release versions 4.0.0 to 4.6:1.    Connect to the master node using SSH.2.    Restart the metastore:sudo stop hive-metastoresudo start hive-metastoresudo status hive-metastore3.    Restart hive-server2:sudo stop hive-server2sudo start hive-server2sudo status hive-server2Related informationHow can I use logs to troubleshoot issues with Hive queries in Amazon EMR?Scaling cluster resourcesTask configurationFollow"


Question: How do I run SQL commands on an Amazon Redshift table before or after writing data in an AWS Glue job?

I have an AWS Glue job that loads data into an Amazon Redshift table. I want to run SQL commands on Amazon Redshift before or after the AWS Glue job completes.

Answer: "I have an AWS Glue job that loads data into an Amazon Redshift table. I want to run SQL commands on Amazon Redshift before or after the AWS Glue job completes.ResolutionPass one of the following parameters in the AWS Glue DynamicFrameWriter class:aws_iam_role: Provides authorization to access data in another AWS resource. Use this parameter with the fully specified ARN of the AWS Identity and Access Management (IAM) role that's attached to the Amazon Redshift cluster. For example, use arn:aws:iam::123456789012:role/redshift_iam_role. For more information, see Authorization parameters.preactions: A semicolon-delimited list of SQL commands that are run before the COPY command. If the commands fail, then Amazon Redshift throws an exception.Note: Be sure that the preaction parameter doesn't contain newline characters.postactions: A semicolon-delimited list of SQL commands that are run after a successful COPY command. If the commands fail, then Amazon Redshift throws an exception.Note: Be sure that the postaction parameter doesn't contain newline characters.extracopyoptions: A list of additional options to append to the Amazon Redshift COPY command when loading data. For example, you might use TRUNCATECOLUMNS or MAXERROR.Example scenariosTruncate an Amazon Redshift table before inserting records in AWS GlueUse the preactions parameter.See the following Python example:datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf(frame= datasource0, catalog_connection = "test_red", connection_options = {"preactions":"truncate table schema.target_table;","dbtable": "schema.target_table", "database": "redshiftdb"}, redshift_tmp_dir = 's3://s3path', transformation_ctx = "datasink4")See the following Scala example:val options = JsonOptions(Map(   "dbtable" -> "schema.target_table",   "database" -> "redshiftdb",   "preactions" -> "truncate table schema.target_table;"   ))glueContext.getJDBCSink(catalogConnection = "test_red", options = options, redshiftTmpDir = 's3://s3path', transformationContext = "datasource0").writeDynamicFrame(datasource0)In these examples, be sure to replace the following values:test_red: the catalog connection to useschema.target_table: the Amazon Redshift database's schema and the Amazon Redshift tables3://s3path: the path of the Amazon Redshift table's temporary directoryUse an IAM role in the connection optionsBecause credentials expire after 1 hour, use an IAM role in the connection options to stop your long running connections from failing.See the following Python example:glueContext.create_dynamic_frame.from_catalog(database = "redshift-database-name",  table_name = "redshift-table-name", redshift_tmp_dir = args["TempDir"], additional_options = {"aws_iam_role": "arn:aws:iam::account-id:role/role-name"})See the following Scala example:val connectionOptions = JsonOptions(Map(      "url" -> "jdbc:redshift://your_redshift_cluster.us-west-2.redshift.amazonaws.com:5439/database",      "dbtable" -> "schema.table",      "user" -> "redshift_user",      "password" -> "redshift_password",      "tempdir" -> "s3://temp_bucket/temp",      "aws_iam_role" -> "arn:aws:iam::your_account_id:role/your_role_name" ))val dyf = glueContext.getSource("redshift", connectionOptions)          .getDynamicFrame()Merge an Amazon Redshift table in AWS Glue (upsert)Create a merge query after loading the data into a staging table.Note: For your merge query to work, target_table must already exist in your Amazon Redshift database.See the following Python example:post_query="begin;delete from schema.target_table using schema.stage_table where schema.stage_table.id = schema.target_table.id ; insert into schema.target_table select * from schema.stage_table; drop table schema.stage_table; end;"datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf(frame = datasource0, catalog_connection = "test_red", connection_options = {"preactions":"drop table if exists schema.stage_table;create table schema.stage_table as select * from schema.target_table where 1=2;","dbtable": "schema.stage_table", "database": "redshiftdb","postactions":post_query},redshift_tmp_dir = 's3://s3path', transformation_ctx = "datasink4")See the following Scala example:val options = JsonOptions(Map(   "dbtable" -> "schema.stage_table",   "database" -> "redshiftdb",   "preactions" -> "drop table if exists schema.stage_table;create table schema.stage_table as select * from schema.target_table where 1=2;",   "postactions" -> "begin;delete from schema.target_table using schema.stage_table where schema.stage_table.id = schema.target_table.id ; insert into schema.target_table select * from schema.stage_table; drop table schema.stage_table; end;"   ))glueContext.getJDBCSink(catalogConnection = "test_red", options = options, redshiftTmpDir = 's3://s3path', transformationContext = "datasink4").writeDynamicFrame(datasink4)In these examples, be sure to replace the following values:schema.target_table: the Amazon Redshift database's schema and the Amazon Redshift tabletest_red: the catalog connection to useschema.stage_table: the Amazon Redshift database's schema and the Amazon Redshift staging tables3://s3path: the path of the Amazon Redshift table's temporary directoryFor more information, see Use a staging table to perform a merge (upsert).Ignore rows that aren't validUse the extracopyoptions parameter to specify a higher MAXERROR value.See the following Python example:datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf(frame = persons_DyF, catalog_connection = "test", connection_options = {"dbtable": "testalblog2", "database": "reddb","postactions":"delete from emp1;","extracopyoptions":"MAXERROR 2"},redshift_tmp_dir = 's3://s3path', transformation_ctx = "datasink4")See the following Scala example:val options = JsonOptions(Map(   "dbtable" -> "testalblog2",   "database" -> "reddb",   "preactions" -> "drop table if exists schema.stage_table;create table schema.stage_table as select * from schema.target_table where 1=2;",   "postactions" -> "delete from emp1;",   "extracopyoptions" -> "MAXERROR 2"   ))glueContext.getJDBCSink(catalogConnection = "test", options = options, redshiftTmpDir = 's3://s3path', transformationContext = "datasink4").writeDynamicFrame(persons_DyF)In these examples, be sure to replace the following values:schema.target_table: the Amazon Redshift database's schema and the Amazon Redshift tableschema.stage_table: the Amazon Redshift database's schema and the Amazon Redshift staging tabletest: the catalog connection to usetestalblog2: the Amazon Redshift table to load data intoreddb: the Amazon Redshift databaseemp1: the Amazon Redshift table to delete the data from, after the data is loaded into testalblog2s3://s3path: the path of the Amazon Redshift table's temporary directoryAdditional informationYou can use the Amazon Redshift Spark connector (redshift-jdbc42-2.1.0.9) when using AWS Glue 4.0 ETL jobs. This connector has the following properties:Supports IAM-based JDBC URLs.Includes performance improvement options like autopushdown, autopushdown.s3_result_cache, and unload_s3_format.Includes the SSE_KMS encryption option that can be used for data in the temporary folder. AWS Glue uses this data when reading from Amazon Redshift tables.Related informationCOPYTRUNCATEData load operationsFollow"


Question: Why can't I increase the Amazon EBS disk volume in my Amazon MSK cluster?

I can't increase the Amazon Elastic Block Store (Amazon EBS) disk volume in my Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster.

Answer: "I can't increase the Amazon Elastic Block Store (Amazon EBS) disk volume in my Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster.ResolutionThe most common causes for this issue are as follows:You don't have the required permissions to modify the Amazon EBS volume.Your cluster is in the optimizing state from a prior Amazon EBS volume upgrade.You don't have the required permissions to modify the Amazon EBS volumeMake sure that you have the required permissions to modify the EBS disk volume for your cluster. Make sure that you attached the AmazonMSKFullAccess or AmazonMSKClusterManagement policy to your AWS Identity and Access Management (IAM) user or role.Your cluster is in the optimizing state from a prior Amazon EBS volume upgradeIn this case, you get the error "An EBS volume in this cluster is under modification from a previous operation".When you increase the size of an EBS volume that's attached to your Amazon MSK broker node, the newly added storage is available to you immediately. However, it takes from 6 to 24 hours or more for the cluster to be optimized. The duration for cluster optimization is proportional to the size of the storage.When you increase the size of the EBS volume, make sure that the added volume is sufficient for the next 48 hours or more. During this period, you can't add more volumes to your cluster. You can't update the volume further until the cluster is optimized.Important: Suppose that your cluster is being optimized and you're already running out of disk space. In this case, be sure to turn off automatic scaling if you previously turned it on. Otherwise, when optimization is complete, the volume is increased only by 10% or 10 GB. This might be too low, causing similar issues with volume modification. After turning off automatic scaling, be sure to analyze your traffic patterns and increase your storage space accordingly.If your cluster is undergoing modification and the KafkaDataLogsDiskUsed metric is less than 94%, you can delete unused or old topics to free up space. Run the following command to determine the space occupied per topic:kafka-topics.sh --command-config /tmp/client.properties --bootstrap-server $bservers --list | xargs -I{} sh -c "echo -n '{} - ' && kafka-log-dirs.sh --command-config /tmp/client.properties --bootstrap-server $bservers --topic-list {} --describe | grep '^{'  | jq '[ ..|.size? | numbers ] | add' | numfmt --to iec " | tee /tmp/topics-by-size.listThen, delete the unused or old topics to free up disk space.Note: If KafkaDataLogsDiskUsed is higher than 94%, then this command might be unresponsive. In this case, you can't delete the topics.Follow"


Question: What is EFS One Zone and how do I convert my existing multi-zone EFS to EFS One Zone?

What is EFS One Zone and how to I convert existing multi-zone Amazon Elastic File System (Amazon EFS) to EFS One Zone?

Answer: "What is EFS One Zone and how to I convert existing multi-zone Amazon Elastic File System (Amazon EFS) to EFS One Zone?ResolutionAmazon EFS offers storage classes for different use cases, including the following:EFS Standard and EFS Standard–Infrequent Access (Standard-IA)EFS One Zone and EFS One Zone–Infrequent Access (EFS One Zone-IA)For more information on the difference between these storage classes, see Comparing Amazon EFS storage classes.How to convert existing multi-zone EFS to EFS One ZoneThere isn't a direct solution to convert the existing multi-zone EFS to EFS One Zone. To move to EFS One Zone, you can do one of the following:Create a new EFS One Zone and then copy the data from the existing EFS volumes. For more information, see Transferring data into and out of Amazon EFS.-or-Use AWS Backup to restore the entire file system with the EFS One Zone file system type. For more information, see How can I back up and restore my Amazon EFS file systems?Follow"


Question: How do I migrate my Elastic Beanstalk environment from one AWS account to another AWS account?

I want to migrate my AWS Elastic Beanstalk environment from one AWS account to another AWS account.

Answer: "I want to migrate my AWS Elastic Beanstalk environment from one AWS account to another AWS account.Short descriptionYou must use saved configurations to migrate an Elastic Beanstalk environment between AWS accounts.Saved configurations are stored in the Amazon Simple Storage Service (Amazon S3) bucket for Elastic Beanstalk in a folder named after your application. For example, the location of a saved configuration for an application named my-app in the us-east-1 AWS Region for account 01234567890 looks like this:s3://elasticbeanstalk-us-east-1-01234567890/resources/templates/my-app/ResolutionIn the following steps, Account A is the account where you have your Elastic Beanstalk environment. Account B is the account that you want to migrate your Elastic Beanstalk Environment to.Note: In this example, both Account A and Account B have Elastic Beanstalk environments already created. The saved configuration is created by Elastic Beanstalk when you create the environment. This environment is uploaded to Account B's Amazon S3 bucket.Create a saved configuration for Account A1.    Create a saved configuration in Account A.2.    Download the saved configuration to your local machine from the following:s3://elasticbeanstalk-us-east-1-account-a-id/resources/templates/application-a-name/Note: In your saved configuration, us-east-1 is the AWS Region where the Elastic Beanstalk environment is located in Account A. The account-a-id is your account ID for Account A. The application-a-name is the application name of the Elastic Beanstalk application in Account A.3.    Change your account-specific parameters in the downloaded configuration file, and then save the changes. For example, change the key pair name, subnet ID, or application name (such as application-b-name).Create an Elastic Beanstalk environment using the saved configuration in Account B1.    Log in to Account B.2.    Upload the saved configuration from your local machine to an S3 bucket in Account B.The S3 path looks similar to the following:s3://elasticbeanstalk-us-east-2-account-b-id/resources/templates/application-b-name/3.    Open the Elastic Beanstalk console.4.    From the Regions list, select your AWS Region.5.    From the navigation pane, choose Applications, and then choose Create a new application.6.    For Application name, enter the name of your new application  (for example, application-b-name).7.    From the left navigation pane, in the section for your new application, choose Saved configurations.Now you can see all the saved configurations that you have in the S3 bucket in a given Region for your account.8.    Choose the saved configuration that you previously uploaded to the S3 bucket in the following account:s3://elasticbeanstalk-us-east-2-account-b-id/resources/templates/application-b-name/9.    Launch your Elastic Beanstalk environment.Follow"


Question: How do I share an Amazon Machine Image (AMI) privately with another AWS account?

I want to make an Amazon Machine Image (AMI) that I've created available to another AWS account.

Answer: "I want to make an Amazon Machine Image (AMI) that I've created available to another AWS account.ResolutionIf you know the account IDs of the AWS accounts that you want to share the AMI with, follow the instructions at Share an AMI with specific AWS accounts.To copy the shared AMI, see Cross-account copying.Note: Sharing an AMI from different Regions isn't available. You can copy the AMI and then share it or launch it in a new Region. For more information, see How do I create and copy an Amazon Machine Image (AMI) from one AWS Region to another?Related informationShared AMIsFollow"


Question: Why does my Amazon Redshift query keep exceeding the WLM timeout that I set?

"I set a workload management (WLM) timeout for an Amazon Redshift query, but the query keeps running after this period expires. Why is this happening?"

Answer: "I set a workload management (WLM) timeout for an Amazon Redshift query, but the query keeps running after this period expires. Why is this happening?Short descriptionA WLM timeout applies to queries only during the query running phase. If WLM doesn’t terminate a query when expected, it’s usually because the query spent time in stages other than the execution stage. For example, the query might wait to be parsed or rewritten, wait on a lock, wait for a spot in the WLM queue, hit the return stage, or hop to another queue.ResolutionWhen querying STV_RECENTS, starttime is the time the query entered the cluster, not the time that the query begins to run. When the query is in the Running state in STV_RECENTS, it is live in the system. However, the query doesn't use compute node resources until it enters STV_INFLIGHT status. For more information about query planning, see Query planning and execution workflow.To view the status of a running query, query STV_INFLIGHT instead of STV_RECENTS:select * from STV_INFLIGHT where query = your_query_id;Use this query for more information about query stages:select * from SVL_QUERY_REPORT where query = your_query_id ORDER BY segment, step, slice;Use the STV_EXEC_STATE table for the current state of any queries that are actively running on compute nodes:select * from STV_EXEC_STATE where query = your_query_id ORDER BY segment, step, slice;Here are some common reasons why a query might appear to run longer than the WLM timeout period:The query is in the "return" phaseThere are two "return" steps. Check STV_EXEC_STATE to see if the query has entered one of these return phases:The return to the leader node from the compute nodesThe return to the client from the leader nodeA rollback is in progressIf a data manipulation language (DML) operation encounters an error and rolls back, the operation doesn't appear to be stopped because it is already in the process of rolling back. You can view rollbacks by querying STV_EXEC_STATE. You can find additional information in STL_UNDONE.The query spends time queuing prior to runningQuery STV_WLM_QUERY_STATE to see queuing time:select * from STV_WLM_QUERY_STATE where query = your_query_id;The query is waiting on a lockIf the query is visible in STV_RECENTS, but not in STV_WLM_QUERY_STATE, the query might be waiting on a lock and hasn't entered the queue. For more information about checking for locks, see How do I detect and release locks in Amazon Redshift?A query hopped to another queueIf a read query reaches the timeout limit for its current WLM queue, or if there's a query monitoring rule that specifies a hop action, then the query is pushed to the next WLM queue. To confirm whether the query hopped to the next queue:While the query is running: query STV_WLM_QUERY_STATEAfter the query runs: query STL_WLM_QUERYTo prevent queries from hopping to another queue, configure the WLM queue or WLM query monitoring rules. For more information about query hopping, see WLM query queue hopping.A networking or firewall issueIf an Amazon Redshift server has a problem communicating with your client, then the server might get stuck in the "return to client" state. Check for conflicts with networking components, such as inbound on-premises firewall settings, outbound security group rules, or outbound network access control list (network ACL) rules. For more information, see Connecting from outside of Amazon EC2 —firewall timeout issue.An issue with the clusterIssues on the cluster itself, such as hardware issues, might cause the query to freeze. When this happens, the cluster is in "hardware-failure" status. To recover a single-node cluster, restore a snapshot. In multi-node clusters, failed nodes are automatically replaced.Related informationTuning query performanceAnalyzing and improving queriesFollow"


Question: Why is my Amazon Redshift snapshot missing some tables?

"I restored a snapshot from an Amazon Redshift cluster, but the snapshot is missing some tables. How do I back up my missing tables?"

Answer: "I restored a snapshot from an Amazon Redshift cluster, but the snapshot is missing some tables. How do I back up my missing tables?Short descriptionTables created as no-backup tables are excluded from Amazon Redshift snapshots. Verify whether Amazon Redshift excluded your table because it was created using the BACKUP NO parameter. Then, perform a deep copy.ResolutionTo back up missing tables from your Amazon Redshift snapshot, perform the following steps:1.    Check the Data Definition Language (DDL) of the tables that are missing from the snapshot.2.    If the table's DDL is unavailable, then run the following query as a superuser:SELECT DISTINCT Rtrim(n.nspname) AS schema_name,                Rtrim(name)      AS table_name,                backupFROM   stv_tbl_perm t       join pg_class c         ON t.id = c.oid       join pg_namespace n         ON n.oid = c.relnamespaceORDER  BY 1,2;The query above identifies tables in the connected database that aren't backed up in the snapshot. It queries STV_TBL_PERM, the system table that is only visible to superuser accounts. For more information about views that are only available to superusers, see Visibility of data in system tables and views.Note: A value of 0 for the backup column indicates that the table was created using the BACKUP NO parameter. You cannot alter an existing table in Amazon Redshift using the BACKUP YES parameter.3.    If your table was created as a no-backup table, then recreate the table without the BACKUP NO parameter.4.    Perform a deep copy of your missing tables.Related informationCREATE TABLE AS ParametersAmazon Redshift SnapshotsFollow"


Question: How do I add or update billing contact information?

I want to add or update a billing contact to my AWS account so they can communicate with AWS Finance teams. How do I do this?

Answer: "I want to add or update a billing contact to my AWS account so they can communicate with AWS Finance teams. How do I do this?Short descriptionApproved billing contacts can communicate with AWS Finance teams about your account to support payments made to AWS. Billing contacts also receive the following:Monthly invoicesConsolidated billing statementsBilling- or payment-related notificationsNote: Notifications might be shared by AWS Accounts Receivable, AWS Support, or other AWS Service teams.Important: Adding or updating a billing contact is available only if you Pay by invoice. If you make payments using a credit card or other method, see Adding, changing, or removing alternate contacts to add an alternate contact for Billing.ResolutionTo access the Billing and Cost Management console, you must sign in with:AWS account root user credentials-or-AWS Identity and Access Management (IAM) user credentials with access permissions for the AWS Billing console. For instructions on activating access, see Activating access to the AWS Billing console.It's a best practice to activate access to the AWS Billing console for IAM users who are responsible for billing-related communications. To communicate with AWS Finance teams, IAM users with AWS Billing console access must also be added to the Billing contact email list.To add or update an approved billing contact email address, do the following:Sign in to the Billing and Cost Management console.In the navigation pane, choose Payment Methods.For Pay by Invoice, choose Edit next to the default billing address.For Billing contact email(s), choose Add additional email, and then enter the billing contact's email address.Repeat this step for each billing contact that you want to add.To update a billing contact, choose Edit next to the email address.Choose Update.Tip: It's a best practice to delegate an IAM user responsible for AWS payments and billing to maintain the billing contact list. Bookmark this page, share it with the responsible IAM user, and repeat this process periodically to maintain an updated record of authorized contacts.Frequently asked questionsCan AWS edit billing contact information on my behalf?No, to maintain the highest level of security, AWS can't edit your billing contact information.Can I add more than one approved billing contact?Yes, you can register up to 13 billing contacts.Can I add a shared email address as a billing contact?Yes, adding a shared email address as a billing contact provides relevant account information as email recipients. Note that emails to AWS must be sent from a registered email address to receive support. If there's one individual contact who's most likely to share in payment responsibilities, this individual should be registered as an alternate billing contact.Can billing contacts view or change my AWS account settings?Adding an alternate billing contact allows AWS to communicate with registered individuals if you're unavailable or prefer to share responsibility of payment-related processes. Only root users or IAM users with certain permissions are able to view or change account settings.If I add billing contacts, will I still receive communications about my account?Yes, primary account holders will continue to receive all account-related email communications.If I add a billing contact on one AWS account, will this contact be automatically added for my other AWS accounts?Billing contacts added on the Payment Methods page are registered at the individual account level and related permissions apply to only that specific account. If you have multiple accounts with AWS, you can follow the steps on this page for each individual account.If you're using AWS Organizations, you can automate the assignment of one central billing contact across all AWS accounts. For more information, see Programmatically managing alternate contacts on member accounts with AWS Organizations. For individual account billing inquiries, additional billing contacts must be added for each individual account.Follow"


Question: How do I detect and troubleshoot ReadProvisionedThroughputExceeded exceptions in Kinesis Data Streams?

I'm encountering a ReadProvisionedThroughputExceeded error in Amazon Kinesis Data Streams. Why is this happening and how do I troubleshoot this?

Answer: "I'm encountering a ReadProvisionedThroughputExceeded error in Amazon Kinesis Data Streams. Why is this happening and how do I troubleshoot this?Short descriptionThe ReadProvisionedThroughputExceeded error occurs when GetRecords calls are throttled by Kinesis Data Streams over a duration of time.Your Amazon Kinesis data stream can be throttled if the following limits are breached:Each shard can support up to five read transactions per second (or five GetRecords calls/second for each shard).Each shard can support up to a maximum read rate of two MiB/second.GetRecords can retrieve up to ten MiB of data per call from a single shard, and up to 10,000 records per call. If a call to GetRecords returns ten MiB of data, then subsequent calls made within the next five seconds result in an error.If you encounter a ReadProvisionedThroughputExceeded error, consider the following approaches:Identify the root cause of your issue.Identify a possible microburst.Follow Kinesis Data Streams best practices.ResolutionIdentify the root cause of your issueTo identify the root cause of the ReadProvisionedThroughputExceeded error in Data Streams, monitor the Amazon Kinesis Data Streams Service with Amazon CloudWatch. Pay attention to the following metrics in CloudWatch:GetRecords.Bytes: The number of bytes retrieved from the Kinesis data stream, measured over a specified time period.GetRecords.Records: The number of records retrieved from the Kinesis data stream over a specified time period.ReadProvisionedThroughputExceeded: The number of GetRecords calls that are throttling in your Kinesis data stream.Set up your CloudWatch dashboard to display your statistics as a Sum with the time period set to one minute. Then, divide Sum by 60 seconds to get an average value.For example, if use the GetRecords.Records metric value, divide Sum by 60 seconds to calculate the average number of records sent per second. Then, check if the average value is less than the records sent per second for the limit that is set for your Kinesis data stream. For more information about shard limits, see Kinesis Data Streams quotas and limits.Note: You can enable the enhanced monitoring feature to ensure that the load is being evenly distributed across all of your shards.You can also use the GetRecords.Records metric with the statistic viewed as a SampleCount and the time period set to one minute. Divide the SampleCount value by 60 seconds to calculate the average number of GetRecords calls made per second for each shard. If the average value is around five GetRecords calls per second and you're getting a ReadProvisionedThroughputExceeded error, verify that your consumers aren't exceeding shard limits. If they're not exceeding shard limits, then the ReadProvisionedThroughputExceeded error could be a result of your consumers making more than five GetRecords calls/second.Finally, check if there is disparity between the ReadProvisionedThroughputExceeded value of your shards. If the distribution of shards is uneven, or if one shard receives more or less data than the other, then there can be a distribution imbalance. To resolve this shard distribution imbalance and to avoid hot shards, use UUID as a partition key in the putRecords API call.Identify a possible microburstAlthough rare, metric values can be below shard limits, causing a Kinesis data stream to throttle during a read.For example, consider a scenario where GetRecords.Bytes Sum:1min represents ten MiB of data read for one minute. At one second, the GetRecords.Bytes call reads two MiB of data without any throttling. Then, at two seconds, the GetRecords.Bytes call reads eight MiB of data. At three seconds, there might not be any read operations or any throttling. Although the shard limit for the minute hasn't been reached (2MiB * 60 = 120MiB of data), you might receive a ReadProvisionedThroughputExceeded error. If you notice a sudden spike in the metric values, look for the microburst that's causing the ReadProvisionedThroughputExceeded exception.Follow Data Streams best practicesTo mitigate ReadProvisionedThroughputExceeded exceptions, apply these best practices:Reshard your stream to increase the number of shards in the stream.Reduce the size of the GetRecords requests. You can do this by configuring the limit parameter or by reducing the frequency of GetRecords requests.Note: If the consumer is Amazon Kinesis Data Firehose, then the Kinesis data stream adjusts to the frequency of the GetRecords calls that are made. If the consumer is an AWS Lambda function with event source mapping, then the stream is polled once every second. The polling frequency can't be modified. If the consumer is an Amazon Kinesis Client Library (KCL) application, then adjust the polling frequency. To adjust the polling frequency, modify the DEFAULT_IDLETIME_BETWEEN_READS_MILLIS parameter value in the KinesisClientLibConfiguration file. You can set this value dynamically in the code. For more information about how to modify this value in the KCL, see Amazon Web Services - Labs on the GitHub website.Distribute read and write operations as evenly as possible across all of the shards in Data Streams.Use consumers with enhanced fan-out. For more information about enhanced fan-out, see Developing custom consumers with dedicated throughput (enhanced fan-out).Note: If your Kinesis data stream uses more than five consumers, it's a best practice to use consumers with enhanced fan-out.Use an error retry and exponential backoff mechanism in the consumer logic if ReadProvisionedThroughputExceeded exceptions are encountered. For consumer applications that use an AWS SDK, the requests are retried by default.Follow"


Question: How can I troubleshoot "429 Too Many Requests" or "Limit Exceeded" errors for my API Gateway API?

I received "429 Too Many Requests" or "Limit Exceeded" errors when sending requests to my Amazon API Gateway API. How do I troubleshoot these errors?

Answer: "I received "429 Too Many Requests" or "Limit Exceeded" errors when sending requests to my Amazon API Gateway API. How do I troubleshoot these errors?Short descriptionAPI Gateway has account-level quotas, per Region. The throttle quota is 10,000 requests per second (RPS) with an additional burst capacity provided by the token bucket algorithm. The maximum bucket capacity is 5,000 requests per account and Region. API Gateway throttling-related settings are applied in the following order:Per-client or per-method throttling limits that you set for an API stage in a usage plan.Per-method throttling limits that you set for an API stage.Account-level throttling per Region.AWS Regional throttling.Exceeding the throttling limit or quota returns a "429 Too Many Requests" or "Limit Exceeded" error response.For more information, see How throttling limit settings are applied in API Gateway.ResolutionBefore you begin, make sure that you have followed the instructions to turn on Amazon CloudWatch Logs for troubleshooting API Gateway. Make sure that you choose ERROR to generate execution logs only for requests to your API that result in an error. Then, view logged API requests and responses using the CloudWatch console."429 Too Many Requests" errorCheck the rate or burst limit for per-client or per-method throttling limits that you set for the API stage for your usage plan. When the rate or burst limits are exceeded, CloudWatch execution logs an exceeded throttle limit error similar to the following:(f277a0b4-2bcd-41b3-8e43-4de770663ffb) API Key **********************************F0yrv6 exceeded throttle limit for API Stage rohkz08x02/dev: Key throttle limit exceeded for Usage Plan ID nnpegc, RestApi rohkz08x02, Stage dev, Resource f646q2, HttpMethod GET. Limit: 5.00 Burst: 10To resolve this error, use retries and an exponential backoff algorithm with jitter, and then resubmit your API request.For more information, see exponential backoff and jitter."Limit Exceeded" errorThis error might indicate that the quota limit is exceeded for your API Gateway usage plan. When the quota limit is exceeded, then CloudWatch execution logs an exceeded quota limit error similar to the following:(7b819c41-e0a0-433a-883e-bc461fd70fd6) API Key **********************************F0yrv6 exceeded quota limit for API Stage rohkz08x02/dev: Key quota exhausted for Usage Plan ID nnpegc. QLimit: 500 Period: DAYTo resolve this error, follow the instructions to extend the remaining quota.Related informationHow do I troubleshoot Lambda function throttling with "Rate exceeded" and 429 "TooManyRequestsException" errors?How do I find API Gateway REST API errors in my CloudWatch logs?Follow"


Question: How do I check running queries and diagnose resource consumption issues for my Amazon RDS or Aurora PostgreSQL DB instance?

I need to see which queries are actively running on an Amazon Relational Database Service (Amazon RDS) or Amazon Aurora PostgreSQL DB instance. How can I do this?

Answer: "I need to see which queries are actively running on an Amazon Relational Database Service (Amazon RDS) or Amazon Aurora PostgreSQL DB instance. How can I do this?ResolutionCheck running queriesYour user account must be granted the rds_superuser role to see all the processes that are running on a DB instance of RDS for PostgreSQL or Aurora PostgreSQL. Otherwise, pg_stat_activity shows only queries that are running for its own processes. For more information, see the PostgreSQL documentation for The Statistics Collector.1.    Connect to the DB instance that is running PostgreSQL or Aurora PostgreSQL.2.    Run the following command:SELECT * FROM pg_stat_activity ORDER BY pid;You can also modify this command to view the list of running queries ordered by when the connections were established:SELECT * FROM pg_stat_activity ORDER BY backend_start;If the column value is null, then there is no transaction opened in that session:SELECT * FROM pg_stat_activity ORDER BY xact_start;Or, view the same list of running queries ordered by when the last query was executed:SELECT * FROM pg_stat_activity ORDER BY query_start;For an aggregated view of the wait events, if there are any, run the following command:select state, wait_event, wait_event_type, count(*) from pg_stat_activity group by 1,2,3 order by wait_event;Diagnose resource consumptionBy using pg_stat_activity and enabling Enhanced Monitoring, you can identify the query/process that is consuming large amounts of system resources. After enabling Enhanced Monitoring, be sure that the granularity set is sufficient to see the information that you need to diagnose the issue. Then, you can check pg_stat_activity to see the current activities in your database and the Enhanced Monitoring metrics at that time.1.    Identify the query that is consuming resources by viewing the OS process list metric. In the following example, the process is consuming about 95% of the CPU time on the RDS DB instance. The process ID (pid) of the process is 14431and the process is executing a SELECTstatement. You can also see the usage of system memory by checking the MEM%.NAMEVIRTRESCPU%MEM%VMLIMITpostgres: master postgres 27.0.3.145(52003) SELECT [14431]457.66 MB27.7 MB95.152.78unlimited2.    Connect to the DB instance that is running PostgreSQL or Aurora PostgreSQL.3.    Identify the current activity of the session by running the following command:SELECT * FROM pg_stat_activity WHERE pid = PID;**Note:**Replace PIDwith the pid that you identified in the step 1.4.    Check the result of the command:datid            | 14008datname          | postgrespid              | 14431usesysid         | 16394usename          | masterapplication_name | psqlclient_addr      | 27.0.3.145client_hostname  |client_port      | 52003backend_start    | 2020-03-11 23:08:55.786031+00xact_start       | 2020-03-11 23:12:16.960942+00query_start      | 2020-03-11 23:12:16.960942+00state_change     | 2020-03-11 23:12:16.960945+00wait_event_type  |wait_event       |state            | activebackend_xid      |backend_xmin     | 812query            | SELECT COUNT(*) FROM columns c1, columns c2, columns c3, columns c4, columns c5;backend_type     | client backendTo stop the process that is executing the query, invoke the following query from another session. Be sure to replace PIDwith the pid of the process that you identified in step 3.SELECT pg_terminate_backend(PID);Important: Before terminating transactions, evaluate the potential impact that each transaction has on the state of your database and your application.Related informationHow can I troubleshoot high CPU utilization for Amazon RDS or Amazon Aurora PostgreSQL?Creating rolesEnhanced MonitoringPostgreSQL documentation for psqlPostgreSQL documentation for pg_stat_activityFollow"


Question: How do I configure a Lambda function to assume an IAM role in another AWS account?

I need my AWS Lambda function to assume an AWS Identity and Access Management (IAM) role in another AWS account. How do I set that up?

Answer: "I need my AWS Lambda function to assume an AWS Identity and Access Management (IAM) role in another AWS account. How do I set that up?Short descriptionTo have your Lambda function assume an IAM role in another AWS account, do the following:Configure your Lambda function's execution role to allow the function to assume an IAM role in another AWS account.Modify your cross-account IAM role's trust policy to allow your Lambda function to assume the role.Add the AWS Security Token Service (AWS STS) AssumeRole API call to your Lambda function's code.Note: A Lambda function can assume an IAM role in another AWS account to do either of the following:**Access resources—**For example, accessing an Amazon Simple Storage Service (Amazon S3) bucket.**Do tasks—**For example, starting and stopping instances.ResolutionNote: The following example procedure references two different types of AWS accounts:A home account that hosts the Lambda function ( 111111111111).A cross-account that includes the IAM role that the Lambda function assumes (222222222222)The procedure assumes:You have created the IAM role that you want to use in the cross-account (222222222222)Configure your Lambda function's execution role to allow the function to assume an IAM role in another AWS accountAdd the following policy statement to your Lambda function's execution role (in account 111111111111) by following the instructions in Adding and removing IAM identity permissions:Important: Replace 222222222222 with the AWS account ID of the cross-account role that your function is assuming. Replace role-on-source-account with the assumed role's name.{    "Version": "2012-10-17",    "Statement": {        "Effect": "Allow",        "Action": "sts:AssumeRole",        "Resource": "arn:aws:iam::222222222222:role/role-on-source-account"    }}Modify your cross-account IAM role's trust policy to allow your Lambda function to assume the roleAdd the following policy statement to your cross-account IAM role's trust policy (in account 222222222222) by following the instructions in Modifying a role trust policy (console):Important: Replace 111111111111 with the AWS account ID of the account that your Lambda function is in. Replace my-lambda-execution-role with the name of your function's execution role.{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Principal": {                "AWS": "arn:aws:iam::111111111111:role/my-lambda-execution-role"            },            "Action": "sts:AssumeRole"        }    ]}Add the AWS STS AssumeRole API call to your Lambda function's codeAdd the AWS STS AssumeRole API call to your function's code by following the instructions in Configuring Lambda function options.Note: The AWS STS AssumeRole API call returns credentials that you can use to create a service client. By using this service client, your Lambda function has the permissions granted to it by the assumed role. For more information, see assume_role in the AWS SDK for Python (Boto 3) documentation.Python function code example that includes the AWS STS AssumeRole API callImportant: Replace 222222222222 with the AWS account ID of the cross-account role that your function is assuming. Replace role-on-source-account with the assumed role's name.import boto3def lambda_handler(event, context):    sts_connection = boto3.client('sts')    acct_b = sts_connection.assume_role(        RoleArn="arn:aws:iam::222222222222:role/role-on-source-account",        RoleSessionName="cross_acct_lambda"    )        ACCESS_KEY = acct_b['Credentials']['AccessKeyId']    SECRET_KEY = acct_b['Credentials']['SecretAccessKey']    SESSION_TOKEN = acct_b['Credentials']['SessionToken']    # create service client using the assumed role credentials, e.g. S3    client = boto3.client(        's3',        aws_access_key_id=ACCESS_KEY,        aws_secret_access_key=SECRET_KEY,        aws_session_token=SESSION_TOKEN,    )    return "Hello from Lambda"Related informationUsing resource-based policies for AWS LambdaAWS Lambda permissionsSwitching to an IAM role (AWS API)Troubleshooting IAM rolesBuilding Lambda functions with PythonFollow"


Question: How do I resolve "The IAM role must delegate access to an Amazon Redshift account" error in Amazon Redshift when using AWS CloudFormation?

"I'm trying to create an Amazon Redshift cluster or scheduled action using AWS CloudFormation. However, I receive an AWS Identity and Access Management (IAM) role error. How do I resolve this error?"

Answer: "I'm trying to create an Amazon Redshift cluster or scheduled action using AWS CloudFormation. However, I receive an AWS Identity and Access Management (IAM) role error. How do I resolve this error?Short descriptionWith AWS CloudFormation, you can create a template in JSON or YAML that describes all the AWS resources that you need. AWS CloudFormation then provisions and configures the AWS resources for you. You can even use the AWS CloudFormation template to create an Amazon Redshift cluster or a scheduled action.However, you must correctly reference the IAM role that authorizes the Amazon Redshift cluster to access the other AWS services. Otherwise, you receive the following error:"The IAM role <role> is not valid. The IAM role must delegate access to an Amazon Redshift account."To resolve this issue, make sure to properly create and attach the AWS IAM role using CloudFormation. After your CloudFormation template file is created, your Amazon Redshift cluster and any specified AWS resources (such as a stack) are then also automatically created. No additional (manual) updates to your stacks are required.ResolutionUpdate your AWS CloudFormation template parameters (in YAML)To update the AWS CloudFormation template parameters in YAML, perform the following steps:1.    Define your parameters:AWSTemplateFormatVersion: 2010-09-09Description: Create Redshift Stack. Parameters:  Environment:    Description: Environment of the resources.    Type: String    Default: staging    AllowedValues:      - production      - staging      - testing  Name:    Description: Cluster name.    Type: String    Default: 'mycluster'  Service:    Description: Service name.    Type: String    Default: redshift    AllowedValues:      - redshift   DatabaseName:    Description:  Database name.    Type: String    Default: dev    AllowedPattern: "([a-z]|[0-9])+"  ClusterType:    Description: The type of cluster    Type: String    Default: multi-node    AllowedValues:    - single-node    - multi-node  NumberOfNodes:    Description: Compute nodes count. For multi-node clusters,      the NumberOfNodes parameter must be greater than 1    Type: Number    Default: '2'  NodeType:    Description: The type of node to be provisioned    Type: String    Default: dc2.large    AllowedValues:     - dc2.large    - dc2.8xlarge    - ra3.4xlarge    - ra3.16xlarge  MasterUsername:    Description: Master user name.    Type: String    Default: awsuser    AllowedPattern: "([a-z])([a-z]|[0-9])*"  MasterUserPassword:    Description: Master user password. Must have a length of 8-64 characters, contain one uppercase letter, one lowercase letter, and one number. Also only contain printable ASCII characters except for '/', '@', '"', ' ', '\' and '\'.    Type: String    NoEcho: 'true'  PortNumber:    Description: The port number on which the cluster accepts incoming connections.    Type: Number    Default: '5439'Conditions:  IsMultiNodeCluster:    Fn::Equals:    - Ref: ClusterType    - multi-nodeNote: Rather than embedding sensitive information in your AWS CloudFormation templates, it is best practice to use dynamic references in your stack template. For more information on best practices, see Security best practices for AWS CloudFormation.2.    Under Resources, create the IAM role that will be assumed by the RedShift service to access other AWS services: Resources: RedshiftRole: Type: AWS::IAM::Role Properties:Resources:  RedshiftRole:    Type: AWS::IAM::Role    Properties:      RoleName: !Sub ${Environment}-${Name}-${Service}      AssumeRolePolicyDocument:        Version: '2012-10-17'        Statement:        - Effect: Allow          Principal:            Service:              - redshift.amazonaws.com          Action:            - sts:AssumeRole          Condition:            StringEquals:              sts:ExternalId: !Sub 'arn:aws:redshift:${AWS::Region}:${AWS::AccountId}:dbuser:${Environment}-${Name}-${Service}/awsuser'      Path: "/"3.    Under Policies, specify the IAM policies to be attached to the IAM role:Policies:    - PolicyName: policy-s3    PolicyDocument:    Version: 2012-10-17    Statement:    - Effect: Allow    Action:      - 's3:AbortMultipartUpload'      - 's3:GetBucketLocation'      - 's3:ListBucket'      - 's3:ListBucketMultipartUploads'      - 's3:GetObject'      - 's3:PutObject'    Resource:      - !Sub "arn:aws:s3:::${Environment}-${Name}-tier1"      - !Sub "arn:aws:s3:::${Environment}-${Name}-tier1/log-internal-${Service}/*"      - !Sub "arn:aws:s3:::${Environment}-${Name}-tier2"      - !Sub "arn:aws:s3:::${Environment}-${Name}-tier2/*"    - Effect: Allow     Action:      - 's3:DeleteObject'    Resource:      - !Sub "arn:aws:s3:::${Environment}-${Name}-tier1/log-internal-${Service}/*"      - !Sub "arn:aws:s3:::${Environment}-${Name}-tier2/*"  - PolicyName: policy-cloudwatch    PolicyDocument:    Version: 2012-10-17    Statement:    - Effect: Allow    Action:      - 'logs:CreateLogGroup'      - 'logs:CreateLogStream'      - 'logs:PutLogEvents'    Resource: "*"4.    Use the Function Fn::GetAtt function to create the Amazon Redshift cluster, and then attach the IAM role (RedshiftRole):RedshiftCluster:  Type: AWS::Redshift::Cluster  Properties:     IamRoles:    - Fn::GetAtt: [ RedshiftRole, Arn ]    AllowVersionUpgrade: true    AutomatedSnapshotRetentionPeriod: 7     ClusterIdentifier: !Sub ${Environment}-${Name}-${Service}     ClusterVersion: 1.0    ClusterType:    Ref: ClusterType    NumberOfNodes:    Fn::If:    - IsMultiNodeCluster    - Ref: NumberOfNodes    - Ref: AWS::NoValue    NodeType:    Ref: NodeType    DBName:    Ref: DatabaseName    MasterUsername:    Ref: MasterUsername    MasterUserPassword:    Ref: MasterUserPassword    Port:    Ref: PortNumber    PreferredMaintenanceWindow: Sun:18:30-Sun:19:30    PubliclyAccessible: yes     AvailabilityZone: !Select [0, !GetAZs ""]The Fn::GetAtt function returns the value for a specified attribute.Note: If you use an incorrect form of reference to attach to your IAM role (such as the Ref function), you receive an IAM role error.Update your AWS CloudFormation template parameters (in JSON)To update the AWS CloudFormation template parameters in JSON, perform the following steps:1.    Define your parameters:{  "AWSTemplateFormatVersion": "2010-09-09",  "Description": "Create Redshift Stack.",  "Parameters": {    "Environment": {      "Description": "Environment of the resources.",      "Type": "String",      "Default": "staging",      "AllowedValues": [        "production",        "staging",        "testing"      ]    },    "Name": {      "Description": "Cluster name.",      "Type": "String",      "Default": "mycluster"    },    "Service": {      "Description": "Service name.",      "Type": "String",      "Default": "redshift",      "AllowedValues": [        "redshift"      ]    },    "DatabaseName": {      "Description": "Database name.",      "Type": "String",      "Default": "dev",      "AllowedPattern": "([a-z]|[0-9])+"    },    "ClusterType": {      "Description": "The type of cluster",      "Type": "String",      "Default": "multi-node",      "AllowedValues": [        "single-node",        "multi-node"      ]    },    "NumberOfNodes": {      "Description": "Compute nodes count. For multi-node clusters, the NumberOfNodes parameter must be greater than 1",      "Type": "Number",      "Default": "2"    },    "NodeType": {      "Description": "The type of node to be provisioned",      "Type": "String",      "Default": "dc2.large",      "AllowedValues": [         "dc2.large",        "dc2.8xlarge",        "ra3.4xlarge",        "ra3.16xlarge"      ]    },    "MasterUsername": {      "Description": "Master user name.",      "Type": "String",      "Default": "awsuser",      "AllowedPattern": "([a-z])([a-z]|[0-9])*"    },    "MasterUserPassword": {      "Description": "Master user password. Must have a length of 8-64 characters, contain one uppercase letter, one lowercase letter, and one number. Also only contain printable ASCII characters except for '/', '@', '\"', ' ', '\\' and '\\'.",      "Type": "String",      "NoEcho": "true"    },    "PortNumber": {      "Description": "The port number on which the cluster accepts incoming connections.",      "Type": "Number",      "Default": "5439"    }  },  "Conditions": {    "IsMultiNodeCluster": {      "Fn::Equals": [        {          "Ref": "ClusterType"        },        "multi-node"      ]    }  },Note: Rather than embedding sensitive information in your AWS CloudFormation templates, it is best practice to use dynamic references in your stack template. For more information on best practices, see Security best practices for AWS CloudFormation.2.    Under Resources, create the IAM role to be used to access your Amazon Redshift cluster:"Resources": {    "RedshiftRole": {      "Type": "AWS::IAM::Role",      "Properties": {        "RoleName": {          "Fn::Sub": "${Environment}-${Name}-${Service}"        },        "AssumeRolePolicyDocument": {          "Version": "2012-10-17",          "Statement": [            {              "Effect": "Allow",              "Principal": {                "Service": [                  "redshift.amazonaws.com"                ]              },              "Action": [                "sts:AssumeRole"              ],              "Condition": {                "StringEquals": {                  "sts:ExternalId": {                    "Fn::Sub": "arn:aws:redshift:${AWS::Region}:${AWS::AccountId}:dbuser:${Environment}-${Name}-${Service}/awsuser"                  }                }              }            }          ]        },        "Path": "/",3.    Under Policies, specify the IAM policies to be attached to the IAM role:"Policies": [      {      "PolicyName": "policy-s3",      "PolicyDocument": {        "Version": "2012-10-17",        "Statement": [        {          "Effect": "Allow",          "Action": [          "s3:AbortMultipartUpload",          "s3:GetBucketLocation",          "s3:ListBucket",          "s3:ListBucketMultipartUploads",          "s3:GetObject",          "s3:PutObject"          ],          "Resource": [          {            "Fn::Sub": "arn:aws:s3:::${Environment}-${Name}-tier1"          },          {            "Fn::Sub": "arn:aws:s3:::${Environment}-${Name}-tier1/log-internal-${Service}/*"          },          {            "Fn::Sub": "arn:aws:s3:::${Environment}-${Name}-tier2"          },          {            "Fn::Sub": "arn:aws:s3:::${Environment}-${Name}-tier2/*"          }          ]        },        {          "Effect": "Allow",          "Action": [          "s3:DeleteObject"          ],          "Resource": [          {            "Fn::Sub": "arn:aws:s3:::${Environment}-${Name}-tier1/log-internal-${Service}/*"          },          {            "Fn::Sub": "arn:aws:s3:::${Environment}-${Name}-tier2/*"          }          ]        }        ]      }      },      {      "PolicyName": "policy-cloudwatch",      "PolicyDocument": {        "Version": "2012-10-17",        "Statement": [        {          "Effect": "Allow",          "Action": [          "logs:CreateLogGroup",          "logs:CreateLogStream",          "logs:PutLogEvents"          ],          "Resource": "*"        }        ]      }      }    ]    }  },4.    Use the Function Fn::GetAtt function to create the Amazon Redshift cluster, and then attach the IAM role (RedshiftRole):"RedshiftCluster": {      "Type": "AWS::Redshift::Cluster",      "Properties": {        "IamRoles": [          {            "Fn::GetAtt": [              "RedshiftRole",              "Arn"            ]          }        ],        "AllowVersionUpgrade": true,        "AutomatedSnapshotRetentionPeriod": 7,        "ClusterIdentifier": {          "Fn::Sub": "${Environment}-${Name}-${Service}"        },        "ClusterVersion": 1,        "ClusterType": {          "Ref": "ClusterType"        },        "NumberOfNodes": {          "Fn::If": [            "IsMultiNodeCluster",            {              "Ref": "NumberOfNodes"            },            {              "Ref": "AWS::NoValue"            }          ]        },        "NodeType": {          "Ref": "NodeType"        },        "DBName": {          "Ref": "DatabaseName"        },        "MasterUsername": {          "Ref": "MasterUsername"        },        "MasterUserPassword": {          "Ref": "MasterUserPassword"        },        "Port": {          "Ref": "PortNumber"        },        "PreferredMaintenanceWindow": "Sun:18:30-Sun:19:30",        "PubliclyAccessible": "true",        "AvailabilityZone": {          "Fn::Select": [            0,            {              "Fn::GetAZs": ""            }          ]        }      }    }  }}The Fn::GetAtt function returns the value for a specified attribute.Note: If you use an incorrect form of reference to attach to your IAM role (such as the Ref function), you receive an IAM role error.Create a new stack in AWS CloudFormationTo create a stack, using your JSON or YAML template, perform the following steps:1.    Open the AWS CloudFormation console.2.    Choose Create stack to create a new stack.3.    Under Prerequisite - Prepare template in Step 1, choose Template is ready.4.    Under Specify template in Step 1, select your template source.5.    (Optional) If your template file must be uploaded, upload your template file.6.    Choose Next.7.    Under Specify stack details in Step 2, specify your Stack name and Parameters.8.    Choose Next.9.    Review all the stack details under Configure stack options in Step 3.10.    (Optional) To modify your stack options, choose Previous until you reach the page you want update. Update the stack options as needed, and then choose Next until you return to the Configure stack options page.11.    Choose Create stack.Related informationAmazon Redshift resource type referenceAmazon Redshift template snippetsAutomate Amazon Redshift cluster creation using AWS CloudFormationFollow"


Question: How can I determine which SageMaker notebook instance made a particular API call if all the instances use the same IAM role?

"I have multiple Amazon SageMaker notebook instances. They all use the same AWS Identity and Access Management (IAM) role. The AWS CloudTrail event for each API action shows the same principalId (session name), no matter which notebook instance performed the action. How can I tell which notebook instance performed which API actions?"

Answer: "I have multiple Amazon SageMaker notebook instances. They all use the same AWS Identity and Access Management (IAM) role. The AWS CloudTrail event for each API action shows the same principalId (session name), no matter which notebook instance performed the action. How can I tell which notebook instance performed which API actions?Short descriptionWhen you have multiple SageMaker instances with the same IAM role, you can't determine which notebook instance performed a particular API action with the CloudTrail event.Example:{    "eventVersion": "1.05",    "userIdentity": {        "type": "AssumedRole",        "principalId": "AAAAAAAAAAAAAAAAAA:SageMaker",           "arn": "arn:aws:sts::111122223333:assumed-role/AmazonSageMaker-ExecutionRole/SageMaker",Resolution1.    Create an IAM execution role for the SageMaker notebook instance. Or, use an existing execution role. In the following steps, the Amazon Resource Name (ARN) for the execution role is arn:aws:iam::111122223333:role/service-role/AmazonSageMaker-ExecutionRole.2.    Attach an IAM policy that includes sts:AssumeRole to the execution role. The sts:AssumeRole action allows the execution role to assume itself using a different session name.Example:{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": "sts:AssumeRole",            "Resource": "arn:aws:iam::111122223333:role/service-role/AmazonSageMaker-ExecutionRole"        }    ]}3.    Create a Start notebook lifecycle configuration script similar to the following. This example script retrieves the notebook instance name and then uses the name as the session name.#Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.#Permission is hereby granted, free of charge, to any person obtaining a copy of this#software and associated documentation files (the "Software"), to deal in the Software#without restriction, including without limitation the rights to use, copy, modify,#merge, publish, distribute, sublicense, and/or sell copies of the Software, and to#permit persons to whom the Software is furnished to do so.#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,#INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A#PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT#HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION#OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE#SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.#!/bin/bashset -ex# Obtain the name of the notebook instancenbname=$(jq -r '.ResourceName' /opt/ml/metadata/resource-metadata.json)echo "Notebook Name = $nbname"# Use the AWS Command Line Interface (AWS CLI) to obtain the Amazon Resource Name (ARN) of the IAM execution rolenbinfo=$(aws sagemaker describe-notebook-instance --notebook-instance-name $nbname)nbrole=$(jq -r '.RoleArn' <<< "$nbinfo")echo "Notebook Role = $nbrole"# Obtain the Region of the notebook instancenbregion=$(aws configure get region)echo "Notebook Region = $nbregion"# Write Assume Role Provider Settings to a new config fileecho "Writing new config file"cat > /home/ec2-user/.aws/config.new <<EOF1[default]region=$nbregionrole_arn = $nbrolecredential_source = Ec2InstanceMetadatarole_session_name = $nbnamests_regional_endpoints = regionalEOF1echo "Moving new config to config file"sudo mv /home/ec2-user/.aws/config.new /home/ec2-user/.aws/config# Secure the "config" file so that it can't be deleted/updated without root user permissionssudo chattr +i /home/ec2-user/.aws/config4.    Create a SageMaker notebook instance (for example, test-2), and then attach the lifecycle configuration script that you created in the previous step.5.    Create a SageMaker notebook instance with root access turned off. This restricts the user ec2-user from deleting or updating the config files.6.    To identify the notebook instance that performed an API action, check the CloudTrail event. Under the userIdentity object, the principalId and arn show the name of the notebook instance. For example, the following event detail shows that the SageMaker notebook instance test-2 made the API call.{    "eventVersion": "1.05",    "userIdentity": {        "type": "AssumedRole",        "principalId": "AAAAAAAAAAAAAAAAAAAA:test-2",        "arn": "arn:aws:sts::111122223333:assumed-role/AmazonSageMaker-ExecutionRole/test-2",        "accountId": "111122223333",        "accessKeyId": "AAAAAAAAAAAAAAAAAAAA",        "sessionContext": {            "sessionIssuer": {                "type": "Role",                "principalId": "AAAAAAAAAAAAAAAAAAAA",                "arn": "arn:aws:iam::111122223333:role/service-role/AmazonSageMaker-ExecutionRole",                "accountId": "111122223333",                "userName": "AmazonSageMaker-ExecutionRole"            },            "webIdFederationData": {},            "attributes": {                "mfaAuthenticated": "false",                "creationDate": "2020-09-12T00:45:04Z"            }        },        "invokedBy": "im.amazonaws.com"    },    "eventTime": "2020-09-12T00:49:04Z",    "eventSource": "sagemaker.amazonaws.com",    "eventName": "CreateEndpoint",    "awsRegion": "us-east-1",    "sourceIPAddress": "im.amazonaws.com",    "userAgent": "im.amazonaws.com",    "requestParameters": {        "endpointName": "sagemaker-mxnet-ep",        "endpointConfigName": "sagemaker-mxnet-epc",        "tags": []    },    "responseElements": {        "endpointArn": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/sagemaker-mxnet-ep"    },    "requestID": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",    "eventID": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",    "eventType": "AwsApiCall",    "recipientAccountId": "111122223333"}Related informationSageMaker rolesFollow"


Question: How can I resolve common issues when using read replicas in Amazon Aurora?

"I have an Amazon Aurora MySQL DB instance, and I am experiencing issues when working with read replicas. How can I troubleshoot common issues when using read replicas with Amazon Aurora?"

Answer: "I have an Amazon Aurora MySQL DB instance, and I am experiencing issues when working with read replicas. How can I troubleshoot common issues when using read replicas with Amazon Aurora?Short descriptionAmazon Aurora MySQL supports Read Replicas that share a common underlying volume with a writer DB instance in same AWS Region. If you change your writer DB instance, the updates are visible to replica instances in the DB cluster. You can also create cross-Region MySQL Read Replicas. These are implemented using the MySQL binlog-based replication engine.It's a best practice to use Aurora replicas when scaling read operations. You do this by reducing the read workload on the writer. Then, increase the availability to handle events that slow or block scaling.ResolutionHow do I promote an Aurora read replica?Manual failover - Perform a manual failover to promote another read replica instance as a writer instance by following these steps:Sign in to the Amazon Relational Database Service (Amazon RDS) console.From the navigation panel, choose Databases.Choose the writer instance for your Aurora DB cluster.Choose Actions, and then choose Failover.Automatic failover - Aurora automatically fails over to a read replica instance if the writer instance becomes unavailable. This can happen for a number of reasons, including resource contention and for maintenance activity. If you have multiple readers, you can give a promotion priority tier to each instance in your cluster. When the writer instance fails, Aurora promotes the replica with the highest priority as the new writer.You can also promote a cross-Region Aurora replica as a standalone DB cluster. The cross-Region replication stops after you initiate the promotion process. The newly promoted cluster functions as an independent DB cluster, and handles both read and write operations.How can I measure replication lag?Because all Aurora DB instances in a single DB cluster share a common data volume, there is minimal replication lag. Usually, lag times are in the 10s of milliseconds. However, you might observe slightly increased lag on the readers in a few specific circumstances.Note: Cross-Region replicas use logical replication, and are influenced by the change/apply rate and delays in network communication between the specific Regions selected. Cross-Region replicas using Aurora databases have a typical lag of under a second.You can measure replication lag using the following Amazon CloudWatch metrics:Use AuroraReplicaLag to measure replica lag between the writer and reader node in milliseconds (same Region).Use AuroraBinlogReplicaLag to measure replica lag between Aurora DB clusters using binary logs.How can I improve replication performance?Follow these recommendations to improve replica lag:If the reader instance is smaller than the writer instance, the volume of changes might be too much for the reader to catch up. It's a best practice that all instances in a cluster are the same size to avoid any workload overload on the reader instances.If there is heavy workload on the writer, you might notice temporary read replica lag. The lag reduces after the reader instance catches up with the writer instance.If there are any long-running transactions in progress, you might observe a replica lag on the readers. To avoid replica lag, run your transactions in smaller batches and run commits more frequently.For more information on troubleshooting high replica lag using native binlog-based MySQL replication, see Overview of backing up and restoring an Aurora DB cluster.Can I use a global transaction identifier (GTID)?A global transaction identifier is a unique string that is associated to a transaction on its Commit. A GTID is unique across all servers, and changes are applied on target, based on the GTID value. For more information, refer to the MySQL documentation for GTID concepts.Aurora doesn't use native binlog replication to replicate data to read replica instances. It's not possible to use GTID to replicate data between instances in the same cluster. However, you can set up GTID-based replication in certain scenarios. For more information on using GTID-based replication in Aurora MySQL, see the AWS Blog on GTID.Note: You can set up GTID-based replication between an Amazon RDS MySQL and an Aurora cluster, and between Aurora Clusters (by assuming the source is an external master). Make sure to enable binlog on the source before you start the replication process.Related informationReplication with Amazon AuroraWhat is Amazon Aurora?Follow"


Question: How can I troubleshoot Pending Verification domain name issues for VPC endpoint services?

I'm configuring a private DNS name for my Amazon Virtual Private Cloud (Amazon VPC) endpoint service (AWS PrivateLink) and setting up a DNS TXT record. The domain verification status is stuck in Pending Verification. How can I troubleshoot this?

Answer: "I'm configuring a private DNS name for my Amazon Virtual Private Cloud (Amazon VPC) endpoint service (AWS PrivateLink) and setting up a DNS TXT record. The domain verification status is stuck in Pending Verification. How can I troubleshoot this?Short descriptionThe following are common reasons for the domain verification status to be stuck in the Pending Verification status:You're using a domain name that isn't owned by you or your organization.You created a TXT record in private zone file instead of public zone file.The domain registrar has the wrong name servers for your domain.Note: For information on setting up a private DNS name for your VPC endpoint services, see How can I set up my VPC endpoint service to use a custom private DNS name?ResolutionYou're using a domain name that isn't owned by you or your organizationThe endpoint service provider can only use a domain name that's owned by you or your organization as a private DNS name. For example, you can't use “amazonaws.com“ as the private DNS name because that domain is owned by Amazon.You created a TXT record in a private zone instead of public zoneAWS verifies the ownership of the domain name provided under endpoint services. This is done by querying TXT records against the authorized public name server configured under the domain registrar. Verification fails if the user configured the TXT record on a private zone file because it can't be queried publicly.You can check whether the TXT record value is created in the publicly-verifiable domain or not by using the nslookup command. This command works in both Windows and Linux machines:nslookup -type=TXT _aksldja21i1.myexampleservice.comIf the TXT record was created in the publicly-verifiable domain, then you receive the domain verification value used while creating the TXT record as the output:_aksldja21i1.myexampleservice.com text = "vpce:asjdakjshd78126eu21”If the TXT record was created in the private hosted zone, then you receive output similar to the following:_aksldja21i1.myexampleservice.com = "v=spf1 -all”Note: You can also verify this information using any 3rd party websites, such as whatsmydns.The domain registrar has the wrong name servers for your domainFor private DNS verification, AWS queries the public name servers that are registered under the domain registrar. So, it's important to make sure that the name servers in your public zone file matches the name servers of your registered domain at the registrar.Note: If the verification status shows Pending Verification for a long time, then you can try manually starting the verification process. For more information, see Manage DNS names for VPC endpoint services.Related informationYou recently transferred DNS service to Amazon Route 53, but you didn't update the name servers with the domain registrarFollow"


Question: How do I avoid "ORA-01653: unable to extend table SYS.AUD$ or FGA_LOG$ in tablespace SYSTEM" errors when using Amazon RDS for Oracle?

"I enabled the Oracle audit feature on my Amazon Relational Database Service (Amazon RDS) database to log events on the database audit tables. But the SYSTEM tablespace is growing more than I expected, or I received the following error message:"ORA-01653: unable to extend table..." for SYS.AUD$ or FGA_LOG$ in the tablespace SYSTEM."How do I resolve this error?"

Answer: "I enabled the Oracle audit feature on my Amazon Relational Database Service (Amazon RDS) database to log events on the database audit tables. But the SYSTEM tablespace is growing more than I expected, or I received the following error message:"ORA-01653: unable to extend table..." for SYS.AUD$ or FGA_LOG$ in the tablespace SYSTEM."How do I resolve this error?Short descriptionThe ORA-01653 error is caused by a lack of free space on the SYS.AUD table in the SYSTEM tablespace. This is where the Oracle audit tables AUD$ and FGA_LOG$ are defined, by default. The error occurs when the table attempts to auto-extend to accommodate additional data, but the table is prevented from expanding. For more information about the auditing actions that can cause full tablespace, see the Oracle documentation for configuring and administering auditing.Note: AUDIT_SYS_OPERATIONS is enabled by default on Amazon RDS for Oracle.ResolutionTo resolve this issue, consider one or more of the following solutions:Migrate the AUDIT tables to a dedicated tablespaceAudit tables grow indefinitely until either a manual or automated purging routine using DELETE statements cleans up the records. Because free space is reclaimed only to the affected segment, purging tables in the SYSTEM tablespace results in no functional change to the size of the AUD$ and FGA_LOG$ tables. This can cause fragmentation issues.Migrating AUDIT tables from the SYSTEM tablespace to a different tablespace allows you more control over the size of your RDS DB instance's audit tables. You can migrate the tables yourself, or if you're running Oracle version 11g R2 or above on your RDS DB instance, you can use the DBMS_AUDIT_MGMT package.To migrate the table manually, follow these steps:1.    Create a new tablespace by running a command similar to the following:CREATE TABLESPACE <name>;Note: Creating a tablespace with AUTOEXTENSIBLE segment space management (enabled by default for new tablespaces) allows you to easily resize the table later.2.    Migrate your AUD$ table to the new tablespace by running commands similar to the following:BEGINSYS.DBMS_AUDIT_MGMT.SET_AUDIT_TRAIL_LOCATION(audit_trail_type => SYS.DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,audit_trail_location_value => '<name>');END;/3.    Migrate your FGA_LOG$ table to the new tablespace by running commands similar to the following:BEGINSYS.DBMS_AUDIT_MGMT.set_audit_trail_location(audit_trail_type => SYS.DBMS_AUDIT_MGMT.AUDIT_TRAIL_FGA_STD, audit_trail_location_value => '<name>');END;/Note: Depending on the size of your table, this operation can take time to complete. If you have large or frequently accessed tables, you might also encounter deadlock errors. To resolve deadlock errors, temporarily set DB_AUDIT_TRAIL to NONE, restart the database, and try the operations again. You can revert DB_AUDIT_TRAIL after the process completes.Follow best practices for auditing tablesTo maintain manageable audit tables, mitigate the space consumed by the audit tables. Also, minimize the occurrence of the ORA-01653 error by auditing the minimum set of pertinent actions. Finally, archive and purge audit trail records on a regular basis. For more information, see the guidelines for auditing typical database activity in the Oracle Database documentation.Turn AUTOEXTEND on for the SYSTEM tablespaceFor RDS DB instances that run Oracle, AUTOEXTEND for the SYSTEM tablespace is set to ON by default. If you turn AUTOEXTEND off manually, then your database's ability to provision additional space for the SYSTEM tablespace can be affected. You can enable this setting for the SYSTEM tablespace by running the ALTER TABLESPACE command:ALTER TABLESPACE SYSTEM AUTOEXTEND ON;For more information, see Creating and sizing tablespaces.Set MAXSIZE for the SYSTEM tablespace to a larger valueIf the SYSTEM tablespace settings for MAXBYTES or MAXBLOCKS are set too low, set MAXSIZE for the SYSTEM tablespace to a larger value.You can also remove limits on the maximum size of the SYSTEM tablespace by running a command similar to the following:ALTER TABLESPACE SYSTEM AUTOEXTEND ON MAXSIZE UNLIMITED;Related informationHow do I truncate the sys.aud$ table on my Amazon RDS instance that is running Oracle?Common DBA database tasks for Oracle DB instancesCommon DBA log tasks for Oracle DB instancesOracle database log filesFollow"


Question: Why are images not loading through CloudFront?

I'm using an Amazon CloudFront distribution to display images in a browser. Why aren't the images loading?

Answer: "I'm using an Amazon CloudFront distribution to display images in a browser. Why aren't the images loading?Short descriptionImages can be blocked because of permission or configuration issues, or because of a 403 or 404 error.To determine whether an error message is preventing the images from loading, open your browser's developer tools. On the Network tab, locate the network calls for the image request. In the response header for the request, check for a 404 or 403 error code.If the value of Server is set to either an Amazon Simple Storage Service (Amazon S3) or custom origin server name, then the origin is returning the error code.ResolutionCloudFront doesn't have permission to access the Amazon S3 originIf you have Origin Access Identity (OAI) configured in CloudFront to access images from an Amazon S3 Rest API endpoint, then confirm that the bucket has a bucket policy that grants OAI access to the content in the bucket. Without the policy, when a request is made from CloudFront, a 403: Access Denied error is returned and the image doesn't load.To resolve this error, check your S3 bucket policy permissions.The image doesn't exist in the S3 origin or custom originIf the requested image file doesn't exist in the bucket or is placed in a different folder, then a 403 or 404 error is returned. The error that is returned depends on your S3 bucket permissions:If you have the s3:ListBucket permission on the bucket, Amazon S3 returns a 404: No Such Key error.If you don't have the s3:ListBucket permission, Amazon S3 returns a 403: Access Denied error.Behavior path pattern directs the request to an incorrect origin or folderIf your distribution has multiple behaviors with different path patterns, then the image might be directed to an incorrect origin or a folder where the file doesn't exist. In this case, you see a 404 error.To resolve this error, review the behavior and path pattern for your distribution. Confirm that the request lands on the correct behavior matching the path pattern and is directed to the correct origin.For example, in the following configuration, Behavior1 has the path pattern "test" that routes the request to the S3 origin. Behavior 2 has the path pattern "Default(*)" which routes the traffic to the Elastic Load Balancing (ELB) origin.Behavior1: Path pattern = test ====> S3 origin Default Behavior: Path pattern = Default (*)=====> ELB originWhen a request for “https://example.com/test/myimage.png” is made, the distribution matches the path with Behavior2's path pattern Default (*). The request doesn't match the Behavior1 path pattern test because there is no wildcard at the end of test. Without a wildcard, this request matches the path pattern for the default behavior and is directed to the ELB origin. Because the object doesn't exist in the ELB origin, the request will return a 404 error and the image won't load.Behavior cache settings aren't configured for header, cookies, or query stringsThe origin can require a certain header, cookies, or query strings to be passed in the request to serve the requested images. In this case, when a request is made using CloudFront, the images don't load. However, if the same request is made directly to the origin, the request is successful and the origin returns the requested resources.To troubleshoot, determine if the origin requires certain headers, cookies, or query strings in order to process the request. Then, make sure the requirements are allowed in the Behavior cache settings.Use an origin request policy to allow the values (headers, cookies, and query strings) that are included in requests that CloudFront sends to your origin. You can also use Legacy cache settings.Metadata for image is not correctly set on the originIf metadata for Content-Type in the origin isn't set to image/png, then the image will be downloaded instead of displaying in your browser.To resolve this issue, check the Content-Type metadata. If needed, update the value to image/png for the images in an S3 or custom origin.CORS error causes the images to not loadIf the CORS configuration in CloudFront or the origin isn't set correctly, you receive the error: "No 'Access-Control-Allow-Origin' header is present on the requested resource."To resolve this issue, confirm the CORS configuration is set up for both CloudFront and the origin. For more information, see How do I resolve the "No'Access-Control-Allow-Origin' header is present on the requested resource" error from CloudFront?Follow"


Question: How can I set up a cross-account AWS Glue ETL job for tables using catalog resource policies?

"I want to access tables in account A from account B using ETL jobs in the AWS Glue Data Catalog, without using AWS Lake Formation."

Answer: "I want to access tables in account A from account B using ETL jobs in the AWS Glue Data Catalog, without using AWS Lake Formation.Short descriptionYou can access cross-accounts in the AWS Glue Data Catalog with or without using AWS Lake Formation. The following sections outline the setup to access a cross-accounts catalog using only AWS Glue.If you're using Lake Formation, see Cross-account data sharing in Lake Formation for more information on setting up a cross-account resource share.Note: The steps describe how you can access cross-accounts within a single AWS Region. They don’t address access to resources located in a different AWS Region.ResolutionSet up access policies in source and target accountsUse the following steps to grant resource-level permissions to account B from account A's AWS Glue Data Catalog.Note: Account A has the AWS Glue Data Catalog resources and account B is the extract, transform, and load (ETL) account. In addition, account A has the resource-based policy modifications while account B holds some of the AWS Identity and Access Management (IAM) policy modifications.Attach a catalog-resource policy in account A1.    Log in to the AWS Management console.2.    In the search bar, search for AWS Glue. Choose Get Started with AWS Glue.3.    From the left panel, choose Catalog settings.4.    Under Permissions, enter the following resource policy. This resource policy lets account B access the databases and tables in account A.{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "AWS": "arn:aws:iam::1111222233334444:root"      },      "Action": "glue:*",      "Resource": [        "arn:aws:glue:us-east-1:5555666677778888:catalog",        "arn:aws:glue:us-east-1:5555666677778888:database/doc_example_DB",        "arn:aws:glue:us-east-1:5555666677778888:table/doc_example_DB/*"      ]    }  ]}Note: Replace the following values in the policy:1111222233334444 with the account ID for account B5555666677778888 with the account ID for account Aus-east-1 with the Region of your choicedoc_example_DB with the name of your database5.    (OPTIONAL) You can limit access to a specific role in account A by including the Amazon Resource Name (ARN) of the role in the policy. For example:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "AWS": "arn:aws:iam::1111222233334444:role/service-role/AWSGlueServiceRole_Glue_Test"      },      "Action": "glue:*",      "Resource": [        "arn:aws:glue:us-east-1:5555666677778888:catalog",        "arn:aws:glue:us-east-1:5555666677778888:database/doc_example_DB",        "arn:aws:glue:us-east-1:5555666677778888:table/doc_example_DB/*"      ]    }  ]}Note: Replace the following values in the policy:1111222233334444 with the account ID for account B5555666677778888 with the account ID for account Aus-east-1 with the Region of your choicedoc_example_DB with the name of your databaseAWSGlueServiceRole_Glue_Test  with the ARN of the role that's used to run the ETL jobAttach an IAM policy in account BThe IAM user in account B that runs the ETL job needs access to the databases and tables in account A.Note: If you're using Amazon Athena with the Data Catalog, then include the default database in the policy. This inclusion makes sure that the GetDatabase and CreateDatabase actions succeed. For more information, see Default database and catalog per AWS Region.1.    Log in to the IAM console with your AWS login credentials.2.    From the left panel, choose Roles.3.    Choose the role name that you're using inside the ETL script.4.    Attach an IAM policy to the AWS Glue ETL job's IAM role in account B. This gives you access to the database and tables in account A:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "glue:GetDatabase",        "glue:GetConnection",        "glue:GetTable",        "glue:GetPartition"      ],      "Resource": [        "arn:aws:glue:us-east-1:5555666677778888:catalog",        "arn:aws:glue:us-east-1:5555666677778888:database/default",        "arn:aws:glue:us-east-1:5555666677778888:database/doc_example_DB",        "arn:aws:glue:us-east-1:5555666677778888:table/doc_example_DB/*"      ]    }  ]}Note: Replace the following values in the policy:5555666677778888 with the account ID for account Aus-east-1 with the Region of your choicedoc_example_DB with the name of your database5.    Verify that the policy you created is attached to the IAM role in account B.6.    Test if account B has access to the Data Catalog in account A. Create an ETL job with the following scripts:Dynamic frame script:df = glueContext.create_dynamic_frame.from_catalog(database="doc_example_DB", table_name="doc_example_table", catalog_id="5555666677778888", region="us-east-1")Data frame script:"""Create Spark Session with cross-account AWS Glue Data Catalog"""from pyspark.sql import SparkSessionspark_session = SparkSession.builder.appName("Spark Glue Example") \.config("hive.metastore.client.factory.class", \"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory") \.config("hive.metastore.glue.catalogid", "5555666677778888") \.enableHiveSupport() \.getOrCreate()table_df = spark_session.sql("SELECT * FROM doc_example_DB.doc_example_table limit 10")table_df.show()Note: Replace the following values in the policy:5555666677778888 with the account ID for account Adoc_example_DB with the name of your databasedoc_example_table with the name of your tableus-east-1 with the Region of your choiceRelated informationGranting cross-account accessSpecifying AWS Glue resource ARNsAbout upgrading to the Lake Formation permissions modelMigration between the Hive metastore and the AWS Glue Data CatalogAWS Glue resource policies for access controlFollow"