Question: Why did my AWS DMS CDC task using Oracle as a source fail with a "Sequence doesn't exist" message?

"I want to migrate data from my on-premise or Amazon Relational Database Service (Amazon RDS) for Oracle database using AWS Database Migration Service (AWS DMS). The AWS DMS change data capture (CDC) task runs as expected, but then fails with an error similar to this:"Oracle CDC maximum retry counter exceeded" (archivelog sequence does not exist)"How can I troubleshoot and resolve this error?"

Answer: "I want to migrate data from my on-premise or Amazon Relational Database Service (Amazon RDS) for Oracle database using AWS Database Migration Service (AWS DMS). The AWS DMS change data capture (CDC) task runs as expected, but then fails with an error similar to this:"Oracle CDC maximum retry counter exceeded" (archivelog sequence does not exist)"How can I troubleshoot and resolve this error?Short descriptionWhen you use an Oracle database as a source for your migration task, AWS DMS gets the data from the table during the full-load phase. During the CDC phase, AWS DMS reads from the archived redo logs. Then, AWS DMS captures the redo logs from the source Oracle database, and applies only the committed changes to the target database.You might see a sequence log error that looks like this:"03980512: 2022-05-23T12:33:11 [SOURCE_CAPTURE ]E: Archived Redo log with the sequence 232488 does not exist, thread 1 [1022318] (oradcdc_thread.c:624"To troubleshoot this error, follow these steps:1.    Run this query on the source Oracle database to check if the archive log sequence is present on the source. For example, this query checks for redo log sequence 232488:select name, dest_id, thread#,sequence#, archived, applied, deleted, status, first_time, next_time, completion_time  from v$archived_log where sequence# =232488;2.    Run this command on the location of the archive log files on the source server. This command checks if the archive log is physically present:ls -l * 232488*3.    Check the archivelog destination (log_archive_dest) on the source Oracle database.The examples in this article investigate this error with two different root causes:AWS DMS is looking for the redo log in the wrong DEST_IDDEL is YES and the archivelog sequence doesn't exist on source OracleResolutionExample 1 - AWS DMS is looking for the redo log in the wrong DEST_IDThe error shows that the archived redo log sequence 232488 is missing on the source Oracle DB. This might happen if the log is deleted from the source Oracle database. This causes the task to fail.01788702: 2022-06-07T17:10:31:206453 [SOURCE_CAPTURE  ]D:  Going to prepare the statement 'select supplemental_log_data_min, DATABASE_ROLE, SUPPLEMENTAL_LOG_DATA_PK, SUPPLEMENTAL_LOG_DATA_ALL from v$database'  (oracle_endpoint_conn.c:114)01788702: 2022-06-07T17:10:31:209648 [SOURCE_CAPTURE  ]I:  Database role is 'PHYSICAL STANDBY'  (oracle_endpoint_conn.c:139)1.    Run this query on the source database to see if archive log sequence 232488 exists in the source:select name, dest_id, thread#, sequence#, archived, applied, deleted, status, first_time, next_time, completion_time  from v$archived_log where sequence# in (232488);Output:NAME                                             DEST_ID    THREAD#  SEQUENCE# ARC APPLIED   DEL S FIRST_TIM NEXT_TIME COMPLETIO--------------------------------------------- ---------- ---------- ---------- --- --------- --- - --------- --------- ---------/orafra/prdsvbo/arc/1_232488_950180179.arc             2          1     232488 YES YES       NO  A 07-JUN-22 07-JUN-22 07-JUN-222.    Check the DEL column. If this column lists NO, then the archivedlog sequence exists in the source database. If it lists YES, then the archiveredlog doesn't exist, and might have been purged from the source because of the retention period.In this example, the output indicates that archivelog sequence exists in the source. But, the AWS DMS task still fails with an error that says the sequence doesn't exist. For troubleshooting when DEL is YES, see Example 2.3.    Next, check the DEST_ID column. By default, AWS DMS captures the redo logs on DEST_ID 1. In this example, you see this snippet in the logs:01788702: 2022-06-07T17:10:31:658376 [SOURCE_CAPTURE  ]I:  Used Oracle archived Redo log destination id is '1'  (oracdc_merger.c:639)01788702: 2022-06-07T17:10:31:658420 [SOURCE_CAPTURE  ]I:  Oracle instance uses more than one archived Redo log destination id. Please configure the correct destination id, if Redo logs of '1' destination cannot be accessed  (oracdc_merger.c:642)So, the AWS DMS task fails because it is looking for the redo log in DEST_ID 1, but the redo log file is present in DEST_ID 2.4.    To mitigate this error, use this extra connection attribute (ECA) in the source endpoint:additionalArchivedLogDestId=2For more information about additionalArchivedLogDestID, see Source data types for Oracle.5.    After you configure the ECA, resume the AWS DMS task. In this example, the logs show that AWS DMS can now capture the available redo log sequence 232488 from DEST_ID 2.01898667: 2022-06-08T05:45:08:535588 [SOURCE_CAPTURE  ]D:  Going to retrieve archived REDO log with sequence 232488, thread 1  (oradcdc_thread.c:510)01898667: 2022-06-08T05:45:08:535607 [SOURCE_CAPTURE  ]T:  Use a prepared statement to access v$archived_log, thread 1  (oradcdc_thread.c:587)01898667: 2022-06-08T05:45:08:598396 [SOURCE_CAPTURE  ]D:  Going to open Redo Log with original name '/orafra/prdsvbo/arc/1_232488_950180179.arc', thread id '1'  (oradcdc_redo.c:492)01898667: 2022-06-08T05:45:08:599614 [SOURCE_CAPTURE  ]D:  archived Redo log '/orafra/prdsvbo/arc/1_232488_950180179.arc' with sequence 232488 is opened, thread id '1'  (oradcdc_redo.c:747)Example 2 - DEL is YES and the archivelog sequence doesn't exist on source OracleAs previously detailed in step 2, if DEL is YES, then it's possible that the archivelog sequence was purged from the source Oracle database.Note: Oracle instances delete the archive log files to minimize the space that the archive logs take up.Steps for On-premises or Amazon Elastic Compute Cloud (Amazon EC2)If you scheduled the Recovery Manager (RMAN) to delete noprompt archivelog until time SYSDATE-1, then this schedule deletes all archive log files older than one day. To increase this, modify this command to SYSDATE-2, or turn off the schedule.Follow these steps to find the current retention period, and then increase it.1.    Connect to RMAN.2.    Show all:CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # defaultNote: The default Archivelog deletion policy value is NONE.3.    Change the deletion policy value to a value that's sufficient to retain the archivelogs on the source on-premises database:CONFIGURE ARCHIVELOG DELETION POLICY BACKED UP integer TIMES TO DEVICE TYPE  CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 2 TIMES TO SBT;Steps for Amazon RDSAmazon RDS deletes the archive log files every five minutes, and keeps a copy in an Amazon Simple Storage Solution (Amazon S3) bucket. You can use this copy to do a point-in-time restore.    1.    Increase the retention of the archive log files using the steps in Performing common log-related tasks for Oracle DB instances.    2.    Verify the retention period that's defined for the archive log files:exec RDSADMIN.RDSADMIN_UTIL.SHOW_CONFIGURATION;3.    Check if the archive log file that you fetched in the previous query is physically available on the source database server.SQL>select name, archived, deleted, status, sequence# from v$archived_log where sequence# = 232488;Note: In Amazon RDS, the deleted column shows NO, even though it was purged. This information comes from the control file.4.    To check if the redo log exists in Amazon RDS, run this query:select * from table(RDSADMIN.RDS_FILE_UTIL.LISTDIR('ARCHIVELOG_DIR')) where filename='redolog-5924417-1-1013939085.arc' order by mtime desc;Note: AWS DMS doesn't control the purging of the archive redo logs. Archive redo logs are purged by the source on-premises or RDS for Oracle database. Instead, AWS DMS confirms that it can't find the log when it tries to process the next LSN.5.    If its available, restore the missing archive log of sequence 232488 in the source destination, and then resume the task. All the redo logs following 232488 must be present on the source before you can successfully resume the AWS DMS task.If you can't restore the missing archive log sequence, try restarting the task from the full-load phase, and then start the migration over.Related informationExtra connection attributes when using Oracle as a source for AWS DMSUsing an Oracle database as a source for AWS DMSPerforming common log-related tasks for Oracle DB instancesFollow"


Question: Why is my TransactWriteItems API call failing in Amazon DynamoDB?

My TransactWriteItems API call is failing and I want to know the root cause of the issue.

Answer: "My TransactWriteItems API call is failing and I want to know the root cause of the issue.ResolutionThere are a number of reasons that a TransactWriteItems request might fail or be rejected by Amazon DynamoDB.A condition in one of the condition expressions isn't metIf a condition that you set for one of the operations in your TransactWriteItems request isn't met, then all operations fail. TransactWriteItems is a synchronous write operation that groups up to 25 action requests. So, either all of the action requests succeed, or all of the action requests fail.If you are experiencing this issue, then you receive an error message similar to this:"message": "Transaction cancelled, please refer cancellation reasons for specific reasons [ConditionalCheckFailed, None, None, None] (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: TransactionCanceledException;..."To resolve this issue, make sure that all the conditions that you set for your request are satisfied.The table in the TransactWriteItems request is in a different Region or accountTransactWriteItems requests allow you to work with items across different tables. But, you can't work with tables that are in different Regions or accounts. Make sure that you are working with tables that are in the same Region as the account where you are making the TransactWriteItems request from.More than one action in the TransactWriteItems operation handles the same itemWhen more than one action in the TransactWriteItems operation handles the same item, then the request fails with a message similar to this:"Transaction request cannot include multiple operations on one item"So, for example if TransactWriteItems tries to run a ConditionCheck and Put operation on the same item, then the request fails. You can monitor this TransactionConflict exception by using the TransactionConflict metric for your DynamoDB table in Amazon CloudWatch.There isn't enough provisioned capacity to complete the transactionLike any other operation, a DynamoDB table might be throttled when you invoke the TransactWriteItems operation. For more information on resolving this issue, see Why is my on-demand Amazon DynamoDB table being throttled? and Why is my Amazon DynamoDB table being throttled?A ValidationError occurredIf an item size becomes larger than 400 KB or a local secondary index (LSI) becomes too large, then a ValidationErrror occurs. Similarly, validation errors can occur because of changes made by the transaction. This means that the input fails to satisfy one or more constraints that are specified by the DynamoDB service. The validation error is accompanied by one of these messages, depending on which constraint isn't satisfied:One or more parameter values are invalid.The update expression attempts to update the secondary index key beyond allowed size limits.The update expression attempts to update the secondary index key to an unsupported type.An operand in the update expression has an incorrect data type.The item to update has exceeded the maximum allowed size.You are attempting to store a number with magnitude larger than the supported range, or number overflow.There is a type mismatch for attribute to update.Nesting levels have exceeded supported limits.The document path provided in the update expression is invalid for update.The provided expression refers to an attribute that doesn't exist in the item.DynamoDB limits the size of each item that you can store in a table. If your application needs to store more data in an item than the size limit allows, compress one or more of the larger attributes. Or, break down the item into multiple items, indexed by sort keys. You can also store the item as an object in Amazon Simple Storage Service (Amazon S3). Then, store the Amazon S3 object identifier in your DynamoDB item. For more information, see Best practices for storing large items and attributes.The aggregate size of the items in a transaction exceeds 4 MBTransactWriteItems operations can't exceed 4 MB. This is a hard limit. For more information, see DynamoDB transactions.A user error occurredA user error, like an invalid data format, can occur for a number of reasons. For example, you might receive a 4xx error for ResourceNotFoundException because the TransactWriteItems operation can't find the underlying DynamoDB table or index. Or, the DynamoDB table or index might not be ACTIVE.Note: When you use the AWS SDK for Java, DynamoDB lists cancellation reasons on the CancellationReasons property. This property isn't set for other languages. Transaction cancellation reasons are listed in the order of requested items. If an item has no error, it lists a NONE code and null message. For more information, see TransactWriteItems.Related informationCommon errorsFollow"


Question: How do I troubleshoot high CPU usage in Amazon Redshift?

"I'm seeing high CPU utilization on my Amazon Redshift cluster. Why is this happening, and what are some best practices to lower my CPU utilization?"

Answer: "I'm seeing high CPU utilization on my Amazon Redshift cluster. Why is this happening, and what are some best practices to lower my CPU utilization?Short descriptionAmazon Redshift is designed to utilize all available resources while running queries. That means that you can expect to see spikes in CPU usage in your Redshift cluster. An increase in CPU utilization can depend on factors such as cluster workload, skewed and unsorted data, or leader node tasks.However, if your CPU usage impacts your query time, then consider the following approaches:Review your Redshift cluster workload.Maintain your data hygiene.Update your table design.Check for maintenance updates.Check for spikes in your leader node CPU usage.Use Amazon CloudWatch to monitor spikes in CPU utilization.ResolutionReview your Redshift cluster workloadThe following factors can impact the CPU utilization on your Redshift cluster:An increased workload (due to more queries running). The increase in workload increases the number of database connections, causing higher query concurrency.The higher number of concurrent queries impacts resource contention, lock wait time, and workload management (WLM) queue wait time.More database connections. This can be a result of idle sessions present in the cluster. Idle sessions can cause additional lock contention issues.While the queries are running, retrieve locking information. To identify long-running sessions, use the following SQL query:select *,datediff(s,txn_start,getdate())/86400||' days '||datediff(s,txn_start,getdate())%86400/3600||' hrs '||datediff(s,txn_start,getdate())%3600/60||' mins '||datediff(s,txn_start,getdate())%60||' secs' as "duration"from svv_transactions where lockable_object_type='transactionid' and pid<>pg_backend_pid() order by 3;<br>Then, run PG_TERMINATE_BACKEND to stop any long-running transactions. To prevent these sessions from remaining open, be sure that all transactions are closed. For example, make sure that all transactions starting with a BEGIN statement are accompanied by an END or COMMIT statement.Then, run the following SQL query to identify queries consuming high CPU:select stq.userid, stq.query, trim(stq.label) as label, stq.xid, stq.pid, svq.service_class,query_cpu_usage_percent as "cpu_%",starttime, endtime, datediff(s,starttime, endtime) as duration_s,substring(stq.querytxt,1,100) as querytext from stl_query stqjoin svl_query_metrics svq on stq.query=svq.query where query_cpu_usage_percent is not null and starttime > sysdate - 1order by query_cpu_usage_percent desc;To analyze segment and slice-level execution steps for each query, run the following query:select query, segment, step, label ,is_rrscan as rrS, is_diskbased as disk, is_delayed_scan as DelayS, min(start_time) as starttime, max(end_time) as endtime, datediff(ms, min(start_time), max(end_time)) as "elapsed_msecs", sum(rows) as row_s , sum(rows_pre_filter) as rows_pf, CASE WHEN sum(rows_pre_filter) = 0 THEN 100 ELSE sum(rows)::float/sum(rows_pre_filter)::float*100 END as pct_filter, SUM(workmem)/1024/1024 as "Memory(MB)", SUM(bytes)/1024/1024 as "MB_produced" from svl_query_report where query in (<query_ids>) group by query, segment, step, label , is_rrscan, is_diskbased , is_delayed_scan order by query, segment, step, label;For more information about tuning these queries, see Top 10 performance tuning techniques for Amazon Redshift.You can also use the wlm_query_trend_hourly view to review the Redshift cluster workload pattern. Then, determine which of the following approaches can help you reduce queue wait time:Reduce query concurrency per queue to provide more memory to each query slot. This reduction helps queries that require more memory to run more efficiently.Enable short query acceleration (SQA) to prioritize short-running queries over long-running queries.Scale the Redshift cluster to accommodate the increased workload. Scaling a cluster provides more memory and computing power, which can help queries to run more quickly. For more information, see How do I resize a Redshift cluster?Maintain your data hygieneData hygiene is gauged by the percentage of stale statistics and unsorted rows present in a table. A high percentage of both can cause the query optimizer to generate an execution plan where queries run inefficiently when referencing tables. Unsorted data can cause queries to scan unnecessary data blocks, which require additional I/O operations. A poorly performing query negatively affects your Redshift cluster's CPU usage.Use the SVV_TABLE_INFO system view to retrieve stats_off and unsorted percentage data for a table. These percentages should remain close to 0. If the percentages are high, run the Analyze & Vacuum schema utility from the AWS Labs GitHub repository to update your tables.Update your table designTable design is governed by the designated sort keys, distribution style, and distribution key. The distribution key and distribution style determine how data is distributed across the nodes.An inappropriate distribution key or distribution style can induce distribution skew across the nodes. To reduce data distribution skew, choose the distribution style and sort key based on query patterns and predicates. The distribution key should support the join conditions in your queries and columns with high cardinality. A proper distribution key selection can help queries perform merge joins instead of hash or nested loop joins, which ultimately affects the amount of time that queries run.To identify tables with skewed distribution, use the table_inspector.sql script. Then, use the Amazon Redshift table design playbook to choose the most appropriate sort keys, distributions keys, and distribution styles for your table.Check for maintenance updatesAmazon Redshift caches compiled code, allowing queries to reuse the code for previously run segments. The cache then is erased during any maintenance updates. As a result, queries that are run for the first time after a patch update will spend some time in compilation. This compilation overhead can increase a Redshift cluster's CPU usage.Use the following SQL query to check how many segments are being compiled each hour:select "hour", count(query) total_queries, count(case when is_compiled='Y' then 1 else null end ) as queries_compiled_count, sum(segements_count) total_segments_count, sum(segments_compiled_count) total_segments_compiled_count from(  select q.query, date_trunc('h',q.starttime) as "hour", q.starttime, q.endtime, q.aborted, (case when compiled_queries.segments_compiled_count = 0 then 'N' ELSE 'Y' end) is_compiled, compiled_queries.segements_count, compiled_queries.segments_compiled_count  from stl_query q  left join (select query, count(segment) segements_count, sum(compile) segments_compiled_count from svl_compile group by query) compiled_queries on q.query = compiled_queries.query  where q.userid > 1  and q.starttime > trunc(sysdate) -7)group by "hour"order by "hour";Check for spikes in your leader node CPU usageLeader node tasks such as parsing and optimizing queries, generating compiled code, and aggregating results from compute nodes consume CPU resources. This consumption leads to increased leader node CPU usage. Leader node CPU usage can also rise if queries are heavily referencing system catalog tables or performing leader node-only functions.If the spike in CPU usage is caused by a leader node, check under Events in the Amazon Redshift console. Verify whether any maintenance has occurred on your Redshift cluster. Use the SQL query provided in Check for maintenance updates to verify whether more segments are being compiled than usual.Use CloudWatch to monitor spikes in CPU utilizationUse CloudWatch metrics to compare the spikes between CPUutilization and Database Connections. Analyze the workload performance by checking the Workload Execution Breakdown chart. The Workload Execution Breakdown chart shows you at which stages the queries are spending the most time.To identify the top 100 queries that consume the most CPU during a specified time, use the following query:select qms.*, substring(q.querytxt,1,100) qtxtfrom svl_query_metrics_summary qmsjoin stl_query q on q.query=qms.querywhere q.starttime > sysdate - 1 and q.userid>1order by qms.query_cpu_time desc nulls last limit 100;To retrieve a list of queries that consume the most resources when CPU reaches 100%, use the following query:select a.userid, service_class, a.query, b.pid, b.xid, a.starttime, slices, max_rows, max_blocks_read, max_blocks_to_disk, max_query_scan_size, segment, round(max_cpu_time/(max_run_time*1.0)*100,2) as max_cpu_pcnt, round(cpu_time/(run_time*1.0)*100,2) as cpu_pcnt, max_cpu_time, max_run_time, case when segment > -1 then 'segment' else 'query' end as metric_lvl, text from pg_catalog.stv_query_metrics a left join stv_inflight b using (query) where step_type=-1 order by query, segment;To check the amount of data that are processed by each node, run the following query:select iq.day_d, sl.node, sum(iq.elapsed_ms) as elapsed, sum(iq.bytes) as bytes from (select start_time::date as day_d, slice,query,segment,datediff('ms',min(start_time),max(end_time)) as elapsed_ms, sum(bytes) as bytes from svl_query_report where end_time > start_time group by 1,2,3,4) iq join stv_slices as sl on (sl.slice = iq.slice) group by 1,2 order by 1 desc, 3 desc;You can use query monitoring rules (QMR) to identify and log any poorly designed queries. For example, QMR rules can be defined to log queries that consume high CPU usage or an extended runtime.Follow"


Question: How do I configure the rds-support-tools GitHub repository for Amazon RDS on my Amazon EC2 instance that is running Linux?

I want to use one of the AWS Support team’s diagnostic scripts with my Amazon Relational Database Service (Amazon RDS) DB instance. How do I clone the rds-support-tools GitHub repository to my Amazon Elastic Compute Cloud (Amazon EC2) instance that is running Linux?

Answer: "I want to use one of the AWS Support team’s diagnostic scripts with my Amazon Relational Database Service (Amazon RDS) DB instance. How do I clone the rds-support-tools GitHub repository to my Amazon Elastic Compute Cloud (Amazon EC2) instance that is running Linux?ResolutionNote: You do not need a GitHub account to clone the rds-support-tools GitHub repository.Before you begin, be sure that your EC2 instance has access to your RDS DB instances. Also, confirm that your EC2 instance can connect to the internet. When you’re ready, clone the rds-support-tools GitHub repository to your EC2 instance by following these steps:1.    Install Git on your Amazon Linux EC2 instance by connecting to the instance through SSH. Then, run the following command:cd $HOMEsudo yum install git-allNote: If you are not using Amazon Linux, use the appropriate command for your specific package manager.2.    Clone the rds-support-tools GitHub repository by using the following commands:% git clone https://github.com/awslabs/rds-support-tools.gitAfter you complete these steps, the default branch of the rds-support-tools repository is cloned to the current user's home directory.% cd $HOME/rds-support-tools% pwd/home/username/rds-support-toolsRelated informationHow can I connect to a private Amazon RDS DB instance from a local machine using an Amazon EC2 instance as a bastion host?Follow"


Question: Can I pay in advance for AWS services?

Can I pay for my AWS usage in a lump sum upfront?

Answer: "Can I pay for my AWS usage in a lump sum upfront?ResolutionAdvance Pay is available to Amazon Web Services, Inc. (AWSI) customers paying in US dollars for eligible AWS charges. For a full list of service restrictions for Advance Pay, see Advance Pay quotas and restrictions. For more information on registering and adding funds to Advanced Pay, see Managing your Advance Pay.Some individual AWS services offer payment options that are similar to prepayment. For example, Amazon Elastic Compute Cloud (Amazon EC2) offers Reserved Instances. These instances allow you to pay an upfront fee to reserve capacity for your account in one-year or three-year terms. For more information about the pricing models for various AWS services, choose that service's pricing page from the Cloud Services Pricing list.Related informationManaging your paymentsManaging your Advance PayAdvance Pay quotas and restrictionsFollow"


Question: What should I consider before purchasing a Savings Plan?

I plan to commit to a specific amount of compute power for a long period of time. What factors should I consider before purchasing a Savings Plan?

Answer: "I plan to commit to a specific amount of compute power for a long period of time. What factors should I consider before purchasing a Savings Plan?ResolutionConsider the following when deciding if Savings Plans are right for you.Long-term contractsSavings Plans have long contract terms (usually, one or three years). Therefore, it's a best practice to purchase Savings Plans when you plan to commit to a consistent amount of usage over the long term.Payment OptionsSavings Plans are available with three different payment options:No Upfront option doesn't require any upfront payment. Your commitment will be charged on a monthly basis.Partial Upfront option requires at least half of your payment upfront. The remaining is charged on a monthly basis.All Upfront option offers the lowest prices. Your entire commitment is charged in one payment.If your account is in an organization's consolidated billing family, then additional considerations might apply. For more information, see How is the pricing benefit of a Savings Plan applied across an organization's consolidated bill?Canceling and modifying a Savings PlanSavings Plans can't be modified or canceled. You can increase your hourly commitment by purchasing additional Savings Plans. You can queue your Savings Plan purchases for a specific date and time in the future. For more information, see Can I schedule my Savings Plan to renew automatically?Savings Plans, like Reserved Instances, are limited to the account that purchases them. They can't be transferred across accounts, but benefits that are purchased through AWS Organizations can be shared within that organization.Capacity reservations and short-term discountsSavings Plans don't provide a Capacity Reservation. If you're interested in the capacity reservation but not Savings Plans, then you might want to use On-Demand Capacity Reservations.For short-term discounts on Amazon Elastic Compute Cloud (Amazon EC2) instances, consider launching a Spot Instance instead.Pricing and purchasing a Savings PlanFor pricing information for Savings Plans, see Compute Savings Plans and Machine Learning Savings Plans.To complete a Savings Plan purchase, see Purchasing Savings Plans.To purchase Savings Plans using APIs, see Savings Plans API Reference.Related informationHow can I use Cost Explorer to analyze my spending and usage?Compute Savings Plans and Reserved InstancesUnderstanding your Savings Plans recommendationsFollow"


Question: How can I change the Session Manager shell to bash on EC2 Linux instances?

I want to change the Session Manager default shell on my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance. How can I do that?

Answer: "I want to change the Session Manager default shell on my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance. How can I do that?Short descriptionSession Manager sessions on Amazon EC2 Linux instances use the Bourne shell (sh) by default. However, bash is the default shell in major Linux distributions. If you’re more comfortable using bash than sh, you can change the default Session Manager shell to bash using configurable shell profiles. Shell profiles allow you to customize preferences within sessions when you use Session Manager, including:Shell preferencesEnvironment variablesWorking directoriesRunning multiple commands when a session startsYou can enable configurable shell profiles using the Systems Manager console or the AWS Command Line Interface (AWS CLI).ResolutionPrerequisitesTo enable shell profiles, you must use SSM Agent version 3.0.196.0 or later. Check the SSM Agent version number on your instance. If necessary, update SSM Agent. For more information, see Automating updates to SSM Agent.To see the available shells on the instance, run the following command:$ sudo cat /etc/shellsNote: You can use any of the shells returned using this command.Enable configurable shell profiles (console)To enable configurable shell profiles and change the shell to bash using the Systems Manager console, follow these steps:1.    Open the Systems manager console, and then choose Session Manager from the navigation pane.2.    Choose the Preferences tab, and then choose Edit.3.    For "linux" under "shellProfile", enter the following:/bin/bash4.    Choose Save.Enable configurable shell profiles (AWS CLI)Note: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.For instructions to enable configurable shell profiles and change the shell to bash using the AWS CLI, see Create Session Manager preferences (command line).Other shell profile configuration optionsBy default, Session Manager starts in the “/usr/bin” directory. You can add a command to your shell profile to change the /usr directory on Linux instances. For example:cd /home/ec2-userYou can also use the shell profile to load specific environment variables, or to run a script at the start of the session. For example, to add the PYTHONPATH environment variable, add the following command to your shell profile:export PYTHONPATH=/usr/lib/python2.7When you are logged in, you can run the following command to verify that the environment variable is correctly set:$ sudo envRelated informationEnable configurable shell profilesGrant or deny user permissions to update Session Manager preferencesSpecify an idle session timeout valueFollow"


Question: How can I troubleshoot Lake Formation permission errors in AWS Glue?

"My AWS Glue crawler or ETL job fails with an AWS Lake Formation related error. But, I configured the required AWS Identity and Access Management (IAM) permissions."

Answer: "My AWS Glue crawler or ETL job fails with an AWS Lake Formation related error. But, I configured the required AWS Identity and Access Management (IAM) permissions.Short descriptionTo access resources such as, the AWS Glue Data Catalog and Amazon Simple Storage Service (Amazon S3), you must have the correct IAM policies and Lake Formation permissions. When an IAM role that's associated with your Amazon S3 crawler or ETL job doesn't have sufficient Lake Formation permissions, you get an error. You must have the correct Lake Formation permissions to read and write to and from the following resources:Database and table in the Data CatalogUnderlying data in Amazon S3ResolutionAccess issues when creating Data Catalog databaseIf Data Catalog database creation issues are causing the error, then you get an error message similar to the following one:Insufficient Lake Formation permission(s): Required Create Database on CatalogTo resolve this error, use the data lake administrator role to access the Lake Formation console. Then, grant the Create database permission to the relevant IAM role.Open the AWS Lake Formation console.In the navigation pane, under Permissions, choose Administrative roles and tasks.Under Database creators, choose Grant.For IAM users and roles, from the dropdown list, choose the IAM role that you want to grand access to.Under Catalog permissions, choose Create database.If you want the IAM role to grant permissions to other roles in your account, then under Grantable permissions, choose Create database.Choose Grant.Access issues with Data Catalog databaseIf Data Catalog database issues are causing the error, then you get an error message similar to one of the following one:Insufficient Lake Formation permission(s) on example_database: Required Create TableInsufficient Lake Formation permission(s) on example_database: (Database name: example_database)Insufficient Lake Formation permission(s) on example_table (Database name: example_database, Table Name: example_table)To resolve the first and third errors, grant the Create table permission for example_database to the IAM role that's associated with the crawler or ETL job.To resolve the second error, grant the Describe permission for example_database to the IAM role that's associated with the crawler or ETL job.Open the Lake Formation console.In the navigation pane, under Permissions, choose Data lake permissions.Choose Grant.Under Principals, choose IAM users and roles.For IAM users and roles, choose the IAM role that's associated with the crawler.Under LF-Tags or catalog resources, choose Named data catalog resources.For Databases, choose the database that your crawler is writing to.Under Database permissions, choose Create table or Describe based on your use case. Note: The crawler role must have both Describe AND Create Table permissions on the database that it's writing to. For AWS Glue ETL, these permissions are also sufficient, unless you're running an UpdateDatabase or DeleteDatabase API call from the job itself. In these cases, grant Alter or Drop permissions.If you want the IAM role to grant permissions to other roles in your account, then set permissions under Grantable permissions.Choose Grant.Access issues with Data Catalog tableIf Data catalog table issues are causing the error is caused, then you get an error message similar to one of the following one:Insufficient Lake Formation permission(s) on example_table (Database name: example_database, Table Name: example_table)Insufficient Lake Formation permission(s): Required Alter on example_tableIf you receive this error, then the AWS Glue crawler or ETL job is accessing the table. For the crawler, grant Describe and Alter permissions for example_table to the IAM role that's associated with the crawler.For ETL jobs that read example_table, grant Describe and Select permissions for example_table to the IAM role that's associated with the job. If the ETL job is updating example_table, then grant Alter permissions for example_table to the IAM role that's associated with the job.Open the Lake Formation console.In the navigation pane, under Permissions, choose Data lake permissions.Choose Grant.Under Principals, choose IAM users and roles.For IAM users and roles, choose the IAM role.Under LF-Tags or catalog resources, choose Named data catalog resources.For Databases, choose the database that your crawler is writing to.For Tables-optional, choose the table that your crawler is accessing.Under Table permissions, choose Select, Describe, or Alter based on your use case.If you want the IAM role to grant permissions to other roles in your account, then set permissions under Grantable permissions.Choose Grant.Access issues with Lake Formation IAM permissionsIf IAM permissions issues for Lake Formation are causing the error, then you get an error message similar to one of the following one:com.amazonaws.services.lakeformation.model.AccessDeniedException: Service Principal: glue.amazonaws.com is not authorized to perform: lakeformation:GetDataAccesson resource: s3://sample-bucket/sample-prefix/ because no identity-based policy allows the lakeformation:GetDataAccess actionYou get this error when the AWS Glue job role or AWS Glue crawler role doesn't have sufficient IAM permissions. The job or role must have permission to check if Lake Formation can vend temporary credentials to the role and the Amazon S3 location.Open the IAM console.Create an IAM policy for your AWS Glue crawler or AWS Glue job role.Add the permission lakeformation:GetDataAccess as the action for the resource in the policy. Note: The API lakeformation:GetDataAccess must use the wildcard as its resource.Attach the policy to your AWS Glue crawler or AWS Glue job role.Access issues with Amazon S3 pathIf Amazon S3 path issues are causing the error is caused, then the error message looks similar to the following one. The error includes the Amazon S3 path:Insufficient Lake Formation permission(s) on s3://s3-example-bucket/example-prefix/ (Database name: example-database, Table Name: example-table)<br>Insufficient Lake Formation permission(s) on s3://s3-example-bucket/example-prefix/This error occurs when the IAM role that's associated with the crawler or ETL job doesn't have the required permission to access the Amazon S3 path.To resolve this error, complete the following steps:Open the Lake Formation console.In the navigation pane, under Register and ingest, choose Data lake locations.Verify that the Amazon S3 path or prefix of the path from the error message is a registered location in the Data lake locations list.If the Amazon S3 path or prefix in the error message isn't listed in the Data lake locations list, then choose Register location.For Amazon S3 path, choose Browse, and then choose the correct Amazon S3 path.For IAM role, keep the default selection of AWSServiceRoleForLakeFormationDataAccess. If you use a custom IAM role, then be sure that the relevant requirements are met. Important: When registering an Amazon S3 location, Lake Formation assumes the preceding IAM role. This grants temporary credentials to integrated AWS services that access data in that location. Verify that the IAM role that's associated with the registered Amazon S3 location has the required permissions to read and write to the S3 bucket. This prevents the AccessDeniederror.In the navigation pane, under Permissions, choose Data locations.Choose Grant.For Grant permissions, choose My account.Under IAM users and roles, choose the IAM role that you want to grant access for.Choose Grant.Related informationManaging Lake Formation permissionsRegistering an Amazon S3 locationFollow"


Question: How do I resolve the "Getting error Too Many Requests" error when I submit an AWS Batch job?

"My AWS Batch job fails, and I receive a "Getting error Too Many Requests" error."

Answer: "My AWS Batch job fails, and I receive a "Getting error Too Many Requests" error.Short descriptionAWS Batch throttles all API requests for AWS accounts based on an account's AWS Region. This makes sure that AWS Batch maintains consistent performance and fair usage for all customers. AWS Batch API quotas typically apply to every API, including the DescribeJobs and SubmitJob API calls.Note: The maximum number of transactions per second (TPS) for each account for SubmitJob operations is 50.When you reach your AWS Batch API quotas, AWS Batch returns the following error message:"Getting error Too Many Requests (Service: AWSBatch; Status Code: 429; Error Code: TooManyRequestsException)."There are two common reasons that Getting error Too Many Requests errors occur:Submitting multiple jobs at the same time programmatically, using the AWS Command Line Interface (CLI) or AWS SDKsUsing AWS Lambda to submit multiple jobsResolutionAs you troubleshoot the error, keep the following points in mind:API quotas apply only to AWS endpoints. These quotas don't apply to your account.AWS throttles Amazon API requests on a per-Region basis to prevent individual accounts from flooding the service endpoints with too many requests.AWS makes sure that all calls to all Amazon APIs don't exceed the maximum allowed API request rate. The maximum API request rate varies across Regions.Use retries and an exponential backoff algorithm with jitter, and then resubmit your jobFor more information, see Exponential backoff and jitter.Use AWS Batch array jobs to submit multiple jobs with a single SubmitJob API callFor more information, see Array jobs and SubmitJob.Note: AWS Batch array jobs allow you to submit up to 10,000 copies of a job.Adjusting API throttling quotasYou can request an increase for API throttling quotas for your AWS account. To request a quota adjustment, contact the AWS Support CenterFollow"


Question: What planning steps can I take when upgrading my Amazon EMR cluster?

I need to plan an Amazon EMR upgrade to keep pace with rapidly changing technology.

Answer: "I need to plan an Amazon EMR upgrade to keep pace with rapidly changing technology.Short descriptionTo keep up with the rapid changes in big data, you must upgrade your version of Amazon EMR. Migrating to a new version of Amazon EMR improves the operational excellence and efficacy of your workload. However, before you upgrade Amazon EMR, you must plan and prepare. There's information that you must review and procedures that you must follow.Benefits of Amazon EMR version upgradesBenefits of upgrading Amazon EMR include:Increased productivity and lowered costs by leveraging the newest features.Updated applications run faster.Up-to-date bug fixes provide a stable infrastructure.The latest security patches strengthen security.Up-to-date access to open-source software features.For example, with Amazon EMR version 6.6 and later, Log4j 1.x and Log4j 2.x are upgraded to Log4j 1.2.17 and Log4j 2.17.1 (or higher), respectively. In the higher versions, bootstrap actions aren't required to mitigate common vulnerabilities and exposures (CVEs).ResolutionAmazon EMR performance optimization features for open-source applicationsAmazon EMR offers performance optimization features for many open-source applications.Spark:Adaptive query executionDynamic partition pruningFlattening scalar subqueriesDISTINCT before INTERSECTBloom filter joinOptimized join reorderImproved Spark performance with Amazon Simple Storage Service (Amazon S3)Spark release history: Before deciding to upgrade Amazon EMR, check the version of Spark and its installed components in Amazon EMR releases.Delta Lake:Using a Delta Lake cluster with SparkUsing a Delta Lake cluster with TrinoDelta release history: Before deciding to upgrade Amazon EMR, check the version of Delta Lake and its installed components in Amazon EMR releases.Flink:Flink supported as a YARN applicationFlink release history: Before deciding to upgrade Amazon EMR, check the version of Flink and its installed components in Amazon EMR releases.Hadoop:Transparent encryption in Hadoop Distributed File System (HDFS)Non-uniform memory access awareness for YARN containersHadoop version history: Before deciding to upgrade Amazon EMR, check the version of Hadoop and its installed components in Amazon EMR releases.HBase:HBase on Amazon S3HBase read replica clustersHBase snapshotsHBase release history: Before deciding to upgrade Amazon EMR, check the version of HBase and its installed components in Amazon EMR releases.HCatalog:Integrations with Amazon EMR releasesUsing the AWS Glue Data Catalog as the metastore for Apache HiveHCatalog release history: Before deciding to upgrade Amazon EMR, check the version of HCatalog and its installed components in Amazon EMR releases.Hive:ACID transactions and Amazon S3Hive Live Long and Process (LLAP)Improve Hive performanceStart the Hive EMR File System (EMRFS) S3 Optimized CommitterUsing S3 Select with Hive to improve performanceMetastore check command (MSCK) optimizationHive release history: Before deciding to upgrade Amazon EMR, check the version of Hive and its installed components in Amazon EMR releases.Hudi:Integrations with Amazon EMR releasesHudi release history: Before deciding to upgrade Amazon EMR, check the version of Hudi and its installed components in Amazon EMR releases.Iceberg:Integrations with Amazon EMR releasesIceberg release history: Before deciding to upgrade Amazon EMR, check the version of Iceberg and its installed components in Amazon EMR releases.Presto and Trino:Integrations with Amazon EMR releasesUsing S3 Select Pushdown with Presto to improve performanceAdding database connectorsActivating Presto strict modeExchange ManagerUsing Presto automatic scaling with Graceful DecommissionPresto release history and Trino release notes: Before deciding to upgrade Amazon EMR, check the version of Presto or Trino and its installed components in Amazon EMR releases.Planning for Amazon EMR version upgradesFollow these steps to prepare for an Amazon EMR version upgrade:Research the issues that you're facing in your current Amazon EMR version.Isolate a small subset of applications or queries that you want to use to test your EMR cluster's performance.Set up an A/B testing strategy to decide the Amazon EMR version that's best for your solution. In A/B testing for Amazon EMR, you test two different versions of the service to compare how they perform in your environment.Gradually migrate the workload to the new version of Amazon EMR. If you discover major problems on the production version of Amazon EMR, you can end the migration process here.After migration is complete, terminate the old Amazon EMR cluster.Fixing issues related to Amazon EMR version upgradesFollow these steps to fix issues that you encounter when upgrading your Amazon EMR version:Reconfigure the application. Observe whether or not the changes improve your application's performance.Check if issues have been resolved by a newer version of the application.Change the application or queries to see if you can avoid issues.Check open defects and workarounds to improve the application. Contact AWS Premium Support to find out if there's a workaround.Stop the Amazon EMR migration until the issue is fixed or a workaround exists.Considerations for Amazon EMR version upgradesWhen you upgrade your version of Amazon EMR, performance regression might cause issues. Upgrades might change the API, which might affect your code's ability to run on a newer interface. Application slowness and failures might occur after an Amazon EMR version upgrade.When you're thinking of upgrading your version of Amazon EMR, it's a best practice to read the What's new? section of the release guide. The What's new? section includes information about Amazon EMR release versions and dates, along with solutions to common issues with open-source applications.Research open-source application changes and outstanding issuesCheck the following release notes and open defects before deciding to migrate to a new Amazon EMR version. The following list of applications are based on Amazon EMR version 6.9.Note: These hyperlinks take you to the third-party application websites, GitHub, or the Apache website.Flink release notes under Upgrade Flink and issue trackingGanglia release notes and issue trackingHadoop release notes and issue trackingHBase release notes and issue trackingHCatalog release notes and issue trackingHive release notes and issue trackingHue release notes and issue trackingJupyterEnterpriseGateway release notes and issue trackingJupyterHub release notes and issue trackingLivy release notes and issue trackingMXNet release notes and issue trackingOozie release notes and issue trackingPhoenix release notes and issue trackingPig release notes and issue trackingPresto release notes and issue trackingSpark release notes and issue trackingSqoop release notes under Releases and issue trackingTensorFlow release notes and issue trackingTez release notes and issue trackingTrino release notes and issue trackingZeppelin release notes and issue trackingZooKeeper release notes and issue trackingFollow"


Question: "How can I use Amazon SES to receive inbound emails, and then store those emails on Amazon S3?"

"I want to use Amazon Simple Email Service (Amazon SES) to receive inbound emails, and store the emails for archiving or further processing."

Answer: "I want to use Amazon Simple Email Service (Amazon SES) to receive inbound emails, and store the emails for archiving or further processing.ResolutionNote: Before you start, verify that your endpoint is in an AWS Region that supports email receiving. Except for Amazon S3 buckets, all the AWS resources that you use for receiving email with Amazon SES must be in the same AWS Region as the Amazon SES endpoint.Verify your email domain with Amazon SESVerify the domain that you want to use for receiving emails.Publish an MX record for Amazon SES to receive emailsFor Amazon SES to receive your domain's inbound emails, publish a mail exchanger record (MX record). Include an Amazon SES inbound receiving endpoint in the domain DNS configuration.Set up an Amazon S3 bucket with the permissions to store emails from Amazon SES1.    Open the Amazon S3 console.2.    Create an Amazon S3 bucket.3.    Select the bucket name from the bucket list.4.    Choose the Permissions tab.5.    Choose Bucket Policy.6.    Under Bucket policy editor, enter the following policy.Replace AWSDOC-EXAMPLE-BUCKET with the name of the S3 bucket that you want to write to.Replace 111122223333 with your AWS account ID. Replace region with the AWS Region where you want to create the receipt rule. Replace rule_set_name with the name of the rule set that contains the receipt rule. Replace receipt_rule_name with the name of the receipt rule. If you don't have a configured rule_set_name and receipt_rule_name in Amazon SES, then you can use any values to configure the bucket policy. Use the same values when you create the rule set and rule in the next section.{  "Version":"2012-10-17",  "Statement":[    {      "Sid":"AllowSESPuts",      "Effect":"Allow",      "Principal":{        "Service":"ses.amazonaws.com"      },      "Action":"s3:PutObject",      "Resource":"arn:aws:s3:::AWSDOC-EXAMPLE-BUCKET/*",      "Condition":{        "StringEquals":{          "AWS:SourceAccount":"111122223333",          "AWS:SourceArn": "arn:aws:ses:region:111122223333:receipt-rule-set/rule_set_name:receipt-rule/receipt_rule_name"        }      }    }  ]}7.    Choose Save.Note: For more information about this policy, see Give Amazon SES permission to write to an Amazon S3 bucket.Create an Amazon SES receipt rule that sends inbound emails to the S3 bucketOpen the Amazon SES console.In the navigation pane, under All rule sets, choose Email Receiving.To add the rule to an active rule set, proceed to step 4. To create a new rule set, choose Create a Rule Set, enter a rule set name, and then choose Create a Rule Set.Note: If you create a new rule set, select the rule set, and then choose Set as Active Rule Set. Only one of your receipt rule sets can be the active rule set at any given time.Choose Active Rule Set.Choose Create Rule.Note: You can also choose to update an existing rule with the same values that are described in the following steps.Enter a unique rule name. If your use case requires TLS or spam and virus scanning, then choose Require TLS or Enable spam and virus scanning. To make this an active rule, select the Enabled checkbox.Choose Next.To store only emails that Amazon SES sends to a specific verified domain, choose Add Recipient conditions, and then enter the email addresses as recipients. To store all emails that Amazon SES sends to all verified domains, don't enter any email addresses. Then, choose Next.Choose Add new action, and then choose Deliver to S3 bucket. Complete the following steps:      For S3 bucket, choose the bucket that you created to store emails.         For Object key prefix, enter an S3 prefix where you want to store the emails. If you leave this field blank, then Amazon SES stores emails at the root of the bucket.(Optional) Choose Message encryption for Amazon SES to use an AWS Key Management Server (AWS KMS) key to encrypt your emails.Note: If you're using a key other than the default root key, then you must grant Amazon SES permissions for using the AWS KMS key. Amazon SES uses Amazon S3 encryption client to encrypt your email before sending the email to the S3 bucket for storage. Amazon SES doesn't use Amazon S3 server-side encryption to encrypt your email.(Optional) For SNS topic, select an Amazon Simple Notification Service (Amazon SNS) topic to notify you when Amazon SES delivers an email to the S3 bucket.Choose Next.Choose Create Rule.Test the Amazon SES ruleOpen the Amazon SES console.In the navigation pane, choose Verified identities.Select your verified domain.Choose Send a Test Email. Then, complete these fields:For Email format, choose Formatted.For From-address, enter the email address that you want to send to the test email from.Under Scenario, choose Custom.For Custom recipient, enter an email address with the domain that you want to test.For Subject and Body, enter an example email text.Choose Send Test Email.Open the Amazon S3 console.Open the bucket that you configured to store the Amazon SES emails.Verify that the bucket contains the test email that you sent. It can take a few minutes for the test email to appear.Related informationWhy aren't my inbound emails on Amazon SES arriving or saving to my Amazon S3 bucket?Setting up Amazon SES email receivingFollow"


Question: How do I resolve issues using short codes and long codes to send messages in Amazon SNS?

I want to use short codes and long codes to successfully send messages in Amazon Simple Notification Service (Amazon SNS).

Answer: "I want to use short codes and long codes to successfully send messages in Amazon Simple Notification Service (Amazon SNS).Short descriptionIf you get failures after sending a message from a short code, go to the Check for a discrepancy between your message and origination identity section.If your short code is approved but messages are still sent by your long codes, go to the Troubleshoot messages sent from long codes section.Note: For information about pricing, see Worldwide SMS Pricing.Important: You can't share short codes or long codes between AWS Regions. Origination numbers are account and Region specific. You can use origination numbers only for the Region or account where the origination number was acquired.ResolutionCheck for a discrepancy between your message and origination identityYou can get message delays or failures for multiple reasons. For example, you can get message delays or failures if your message type and the short code route type don't match.Important: You must send the appropriate message type through the same type of origination number that's acquired. That is, transactional messages must use transactional short codes, and promotional messages must use promotional short codes.To check for a discrepancy, complete the following steps:Open the Amazon SNS console.On the navigation pane, expand Mobile, and then choose Origination numbers.Check the value of the Route type for your short code.To check the message type that's set when a message is sent, check the smsType field in your delivery logs.Troubleshoot messages sent from long codesConfirm that you're sending messages from the right short code.Update the keywords and responses in your SMS settings in Amazon Pinpoint. The keywords and responses in your SMS settings must match the keywords and responses that you registered when you requested your short code.Confirm that the country that you're sending messages to with your short code is the same country where you acquired your short code.Important: You can't send messages to multiple countries with the same short code.Note: You can only send messages to US numbers using a US short code. If you send a message to a CA number from a US short code, the message is delivered from a random CA long code.Related informationShort codesLong codesFollow"


Question: How can I enable communication between multiple VPCs from a single VPN connection attached to my transit gateway without allowing access between the VPCs?

"I have two virtual private clouds (VPCs): one for production and one for development. On-premises users need access to both VPCs with a single VPN connection. I must establish network connectivity between the VPCs and the on-premises network through a VPN connection. Additionally, I need to block access between the VPCs. How can I enable communication between multiple VPCs and a VPN connection attached to my transit gateway without allowing access between the VPCs?"

Answer: "I have two virtual private clouds (VPCs): one for production and one for development. On-premises users need access to both VPCs with a single VPN connection. I must establish network connectivity between the VPCs and the on-premises network through a VPN connection. Additionally, I need to block access between the VPCs. How can I enable communication between multiple VPCs and a VPN connection attached to my transit gateway without allowing access between the VPCs?Short DescriptionComplete the steps below to establish network connectivity between resources in multiple VPCs so that:On-premises users can access resources from all VPCs across the VPNVPC resources cannot access resources in the other VPCsResolutionCreate a transit gateway, and then attach your VPCs and a site-to-site VPNCreate a transit gateway.Attach your VPCs to your transit gateway.Create asite-to-site VPN connection and attach it to your transit gateway.Notes:Disable the Default association route tablesetting when creating your transit gateway.To automatically propagate VPN routes to the transit gateway route table, choose Dynamic(requires Border Gateway Protocol) for Routing option.Create a transit gateway route table and associate your VPCs to itOpen the Amazon Virtual Private Cloud (Amazon VPC) console.From the navigation pane, choose Transit Gateways.Verify that the Default association route tablesetting for your transit gateway is set to Disable.**Note:**If the setting is set to Enable, skip to step 8.Choose Transit Gateway Route Tables.Choose Create Transit Gateway Route Table, and then complete the following:For Name tag, enter Route Table A.For Transit Gateway ID, choose the Transit Gateway IDfor your transit gateway.Choose Create Transit Gateway Route Table.Choose Route Table A(or the default route table of your transit gateway). Then, choose Associations, Create Association.For Choose attachment to associate, choose the association IDs for your VPCs. Then, choose Create Association. Repeat this step until all of your VPCs display under Association.Delete the VPN association from default transit gateway route table.Create a second transit gateway route table and associate your VPN connection association to itChoose Transit Gateway Route Tables.Choose Create Transit Gateway Route Table, and then complete the following:For Name tag, enter Route Table B.For Transit Gateway ID, choose the Transit Gateway IDfor your transit gateway.Choose Create Transit Gateway Route Table.Choose Route Table B(or the default route table of your transit gateway). Then, choose Associations, Create Association.Associate the VPN connection that you just created with Route Table B.Propagate routes from your VPCs and VPN on both route tablesChoose Route Table A Propagation.Choose Propagation. For Choose attachment to propagate, choose the propagation for the VPN. If you have propagation enabled for all of the attachments, verify that the VPN connection association is not enabled in this route table.**Important:**If you created a VPN connection with the Static Routeoption (rather than Dynamic Routing), you must create a static route for the on-premises network to the VPN on Route Table A instead of enabling route propagation from the VPN connection.Choose Route Table B Propagation.Choose Propagation. For Choose attachment to propagate, choose the propagation for all of the VPCs.Configure the route table associated with your VPC and attachment subnetOpen the Amazon VPC console.From the navigation pane, choose Route Tables.Choose the route table that's attached to the attachment subnet.Choose the Routestab, and then choose Edit Routes.Choose the Add Routetab, and then complete the following:For Destination, choose the subnet of the on-premises network.For Target, choose your transit gateway.Choose Save routes.**Note:**If your use case requires more restrictive access between your VPCs, you can create a separate route table for each VPC and configure the routes. Keep in mind:Routing in the transit gateway route table is based on the association of the transit gateway association and the transit gateway route table.You can configure routes to any destination in the transit gateway attachment in any transit gateway route table. The transit gateway attachment doesn't need to be associated with that specific route table.Follow"


Question: "How do I configure the ATOP and SAR monitoring tools for my EC2 instance running Amazon Linux, RHEL, CentOS, or Ubuntu?"

"I have an Amazon Elastic Compute Cloud (Amazon EC2) instance that is running Amazon Linux, Red Hat Enterprise Linux (RHEL), CentOS, or Ubuntu. How do I configure the ATOP and SAR monitoring tools to collect granular data about process utilization?"

Answer: "I have an Amazon Elastic Compute Cloud (Amazon EC2) instance that is running Amazon Linux, Red Hat Enterprise Linux (RHEL), CentOS, or Ubuntu. How do I configure the ATOP and SAR monitoring tools to collect granular data about process utilization?Short descriptionATOP and SAR monitoring tools aren't configured on standard Amazon Machine Images (AMIs). You can install these monitoring tools on Amazon Linux, RHEL, CentOS, or Ubuntu Linux-based distributions. Enabling ATOP and SAR monitoring provides a more granular view into the instance's process utilization with a 30-day retention history.By default, ATOP and SAR monitoring tools collect metrics in 10-minutes intervals. You can reconfigure the interval from the default value to 1 minute intervals to collect more granular data.ResolutionAmazon Linux 21.    Install the EPEL release package for RHEL 7 using the amazon-linux-extras command:sudo amazon-linux-extras install epel2.    Install the monitoring tools:sudo yum -y install sysstat atop --enablerepo=epel3.    Change the log collection interval:sudo sed -i 's/^LOGINTERVAL=600.*/LOGINTERVAL=60/' /etc/sysconfig/atopsudo sed -i -e 's|*/10|*/1|' -e 's|every 10 minutes|every 1 minute|' /etc/cron.d/sysstat4.    Enable and restart services:sudo systemctl enable atop.service crond.service sysstat.servicesudo systemctl restart atop.service crond.service sysstat.serviceAmazon Linux AMI1.    Install the monitoring tools.Note: Amazon Linux already has the EPEL repository installed.sudo yum -y install sysstat atop --enablerepo=epelsudo sed -i 's/^INTERVAL=600.*/INTERVAL=60/' /etc/sysconfig/atopsudo sed -i -e 's|*/10|*/1|' -e 's|every 10 minutes|every 1 minute|' /etc/cron.d/sysstat2.    Enable and restart services:for _service in atop crond sysstat; do sudo chkconfig ${_service} on; sudo service ${_service} start; doneRHEL and CentOSThe following configuration steps are the same for each major release of RHEL and CentOS.RHEL 8 and CentOS 8:1.    Install the EPEL release package for RHEL 8:sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y2.    Install the monitoring tools:sudo dnf -y install sysstat atop --enablerepo=epel3.    Change the log collection interval:sudo sed -i 's/^LOGINTERVAL=600.*/LOGINTERVAL=60/' /etc/sysconfig/atopsudo mkdir -v /etc/systemd/system/sysstat-collect.timer.d/sudo bash -c "sed -e 's|every 10 minutes|every 1 minute|g' -e '/^OnCalendar=/ s|/10$|/1|' /usr/lib/systemd/system/sysstat-collect.timer > /etc/systemd/system/sysstat-collect.timer.d/override.conf"sudo sed -i 's|^SADC_OPTIONS=.*|SADC_OPTIONS=" -S XALL"|' /etc/sysconfig/sysstat4.    Enable and restart services:sudo systemctl enable atop.service crond.service sysstat.servicesudo systemctl restart atop.service crond.service sysstat.serviceRHEL 7 and CentOS 7:1.    Install the EPEL release package for RHEL 7:sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm2.    Install the monitoring tools:sudo yum -y install sysstat atop --enablerepo=epel3.    Change the log collection interval:sudo sed -i 's/^LOGINTERVAL=600.*/LOGINTERVAL=60/' /etc/sysconfig/atopsudo sed -i -e 's|*/10|*/1|' -e 's|every 10 minutes|every 1 minute|' /etc/cron.d/sysstat4.    Enable and restart services:sudo systemctl enable atop.service crond.service sysstat.servicesudo systemctl restart atop.service crond.service sysstat.serviceRHEL 6 and CentOS 6:1.    Install the EPEL release package for RHEL 6:sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm2.    Install the monitoring tools:sudo yum -y install sysstat atop --enablerepo=epel3.    Change the log collection interval:sudo sed -i 's/^INTERVAL=600.*/INTERVAL=60/' /etc/sysconfig/atopsudo sed -i -e 's|*/10|*/1|' -e 's|every 10 minutes|every 1 minute|' /etc/cron.d/sysstat4.    Enable and restart services:for _service in atop crond sysstat; do sudo chkconfig ${_service} on; sudo service ${_service} start; doneUbuntuUbuntu 20.04:1.    Install the monitoring tools:sudo apt-get updatesudo apt-get -y install atop sysstat2.    Change the log collection interval and configure sysstat to report disk and inodes usage by adding -S XALL in the configuration file:sudo sed -i 's/^LOGINTERVAL=600.*/LOGINTERVAL=60/' /usr/share/atop/atop.dailysudo sed -i -e 's|5-55/10|*/1|' -e 's|every 10 minutes|every 1 minute|' -e 's|debian-sa1|debian-sa1 -S XALL|g' /etc/cron.d/sysstatsudo bash -c "echo 'SA1_OPTIONS=\"-S XALL\"' >> /etc/default/sysstat"3.    Enable and restart services:sudo sed -i 's|ENABLED="false"|ENABLED="true"|' /etc/default/sysstatsudo systemctl enable atop.service cron.service sysstat.servicesudo systemctl restart atop.service cron.service sysstat.serviceUbuntu 18.04:1.    Install the monitoring tools:sudo apt-get updatesudo apt-get -y install atop sysstat2.    Change the log collection interval and configure sysstat to report disk and inodes usage by adding -S XALL in the configuration file:sudo sed -i 's/^INTERVAL=600.*/INTERVAL=60/' /usr/share/atop/atop.dailysudo sed -i -e 's|5-55/10|*/1|' -e 's|every 10 minutes|every 1 minute|' -e 's|debian-sa1|debian-sa1 -S XALL|g' /etc/cron.d/sysstatsudo bash -c "echo 'SA1_OPTIONS=\"-S XALL\"' >> /etc/default/sysstat"3.    Enable and restart services:sudo sed -i 's|ENABLED="false"|ENABLED="true"|' /etc/default/sysstatsudo systemctl enable atop.service cron.service sysstat.servicesudo systemctl restart atop.service cron.service sysstat.serviceUbuntu 16.04:1.    Install the monitoring tools:sudo apt-get updatesudo apt-get -y install atop sysstat2.    Change the log collection interval and configure sysstat to report disk and inodes usage by adding -S XALL in the configuration file:sudo sed -i 's/^INTERVAL=600.*/INTERVAL=60/' /etc/default/atopsudo sed -i -e 's|5-55/10|*/1|' -e 's|every 10 minutes|every 1 minute|' -e 's|debian-sa1|debian-sa1 -S XALL|g' /etc/cron.d/sysstatsudo bash -c "echo 'SA1_OPTIONS=\"-S XALL\"' >> /etc/default/sysstat"3.    Enable and restart services:sudo sed -i 's|ENABLED="false"|ENABLED="true"|' /etc/default/sysstatsudo systemctl enable atop.service cron.service sysstat.servicesudo systemctl restart atop.service cron.service sysstat.serviceRelated informationAdding repositories on an Amazon Linux instanceExtras library (Amazon Linux 2)Amazon Linux 2 FAQsEPEL websiteFollow"


Question: How can I prevent my Lambda function connected to an Amazon VPC from timing out?

I want to prevent my AWS Lambda function connection to an Amazon Virtual Private Cloud (Amazon VPC) from timing out.

Answer: "I want to prevent my AWS Lambda function connection to an Amazon Virtual Private Cloud (Amazon VPC) from timing out.Short descriptionInvoking Lambda function API calls behind a load balancer or NAT gateway without a response might be due to a connection idle timeout issue. The load balancer idle timeout value is 350 seconds for TCP flows.If the API call response delay is intermittent or less than 350 seconds, this might happen due to retry and timeout issues. For more information, see How do I troubleshoot retry and timeout issues when invoking a Lambda function using an AWS SDK?ResolutionTo prevent the connection from dropping, you can turn on TCP keepalive with a value of less than 350 seconds to reset the idle timeout. TCP keepalive prevents the connection from being idle by sending packets to the front-end and backend connections every 20 seconds.You can call setKeepAlive on the socket object similar to the following:Node.js 14 and Node.js 16const http = require('http');function invokeApi() {  return new Promise(resolve => {       var callback = function(response) {      var str = '';          response.on('data', function (chunk) {        str += chunk;      });           response.on('end', function () {        response = '**** ALB response: ' + str;        resolve(response)      });    }    const url = 'http://<alb-dns>';    const myTimeout = 1000*60*15; // 15 minutes    let req = http.get(url, callback);    req.on('error', (e) => {      console.error(`Got error: ${e.message}`);    });    req.on('socket', function (socket) {      socket.setKeepAlive(true, 1000); //set SO_KEEPALIVE         socket.setTimeout(myTimeout);        socket.on('timeout', function() {        console.log('request is about to timeout');        req.abort();      });    });    req.end();  })}exports.handler = async (event) => {    let output = await invokeApi();    const response = {        statusCode: 200,        body: output,    };    return response;};Node.js 18import * as http from "http"function invokeApi() {  return new Promise(resolve => {    var callback = function(response) {      var str = '';      response.on('data', function (chunk) {        str += chunk;      });      response.on('end', function () {        response = '**** ALB response: ' + str;        resolve(response);      });    }    const url = 'http://<alb-dns>';    const myTimeout = 1000*60*15; // 15 minutes    let req = http.get(url, callback);    req.on('error', (e) => {      console.error(`Got error: ${e.message}`);    });    req.on('socket', function (socket) {      socket.setKeepAlive(true, 1000); //set SO_KEEPALIVE      socket.setTimeout(myTimeout);        socket.on('timeout', function() {        console.log('request is about to timeout');        req.abort();      });    });    req.end();  })}export const handler = async(event) => {    let output = await invokeApi();    const response = {        statusCode: 200,        body: output,    };    return response;};Python 3.ximport jsonimport socketimport requestsfrom urllib3.connection import HTTPConnectionHTTPConnection.default_socket_options = HTTPConnection.default_socket_options + [(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)]def lambda_handler(event, context):    try:        r = requests.get("http://<alb-dns>",timeout=(5,900))        output = r.text    except requests.RequestException as e:        print(e)        raise e    return {        "statusCode": 200,        "body": json.dumps({            "message": output,        }),    }Using AWS SDK for Java2, you can pass a pre-configured HTTP client similar to the following:LambdaClient awsLambda = LambdaClient.builder()    .region(region    .httpClient(ApacheHttpClient.builder()                   .tcpKeepAlive(true)                   .connectionTimeout(Duration.ofSeconds(5))                   .connectionMaxIdleTime(Duration.ofMinutes(15))                   .socketTimeout(Duration.ofMinutes(15)).build())    .build();Using AWS SDK for Python (Boto3) allows you to use a configuration file. Create a .config file with following contents:[default]tcp_keepalive=trueThen, create a Lambda environment variable AWS_CONFIG_FILE with the value of the .config file in the directory /var/task/config.For more information, see Connection idle timeout.Related informationAnnouncing improved VPC networking for AWS Lambda functionsHow can I troubleshoot connectivity issues when using NAT gateway on my private VPC?Follow"


Question: How do I resolve query timeout issues in QuickSight?

"I'm trying to import data into Amazon QuickSight using direct query mode, but I get a query timeout error. How to resolve this issue?"

Answer: "I'm trying to import data into Amazon QuickSight using direct query mode, but I get a query timeout error. How to resolve this issue?Short descriptionThe following are common reasons for getting timeout errors when you're importing data into QuickSight using direct query mode:Data preparation takes more than 50 seconds.Generating a visual takes more than 2 minutes.The query runtime exceeds the data source timeout quota for the Amazon service that you're importing from. To see the data source timeout quota for your Amazon service, see AWS service quotas.Note: You can't increase the 2-minute visual generation timeout quota.ResolutionLimit the data that you're importingThe following are steps that you can take to limit the amount of data that you're importing into QuickSight:Unselect columns that you don't need.Add filters to your dataset in QuickSight.Use predicates in your custom SQL query such as WHERE and HAVING.Import the data into SPICEWhen you import your data into SPICE, queries process faster.Note: The 50-second timeout quota for data preparation and 2-minute timeout for generating visuals still apply when you use SPICE. However, if you're using Amazon Athena to import data into QuickSight, then the query timeout is 30 minutes.For more information on SPICE quotas for imported data and quotas for direct SQL queries, see Data source quotas.Follow"


Question: How do I update a CloudFormation stack that's failing because of a resource that I manually deleted?

"I manually deleted a resource that was created by one of my AWS CloudFormation stacks, and now my stack won't update."

Answer: "I manually deleted a resource that was created by one of my AWS CloudFormation stacks, and now my stack won't update.Short descriptionIf you delete a resource that was created by a CloudFormation stack, then your stack fails to update, and you get an error message.Choose one of the following resolutions based on your resource type:Update a resource with a unique nameUpdate a resource with a unique IDResolutionUpdate a resource with a unique nameIf the deleted resource supports a unique name, you can get your stack to update by manually creating the resource. The new resource, however, must have the same name as the deleted resource. For example, you can impact how CloudFormation handles a resource when you change a resource name. This is because AWS Identity and Access Management (IAM) roles are tracked by name. If you accidentally delete an IAM role, you can manually recreate that role with the same name.To update an IAM role or any other resource with a unique name, complete the following steps:Open the AWS CloudFormation console.Select your stack, and then choose the Resources view.In the Physical ID column, find the Physical ID of the IAM role that you want to replace with your new IAM role.Note: The Physical ID is the name of the resource.Create a new IAM role, and then give the new role the same Physical ID that you found in step 3.Update your stack.Note: For more information, see Modifying a stack template.Update a resource with a unique IDYou can't manually recreate a resource, such as a route table, with a unique ID. Instead, you must remove the resource and any references to that resource from your CloudFormation template.Note: The following resolution also works for resources that don't support a unique name or a unique ID. For example, a record set in a hosted zone.To allow CloudFormation to recreate a route table or any other resource with a unique ID, complete the following steps:In your CloudFormation template, remove the route table and any references to it.Update the stack.Note: CloudFormation tries to delete the route table. Because the resource has already been deleted, CloudFormation considers the deletion a successful operation.Add the route table and any additional resources or references that you require back to your CloudFormation template.Update your stack again.Note: CloudFormation recreates the deleted resource.Related informationTroubleshooting AWS CloudFormationFollow"


Question: Why is my Amazon DynamoDB table being throttled?

Read or write operations on my Amazon DynamoDB table are being throttled.-or-I'm getting ProvisionedThroughputExceededException when I perform a read or write operation on my DynamoDB table.

Answer: "Read or write operations on my Amazon DynamoDB table are being throttled.-or-I'm getting ProvisionedThroughputExceededException when I perform a read or write operation on my DynamoDB table.Short descriptionHere are some of the common throttling issues that you might face:Your DynamoDB table has adequate provisioned capacity, but most of the requests are being throttled.You activated AWS Application Auto Scaling for DynamoDB, but your DynamoDB table is being throttled.Your DynamoDB table is in on-demand capacity mode, but the table is being throttled.You have a hot partition in your table.Your table's traffic is exceeding your account throughput quotas.ResolutionFor information on DynamoDB metrics that must be monitored during throttling events, see DynamoDB metrics and dimensions. These metrics might help you to locate the operations creating throttled requests and identify the cause for throttling. Use one or more of the following troubleshooting options based on your use case.Your DynamoDB table has adequate provisioned capacity, but most of the requests are being throttledDynamoDB reports minute-level metrics to Amazon CloudWatch. The metrics are calculated as the sum for a minute and then averaged. However, the DynamoDB rate limits are applied per second. For example, if you provisioned 60 write capacity units for your DynamoDB table, then you can perform 3600 writes in one minute. However, driving all 3600 requests in one second with no requests for the rest of that minute might result in throttling. The total number of read capacity units or write capacity units per minute might be lower than the provisioned throughput for the table. However, if all the workload falls within a couple of seconds, then the requests might be throttled.To resolve this issue, make sure that your table has enough capacity to serve your traffic and retry throttled requests using exponential backoff. If you are using the AWS SDK, then this logic is implemented by default. For more information, see Error retries and exponential backoff.Note: DynamoDB doesn't necessarily start throttling the table after the consumed capacity per second exceeds the provisioned capacity. With the burst capacity feature, DynamoDB reserves a portion of the unused capacity for later bursts of throughput to handle usage spikes.You activated AWS Application Auto Scaling, but your table is still being throttledAWS Application Auto Scaling isn't a suitable solution to address sudden spikes in traffic with DynamoDB tables. Application Auto Scaling initiates a scale up only when two consecutive data points for consumed capacity units exceed the configured target utilization value within a one-minute span. Application Auto Scaling automatically scales the provisioned capacity only when the consumed capacity is higher than target utilization for two consistent minutes. Also, a scale-down event is initiated when 15 consecutive data points for consumed capacity in CloudWatch are lower than the target utilization. After Application Auto Scaling is initiated, an UpdateTable API call is invoked that might take a couple of minutes to update the provisioned capacity for your DynamoDB table or index. Application Auto Scaling requires consecutive data points with higher target utilization values to scale up the provisioned capacity of the DynamoDB table. During this period, any requests that exceed the provisioned capacity of the table are throttled. Therefore, it's not a best practice to use Application Auto Scaling to handle spiky workloads in DynamoDB and you might consider switching to on-demand mode in that use case. For more information, see Managing throughput capacity automatically with DynamoDB auto scaling.Your table uses the on-demand capacity mode, but the table is still being throttledWhen the table uses the on-demand capacity mode, the table doesn't throttle as long as the following conditions are true:The access pattern is distributed evenly across partitions to avoid issues related to hot partition.The table doesn't exceed double its previous peak traffic.For on-demand tables, DynamoDB automatically allocates more capacity as your traffic volume increases to make sure that your workload doesn't experience throttling. However, throttling can occur if the traffic volume is more than double the previous peak within a span of 30 minutes. For more information, see Peak traffic and scaling properties.You have a hot partition in your tableIn DynamoDB, a partition key that doesn't have a high cardinality can result in many requests targeting only a few partitions and resulting in a hot partition. A hot partition can cause throttling if the partition limits of 3000 RCU or 1000 WCU (or a combination of both) per second are exceeded.To find the most accessed and throttled items in your table, use the Amazon CloudWatch Contributor Insights. Amazon CloudWatch Contributor Insights is a diagnostic tool that provides a summarized view of your DynamoDB tables traffic trends and helps you identify the most frequently accessed partition keys. With this tool, you can continuously monitor the graphs for your table’s item access patterns. A hot partition can degrade the overall performance of your table. To avoid this poor performance, distribute the read and write operations as evenly as possible across your table. For more information, see Designing partition keys to distribute your workload evenly and Choosing the right DynamoDB partition key.Note: Using the CloudWatch Contributor Insights tool for DynamoDB incurs additional charges. For more information, see CloudWatch Contributor Insights for DynamoDB billing.Your table's traffic is exceeding your account throughput quotasThe table-level read throughput and table-level write throughput quotas apply at the account level in any Region. These quotas apply for tables with both provisioned capacity mode and on-demand capacity mode. By default, the throughput quota placed on your table is 40,000 read requests units and 40,000 write requests units. If the traffic to your table exceeds this quota, then the table might be throttled. To resolve this issue, use the Service Quotas console to increase the table-level read or write throughput quota for your account.Related informationUsing write sharding to distribute workloads evenlyFollow"


Question: How can I troubleshoot problems with viewing the Spark UI for AWS Glue ETL jobs?

I can't see the Apache Spark UI for AWS Glue ETL jobs.

Answer: "I can't see the Apache Spark UI for AWS Glue ETL jobs.ResolutionChoose one of the following solutions, depending on how you're accessing the Spark UI with an AWS CloudFormation stack or with Docker.AWS CloudFormation stackWhen you use an AWS CloudFormation stack to view the Spark UI, an Amazon Elastic Compute Cloud (Amazon EC2) instance makes an HTTPS request to confirm that the Spark UI is working. If that request fails, then you get the error "WaitCondition timed out. Received 0 conditions when expecting 1," and the AWS CloudFormation stack is rolled back.Check the following to resolve this issue:Subnet: Confirm that the subnet can reach the Amazon Simple Storage Service (Amazon S3) API endpoint. For example, if you're using a private subnet, then confirm that the subnet has a VPC endpoint or a NAT gateway.History server port: Confirm that you can access the subnet through the Spark history server port. For example, a firewall could be blocking the port.Event log directory: Confirm that you entered a valid Amazon S3 path for the event log directory. You must use s3a:// for the event logs path scheme. If there are event log files in the Amazon S3 path that you specified, then the path is valid.If you still get an error, then check the following log groups in Amazon CloudWatch Logs:/aws-glue/sparkui_cfn/cfn-init.log/aws-glue/sparkui_cfn/spark_history_server.logNote: The history server EC2 instance is terminated when the CloudFormation stack rolls back. To prevent the instance from being terminated, enable termination protection for the stack.DockerIf you're using Docker to view the Spark UI and you can't connect to the Spark history server from your web browser, then check the following:Confirm that the AWS credentials (access key and secret key) are valid. If you want to use temporary credentials, then you must use spark.hadoop.fs.s3a.session.token in the command. Example:$ docker run -itd -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \-Dspark.history.fs.logDirectory=s3a://path_to_eventlog \-Dspark.hadoop.fs.s3a.access.key=AWS_ACCESS_KEY_ID-Dspark.hadoop.fs.s3a.secret.key=AWS_SECRET_ACCESS_KEY \-Dspark.hadoop.fs.s3a.session.token=SESSION_TOKEN \-Dspark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider" \-p 18080:18080 glue/sparkui:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer"Confirm that you entered a valid Amazon S3 path for the event log directory. You must use s3a:// for the event logs path scheme. If there are event log files in the Amazon S3 path that you specified, then the path is valid.Confirm that you entered the correct port number in the browser. By default, the port number is 18080 (for example, http://localhost:18080). To change the port number, change the -p parameter in the command and the spark.history.ui.port parameter in the Dockerfile.If you still can't view the Spark UI, then check the logs. To get the stdout and stderr logs for the Docker container, run docker run with the -it parameter instead of the -itd parameter. Example:$ docker run -it -e SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS \-Dspark.history.fs.logDirectory=s3a://path_to_eventlog \-Dspark.hadoop.fs.s3a.access.key=AWS_ACCESS_KEY_ID-Dspark.hadoop.fs.s3a.secret.key=AWS_SECRET_ACCESS_KEY \-Dspark.hadoop.fs.s3a.session.token=SESSION_TOKEN \-Dspark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider" \-p 18080:18080 glue/sparkui:latest "/opt/spark/bin/spark-class org.apache.spark.deploy.history.HistoryServer"Related informationMonitoring jobs using the Apache Spark web UIEnabling the Apache Spark web UI for AWS Glue jobsFollow"


Question: Does CloudFront support static IP addresses?

Can I associate a static IP address with my Amazon CloudFront distribution?

Answer: "Can I associate a static IP address with my Amazon CloudFront distribution?ResolutionCloudFront doesn't support assigning a static IP address to distributions.When a user requests content from CloudFront, DNS returns 4 IP addresses of the edge location with the lowest latency. This allows content to be delivered with the best possible performance. Since edge locations are associated with different sets of IP addresses, you don't receive the same static address for your distribution when making a request.The IP address that is returned for the edge location is allocated dynamically from the IP address ranges of CloudFront edge servers list.The IP address ranges for CloudFront edge servers are subject to change. To be notified of IP address changes, subscribe to AWS Public IP Address Changes via Amazon SNS.Related informationUsing a dedicated IP address to serve HTTPS requests (works for all clients)Follow"


Question: Why can't a target behind my Network Load Balancer connect to its own Network Load Balancer?

"A target behind my Network Load Balancer is trying to connect to the same Network Load Balancer, but the connection fails. Why?"

Answer: "A target behind my Network Load Balancer is trying to connect to the same Network Load Balancer, but the connection fails. Why?Short descriptionWhen the target of an internal Network Load Balancer establishes a TCP connection to its own Network Load Balancer, the target can get routed to itself.Network Load Balancers preserve the source IP, so both the source and destination of the arriving packet are the private IP address of the target. Then, the host operating system sees the packet as not valid so it doesn't send response traffic, and the connection fails.Note: This connection failure happens only when the source and the target are the same. As a result, the connection issues happen intermittently, depending on the number of targets available to the Network Load Balancer.To correct this type of Network Load Balancer connection failure, deactivate client IP preservation for TCP and TLS target groups using the preserve_client_ip.enabled target group attribute.ResolutionTo activate or deactivate client IP preservation using the new console, do the following:Open the Amazon EC2 console.On the navigation pane, under Load Balancing, choose Target Groups.Choose the name of the target group to open its details page.On the Attributes tab, choose Edit.To activate client IP preservation, choose Preserve client IP addresses. To deactivate client IP preservation, clear Preserve client IP addresses.Choose Save changes.If your application needs the IP addresses of the clients, activate proxy protocol support and access the client IP addresses from the proxy protocol header.Related informationConnections time out for requests from a target to its load balancerFollow"


Question: How do I resolve the “VpcPeeringConnection failed to stabilize” error in AWS CloudFormation?

I get a "VpcPeeringConnection failed to stabilize" error in AWS CloudFormation when I try to create an Amazon Virtual Private Cloud (Amazon VPC) peering connection between an accepter VPC and requester VPC.

Answer: "I get a "VpcPeeringConnection failed to stabilize" error in AWS CloudFormation when I try to create an Amazon Virtual Private Cloud (Amazon VPC) peering connection between an accepter VPC and requester VPC.Short descriptionYou can receive this error for the following reasons:Your AWS::EC2::VPCPeeringConnection resource was created in the accepter account.IPv4 CIDR ranges overlap.The PeerRoleArn property isn't passed correctly when you're creating a VPC peering connection between VPCs in different accounts.The AWS Identity and Access Management (IAM) role in the accepter account doesn't have the right permissions.The PeerRegion property isn't passed correctly when you're creating a VPC peering connection between VPCs in different AWS Regions.ResolutionIf your AWS::EC2::VPCPeeringConnection resource was created in the accepter accountCreate your AWS CloudFormation stack with the AWS::EC2::VPCPeeringConnection resource in the requester account, not the accepter account.If IPv4 CIDR ranges overlapUse different IPv4 CIDR blocks for the VPCs in your accepter account and requester account.If the PeerRoleArn property isn't passed correctly when you create a VPC peering connection between VPCs in different accountsIf you're creating a VPC peering connection between VPCs in different accounts, then use the PeerRoleArn property to pass your cross-account IAM role from your accepter account in your AWS CloudFormation template. For more information, see AWS::EC2::VPCPeeringConnection. See the following JSON and YAML examples:JSON:{    "myVPCPeeringConnection": {        "Type": "AWS::EC2::VPCPeeringConnection",        "Properties": {            ......            "PeerRoleArn": "arn:aws:iam::Accepter-Account-ID:role/PeerRole"        }    }}YAML:myVPCPeeringConnection:  Type: 'AWS::EC2::VPCPeeringConnection'  Properties:    .......    PeerRoleArn: 'arn:aws:iam::Accepter-Account-ID:role/PeerRole'If the IAM role in the accepter account doesn't have the right permissionsTo allow the IAM role to accept a VPC peering connection in the accepter account, include the following permissions:{    "Version": "2012-10-17",    "Statement": [        {            "Action": "ec2:AcceptVpcPeeringConnection",            "Resource": "arn:${Partition}:ec2:${Region}:${Account}:vpc-peering-connection/${VpcPeeringConnectionId}",            "Effect": "Allow"        }    ]}To allow the requester account to assume the IAM role, configure a trust relationship for the IAM role. For example:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "AWS": "arn:aws:iam::Requester-Account-ID:root"      },      "Action": "sts:AssumeRole"    }  ]}If the PeerRegion property isn't passed correctly when you're creating a VPC peering connection between VPCs in different AWS RegionsIf the VPCs are located in different AWS Regions, then you must include the PeerRegion in your AWS CloudFormation template. Then, specify the AWS Region where your accepter account VPC is located. See the following JSON and YAML examples:JSON:{    "myVPCPeeringConnection": {        "Type": "AWS::EC2::VPCPeeringConnection",        "Properties": {            ......            "PeerRegion": Accepter-VPC-Region-Code        }    }}YAML:myVPCPeeringConnection:  Type: 'AWS::EC2::VPCPeeringConnection'  Properties:    ......    PeerRegion: Accepter-VPC-Region-CodeRelated informationWalkthrough: Peer with an Amazon VPC in another AWS accountCreate a VPC peering connectionFollow"


Question: Why is my AWS Config rule with required-tags status stuck on "Evaluating"?

"I created an AWS Config managed rule using required-tags to check for specific resources. However, my AWS Config rule with required-tags status is still "Evaluating" resources, or reporting unexpected results."

Answer: "I created an AWS Config managed rule using required-tags to check for specific resources. However, my AWS Config rule with required-tags status is still "Evaluating" resources, or reporting unexpected results.ResolutionUse the following best practices when creating and editing AWS Config rules using required-tags.The AWS Config configuration recorder must be turned on and configured in the same AWS Region as the AWS Config rule. To turn the configuration recorder on or off, see managing the configuration recorder.AWS Config rules using required-tags typically return results in 20 minutes or less. Results can vary depending on the service or resource type due to downstream dependencies.If the AWS Config rules Scope of changes is set to Resources, then verify that the resource type (for example, EC2::Instance) is specified for the trigger.If the AWS Config rules Scope of changes is set to Tags with or without a tag value, then a supported resource must be tagged with a tag key.Verify that the AWS Identity and Access Management (IAM) role assigned to AWS Config has the AWS_ConfigRole managed policy attached. For more information, see IAM role policy for getting configuration details.For configuring AWS Config to record to Amazon Simple Storage Service (Amazon S3) buckets, verify that the permissions allow recording changes. For more information, see Troubleshooting for recording S3 buckets.Related informationHow can I troubleshoot AWS Config console error messages?How can I be notified when an AWS resource is non-compliant using AWS Config?IAM role policy for getting configuration detailsFollow"


Question: How do I control access to my WorkSpaces?

I want to set access controls in Amazon WorkSpaces so that specific users or specific IP addresses can access my WorkSpace. How can I do that?

Answer: "I want to set access controls in Amazon WorkSpaces so that specific users or specific IP addresses can access my WorkSpace. How can I do that?ResolutionWorkSpaces access is managed in the directory details for your WorkSpace. The Access Control Options settings define the parameters for trusted devices. The IP Access Control Groups settings apply public IP addresses for the client connection.To access these settings for your WorkSpace, in the WorkSpaces console, choose Directories from the navigation pane. In the Actions dropdown, choose Update Details.Access Control OptionsUse Access Control Options to limit access to your WorkSpace. Do this by specifying the operating systems and certificates installed on the systems, or by choosing client platforms.To use certificates, you must have an internal certificate authority (CA) and provide two certificates:For the WorkSpaces directory: A root certificate generated by an internal CA.For a client device: An installed client certificate that links to a root certificate for the client device.For more information about creating and deploying certificates in WorkSpaces, see Restrict WorkSpaces access to trusted devices.IP Access Control GroupsUse IP Access Control Groups to restrict access based on the public IP address ranges that are allowed to connect to the WorkSpace. The allowed IP ranges are managed with groups created in the IP Access Controls navigation pane.These are the limitations of using this feature:Up to 25 IP access control groups are allowed with a single directory.100 IP access control groups are allowed in each AWS Region.For more information about IP access control groups, see IP access control groups for your WorkSpaces.Note: To find the public IP address of a user, use http://checkip.amazonaws.com/.Follow"