Question: Terminating connection due to serverless scale event timeout

I am getting sporadic connection timeouts when running on Aurora withe the following message:"terminating connection due to serverless scale event timeout"This happens when increased load on the server causes the cluster to scale.Has anyone seen this on their cluster and have any advice on how to resolve it?Cluster DetailsEngineAurora PostgreSQLRegion & AZus-west-2Capacity settingsMinimum Aurora capacity unit2 capacity unitsMaximum Aurora capacity unit384 capacity unitsPause compute capacity after consecutive minutes of inactivity5 minutesForce scaling the capacity to the specified values when the timeout is reachedEnabledEncryptionEnabledMaintenance windowsat:12:30-sat:13:00 UTC (GMT)FollowComment

Answer: "0Accepted AnswerLooks to me like this is intended behavior: you have enabled "Force scaling the capacity to the specified values when the timeout is reached", which means that connections which block a scaling event due to active (probably long-running) transactions will be terminated when the scaling timeout is reached. If you want the scaling event to wait for all active transactions to finish, you'll need to disable that setting...-Kevin J (PM for Aurora PostgreSQL)CommentShareAWS-User-3578276answered 3 years ago0Hi, I am currently having the very same issue.But at least for me, if I disable the option to "force autoscale", I get an error in the Aurora events log with the message: "Scaling point wasn't found".How to proceed?CommentSharebressanmarcosanswered 2 years ago"


Question: Does AWS Elemental Media Convert have an VPC endpoint?

"My Media Convert API endpoint is currently resolving toa public IP and I would like to access the Media Convert endpoint from within the VPC(without the internet), does it have a VPC endpoint?FollowComment"

Answer: 1AWS Elemental Media Convert does not currently support PrivateLink (VPC Endpoint).  You can see a list of services that do on the following page: https://docs.aws.amazon.com/vpc/latest/privatelink/aws-services-privatelink-support.htmlCommentShareBill_Sanswered 9 days ago


Question: How long so we have to upgrade?

We have received a notification that Amazon RDS for PostgreSQL is deprecating major version 10 and we need to up grade.This presents some problems for us and I would like to know how long we have until the issue of deprecating version 10 will affect our database? We need as long as possible.  What is possible?FollowComment

Answer: "1Hello,Thank you again for your question and I believe the answer you are asking is in the third reference that I provided earlier which mentions as follows:" RDS will upgrade your PostgreSQL 10 databases to PostgreSQL 14 during a scheduled maintenance window between April 17, 2023 00:00:01 UTC and July 18, 2023 00:00:01 UTC. On July 18, 2023 00:00:01 AM UTC, any PostgreSQL 10 databases that remain will be force upgraded to version 14 regardless of instances’ scheduled maintenance window."So, starting February 14th, even if you wont be able to create new RDS with POSTgreSQL, you will still be able to keep using your database. RDS will upgrade your PostgreSQL 10 databases to PostgreSQL 14 during a scheduled maintenance window between April 17, 2023 00:00:01 UTC and July 18, 2023 00:00:01 UTC. So, in your specific case, our recommendation is to fix the problem you have currently and start the upgrade. RDS will upgrade your PostgreSQL version during a scheduled maintenance window. So, you will know ahead of time when that will happen. If you donot want that maintenance to happen during that time, then you can select to adjust that maintenance window.https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.Maintenance.html#AdjustingTheMaintenanceWindowWe donot have visibility to any  other information than what is publicly available in our docs.I hope this helps. If you need further info, let me know in the comments; otherwise I'd appreciate if you mark my answer as "accepted".Kind regards, Rochak from AWSReferences:https://repost.aws/questions/QUph1IFLkkRiyc0pCdTH493Q/announcement-amazon-rds-for-postgre-sql-10-deprecationCommentShareAWS-Rochakanswered 3 months ago0Hello ,Thank you for posting your question on the AWS Repost, my name is Rochak and it will be a pleasure assisting you with this today.I understand you received a notification thay RDS for PostgreSQL is deprecating version 10 and you would like to know how long you have until it affects your database.Please, let me know if my understanding is incorrect.Yes, you are correct and Amazon RDS plans to deprecate PostGreSQL 10 on Aril 17, 2023. [1]We strongly recommend that you take action as soon as possible and upgrade your PostgreSQL databases running on major version 10 to a later major version, such as version 14. [2]Also, regarding the details on how the whole deprecation process will affect you and what you can do, please follow this announcement from AWS which includes various other valuable resources regarding the topic.[3]I hope this helps. If you need further info, let me know in the comments; otherwise I'd appreciate if you mark my answer as "accepted".Kind regards,Rochak from AWSReferences:[1] Deprecation of PostgreSQL version 10https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.DBVersions.Deprecation10[2] Upgrading the PostgreSQL DB engine for Amazon RDShttps://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html[3] [Announcement] Amazon RDS for PostgreSQL 10 deprecationhttps://repost.aws/questions/QUph1IFLkkRiyc0pCdTH493Q/announcement-amazon-rds-for-postgre-sql-10-deprecationCommentShareAWS-Rochakanswered 3 months ago0Thank you for this but can you be more exact?We know it will take some time to fix a problem we have before we can upgrade.Can you indicate how long it would be before not upgrading would affect the operation of our site? Two weeks? Six weeks? Eight weeks?  Any indication would be very useful.Thank you.CommentSharerePost-User-8337090answered 3 months ago"


Question: PostGIS - Aurora database Raster GIST index support

"Hello,I have created a PostgreSQL database, including the PostGIS extension, however, I'm unable to create GiST index on my rast column.I'm trying to execute the following line:CREATE INDEX IF NOT EXISTS rast_index ON public.raster_data USING gist(rast);I get the following response:ERROR:  data type raster has no default operator class for access method "gist"HINT:  You must specify an operator class for the index or define a default operator class for the data type.SQL state: 42704FollowComment"

Answer: "0Hi,I understand you are not able to create GiST index on my rast column using the below commandCREATE INDEX IF NOT EXISTS rast_index ON public.raster_data USING gist(rast);Kindly note that GiST indexes enable building general B-Tree structures that you can use for operations more complex than equality and range comparisons. They are mainly used to create indexes for geometric data types and they support full-text search indexing.With that said above you can only create GIST indexes for geometric data types such as ( point , polygon , lseg , box , circle , and path) for PostgreSQL.See the example below:postgres=> CREATE TABLE my_points (id serial PRIMARY KEY,p geometry(POINT),pz geometry(POINTZ),pm geometry(POINTM),pzm geometry(POINTZM),p_srid geometry(POINT,4269));CREATE TABLEpostgres=> CREATE INDEX IF NOT EXISTS rast_index ON public.my_points USING gist(p);CREATE INDEXFor more information about GIST, kindly visit the below link:[+]https://docs.aws.amazon.com/dms/latest/oracle-to-aurora-postgresql-migration-playbook/chap-oracle-aurora-pg.tables.indexes.html#chap-oracle-aurora-pg.tables.indexes.pg[+]https://www.postgresql.org/docs/9.0/textsearch-indexes.htmlCommentShareWinnieanswered a year ago"


Question: How to handle an access pattern in which you need to "list values of all partions with some`userid` pk"

"In DynamoDB, how do you handle an access pattern in which you need to get a list of partition values?For instance, say the partition-key is userid, and the attributes are firstname and lastname for the sake of simplicity (but in reality there is a lot more attributes). How do I then get a list of all the value pairs [(uiserid_1-firstname, uiserid_1-lastname), (uiserid_2-firstname, uiserid_2-lastname), ...]? Obviously, doing a scan or a query for each partition will consume too many read requests.So far, I think the best option would be to create a composite key: The pk is a random number prefixed with, say, list-users- to use as a partition-key; The sk is the userid. Then, all the items in these partitions are going to be the value of the userid partition duplicated. (I hope this makes sense, if not I can create a visualisation to better convey the idea).Doing it this way, you only need to perform one query on that random numbered partition-key; hence much more efficient.Is this the correct dynamodb-way of handling this access-pattern?How would you handle this access pattern?FollowCommentjzhunter 9 months agoIt's unclear exactly what you want to achieve. It's also not obvious yet why a scan is wrong. Perhaps you can clarify?Share"

Answer: "0You're saying that doing a scan in your scenario will consume more RCUs than what you expect for reading the data you require. Is this due to a large item size, multiple sk per pk, or something else?If this is due to your large item size, consider using indexes and projecting attributes carefully: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-indexes-general.html#bp-indexes-general-efficiencyIf this is due to multiple items per pk, you could leverage a sparse index design pattern. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-indexes-general-sparse-indexes.htmlHope this helps.CommentSharePadma_awsanswered 4 months ago"


Question: [NAT instance] Disable source/destination checks not working

"I tried creating an NAT instance follow by https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCheck.But when I selected the NAT instance, choose Actions, Networking, Change source/destination check, enabling popup was not showing.I want to make sure this AWS feature is working or not?ThanksFollowComment"

Answer: "0Accepted AnswerI changed the UI mode to default instead of "New EC2 Experience"(switch on the left top screen).After that "Disable source/destination" works.CommentSharerePost-User-5902367answered 5 months agoEXPERTTushar_Jreviewed 4 months ago0I'm seeing the popup. Could be a browser issue. Try switching to another browser. Or fire up CloudShell and just issue the CLI command.CommentShareEXPERTkentradanswered 5 months agorePost-User-5902367 5 months agoCan you tell me what browser you are using? I tested on chrome(windows, mac), safari, opera(mac), firefox(mac) and it did not work at all.Sharekentrad EXPERT5 months agoChrome on Windows.Share0I'm seeing the same - "Actions, Networking, Change source/destination check" does nothing.CommentShareEXPERTskinsmananswered 5 months agorePost-User-5902367 5 months agoThank you for the confirmation.Share0I'm having the same issue. Changing browser didn't helpCommentSharerePost-User-6175416answered 5 months ago0You need to select the NAT gateway and then under Details you will see Primary Network Interface ID or ENI. Clicking on the ENI will redirect you to the ENI page and from there you can click on the ENI and then the option to change Source/Destination check is available for the user to change.CommentSharemaverick_tomahawkanswered 2 months ago"


Question: sagemaker xgboost estimator fail to fit

"I used the training script from https://sagemaker.readthedocs.io/en/stable/frameworks/xgboost/using_xgboost.html,and trying to train the model. And the here is the code I used for configuring the model.`    sess = sagemaker.Session()hyperparams = {"max_depth": "5","eta": "0.2","gamma": "4","min_child_weight": "6","subsample": "0.7","objective": "reg:squarederror","num_round": "50","verbosity": "2"}instance_type = "ml.m5.2xlarge"output_path = f's3://{bucket}/Model-Resource-Production/dr_training_folder'content_type = "libsvm"# Create XGBoost estimatorxgboost_estimator = XGBoost(    entry_point="script.py",    framework_version="1.7-1",  # Note: framework_version is mandatory    hyperparameters=hyperparams,    role=role,    instance_count=1,    instance_type=instance_type,    output_path=output_path,)file_path = 'Model-Resource-Production/dr_training_folder'# Read train test file from S3training_input_config = TrainingInput(s3_data=f's3://{bucket}/{file_path}/DR_train', content_type='csv')validation_input_config = TrainingInput(s3_data=f's3://{bucket}/{file_path}/DR_valid', content_type='csv')# Fit the modelxgboost_estimator.fit({'train': training_input_config, 'validation': validation_input_config})`However, it returns the error during the "Training in progress" and here is the error:`    sess = sagemaker.Session()hyperparams = {"max_depth": "5","eta": "0.2","gamma": "4","min_child_weight": "6","subsample": "0.7","objective": "reg:squarederror","num_round": "50","verbosity": "2"}instance_type = "ml.m5.2xlarge"output_path = f's3://{bucket}/Model-Resource-Production/dr_training_folder'content_type = "libsvm"# Create XGBoost estimatorxgboost_estimator = XGBoost(    entry_point="script.py",    framework_version="1.7-1",  # Note: framework_version is mandatory    hyperparameters=hyperparams,    role=role,    instance_count=1,    instance_type=instance_type,    output_path=output_path,)file_path = 'Model-Resource-Production/dr_training_folder'# Read train test file from S3training_input_config = TrainingInput(s3_data=f's3://{bucket}/{file_path}/DR_train', content_type='csv')validation_input_config = TrainingInput(s3_data=f's3://{bucket}/{file_path}/DR_valid', content_type='csv')# Fit the modelxgboost_estimator.fit({'train': training_input_config, 'validation': validation_input_config})``---------------------------------------------------------------------------UnexpectedStatusException                 Traceback (most recent call last)Cell In[4], line 7168 validation_input_config = TrainingInput(s3_data=f's3://{bucket}/{file_path}/DR_valid', content_type='csv')70 # Fit the model---> 71 xgboost_estimator.fit({'train': training_input_config, 'validation': validation_input_config})File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/workflow/pipeline_context.py:284, in runnable_by_pipeline.<locals>.wrapper(*args, **kwargs)280         return context282     return _StepArguments(retrieve_caller_name(self_instance), run_func, *args, **kwargs)--> 284 return run_func(*args, **kwargs)File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/estimator.py:1198, in EstimatorBase.fit(self, inputs, wait, logs, job_name, experiment_config)1196 self.jobs.append(self.latest_training_job)1197 if wait:-> 1198     self.latest_training_job.wait(logs=logs)File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/estimator.py:2344, in _TrainingJob.wait(self, logs)2342 # If logs are requested, call logs_for_jobs.2343 if logs != "None":-> 2344     self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)2345 else:2346     self.sagemaker_session.wait_for_job(self.job_name)File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/session.py:4756, in Session.logs_for_job(self, job_name, wait, poll, log_type)4753             last_profiler_rule_statuses = profiler_rule_statuses4755 if wait:-> 4756     self._check_job_status(job_name, description, "TrainingJobStatus")4757     if dot:4758         print()File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sagemaker/session.py:4263, in Session._check_job_status(self, job, desc, status_key_name)4257 if "CapacityError" in str(reason):4258     raise exceptions.CapacityError(4259         message=message,4260         allowed_statuses=["Completed", "Stopped"],4261         actual_status=status,4262     )-> 4263 raise exceptions.UnexpectedStatusException(4264     message=message,4265     allowed_statuses=["Completed", "Stopped"],4266     actual_status=status,4267 )UnexpectedStatusException: Error for Training job sagemaker-xgboost-2023-04-18-13-58-00-099: Failed. Reason: AlgorithmError: ExecuteUserScriptError:Command "/miniconda3/bin/python3 -m script --eta 0.2 --gamma 4 --max_depth 5 --min_child_weight 6 --num_round 50 --objective reg:squarederror --subsample 0.7 --verbosity 2", exit code: 1`It seems some problems during passing the hyperparameters but I don't know what actually is going on. Thanks so much in advance.FollowComment"


Question: How to escape a comma in a csv file in AWS Glue?

"Customer is using Glue to parse a CSV file.The issue happens when trying to load a CSV file and one of its text column EQUIPMENT_DESCRIPTION has “,” (Comma) value in it. This column value is identified with text qualifier (double quotes). But the way we currently have our crawler/glue configured, we are not able to parse the column with the text qualifier to ignore comma values in it.This is causing all the other columns following EQUIPMENT_DESCRIPTION, to be filled with incorrect data including the DATE_COLLECTED column.How do you configure Glue to ignore this comma?FollowComment"

Answer: 0Accepted AnswerYou should confirm that the quote character is configured appropriately in the SerDe properties of the table?Have a look at Working with CSV Files in Best Practices When Using Athena With AWS Glue.CommentShareEXPERTJDBanswered 4 years ago


Question: Trouble joining svv_table_info and svv_interleaved_columns tables

"We earlier used to run the following query to get the skew for tables with interleaved sort keys and it suddenly started failing:Query: " select svv_table_info.table as tablename, svv_table_info.schema as schemaname, interleaved_skewfrom svv_interleaved_columns join  svv_table_infowhere svv_interleaved_columns.tbl = svv_table_info.table_id"Failing with Error: " ERROR: This type of correlated subquery pattern is not supported due to internal error"FollowCommentHi5 9 days agoWe are also facing similar issue. Earlier query used to work but has started failing staring 03/30.SELECT 1 FROM svv_table_info svvti                JOIN svv_interleaved_columns svvic ON svvti.table_id = svvic.tbl;[XX000] ERROR: This type of correlated subquery pattern is not supported due to internal errorShare"

Answer: "0A temporary fix can be new view creation based on definition of svv_interleaved_columns by removing all correlated subqueries and use it in join condition. Refer below code for the same.create or replace view <schema_name>.new_svv_interleaved_columns(tbl, col, interleaved_skew, last_reindex) asSELECT skew_d.tbl, skew_d.col, skew_d.interleaved_skew, vac_d.end_time AS last_reindexFROM (SELECT smoothed_dtls.tbl,             smoothed_dtls.col,             CASE                 WHEN sum(smoothed_dtls.smoothed_count) = 0 THEN NULL::numeric                 ELSE ("max"(smoothed_dtls.smoothed_count)::double precision /                       (sum(smoothed_dtls.smoothed_count)::double precision /                        count(DISTINCT smoothed_dtls.compressed_val)::double precision))::numeric(19, 2)                 END AS interleaved_skew      FROM (SELECT derived_table13.tbl,                   derived_table13.col,                   derived_table13.compressed_val,                   derived_table13.orig_count,                   CASE                       WHEN derived_table13.orig_count <> 0 AND                            (derived_table13.prec_count <> 0 OR derived_table13.prec_count IS NULL)                           THEN derived_table13.orig_count                       WHEN (derived_table13.group_size - derived_table13.rank) <                            (derived_table13.count % derived_table13.group_size)                           THEN derived_table13.count / derived_table13.group_size + 1                       ELSE derived_table13.count / derived_table13.group_size                       END AS smoothed_count            FROM (SELECT wt.tbl,                         wt.col,                         indices.compressed_val,                         indices.count                                                 AS orig_count,                         wt.count,                         pg_catalog.lead(indices.count, 1)                         OVER (                             PARTITION BY wt.tbl, wt.col                             ORDER BY indices.compressed_val DESC)                     AS prec_count,                         pg_catalog.rank()                         OVER (                             PARTITION BY wt.tbl, wt.col, indices.idx                             ORDER BY indices.compressed_val)                          AS rank,                         count(*)                         OVER (                             PARTITION BY wt.tbl, wt.col, indices.idx                             ORDER BY indices.compressed_val                             ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS group_size                  FROM (SELECT a.tbl,                               a.col,                               a.compressed_val,                               a.count,                               min(COALESCE(b.compressed_val, a.compressed_val)) AS idx                        FROM (SELECT counts_sum.tbl, counts_sum.col, counts_sum.compressed_val, counts_sum.count                              FROM (SELECT stv_interleaved_counts.tbl,                                           stv_interleaved_counts.col,                                           stv_interleaved_counts."index",                                           stv_interleaved_counts.active,                                           stv_interleaved_counts.compressed_val,                                           sum(stv_interleaved_counts.count) AS count                                    FROM stv_interleaved_counts                                    GROUP BY stv_interleaved_counts.tbl, stv_interleaved_counts.col,                                             stv_interleaved_counts."index", stv_interleaved_counts.active,                                             stv_interleaved_counts.compressed_val)                                    counts_sum                                JOIN                                   (SELECT tbl, min(stv_interleaved_counts.active) AS min_val                                    FROM stv_interleaved_counts                                    GROUP BY tbl                                   ) min_sic                                   ON counts_sum.active = min_sic.min_val                                    AND counts_sum.tbl = min_sic.tbl                              GROUP BY counts_sum.tbl, counts_sum.col, counts_sum.compressed_val, counts_sum.count                            ) a                                 LEFT JOIN (SELECT counts_sum.tbl,                                                   counts_sum.col,                                                   counts_sum.compressed_val,                                                   counts_sum.count                                            FROM (SELECT stv_interleaved_counts.tbl,                                                         stv_interleaved_counts.col,                                                         stv_interleaved_counts."index",                                                         stv_interleaved_counts.active,                                                         stv_interleaved_counts.compressed_val,                                                         sum(stv_interleaved_counts.count) AS count                                                  FROM stv_interleaved_counts                                                  GROUP BY stv_interleaved_counts.tbl, stv_interleaved_counts.col,                                                           stv_interleaved_counts."index",                                                           stv_interleaved_counts.active,                                                           stv_interleaved_counts.compressed_val                                                ) counts_sum                                JOIN                                   (SELECT tbl, min(stv_interleaved_counts.active) AS min_val                                    FROM stv_interleaved_counts                                    GROUP BY tbl                                   ) min_sic                                   ON counts_sum.active = min_sic.min_val                                    AND counts_sum.tbl = min_sic.tbl                              GROUP BY counts_sum.tbl, counts_sum.col, counts_sum.compressed_val, counts_sum.count                                     ) b                                           ON a.tbl = b.tbl AND a.col = b.col AND                                              b.compressed_val >= a.compressed_val AND                                              (a.count <> 0 OR b.count > a.count)                        GROUP BY a.tbl, a.col, a.compressed_val, a.count) indices,                       (SELECT counts_sum.tbl, counts_sum.col, counts_sum.compressed_val, counts_sum.count                        FROM (SELECT stv_interleaved_counts.tbl,                                     stv_interleaved_counts.col,                                     stv_interleaved_counts."index",                                     stv_interleaved_counts.active,                                     stv_interleaved_counts.compressed_val,                                     sum(stv_interleaved_counts.count) AS count                              FROM stv_interleaved_counts                              GROUP BY stv_interleaved_counts.tbl, stv_interleaved_counts.col,                                       stv_interleaved_counts."index", stv_interleaved_counts.active,                                       stv_interleaved_counts.compressed_val                                   ) counts_sum                                JOIN                                   (SELECT tbl, min(stv_interleaved_counts.active) AS min_val                                    FROM stv_interleaved_counts                                    GROUP BY tbl                                   ) min_sic                                   ON counts_sum.active = min_sic.min_val                                    AND counts_sum.tbl = min_sic.tbl                              GROUP BY counts_sum.tbl, counts_sum.col, counts_sum.compressed_val, counts_sum.count                           ) wt                  WHERE indices.idx = wt.compressed_val                    AND indices.tbl = wt.tbl                    AND indices.col = wt.col) derived_table13) smoothed_dtls      GROUP BY smoothed_dtls.tbl, smoothed_dtls.col) skew_d      LEFT JOIN (SELECT outer_pairs.table_id, outer_pairs.end_time                    FROM (SELECT vac_start.table_id,                                 vac_start.eventtime    AS start_time,                                 min(vac_end.eventtime) AS end_time                          FROM stl_vacuum vac_start,                               (SELECT e.table_id, e.eventtime                                FROM stl_vacuum s,                                     stl_vacuum e                                WHERE e.status = 'Finished'::bpchar                                  AND (s.status = 'Started'::bpchar OR s.status = 'Started Sort Only'::bpchar OR                                       s.status = 'Started Reindex'::bpchar)                                  AND s.xid = e.xid                                  AND s.table_id = e.table_id) vac_end                          WHERE vac_start.table_id = vac_end.table_id                            AND vac_start.status = 'Started Reindex'::bpchar                            AND vac_start.eventtime < vac_end.eventtime                          GROUP BY vac_start.table_id, vac_start.eventtime                         ) outer_pairs                             JOIN (SELECT temp_inner.table_id, max(temp_inner.start_time) AS max_start_time                                   FROM (SELECT vac_start.table_id,                                                vac_start.eventtime    AS start_time,                                                min(vac_end.eventtime) AS end_time                                         FROM stl_vacuum vac_start,                                              (SELECT e.table_id, e.eventtime                                               FROM stl_vacuum s,                                                    stl_vacuum e                                               WHERE e.status = 'Finished'::bpchar                                                 AND (s.status = 'Started'::bpchar OR                                                      s.status = 'Started Sort Only'::bpchar OR                                                      s.status = 'Started Reindex'::bpchar)                                                 AND s.xid = e.xid                                                 AND s.table_id = e.table_id) vac_end                                         WHERE vac_start.table_id = vac_end.table_id                                           AND vac_start.status = 'Started Reindex'::bpchar                                           AND vac_start.eventtime < vac_end.eventtime                                         GROUP BY vac_start.table_id, vac_start.eventtime                                        ) temp_inner                                   WHERE temp_inner.end_time IS NOT NULL                                   GROUP BY temp_inner.table_id                    ) inner_pairs                                  ON outer_pairs.start_time = inner_pairs.max_start_time                                      AND inner_pairs.table_id = outer_pairs.table_id) vac_d                   ON skew_d.tbl = vac_d.table_id;CommentShareHi5answered 8 days ago0@AWS Support - I'm wondering why there are so many correlated subqueries in Redshift system views. Converting them to joins will make code easier to read and performant. Also, use of WITH clauses will make codes further more readable.Sample codeCREATE OR REPLACE VIEW <view_name>AS    WITH a AS (SELECT 1 aa) /*Doing task A*/, b AS (SELECT 1 bb) /*Doing task B*/SELECT * FROM a JOIN bON a.aa = b.bb;CommentShareHi5answered 8 days ago"


Question: AWS Compute Savings Plan

"Hi Team,Our customer has purchased $4 compute savings plan. When observed the savings plan coverage, it shows applied 67% for one instance type, 80% for another instance type, 79% for 3rd instance type. My understanding were, it should first cover 100% for one instance type (with max % savings) and then apply for other. But the above observation is confusion. Please someone clarify the doubt.Regards,Nikhil ShahFollowComment"

Answer: "1You are correct, compute savings plans will cover the most expensive compute resource. When you say that these 3 instance types are being covered, are the resources running constantly or could it be that as things scale up/down that the savings plan is covering the most expensive resource for that hour?CommentShareEXPERTShlomo Dubrowinanswered a year agoAWS-User-1576695 a year agoYou mentioned, it will apply to most expensive resource of the hour? I thought it will apply to most discounted resource.Also, does it mean compute savings plan apply hourly and not by end of billing on most discounted resource>?Share0Just to clarify in any hour your Savings Plans apply to your usage starting with the highest discount percentage. [1]* Q. Are Saving Plans applied hourly* A. Yes, they are calculated hourly to the usage that gets the largest discount from the saving plans vs on-demand cost. Q. Why am I not seeing my saving plan applied 100% on one resources?A. Probably because the resources isn't active for that time period or wasn't the resources with the largest discount.Below is the reference to the AWS documentation[1] Understanding how Savings Plans apply to your AWS usage - https://docs.aws.amazon.com/savingsplans/latest/userguide/sp-applying.htmlCommentSharenitro24x7answered 6 months agoShlomo Dubrowin EXPERT3 months agoYou are correct, SPs will apply to the most discounted. Usually, this would be the most expensive instance type, but I can think of times this may not be the case.RIs and SPs are applied hourly and can change every hour depending on the application of the RI/SP according to our documentation (that you linked). You can see the application of RIs and SPs in the CUR as noted in the answer by Brian.Share0Hello, I ran into this same issue a few times! I put together some resources for other AWS users to understand what their Savings Plans are covering (by the hour, if that is what you want to know).For the full rundown of queries, check out this blog post on Savings Plans. For now, here is a sample query you can run against your Cost and Usage Report to see which instance types your Savings plans are being applied to by region. Instance types in different regions can have different pricing, which is another reason you could be seeing the data you're seeing.--Savings plan existence cost--Savings plan existence cost by instance type, regionSELECT   [product/instanceType],  [product/region],    round(sum([savingsPlan/SavingsPlanEffectiveCost]), 4) as existence_costFROM CURWHERE  [lineItem/ProductCode] = 'AmazonEC2'    and [product/instanceType] <> ""    and [lineItem/LineItemType] is 'SavingsPlanCoveredUsage'GROUP BY   [product/instanceType],  [product/region]ORDER BY  sum([savingsPlan/SavingsPlanEffectiveCost]);Best of luck!CommentShareBriananswered 3 months ago"


Question: Download big Oracle DataPump-ImportFile from S3 fails

"Hello,we have a big Oracle DataPump-ImportFile of about 165 GB in a S3-Bucket that we to import into a RDS Oracle 12.1 Database. The download begins but fails after having downloaded about 117 GB.RDS Database trace/alert.log from when it starts to fail:Mon Jul 01 12:09:57 2019WARN: ARCH: Terminating process hung on an operation (PID:16174)Mon Jul 01 12:10:57 2019Killing 1 processes (PIDS:16174) (Process by index) in order to remove hung processes. Requested by OS process 16170Mon Jul 01 12:12:14 2019ARCH: krsv_hung_process_check: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_1Mon Jul 01 12:13:44 2019Thread 1 cannot allocate new log, sequence 57Checkpoint not completeCurrent log# 4 seq# 56 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_4_gf312q1j_.logMon Jul 01 12:14:54 2019Thread 1 advanced to log sequence 57 (LGWR switch)Current log# 1 seq# 57 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_1_gf312p3q_.logMon Jul 01 12:17:31 2019ARC0: Detected ARCH process failureARC0: STARTING ARCH PROCESSESStarting background process ARC1Mon Jul 01 12:17:31 2019ARC1 started with pid=40, OS id=10593 ARC1: Archival startedARC0: STARTING ARCH PROCESSES COMPLETEARC0: Becoming the 'no FAL' ARCHARC0: Becoming the 'no SRL' ARCHMon Jul 01 12:19:31 2019Thread 1 cannot allocate new log, sequence 58Checkpoint not completeCurrent log# 1 seq# 57 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_1_gf312p3q_.logMon Jul 01 12:21:04 2019ORACLE Instance FAB001 - Cannot allocate log, archival requiredMon Jul 01 12:21:04 2019Thread 1 cannot allocate new log, sequence 58All online logs need archivingExamine archive trace files for archiving errorsCurrent log# 1 seq# 57 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_1_gf312p3q_.logMon Jul 01 12:24:14 2019WARN: ARCH: Terminating process hung on an operation (PID:16176)Mon Jul 01 12:25:14 2019Killing 1 processes (PIDS:16176) (Process by index) in order to remove hung processes. Requested by OS process 16170Mon Jul 01 12:34:14 2019WARN: ARCH: Terminating process hung on an operation (PID:16178)Mon Jul 01 12:37:22 2019Killing 1 processes (PIDS:16178) (Process by index) in order to remove hung processes. Requested by OS process 16170Mon Jul 01 12:38:24 2019ARCH: Detected ARCH process failureMon Jul 01 12:42:24 2019WARN: ARCH: Terminating process hung on an operation (PID:16172)Mon Jul 01 12:43:24 2019Killing 1 processes (PIDS:16172) (Process by index) in order to remove hung processes. Requested by OS process 16170Mon Jul 01 12:47:40 2019Archived Log entry 16875 added for thread 1 sequence 54 ID 0x5bfbce85 dest 1:ARC1: Detected ARCH process failureARC1: Detected ARCH process failureARC1: STARTING ARCH PROCESSESStarting background process ARC0Mon Jul 01 12:47:41 2019ARC0 started with pid=25, OS id=13561 Starting background process ARC2Mon Jul 01 12:47:41 2019ARC2 started with pid=26, OS id=13564 Starting background process ARC3Mon Jul 01 12:47:41 2019ARC3 started with pid=27, OS id=13568 ARC0: Archival startedARC2: Archival startedARC3: Archival startedARC1: STARTING ARCH PROCESSES COMPLETEARC1: Becoming the 'no FAL' ARCHARC1: Becoming the 'no SRL' ARCHMon Jul 01 12:47:41 2019ARC0: Becoming the heartbeat ARCHUsing STANDBY_ARCHIVE_DEST parameter default value as /rdsdbdata/db/FAB001_A/arch/redologMon Jul 01 12:47:44 2019Thread 1 advanced to log sequence 58 (LGWR switch)Current log# 2 seq# 58 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_2_gf312pf4_.logMon Jul 01 12:52:51 2019Thread 1 cannot allocate new log, sequence 59Checkpoint not completeCurrent log# 2 seq# 58 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_2_gf312pf4_.logMon Jul 01 12:54:38 2019ORACLE Instance FAB001 - Cannot allocate log, archival requiredMon Jul 01 12:54:38 2019Thread 1 cannot allocate new log, sequence 59All online logs need archivingExamine archive trace files for archiving errorsCurrent log# 2 seq# 58 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_2_gf312pf4_.logMon Jul 01 13:00:04 2019WARN: ARCH: Terminating process hung on an operation (PID:10593)Mon Jul 01 13:02:52 2019Killing 1 processes (PIDS:10593) (Process by index) in order to remove hung processes. Requested by OS process 16170Mon Jul 01 13:04:03 2019ARCH: krsv_hung_process_check: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_1Mon Jul 01 13:09:33 2019WARN: ARCH: Terminating process hung on an operation (PID:13568)Mon Jul 01 13:10:33 2019Killing 1 processes (PIDS:13568) (Process by index) in order to remove hung processes. Requested by OS process 16170Mon Jul 01 13:12:34 2019ARC0: Detected ARCH process failureMon Jul 01 13:12:51 2019ARC0: STARTING ARCH PROCESSESStarting background process ARC1Mon Jul 01 13:12:51 2019ARC2: Becoming the 'no FAL' ARCHARC2: Becoming the 'no SRL' ARCHMon Jul 01 13:12:51 2019ARC1 started with pid=24, OS id=16168 Mon Jul 01 13:12:51 2019ARC1: Archival startedARC0: STARTING ARCH PROCESSES COMPLETEMon Jul 01 13:22:08 2019ARC2: Detected ARCH process failureARC2: STARTING ARCH PROCESSESStarting background process ARC3Mon Jul 01 13:22:08 2019ARC3 started with pid=27, OS id=17139 ARC3: Archival startedARC2: STARTING ARCH PROCESSES COMPLETEMon Jul 01 13:24:33 2019Archived Log entry 16876 added for thread 1 sequence 55 ID 0x5bfbce85 dest 1:Mon Jul 01 13:24:33 2019Archived Log entry 16877 added for thread 1 sequence 56 ID 0x5bfbce85 dest 1:krse_arc_driver_core: Successful archiving of previously failed ORLIs there a limitation on file size or the amount of time this operation is allowed to consume?The log file indicates that something about archiving of log files is going wrong. Is it possible to deactivate those processes temporarily or is it just something else that is going to fail then?Please advice! Thank you.Best Regards,slEdited by: sl-aws-forums on Jul 2, 2019 8:30 AMFollowComment"

Answer: "0Accepted AnswerHi there,The S3_INTEGRATION option for Amazon RDS for Oracle does not have limits on file size or transfer duration. Based on the error information in the alert log it sounds like you may be exceeding I/O limits for the database instance that you are using, so I would suggest first checking CloudWatch metrics to make sure that you have sufficient IOPS and/or network throughput to perform a file transfer of this size.https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.htmlhttps://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MonitoringOverview.htmlFile copying should not itself increase the amount of redo logging generated on your instance, but I/O saturation can cause internal database processes to appear hung. You do have the option of running the database in NOARCHIVELOG mode which can be done by setting backup retention to "0" - note that this will require a brief outage to change the logging mode of the database and will delete existing automated backups - and then setting it back to its initial value after loading is completed (again, will require a brief outage).If you continue to encounter errors please open a case with AWS Support and the RDS team can investigate further.Thanks!MichaelCommentShareAWS-User-9938760answered 4 years ago"


Question: Unable to create Measurement definition in Asset Model properties (IoT_Site_Wise) through AWS Cloud Formation Template

"Resources:AssetModel:Type: AWS::IoTSiteWise::AssetModelProperties:AssetModelName: TestmodelAssetModelDescription: TestmodelAssetModelProperties:- LogicalId: MyLogicalId_for_Accumulator_LevelName: Accumulator LevelDataType: INTEGERDataTypeSpec: Accumulator LevelType:TypeName: MeasurementUnable to create Measurement definition in Asset Model properties (IoT_Site_Wise) through AWS Cloud Formation Template. Tried to create in many ways by changing Logical id, Datatype, Name etc. Please help me to create Measurement definitions. It will be a great help if anyone can provide sample template for Measurement creations.FollowComment"

Answer: "0DataTypeSpec is for when the STRUCT DataType is being used. Since your property is an Integer, you can omit this property.Resources:  AssetModel:    Type: AWS::IoTSiteWise::AssetModel    Properties:      AssetModelName: Testmodel      AssetModelDescription: Testmodel      AssetModelProperties:        - LogicalId: MyLogicalId_for_Accumulator_Level          Name: Accumulator Level          DataType: INTEGER          Type:            TypeName: MeasurementCommentShareAWS-User-7434318answered a year ago"


Question: Amazon AWS hadoop- how to download a file from s3 in map-reduce program in

"I am trying to make a quite simple map-reduce program that as part of the map task it needs to load a small file from S3 (something that can be saved in the local memory).In the map reduce the task is to check each line against this file and generate a feature vector (and so on...)I am having troubles with making my setup method downloading and accessing this file.I think the problem is with forwarding the credentials but it might as well be in the way I am accessing the file in the setup function (I put code snippets below).In my map-reduce logs the error I'm getting is:"profile file cannot be null"about this line:            AWSCredentialsProvider credentialsProvider = new AWSStaticCredentialsProvider(new ProfileCredentialsProvider().getCredentials());I tried many other ways and got nowhere, if there is anyway you can guide me it will be great.In my main (running local on my pc) java file I do:AmazonElasticMapReduce mapReduce =AmazonElasticMapReduceClientBuilder.standard().withRegion("us-east-1").build();HadoopJarStepConfig hadoopJarStep = new HadoopJarStepConfig()                .withJar("MYPATH")  // This should be a full map reduce application.                .withMainClass("MYMAIN");       in the mapreduce jar (the one that runs on the ec2 task) I define the relevant jobs and the job contorl, and in the setup function of the map class I do:AWSCredentialsProvider credentialsProvider = new AWSStaticCredentialsProvider(new ProfileCredentialsProvider().getCredentials());            AmazonS3 s3 = AmazonS3ClientBuilder.standard()                    .withCredentials(credentialsProvider)                    .withRegion("us-east-1")                    .build();            String S3Bucket = "MYBUCKET";  // the bucket where the file is located            String S3Key = "MYKEY"; // the name of the file            S3Object object = s3.getObject(new GetObjectRequest(S3Bucket, S3Key)); // bucket, key            S3ObjectInputStream summaryInputStream = object.getObjectContent();            BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(summaryInputStream));I also tried to hard code my credentials in the main (run local on my pc) like this:String access_key = "aws_access_key_id=MYKEY";        String secret_key = "aws_secret_access_key=MYSECRETKEY";        BasicAWSCredentials creds = new BasicAWSCredentials(access_key, secret_key);        AWSStaticCredentialsProvider awsCred = new AWSStaticCredentialsProvider(creds);        AmazonElasticMapReduce mapReduce =                AmazonElasticMapReduceClientBuilder.standard().withRegion("us-east-1").withCredentials(awsCred).build();also with no success (I tried with excluding the "aws_access_key_id=" and the "aws_secret_access_key=" from the strings)Thanks!FollowComment"

Answer: "0Accepted AnswerYour first issue looks like you are having a credential issue with Java SDK V1.Also - I would recommend moving to V2: https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/javav2.For V1 - simply try to follow this code example to create a bucket so we can focus on your cred issue.https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/java/example_code/s3/src/main/java/aws/example/s3/CreateBucket.javaI recommend placing your credentials here - as discussed in the docs:The default credential profiles file– typically located at ~/.aws/credentials (location can vary per platform), and shared by many of the AWS SDKs and by the AWS CLI. The AWS SDK for Java uses the ProfileCredentialsProvider to load these credentials.Then you can create your S3 Service client like this:AmazonS3 s3 = AmazonS3ClientBuilder.standard().withRegion(Regions.DEFAULT_REGION).build();I just ran this Java V1 example and it created a bucket perfectly.Try this and post back what happened.CommentSharePowerUserScottanswered 3 years ago0Thanks a lot, that did the trickCommentSharenoampaanswered 3 years ago0I am glad it worked for you!!!Edited by: PowerUserScott on Jul 9, 2020 10:24 AMCommentSharePowerUserScottanswered 3 years ago"


Question: BatchWrite limits

"Related a bit to forum https://forums.aws.amazon.com/post!reply.jspa?messageID=821050, but I don't think I should completely hijack that topic.I'm performing the following batch_write operation using Python:{    'DirectoryArn': '<snip>',     'Operations': [        {'CreateObject': {'SchemaFacet': [{'SchemaArn': '<snip>', 'FacetName': 'Organization'}], 'ObjectAttributeList': [{'Key': {'SchemaArn': '<snip>', 'FacetName': 'Organization', 'Name': 'display_name'}, 'Value': {'StringValue': 'Object #1'}}], 'ParentReference': {'Selector': '/'}, 'LinkName': 'object', 'BatchReferenceName': '5bd5059b-b793-4ebe-b45b-b0a136a4ac8d'}},        {'CreateObject': {'SchemaFacet': [{'SchemaArn': '<snip>', 'FacetName': 'Meta'}], 'ObjectAttributeList': [{'Key': {'SchemaArn': '<snip>', 'FacetName': 'Meta', 'Name': 'display_name'}, 'Value': {'StringValue': 'Child #1'}}], 'ParentReference': {'Selector': '#5bd5059b-b793-4ebe-b45b-b0a136a4ac8d'}, 'LinkName': 'organization', 'BatchReferenceName': '01e66be7-d47f-40ed-b7d0-faa638db9ced'}},        {'CreateObject': {'SchemaFacet': [{'SchemaArn': '<snip>', 'FacetName': 'Meta'}], 'ObjectAttributeList': [{'Key': {'SchemaArn': '<snip>', 'FacetName': 'Meta', 'Name': 'display_name'}, 'Value': {'StringValue': 'Child #2'}}], 'ParentReference': {'Selector': '#5bd5059b-b793-4ebe-b45b-b0a136a4ac8d'}, 'LinkName': 'index', 'BatchReferenceName': 'f8b672b9-fef5-40c8-8c55-94152d4db745'}},        {'CreateObject': {'SchemaFacet': [{'SchemaArn': '<snip>', 'FacetName': 'INDEX'}], 'ObjectAttributeList': [], 'ParentReference': {'Selector': '#f8b672b9-fef5-40c8-8c55-94152d4db745'}, 'LinkName': 'index1', 'BatchReferenceName': '71b75012-516b-4cb6-8354-976ad1cc4589'}},        {'CreateObject': {'SchemaFacet': [{'SchemaArn': '<snip>', 'FacetName': 'INDEX'}], 'ObjectAttributeList': [], 'ParentReference': {'Selector': '#f8b672b9-fef5-40c8-8c55-94152d4db745'}, 'LinkName': 'index2', 'BatchReferenceName': 'c6824a61-1338-4cd5-8d3a-553b67e50a18'}},        {'CreateObject': {'SchemaFacet': [{'SchemaArn': '<snip>', 'FacetName': 'INDEX'}], 'ObjectAttributeList': [], 'ParentReference': {'Selector': '#f8b672b9-fef5-40c8-8c55-94152d4db745'}, 'LinkName': 'index3', 'BatchReferenceName': 'c72c0bab-25f1-4a83-a9c5-569d34d1a90d'}},        {'CreateObject': {'SchemaFacet': [{'SchemaArn': '<snip>', 'FacetName': 'INDEX'}], 'ObjectAttributeList': [], 'ParentReference': {'Selector': '#f8b672b9-fef5-40c8-8c55-94152d4db745'}, 'LinkName': 'index4', 'BatchReferenceName': '8b4368e2-2d29-4666-8eb0-4e93105f1efb'}},        {'CreateObject': {'SchemaFacet': [{'SchemaArn': '<snip>', 'FacetName': 'INDEX'}], 'ObjectAttributeList': [], 'ParentReference': {'Selector': '#f8b672b9-fef5-40c8-8c55-94152d4db745'}, 'LinkName': 'index5', 'BatchReferenceName': 'e5ba15a8-4cdd-43a7-873e-0afd8c0e6870'}},        {'CreateObject': {'SchemaFacet': [{'SchemaArn': '<snip>', 'FacetName': 'INDEX'}], 'ObjectAttributeList': [], 'ParentReference': {'Selector': '#f8b672b9-fef5-40c8-8c55-94152d4db745'}, 'LinkName': 'index6', 'BatchReferenceName': 'afba33be-d2a2-461f-8742-ec0cb805b0dc'}},        {'CreateObject': {'SchemaFacet': [{'SchemaArn': '<snip>', 'FacetName': 'INDEX'}], 'ObjectAttributeList': [], 'ParentReference': {'Selector': '#f8b672b9-fef5-40c8-8c55-94152d4db745'}, 'LinkName': 'index7', 'BatchReferenceName': 'c4b70cd5-1a39-45ba-a5ee-efb0fbc424e7'}},        {'CreateObject': {'SchemaFacet': [{'SchemaArn': '<snip>', 'FacetName': 'INDEX'}], 'ObjectAttributeList': [], 'ParentReference': {'Selector': '#f8b672b9-fef5-40c8-8c55-94152d4db745'}, 'LinkName': 'index8', 'BatchReferenceName': '1ac0290c-339f-4b50-9080-d1c5c8d4159a'}}    ]}which fails with the error:An error occurred (LimitExceededException) when calling the BatchWrite operation: Limits exceeded on the request. Write objects, max: 20, actual: 22. Read objects, max: 200, actual: 10. Write attributes, max: 1000, actual: 4. Read attributes, max: 1000, actual: 0.I'm creating 11 objects. The selector paths would look like:/object/object/organization/object/index/object/index/index1/object/index/index2/object/index/index3/object/index/index4/object/index/index5/object/index/index6/object/index/index7/object/index/index8Although I can understand where the 22 Writes come from, I wonder if this is a fair assumption as the object with "/object/index" is for example counted 9 times (1 create, 8 updates). I'm using the BatchReference so during the counting it can be taken into account that I intend to write the same object multiple times, and as it is documented that the batch_write is going to be a single transaction, I would expect my "Write object count" to be 12 (11 objects created + root node update.This comment can also be extended a bit to the Read Objects count of 10, but I'll that one out of the discussion. :)FollowComment"

Answer: "0Apologies, the wording of the error message could be improved. I'll create an item in our internal bug tracker.You are correct that there are 11 Objects created. When the error message says "Write objects, max: 20, actual: 22", it is using the "little o" definition of object (anything in your directory) rather than the "big O" definition of Object (a node in the directory's graph). This can certainly cause confusion. In your case, the additional writes are coming from the creation of the links from "/object/index" to each child. Both Links and Objects count towards this write limit.-MattCommentShareAWS-User-6675827answered 4 years ago0Thanks for the clarification.CommentShareidoornanswered 4 years ago"


Question: What is the difference between Lightsail availability and EC2 availability?

"As of now, I have been using Lightsail-related computing business for several months, but I have found that sometimes my customers cannot connect to the server in recent months, and this time lasts for a short time, and it will be restored in a few minutes.Still, it makes me worry about its usability. After my investigation, when my client can't access the server, he can't ping the target IP address. Does this explain the difference in the availability of EC2 and Lightsail? Or is it an issue with the AWS network? If my business requires high availability of the network, will using more expensive EC2 improve availability?FollowComment"

Answer: 0Hello!The largest available Lightsail virtual machines are not nearly as powerful as the high-end virtual machines available in EC2. Regarding your customers' problems there might be too much traffic to a single virtual machine so it can't handle the load. I would recommend adding additional virtual machines and a load balancer to spread the load between virtual machines. See more info here: https://lightsail.aws.amazon.com/ls/docs/en_us/articles/understanding-lightsail-load-balancersCommentSharemasseanswered 6 months ago


Question: Best way to expose files from EFS over HTTP(S)?

"I have some dynamically-generated files (more context below) stored on EFS and need to expose these files over HTTPS.I'm wondering what the best way to do this would be…I've thought of a few ideas, some might be doable and others might not, I'm curious to see what other people think:Setup a Cloudfront distribution and register my EFS as an Origin. This works fine for S3 but doesn't seem to be possible for EFS :-(Setup some replication mechanism that would upload files to S3 as soon as they are created in EFS. I haven't checked yet if EFS can generate an Event (maybe to EventBridge?) when a file has just been created, but if it can, plugging another Lambda to copy from EFS to S3 would work… Or maybe a managed service would be able to do that for me? (I don't really want to update my code to raise an event when a file has been generated, I'd rather have AWS generate that event automatically)Setup a Cloudfront -> API Gateway -> Lambda that would serve the file from EFS. Executing a lambda to serve a file is not optimal from a "cost" point of view, but  those files could be cached by Cloudfront forever, making this approach OK-ish.Does one of these approaches sound like what you would do?Do you have another idea / recommendation?.More context:The files are created on EFS by a lambda function -- when that Lambda function is called, it downloads an image and generates a thumbnail. That thumbnail is stored, as a not-too-big file, on EFS.If the Lambda was running my own code, I would change it to write the thumbnail to S3 (and set up a Cloudfront distribution to serve the thumbnails over HTTPS, idea #1). But this is not my code and I'm not too fond of modifying it…When a thumbnail is generated, it needs to be available over HTTP quickly (delay of 1-5 seconds is Okay-ish, 1-5 minutes is not OK).After a thumbnail has been generated, it is never updated. And thumbnails are rarely deleted (and keeping old "deleted" thumbnails for even days is OK)Estimates: there will be between one and ten thousands thumbnails on EFS. Total size will be between 1 and 10 GB or so.I expect only a few (a dozen, max) new thumbnails will be generated each day, which means a non-serverless and always-running approach will not be optimal from a "cost" point of view.FollowComment"

Answer: "7As you say, EFS is not ideal for this use case. EFS provides low-latency access to large volumes of information, but only through an otherwise-opaque NFS file system interface. It does not provide an HTTP API for its contents, nor does it produce lifecycle events for files in it.I'd say your best bet is #2, replicating to S3. While it requires more modifications to your existing code, it has fewer moving parts overall and should be lower maintenance. You'll have to hook into your existing thumbnail generation process, but there are a few options for that. Without modifying your thumbnail generation Lambda function, probably the easiest is to add a Lambda destination onto the thumbnail generation function. You'll create a new Lambda function, which will get the input and output of the thumbnail generation function, and can pull the thumbnail from EFS and put it into Lambda.CommentShareben11kehoeanswered 2 years agoEXPERTRaphaelreviewed 2 years ago1AWS DataSyncThis becomes your replication mechanism.In using a managed service to simplify the process of transferring data from an EFS filesystem to an S3 bucket.You can also schedule periodic replication of an EFS file system to an S3 bucket.https://aws.amazon.com/premiumsupport/knowledge-center/datasync-transfer-efs-s3/Note/Cost Optimization - VPC Endpoints for DataSync use AWS PrivateLink, when you use DataSync in a VPC the agent can communicate directly without the need to cross public internet.https://docs.aws.amazon.com/datasync/latest/userguide/datasync-in-vpc.htmlCommentShareNickanswered 2 years ago"


Question: Azure Cloud MFA support for Workspaces

"My customer is using AWS workspaces with Azure MFA server which was implemented following this guide. MFA server is no longer being made available by Microsoft, and is now unavailable for new licensees as per detail here. Instead, Microsoft users are being pushed towards "Azure Cloud MFA”. Is there any way to configure Azure Cloud MFA to work with Workspaces, or any suggested workarounds if this is not possible?FollowComment"

Answer: 0Accepted AnswerYou can use Azure MFA with workspace using NPS extension on RADIUS server .please read the below post for better understanding.https://docs.microsoft.com/en-us/azure/active-directory/authentication/howto-mfa-nps-extensionMFA Server may have been deprecated but it will work the same way with NPS extension running on the Radius server.https://aws.amazon.com/blogs/desktop-and-application-streaming/integrating-microsoft-azure-mfa-server-with-amazon-workspaces/CommentShareAWS-User-7823728answered 3 years ago


Question: Redshift chaos testing

We are planning to redshift chaos testing. Just to know what feasible chaos experients can be  considered for testing. Thanks.FollowComment

Answer: "1At this moment the AWS Fault Injection Simulator does not have direct integration with Redshift. However your chaos testing experiments/scenarios will depends on how your application is interacting with Redshift and what layers of failures you want to test, basically requirements. As an example you can do like simulate connection issues in Amazon Redshift, start and stop etc. but have to do with custom implementation and also do on test env which you can easily re-create.CommentShareEXPERTAWS-User-Nitinanswered 7 months agoEXPERTFabrizio@AWSreviewed 7 months ago"


Question: How to backup server settings and users?

We are writing a BCP for this service and are not finding any documentation for backing up user accounts. Is it possible to export user accounts and import to a new/replacement server?FollowComment

Answer: "0To retrieve list of users created and stored in the service, we recommend using the ListUsers API - https://docs.aws.amazon.com/transfer/latest/userguide/API_ListUsers.html. You can then use the output of this to create users on a new SFTP server via the CreateUsers API. Let us know if this will help meet your requirements for recreating your server for BC purposes.CommentShareAWS-User-9826533answered 4 years ago0ThanksCommentShareFredHueanswered 3 years ago"


Question: EC2 instance inaccessible after RDP change

"My RDP port was changed in the registry of my EC2 Windows Server instance, and now the server is inaccessible after a reboot.  Is there any way to recover the instance to how it was before?  The securitygroup allows the new port and is assigned to the server.FollowComment"

Answer: "0Took some doing, but I spun up a new instance, mounted the dead server's disk, and changed the registry setting back.  Back in business.  Will be doing snapshots in the future. :)CommentShareKyHackanswered 3 years ago"


Question: Stuck on UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS state

how do I completely delete this stack? my stack ID was : arn:aws:cloudformation:ap-northeast-1:234758500005:stack/appsync-qldb-demo-dev/c5b800e0-146b-11ed-b082-0aa7231467b7 it keeps on this state for a while now after I cancel the deployment process manually on AWS CloudFormation console which also stuck on a state when updating the stacks.thank you.FollowComment

Answer: "0Hello,Generally, a CloudFormation stack gets stuck in the UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS state when CloudFormation is still in the process of removing old resources, or can't remove those resources due to a dependency issue.In most cases, CloudFormation wait for resources to delete until it reaches to resource stabilization time before its timeout. Once it will be timed out, stack should move to UPDATE_ROLLBACK_COMPLETE.You can also refer to KC article for some more information and resolutions.Hope above was useful to resolve to the issue.CommentShareSUPPORT ENGINEERSatyam_Ganswered 10 months ago"


Question: Is there a way to automatically set to available from Missed state in amazon connect CCP?

"Is there a way to automatically set to available from Missed state on CCP?FollowCommentrePost-User-1699494 16 days agoHello,To automatically set to available from Missed state on CCP, unfortunately this feature is not available as of now in naive or default CCP in Amazon Connect.As a workaround, you can leverage custom CCP to achieve this requirement:agent.onStateChange()   if agent.state == missed        agent.setState() / agent.setStatus() = AvailableUsing the above method, every time the agent state changes, the CCP will fetch the current state and if the current state is "Missed call", the CCP will change the state back to Available using the agent.setState() / agent.setStatus() method.a)agent.onStateChange()[+]https://github.com/amazon-connect/amazon-connect-streams/blob/master/Documentation.md#agentonstatechangeb)agent.setState() / agent.setStatus()[+]https://github.com/amazon-connect/amazon-connect-streams/blob/master/Documentation.md#agentsetstate--agentsetstatusShare"

Answer: "1I would caution to do this as the call will keep sending to this agent if this person is the only one around but stepped out.  This will also cause a lot of routing decision rules to break if you are checking if there are any agents available to take calls before putting the call in queue.There is a downside to use Stream API if the browser crashed as it changed to missed.Another way to do this is monitoring Agent Event Stream, for Agent State to be MISSED.  A new API has been made to resolve this DismissUserContact:https://docs.aws.amazon.com/connect/latest/APIReference/API_DismissUserContact.htmlin the Agent Event Stream you will get both the contactID and the AgentID which is what you need to set the agent back.Again, not something you should do unless you know the agent condition and environment through some telemetry check of your agent physical status potentially through their headset if it give telemetry data like JabraCommentShareClarenceChoianswered 16 days ago"


Question: AWS EventBridge Customise Output to include EC2 Tag Name

"I've created an EventBridge Rule that notifies me using an SNS email when the Instance State of an EC2 changes, this works fine.I'm using the input transformer to customise the message and want to include the Name of the EC2 as well as the instance ID.  Is this possible as the Name is a Tag ?    The input so far is :{"account":"$.account","instance-id":"$.detail.instance-id","region":"$.region","state":"$.detail.state","time":"$.time"}Can this be tweaked to provide a tag or get the EC2 Name another way ?FollowComment"

Answer: "0You can add more information to the original event  (from producer) and create your own event to be handled by the consumer. Use SDK  DescribeTags with the instanceID from the original event to Instance tags and add it to the custom event. See diagram:                           ┌-----------------Custom event---------------- ┐                           |                                              |  | Producer | ---- >|Event bus|------ Event does not match tags ----> | Lambda Enrich |                           |                           └ --------match tags------>| Consumer| CommentShareIgviranswered a year agoBalu 2 months agoHello @Igvir,I hope you doing well. Could you please elaborate on the diagram of how it works exactly? (I am pretty new to AWS) Did you mean need to create 3 rules? What does the mean Producer/Consumer/Lambda Enrich?Here are my input transformers:{"account":"$.account","instance":"$.detail.instance-id","region":"$.region","state":"$.detail.state","tags":"$.Key|Value","time":"$.time"}It would very helpful If you help me display the "instance name".Thanks in Advance!Share0Will give that a try thank you.CommentShareRobWoodcockanswered a year agoBalu 2 months agoHey @RobWoodcock, Did you find the solution? we are in the same boat!Share"


Question: How to check the default route on private and isolated subnets?

"I am trying to use describe_route_tables on boto3 to describe routes that are going to a particular subnet and going to a particular gateway/network firewall.Furthermore, is there a way to check the default route in a subnet? Using bogo3?FollowComment"

Answer: 1The default route is no different to any other route - there's no special marker for it. Look for a route that has a DestinationCidrBlock of 0.0.0.0/0 - that's the default route.CommentShareEXPERTBrettski-AWSanswered 6 months ago0You can use that boto3 command. You can further use this filter to only grep default routeroute.destination-cidr-blockAccept answer if helped.CommentShareMuhammadanswered 5 months ago


Question: Golang XRAY SDK with redis

"Hi, I am trying to implement golang xray sdk in LAMBDA to capture calls going to redis (AWS managed). Is there a documentation that I can refer for that or any directions to do that.I am using go-redis to makes calls to redis server.FollowComment"

Answer: "0Greetings,Kindly note that currently there is no official documentation for the implementation of the golang xray sdk in LAMBDA to capture calls going to redis. But I did find the relevant documentation for your use case where when your application makes calls to AWS services to store data, write to a queue, or send notifications, the X-Ray SDK for Go tracks the calls downstream in subsegments, but this reference is for Dynamodb ; and the other one is for the Serverless tracking.[+] Tracing AWS SDK calls with the X-Ray SDK for Go :https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-go-awssdkclients.html[+] AWS Lambda and AWS X-Ray :https://docs.aws.amazon.com/xray/latest/devguide/xray-services-lambda.html[+] Tracing serverless application with AWS X-Ray :https://medium.com/nordcloud-engineering/tracing-serverless-application-with-aws-x-ray-2b5e1a9e9447Going ahead, I was also able to find on the aws/aws-xray-sdk-node github link that 'the SDK uses CLS to bind context. Since the SDK doesn't officially support any Redis client, they may or may not work with CLS.' Please refer to the below mentioned links for more information on the same.[+] Redis Support via Lambda :https://github.com/aws/aws-xray-sdk-node/issues/26[+] https://github.com/aws/aws-xray-sdk-node/issues/28Having said that, if need further deep diving or any possible solutions/workarounds for your concern, please reach out to the ElastiCache for Redis team through the AWS support and we will be glad to assist you.[+] Getting started with AWS Support :https://docs.aws.amazon.com/awssupport/latest/user/getting-started.html[+] Creating support cases and case management :https://docs.aws.amazon.com/awssupport/latest/user/case-management.htmlThank you! Have a great day ahead!CommentShareOmkar_Sanswered 10 months ago"


Question: EC2: telnet one instance from another instance

"Hi Guys,I have 2 AWS Instance, I need to telnet Instance 1 to Instance 2, I have added security groups ICMP ALL Ports, but telnet error message getting:  telnet: connect to address XX.XX.XX.XXX: Connection timed outFollowComment"

Answer: "1It sounds like you don't have an ingress rule allowing TCP connections on the right port in any of the instance's (or ENI's) Security Groups. ICMP is a different IP protocol that allows software such as ping to work. But it would not allow a telnet session to work.CommentShareEXPERTMichael_Fanswered a year ago0Most linux distros do not come with telnet server installed. Have you verified the telnet server is running on the instance you want to telnet into? What linux are you using? I'm a bit confused as to which ports are open on "ICMP ALL Ports." Telnet runs on port 23 and is not ICMP traffic. Try adding port 23 to your security group's inbound rule.CommentShareJacksonanswered a year ago0Is there a specific purpose for trying to use Telnet?It has been considered bad practice to use insecure protocols like Telnet for a long time now because they transmit plain text over the network.  I found an AWS article from 2008 specifically recommending use of SSH over Telnet.SSH is secure by design and comes pre-installed/running on nearly every flavour of Linux.  Another advantage is the ability to use SSH Keys for authentication saving the need for/admin of passwords.CommentShareDaniel Craigieanswered a year ago0Add an ingress TCP rule for the port you are trying to telnet into. For e.g. to telnet to port 8080, add TCP inbound 8080 to the Incoming Security Group of the server you're telneting into. Then from the source server type:telnet <target server IP> 8080CommentShareViper25answered a year ago"


Question: FQDN value in strust for SAP Fiori working with Route 53

"The customer wants to keep custom FQDN for DEV & QAS when lifting to AWS, and they need to create certs in strust for access via Fiori. In this case, do we put the custom FQDN or Route 53's FQDN when creating the cert in strust? I assume that it should just stay as is, because the connection to the DEV/QAS instances is resolved via private hosted zone of Route 53, where the custom FQDN is registered. Would you help confirm?FollowComment"

Answer: "0Hi Ash, yes, customer can choose to keep custom FQDN. The cert in STRUST should then follow this custom FQDN.CommentShareferrymulanswered 13 days ago"


Question: PSI exam didnot load

My secure browser session didnot launch i tried for more than one hour.i had sent you chat within 5-10 mins of my exam start that my secure browser is not launching.My system had passed all the system checks and my connectivity was also good.Its been more than one hour of me trying to launch the exam but i dont know it dint launch the secure browser.I sent mail to psi exam support attaching a video that i prepared when i was trying to launch the secure browser but nothing was happening after several attempts.This exam was very important for me and i studied so hard and now i am just helpless as I didnot get even an attempt to take the test after paying so much and following every instruction that was provided before exam.FollowComment

Answer: "0Please submit a ticket to the AWS Training & Certification customer support team, and they will provide assistance. If you already submitted a case, please allow at least 48 hrs to get a response.CommentShareNelliT-AWSanswered 3 months ago"


Question: AWS S3 Cross Replication - FAILED replication status for prefix

"Hi there,We are utilizing cross-region replication to replicate a large bucket with tens of millions of objects in it to another AWS account for backup purposes.Originally, we had configured the replication rules to replicate the entire bucket. However, we recently noticed that some objects were not being replicated to the destination bucket in the backup account and appeared with a replication status of FAILED (screenshot: https://pasteboard.co/IgdSZf3.png).Thinking that perhaps the bucket had too many objects in it and that perhaps CRR was not capable of reliably replicating an entire bucket with that many objects, we created multiple replication rules at the prefix level (i.e. instead of one CRR rule for "bucket-name" we created ~10 for each "subfolder" prefix in the bucket, eg "bucket-name/subfolder1" "bucket-name/subfolder2" "bucket-name/subfolder3"). After doing so, and doing some testing, I have noticed that replication is working fine in all but one of the bucket prefixes. Objects under this prefix can't be replicated, and the replication status shows as FAILED for each new object added to the bucket. This particular prefix has a lot of objects under it.Replication is working for this bucket for certain prefixes, so it's obviously not a policy or permissions issue. What else can we do to troubleshoot this so we can get CRR working reliably for this bucket?FollowComment"

Answer: "0Hi,Do you have a support plan with AWS? You would need to submit a support ticket and provide couple of keys that failed replication so that we could check and see exactly what happened.Regards,YuriyAWS S3CommentShareyzubarev-s3answered 4 years ago0Thanks, we're not on a paid support plan at the moment, but fortunately, I don't think we'll need to create a ticket for this. I believe I have FINALLY figured this out.This issue appears to have been caused by public access settings on the destination bucket. The objects in the problematic "subfolder" in the source bucket are public (which is intentional), but are not supposed to be public in the destination bucket. When I disabled the "Block public access to buckets and objects granted through new access control lists (ACLs)" setting, replication started working (screenshot: https://pasteboard.co/Ii4SQJD.png). Reference from AWS docs on what I'm talking about: https://docs.aws.amazon.com/AmazonS3/latest/user-guide/block-public-access-bucket.html. I believe these settings were somewhat recently introduced.I've confirmed that the ACL on the replicated object does not, in fact, grant public read access to the object in the destination bucket once it has been replicated. There must be some process that takes place behind the scenes that copies over the original ACL for the object (which the public access policy must block, since it sees it as a "new" ACL that grants public read permission) and then changes the ownership of the object to the destination bucket, probably by changing the ACL.CommentSharegbdananswered 4 years ago0Marked as "answered" - see above.Edited by: gbdan on Jun 5, 2019 5:16 PMCommentSharegbdananswered 4 years ago"


Question: Can't update environment variables in elastic beanstalk environment

"Have an elastic beanstalk environment that's a clone of a different environment that can't do cloudwatch log streaming for some reason but that's another story.I'm trying to update this environment's configuration. Specifically, I'm trying to toggle an environment variable value from "true" to "false". When I do this in the web console, this is the output:2022-01-05 14:26:57 UTC-0500INFOThe environment was reverted to the previous configuration setting.2022-01-05 14:26:24 UTC-0500ERRORDuring an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.2022-01-05 14:26:24 UTC-0500ERRORFailed to deploy configuration.2022-01-05 14:26:23 UTC-0500ERRORUnsuccessful command execution on instance id(s) 'i-077df8c5051489285'. Aborting the operation.2022-01-05 14:26:23 UTC-0500INFOCommand execution completed on all instances. Summary: [Successful: 0, Failed: 1].2022-01-05 14:26:23 UTC-0500ERROR[Instance: i-077df8c5051489285] Command failed on instance. An unexpected error has occurred [ErrorCode: 0000000001].2022-01-05 14:25:18 UTC-0500INFOUpdating environment api-prod-clone's configuration settings.2022-01-05 14:25:09 UTC-0500INFOEnvironment update is starting.Have tried this about a half dozen times now, the failure is the same every time. The single instance in this environment winds up degraded and I have to terminate it to get the environment back to OK status.Figuring maybe the web console was just broken, I next tried the eb CLI.eb setenv DISABLE_CONSOLE_LOGS=false -e api-prod-clone2022-01-05 19:25:09    INFO    Environment update is starting.      2022-01-05 19:25:18    INFO    Updating environment api-prod-clone's configuration settings.2022-01-05 19:26:23    ERROR   [Instance: i-077df8c5051489285] Command failed on instance. An unexpected error has occurred [ErrorCode: 0000000001].2022-01-05 19:26:23    INFO    Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].2022-01-05 19:26:23    ERROR   Unsuccessful command execution on instance id(s) 'i-077df8c5051489285'. Aborting the operation.2022-01-05 19:26:24    ERROR   Failed to deploy configuration.                                                                            ERROR: ServiceError - Failed to deploy configuration.So, I give up. How do you modify the environment variables for an eb environment?FollowComment"

Answer: 0EC2 instances in your Elastic Beanstalk environment generate logs that you can view to troubleshoot issues with your application or configuration files. You can easily retrieve them by using the environment management console or the EB CLI.Via the console you can pull the last 100 lines of the most commonly used log files. The ones probably of most interest will be eb-activity.log and eb-commandprocessor.log as these logs contain messages about deployment activities and particular stack traces associated with errors during deployment.Its possible there is a more detailed error message which may help to understand what the issue is.CommentShareRoBanswered a year ago0I have also noticed that after setting up an environment (cloning or from scratch) it says it's "ready" sooner then it really is.I found I have to wait another 10-20 minutes sometimes after the env says "ready" before I can change the Configurations or Deploy to it.CommentSharemehiattanswered a year ago


Question: How can customers find out the S3 bucket with the highest cost or sort the cost for each S3 bucket?

How can customers find out the S3 bucket with the highest cost or sort the cost for each S3 bucket?A customer is currently going through cost optimization and they want to find out the S3 bucket with the highest cost.  They are also looking for a way to sort the cost for the S3 buckets.Thanks for any information!FollowComment

Answer: "1Accepted AnswerHi, does this help?https://aws.amazon.com/premiumsupport/knowledge-center/s3-find-bucket-cost/CommentShareAWS-User-2028759answered 3 years agoEXPERTNataliya_Greviewed 2 years ago"


Question: AアカウントからBアカウントにS3バケットのデータを移行する際、最小コストで実施する方法とは？

"I'm thinking of migrating S3 bucket data using the following method.-Create an AWS CLI user at the migration source, create an S3 bucket at the migration destination, and attach a bucket policy.-Execute the CLI command from the migration source to copy the objects in the bucket.In that case, the cost will be about 50,000 yen because the bucket x 4 and the total data amount: 4.0TB.Please let me know if there is a way to migrate with minimal cost.FollowComment"

Answer: "0HelloYou use the AWS Data Sync to migrate the objects, Here is the link https://aws.amazon.com/blogs/storage/how-to-use-aws-datasync-to-migrate-data-between-amazon-s3-buckets/ThanksGKCommentShareGKanswered a year agoAWS-User-5680610 a year agoThank u！Check and try.Share"


Question: Unable to delete RDS Multi AZ deployment with status Incompatible-network

"The RDS DB Cluster is a MySQL Multi-AZ DB Cluster with 3 instances (db.m5d.large) that currently are in the Incompatible-network state, with the Cluster state as Backing-up. AWS-Nuke was run to cleanup the resources but due to deletion protection being enabled the RDS was not deleted but somehow the VPC attached to the RDS was cleaned up which let to the Incompatible-network state.And now in this state no modify operation can be done, and when deleting the cluster (Not creating a final snapshot). We get the deleting message for some time which changes to "We're sorry, your request to delete DB cluster database-1 has failed".I don't need to recover the the db, just need to delete it and its costing me a lot as of now, any help would be appreciated, have tried all the available solutions but most ways to get the instances out of the Incompatible-network state are requiring to modify the cluster, which it doesn't allow. Also its stuck at creating a snapshot at 0% from many hours, which most probably is not letting us modify the cluster.FollowComment"

Answer: 0I would suggest you to raise AWS Support Case as you may hit a bug in RDS service. You may not be able to delete that RDS database by yourself and help from AWS RDS internal team is required.CommentShareMaciej Malekanswered 5 months ago


Question: S3 hosted website through Cloudfront -  302 (temporary redirect) response for objects

My customer is looking to have HTTP 302 (temporary redirects) for objects in S3. Our APIs allows to setup 301 redirects. Is there a way to achieve 302 redirects?FollowComment

Answer: 0Accepted AnswerYou can specify the response code using HttpRedirectCode in routing rules.http://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.htmlCommentShareEXPERTLeeAtk@AWSanswered 6 years ago


Question: Failure and Disaster Prevention in Aurora PostgreSQL

"Failure and Disaster Prevention in Aurora PostgreSQLWe had assumed cross-region replication as a countermeasure method, but learned that it is not supported by Aurora PostgreSQL.How to implement cross-region replication even though it is not supportedAn alternative implementation that would satisfy almost the same requirements as cross-region replication.Please tell us either or both of the above.FollowComment"

Answer: "1Accepted AnswerDear Customer, Adding to the response already provided - there are some ways you can achieve thisFirstly please allow me to inform you that when it comes to replication with Aurora PostgreSQL we have the following options:PostgreSQL logical replication using the pglogical extensionPhysical replication with Aurora global databasePostgreSQL native logical replicationSame has been mentioned in details in below AWS blog.[+] https://aws.amazon.com/blogs/database/understand-replication-capabilities-in-amazon-aurora-postgresql/Also please note that Aurora PostgreSQL DB clusters don't support Aurora Replicas in different AWS Regions, so we can't use Aurora Replicas for cross-Region replication.PostgreSQL native logical replication:Now you asked if for PostgreSQL native logical replication if we have any AWS managed service to set up the same then please allow me to inform you that unfortunately we don't have anything like that. If we want to enable native PostgreSQL logical replication then there are certain steps that we need to follow.Below are the two links which talks about native PostgreSQL logical replication in detail.[+] Using PostgreSQL logical replication with Aurora - https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Replication.Logical.html[+] https://aws.amazon.com/blogs/database/using-logical-replication-to-replicate-managed-amazon-rds-for-postgresql-and-amazon-aurora-to-self-managed-postgresql/PostgreSQL logical replication using the pglogical extension:If we talk about PostgreSQL logical replication using the pglogical extension then you can go through below AWS blog which talks about the same with step by step instructions.[+] https://aws.amazon.com/blogs/database/part-2-upgrade-your-amazon-rds-for-postgresql-database-using-the-pglogical-extension/Physical replication with Aurora global database:We can't use Aurora Replicas for cross-Region replication but we have a concept of Aurora global database.AWS Aurora global databases span multiple AWS Regions, enabling low latency global reads and providing fast recovery from the rare outage that might affect an entire AWS Region. An Aurora global database has a primary DB cluster in one Region, and up to five secondary DB clusters in different Regions.An Aurora global database consists of one primary AWS Region where your data is written, and up to five read-only secondary AWS Regions. You issue write operations directly to the primary DB cluster in the primary AWS Region. Aurora replicates data to the secondary AWS Regions using dedicated infrastructure, with latency typically under a second.Please go through below AWS documentation which talks about "Using Amazon Aurora global databases" in detail.[+] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.htmlAWS Database Migration Service (AWS DMS):In the end you can also leverage AWS DMS service to set up replication between two Aurora PostgreSQL cluster both in different region.AWS  Database Migration Service (AWS DMS) is a cloud service that makes it easy to migrate relational databases, data warehouses, NoSQL databases, and other types of data stores. You can use AWS DMS to migrate your data into the AWS Cloud or between combinations of cloud and on-premises setups.With AWS DMS, you can perform one-time migrations, and you can replicate ongoing changes to keep sources and targets in sync.Below are the some documentation links for AWS DMS.[+] What is AWS Database Migration Service? - https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html[+] Using a PostgreSQL database as an AWS DMS source - https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html[+] Using a PostgreSQL database as a target for AWS Database Migration Service - https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.htmlLimitations:[+]  Limitations on using PostgreSQL as a target for AWS Database Migration Service - https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.PostgreSQL.html#CHAP_Target.PostgreSQL.LimitaTIONSCommentShareEXPERTNIRUPAM_danswered 10 months ago1Hello User,if I understood correctly, you want to achieve a cross-region resiliency for Aurora pg.While a "normal" Aurora pg cluster doesn't allow this (as you've already mentioned), Aurora Global Database should provide the level of disaster resiliency you're looking for.A concise overview is given on our AWS Database Blog.We also have a hands-on tutorial to get you started.If you want to read upon a practical example, check out the GS Developer Blog.Does this satisfy your requirements? Then I'd appreciate if you mark my answer as "accepted".Any questions left? Feel free to reach out via the comment section.Kind regards, UweCommentShareUwe  Kanswered 10 months ago"


Question: MIgrating from self-managed Cassandra to Keyspaces

Running a Cassandra cluster on Kubernetes and is considering a move to Keyspaces.It is unclear on how to export the data from the existing cluster and import it into Keyspaces?Would they just use cqlsh COPY?FollowComment

Answer: "0Accepted AnswerYes,  use cqlsh:https://aws.amazon.com/blogs/database/loading-data-into-amazon-mcs-with-cqlsh/At the moment, doesn't look like Keyspaces is a supported target for DMS.CommentShareAWS-User-9164621answered 3 years ago0Yes, but is not the only option.for small datasets or tables you can load data into Amazon Keyspaces using cqlsh.for production workloads with large datasets, you can use the DataStax Bulk Loader for Apache Cassandra to load data into Amazon Keyspaces using the dsbulk command.CommentSharerafamsanswered a year ago"


Question: Access Denied error when accessing Cloudfront url image link

"Hi,All of a sudden our site wasn't saving images to the S3 bucket and also getting Access Denied from Cloudfront. We have a WordPress site and aren't using Plugins to save to our S3 bucket. Does Amazon change permissions over time? As its a very strange issue that images just stopped uploading..FollowComment"

Answer: "0Hello -I understand your experiencing an access denied error unexpectedly when accessing CloudFront URL image links and S3 buckets.  This article might help you troubleshoot the issue. https://aws.amazon.com/premiumsupport/knowledge-center/s3-website-cloudfront-error-403/ Steps you could possibly check:1. Confirm that the objects in the bucket are not encrypted by AWS KMS2. Confirm that there is no explicit "Deny" in the bucket policy for the s3:GetObject action3. Confirm that Amazon s3 block public access is turned off for the bucket4. Confirm that objects in the bucket are publicly accessible5. If you're using a Referer header to restrict access from CloudFront to your S3 origin, then review the custom header6. Confirm that there's no explicit "Deny" Service control policies (SCPs) attached by your organization's management account I hope that helps you troubleshoot the access denied error.CommentShareamandahyanswered 10 months agoSUPPORT ENGINEERAjinkya B-AWSreviewed 10 months ago"


Question: Rename RDS MSSQL physical files after restore

"Hi, I recently migrated a database to RDS MSSQL (i.e. - restored from *.bak file) and want to change the names of the physical files.  When I executed "ALTER DATABASE [db_name] MODIFY FILE (Name=[logical_name], FileName=[full_file_path])" command I received this message from my RDS instance: "Modifying physical file name is not supported on RDS Instances. Modify reverted".  Has anyone seen this message before and if so, what should I do so I can rename the files?FollowComment"

Answer: "0The great thing about RDS is that it's a managed service.  This means that under the shared responsibility model AWS takes care of all management tasks for the DBMS leaving the customer responsible for optimising the application to suit their needs.In your case, the downside of this amount of AWS control is a limitation of the commands that can be executed.  You have the ability to perform high level "standard tasks" (naming DB, Tables, Indexes, Procedures, etc) but loose control of the underlying OS and file management.I fully understand that some people require/insist on having full control over their DBMS and in these situations you would probably have to look at running your DBMS directly on an EC2 instance, however, I strongly advise weighing up the pros/cons for realising this functionality. Especially the extra costs that can come with appropriate instance types/block storage/networking.I sleep very well at night no longer having to worry about managing replication/backups/uptime/etc as all of that is taken care of by AWS.CommentShareDaniel Craigieanswered a year ago"


Question: Uploading a Dataframe to AWS S3 Bucket from SageMaker

"After successfully uploading CSV files from S3 to SageMaker notebook instance, I am stuck on doing the reverse.I have a dataframe and want to upload that to S3 Bucket as CSV or JSON. The code that I have is below:bucket='bucketname'data_key = 'test.csv'data_location = 's3://{}/{}'.format(bucket, data_key)df.to_csv(data_location)I assumed since I successfully used pd.read_csv() while loading, using df.to_csv() would also work but it didn't. Probably it is generating error because this way I cannot pick the privacy options while uploading a file manually to S3. Is there a way to upload the data to S3 from SageMaker?FollowComment"

Answer: "0Accepted AnswerOne way to solve this would be to save the CSV to the local storage on the SageMaker notebook instance, and then use the S3 API's via boto3 to upload the file as an s3 object. S3 docs for upload_file() available here.Note, you'll need to ensure that your SageMaker hosted notebook instance has proper ReadWrite permissions in its IAM role, otherwise you'll receive a permissions error.code you already have, saving the file locally to whatever directory you wishfile_name = "mydata.csv"df.to_csv(file_name)instantiate S3 client and upload to s3import boto3s3 = boto3.resource('s3')s3.meta.client.upload_file(file_name, 'YOUR_S3_BUCKET_NAME', 'DESIRED_S3_OBJECT_NAME')Alternatively, upload_fileobj() may help for parallelizing as a multi-part upload.CommentShareharshmanvaranswered 4 years ago"


Question: AWS IoT 1-Click is not updating wifi on new Seeed Button

"Hi there,I have recently purchased the Seeed IoT Button For AWS but I cannot find a way to update the button’s wifi configuration with the AWS IoT 1-Click app. On an iOS devices, when I get to the “Choose the Wi-Fi Network” screen I can enter network details but I cannot press the “Confirm” button to progress to the next screen. The button flashes like it has been pressed but nothing happens. On an Android device I can click the “Confirm” button but I get a “Wifi Error: Check for Wifi Username and Pwd” message. I have tried different networks and verified the passwords so I am confused about this error.I was able to setup a new wireless network with the SSID of “TE_TEST” with a password of “123456789” and the button was able to connect to that. Using that network I was able to do a firmware update and claim the button in AWS IoT 1-Click web console. Unfortunately the firmware update did not fix my issue and I am still not able to update the wifi configuration on this button.Does someone know what I can do to update the wifi configuration on this devices? Thanks for your help!JordanFollowComment"


Question: Migrating table structures and data

"I'm working on a DynamoDB solution that will need to be migrated to a customer's AWS instance at some point. I have a table that I would like to script out so I can use that to create the table on the customer's instance using AWS CLI. I can use describe-table to create a script, so surely there's a way to use it to create the table, but create-table doesn't take a file as a parameter?Likewise, with the data, you can create a json file of all records, but it's in a different format to the file you specify in the --request-items parameter of batch-write-item. So, how do you migrate data from one instance of a table to another?FollowComment"

Answer: "0Migrate the DynamoDB table data by exporting the table to an Amazon S3 bucket in the target account (https://aws.amazon.com/pt/premiumsupport/knowledge-center/back-up-dynamodb-s3/) . Be sure that DynamoDB has s3:ListBucket permissions to this S3 bucket. Be sure that the S3 bucket doesn't have any access control lists that deny access to the exported data.After the export is completed, import the data from the S3 bucket into a new table in the target account. https://aws.amazon.com/pt/blogs/database/amazon-dynamodb-can-now-import-amazon-s3-data-into-a-new-table/CommentShareEXPERTDaniel ABIBanswered 5 months agoNick 5 months agothanks, will try thatShare"


Question: AdminGetUser method does not return all fields in the list of UserAttrubutes

"Hello I'm trying to get user information by userid but it does not return all list of UserAttributesTrying to get using this method const user = await this.cognito      .adminGetUser({        ...this.defaultParams,        Username: userId,      })      .promise();Actual Response{  "Username": "XXXXXXXXXXXXXXXXXXXX",  "UserAttributes": [    { "Name": "sub", "Value": "Any" },    { "Name": "email_verified", "Value": "true" },    { "Name": "email", "Value": "abc@gmail.com" }  ]}Expected Response{  "Username": "XXXXXXXXXXXXXXXXXXXX",  "UserAttributes": [    { "Name": "sub", "Value": "Any" },    { "Name": "email_verified", "Value": "true" },    { "Name": "email", "Value": "abc@gmail.com" },      { "Name": "name", "Value": "first name" } ,    { "Name": "family_name", "Value": "last name" }  ]}Could someone please let me know why AdminGetUser does not return name and family_name in the list of UserAttributes?FollowCommentAmplify_lover 10 months agoHey bro can you share code snippet please I'm in trouble Please help please.Share"


Question: Changing data retention settings has no affect

"Hello!I was updating the data retention settings of a table within my Timeseries DB, and even though I can see settings have been updated in the console, I can see that the table still operates using the old settings.I know this because if I were to send a new telemetry entry that's older than 6hrs (which was the default memoryStoreRetentionPeriodInHours), I would get an error.So, that means the new memoryStoreRetentionPeriodInHours hasn't actually been applied - even though the console says so.Hope I've managed to explain my situation. Happy to share more info if needed.FollowComment"

Answer: "1Hi,You may be running into the behavior described in the note on this page: https://docs.aws.amazon.com/timestream/latest/developerguide/storage.html The retention change will take effect from the point that you make the change onward. If you were to update your memory store retention period from 6hrs to say 24hrs, it would take 18hrs for the memory store to reflect the full 24hr window. Does this match the behavior you see?Thanks for your feedback on Amazon Timestream!CommentShareAudrey Lanswered a year agoAdrian a year agoThank you for the response and sorry for the delay on my end.I kinda understand your answer, but, on the other hand, still not sure if that's it.So, it seems the doc says that, if I were to change the memoryStoreRetentionPeriodInHours, to say, 7 days, then I'd need to wait 7 days in order to be able to insert records that are 7 days old? Is that the case?Thank you!P.S. Luckily we're still not live with our project, so I was able to circumvent my issue by just destroying/deploying the same table, this time with new settings.Share"


Question: Amazon Transcribe Creating a custom vocabulary fail at line 2

Here is my process:read https://docs.aws.amazon.com/transcribe/latest/dg/custom-vocabulary-create.html#create-vocabulary-tableused https://tophonetics.com/ to convert my textbuild file in Google sheetdownload as TSVedit in Ultra edit change the line terminator on save to LFupload to S3 bucketnavigate  to create custom vocabupload from s3Validation error: File contains invalid characters or format in the IPA column. Error at line 2.Which of these is invalid?siːˈaɪˈɛsˈɛsˈpiːI placed the [tab] below for readability.Phrase[tab]IPA[tab]DisplayAs[tab]SoundsLikeCISSP[tab]ˈsiːˈaɪˈɛsˈɛsˈpiː[tab]CISSP[tab]biometric[tab]ˌbaɪoˈmɛtrɪk[tab]biometric[tab]SDLC[tab]ˈɛsˈdiːˈɛlˈsiː[tab]SDLC[tab]ISC squared[tab]aɪ-ɛs-siː skweədtab2[tab]gonna[tab]ˈgɑnə[tab]going to[tab]FollowComment

Answer: "0Dean, check this outhttps://forums.aws.amazon.com/thread.jspa?threadID=318735we are researching this issue as well..CommentShareFeranswered a year ago"


Question: Remove user id 1000

"Hi ladies/gentlemen,I am in curious trouble here. We have a system that is installed by script it mandatory requires exclusivity over id 1000 on /etc/passwd.Yes, you understand right, I have to remove  ec2-user from my linux box.I know, that I just need to run "deluser ec2-user" and "voi lá", but which impacts it will cause me on the future?Is it safe, or it better to ask to mantainers to change their software?Regards,FollowComment"

Answer: "2Accepted AnswerYou are free to repurpose the default ec2-user account as you wish. If you would like to create a different user ID for the purpose of logging in, you can do so. You'll just need to ensure that the user's home directory is set up and has the appropriate public ssh keys listed in its $HOME/.ssh/authorized-keys file.In fact, if you run AWS Systems Manager Session Manager on your EC2 instance, you don't need to use ssh to log in at all; you can use Session Manager to log in. Session Manager uses a different user ID for logins (ssm-user, id 1001).CommentShareEXPERTMichael_Fanswered a year agoAnotherBrickIntheWall a year agoThanks @Michael_F, this is what I suspected.Share"