Question: RDS: Long_query_time logging works curious

"Hello. I have RDS (8.0.mysql_aurora.3.02.0) instance. I modify it by turning ON the "Slow query log" feature to see all slow requests in my CloudWatch. Also, I attached a custom parameter group with slow_query_log = 1, log_output = FILE and long_query_time = 0.3. I send some requests to see how it works.Using MySQL workbench I see request time like: Duration = 0,468 sec and Fetch = 0,620 sec. This request doesn't register in CloudWatch. So I created more powerful request to see even 1 row in Cloudwatch and I reach my aim: Duration = 0,376 sec and Fetch = 0,923 sec and in the CloudWatch I saw next request time: Query_time: 0.830316 Lock_time: 0.000123. Using different SQL requests I can't rich CloudWatch measure approx 0.3 sec. The minimal registered request was 0.7 sec.So I have questions:How to measure request time in AWS RDS?Why request times are different between MySQL Workbench and CloudWatch records?FollowComment"

Answer: "1Accepted AnswerIn MySQLWorkBench,Duration time - is the time that query needs to be executed. You should try to minimize it when optimizing performance of sql query.Fetch time - measures how long transferring fetched results take, which has nothing to do with query execution. I would not consider it as sql query debugging/optimization option since fetch time depends on network connection, which itself does not have anything to do with query optimization. If fetch time is bottleneck then more likely there's some networking problem.Note: fetch time may vary on each query execution because sometimes results are cached, so they can be sent out quicker.Please refer below forum for more information on Duration and Fetch times in MySQLWorkBench. We do have several third party articles explaining the same.[+] https://forums.mysql.com/read.php?152,560430,560430#msg-560430When it comes to MySQL Slow Query log,[+] https://dev.mysql.com/doc/refman/8.0/en/slow-query-log.htmlQuery_time: durationThe statement execution time in seconds.Lock_time: durationThe time to acquire locks in seconds.I have performed simple testing with below setting from MySQL Client and MySQL Workbench, however I do not observe any differences in terms of execution time in SlowQuery logs in console and SlowQuery logs in CloudWatch.Version: 8.0.mysql_aurora.3.02.0slow_query_log = 1log_output = FILElong_query_time = 0.3From MySQL Terminal:mysql> select sleep(10);+-----------+| sleep(10) |+-----------+|         0 |+-----------+1 row in set (10.00 sec)mysql> select sleep(1);+----------+| sleep(1) |+----------+|        0 |+----------+1 row in set (1.01 sec)mysql> select sleep(0.5);+------------+| sleep(0.5) |+------------+|          0 |+------------+1 row in set (0.50 sec)mysql> select sleep(0.2);+------------+| sleep(0.2) |+------------+|          0 |+------------+1 row in set (0.20 sec)mysql> select sleep(0.8);+------------+| sleep(0.8) |+------------+|          0 |+------------+1 row in set (0.80 sec)MySQL SlowQuery Log in Console:Time Id Command ArgumentTime: 2022-10-06T05:16:48.858424ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 14Query_time: 10.000159 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1use test;SET timestamp=1665033398;select sleep(10);Time: 2022-10-06T05:17:12.159499ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 14Query_time: 1.000166 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1SET timestamp=1665033431;select sleep(1);Time: 2022-10-06T05:17:34.905054ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 14Query_time: 0.500164 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1SET timestamp=1665033454;select sleep(0.5);Time: 2022-10-06T05:19:21.209179ZUser@Host: admin[admin] @ [172.x1.xx.x] Id: 14Query_time: 0.800173 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1SET timestamp=1665033560;select sleep(0.8);----------------------- END OF LOG ———————————CloudWatch logs:Time: 2022-10-06T05:16:48.858424ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 14Query_time: 10.000159 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1use test;SET timestamp=1665033398;select sleep(10);Time: 2022-10-06T05:17:12.159499ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 14Query_time: 1.000166 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1SET timestamp=1665033431;select sleep(1);Time: 2022-10-06T05:17:34.905054ZUser@Host: admin[admin] @ [172.x1.xx.x] Id: 14Query_time: 0.500164 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1SET timestamp=1665033454;select sleep(0.5);Time: 2022-10-06T05:19:21.209179ZUser@Host: admin[admin] @ [172.x1.xx.x] Id: 14Query_time: 0.800173 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1SET timestamp=1665033560;select sleep(0.8);From MySQLWorkBench:06:27:18select sleep(10) LIMIT 0, 10001 row(s) returned10.000 sec / 0.000 sec06:28:19select sleep(1) LIMIT 0, 10001 row(s) returned1.000 sec / 0.000 sec06:29:49select sleep(0.5) LIMIT 0, 10001 row(s) returned0.500 sec / 0.000 sec06:30:08select sleep(0.2) LIMIT 0, 10001 row(s) returned0.203 sec / 0.000 sec06:30:34select sleep(0.8) LIMIT 0, 10001 row(s) returned0.797 sec / 0.000 secMySQL SlowQuery Log in Console:Time Id Command ArgumentTime: 2022-10-06T06:27:28.822387ZUser@Host: admin[admin] @ [172.x1.xx.xxx] Id: 30Query_time: 10.000138 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1use test;SET timestamp=1665037638;select sleep(10)LIMIT 0, 1000;Time: 2022-10-06T06:28:20.709416ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 30Query_time: 1.000203 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1SET timestamp=1665037699;select sleep(1)LIMIT 0, 1000;Time: 2022-10-06T06:29:50.340710ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 30Query_time: 0.500165 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1SET timestamp=1665037789;select sleep(0.5)LIMIT 0, 1000;/rdsdbbin/oscar/bin/mysqld, Version: 8.0.23 (Source distribution). started with:Tcp port: 3306 Unix socket: /tmp/mysql.sockTime Id Command ArgumentTime: 2022-10-06T06:30:35.104460ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 30Query_time: 0.800165 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1use test;SET timestamp=1665037834;select sleep(0.8)LIMIT 0, 1000;----------------------- END OF LOG ----------------------CloudWatch logs:Time: 2022-10-06T06:27:28.822387ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 30Query_time: 10.000138 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1use test;SET timestamp=1665037638;select sleep(10)LIMIT 0, 1000;Time: 2022-10-06T06:28:20.709416ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 30Query_time: 1.000203 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1SET timestamp=1665037699;select sleep(1)LIMIT 0, 1000;Time: 2022-10-06T06:29:50.340710ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 30Query_time: 0.500165 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1SET timestamp=1665037789;select sleep(0.5)LIMIT 0, 1000;Time: 2022-10-06T06:30:35.104460ZUser@Host: admin[admin] @ [172.x1.xx.xx] Id: 30Query_time: 0.800165 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1use test;SET timestamp=1665037834;select sleep(0.8)LIMIT 0, 1000;CommentShareSUPPORT ENGINEERawsdinakarananswered 8 months agoKonstantinZ 8 months agoThank you! I tested it through Workbench via a graphical interface. In this case, executing time on the Workbench side is too long, but in Cloudwatch all expected requests were fixed.Share"


Question: Redshift Datashare: Grant CREATE DATASHARE permissions to users

"Hi,Is it possible to grant users permission to create a datashare?Running command GRANT CREATE DATASHARE TO user;errors out with:Grant/Revoke system privilege on User is not supported.I see that its possible to grant it on a role level, but I'm hoping it's possible to provision it on a user level.Thanks!FollowComment"

Answer: "1Accepted AnswerHello,Thank you for reaching out to AWS. Kindly find the details you are looking for :-ALTER and SHARE are the only privileges that you can grant to users and user groups.The following is the syntax for using GRANT for datashare privileges on Amazon Redshift. ALTER and SHARE are the only privileges that you can grant to users and user groups in this case.GRANT { ALTER | SHARE } ON DATASHARE datashare_nameTO { username [ WITH GRANT OPTION ] | ROLE role_name | GROUP group_name | PUBLIC } [, ...]https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.htmlI Hope this is helpful.Thank you !CommentSharePriyaAWS2020answered a year ago"


Question: Transfer Data from EC2 to S3

"I plan on deleting/deactivating my instances of EC2, but before I do so, I want to transfer all my existing data from EC2 to S3 so I have a copy for permanent storage. How do I do this?FollowComment"

Answer: "3Create a role for EC2 with a policy including PutObject to the objects in the bucket. The policy looks like this:{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "Stmt1647624289157",      "Action": [        "s3:PutObject"      ],      "Effect": "Allow",      "Resource": "arn:aws:s3:::mybucket/*"    }  ]}Assign the role to the instance.On the instance, copy the files:aws s3 cp . s3://mybucket --recursiveRemove role from instance & delete the role and policyCommentShareEXPERTkentradanswered a year ago0Assuming you have the data in an EBS, you could do this via DataSync.Since it is currently not possible to transfer via DataSync from an EBS you need an NFS server to share the data directories with a DataSync agent. You can install an NFS server on your EC2 instance or create a snapshot of it, then create a volume from the snapshot and attach it to an instance that will act as an NFS server.Once you have the NFS, follow the necessary steps in DataSync:create an agent.create a source location which is the NFS and a destination location which is your S3 bucket.create a copy task and launch it.CommentShareImobachanswered a year ago"


Question: "How do we give recommendations when users create/post content? Like in YouTube, TikTok etc"

"I've explored amazon personalize, etc for generating recommendations. Amazon personalize can be used when all the content is with the company/a single entity. For example, in Netflix, all the content (the catalogue of movies, tv shows etc.) is with them and they generate personalized movie/tv show recommendations.But what if there's a platform similar to Youtube, TikTok, where users can:post content (users are continuously generating content)view other users content and interact(like, share, repost, comment)follow other usersWhen there is user generated content like this and users follow other users (meaning they probably want recommendations from users they follow), how do we give recommendations? Can we do it with Amazon Personalize? What algorithms and tools can be used?Lots of content - handling the cold start problemAnd when there is user generated content, there is going to be lots of content being generated every minute. So how do we handle the cold start problem (i.e. how do we decide who to recommend all of this new influx of content too)? Usually we might experiment with this new content, like recommend it to some users , see how they're responding and appropriately decide how to recommend this content. But when there is a very high frequency of content being created, how do we reduce the amount of time it takes to give recommendations/push the new content to users quickly?And does anybody know if the questions mentioned above can be addressed using Amazon Personalize (in any way)?Open to any and all suggestions. Thank you!FollowComment"

Answer: "0Disclaimer: I haven't still used the domain datasets as they are fairly new.I believe you can do both parts of your questions by using the VIDEO_ON_DEMAND datasets of Amazon Personalize.For the first part of the problem, the VIDEO_ON_DEMAND dataset gives you the option to add metadata such as the CONTENT_OWNER. [1] Even if Personalize wouldn't know who you are following, it will know you are interacting with content from a user, and that should have a similar effect.For the second part, handling the cold start problem, you can use incremental uploads, and Personalize will retrain the model every two hours [2]. If you also use events for tracking user interactions, recommendations for new users will get relevant faster. [3][1] https://docs.aws.amazon.com/personalize/latest/dg/VIDEO-ON-DEMAND-datasets-and-schemas.html[2] https://docs.aws.amazon.com/personalize/latest/dg/importing-domain-dsg-data.html#incremental-import-domain-dsg[3] https://docs.aws.amazon.com/personalize/latest/dg/recording-events.htmlCommentSharejavieranswered a year ago"


Question: Impossible to connect to Windows Lightsail instance

"I have a Lightsail Windows Server Instance that is impossible to connect to and, hence, impossible to fine the issue it has for me. I tried everything and pay now for multiple months money for the instance and cannot use it. I restarted multiple times and tried to use the created backup. But it is also not responsive.I ported the instance to EC2 and started it there. All status checks green and running but I cannot access it there too (Error Error code: 0x204).I before did a backup exactly as described in the documentation (at least to my knowledge) here:https://lightsail.aws.amazon.com/ls/docs/en_us/articles/prepare-windows-based-instance-and-create-snapshotHow can I trace errors to get the instance running again.FollowComment"

Answer: "0Hello,Thank you for reaching out. For what I can see the backup has been transferred to EC2 for further troubleshooting and now facing an OS issue.Since  Snapshots are not done automatically, you can go back only as far as they were taken manually, this is for either EC2 or Lightsail. Now that you are using EC2 for troubleshooting, you can try using Ec2Rescue with a helper Instance by following the documentation below (1). Here you should be able to go back to a Last Known Good Configuration by attempting to overwrite disk signature on BCD, remove Firewalls or check what else could be wrong.As you would be detaching the root volume and attaching it to a helper Instance, here you can also check on logs, event viewer, folders and even recover some files in the event you decide to rebuild.I hope this information is helpful. As a quick reminder and to set expectations, while system administration tasks, OS and application issues fall outside of the scope of AWS support and the Shared Responsibility Model (2-3), in AWS we are always happy to assist as every engineer provides their best knowledge to help resolve all we can even beyond our scope as best efforts.References(1) Use EC2Rescue for Windows Server GUI(2) AWS Premium Support Scope(3) AWS Shared Responsibility ModelCommentShareSUPPORT ENGINEERJosueAWSanswered 8 months ago0Adding to the above, please consider troubleshooting this issue using the AWSSupport-TroubleshootRDP automation runbook.NOTE: You can only run this on/against the instance you have migrated to EC2 and NOT on LightSail.There is a video on how to use run the AWSSupport-TroubleshootRDP automation runbook here.CommentShareCharles Manswered 8 months ago"


Question: How to set spark configuration parameters in PySparkProcessor() in sagemaker processing job?

"Hi folks,I'm trying to set the spark executor instances & memory, driver memory and switch of dynamic allocation. What is the correct way to do it?FollowComment"

Answer: "1Hi!You can achieve this by passing a "configuration" dictionary to the PySparkProcessor. Have a look at the example below to see exactly how to achieve this:https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_processing.html#configuration-overridehappy codingCommentSharegeorgios_sanswered a year agoEXPERTTasioreviewed a year agorePost-User-6470938 a year agoHI! Thanks for the prompt response. I tried the approach above and here is how my configuration looksconfiguration = [{"Classification": "spark-defaults","Properties": {"spark.executor.memory":"45g", "spark.executor.instance":"45","spark.executor.cores":"6","spark.driver.memory":"30g","spark.dynamicAllocation.enabled":"false"},}]and couldn't update the executor instances,  i.e., spark.executor.instances.To confirm passing values via "spark-deafults" is equivalent to “--conf” on an EMR spark-submit job.Share"


Question: t2.micro becomes slower and slower

"Hi !We're testing our application and found that t2.micro is able to run but with not heavy computation (about 30% CPU usage), after a whole night, the performance degenerated for about 5 times e.g. computation time for a unit costs 5 times more, and CPU usage became about 70%, so it's not stable at all, is there any tips on this behavior ? Thanks a lot.FollowComment"

Answer: "0Accepted AnswerHello therePlease note that t2.micro instances have 1 vCPU and baseline performance is 10% of this vCPU. So if workload burst more than 10% CPU usage, it will start consuming burst credits. Once this burst credits becomes 0, the CPU usage will be throttled at 10% baseline. This will be seen as slowness.Please refer https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html for more information.You can choose higher instance type or move to t2 unlimited optionhttps://aws.amazon.com/blogs/aws/new-t2-unlimited-going-beyond-the-burst-with-high-performance/CommentShareAWS-User-7041766answered 3 years ago"


Question: Not Able To Create a New Bucket - Policy Permissions Appear Correct

"Hello,I wanted to see if someone could point me in the right direction towards an AccessDenied error that I receive when creating a new Bucket via the CLI (I am not able to create via the web GUI, as that is not allowed for this account via my company's IT policy).I do not seem to get any more specific info, other than CLI output of " An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied ".  I receive this same error, whether I create directly via the commands s3 mb or s3api, or create as part of a cloudformation template (the stack create fails once this happens).Checking Cloudtrail event logs, I confirmed that the correct role is being passed through the CLI, that I think should have more than sufficient permissions in its policy (listed at the bottom).  Cloudtrail does not appear to show any other error details outside of "Access Denied".Wanted to see if there may be an action or resource type that I may be missing generally to create a bucket?Thanks,Justin{"Effect": "Allow","Action": ["cloudformation:","logs:","s3:","kms:"],"Resource": ["arn:aws:cloudformation:us-east-1:ACCOUNT_NUMBER_HERE:stack/devops*","arn:aws:cloudformation:us-east-1:ACCOUNT_NUMBER_HERE:stack/IoT-Sitewise*","arn:aws:logs:us-east-1:ACCOUNT_NUMBER_HERE:log-group:/aws/","arn:aws:s3:::bucket","arn:aws:s3:::${BucketName}","arn:aws:s3:::AccountPublicAccessBlock*","arn:aws:s3:::object*","arn:aws:s3:::ACCOUNT_NUMBER_HERE-sitewises3*","arn:aws:s3:::ACCOUNT_NUMBER_HERE-sitewises3/","arn:aws:kms:us-east-1:ACCOUNT_NUMBER_HERE:key","arn:aws:kms:us-east-1:ACCOUNT_NUMBER_HERE:alias*"]}Edited by: jf04145 on Nov 5, 2021 12:26 PMFollowComment"

Answer: "0The statement is correct, but you need to separate out the List Bucket and create bucket actions into another statement, as you need to apply a wild card for the resource-id part of the ARN. Add this to your statement and it should work. Worked for me when I tested it in the IAM Policy Simulator.    {          "Sid": "s3statement",          "Effect": "Allow",          "Action": \[              "s3:CreateBucket",              "s3:ListAllMyBuckets",              "s3:GetBucketLocation"          ],          "Resource": \[              "arn:aws:s3:::*"          ]      }CommentShareTomB-AWSanswered 2 years ago0Gotcha, thanks, I will give that a tryCommentSharejf04145answered 2 years ago0Thanks again.  I was able to separate out the S3 permissions from the rest, but in order to put a wildcard on the S3 resource-id, I added a resource tag filter for the bucket per my organization's security requirements.I have added this same resource tag/value to the S3 bucket resource in my Cloudformation template, and I think my syntax is correct (syntax verification passed at least), but I receive AccessDenied still on bucket creation within the CF stack.  I've attached the new S3 policy items and CF template resource, in case there's something sticking out that I missed.I can also attach the error event from CloudTrail if helpful.Thanks again{"Effect": "Allow","Action": ["s3:Create*","s3:List*","s3:Get*","s3:Describe*"],"Condition": {"StringEquals": {"aws:ResourceTag/component": ["sitewise_bridge"]}},"Resource": ["arn:aws:s3:::*"]}Resources:IoTSitewiseExportToS3DestinationBucket:Type: AWS::S3::BucketDeletionPolicy: RetainProperties:BucketName: !Sub ${BucketName}AccessControl: PrivateVersioningConfiguration:Status: EnabledBucketEncryption:ServerSideEncryptionConfiguration:- ServerSideEncryptionByDefault:SSEAlgorithm: AES256Tags:- Key: "component"Value: "sitewise_bridge"Edited by: jf04145 on Nov 10, 2021 8:56 AMCommentSharejf04145answered 2 years ago0Are you able to share you cloud formation template?Happy for you to send me this as a private message if you don't want to share it on a public forum.Also just to check, you are including the policy statement within the following JSON:{"Version": "2012-10-17","Id": "ExamplePolicy01","Statement": [<<statement goes here>>]}CommentShareTomB-AWSanswered 2 years ago0Thanks Tom - I will PM over my CF template and the policy in it's entirety if that works.To your point, I am missing the ID entry in the policy, so I can definitely add that, have seen some other policies in my org that do not have it (for what it's worth).Let me know if you don't see the message and I can resend (it appears to have sent but doesn't show in my sent messages yet)Edited by: jf04145 on Nov 16, 2021 10:41 AMCommentSharejf04145answered 2 years ago0Ok, I think I've identified the problem. You've applied a resource tag condition to the Bucket Creation, however tags wouldn't be applied until after a bucket is created. You need to separate out the create bucket, and don't apply a condition to it. Try this policy and let me know how you get on:{"Sid": "VisualEditor0","Effect": "Allow","Action": ["s3:Create*","s3:List*","s3:Get*","s3:Describe*"],"Condition": {"StringEquals": {"aws:ResourceTag/component": ["sitewise_bridge"]}},"Resource": ["arn:aws:s3:::"]},{"Sid": "VisualEditor2","Effect": "Allow","Action": ["s3:CreateBucket"],"Resource": ["arn:aws:s3:::"]}CommentShareTomB-AWSanswered 2 years ago0Thanks a bunch Tom, I will give that a try and report back.CommentSharejf04145answered 2 years ago0repostEdited by: jf04145 on Nov 24, 2021 10:39 AMCommentSharejf04145answered 2 years ago0Thanks, Tom, that looks to have resolved my issue, I'm able to create the bucket via my Cloudformation template now.  I really appreciate the help digging into this.CommentSharejf04145answered 2 years ago"


Question: Unable to push to ElasticContainerRegistry

"Despite having full permission policy, user gets blocked when pushing to Registry with:User is not authorized to perform: ecr:InitiateLayerUploadPermissions:User has (managed) AmazonEC2ContainerRegistryFullAccess policy directly attached{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "ecr:*",                "cloudtrail:LookupEvents"            ],            "Resource": "*"        },        {            "Effect": "Allow",            "Action": [                "iam:CreateServiceLinkedRole"            ],            "Resource": "*",            "Condition": {                "StringEquals": {                    "iam:AWSServiceName": [                        "replication.ecr.amazonaws.com"                    ]                }            }        }    ]}Registry has this policy in place:{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "AllowPushPull",            "Effect": "Allow",            "Principal": {                "AWS": [                    "arn:aws:iam::<account>:user/<user>"                ]            },            "Action": [                "ecr:BatchGetImage",                "ecr:BatchCheckLayerAvailability",                "ecr:CompleteLayerUpload",                "ecr:GetDownloadUrlForLayer",                "ecr:InitiateLayerUpload",                "ecr:PutImage",                "ecr:UploadLayerPart"            ]        }    ]}When attempting to push:$ docker push <account>.dkr.ecr.us-east-2.amazonaws.com/myapp:latestI get this response:The push refers to repository [<account>.dkr.ecr.us-east-2.amazonaws.com/myapp]e2eb06d8af82: Preparingdenied: User: arn:aws:iam::<account>:user/<user> is not authorized to perform: ecr:InitiateLayerUpload on resource: arn:aws:ecr:us-east-2:<account>:repository/myapp with an explicit denyEdited by: ecruser on Sep 9, 2021 9:47 AMFollowComment"

Answer: 0The issue was due to an MFA policy.  User had to get a session token via command line MFA loginCommentShareecruseranswered 2 years ago


Question: Glue crawler unable to detect array of strings column

"I have some CSV data that have an array of strings.  For example, I want to extract two columns:USGLS,"['MXMAR', 'BZBZE', 'MXCZM']",TESTUnfortunately, the Glue CSV crawler (after executing it) determines the second column to be a string.  Moreover, the crawler believes the second column value is:"['MXMAR'while the third column value is:'BZBZE'Editing the schema for the data catalog doesn't allow me to change the data type of the column to an array type.  How can I change the column data type to an array of strings?FollowComment"


Question: Unmanaged instance lost password or disabled admin account

Lost Admin account or admin account disabled on unmanaged instance. Lost RDP access to Windows instance. Cannot retrieve password with pem. Any method to reset this without relying on system manager.FollowComment

Answer: "0Hi.You can run EC2 Rescue manually.https://aws.amazon.com/premiumsupport/knowledge-center/ec2rescue-windows-troubleshoot/?nc1=h_lsThen try resetting your password using EC2SetPassword.https://aws.amazon.com/premiumsupport/knowledge-center/reset-admin-password/?nc1=h_lsCommentShareEXPERTiwasaanswered a year agoiwasa EXPERTa year agoI actually tried it in an article, but I was able to reset the administrator password. Please refer to here.https://dev.classmethod.jp/articles/ec2-windows-password-reset/Share"


Question: Can my friend use my 50% off promo code to get a discount on her AWS exam?

"Hello,Can my friend use my 50% off promo code to get a discount on her AWS exam?My current company supports/reimburses employees to take the exam, so there is no point for me to use the code… but my friend doesn’t have an IT job and she is studying 100% on her own.I really want to support her, and endorse her to get into IT industry by taking the AWS cert.Now, I was wondering if my 50% discount code from Benefit section is restricted to my account only, or can she use my code to get the discount to take the exam?She and I both live in Southern California.FollowComment"

Answer: "0It is very kind of you to want to share your 50% benefit voucher with a friend. However, per our policies: "All benefits are non-transferable and intended for use solely by the individual who earned the benefit and by the AWS Certification account to which the benefit was originally assigned. If AWS, in its sole discretion, determines you misused or transferred a benefit, AWS may invalidate the exam result related to the misuse or transfer and the benefit will not be reinstated."CommentShareKimWD-AWSanswered a year ago"


Question: Bucket Policy to Restrict Access to an Instance Profile

"I am trying to restrict access to an S3 bucket to only EC2 instances with a specific instance profile.Here's the scenario. I have an S3 bucket into which a file is written. There is a lambda trigger that submits a job to an AWS Batch queue. The Batch compute environment launches an EC2 instance with a specific instance profile that attempts to read the file from S3 for processing. Using the roleId of the role attached to the instance profile I've written the following Bucket Policy,{"Version": "2012-10-17","Statement": [{"Effect": "Deny","Principal": "","Action": "s3:","Resource": ["arn:aws:s3:::MY_BUCKET","arn:aws:s3:::MY_BUCKET/"],"Condition": {"StringNotLike": {"aws:userId": ["AROAMY_ROLEID:"]}}}]}The read fails.This policy seems to work fine in other scenarios where I'm not in an AWS Batch environment. For example, a lambda function that uses the same role as it's execution role. The file is readable, no problem.Is there something different about doing this from with a Batch execution environment that is causing this policy to not work?Thanks for any help...FollowComment"

Answer: 0User errorCommentSharevx2112answered 3 years ago


Question: Announcement: Amazon RDS for Oracle - End of Support Timeline for 12c Oracle Release 2 (12.2.0.1) and Oracle Release 1 (12.1.0.2) Major Version

"Oracle Corporation has announced the end of support for Oracle Database version 12.1.0.2 on July 31, 2022, and version 12.2.0.1 on March 31, 2022 [1]. After these dates, Oracle Support will no longer release Critical Patch Updates for these database versions, and these versions will no longer be available in Amazon RDS for Oracle.For Oracle Database version 12.2.0.1:Amazon RDS for Oracle will end the support for Oracle Database version 12.2.0.1 (all editions) for both License Included (LI) and Bring Your Own License (BYOL) models on March 31, 2022.For Oracle Database version 12.1.0.2:Amazon RDS for Oracle will end the support for Oracle Database version 12.1.0.2 (all editions) for both License Included (LI) and Bring Your Own License (BYOL) models on July 31, 2022.If you have already upgraded your DB instance(s) to 19c, no further action is required. We highly recommend you upgrade your existing 12.1.0.2 and 12.2.0.1 DB instances to version 19c and validate your applications before the automatic upgrades begin.Please keep in mind the following timeline for the 19c version:• 19c: The Premier Support for Oracle Database 19c ends on April 30, 2024, while the Extended Support ends on April 30, 2027. Amazon RDS for Oracle plans to support Oracle Database 19c until April 30, 2027.12.2.0.1 Deprecation - Timeline  summary:• Now - March 31, 2022 - You can upgrade 12.2.0.1 DB instances manually to the version of your choice.• From February 14, 2022 - You can upgrade 12.2.0.1 snapshots manually to the version of your choice.• From February 14, 2022 - Amazon RDS for Oracle disables new instance creates on 12.2.0.1, but you will be able to continue to restore 12.2.0.1 DB snapshots without being auto upgraded until March 31, 2022.• From April 1, 2022 - Amazon RDS for Oracle starts automatic upgrades of 12.2.0.1 DB instances to 19c.• From April 1, 2022 - Amazon RDS for Oracle starts automatic upgrades to 19c of DB instances restored from 12.2.0.1 snapshots.12.2.0.1 Deprecation Timeline by Amazon RDS for Oracle (for both BYOL and LI):Starting on February 14, 2022, Amazon RDS for Oracle will disable the ability to create new DB instances using 12.2.0.1. If you are using 12.2.0.1, please upgrade your instance(s) to a later major version before March 31, 2022. Starting on April 1, 2022, Amazon RDS Oracle will automatically upgrade DB instances from 12.2.0.1 to the latest Release Update (RU) on RDS for Oracle Database 19c. We highly recommend that you test your application on 19c or whichever major version you plan on upgrading to.Starting on April 1, 2022, any 12.2.0.1 DB instance created from a snapshot restore or point-in-time restore will be automatically upgraded to the latest RU on RDS for Oracle 19c.Starting on February 14, 2022, you may upgrade your snapshots manually from 12.2.0.1 to a newer major engine version. For more information, see Upgrading an Oracle DB Snapshot [3]. If you have encrypted snapshots do plan to perform manual snapshot upgrade.12.1.0.2 Deprecation - Timeline  summary:• Now - July 31, 2022 - You can upgrade 12.1.0.2 DB instances manually to the version of your choice.• From June 1, 2022 - You can upgrade 12.1.0.2 snapshots manually to the version of your choice.• From June 1, 2022 - Amazon RDS for Oracle disables new instance creates on 12.1.0.2, but you will be able to continue to restore 12.1.0.2 DB snapshots without being auto upgraded until July 31, 2022.• From August 1, 2022 - Amazon RDS for Oracle starts automatic upgrades of 12.1.0.2 DB instances to 19c.• From August 1, 2022 - Amazon RDS for Oracle starts automatic upgrades to 19c of DB instances restored from 12.1.0.2 snapshots.12.1.0.2 Deprecation Timeline by Amazon RDS for Oracle (for both BYOL and LI):Starting on June 1, 2022, Amazon RDS for Oracle will disable the ability to create new DB instances using 12.1.0.2. If you are using 12.1.0.2, please upgrade your instance(s) to a later major version before July 31, 2022. Starting on August 1, 2022, Amazon RDS Oracle will automatically upgrade DB instances from 12.1.0.2 to the latest Release Update (RU) on RDS for Oracle Database 19c. We highly recommend that you test your application on 19c or whichever major version you plan on upgrading to.Starting on August 1, 2022, any 12.1.0.2 DB instance created from a snapshot restore or point-in-time restore will be automatically upgraded to the latest RU on RDS for Oracle 19c.Starting on June 1, 2022, you may upgrade your snapshots manually from 12.1.0.2 to a newer major engine version. For more information, see Upgrading an Oracle DB Snapshot [3]. If you have encrypted snapshots, you need to plan to perform manual snapshot upgrade.Additional Notes:After the upgrade, if the SQL statements perform in an unexpected manner due to the change in the plans by the 19c optimizer, you can use the OPTIMIZER_FEATURES_ENABLE parameter to retain the behavior of the 12c optimizer.There is no impact to your current Reserved Instances (RI) due to the engine version deprecation of 12.1.0.2 and 12.2.0.1.For versions in Extended Support, BYOL model customers must have purchased Extended Support agreements from Oracle Support or upgrade to a version for which they have support. For the details on the Licensing and Support requirements for the BYOL model customers, refer to the Amazon RDS for Oracle FAQs [4].To learn more about upgrading Oracle major versions in RDS, review Upgrading the Oracle DB engine [5].Please contact us through AWS Support [6] should you have any questions or concerns.Oracle Database Version 19c:The Extended Support for Oracle Database 19c ends on April 30, 2027.Amazon RDS for Oracle will support Oracle Database 19c until  April 30, 2027.We recommend you to upgrade your existing 12.1.0.2 or 12.2.0.1 DB instances to version 19c [7] because it is the long-term support release.Note for versions in Extended Support, BYOL model customers must have purchased Extended Support agreements from Oracle Support or upgrade to a version for which they have support.For the details on the Licensing and Support requirements for the BYOL model customers, refer to the Amazon RDS for Oracle FAQs [4]. Review the version specific deprecation details on the AWS forum [2]. To learn more about Amazon RDS policy for supporting database versions, please see the RDS FAQs and search for “guidelines for deprecating database engine versions” [8].Please contact us through AWS Support [6] or the AWS Developer Forums [9] should you have any questions or concerns.[1] https://www.oracle.com/us/assets/lifetime-support-technology-069183.pdf[2] https://forums.aws.amazon.com/ann.jspa?annID=8593[3] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBSnapshot.Oracle.html[4] https://aws.amazon.com/rds/oracle/faqs/[5] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.Oracle.html[6] https://aws.amazon.com/support[7] https://aws.amazon.com/about-aws/whats-new/2019/11/amazon-rds-for-oracle-now-supports-oracle-database-19c/[8] https://aws.amazon.com/rds/faqs[9] https://forums.aws.amazon.com/forum.jspa?forumID=60FollowComment"

Answer: 0[Announcement] Does not require an answer.CommentShareEXPERTIsraa-Nanswered 18 days ago


Question: Unable to deploy ggv2 component locally

"Hi there,I try to test my component locally, I have this AWS_ERROR_SUCCESS, what is this error btw ?sudo /greengrass/v2/bin/greengrass-cli deployment create  --recipeDir ~/Desktop/ms5/greengrass-build/recipes  --artifactDir ~/Desktop/ms5/greengrass-build/artifacts --merge ms5=1.0.0Mar 30, 2022 4:41:56 PM software.amazon.awssdk.eventstreamrpc.EventStreamRPCConnection$1 onConnectionSetupINFO: Socket connection /greengrass/v2/ipc.socket:8033 to server result [AWS_ERROR_SUCCESS]Mar 30, 2022 4:41:56 PM software.amazon.awssdk.eventstreamrpc.EventStreamRPCConnection$1 onProtocolMessageINFO: Connection established with event stream RPC serverLocal deployment submitted! Deployment Id: 7a45ceaf-8a5b-4c80-9c7e-cf74f4a35344 sudo /greengrass/v2/bin/greengrass-cli deployment status -i 7a45ceaf-8a5b-4c80-9c7e-cf74f4a35344Mar 30, 2022 4:42:48 PM software.amazon.awssdk.eventstreamrpc.EventStreamRPCConnection$1 onConnectionSetupINFO: Socket connection /greengrass/v2/ipc.socket:8033 to server result [AWS_ERROR_SUCCESS]Mar 30, 2022 4:42:48 PM software.amazon.awssdk.eventstreamrpc.EventStreamRPCConnection$1 onProtocolMessageINFO: Connection established with event stream RPC server7a45ceaf-8a5b-4c80-9c7e-cf74f4a35344: FAILEDFollowComment"

Answer: "1Accepted AnswerHi. AWS_ERROR_SUCCESS means success. It's not an error.https://catalog.us-east-1.prod.workshops.aws/workshops/5ecc2416-f956-4273-b729-d0d30556013f/en-US/chapter8-tips-troubleshoot#tip6.-i'm-seeing-aws_error_successhttps://github.com/awslabs/aws-c-common/blob/eb4cff6d50222a5352efd4f761563348bc1ecd1d/include/aws/common/error.h#L144To know more about the cause of your deployment failure, you will need to look in either greengrass.log or the component's log file.CommentShareEXPERTGreg_Banswered a year ago"


Question: Scheduled Aurora Read Replica Scaling

"Hi! We are currently using Amazon Aurora and want to scale the Read Replicas based on schedule instead of a Target Tracking Policy. While this is not natively available through the console, we plan on doing it through a Lambda function and an EventBridge rule. Is there a  better way to approach our method?FollowComment"

Answer: "1Accepted AnswerHelloSince you mentioned target tracking, I assume you have AutoScaling setup on the Aurora cluster.  You can use Scheduled Scaling policies, which are a native part of Application AutoScaling (the service which tells RDS when to scale)https://docs.aws.amazon.com/cli/latest/reference/application-autoscaling/put-scheduled-action.htmlThis will be simpler and cheaper (no Lambda costs) than EventBridge -> Lambda.  Make sure you select a timezone when you set it up if you live in a country with time changes to account for those.Summary: if you're only using scheduled actions and not target tracking, just set both the Min and Max in each scheduled action to the same value, and that's what the desired capacity will change to.  If you're also using target tracking to handle bursts, you'll want to set the Min to the value you want desired to be at when scaling-out, and have the max be a bit higher to allow for additional scale-out to handle unexpected bursts of trafficDetails: The scheduled action sets the min and/or max values of the scalable target.  If the desired capacity would be outside this new min/max range, then it will get dragged along with the new min/max values.  A couple quick examples for scale-out, but the same concepts apply for scale-in (first column values before the action; middle column is the min/max set in the scheduled action; last column the values after the scheduled action triggers.  NULL means the value wasn't included in the scheduled action)Min/Max/DesiredMin/MaxMin/Max/DesiredComment0/0/02/22/2/2Min and max set to 2, so desired gets increased to 2 to not be below min0/1/1NULL/40/1/4Desired didn't go up since only the Max increased, and desired was still between Min and Max0/1/22/42/2/4Desired went up by 1 so that it wasn't below Min.  Max is 4 to allow for dynamic scaling1/4/42/62/4/6Desired doesn't change since it was already higher (presumably from target tracking), and is still between Min and Max after the scheduled action.CommentShareShahad_Canswered a year agoburgerguy a year agoThank you so much for this! I hope AWS will make this available through the console as well, not just via CLI.Cheers!Share"


Question: NLB doesn't send traffic to new healthy Fargate tasks

"I have ECS with Fargate. Traffic is routed with NLB.When service is just started NLB routes traffic equally to all tasks with no problem.However, when service is automatically sales up (or if I manually increase tasks count) NLB is not routing traffic to a newly created tasks.Despite the fact that new task are marked as healthy in a Target group. I can see it clearly form new tasks log and from "about zero" minimum CPU on the Service.FollowCommentRodney Lester a year agoI presume they are all registering with IP, not instance ID in the target group. This is the default in Fargate. Are they in an AZ that the NLB is not configured to support?ShareSmotrov a year agoYes, NLB is available in the same 3 AZ as tasks. And yes, they are registered with IP.ShareBen Bridts EXPERTa year agoDo you have sticky sessions enabled? How are you testing the routing?ShareMonica_Lluis SUPPORT ENGINEERa year agoHello Smotrov, has your issue been resolved? If not, can you reply to Ben's questions? Thank you for your participation in AWS re:Post community.Share"

Answer: "0Hi Smotrov, I understand that you have ECS Fargate tasks which are being load balanced using an NLB with a Target Group configured for IP Registration. However, connections are not being routed to the new healthy tasks as they are registered with the target group. In the NLB overview section of AWS Docs you will find that each individual TCP connection is routed to a single target for the life of the connection. With ALB you can specify the load balancing algorithm of either round robin or least outstanding requests.CommentShareclintwyckoff_awsanswered a year agoEXPERTRob_Hreviewed a year ago0This question was answered live in great detail at twitch https://www.twitch.tv/aws/video/1420471067.CommentShareSUPPORT ENGINEERMonica_Lluisanswered a year agoEXPERTRob_Hreviewed a year ago"


Question: ec2:ResourceTag/Name not working with IAM Policy

"Hi,My IAM Policy is as below but not work.  I can modify security groups after removing "ec2:ResourceTag/Name".aws ec2 modify-instance-attribute --instance-id  $INSTANCEID --groups $SGID --region ap-southeast-1How to fix it?        {            "Sid": "AllowModifyDevOpsInstances",            "Effect": "Allow",            "Action": [                "ec2:ModifyInstanceAttribute"            ],            "Resource": "*",            "Condition": {                "StringLike": {                    "ec2:ResourceTag/Name": [                        "DevOps*"                    ]                },                "StringEquals": {                    "aws:RequestedRegion": "ap-southeast-1"                }            }        },FollowCommentBrettski-AWS EXPERT4 months agoFor background: What are you intending your policy to do?ShareLinda 4 months agoI want this iam role can modify security groupsShareErcan 4 months agoHi Linda,Could you paste your aws cli exact output, please?ShareLinda 4 months agoHi  Ercan,my command:aws ec2 modify-instance-attribute --instance-id $(curl -s http://169.254.169.254/latest/meta-data/instance-id) --groups sg-07xxxxxxxxxff sg-01xxxxxxxxxxxb4 --region ap-southeast-1Response:An error occurred (UnauthorizedOperation) when calling the ModifyInstanceAttribute operation: You are not authorized to perform this operation. Encoded authorization failure message: yi4-pN3H8UP_XN9V58g3tszKNQ2WG7NWzHDma4N3xmN3AmEUdJ2F-7di_0h0YDD3y5vk1tKYt8Z9WDlGH0yt1O4EdZUGyyjGgtzP3rPAfJrUPX2Khilm3sjVCzIoCYZU_xF2X9VEXwNxgMDPwgD37mRjdP5L-cOQytRP8FP6cya7hxS1aDJxfT1i7ZAxUiEnV0uKJshAEN-G3LSpEI6ULhFijNTMvSSnjqnD54muUv0fmJQgyH_PWyj_wDRLXmtu624_vATuugwl5Nrtafr2ch263GSMSeePTArNIr_W-Wz4NENpGnuJ7XpgDQbzFwqrUEJ_W6bvb55Ugjx4RQGv0g6vGax8gljHOACyzfZzxhVr1coQWpw_DUMpksy7Z_KrmG5oO8p0fq_3ddVtBzGWp7zHA0A3yZgZyMMs3byvc4hrotV2l6ZmG8vwUR5biWGp7bG0YUCZKH0RC3Cwc_MyeKOz4sSWd9bl08aC2MER_qoh4QrJa0PA8iBAIvFjetifoWFOdSWUXNnYwnoj-LyQ3EWadG-7T1Pvqd5b_22dDVDuK7LPNf7uGmhRVbH8R8NRnV0KZ1Ae2oLL-3muMlp0XA1NlED5ppNhcUgNpEf0LZG7JT7Ldv3WHwqScLmn8Llex28lZmxJi8soCdH_DwRzjTpUKWXRPqwd8HIKPkUQWTRzNKHwuw4Share"

Answer: "1Hello, this  IAM policy will only allow  ec2:ModifyInstanceAttribute  action on instances that have a tag key of Name and a tag value that starts with DevOps. It's not allowing the action on  any SGs.To fix it, you can add the ec2:ModifyInstanceSecurityGroups action to your policy's "Action" array:"Action": [    "ec2:ModifyInstanceAttribute",    "ec2:ModifyInstanceSecurityGroups"],CommentShareOlwi Mamdouhanswered 4 months agoLinda 4 months agoI will got the following error:"Invalid Action: The action ec2:ModifyInstanceSecurityGroups does not exist"Share"


Question: What IAM Role permissions required to restore CMK encrypted EC2 instances ?

"Hi there,Apologies if this question has been answered earlier, unfortunately I am not able to locate it easily :(When I try to restore an EC2 instance that has been backed up by AWS Backup, it fails with error "Client.InternalError: Client error on launch"Looks like it is an encryption issue...The following are the details:The EC2 instance being backed up has been setup with Volume(s) that are encrypted with Customer Managed Keys (CMK)We provided full KMS permissions for the role (not my preferred option, but just to test the restores)The restores attempts terminates the target EC2 instance almost immediately on launch of the restore operation with the error "Client.InternalError: Client error on launch"Note: I tested the backup and restore of the EC2 instance with unencrypted volume(s) and it completed successfullyPlease provide guidance to complete restores for EC2 instances setup with CMKThank you,ShyamFollowComment"

Answer: "0Issue was caused by the KMS CMK key policy, it has been resolved after providing grant to the backup roleCommentShareshyamkanswered 3 years ago"


Question: Reason for providing the 'source-dbi-resource-id' parameter in the AWS RDS CLI command 'restore-db-instance-to-point-in-time'?

"Hi,I have come across the parameter 'source-dbi-resource-id' in the AWS CLI command for **Point-in-Time Restore **of RDS DB instances. The parameter accepts the Resource ID of the DB instance which acts as the source for Point-in-Time restore.Can someone let me know why this parameter exists in the CLI command/AWS API, since it is always possible to refer to the source DB instance using the parameter 'source-db-instance-identifier'. Does the 'source-dbi-resource-id' parameter serve a special use-case that is otherwise not possible using the instance identifier?Thanks!References -AWS CLI command 'restore-db-instance-to-point-in-time' : https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/restore-db-instance-to-point-in-time.htmlAWS REST API documentation for RestoreDBInstanceToPointInTime :https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_RestoreDBInstanceToPointInTime.htmlFollowComment"

Answer: "0Hi there,You mention that you would like to know the difference between "source-dbi-resource-id" and "source-db-instance-identifier" in AWS CLI point in time restore.With that being said, on the command line interface is where you would use the “source-db-instance-identifier” to get a point in time restore on an existing resource. Where as the command “source-dpi-resource-id” is used to restore your instance to a specific point in time whether your instance is deleted or not. So the main difference is that you are able to restore to a point in time with the "source-db-instance-identifier”  command and with the "source-dbi-resource-id" command you may take it a step further and restore from a deleted instance and not just an existing one.You could also perform test cases for the two commands which helps better understand the differences:For the identifier restore command you may run the command on an existing DB and restore it to a specific point in time:aws rds restore-db-instance-to-point-in-time \    --source-db-instance-identifier test-instance \    --target-db-instance restored-test-instance \    --restore-time 2018-07-30T23:45:00.000ZFor the resource ID command,delete your instance and from an automated back up you may restore it to a point in time:aws rds restore-db-instance-to-point-in-time \--source-dbi-resource-id db-SFYEN53K4FSUXXXXXXXXX \--target-db-instance restored-after-delete \--restore-time 2018-07-30T23:45:00.000ZCommentShareAntonioanswered a year ago0How can we find db source-dbi-resource-id after Instance is deleted ?CommentSharerePost-User-5346999answered 4 months ago"


Question: Connect to MySQL DB Securely using MySQL Workbench

"Trying to connect to MySQL DB Securely using MySQL Workbench or MySQL CLI over SSL. Not sure the process of obtaining, downloading and applying the certificate and connecting MySQL Workbench to it using SSL. I can connect via port 3306 so its only SSL. Appreciate the help.FollowComment"

Answer: "0Without knowing your exact version, you could refer to this or find similar: https://dev.mysql.com/doc/workbench/en/wb-mysql-connections-ssl-wizard.htmlIn general you need to download or generate a trusted certificate (typically a PEM file) which then can be provided as connection properties.CommentShareEXPERTalatechanswered 4 months ago"


Question: How to specify a character which contains hyphen in ExportOnly of start_export_task

"I am using a method start_export_task of boto3 (python)  to export a snapshot of the RDS.When I try to specify the schema “something-v2” (contains hyphen) in Aurola DB For MySql using its ExportOnly option, I get the following error.ClientError: An error occurred (InvalidParameterValue) when calling the StartExportTask operation: something-v2.* Table names can contain only basic Latin letters, digits 0-9, dollar symbols ($), and underscores (_).Is it not possible to specify a schema name that contains hyphens?We are waiting for your answer.FollowComment"


Question: What Happens to Running Full-Load Tasks During a DMS Instance Scale-Up/Down.

"In scale-up/scale-down solution for DMS Replication Instances, there are multiple instances, and each instance is monitored via CloudWatch Alarms. When an alarm is triggered, a message is sent to a topic that in turn calls a lambda to scale the instance appropriately.  Eventually, the alarm sources DMS instance is either scaled to a smaller or larger instance.We are attempting to determine the effects of the scale-up/down occurring while an attached task is in the middle of a "Full-load". The results appear to be inconsistent. Here is what has been observed.Restart a full load task attached to a target instance.Trip the alarm for that target instance via set_alarm_state.Very quickly, the target DMS instance state changes to "modifying" and the links in the UI are greyed out.While the DMS instance appears, in the UI, to be in a "Modifying" state, the Full-Load Task continues for another 5 minutes and completes.Is this just happenstance?Can anyone confirm that a state change from "Available" to "Modifying" for a DMS Instance is smart enough to wait for active full-loads to complete before rebooting or is this just some sort of latency between state changes and the UI?It would be nice to have an explanation of what happens when a 30-minute full load task starts and the associated DMS instance scales up while that task is still running.FollowComment"

Answer: "1Hi,When you scale up / down your replication instance, its restarted as part of this modification. Upon restart, each DMS task running on the instance is stopped and resumed after new instance is available.DMS doesn't wait for any task to complete before it restarts the replication instance. Behavior that you noticed that full load task continued to run and completed before instance was restarted, is a coincidence.Below is the behavior of a full-load task when it resumes, after replication instance restart.Tables for which load was completed are not touched/affectedTables for which full load was in progress when instance restarted, load of such tables is restartedTables for which full load was yet to start, will be queued and started as task progressesCommentShareSUPPORT ENGINEERGaurav_Kanswered 2 months agoRoss Bush 2 months agoExactly the kind of answer I was hoping to get. Thanks again for the fast response!Share"


Question: Use getQueueUrl API to generate sqsUrl using sqsName and awsAccountId or simply hardcode the sqsUrl string. What is the recommendation?

"There are two ways to get the sqsUrl:A. We can simply hardcode the static url string in the code.Pros:a. No API call, hence less latency.b. Simple to write and less line of code. Will have to define just the sqsUrl for each sqs in Configuration.c. Many services in Amazon use this only, probably to save latency.Cons:a. If AWS changes the url structure in future, then the code will break. And we have to go change the url in code. But slim chance of happening this. They will make sure of compatibility.b. Kind of hardcoding, not a good coding practice. (My opinion)B. Other way is to programatically create the sqsUrl using getQueueUrl api, which accepts sqsName and awsAccountId (optional, only required if sqs’s account is different than caller’s account.)Pros:a. If we want to send the message to some random team’s sqs, but they didn’t provide the sqsUrl. They just provided the name of sqs and account id. In that case, generating url programatically can help.b. If AWS changes the url structure in future, the code will require no changes, and things will work as it is.c.There must be some reason why SQS provided getQueueUrl api, to avoid directly using url/arn. (Yet to find any documentation where AWS recommends using this. )Cons:a. Extra dependency on an api getQueueUrl, extra latency of ~200 ms. But should not be a concern when this much latency doesn’t matter. For ex: Offline systems, Postprocessing scenario, etc.b. Less lines of codes.FollowComment"

Answer: "0If you expect the SQS URL to remain constant and do not anticipate needing to frequently change the queue or account, the simpler approach of hardcoding the URL may be suitable. This approach indeed reduces latency and the amount of code you need to maintain. The risk, as you pointed out, is that if Amazon were to change the URL structure in the future, your code might break. However, it's worth noting that such changes are typically rare and are usually accompanied by lengthy deprecation periods, during which both the old and new formats would continue to work. On the other hand, using the getQueueUrl API to dynamically generate the SQS URL provides more flexibility. This method would be beneficial if you foresee needing to interact with multiple queues or accounts, or if the queue name might change. Furthermore, this approach is more robust to potential future changes in AWS's URL structure.CommentShareEXPERTsdtslmnanswered 6 days ago"


Question: Internal Server Error - SAML Federation for User Pools

"Hi!Attempting to get SAML federation for User Pools working. Have followed instructions as outlined here:https://aws.amazon.com/blogs/mobile/amazon-cognito-user-pools-supports-federation-with-saml/To trigger sign-in, we're running a local version of this example app:https://github.com/aws/amazon-cognito-auth-js/tree/master/sampleWhen selecting to sign in via our SAML IdP, we're being redirected to the IdP login and after authentication we're successfully receiving a POST back to ourhttps://myapp.auth.eu-west-1.amazoncognito.com/login/redirect with a successful SAML assertion. NameID is set in the saml:Subject, and we're using the urn:oasis:names:tc:SAML:2.0:nameid-format:persistent (as specified in the above instructions)We're then supposed to be redirected to our redirect_url with an idp token, but instead we see a GET to https://myapp.auth.eu-west-1.amazoncognito.com/error?error=Error%20occured%20while%20reading%20the%20IdP%20token.%20Internal%20server%20error (in the Cognito provided web pages)"Error occured while reading the IdP token. Internal server error."As a side note: we have also tried establishing a user pool in another region, but with the same result...How can we find out what is going wrong? Would apprectiate any help :)Regards, KennethFollowComment"

Answer: "0Thanks for reporting this. We are in process of fixing couple of bugs identified in our implementation.Name id longer than 128 char results in error.SAML response containing invalid base64 chars  (new line or whitespace) results in error.We are in process of fixing these and will update here once it is done.Meanwhile if above two cases are not applicable for your scenario, can you please PM me approximate timestamp and user pool id so that i can investigate further.CommentShareAWS-User-2053392answered 6 years ago0Hi Vinay, thanks for the response!The nameId is a lot shorter than 128 chars. And the encoded SAML response does not contain any newline or whitespace characters.Have also tried both with users that don't already exist in the user pool AND users that are already established (with user self-signup) within the user pool. Same problem for both.Will PM you with details so that you can investigate furter :-)Thanks a lot, KennethCommentSharekschulstanswered 6 years ago0A follow up for others that might experience the same problem...It did not look like our SAML response contained invalid chars, but after inspecting AWS logs:Vinay:"I can confirm that the issue is related to the invalid base-64 chars (new line). You will see %0D%0A chars in the url encoded base-64 response, which typical SAML decoder ignore. This is why you don't see it in the base-64 response.We have the fix rolled out in Frankfurt, Mumbai and London region already. So, you should be able to get it working in any of these regions. Meanwhile we are rolling out the fix in other regions."I attempted to get it working with a user pool in Frankfurt, but experienced the same problem there. However, when I tried London, things started to work! :-DGuess the fix will be rolled out in multiple regions soon...CommentSharekschulstanswered 6 years ago0Can you confirm this has rolled out on all regions, including us-west-2? I'm receiving a similar error, even when the IdP is showing success.CommentSharetravisnanswered 6 years ago0The fix has been deployed to handle the new line chars in the URL encoded SAML response.CommentShareAWS-User-2053392answered 6 years ago"


Question: Running services in Lightsail

"Dear all,First thanks in advance for any help.How can I see the running services on Lightsail? I removed all the instances, containers, etc, but yhe bill is still increasing, so there still must be running something.Can anybode please explain where to check this?regards HenkFollowComment"

Answer: "3Accepted Answeractually this is the correct plase to check the instances but if you still does not see anything please check the billSelect "Cost Explorer"Use the filters and grouping options to analyze your costs by service, region, or other dimensionsCommentShareEXPERTsdtslmnanswered 2 months ago2Thanks a lot for your help.This way I found two volumes still running :)CommentSharerePost-User-8926349answered 2 months ago"


Question: lambda permissions: source arn cannot have wildcard for region

"I'm trying to add a permission to a (regular) lambda for another lambda@edge's loggroups subscription.Since the loggroups are being created by lambda@edge in their respective region I need a permission for every region. It would be simpler to make the permission work for any region in that account and function.Example code that fails:await lambdaClient.send(    new AddPermissionCommand({        StatementId: `${lambdaEdgeFunctionName}-subscription-permission`,        FunctionName: loggroupSubscriptionTargetFunctionName,        Principal: "logs.amazonaws.com",        SourceArn: `arn:aws:logs:*:${accountId}:log-group:/aws/lambda/us-east-1.${lambdaEdgeFunctionName}:*`,        Action: "lambda:InvokeFunction",    }))Since above command fails the regex check I have to add a permission for every region the lambda@edge will be run in.SourceArn (that works)`arn:aws:logs:us-east-1:${accountId}:log-group:/aws/lambda/us-east-1.${lambdaEdgeFunctionName}:*`aws error message:  at 'sourceArn' failed to satisfy constraint: Member must satisfy regular expression pattern: arn:(aws[a-zA-Z0-9-]*):([a-zA-Z0-9\-])+:([a-z]{2}((-gov)|(-iso(b?)))?-[a-z]+-\d{1})?:(\d{12})?:(.*)different related github issue: https://github.com/hashicorp/terraform-provider-aws/issues/9972FollowComment"

Answer: "1Hi,Thanks for sharing your steps and the error which is being returned. However, I would like to clarify that this is indeed expected behavior. The source arn for the Lambda resource policy cannot have a wildcard region or account ID which is why we are getting this error when using the AddPermission API.I do see that there is an existing feature request which aims to add the functionality of using wildcards in the region or account ID. While I am unable to comment on if/when this feature may get released, I request you to keep an eye on our What's New and Blog pages for any new feature announcements.CommentShareSUPPORT ENGINEERRyan_Aanswered a year ago"


Question: Do I need to create two kinesis delivery streams to send messages under two different paths in the same s3 bucket?

I have a lambda function which receives logs from cloudwatch using a subscription filter and has to send them to firehose to be delivered to an s3 bucket.The same lambda function receives two kinds of logs. These two log formats need to be written into two different paths under the same s3 destination.For eg.1)Log type A should be stored under - /logTypeA directory.2)Log type B should be stored under - /logTypeB directory.I am using cdk to create the necessary firehose and s3 setup.Do I need to create two different delivery streams - one for log type A and one for log type B or is there a way to do this using a single delivery stream ? Didn't find anything useful in the tech docs.FollowComment

Answer: "0If you have some field whose value you can use to discriminate, you can use dynamic partitioning.CommentShareEXPERTTasioanswered a year ago"


Question: Flexmatch is filled and status is "COMPLETED" but there is no Gamesession

"REMOVEDUPLOADas described in the title, i succesfuly made a match and used DescribeMatchmakingCommand to check on the status of a ticket, after i forced a match during my test, the status is COMPLETED, but there is no gamesession created and response.TicketList[0].GameSessionConnectionInfo is undefined.What did I do wrong and how to fix it?FollowComment"

Answer: "0I have cut a ticket for the gamelift team to look into this. Thanks!P62389542CommentSharerePost-User-7108006answered a year ago0hi @REDACTEDUSERany update on this?CommentSharerePost-User-9801317answered a year ago0Hey @REDACTEDUSERSorry for the delayed response!In order to create a GameSession, a matchmaking configuration must have an associated queue. You can update your configuration via the AWS CLI (https://docs.aws.amazon.com/cli/latest/reference/gamelift/update-matchmaking-configuration.html) or our Console (https://console.aws.amazon.com/gamelift/home#/r/matchmaking-configurations) to add your queue.Hope that helps!CommentSharerePost-User-1696513answered a year ago"


Question: CloudFront alternate domain name Certificates

"My custmer uses Cloudfront to run shops for customers under their own subdomain but also under the mobile subdomain of the customer, which they don't manage themselves.Before this CloudFront security change they were able to add the alternate domain using only their certificate and then use it to validate and get a Let's Encrypt certificate for the customer subdomain.Is this still possible through some other means?FollowComment"

Answer: "0Accepted AnswerI understand that this company (for example shops.com) creates a distribution covering two CNAMEs for each of their customers (customer.shops.com and m.customer.com).  Then they ask their customer to point m.customer.com to the created distribution, then they use Lets Encrypt with HTTP validation to issue a certificate covering both domain, and finally they attach the certificate to this distribution using ACM.That will not work any more with the new security enhancements in CloudFront. I suggest that the customer first issue a certificate using ACM and DNS validation covering both domains (first action from their customer), then creates the distribution with this certificate and finally ask the customer to create the necessary CNAME in their DNS configuration.CommentShareEXPERTachrafanswered 4 years ago"


Question: ECS Service connect could not resolve host

"Hello.I've been looking into using ECS Service connect for one of the projects. I've configured ECS service with Service connect functionality but when I try to connect to it from other service in the same namespace I get curl: (6) Could not resolve host: coreService configuration"serviceConnectConfiguration": {    "enabled": true,    "namespace": "arn:aws:servicediscovery:us-east-1:***:namespace/ns-awd3wsdifs2nsrla",    "services": [        {            "portName": "core",            "discoveryName": "core",            "clientAliases": [                {                    "port": 80,                    "dnsName": "core"                }            ]        }    ]},"serviceConnectResources": [    {        "discoveryName": "core",        "discoveryArn": "arn:aws:servicediscovery:us-east-1:***:service/srv-rirfgcxi4bpplqyj"    }]Curl resultscurl -v http://core:80* Could not resolve host: core* Closing connection 0curl: (6) Could not resolve host: corecurl -v http://core.skc:80* Could not resolve host: core.skc* Closing connection 0curl: (6) Could not resolve host: core.skcSource service config"serviceConnectConfiguration": {    "enabled": true,    "namespace": "arn:aws:servicediscovery:us-east-1:***:namespace/ns-awd3wsdifs2nsrla",    "services": [        {            "portName": "admin",            "discoveryName": "admin",            "clientAliases": [                {                    "port": 77,                    "dnsName": "admin"                }            ]        }    ]},"serviceConnectResources": [    {        "discoveryName": "admin",        "discoveryArn": "arn:aws:servicediscovery:us-east-1:***:service/srv-niqcn3ec7jgsz27i"    }]VPC has DNS and DNS hostnames enabled{    "VpcId": "vpc-00bdb82e8d17a444a",    "EnableDnsSupport": {        "Value": true    }}{    "VpcId": "vpc-00bdb82e8d17a444a",    "EnableDnsHostnames": {        "Value": true    }}What am I missing?Thank you.FollowComment"

Answer: "0Accepted AnswerActually, figured it out.Sequence of events:Deploy service1Deploy service2Connect to container in ECS service1 and try curl http://service2:80 - this results in could not resolve hostConnect to container in ECS service2 and try curl http://service1:80 - it worksRestart service1 with --force-new-deployment and repeat step 3 - and now it works.Apparently, ECS adds those ECS Service connect related records in /etc/hosts. Line Service Connect configures DNS names for your services in the task itself, and doesn't require nor create DNS records in your hosted zones. suggests this./etc/hosts on service1 in step 3127.0.0.1 localhost10.0.20.61 ip-10-0-20-61.eu-west-2.compute.internal127.255.0.1 service12600:f0f0:0:0:0:0:0:1 service1/etc/hosts on service2 in step 4127.0.0.1 localhost10.0.20.167 ip-10-0-20-167.eu-west-2.compute.internal127.255.0.1 service12600:f0f0:0:0:0:0:0:1 service1127.255.0.2 service22600:f0f0:0:0:0:0:0:2 service2/etc/hosts on service1 in step 5127.0.0.1 localhost10.0.20.13 ip-10-0-20-13.eu-west-2.compute.internal127.255.0.1 service12600:f0f0:0:0:0:0:0:1 service1127.255.0.2 service22600:f0f0:0:0:0:0:0:2 service2Looks like ECS does not update /etc/hosts dynamically when new endpoints are added into CloudMap namespace.CommentSharerePost-User-3914566answered 6 months ago"


Question: No longer possible to restore snapshot to serverless?

"Hello,during our QA process we routinely restore a snapshot from production system to a serverless Aurora MySQL cluster and from there reload the staging environment. Today I noticed that I can no longer restore a snapshot to a serverless instance but instead had to select provisioned.Is this a know issue due to deprecation of old versions? Snapshot created from a RDS running 5.7.24 of MySQL.FollowComment"

Answer: "0Accepted AnswerI heard that the restoration can be done with AWS CLI, but do I get any errors when I run it with the command?https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.restorefromsnapshot.htmlaws rds restore-db-cluster-from-snapshot \    --db-cluster-identifier mynewdbcluster \    --snapshot-identifier mydbclustersnapshot \    --engine-mode serverless \    --engine aurora-mysql \    --engine-version 5.7The following document shows that 5.7.24 is already an unsupported version, so it is important to check if it is possible to restore server-less if the version is upgraded.https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Concepts.VersionMgmt.htmlCommentShareEXPERTRiku_Kobayashianswered 18 days agomsandstrom 18 days agoHaven't tested through the CLI. Could be that some issues have been identified, I have noticed the UI for restoring has changed multiple times the last couple of months.ShareRiku_Kobayashi EXPERT18 days agoThe AWS UI can change frequently, so it is recommended to use the CLI if you can.Share"


Question: I accidentally deleted var folder

"I accidentally deleted var folder. So basically I lost everything including databases and folders within var/www/htmlIs there any way to recover them, any paid support can help. Please advise.FollowComment"

Answer: "1Accepted AnswerThe AWS Shared Reponsibility Model means that this data is looked after by you not AWS.  AWS however provides great tools for backing up your important data based around EBS snapshots which can be done directly or via the AWS Backup service or Amazon Data Lifecycle Manager.  So you can restore a snapshot to another instance to replace the current one, or copy the data you need from it.CommentShareEXPERTskinsmananswered 3 months agoEXPERTTushar_Jreviewed 3 months agorePost-User-0901117 3 months agothank you so much for your quick response. Actually I',m new to AWS. I checked under EBS there is no snapshot available.Could you please advise step by step. Also please advise, now I need to create a new instance right? the older one won't work anymore?Share"


Question: Error while calling API DetectDocumentTextAsync - One or more errors occurred. (Unable to get IAM security credentials from EC2 Instance Metadata Service

"Hi I am sharing below error, dont know why this error showing.2-3 days before this code was working and now i am getting this error, i have not changed any setting. Please help to understand this errorCode showing errorvar result = new DetectDocumentTextResponse();var s3Object = new S3Object {Bucket = bucketName,Name = key};var request = new DetectDocumentTextRequest();request.Document = new Document {S3Object = s3Object};// line returning errorreturn await this.textract.DetectDocumentTextAsync(request);System.AggregateExceptionHResult=0x80131500Message=One or more errors occurred. (Unable to get IAM security credentials from EC2 Instance Metadata Service.)Source=System.Private.CoreLibStackTrace:at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)at System.Threading.Tasks.Task.Wait()at Dotnet_Core.ArgHandlers.DetectTextS3Handler.Handle(String bucketName, String s3File) in D:\amazon-textract-code-samples-master\amazon-textract-code-samples-master\src-csharp\ArgHandlers\DetectTextS3Handler.cs:line 14at Dotnet_Core.Program.Main(String[] args) in D:\amazon-textract-code-samples-master\amazon-textract-code-samples-master\src-csharp\Program.cs:line 55This exception was originally thrown at this call stack:[External Code]Dotnet_Core.Services.TextractTextDetectionService.DetectTextS3(string, string) in TextractTextDetectionService.csInner Exception 1:AmazonServiceException: Unable to get IAM security credentials from EC2 Instance Metadata Service.FollowComment"

Answer: "0Hi,To better understand your issue, could you pleaseHow did your code get the necessary permission? if you use AWS SDK could you please provide the code block.As it was working before, could you please verify your EC2 instance is healthy and has correct IAM attached, if possible, could you please try to relaunch a new instance and deploy the code to see if it works on new instance.ThanksCommentShareJadyanswered 9 months ago0Dear Jady,Thankyou for your response, below is peace of code i am usingI am not using ec2 instance, even not created, using dotnet core application on laptop and accessing Textract sdkvar builder = new ConfigurationBuilder().SetBasePath(Environment.CurrentDirectory).AddJsonFile("appsettings.json", optional: false, reloadOnChange: true).AddEnvironmentVariables().Build();var awsOptions = builder.GetAWSOptions();Console.WriteLine(awsOptions.Profile + ":" + awsOptions.ProfilesLocation + ": " + awsOptions.Region.DisplayName);IAmazonTextract textract =awsOptions.CreateServiceClient<IAmazonTextract>();var request = new StartDocumentTextDetectionRequest();request.DocumentLocation = new DocumentLocation{   S3Object = new S3Object{  Bucket =  "testautomatelab",Name = "test-files/employmentapp.png"                }};var response =  textract.StartDocumentTextDetectionAsync(request);}Receiving responseId = 263, Status = Faulted, Method = "{null}", Result = "{Not yet computed}"Response Inner exceptionOne or more errors occurred. (Unable to get IAM security credentials from EC2 Instance Metadata ServiceCommentSharerePost-User-4989193answered 9 months ago"


Question: "I have moved my domain to another account, I have created the hosted zone but it does not forward to my ec2 instance and not to my load balancer"

"I moved my domain that I bought route 53 to another aws account through Terminal-CMD, it worked perfectly, I created the zones hosted on router 53, but when I create the type A record so that my ec2 instance appears when they search for my domain on the internet it does not appear Nothing, it just gives me an error and adding the load balancer doesn't work either, it doesn't work with a static IP, balancer or latency. I don't know what to do anymore, I need help. I am very new to aws.FollowComment"


Question: All nodes have disappeared from the elastic search cluster.

"They simply do not exist. An attempt to reconfigure the cluster, increase capacity, and select other types of virtual machines did not lead to anything.The cluster has been "processing" for more than a dayWhy did this happen?Can this cluster be repaired or its data restored?FollowComment"

Answer: "0Hi, in that situation, please open a support ticket to unblock your domain and for investigation.The service is taking automated snapshots every hour and keeps them securely for 14 days so your data should be safe within this time frame up to the last successful snapshot.CommentSharecpelvetanswered a year ago"


Question: Host name value in Custom Identity Provider

"Does anyone know if it is possible to get the hostname used to connect to the SFTP inside the Custom Identity Provider?The host name is something that could be used in the authentication process.Also, the documentation suggests that multiple hostname(s) are possible.  However, the examples I've seen suggest the config of only one.  Are multiple hostnames suppported for a single SFTP server?Thanks.FollowComment"

Answer: "0Accepted AnswerUnfortunately that information is lost by the time it reaches the Lambda that queries your Custom Identity Provider. Are you trying to use custom hostnames for namespaces? There may be other was by putting it as a part of the user name and separating by underscores or hyphens?CommentShareAWS-User-9826533answered 4 years ago0The hostname for your server created using a PUBLIC endpoint is:<server-id>.server.transfer.<region>.amazonaws.comso it looks like:s-xxxxxx.server.transfer.us-east-1.amazonaws.comIf you are using the Console, we create a CNAME record from your custom hostname to the server hostname above. To learn more about how we do this, refer to the docs below:https://docs.aws.amazon.com/transfer/latest/userguide/requirements-dns.htmlAlternatively you can map any custom hostname (and any number of them) yourself by pointing them to resolve to the server hostname above.Let me know if this answers your questionThanksCommentSharesrsmithaanswered 4 years ago0Thanks, that helps.Do you know if it is possible to get the custom hostname value used to connect to the server inside the Custom Identity Provider?CommentShareTTF2019answered 4 years ago0We work with multiple clients who would like the appearance of their own SFTP server and that custom hostname would serve as a sort of user context.  The user could potentially have access to different user contexts.  We would prefer that the same user (userid, password) could be used to access different sites.  But their home folder visibility under each one should be specific to the context they log in to.For example user1 has access to sftp.site1.com and sftp.site2.com.  We don't want the user to have to manage separate accounts/passwords for site1 and site2.  At the same time, we would rather avoid the user having the same home folder, and have to manually separate their files by using subfolders like /site1 and /site2.So the desired behaviour would be, the user logs into site1, the custom auth identifies the login was from site1 and maybe sets their home directory to be /site1/user1/.  If they log into site2, then similarily they would have a home directory of /site2/user1/.  While the user has access to both sites, their data should not really mix together, so that is why we were hoping to avoid simply using subfolders like /site1 and /site2 within a single home directory.Can you think of any way we could accomplish something like this?Edited by: TTF2019 on Apr 15, 2019 4:40 AMCommentShareTTF2019answered 4 years ago0In addition to accessing the hostname in the custom identity provider, we would see value in being able to identify the hostname that was used to upload a file.We will be using the Metadata 'user-agent-id' on the s3 object to determine who uploaded the file and it would be great if another Metadata field could include the hostname used for the upload.Heck, maybe the public IP address of the uploader could be added to the Metadata too?Maybe even better if the custom identity provider could return values during auth to be added to the metadata of uploaded files by that authenticated user.CommentShareTTF2019answered 4 years ago0That's an interesting use case of contextual access for an authenticated user. The details are helpful. If i'm understanding correctly any context would be helpful - even source IP address of the end user?CommentShareAWS-User-9826533answered 4 years ago0Yeah I started with hostname with the idea that a username could be unique, per hostname, rather than require a globally unique username.  With no other available info the identity provider has to maintain a single globally unique user pool of usernames.The IP would be of interest for audit logging, and possibly a manual implementation of Account locking or brute force attack monitoring, but I believe that AWS may already be working on whitelisting support?  If true, the IP might be of more interest in the s3 object metadata for a bit of an audit trail.CommentShareTTF2019answered 4 years ago0For anyone else that may be looking at this, it appears aws has added the ability to get the sourceIP value within the custom identity provider lambda function.https://aws.amazon.com/about-aws/whats-new/2020/06/aws-transfer-family-enables-source-ip-as-a-factor-for-authorization/It appears that getting the hostname is still not supported.CommentShareTTF2019answered 2 years ago"


Question: Clip stiching limits

"Hi!Is it possible to raise the number of allowed inputs to the job? I get the following error when creating a job with lots of inputs:'inputs' failed to satisfy constraint: Member must have length less than or equal to 200I need to handle more than 200 inputs and while I could do that in multiple passes, it would be a lot more convenient to do it with a single job.FollowComment"

Answer: "1Replying to myself. According to AWS support this is a hard limit.I received an answer from Elastic Transcoder (ETS) service team, and they said that ETS has a hard limit of 200 inputs for a single job. However, I'll create a feature request about it to AWS Elastic Transcoder Team, I wouldn't be able to provide an ETA regarding when them will decide to implement this. All the latest announcements can be found here: https://aws.amazon.com/new/ .CommentSharetszymczyszyn-hanswered 2 years ago0I am also dealing with this 200 inputs hard limit and would like to have it removed or at least much bigger.CommentShareSpasableanswered 2 days ago"


Question: File Storage Gateway temporary storage / option to delay uploads

"Hi,So a random one, is there a means to either have a temporary storage area on the the file gateway appliance on prem, or an option to delay file uploads to S3 by a set amount of time?The rational is that on prem we want to move data to the file gateway via NFS BUT also want to be prudent (as its archive data) and hash the file data after our internal transfer to ensure no corruption occurred as data transferred across our network (overkill - possibly, but the data is really important, so safe rather than sorry - and the average file size is large enough 50 - 100GB that its is a real possibility with some being hundreds GBs to TB range).Ideally we don't want to have data uploaded whilst hashing just in case there is an issue with it and our systems have to delete and re-copy the data.If there is a way to delay data being uploaded, or a means to have a temporary storage area no linked to a bucket from which good data can then be copied after the fact that would be awesome. It would also overcome the pitfalls of using rsync and an S3 bucket with version locking (which seems a bit of an oversight).Thanks in advance,Owen.FollowComment"

Answer: "0Accepted AnswerCurrently there is no provision to delay the upload. Since file gateway can support files larger than the size of the Cache disk, as soon as 5 MiB of data is available, it will start uploading that data.Your use case is interesting, we will consider that for a future consideration.CommentShareShashi-AWSanswered a year agoommps a year agoThank you for the info. It would be amazing if it could be added as like a system-wide user configurable option in the future for those who need just a bit of time between it hitting the cache disks and starting upload. And as mentioned it overcomes the 'issue' of rsync's temp files and S3 version locking.Share"


Question: To find the tags attached to a given AWS Resource at the OU Level

"Hi, I am trying to fetch the list of tags attached to an AWS resource when evaluated at the centralized level (OU level). Is there any direct way for implementing this problem set?If not, can you please help me the potential ways of achieving it. I also want an additional feature where I can download the csv format of the generated data rather than further cleaning and extracting the useful data out of ton of the information collected.Please helpFollowComment"

Answer: "0You can do this with AWS Config Advanced Query using the Organization Config Aggregator. See: Querying the Current Configuration State of AWS Resources.Something like this:SELECT  resourceId,  tagsWHERE  resourceId = 'AROAQCPVTFYGIWRJVM63L'You can then export the results to JSON or CSV.CommentShareEXPERTkentradanswered 6 months ago"


Question: Emails in WorkMail will not delete

"Hello All,New to the community here.  I've got a strange one here.  I have a email message that has duplicated itself countless times and will not delete from AWS WorkMail.  No matter how many I select, delete and empty the folder...they keep coming back.FollowComment"

Answer: "0Hi,I'm sorry to hear you're experiencing problems with deleting email. With what client did you try to delete the mails? Have you tried to delete the mails from another client?Kind regards,RobinCommentShareMODERATORrobinkawsanswered 7 months ago"


Question: S3 Bucket Keys and CRR

Has anyone been able to get S3 bucket keys to work with CRR? I tried enabling bucket keys (on both source and destination buckets) and my previously working CRR stopped working. I disabled bucket keys and CRR started working again. Documentation doesn't list any additional steps https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-key.htmlThanks!FollowComment

Answer: "0Accepted AnswerIf your replication role policy previously match by this field "kms:EncryptionContext:aws:s3:arn", you will need to update it so it's just the bucket ARN. See "Changes to note before enabling an S3 Bucket Key" on https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-key.html#bucket-key-changes - S3 Bucket Keys uses bucket ARN as encryption context, while previously KMS encryption uses object ARN. Also, you do not have to enable S3 Bucket Keys on both source and destination buckets for replication to work.CommentShareAWS-User-0154594answered 2 years ago"


Question: Working example of AWS::EC2::TransitGatewayMulticastDomainAssociation?

"I have run into something that I could use some help with: I'm trying to setup an Cloudformation template for a VPC with support for multicasting, and I have it except for the AWS::EC2::TransitGatewayMulticastDomainAssociation which forms the "last mile" connection.  That resource type seems to have some contradictory information in the documentation1, which has the property named SubnetId (singular) and String type that's described as holding multiple subnets. I need three subnets associated with the multicast domain and I'm not sure how to accomplish that.  As far as I can tell from searching github, there are no public examples of using this resource type.Help?FollowComment"

Answer: "0Accepted AnswerWhat if we create an "AWS::EC2::TransitGatewayMulticastDomainAssociation" for each subnet?  Association1:    Type: AWS::EC2::TransitGatewayMulticastDomainAssociation    Properties:       SubnetId: !Ref Subnet1      TransitGatewayAttachmentId: !Ref TransitGatewayAttachment      TransitGatewayMulticastDomainId: !Ref TransitGatewayMulticastDomain  Association2:    Type: AWS::EC2::TransitGatewayMulticastDomainAssociation    Properties:       SubnetId: !Ref Subnet2      TransitGatewayAttachmentId: !Ref TransitGatewayAttachment      TransitGatewayMulticastDomainId: !Ref TransitGatewayMulticastDomain  Association3:    Type: AWS::EC2::TransitGatewayMulticastDomainAssociation    Properties:       SubnetId: !Ref Subnet3      TransitGatewayAttachmentId: !Ref TransitGatewayAttachment      TransitGatewayMulticastDomainId: !Ref TransitGatewayMulticastDomainCommentShareEXPERTRiku_Kobayashianswered 4 days agomlhpdx 2 days agoYep, the documentation is misleading and it isn't clear that the TransitGatewayMulticastDomainAssociation type can be instantiated multiple times.  It didn't occur to be to try that for some reason so thanks for the hint.Share"


Question: AWS Authorized Instructor Requirements

"Hello,What do I require to qualify for AWS Authorized Instructor program. And is there any payment associated with it?I have 2 Certification - Advanced Networking and SA AssociateI am self employedThank you.Aman.FollowComment"

Answer: "3Accepted AnswerHi Aman,This link might help with your question: https://aws.amazon.com/training/aws-authorized-instructor-program/CommentShareSUPPORT ENGINEERLundianswered 6 months agoEXPERTkentradreviewed 6 months ago"


Question: API Gateway as Reverse HTTP Proxy to SQS

"I am trying to use AWS API Gateway as a reverse (forwarding) proxy to AWS SQS. I essentially want to send a REST request to the API Gateway which then gets forwarded directly to the SQS REST API and returns the response.When I send a request to the gateway, I immediately get back<?xml version="1.0"?>  <ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/">    <Error>      <Type>Sender</Type>      AccessDenied      <Message>Access to the resource https://sqs.us-east-1.amazonaws.com/ is denied.</Message>      <Detail/>    </Error>    <RequestId>51c903b2-4da3-5d5e-a3b8-589ee72167de</RequestId>  </ErrorResponse>However, when I switch the request URL to SQS directly (https://sqs.us-east-1.amazonaws.com) the request succeeds.What am I missing?curl --request POST 'https://my-api-gateway.com/sqs' \--header 'X-Amz-Date: <date>' \--header 'X-Amz-Security-Token: <token>' \--header 'Authorization: <auth>' \--header 'Amz-Sdk-Invocation-Id: <invocation>' \--header 'Amz-Sdk-Request: attempt=1; max=10' \--header 'User-Agent: aws-sdk-go-v2/1.16.5 os/macos lang/go/1.18.3 md/GOOS/darwin md/GOARCH/arm64 api/sqs/1.18.6' \--header 'Content-Length: 206' \--data-urlencode 'Action=ReceiveMessage' \--data-urlencode 'MaxNumberOfMessages=10' \--data-urlencode 'QueueUrl=<my-queue-url>' \--data-urlencode 'Version=2012-11-05' \--data-urlencode 'WaitTimeSeconds=20'Configuration:IntegrationsRoutesParameter MappingsFollowComment"

Answer: "1Accepted AnswerIn this case it is required to configure an IAM Role for API Gateway to assume with the required sqs:SendMessage permission as API Gateway will drop the Authorization header if it contains an AWS Signature. It's in the docs but well hidden under the header mapping table.The Authorization header is dropped if it contains a Signature Version 4 signature.CommentShareMatthias Pichleranswered a year agoaustinarbor a year agoThanks! I missed thatShare0Did you create a role for API Gateway that has a policy that allows API Gateway to interact with the SQS queue?CommentShareEXPERTIndranil Banerjee AWSanswered a year agoaustinarbor a year agojust curious but why would that be required? I am strictly using the REST API which is passing the authentication headers to SQS. The gateway itself shouldn't really know that it's calling SQS should it?ShareIndranil Banerjee AWS EXPERTa year agoIn this case the API Gateway receives the REST request but has to make an SQS API call behind the scenes, so it will need to be granted the access based on the principle of least privilege.Share"


Question: Eventbridge schema

"I have many glue jobs with similar job names, and I'm using Eventbridge to monitor these jobs, once a job failed, trigger lambda to send email.below is current schema, there're too many glue jobs, the length of json exceeded the limit, so I have to create 2 rules to cover all the jobs.Q: can I use regular expression in "jobName" ? or can I use another parameter instead of "jobName" to filter glue jobs?{  "detail-type": ["Glue Job State Change"],  "source": ["aws.glue"],  "detail": {    "jobName": ["glue_job_1", "glue_job_2", "glue_job_3", "glue_job_4", "glue_job_5".....],    "state": ["FAILED"]  },  "region": ["ap-southeast-1"],  "account": ["xxxxxxxx"]}Thanks for your answer!FollowComment"

Answer: "0First, if you do not need the job name, you can remove it from the rule. If you do need you can use a Prefix to catch all the job names that starts with some text, e.g., "jobName": [ {"prefix": "glue_job_" } ].You can find more information about the available event patterns here.CommentShareEXPERTUrianswered a year ago"


Question: Deploying large scale ML model

"Hi, I am deploying an ML model with a retrieval component from AWS and it had two parts:ML model that is deployed using Sagemaker. The model isn't big, so this is simple.Retrieval: The ML model first retrieves information from a database using ANN algorithm(like Annoy or Scann). The database needs to be loaded into memory at all times for really fast inference. However, the database is big(around 500GB). What is the best way to deploy this database? Is Sagemaker the best bet?FollowCommentSwwapnil a year agoCan you please clarify, What database are you using to store the data? What kind of data is stored in the database?ShareAlex_T EXPERTa year agoAlso I wonder if you could give an idea of how many times the DB would be queried for one model inference? Once? Many? The acceptable latency here might guide whether it's better/practical to have the endpoint call out to a separate DB service, or necessary to try and wedge everything into the endpoint container's RAM.Share"


Question: Can Pinpoint send a notification-message?

"I am currently developing an app using React Native. For the management screen, I am using Laravel and creating campaigns on Amazon Pinpoint through createCampaign of aws-sdk for PHP to deliver push notifications for both iOS and Android. At this point, I was able to display push notifications as banners successfully. However, I noticed that push notifications could not be received after killing the Android app. This also applies when the app is restarted and placed in the foreground. I have researched and found out that there are types of FCM push notifications that can be received when the app is killed and some that cannot. Is it impossible to send notification messages that can be received even after the app is killed via Pinpoint? I apologize for my amateur knowledge and poor expression, and would appreciate any help or wisdom.FollowComment"


Question: Unable to set Lambda memory over 3008MB

"I have a Lambda with the following configuration:Region: us-east-2 (Ohio)Runtime: nodejs12 (also tested nodejs14)Inside my accounts service quotas, I can see that my maximum function size is 10240 (10gb).However, attempting to update to anything beyond 3008MB results in the following error:'MemorySize' value failed to satisfy constraint: Member must have value less than or equal to 3008According to this post, Ohio is a supported region.I've tried setting this via Cloudformation as well as via the console, receiving the same error both times.Does anybody know what I'm doing wrong?Thanks!FollowCommentrePost-User-9925307 7 months agoHave the same issue.eu-central-1python 3.9ShareBasti 7 months agoSame issue here! eu-central-1Shareakali93 7 months agoHad the same issue in us-east-1. Tried again a few weeks later, it just worked. No idea what changedShare"

Answer: "2To give a more recent answer to this:I had the same error (13 month old account, fresh out of free tier, basic plan, Frankfurt / eu-central-1 region, Dotnet 6). The problem got resolved with the AWS support.Because technical support was not possible in my support tier, I filed a request in the Support Center as:Account & Billing, Service: "Account", Category: "Other Account Issues", Severity: "General Question".I have chosen "Chat" as communication method.After a short chat, a request was filed (at 9-10 am local time), and resolved (6am the following day).Now I can set the memory limit to 10,240MB without issues. Tested in Frankfurt and Ireland region.(Sorry for the lengthy answer, but maybe it saves someone a few minutes piecing together information)CommentSharerePost-User-7909596answered a month ago0Lambda functions with memory configuration greater than 3GB are currently unavailable for first time use in some regions. We are working on restoring this feature and mitigations are in progress. If you urgently require to use your function with memory greater than 3GB, please raise a support ticket and request that the limit get raised for your account.CommentShareEXPERTBrettski-AWSanswered a year agoakali93 a year agoHi, was this fixed? Im still getting this error in my free-tier account in us-east-1Sharenils 6 months agoThere is no way to request a quota increase through the service console since the "Function memory maximum" incorrectly shows the maximum of 10,240 megabytes already. Can you please provide instructions as to how one should go about contacting AWS support without paying for a support plan?Sharetsnobip a month agothis seems to still be a problem, and you cannot raise a support ticket without a support plan of course, this is quite a pain!Share0If your account quota shows 10GB, and you are getting this error, it could be a service glitch or capacity issue.You can either try again later, or choose another supported region and try again.CommentShareJason_Sanswered a year agoHarrison a year agoThanks Jason, I tried another region (us-east-1) and did not receive the error.  I also tried changing a modifying a different lambda in the problematic region and received the error again, so it's clearly something wrong with the region.  Thanks, I'm assuming I'll need to reach out to support on thisShareJason_S a year agoHarrison - as Brettski pointed out you can escalate to support and request a limit lift. Hope this get resolved soon.Share"