Question: Free 12 months AWS but Lightsail paid?

"I just signed up for the S3 12-month free tier. When going to start an instance, Lightsail says it will be free for 3 months, then begin charging. Which is it - 12 months or 3 months? I want to use whatever is free for 12 months, if that doesn't include Lightsail then I either need to install Linux through ssh or delete my AWS account. Thanks.FollowComment"

Answer: "0Lightsail bundles are only free for the first 3 months for certain types. Here is a link to the pricing page:https://aws.amazon.com/lightsail/pricing/CommentShareGeno_AWSanswered 10 months agoBman70 10 months agoAh OK I'm starting to get that it's a separate service, with different pricing. I was under the impression I had AWS services free for 12 months, and the way Lighsail popped up first thing made me think it was part of the free tier service. I'll have to set up a bucket without Lightsail.ShareShlomo Dubrowin EXPERT10 months agoAs you noted each service may or may not have a free tier and those tiers have different requirements. You can see all the free tier options at this page [1]. If you are interested in 12 months of free tier for Linux and an S3 Bucket, then you can look at an EC2 Instance and S3, both have a 12 month free tier option. See the link for details.[1] https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRank&all-free-tier.sort-order=asc&awsf.Free%20Tier%20Types=*all&awsf.Free%20Tier%20Categories=*allShare0Thanks I was wondering to know too. Is super more easy this way. but just 3 months.CommentSharelvillaltaanswered 5 months ago"


Question: AWS Ground Truth authorization errors on SOME images

"I've followed these tutorials:Amazon's - https://docs.aws.amazon.com/sagemaker/latest/dg/sms-getting-started.htmlMedium.com - https://medium.com/@bruceyanghy/end-to-end-guide-for-setting-up-aws-sagemaker-ground-truth-public-data-labeling-jobs-8732cf49b865I'm attempting to run a basic object detection (images with bounding boxes) Ground Truth labeling job with Mechanical Turk. I've tried multiple times, opening up any and all security on the S3 bucket containing the images. My past two attempts have been the closest, but still only manage 32/389 images labeled (357 Failed) and 62/389 images labeled (327 Failed). The images are exactly the same in both attempts.Checking the manifest file for clues, the only error I get is:InvalidArgument<Message>Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>null</ArgumentValue>I can't find any references anywhere as to what I need to change to make this work. I've tried with and without server-side encryption on the S3 bucket.How do I change this signature version?FollowComment"

Answer: -1answer is hereCommentSharepkanswered 6 months ago


Question: Is it possible to check the connection after setting the AWS DMS target endpoint as a server of another Cloud Service Provider?

"Is it possible to designate mariadb, which is operated on the naver cloud platform from AWS Oracle db as the target endpoint with AWS DMS? After that, can I access the server installed on the Naver Cloud Platform from aws database (oracle db) through aws dms?FollowComment"

Answer: 0if I understand correctly you are looking to replace dms task target from aws oracle to mariadb. please note dms target endpoint engine cannot be changed. you need to use a fresh endpoint and fresh task. but again we need to make sure test and run full load and cdcCommentShareAWS-User-8174525-subhashanswered a year ago


Question: What is the maximum supported throughput of a transit gateway

"Hi All,I wanted to know about the actual supported bandwidth by a single TGW!As per the document linked below, a TGW supports VPC attachments up to 5000 and TGW provides 50Gbps of throughput per VPC.So should I assume a TGW can give throughput of 5000x50Gbps = 250Tbps (Approx)?https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-quotas.htmlPlease help me in getting the clear information, it will help me designing the efficient architecture.FollowComment"

Answer: "2Hello,I agree to the above suggestion provided regarding reaching out to the Accounts team/AWS Premium Support Team for TGW.The reason being that Transit gateway works with multiple elements VPC, DX, VPN and hence there can be a lot of design implications & improvements that can be recommended to you by the AWS team to maximize your throughput without increasing the costs on your account.CommentShareSUPPORT ENGINEERAWS-User-Chiraganswered a year ago0Given the magnitude of the network you're designing I strongly recommend that you reach out to your local AWS account team (and Solutions Architect) to discuss your requirements.CommentShareEXPERTBrettski-AWSanswered a year agoAWS-User-787786 a year agoQuestion is specific to collective maximum supported throughput by TGW with multiple VPC attachments.ShareBrettski-AWS EXPERTa year agoAt the scale of the network you mention in the question there are many variables; please reach out to your account team as they can give you far more detailed information than can be shared in this forum.Share"


Question: The database process was killed by the OS due to excessive memory consumpti

"Hi,Our RDS instance crash daily, we got:The database process was killed by the OS due to excessive memory consumption. It is recommended to tune your database workload and/or parameter usage to reduce memory consumption.We have default parameters, why it is killed?Instance: wpaurora1FollowComment"

Answer: "0Hi,The above error message makes it clear that the instance is crashing due to "Out of memory" (OOM) error on the Aurora instance.To troubleshoot, I would recommend you to:Check the CloudWatch graph for "FreeableMemory" to understand the memory usage.Use monitoring tools like Enhanced monitoring which provides a process list (RDS, OS and MySQL processes) with details about CPU, memory etc consumed by each process (Note: EM is at an additional cost).I understand you are using default parameter group but sometimes the available memory may not be enough to handle the workload, in this case you would need to scale up your instance class (with more memory resource). You may choose from:[+]https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html#Concepts.DBInstanceClass.SummaryYou can consider enabling "slow_query_log" with appropriate "long_query_time". Then dig into the slow query logs for the exact time (before the restart) and review them with your DBA to see if there were any long running transaction active at the time of the issue. (Note: Logs take up space on the instance)Make use of Performance insights (if supported for the version/instance class) to identify the CPU bottle neck and troubleshoot performance issues.Please upgrade your cluster if running an old version of Aurora (as it could be a bug as well). You can refer the below link to see what improvements have been made in later versions:[+]For Aurora 5.6: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.11Updates.html[+]For Aurora 5.7: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20Updates.htmlI recommend you to upgrade your Aurora cluster to the latest version available which has all the bug fixes, major improvements.[+]https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.htmlCommentShareAWS-User-0226969answered 3 years ago"


Question: MEAN LightSail Instance Stopped Working After Changing to Static IP Adress

"HI Everyone,I desperately need your help. I have a MEAN instance in LightSail, it has been working fine for many months. I have changed the public ip address to static public address and I start receiving 503 error message.I use third party domain and I have updated A records too.I reboot, stop and start the instance many times but it did not workYour response is really appreciated.FollowCommentyensteveAWS 2 months agoHi, I'm sorry you are seeing this issue. Are you able to ping the static ip successfully? If so, the static ip should be working properly. You may want to contact your domain provider to confirm whether there are any additional steps necessary to integrate with the static ip.Share"

Answer: "0Hi ThereHave you checked the logs on the web server.  See Debug Apache Errors from the MEAN packaged by Bitnami for AWS Cloud guide.CommentShareEXPERTMatt-Banswered 2 months ago0Thank you for your responseI checked error log and I found the below errors, do you have any suggestions[proxy:error] [pid 1303:tid 139751247963904] (111)Connection refused: AH00957: http: attempt to connect to 127.0.0.1:4700 (localhost:4700) failed[proxy_http:error] [pid 1303:tid 139751247963904] [client 121.45.104.192:61372] AH01114: HTTP: failed to make connection to backend: localhost,CommentSharerePost-User-2166452answered 2 months ago"


Question: SES Production Assess

After granting SES production access I am facing the following issue on sending an email on the SES console or on a production website.(The message can't be sent because the account's ability to send email is currently paused.  Sending is paused for this account.)please guide.FollowComment

Answer: "0Hi,Check the SES console reputation metric is healthy, sometime it might be disabled, paused if bounce rate and complaint rate cross the limit, if diabled/pause you need to request support to enable.CommentSharesouravanswered a year agoExportminers 9 months agoNo. There is no way.I have the same problem. I gave them some new fresh information. I got nothing. Every time i add some information; the case appers as re-opened, after 2 minutes it will go automaticly as "SOLVED ".So answer is no. They are SES (!) , they are untouchables (!), they are unreachables...Share0No. There is no way.I have the same problem. I gave them some new fresh information. I got nothing. Every time i add some information; the case appers as re-opened, after 2 minutes it will go automaticly as "SOLVED ".So answer is no. They are SES (!) , they are untouchables (!), they are unreachables...CommentShareExportminersanswered 9 months ago"


Question: CSV crawler tables name

"Hi team,I have an issue with the glue crawler, when I leave in my S3 bucket only 1 CSV file and run the CSV crawler, it generates the crawled table with the name of the s3 bucket itself non with the name of my CSV file.When I have 1+ files in the bucket I have the crawled table that have the same names as the CSV files but if I have only 1 file on my s3 bucket the crawled table takes the name of the s3 bucket not of the crawled file.any idea to resolve that issue?ThanksFollowComment"

Answer: "0By default, if a prefix contains 1 or more files of similar schema, Crawler creates one table with name as that of parent prefix.However, if the crawler is not able to determine similar schema between the files under a prefix, it will create multiple tables each with name similar to the file name.CommentShareSUPPORT ENGINEERManu_Ganswered a year ago"


Question: Cloudwatch 'number' widget that displays number of active alarms

"Hi , I am trying to configure a simple cloudwatch 'number' widget that displays the total number of alarms currently in 'in-alarm' status ...and likewise another one that displays the total number of alarms in  'ok' status. (i will colour the widgets red and green accordingly) ...our OPs team need this very simple view as they are running out of screen real estate to display much elseMy question is, how can i get the widget to display this count of alarms? it seems widgets can only be based on metrics ...would i need to create a custom metric that somehow counts the alarms based on a query? (such as alarm status or a string match in the alarm name etc) ....is this even possible?...or would i have to have some type of lambda to do this?Any guidance would be greatly appreciated thanksFollowComment"

Answer: "1Hi, emitting a metric is one possibility. You would need though to create this metric in a custom way, e.g. call a lambda on a regular schedule (using eventbridge) to count your alarms and emit the number to a metric.You can also create a custom widget on your dashboard? You can write the same lambda that counts your alarms, the difference is the lambda would output the number in HTML to feed the custom widget, and you don't need to call your lambda on a schedule. As an optimization you can save the count from the lambda to not run the query every time a dashboard is displayed or refreshed but only at whatever refresh rate you want.CommentShareJscanswered 8 months ago0Thanks ...I found this really good article to help with the abovehttps://medium.com/@anatoliydadashev/aws-cloudwatch-alarms-count-in-grafana-53932f5bc901CommentSharerePost-User-6610418answered 8 months ago"


Question: my site refuses to work

I have a wordpress site that is on the route 53 dns and on an Ec2 instance that are all working normally. but despite this the site is not working for some timeFollowComment

Answer: "0https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-hosting-unresponsive-website/Also is it a single EC2 machine or are you using an Auto Scaling Group with an Application Load Balancer in front? If not, I would recommend doing sohttps://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.htmlhttps://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.htmlAlso if you haven't read this, it is recommended to go through this - https://docs.aws.amazon.com/whitepapers/latest/best-practices-wordpress/best-practices-wordpress.pdfCommentShareEXPERTIndranil Banerjee AWSanswered 10 months ago"


Question: OpsWorks support for Puppet Enterprise 2019.0

"Any timeline?I'm stuck on PE 2018.1.2, which does not include an Ubuntu 18.04 agent.FollowComment"

Answer: "0Accepted AnswerAWS OpsWorks for Puppet Enterprise now supports Puppet Enterprise version 2018.1.7. With this upgrade, Ubuntu 18.04 is supported. AWS OpsWorks for Puppet Enterprise will automatically update the Puppet master server to 2018.1.7 during the weekly system maintenance window.CommentShareAWS-User-1476813answered 4 years ago0I'll take it! Thanks guys, your efforts are appreciated.CommentShareJonInVAanswered 4 years ago"


Question: How to setup alerts when PHP is updated by AWS

My website went down as AWS updated PHP at their end and I didn't receive any advance communication about it. Is there a way to set alerts of configure when to accept these updates?FollowComment

Answer: "0If you're referring to Elastic Beanstalk (?), you can define a maintenance window for platform updates where you can also configure to receive notification when an upgrade is executed. The following pages in the documentation provide information on these features:https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environment-platform-update-managed.htmlhttps://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.eventbridge.htmlPlease note that this is applicable only if Managed Updates are enabled. Notifications for the event that an update for the platform is available are currently not supported.CommentShareDanielanswered 6 months ago"


Question: Need AWS API for fetching Access Level for the aws service actions

Is there an API for fetching Access Level for the AWS service actions.For Example: iam:ChangePassword has a write Level access.Ref: https://docs.aws.amazon.com/service-authorization/latest/reference/list_identityandaccessmanagement.htmlFollowComment

Answer: 0There is no API for fetching Access Level for the AWS Service actions. But you can see the Access Level of policy you created by accessing the policy using AWS management console. Please refer to "Viewing service summaries" in below link.https://docs.amazonaws.cn/en_us/IAM/latest/UserGuide/access_policies_understand-service-summary.htmlCommentShareEunsu_Shinanswered 6 months ago


Question: Amazon Connect Time Zone for India Standard Time (IST)

"What time zone should hours of operation assigned for use in India use? India has it's own time zone defined at GMT+5:30. This is not available in the list provided. While there is an entry for Asia/Calcutta, I'm unsure if this actually matches as Calcutta was known to have a separate time zone at one time.FollowComment"

Answer: "3ou can safely use the Asia/Calcutta time zone for your Amazon Connect hours of operation in India. The Asia/Calcutta time zone in Amazon Connect corresponds to the current India Standard Time (IST), which is GMT+5:30.The city of Calcutta, now known as Kolkata, does not have a separate time zone. The time zone for Kolkata is the same as IST, and Amazon Connect uses the older city name for historical reasons. Many systems, including IANA Time Zone Database, still use the "Asia/Calcutta" identifier for compatibility purposes.So, you can confidently use the Asia/Calcutta time zone for your Amazon Connect instance to set up hours of operation in India.CommentShareEXPERTsedat_salmananswered 2 months agoEXPERTAWS-samfredreviewed 2 months ago"


Question: Bug in RDS migrate snapshot feature

I am trying to create an RDS instance from a backup from AWS console. But it seems the functionality is broken.I see this error in the network error log.<ErrorResponse xmlns="http://rds.amazonaws.com/doc/2014-10-31/">  <Error>    <Type>Sender</Type>    InvalidParameterCombination    <Message>Cannot find version 10.0.17 for aurora-mysql</Message>  </Error>  <RequestId>12ff6fc8-f796-4215-8826-719174b9c358</RequestId></ErrorResponse>FollowComment


Question: What is "No Tenancy" in Cost Explorer?

"I was getting two very different sets of numbers in Cost Explorer between the Cost Explorer details linked from the EC2 Reserved Instances Recommendations, and doing my own filtering based on known instance types in tags in another window (costs and hours in my window were about double).The filter that made them match up turned out to be Tenancy, where the RI page was specifically specifying Shared Tenancy, but I had no filter on, which was including both Shared and No Tenancy.What does No Tenancy mean? I can find info on Dedicated, Shared and Default, but not "No Tenancy".Note, there are no dedicated hosts being used.FollowComment"

Answer: "1Accepted AnswerEC2 has "%BoxUsage%" related charges that will have one of your tenancy options listed. EC2 has other charges (like data transfer) that will be classified as "No Tenancy". You can first group by tenancy, get the total for "No Tenancy", then group by Usage Type, and add up all the non "%BoxUsage%" charges and you will find them to be equal.CommentShareAWS-User-3911429answered 3 years ago"


Question: "ImageChecker" fails to launch - "Line 1812. Security error."

"Workspaces: Custom image based on "Windows 10 Experience provided by Windows Server 2016"Pending Windows updates? NoNecessary free diskspace on C and less then 10GB for the userprofile on the D-drive? YesRebooted Workspace? YesExecutionPolicy? BypassHello,This Workspace have worked before to create images from and the last time it worked was May 2020.However now when I was going to create a custom image with minor modifications:Updated through Windows Update ("winver" reports Windows Server 2016 Ver 1607 Build 14393.3866)Modified an already existing PS-scriptand now image creation fails with:"Error during validation of Workspaces image"Trying to run "imagechecker.exe" located in C:\Program Files\Amazon does... nothing.The application does not start at all, it gives an UAC-prompt but then nothing happens.Tried both running it using CMD (w. and w/o running CMD as admin) and executing the exe directly.The imagechecker-log shows this:"[Error] (ImageChecker.exe) Line 1812. Security error."I tried going through this list here for reasons why image creation fails:https://docs.aws.amazon.com/workspaces/latest/adminguide/create-custom-bundle.html#image_checker_tipsbut I can't figure out why it fails and why even the "imagechecker.exe" doesn't start.I never used "imagechecker" before so I can't give you any history that it has worked ever since I never used it before.Tried downloading it using the dedicated URL if the download of the exe was corrupted but no change.EventViewer > "Applications and Services Logs" > "Amazon_Workspaces" shows me the same as the manual log located in C:\Program Files\Amazon i.e:"[Error] (ImageChecker.exe) Line 1812. Security error."That is the last line in the log.Tried to find out exactly what that error means but its just the general error of "Security error" occurring on line 1812 and I haven't found any documentation or forum posts.Running this script:C:\Program Files\Amazon\WorkspacesConfig\scripts\imagePrepValidation.ps1simply returns "True"the script seems to be the real backend that "imagechecker" possibly provides a GUI / frontend to and contain various functions checking that all requirements are met but all I get returned by the script is "True".Need to delve deeper into the script at a later stage since its quite late here.Would be grateful for some clarification of the error message and/or help on how to resolve the issue.Thanks in advance!FollowComment"

Answer: "0This is still an issue, tried to download the "imagechecker.exe" but it still behaves as my old post:I don't get any GUI / window at all but I can see that a log-file is createdThat log ends with:2021-02-17 17:02:30Z [Error] (ImageChecker.exe) Line 1812. Security error.Any ideas or anything anyone has encountered?Would REALLY appreciate an answer.KrgdsCommentShareTheSwede86answered 2 years ago0HelloThe same issue, tried from multiple workspaces.Tried looking into Event viewer if you get any more app info there but says the same as the logI am curious what Line - 1812 is because that would most likely narrow down possible culpritsCommentSharetcvall86answered 2 years ago0I seeing the same issue.[Error] (ImageChecker.exe) Line 1812. Security error.CommentShareJV1answered 2 years ago0I have the same error, has this been resolved??2021-03-25 20:16:01Z [Information] (ImageChecker.exe) Successfully downloaded and validated file 'imagePrepValidation.ps1' under 'updates/scripts/casigned' from server '198.19.64.253' to 'C:\Program Files\Amazon\WorkspacesConfig\scripts\imagePrepValidation.ps1'.2021-03-25 20:16:01Z [Error] (ImageChecker.exe) Line 1812. Security error.CommentShareraskov64answered 2 years ago"


Question: SAP Enable now integration with Amazon Polly

"SAP Enable Now (SEN) an SAP Product, is a learning content development tool. They have many training courses developed and currently maintained using this tool. To improve and enhance SAP training content customer would like to be able to use Amazon Polly  to add voice to the courses. Text-to-speech (TTS) is an assistive technology in which a computer or tablet reads the words on the screen out loud to the user. This technology is essential for learners who have difficulties with reading, especially those who struggle with decoding.Do we have Amazon polly integration with SEN ?FollowCommentAWS-User-0741373 4 months agoYou can use Amazon polly api's to build integration with the application. There is no off the shelf integration with SEN.You can check out with our partner e-zest, they provide polly integration services for our customers.https://www.e-zest.com/amazon-web-services-aws-polly-integration-serviceshttps://docs.aws.amazon.com/polly/latest/dg/API_Operations.htmlShare"


Question: API Gateway Dynamic routing

"My customer has a multi-tenant application that wants to route all tenant traffic to a single endpoint. Based on content in the request header the request would be routed to the appropriate tenant's endpoint similar to a how a Lambda authorizer works but to change routing. The tenant id/route mapping can change over time, so would need to be looked up in something like DynamoDB. What is the best way to accomplish this?FollowComment"

Answer: "0Accepted AnswerYou can do this using Lambda@Edge using CloudFront. The Lambda function could use a Route 53 hostname in it to make it more resilient and easy to modify the weights + hostnames of DCs.To add -If the incoming request doesn't have that specific header, you can let Lambda@Edge do the routing based on the routing_logic. Once the request is mapped to a backend, you would want to stick a cookie (session/persistent) in the browser with that Origin value.When subsequent requests come to CloudFront, your logic would be like:{   IF:       Request Header: A   THEN:      Go to, Origin = A}{   IF:       Request Header: B   THEN:      Go to, Origin = B}Even for the above snippet that I shared, my vote would be to 'not' hardcode A and B Origin hostnames in Lambda functions and still create R53 failover properties, and use them in there. The advantage is that; if A goes down, the failover property would automatically send the requests to the failover target.CommentShareImEavesDroppinanswered 3 years ago"


Question: You are currently not allowed to send this e-mail. Please try again later.

"2022-05-24T11:40:42.292+05:30Copy{"accountId": "231935139027","organizationId": "m-8ef9cd271dc24eaea718b459f9cbaae6","traceId": "d5ded513f32c2f87ee09300bbd46657b","eventTime": 1653372642292,"event": {"bouncedRecipient": {"status": "5.0.0","action": "Failed","emailAddress": "cetinalihan@domain.com","reason": "You are currently not allowed to send this e-mail. Please try again later."},"eventName": "OUTGOING_EMAIL_BOUNCED"}}I am using Email Client to send emails, But emails are bounced, and when I check log files, the above issue was listed in that.Pls help. How we can fix this issue.FollowComment"

Answer: "0Hi,If you're using a custom domain you can still be over the allowed sending limit. Have you been sending a lot of mail? I can be that your account has been limited due to abuse concerns. If this is the case please check you Personal health dashboard in your AWS account for a reach out message.Kind regards,RobinCommentShareMODERATORrobinkawsanswered a year agoMTarun a year agoThank you Robin, For your Time and Reply.The personal health dashboard has no entry, which means nothing in that.. all clean.Is there any panel where I can check the allowed sending limit ( WorkMail allowed sending limit for my domain or Organization )Sharerobinkaws MODERATORa year agoHi,No there is no way to verify what your limit is. If you're entitled to support I would suggest to create a support case to see what account limits are applied.Kind regards,RobinShareMTarun a year agoHello Robin,Thank you for your Time and Reply.Yes, I want to know what are my account limitations.Looking forward to your reply with the required details.Kind regards,TarunShare0Hi,I'm sorry to hear you're experiencing problems sending mail. This error is returned when you have reached the sending limit of your account. Are you using a custom domain or the free awsapps.com domain? The free awsapps.com domain has a very low limit.Kind regards,RobinCommentShareMODERATORrobinkawsanswered a year agoMTarun a year agoThank you for your Time and Reply.Que# Are you using a custom domain or the free awsapps.com domain? The free awsapps.com domain has a very low limit.Reply# No we are using our own domains, And all domains are verified, All records are added in the DNS zone to verify # Domain ownership, WorkMail configuration, Improved security, and Improved email delivery.Que # What is sending limit per hour for the WorkMail account?Share0Thank you for your Time and Reply.Que# Are you using a custom domain or the free awsapps.com domain? The free awsapps.com domain has a very low limit.Reply# No we are using our own domains, And all domains are verified, All records are added in the DNS zone to verify # Domain ownership, WorkMail configuration, Improved security, and Improved email delivery.Que # What is sending limit per hour for the WorkMail account?CommentShareMTarunanswered a year ago"


Question: The proxy IP we operate is registered in AWS WAF's ip reputataion.

"It is being used for a normal service purpose, but a request to the AWS website is not possible.Is it possible to request an IP release?please tell me howFollowComment"

Answer: "0I understand the frustration of not being able to access your website via your proxy IP address. An Amazon IP reputation list managed rule groups allow you to block requests based on their source IP address that are typically associated with bots or other threats. While it is not possible for you to have your proxy IP address released there are ways to allow a specific IP address or addresses, using one of the following methods to resolve this problem: Scope-down statements to narrow the scope of the requests that the rule evaluates. Choose this option for addressing logic in a single rule group. Or, you can use Labels on web requests to allow a rule that matches the request to communicate the match results to rules that are evaluated later in the same web ALC. Choose this option to reuse the same logic across multiple rules.https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-ip-rep.html#aws-managed-rule-groups-ip-rep-amazonhttps://docs.aws.amazon.com/waf/latest/developerguide/waf-rule-scope-down-statements.htmlhttps://docs.aws.amazon.com/waf/latest/developerguide/waf-labels.htmlConsidering your use case, creating a IP Set on WAF would be the most convenient solution. You can allow traffic from your proxy IP while still utilizing the IP reputation list managed rule group. Please follow these steps:Sign in to the AWS Management Console and open the WAF console at https://console.aws.amazon.com/wafv2/In the navigation pane, choose IP sets, and then choose Create IP set.Enter an IP set name and Description - optional for the IP set. For example: MyTrustedIPs. Note: You can't change the IP set name after you create the IP set.For Region, choose the AWS Region where you want to store the IP set. To use an IP set in web ACLs that protect Amazon CloudFront distributions, you must use Global (CloudFront).For IP version, choose the version you want to use.For IP addresses, enter one IP address or IP address range per line that you want to allow in CIDR notation. Note: AWS WAF supports all IPv4 and IPv6 CIDR ranges except for /0. Examples: To specify the IPv4 address 192.168.0.26, enter 192.168.0.26/32. To specify the IPv6 address 0:0:0:0:0:ffff:c000:22c, enter 0:0:0:0:0:ffff:c000:22c/128. To specify the range of IPv4 addresses from 192.168.20.0 to 192.168.20.255, enter 192.168.20.0/24. To specify the range of IPv6 addresses from 2620:0:2d0:200:0:0:0:0 to 2620:0:2d0:200:ffff:ffff:ffff:ffff, enter 2620:0:2d0:200::/64.Review the settings for the IP set. If it matches your specifications, choose Create IP set.In the navigation pane, under AWS WAF, choose Web ACLs.For Region, select the AWS Region where you created your web ACL. Note: Select Global if your web ACL is set up for Amazon CloudFront.Select your web ACL.In the web ACL Rules tab, choose the specific AWS Managed Rule group that is blocking your request, and then choose Edit.Choose Rules, and then choose Add Rules, Add my own rules and rule groups.For Scope-down statement - optional, choose the Enable scope-down statement.For If a request, choose doesn't match the statement (NOT).On Statement, for Inspect, choose Originates from IP address in.For IP Set, choose the IP Set you created earlier. For example: MyTrustedIPs.For IP address to use as the originating address, choose Source IP address.Choose Save rule.https://aws.amazon.com/premiumsupport/knowledge-center/waf-allow-ip-using-reputation-anon-list/Please contact if you have any further questions, and feel free to reach out to us via a support case to facilitate a discussion on the specifics of your resourceshttps://console.aws.amazon.com/support/home#/case/create?issueType=technicalCommentShareKevin_Janswered a year ago"


Question: Error pulling image from ECR through Internet proxy

"Hi, we are getting an error pulling images from ECR private repositories from a server that get access to Internet through a web proxy. We ask proxy admin to allow access to <ACCOUNT_ID>.dkr.ecr.us-east-2.amazonaws.com. We are using aws cli to authenticate with amazon and docker, docker login is successful with aws token, however when we pull the image we get an error like this, after some timeouts reported by docker trying to get image layers:error pulling image configuration: download failed after attempts=6: dial tcp 52.219.102.138:443: i/o timeoutDocker pull command is something like:docker pull <ACCOUNT_ID>.dkr.ecr.us-east-2.amazonaws.com/<repository>:<TAG>Appreciate your help, what are we missing ? Is there some additional domain/endpoint we have to enable access on web proxy configuration ? Do we need add  https://api.ecr.us-east-2.amazonaws.com/ to web proxy allowed targets ? Something else ?Appreciate your help.FollowComment"

Answer: "0Need to communicate with the following endpoints.dkr.ecr.us-east-2.amazonaws.comapi.ecr.us-east-2.amazonaws.coms3.us-east-2.amazonaws.comAlso consider using VPC Endpoints.https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.htmlCommentSharehayao-kanswered 8 months agoAdorian 8 months agoThanks, your help is appreciated. We validated successfully we can reach these endpoints with curl, however we get the same error when tried to pull the image. Getting the ECR token and therefor docker login to ECR is successful. However image pull fails miserably with a timeout error.We noted that s3 https://s3.us-east-2.amazonaws.com endpoint returns a redirect to https://aws.amazon.com/s3/ , we follow the redirect successfully also.Is like docker pull operation use another endpoint/domain, or get in conflict with proxy. We ensure also proxy env variables targeting to we proxy were set before all tests.Appreciate your help.Best !Share"


Question: Protect Public Restful APIs with API Gateway and oauth2.0

I am working on Airbnb like project. There are Public RESTful APIs that need to be secured with API Gateway and oauth 2.0 I want a solution to secure the public RESTful APIs with OAuth 2.0. ThanksFollowComment

Answer: "2Accepted AnswerHi zeeshan.First of all you need to decide whether you are gonna use an API Gateway of type REST or HTTP. https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-vs-rest.html.This is because they provide different security mechanisms:Rest: https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-control-access-to-api.htmlHttp: https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-access-control.htmlYou can check the possibilities above, but in general, in terms of Oauth2 support you should look into Lambda Authorizer (max level of control), Cognito pools and JWT Authorizers.Other security mechanisms to be taken into account are about adoption of WAF (Web Application Firewall). Even in that case, it will depend on Rest vs Http. In general, you can attach a WAF directly to an API GW of type REST, while for HTTP you can achieve only if you have CloudFront in front of the api (as edge-optimized api) or an Application Load balancer.Finally you should consider different levels of protections  and possibly Packet Inspection and/or Detection. See a reference here https://aws.amazon.com/blogs/networking-and-content-delivery/scaling-network-traffic-inspection-using-aws-gateway-load-balancer/ which leverages Gateway Load Balancers to perform IPS scanning for internet ingress traffic.Hope it helps ;)CommentShareEXPERTalatechanswered 3 months ago"


Question: Cloud 9 environment updating title and description fail to persistt

"Hi,When I try to update the descriptive text for an environment, I get the message "Environment was successfully updated," however, nothing changes. I have tried several times, logged out and in again.Please adviceFollowComment"

Answer: 0Hi tovare!Thanks for reporting this and sorry to hear that you have issues.We identified the problem and working on a fix for it.I'll keep this thread updated.CommentShareanester-awsanswered 3 years ago


Question: How to reuse existing resources with cloudformation yaml?

"I want to make a secret in secretsmanager by using cloudformation. Since I want to add the secret value manually and keep the value everytime I run the cloudformation, I specified "Retain" policy as follows.SecretsManagerMyApiKey:    Type: AWS::SecretsManager::Secret    DeletionPolicy: Retain    Properties:      Name: my-api-keyHowever, if I delete the stack once and run the cloudformation again, I get the following error.The operation failed because the secret my-api-key already exists.How can I avoid this behavior and reuse the existing secret key?I know that I can reuse existing resources as follows with CDK python. What I need is a way to do like this with yaml. Please let me know.sg = ec2.SecurityGroup.from_security_group_id(            self,"SG",            security_group_id=sgid,            mutable=False        );FollowComment"

Answer: "0Hi, @nemy.How would you like to use existing secrets?If you want to refer to another resource, use the resource's ARN or ID.When referencing resources in CloudFormation, parameters are simply obtained and used according to the specification of each resource.For example, AWS::SecretsManager::Secret will get the ARN by "Ref".https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-secretsmanager-secret.html#aws-resource-secretsmanager-secret-return-valuesBy using a custom function, I think that it is possible to create something that obtains an ARN string from a resource name that is not managed by the stack.If you want to manage it as a CloudFormation stack, consider incorporating existing resources into the CloudFormation stack by referring to the following.https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.htmlCommentShareEXPERTiwasaanswered 9 months agonemy 9 months agoThank you for the answer.How can I define a resource in the template by using ARN of an existing resource?What I want to do is something like below.'''MyBucket: "ARN of the existing bucket"MyPipeline:Type: AWS::CodePipeline::PipelineProperties:RoleArn: !GetAtt CodePipelineRole.ArnName: my-pipelineArtifactStore:Type: S3Location: !Ref MyBucket'''If I simply write ARN instead of "!Ref MyBucket", I get the following error.'''1 validation error detected: Value at 'pipeline.artifactStore.location' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9-.]+ (Service: AWSCodePipeline; Status Code: 400; Error Code: ValidationException;'''.Share"


Question: "After running a yum update on Centos 8, instance is not working."

"Hello,I just ran an update. It ran out of memory doing so. It said: Cannot allocate memory. I do not know if it completed the update or not so I re-ran yum update and get the following: Config error: [Errno 30] Read-only file system: '/var/log/dnf.log': '/var/log/dnf.log'Googling this error, it appears it may be a hardware failure. Obviously I can't fix that.I use this instance to run MySQL for my website. My website is now down because of this.Need AWS tech support!!ThanksFollowComment"


Question: Athena recursive cte NOT_SUPPORTED: Recursion depth limit exceeded (10). Use 'max_recursion_depth'

"Hi,Trying to use a recursive cte with Athena and I get this error :NOT_SUPPORTED: Recursion depth limit exceeded (10). Use 'max_recursion_depth' session property to modify the limit.How would I set this session property from the console ?Thanks!FollowComment"

Answer: 0Using the WITH clause to create recursive queries is not supported.https://docs.aws.amazon.com/athena/latest/ug/select.html#select-parametersPlease share your SQL if you are not using WITH clause.CommentShareMilindOkeanswered 3 months ago


Question: Glue Tables - Tag Parameters in PartitionKeys

"Hello,I'm creating a table following the documentation boto3, but when I run the create_table command, I get the error An error occurred (InvalidInputException) when calling the CreateTable operation: Parameters not supported for partition columns.In the AWS documentation , the parameters tag inside the partitionKeys is supported but I tried to create the tag in the aws console and got the same error.I used a code similar to this using the boto3 APIimport boto3glue = boto3.client('glue', region_name='us-east-1')table_input = {    'Name': 'dummyTable',    'Description': 'Test parameters on partitionKeys',    'TableType': 'EXTERNAL_TABLE',    'StorageDescriptor': {        'Columns': [                      {                        "Name": "id",                        "Type": "int",                        "Parameters": {                          "type": "int",                          "index": "true"                        }                      },                      {                        "Name": "name",                        "Type": "string"                      },                      {                        "Name": "age",                        "Type": "bigint"                      },                      {                        "Name": "city",                        "Type": "string"                      }                    ],        'Location': "s3://my-bucket/dummytable",        "Parameters": {            "classification": "parquet",            "typeOfData": "file"        },        'InputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat',        'OutputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat',        'SerdeInfo': {'SerializationLibrary': 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'        }    },    'PartitionKeys': [            {                'Name': 'active',                'Type': 'int',                'Comment': 'Test Partition',                'Parameters': {                    'index': 'True'                }            },        ]}createTable = glue.create_table(        DatabaseName='sample',        TableInput=table_input)FollowComment"

Answer: "0Hello,I have tested this and can confirm that adding Parameters under PartitionKeys is throwing below error.Error: botocore.errorfactory.InvalidInputException: An error occurred (InvalidInputException) when calling the CreateTable operation: Parameters not supported for partition columnsI am going to further test this and will update soon.ThanksNiharCommentShareNIHAR-AWSanswered 2 months ago0Hi,This seems to be a documentation issue. The PartitionKeys object is an array of Column objects, which contain the Parameters property. This property can be set in other Column objects, but not when that Column represents a partition column in a table, according to our code. It looks like our documentation was made with the knowledge that columns can have parameters - but not knowing this limitation. We support parameters on normal table columns for the Lake Formation Michigan use cases - so we are going to open discussion as to whether this capability should be opened everywhere. In the meantime, we have put in a request to change the documentation to reflect what is currently possible.ThanksNiharNote : Please upvote/accept my answer if it helped you.CommentShareNIHAR-AWSanswered 2 months ago0Ninar, do you think we can use this parameter in future updates?CommentSharemarcusanswered 21 days ago"


Question: Lua Script not working

"I am attempting to make my own moving mechanics in my own project (which i have built and added all the gems i desire) and i have the following code: (i have made the Input Mapping files)local move ={Properties ={movementSpeed = .03}}function move:OnActivate()self.forwardId = InputEventNotificationID("Forward")self.ForwardBus = InputEventNotificationBus.Connect(self, self.forwardId);self.backwardId = InputEventNotificationID("Backward")self.BackBus = InputEventNotificationBus.Connect(self, self.backwardId);self.leftId = InputEventNotificationID("Left")self.LeftBus = InputEventNotificationBus.Connect(self, self.leftId);self.rightId = InputEventNotificationID("Right")self.RightBus = InputEventNotificationBus.Connect(self, self.rightId);endfunction move:OnHeld(floatvalue)if(InputEventNotificationBus.GetCurrentBusId() == self.forwardId)thenTransformBus.Event.MoveEntity(self.entityId, Vector3(0,self.Properties.movementSpeed,0));endendThe problem is that it doesn't work in game and i get these errors:general.enter_game_modeReturned:[Warning] (Cloud Canvas) [Cloud Canvas] Resource manager was invoked but there are no deployments or resource groups.Disable Accelerators[Warning] Access to undeclared global variable 'InputEventNotificationID'---------- START STACK TRACE ---------- =[C](-1): ?code/movement.lua(14): ? ---------- END STACK TRACE ----------code/movement.lua:14: attempt to call global 'InputEventNotificationID' (a nil value)---------- START STACK TRACE ---------- =[C](-1): InputEventNotificationIDcode/movement.lua(14): ? ---------- END STACK TRACE ----------[Error] code/movement.lua:14: attempt to call global 'InputEventNotificationID' (a nil value)---------- START STACK TRACE ---------- =[C](-1): InputEventNotificationIDcode/movement.lua(14): ? ---------- END STACK TRACE ----------2 animation-assets loade`general.exit_game_modeReturned:general.exit_game_modeReturned:code/movement.lua:42: attempt to index field 'InputEventNotificationBus' (a nil value)---------- START STACK TRACE ---------- =[C](-1): ?code/movement.lua(42): ? ---------- END STACK TRACE ----------[Error] code/movement.lua:42: attempt to index field 'InputEventNotificationBus' (a nil value)---------- START STACK TRACE ---------- =[C](-1): ?code/movement.lua(42): ? ---------- END STACK TRACE ----------Enable Accelerators2 animation-assets loaded (total assets: 1)and :Traffic control: We did not received any packets for the last second 1498067124084!Traffic control: We did not received any packets for the last second 1498067125095!Traffic control: We did not received any packets for the last second 1498067126072!Connection 000001892B2C5DF8 rtt 31.41 ms (max. 500.00) and packetLoss 0.32 (max. 0.30). Disconnecting...[928a9f08] Disconnecting 127.0.0.1|58566 to due to bad traffic conditions!Disconnect 192.168.0.13|50205 => (127.0.0.1|58566,Bad connection)Member 000001893D412108(Peer) leaving TM session 000001899230BDC8.Lost contact with node from member c43b3364.Disabling debugging for script context(0x0000018967A42578).Remote debugger LuaIDE has detached from context 0x0000018967A42578.general.exit_game_modeReturned:general.exit_game_modeWhat am I doing wrong? Help would be appreciated!Follow"

Answer: "0Accepted Answerit should be "InputEventNotificationId" not "InputEventNotificationID"SharerePost-User-9018410answered 6 years ago0Fixed it! I forgot to use this code for every movement since right now it only does it for forward. Thanks for your help!:    if(InputEventNotificationBus.GetCurrentBusId() == self.forwardId)thenTransformBus.Event.MoveEntity(self.entityId, Vector3(0,self.Properties.movementSpeed,0));SharerePost-User-8843326answered 6 years ago0@REDACTEDUSERcode/movement.lua:42: attempt to index field 'InputEventNotificationBus' (a nil value)---------- START STACK TRACE ---------- =[C](-1): ?code/movement.lua(42): ? ---------- END STACK TRACE ----------[Error] code/movement.lua:42: attempt to index field 'InputEventNotificationBus' (a nil value)---------- START STACK TRACE ---------- =[C](-1): ?code/movement.lua(42): ? ---------- END STACK TRACE ----------<br>SharerePost-User-8843326answered 6 years ago"


Question: Question about Control Tower aws-controltower-SecurityNotifications

Hello is it possible to set a subscription filter policy on the topic to just get the "complianceType": "NON_COMPLIANT" messages. Im lost in this filter policies.tried{   "complianceType": [     "NON_COMPLIANT"   ] }but that didnt work.FollowComment

Answer: 0No suggestions?CommentShareAlexanswered 10 months ago


Question: Monitor Linux services via CloudWatch

I have installed CWAgent and have everything monitored accept for specific service running on my RHEL 7.9 instances.  How do I configure cloudwatch to monitor a specific service?FollowComment

Answer: "0Hi,If you have installed CWAgent, you can configure procstat section. Like that, you can target a specific process.https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-procstat-process-metrics.htmlCommentShareDonovanswered a month ago0You can publish a custom metric to CloudWatch in the form of a "heart beat".Have a small script running via cron on your server checking the process list to see whether my_service is running and if it is, make a put-metric-data call to CloudWatch.The metric could be as simple as pushing the number "1" to your custom metric in CloudWatch.Set up a CloudWatch alarm that triggers if the average for the metric falls below 1Make the period of the alarm be >= the period that the cron runs e.g. cron runs every 5 minutes, make the alarm alarm if it sees the average is below 1 for two 5 minute periods.Make sure you also handle the situation in which the metric is not published (e. g. cron fails to run or whole machine dies). you would want to setup an alert in case the metric is missing. (see here: AWS Cloudwatch Heartbeat Alarm)Be aware that the custom metric will add an additional cost of 50c to your AWS bill (not a big deal for one metric - but the equation changes drastically if you want to push hundred/thousands of metrics - i.e. good to know it's not free as one would expect)See here for how to publish a custom metric: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/publishingMetrics.htmlCommentShareAWS-User-6621777answered a month ago"


Question: Connecting a Linux box to AWS-VPN using OKTA Authentication/Authorization

"First of all, a rookie, related to VPN/Security issues, so really forgive me for whatever error I make while describing my problem, and hope I'm able to make it clear.Our contractors changed AVIATRIX-OKTA VPN for AWS-VPN with OKTA Authentication, they send as an .ovpn file, that works ok for Windows/MAC using AWS-Vpn-Client application software, but a couple of us using Linux boxes (Ubuntu specifically) run the described method in AWS which is:openvn config-file.ovpnbut it fails to authenticate.It simply asks for usr/pwd an then it fails with auth error (we use our OKTA credentials) , seems nothing is configured to go to OKTA, open a browser or whatever it needs to do.As an aside note, we can connect without any trouble to our aws k8s cluster using OKTA client libraries, no sure is this is useful or not, just in case.The .ovpn file looks like thisclientdev tunproto tcpremote random.cvpn-endpoint-xxxxxx.yyy.clientvpn.us-west-2.amazonaws.com 443remote-random-hostnameresolv-retry infinitenobindpersist-keypersist-tunremote-cert-tls servercipher AES-256-GCMverb 5<ca>............</ca>auth-user-passauth-federateauth-retry interactauth-nocachereneg-sec 0An interesting thing to notice is that openvpn complains about auth-federateseems not to recognize it,  so I started using gnome network-manager which seemsto accept this configuration, but getting Auth error too.After this I tried openvpn3 which didn't complain about configuration,but still getting the same error.Any help on how to configure it, or just know if it is possible, will be greatly welcome , seems there is very little information around this in the net and we are really stuck on this, we are willing not to change OS or machines as they are asking to, or using VM just to connect.Thanks in advance,FollowComment"

Answer: "0Finally I got an answer from AWS people:If the Client VPN endpoint is configured using SAML-based authentication (such as Okta), then you > > have to use the AWS-provided client to connect:https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/client-authentication.html#saml-requirementsAnd the promise to update the client documentation with a WARNING about this.CommentSharetjcanswered 3 years ago0I hope for a Linux client..The general theme of getting Linux client stuff from AWS is getting better, they provided the NOSQL workbench in Linux form recently.CommentShareStevenHarperanswered 3 years ago0Definitely, not sure why there is not, okta is kind of linux friendly openvpn no need to explain, so why there is not one, is probably the question.Just to mention, previously as I stated, we used aviatrix, which provided a token that we can use as a password wehen askled and use standard OVPN configuration without any trouble, why not a simple similar solution.Thanks Steven for triggering this response :-)Edited by: tjc on Jun 26, 2020 6:32 AMCommentSharetjcanswered 3 years ago"


Question: API Gateway User Authentication Best Practices

"Hello Everybody!  As a best practices question, how would you recommend providing authentication for an API gateway endpoint exposed to the internet?  Think S3 website, HTML form, JavaScript form handler making a request to API Gateway backed by a Lambda function.  I'm very interested to read about the techniques you're using.FollowComment"

Answer: "1Integrate AWS API Gateway with Web Application Firewall to prevent OWASP Vulnerabilities. Use Predefined or create Custom rules based on your regulatory requirements.Lambda authorizer functions for controlling access to API methods using token authentication(JWT Validation).Enforce API Keys/Tokens to the API Users and implement API access plans and rate limiting by leveraging AWS API Gateway OOB functionalities.Create an Amazon API Gateway Private API with AWS VPC Lambda proxy integration.https://serverlessland.com/patterns/cdk-private-api-lambdaEnable real time security event monitoring with proper audit trails and logs . Raise alerts/Notification in case of deviation from normal access patternsCommentShareArunkumaranswered a year ago0API Gateway supports multiple mechanisms for controlling and managing access to your API.You can use the following mechanisms for authentication and authorization:Resource policies let you create resource-based policies to allow or deny access to your APIs and methods from specified source IP addresses or VPC endpoints.Standard AWS IAM roles and policies offer flexible and robust access controls that can be applied to an entire API or individual methods. IAM roles and policies can be used for controlling who can create and manage your APIs, as well as who can invoke them.IAM tags can be used together with IAM policies to control access.Endpoint policies for interface VPC endpoints allow you to attach IAM resource policies to interface VPC endpoints to improve the security of your private APIs.Lambda authorizers are Lambda functions that control access to REST API methods using bearer token authentication—as well as information described by headers, paths, query strings, stage variables, or context variables request parameters. Lambda authorizers are used to control who can invoke REST API methods.Amazon Cognito user pools let you create customizable authentication and authorization solutions for your REST APIs. Amazon Cognito user pools are used to control who can invoke REST API methods.Refer to Controlling and managing access to a REST API in API GatewayCommentShareRoBanswered a year agoJackson a year agoI think OP question is what would be the best practice given the use case mentioned.Share"


Question: AmazonLinux 2022 Package availability seems really lacking

"Hello,I am trying out AL2022 RC, and on thing that strikes me as odd is that many packages have been removed.For example, mlocate and libreswan have been removed and are nowhere to be found.This seems very odd as those packages are available by default on debian 22.04, RH8, etc, but are a real pain to install on amazonlinux2022.The only way I found to add them was downloading the rpm manually but this would cause a lot of headache to maintain for a multitude of production servers.Are there additional repositories for extra packages available for al2022 that I might have missed ?Is there a chance the list of removed packages could be reconsidered ?Thanks a lot for your help,XavierFollowComment"

Answer: "0Hello Xavier ,The packages you mentioned are provided in the list of the packages removed as seen here -- [1]However, Amazon Linux 2022 is in preview release, and the features, packages, and package versions that are included in the preview release are subject to change. Before Amazon Linux 2022 is released for general availability, we will make changes and improvements to the preview version.During the preview, we're actively seeking your feedback about what to add to and modify in Amazon Linux 2022. We also have a clear roadmap moving forward.Feedback on Amazon Linux 2022 can be provided through your designated AWS representative or by filing an issue in the amazon-linux-2022 repo on GitHub - https://github.com/amazonlinux/amazon-linux-2022/issues -- [2]  Hope this helps !![1] -- https://docs.aws.amazon.com/linux/al2022/release-notes/removed-packages-al2022.htmlCommentShareSUPPORT ENGINEERAWS-User-Chiraganswered 9 months ago"


Question: Cannot delete because cluster <cluster name> currently has an update in progress

"I have deployed an EKS cluster with a node group using Terraform a little bit ago but now when I want to delete it, I get the error "Cannot delete because cluster my-cluster currently has an update in progress"There is no update in progress from what I can tell, meaning that there is no update to stop and no way to delete my cluster now. Both Terraform and the AWS console show the same error and I have no idea how to delete it.There are no more nodes, autoscalers, load balancers, network interfaces or even a VPC associated with the cluster anymore. The cluster shows up as "Active".I did perform updates on the cluster without syncing Terraform though. The VPC-CNI needed updating and the node group version as well. Terraform had that synced before deleting, so that shouldn't've been an issue.How do I delete my cluster?FollowComment"

Answer: "0Accepted AnswerAfter waiting for a little bit, I am not sure what it was that was updating, but for some reason that finished and now I can delete the cluster. Very odd, but alright. Question resolved itself I suppose..CommentShareDutchEllieanswered a year ago"


Question: AWS Amplify Data Modeling - is there an option for enumerations?

"Are enumerations available in AWS Amplify Studio data modeling? Below is a simplified example of something I am trying to do.enum Example {     optionA(char)     optionB(AnotherEnum)}enum AnotherEnum {...My workaround at the moment is to make the Example from above a custom type that contains optional fields pointing to a String or an enum AnotherEnum. It seems to be working, so it is not a critical issue, but curious to know any workarounds.FollowComment"


Question: Unable to Connect to Cloud9 Environment

"Hello,I have a Cloud9 environment, the environment has been running fine for months, today I encountered that I can not login anymore.The EC2 instance exist and Cloud9 cand start the instance, I can also connect to the instance through Systems Manager but when I try through the Cloud9 service it fails.Also the Status of the Cloud9 Env shows as "Connecting" and never change.Does someone know what I can check? Are there any log in the EC2 about cloud9 ?I'm trying to find out what is failing as the instance has Internet access and everything was fine before.Thank you.FollowComment"

Answer: "0How about scaling up your instance ? Lack of performance could be the cause.Scale up your instance (e.g. t2.small to t2.medium)Scale up your EBS volumeor you can check Troubleshooting AWS Cloud9.https://docs.aws.amazon.com/cloud9/latest/user-guide/troubleshooting.html#troubleshooting-connectionCommentSharefu3ak1answered 4 months agoHigher 4 months agoThanks for your answer, I have check the Instance size and the CPU is idle, also the EBS is big enough and reamins with low utilization. I will review again the Troubleshooting but I would like to know if there is a log in the EC2 that I could check, the AWS Console error is too generic.Share"


Question: How do I restrict AWS secret by Lambda alias?

"I want to have a QA secret and a Production secret, and I want a Lambda QA alias to only be able to access the QA secret, and likewise with the Lambda Production alias and the Production secret.Each Lambda function alias uses a single execution role, set at the function level. So any IAM policies attached to this role would apply to all aliases. This means that the QA alias lambda could access the Production secret. So an execution-role based policy does not appear to be the answer.I am trying to understand how to apply resource-based policies to the secrets, but it appears that I cannot choose the Principal to be a fully-qualified ARN to the Lambda alias. Is it possible to do this?Unless I am misusing these systems, it seems impossible to keep secrets separate and secure with Lambda aliases.FollowComment"

Answer: "2I am not sure if it is possible or not, but I would recommend that you do not use aliases for this purpose but rather separate into two (or more) different environments. At least dev and prod. This way you reduce the blast radius of mistakes happening in your dev account, e.g., consuming all account's concurrency which may affect production.CommentShareEXPERTUrianswered a year agoAWS-User-4223059 a year agoYour point about blast radius is a good one. However, I'm not ready to move to that architecture yet, but would like to increase security. iwasa had an interesting discovery in the second part of their answer, which is that execution roles can be changed at a per-version level. I'm already changing the docker image per-version when I deploy, so adding this is not much more work, and it enables what I am looking for.Share1Accepted AnswerHi.I saw this question and tried various things.But I couldn't.Unfortunately, neither Lambda's Execution Role nor Secret's Rolebase Policy seem to be able to include a specific ARN (eg alias) as a condition.If you want to limit with IAM roles, I think you should separate QA and Production at the function level and above, as @Uri advises.Alternatively, Lambda has a different Execution role for each version.You can indirectly separate roles if the alias points to another version.This also enables role-based secret access control.CommentShareEXPERTiwasaanswered a year agoAWS-User-4223059 a year agoInteresting! I didn't realize that the Lambda roles are inherited/instantiated similar to how the container image on a Lambda is changed: you change it at the function level, then create a new version from that function template. In theory, I could do the same thing: change the role AND the docker image, then create the new version from that function template, then re-point the alias. Since this actually solves the question that I had, I'm marking this as the answer, though Uri does make good points about blast radius.Share0If you need to restrict secrets to certain functions, you need to create a new role. You can't restrict by alias IIUC.{  "Effect": "Allow",  "Action": ["secretsmanager:GetSecretValue"],  "Resource": "arn:aws:secretsmanager:ap-southeast-1:${ACCOUNT_ID}:secret:/${ENVIRONMENT}/secretA"},Typically each role would have some ${ENVIRONMENT} variable in it too btw, usually isolated to an account.CommentSharekaianswered a year agoAWS-User-4223059 a year agoThat won't work, since a Lambda function's role applies to all of its aliases. Giving that role access to a production secret would allow the QA lambda to access it too.Share"


Question: Step Function - dynamodb query with Boolean Filter

"Using Step function AWS SDK integration for querying dynamodb table with key condition and filter condition, below is the query works fine outside of step-function. When trying to save the state machine after adding this, get the below Error.Ddb Query{  "TableName": "test",  "ScanIndexForward": true,  "KeyConditionExpression": "pk = :pk",  "FilterExpression":"scheduled = :scheduled",  "ExpressionAttributeValues": {    ":pk": {      "S": "test"    },    ":scheduled": {      "BOOL":true    }  }}** Error Saving State Machine **There are Amazon States Language errors in your state machine definition. Fix the errors to continue.    The field "BOOL" is not supported by Step FunctionsFor more information, see Amazon States LanguageFollowComment"

Answer: "2Accepted AnswerThe optimized service integrations accept BOOL, however there is a difference between how AWS SDK integrations handle request parameters. If you switch from "BOOL" to "Bool" it will work. All parameters for AWS SDK integrations must be in PascalCase.Source:The API action will always be camel case, and parameter names will be Pascal case. For example, you could use Step Functions' startSyncExecution API action and specify the parameter StateMachineArn.https://docs.aws.amazon.com/step-functions/latest/dg/supported-services-awssdk.htmlCommentSharewongadaanswered a year ago1According to this, you should change":scheduled": {       "BOOL":true     }to":scheduled": trueDid not try it.CommentShareEXPERTUrianswered a year agomuthumv a year agoThanks for your suggestion. But unfortunately this doesn't work as its javascript. Looks like internally the step-function uses JAVA SDK. Was able to figure this out from the error:There are Amazon States Language errors in your state machine definition. Fix the errors to continue.Cannot construct instance of `software.amazon.awssdk.services.dynamodb.model.AttributeValue$BuilderImpl` (although at least one Creator exists): no boolean/Boolean-argument constructor/factory method to deserialize from boolean value (true)For more information, seeHere is the Java doc for Filter Expression. Its like chicken and egg situation here:Java sdk for dynamodb accepts only a map of <String, AttributeValue> ({"pk":{"S":"data"}, "scheduled":{"BOOL":true}}in State Language "BOOL" is a reserved keyword probably and not being acceptedShare-1Querying is not supported in SFN, just get, put, delete, and update.CommentSharejamessouthanswered a year agoUri EXPERTa year agoQuery is supported by StepFunctions.Sharejamessouth a year agooops i was looking at optimized integrations. sorry!Share"


Question: NoCredentialsError: Unable to locate credentials when using dask_ml inside sagemaker

"I'm trying to use dask in sagemaker because I have over 1B+ rows in a single dataset. Creating a dask.dataframe works fine, when I create a client through dask, it also works:client = Client(n_workers=6, threads_per_worker=20)clientHowever when I try to use dask_ml.preprocessing.Categorizer, I get the error NoCredentialsError: Unable to locate credentials.I understand the issue might be dask distributed client doesn't have authorization to sagemaker client. Its confusing but how do I have them connect somehow? Code below:import dask.dataframe as dddf3 = dd.read_parquet('s3://bucket/parquetfiles2/data_*.parquet',                     storage_options={'token': 'anon'})if __name__ == "__main__":    obj_df = df3.select_dtypes(include=['object','datetime64'])    num_df = df3.select_dtypes(exclude=['object','datetime64'])    ce = Categorizer()    obj_df_1 = ce.fit_transform(obj_df)FollowComment"

Answer: "0Hello,Thank you for contacting us and for using Amazon Sagemaker.I understand that you encountered a "NoCredentialsError: Unable to locate credentials" when trying to use dask_ml.preprocessing.Categorizer.It looks like you're running the code locally on your machine. Please feel free to correct me if I have misunderstood anything here. The error is usually seen when you don’t have AWS credentials correctly configured on your machine.However, I see you're using Dask Distributed client to use Amazon SageMaker Processing. Please have a look here : https://sagemaker-examples.readthedocs.io/en/latest/sagemaker_processing/feature_transformation_with_sagemaker_processing_dask/feature_transformation_with_sagemaker_processing_dask.html#Build-a-Dask-container-for-running-the-preprocessing-job where we have built Dask enabled containers for SageMaker Processing.You might need to run aws configure so set up IAM Credentials (User) on your machine. [1] However, to be able to replicate and investigate into this further, we'd need your IAM role arn and other details. Hence, for further investigation on this issue, I recommend to cut a support case and provide more detail about your account information and script/config. Due to security reason, we cannot discuss account specific issue in the public posts.Please open a support case with AWS using the link:https://console.aws.amazon.com/support/home?#/case/createThank youReferences :[1] https://aws.amazon.com/premiumsupport/knowledge-center/s3-locate-credentials-error/[2] https://youtu.be/UMUQs2PojdECommentShareSUPPORT ENGINEEROjasvi_Banswered a year ago"


Question: Connect EventBridge Bus directly to a EventBridge Pipes as a Source

How do I connect an EventBridge Bus directly to an EventBridge Pipes as a Source.  So EventBridge Bus -> EventBridge Pipes -> Enrichment (Lambda) -> Pipes Target Event Pattern -> Target (Lambda).  As far as I can tell by the documentation and console ops I can only select Steaming services as Pipes Sources.  Is this a limitation that is fixed forever?The scenario I was wanting to implement was my EventBridge Bus events being enriched with feature flag detail pre-populated based on identity and detail-type and to discourage target services making any tightly coupled call(s) to feature flag service.  I thought EventBridge Pipes sound best idea as no code would have to be written to plum messages along the "Pipeline" just the Lambda code to enrich messages.One possible work around I was planning to try was to setup my pipeline.  EventBridge Bus -> Rule Event Pattern (*) -> Lambda Target (enriches events based on data from DynamoDb Table w/ Cache) and then code to push events to a second EventBridge Bus -> EventBridge Bus -> Rule Event Pattern(s) -> Target(s).Would love expert suggestions for alternatives or maybe that this is a planned feature change.ThanksFollowComment

Answer: "1Accepted AnswerHi,You can set Eventbridge bus to send to SQS and Eventbridge Pipe to read from SQS. That way you keep your intended flow with the introduction of one hop ie SQS--SydCommentShareSydanswered 3 months ago"


Question: AWS Backup and software level backup

"Hi,I have EC2 mail server and AWS backup is used to backup my entire AWS EC2.Do I still need to use software level backup?FollowComment"

Answer: "2If you have tested restoration and it works for you, then you do not. The true test of a backup is to restore to it and see that it works as you need it to.CommentShareRodney Lesteranswered 9 months ago"


Question: Following code Pipeline starter tutorial. CodeDeployAgent throwing ERROR

"HI, I am following the tutorial here: https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-simple-s3.htmlBut the deployment fails. Where I run status on the CodeDeployAgent Service using:sudo service codedeploy-agent statusI get this error:`internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:151:in require': cannot load such file -- net/smtp (LoadError)         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:151:in require'from /opt/codedeploy-agent/vendor/gems/logging-1.8.2/lib/logging/appenders/email.rb:2:in <top (required)>'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:151:in require'from internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:151:in require'         from /opt/codedeploy-agent/vendor/gems/logging-1.8.2/lib/logging/appenders.rb:57:in module:Logging'from /opt/codedeploy-agent/vendor/gems/logging-1.8.2/lib/logging/appenders.rb:2:in <top (required)>'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:151:in require'from internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:151:in require'         from /opt/codedeploy-agent/vendor/gems/logging-1.8.2/lib/logging.rb:537:in module:Logging'from /opt/codedeploy-agent/vendor/gems/logging-1.8.2/lib/logging.rb:18:in <top (required)>'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:162:in require'from internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:162:in rescue in require'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:152:in require'from /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/log.rb:2:in <top (required)>'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:99:in require'from internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:99:in require'         from /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager.rb:9:in <top (required)>'from internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:162:in require'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:162:in rescue in require'from internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:152:in require'         from /opt/codedeploy-agent/lib/instance_agent.rb:10:in <top (required)>'from internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:88:in require'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:88:in require'from /opt/codedeploy-agent/bin/../lib/codedeploy-agent.rb:22:in <main>' <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:151:in require': cannot load such file -- logging (LoadError)Did you mean?  loggerfrom internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:151:in require'         from /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/log.rb:2:in <top (required)>'from internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:99:in require'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:99:in require'from /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager.rb:9:in <top (required)>'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:162:in require'from internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:162:in rescue in require'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:152:in require'from /opt/codedeploy-agent/lib/instance_agent.rb:10:in <top (required)>'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:88:in require'from internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:88:in require'         from /opt/codedeploy-agent/bin/../lib/codedeploy-agent.rb:22:in <main>'internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:88:in require': cannot load such file -- process_manager (LoadError)         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:88:in require'from /opt/codedeploy-agent/lib/instance_agent.rb:10:in <top (required)>'         from <internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb>:88:in require'from internal:/usr/share/ruby3.2-rubygems/rubygems/core_ext/kernel_require.rb:88:in require'         from /opt/codedeploy-agent/bin/../lib/codedeploy-agent.rb:22:in <main>'`Has anyone experienced this before.FollowCommentMassimilianoAWS EXPERT2 months agoPlease provide additional information on the OS version you are running?Share"


Question: IoT Rule SQL error for reserved topic

"I'm happy to see that AWS added the connection status of devices & LoRa gateways.I was excited to use a rule to process those messages downstream, but can't figure out how.The connect/disconnect events are published here:$aws/iotwireless/events/connection_status/+/lorawan/wireless_gatewaysBut when making the rule the following query:SELECT * FROM $aws/iotwireless/events/connection_status/+/lorawan/wireless_gatewaysThrows an error:SqlParseExceptionUnexpected character '$' $aws/iotwireless/events/connection_status/+/lorawan/wireless_gateways --------------^ at 1:15Please tell me that this is possible and I'm just not understanding...FollowComment"

Answer: "0Whoops, my mistake.  Looks like I had a simple formatting issue.  Forgot the quotes... Should have been.SELECT * FROM '$aws/iotwireless/events/connection_status/+/lorawan/wireless_gateways'CommentShareAWS-User-9404308answered a year ago"


Question: Translating a SUMIF equation from Excel to WorkDocs

"I am having trouble translating the WorkDocs equivalent to the excel formula =SUMIF(E14:U14,"<>#N/A"). Any suggestions are greatly appreciated.FollowComment"


Question: IAM AUTH Too many requests

We have a Django app running in lambda. It connects to the RDS Database through RDS PROXY Using IAM Auth. When we are doing load testing after certain load we start getting error saying Too many requests to the IAM AUTH service. When we tried creating a new AWS RDS PROXY without IAM AUTH configuration our load tests performed much better. But we wanted to check what is the best and scalable architecture. Should we remove the IAM AUTH and keep the communication direct BW the application and the IAM AUTH or there is a better way to do this. Also is there a document I can follow to understand the interaction between the RDS PROXY and IAM AUTH.FollowComment

Answer: "0When using IAM authentication with RDS Proxy, you might encounter rate limits on the AWS Security Token Service (STS) when generating temporary credentials under high load. This can cause the "Too many requests" error that you're experiencing.To reduce the number of requests to the IAM authentication service, you can implement caching of IAM credentials in your Django application. Store the generated credentials and use them until they're close to expiration. This way, you'll minimize the number of calls to the STS service, which can help you avoid rate limitsIf caching IAM credentials does not provide the desired performance, you can consider using database credentials with RDS Proxy instead of IAM authentication. This approach might not be as secure as IAM authentication, but it can help you avoid rate limits and improve the overall performance of your application under load.: Regardless of whether you use IAM authentication or database credentials, implementing connection pooling and reusing database connections in your Django application can help improve performance. This can help reduce the overhead of establishing new database connections and the associated IAM authentication requests.If the "Too many requests" error persists, you can try contacting AWS Support to request an increase in the rate limits for your account.CommentShareEXPERTsedat_salmananswered 2 months ago"


Question: DynamoDB .Net SDK query timed out

"I am using the AWS .Net SDK to access DynamoDB in my alexa lambda function.Frequently, (about an hour every day), I get a query timed out error.  This is a query that runs every minute from CloudWatch Events and it fetches the value from a table that has only one row.Most of the time it works fine and returns in milliseconds.  However, several times during the day it times out.I have tried increasing and decreasing the timeout period, but it does not help.How do I resolve such errors ?  What is causing this ?Will it help if I have my DynamoDB within the same VPC endpoint as the lambda ?FollowComment"

Answer: "0Accessing DynamoDB from Lambda is not a problem as long as the IAM role is set up, so no VPC endpoints are required.VPC endpoints are needed when Lambda is running inside a VPC.Your Lambda is not running in a VPC, is it?If Lambda is running in a VPC, it is necessary to check the configuration around the network as it may be incorrect.CommentShareEXPERTRiku_Kobayashianswered 9 days ago"


Question: DNS not resolving correctly

"We had a load balancer with one target group rotating between 3 private IPS.Then used haproxy to tunnel traffic in a round robin way to one of the 3 instances using the nlb something like this in the haproxy config:backned our-backend    server          nlb        our-nlb-amazonaws.com:62015 check resolvers awsresolverBut this morning this stopped working, using dig from the haproxy(and locally) it resolves to 3 totally different ips then the ones in the target group.We've fixed this for now by bypassing the load balancer and listing the 3 instances in the haproxy directly which works fine for now, but it's not the best since deploying or adding more of those instances would mean that this config would have to change as well. This is a bit fragile and the previous approach is preferred.FollowComment"

Answer: "0Hi njichev,I understand your traffic flow is as follows:Client ----> HaProxy ------> NLB --------> Three targetsSo based on your notes, you ran a dig to the NLB DNS and you saw three IP addresses that are not of the targets. Please correct me if I misunderstood.When running a dig to the ELB DNS, the IP Addresses that appears belong to the NLB nodes and not of the targets, hence you saw different IPs  of the targets . For more information on NLB IPs, you can refer to the following document: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.htmlEnlisting the IP addresses of the targets, it changes your traffic flow as Client ----> HaProxy ---> Three targets. Your HaProxy hits the target directly without going through the load balancer. I would like to suggest, enlisting the NLB DNS name back and then troubleshoot the NLB issue.You can refer to the following document on troubleshooting issues: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.htmlCommentShareWinnieanswered a year ago"


Question: AWS Glue Job: Error calling o92.pyWriteDynamicFrame

"I would like to resolve an error for job run id:  jr_d67e4c6d24e9feb7e071bc5a2989be9acdce32bbea82596b8f953e96808c3d5eThe error:Traceback (most recent call last):  File "/tmp/ga-overview-gluejob", line 52, in <module>    datasink1 = glueContext.write_dynamic_frame.from_catalog(frame = transform1, name_space = "test-hud", table_name = "ga_overview", transformation_ctx = "datasink1")  File "/opt/amazon/lib/python3.6/site-packages/awsglue/dynamicframe.py", line 657, in from_catalog    return self._glue_context.write_dynamic_frame_from_catalog(frame, db, table_name, redshift_tmp_dir, transformation_ctx, additional_options, catalog_id)  File "/opt/amazon/lib/python3.6/site-packages/awsglue/context.py", line 298, in write_dynamic_frame_from_catalog    return DataSink(j_sink, self).write(frame)  File "/opt/amazon/lib/python3.6/site-packages/awsglue/data_sink.py", line 35, in write    return self.writeFrame(dynamic_frame_or_dfc, info)  File "/opt/amazon/lib/python3.6/site-packages/awsglue/data_sink.py", line 31, in writeFrame    return DynamicFrame(self._jsink.pyWriteDynamicFrame(dynamic_frame._jdf, callsite(), info), dynamic_frame.glue_ctx, dynamic_frame.name + "_errors")  File "/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in call    answer, self.gateway_client, self.target_id, self.name)  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco    return f(**a, **kw)  File "/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value    format(target_id, ".", name), value)py4j.protocol.Py4JJavaError: An error occurred while calling o92.pyWriteDynamicFrame.: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 38, 172.36.108.132, executor 1): org.apache.spark.api.python.PythonException: Traceback (most recent call last):  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 377, in main    process()  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 372, in process    serializer.dump_stream(func(split_index, iterator), outfile)  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 345, in dump_stream    self.serializer.dump_stream(self._batched(iterator), stream)  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 141, in dump_stream    for obj in iterator:  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 334, in _batched    for item in iterator:  File "<string>", line 1, in <lambda>  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 83, in <lambda>    return lambda **a: toInternal(f(**a))  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/util.py", line 99, in wrapper    return f(**args, **kwargs)  File "/tmp/ga-overview-gluejob", line 35, in <lambda>TypeError: strptime() argument 1 must be str, not intat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:462)at com.amazonaws.services.glue.sinks.HadoopWriters$$anonfun$com$amazonaws$services$glue$sinks$HadoopWriters$$writeNotPartitioned$1.apply$mcV$sp(HadoopWriters.scala:121)at com.amazonaws.services.glue.sinks.HadoopWriters$$anonfun$com$amazonaws$services$glue$sinks$HadoopWriters$$writeNotPartitioned$1.apply(HadoopWriters.scala:121)at com.amazonaws.services.glue.sinks.HadoopWriters$$anonfun$com$amazonaws$services$glue$sinks$HadoopWriters$$writeNotPartitioned$1.apply(HadoopWriters.scala:121)at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)at org.apache.spark.sql.glue.SparkUtility$.tryWithSafeFinallyAndFailureCallbacks(SparkUtility.scala:39)at com.amazonaws.services.glue.sinks.HadoopWriters$.com$amazonaws$services$glue$sinks$HadoopWriters$$writeNotPartitioned(HadoopWriters.scala:125)at com.amazonaws.services.glue.sinks.HadoopWriters$$anonfun$doStreamWrite$1.apply(HadoopWriters.scala:138)at com.amazonaws.services.glue.sinks.HadoopWriters$$anonfun$doStreamWrite$1.apply(HadoopWriters.scala:129)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:121)at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Driver stacktrace:at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)at scala.Option.foreach(Option.scala:257)at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)at org.apache.spark.SparkContext.runJob(SparkContext.scala:2114)at com.amazonaws.services.glue.sinks.HadoopWriters$.doStreamWrite(HadoopWriters.scala:129)at com.amazonaws.services.glue.sinks.HadoopDataSink$$anonfun$writeDynamicFrame$1.apply(HadoopDataSink.scala:258)at com.amazonaws.services.glue.sinks.HadoopDataSink$$anonfun$writeDynamicFrame$1.apply(HadoopDataSink.scala:148)at com.amazonaws.services.glue.util.FileSchemeWrapper$$anonfun$executeWithQualifiedScheme$1.apply(FileSchemeWrapper.scala:66)at com.amazonaws.services.glue.util.FileSchemeWrapper$$anonfun$executeWithQualifiedScheme$1.apply(FileSchemeWrapper.scala:66)at com.amazonaws.services.glue.util.FileSchemeWrapper.executeWith(FileSchemeWrapper.scala:58)at com.amazonaws.services.glue.util.FileSchemeWrapper.executeWithQualifiedScheme(FileSchemeWrapper.scala:66)at com.amazonaws.services.glue.sinks.HadoopDataSink.writeDynamicFrame(HadoopDataSink.scala:147)at com.amazonaws.services.glue.DataSink.pyWriteDynamicFrame(DataSink.scala:63)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)at py4j.Gateway.invoke(Gateway.java:282)at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)at py4j.commands.CallCommand.execute(CallCommand.java:79)at py4j.GatewayConnection.run(GatewayConnection.java:238)at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 377, in main    process()  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 372, in process    serializer.dump_stream(func(split_index, iterator), outfile)  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 345, in dump_stream    self.serializer.dump_stream(self._batched(iterator), stream)  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 141, in dump_stream    for obj in iterator:  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 334, in _batched    for item in iterator:  File "<string>", line 1, in <lambda>  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/worker.py", line 83, in <lambda>    return lambda **a: toInternal(f(**a))  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/util.py", line 99, in wrapper    return f(*args, **kwargs)  File "/tmp/ga-overview-gluejob", line 35, in <lambda>TypeError: strptime() argument 1 must be str, not intat org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:81)at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.read(PythonUDFRunner.scala:64)at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:462)at com.amazonaws.services.glue.sinks.HadoopWriters$$anonfun$com$amazonaws$services$glue$sinks$HadoopWriters$$writeNotPartitioned$1.apply$mcV$sp(HadoopWriters.scala:121)at com.amazonaws.services.glue.sinks.HadoopWriters$$anonfun$com$amazonaws$services$glue$sinks$HadoopWriters$$writeNotPartitioned$1.apply(HadoopWriters.scala:121)at com.amazonaws.services.glue.sinks.HadoopWriters$$anonfun$com$amazonaws$services$glue$sinks$HadoopWriters$$writeNotPartitioned$1.apply(HadoopWriters.scala:121)at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)at org.apache.spark.sql.glue.SparkUtility$.tryWithSafeFinallyAndFailureCallbacks(SparkUtility.scala:39)at com.amazonaws.services.glue.sinks.HadoopWriters$.com$amazonaws$services$glue$sinks$HadoopWriters$$writeNotPartitioned(HadoopWriters.scala:125)at com.amazonaws.services.glue.sinks.HadoopWriters$$anonfun$doStreamWrite$1.apply(HadoopWriters.scala:138)at com.amazonaws.services.glue.sinks.HadoopWriters$$anonfun$doStreamWrite$1.apply(HadoopWriters.scala:129)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)at org.apache.spark.scheduler.Task.run(Task.scala:121)at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)... 1 moreThe script:import sysfrom awsglue.transforms import *from awsglue.utils import getResolvedOptionsfrom pyspark.context import SparkContextfrom awsglue.context import GlueContextfrom awsglue.dynamicframe import DynamicFramefrom pyspark.sql.functions import *import pyspark.sql.functions as Ffrom pyspark.sql.types import DateTypefrom datetime import datetimefrom awsglue.job import JobGet job nameargs = getResolvedOptions(sys.argv, ['JOB_NAME'])Creat Glue and Spark ContextsglueContext = GlueContext(SparkContext.getOrCreate())spark = glueContext.spark_sessionjob = Job(glueContext)job.init(args['JOB_NAME'], args)EXTRACT data from redshift tabledatasource0 = glueContext.create_dynamic_frame.from_catalog(database = "test-hud", table_name = "ga_overview")Convert from Glue DynamicFrame to Spark Dataframedf = datasource0.toDF()TRANSFORM - page cleaning and date formattingUDF to convert string to datedate_func = udf(lambda x: datetime.strptime(x, '%Y%m%d'), DateType())df1 = df.withColumn('ga:date', date_format(date_func(col('ga:date')), 'yyyy-dd-MM'))Trim URL parameters from page path columns, retain string dtypeurl_cols_to_clean = ['ga:pagePath', 'ga:pagePathLevel1']for url_col in url_cols_to_clean:df2 = df1.withColumn(url_col, F.regexp_extract(url_col, r'([a-z0-9-._~%!$&()+,;=:@/])', 0))Transforming Spark Dataframes back to Glue DynamicFramestransform1 = DynamicFrame.fromDF(df2, glueContext, 'transform1')LOADStoring the transformed data in same redshift tabledatasink1 = glueContext.write_dynamic_frame.from_catalog(frame = transform1, name_space = "test-hud", table_name = "ga_overview", transformation_ctx = "datasink1")job.commit()FollowComment"

Answer: "0Were you able to solve this error, apparently I'm having a similar error.CommentSharebmartinvranswered 2 years ago"


Question: Why hasn't my domain been verified?

"Hello,I added a new domain to my account, and added the TXT record to the DNS. As expected, it immediately showed as pending verification.However, it's been like that for a few days now. I haven't had any notification of any success or failure to verify, and the status is still pending.Anyone any idea how I get this verified? The other domains I've added were verified within a few hours.ThanksFollowComment"

Answer: "0Amazing, been waiting days, and within an hour of posting, I got the verification email!Someone somewhere has powers of prophecy!CommentShareFredanswered 4 years ago"