Question: Excessive Memory use when deploying PipelineModel using the pre-build Scikit container in Sagemaker

"I am using the pre-build Scikit container in Sagemaker to deploy an endpoint based on a model that contains a 59.4 MB model.tar.gz file. The following line was used to deploy the endpoint:sm_model.deploy(initial_instance_count=1, instance_type="ml.m5.xlarge", endpoint_name=endpoint_name)However, the after the endpoint was created, it fails to allocate memory to works. These error messages and warnings keep showing in the logs:[Errno 12] Cannot allocate memory[WARNING] Worker with pid 242 was terminated due to signal 9As far as I know, the xlarge instance has 16 GB of memory. The endpoint memory usage is at 60% while it still fails to allocate memory to workers. May I ask if anyone has any insight on why this is happening and how to solve this issue without using an instance that has more memory?FollowComment"


Question: Oracle datapump import error - AWS RDS for Oracle SE2 with S3 integration

"Hi,I am trying to import the Oracle data dump file using S3 integration. We received the Dump file via SFTP from other customer and I could successfully copy it to DATA_PUMP_DIR, but when I try to import it using DBMS_DATAPUMP operation, I am getting the below error sayingORA-31694 : XXXX table "DBNAME.SYS_IMPORT_SCHEMA_02" failed to load/unload, ORA-02369: internal number in header in file <**.dmp> is not valid. did anyone face this issue before? if yes, can you plz provide some insights? I want to rule out RDS side of issues before asking the customer to check the file on their sideFollowComment"

Answer: 0I do not think its a RDS issue. Below are the possible reasons for this error:The transfer of the dump files from source environment to target environment has been interrupted.The dump file are truncated by disk space of file system size limitations.The export job has not been completed successfully.CommentShareD-Raoanswered a year ago


Question: "EC2, PHP7.4, php-zip"

"Hi all.  Standard EC2 instance.sudo yum updatesudo amazon-linux-extras install php7.4sudo yum install php-mbstring php-dom php-pgsql - all good.sudo yum install php-zipwhy does php-zip always want to install "php-common-5.4.16-46.amzn2.0.2.x86_64"  ??any suggestions appreciated.[]$ sudo yum install php-zipLoaded plugins: extras_suggestions, langpacks, priorities, update-motdResolving Dependencies--> Running transaction check---> Package php-common.x86_64 0:5.4.16-46.amzn2.0.2 will be installed--> Processing Dependency: libzip.so.2()(64bit) for package: php-common-5.4.16-46.amzn2.0.2.x86_64--> Running transaction check---> Package libzip010-compat.x86_64 0:0.10.1-9.amzn2.0.5 will be installed--> Finished Dependency ResolutionEdited by: Zitan on Mar 15, 2020 2:28 PMFollowComment"

Answer: 0sudo amazon-linux-extras install epelsudo yum install epel-releaseyum install http://rpms.remirepo.net/enterprise/remi-release-7.rpmsudo yum install php74-php-pecl-zip.x86_64sudo cp /opt/remi/php74/root/usr/lib64/php/modules/zip.so /usr/lib64/php/modules/zip.sosudo nano /etc/php.d/40-zip.iniextension=zip.soCommentShareZitananswered 3 years ago0Be great to have repos updated to not have to do the above cheers.  Php 7.4CommentShareZitananswered 3 years ago0It seems that the repository has been fixed.The zip extension was bundled with php-cli-7.4.4-1.amzn2.$ docker pull amazonlinux2:latest$ docker run -it --rm amazonlinux:latest /bin/bash -c 'amazon-linux-extras install -y php7.4 && php -i'...zipZip => enabledZip version => 1.15.6Libzip headers version => 1.3.2Libzip library version => 1.3.2...CommentShareKentarouTakedaanswered 3 years ago


Question: Can't create Lambda code hook for Lex's TestBotAlias with CloudFormation

"My team is building a Lex Bot backed by AWS Lambda as a fulfillment code hook. We're using CloudFormation to deploy the stack.We're facing an issue where with each deployment introducing changes to the bot, the association between the Lambda and the TestBotAlias gets removed.We've tried to build an AWS::Lex::BotAlias, but it doesn't work because CloudFormation tries to create a second alias with the same name. Moreover, references to the DRAFT version of the bot are not possible because AWS::Lex::BotVersion allows us only to use numerical values.Is there a recommended way to associate the Lex bot with Lambda for tests so that it doesn't get removed every time we deploy a stack?FollowComment"

Answer: "0Hello!With the way that the CloudFormation workflow is set up, there is currently no mechanism of modifying the TestBotAlias within a CFN template. In the meantime a possible solution you can use is to create a new BotVersion when you are ready to test your bot and associate an AWS::Lex::BotAlias resource with templated bot version.If there are any other questions please feel free to reach out again!CommentShareKevinCanswered a year agowcheek a year agoThis is really inconvenient..Is there a way to modify the TestBotAlias after deployment?Share0I was finally able to get this working (in AWS CDK)While defining the lex bot:test_bot_alias_settings = (    lex.CfnBot.TestBotAliasSettingsProperty(        bot_alias_locale_settings=[            lex.CfnBot.BotAliasLocaleSettingsItemProperty(                locale_id="en_US",                bot_alias_locale_setting=lex.CfnBot.BotAliasLocaleSettingsProperty(                    enabled=True,                    code_hook_specification=lex.CfnBot.CodeHookSpecificationProperty(                        lambda_code_hook=lex.CfnBot.LambdaCodeHookProperty(                            code_hook_interface_version="1.0",                            lambda_arn=self.my_lex_lambda.function_arn,                        )                    ),                ),            )        ],    ),)CommentSharewcheekanswered a year ago"


Question: Unable to reach application hosted in EKS worker node by secondary IP from a Window server hosted in another VPC

"We have 2 VPCs, VPC A and B. They have connected with a PCX.In VPC A there is a EKS cluster, I have a K8s service with the following endpoint:me$ kubectl get endpoints | grep my-appmy-app                           10.0.102.250:80                   2y200dAnd this service hosts in a EKS worker node EC2 instance with 4 ENIs, so there are 4 private IPs. One of it is 10.0.100.200 for example.In VPC A, when I curl from a EC2 to 10.0.102.250:80 it responds properly and I can ping to the EC2 by the host private IP 10.0.100.200.However when I curl from a EC2 in VPC B, I can only ping to 10.0.100.200 with correct response, but I cannot access the K8s service with 10.0.102.250:80. The K8s worker node has whitelisted all traffic from VPC B's EC2.When I run tracert 10.0.102.250 from the VPC B EC2, it shows the followings.C:\Users\Administrator>tracert 10.0.102.250Tracing route to ip-10-0-102-240.us-west-2.compute.internal [10.0.102.250]Is it because the DNS in VPC A passed to VPC B via PCX contains information for EC2 with ENI attached, those secondary private IPs associated were excluded making VPC B's EC2 not sure how to forward request to?For the PCX we forward VPC A's CIDR to PCX, it should be correct(10.0.0.0/16)FollowComment"

Answer: "0It's possible that the issue is related to DNS resolution or routing. When you try to access the K8s service from a EC2 instance in VPC B using the endpoint 10.0.102.250:80, the request is being sent to the IP address, but it's not reaching the service.Since you can ping the worker node's IP address 10.0.100.200 from the EC2 instance in VPC B, it seems that routing between the VPCs is working correctly. However, if you're not able to access the K8s service, it could be due to a DNS resolution issue.When you run tracert 10.0.102.250 from the EC2 instance in VPC B, it shows the hostname ip-10-0-102-240.us-west-2.compute.internal. This hostname is likely being resolved by a DNS server in VPC A, and it's possible that the DNS resolution is not working correctly from VPC B.To troubleshoot the issue, you could try to manually resolve the hostname ip-10-0-102-240.us-west-2.compute.internal to the IP address 10.0.102.250 from the EC2 instance in VPC B using the nslookup command. If the DNS resolution is not working correctly, you could try to update the DNS configuration in VPC B to use the same DNS server as VPC A or configure a DNS resolver that can resolve the hostname correctly.Additionally, you could try to access the K8s service using the IP address 10.0.100.200 instead of the endpoint 10.0.102.250:80 to see if it works. If it does, then the issue is likely related to DNS resolution, and you may need to update the DNS configuration in VPC B.Ref links:https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-troubleshootinghttps://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html#service-load-balancer-issuesCommentShareMina Gobrialanswered 3 months ago"


Question: How to see actual requests by ELB?

"My load balancer for a new "Linux 2" environment is in “Severe” health because the health checks are experiencing: “Target.ResponseCodeMismatch”.Is there any way I can see the exact requests being made by the ELB to my servers? I want to know the hostname, the ports, etc., so I can exactly pinpoint what’s going on. I turned on logging in the Elastic Beanstalk configuration for the load balancer, but I downloaded those  files from s3 and I’m not seeing the health check requests.This page has some helpful ideas https://aws.amazon.com/premiumsupport/knowledge-center/elb-fix-failing-health-checks-alb/ but it’s all hearsay without being able to see what the ELB is actually doing.FollowComment"

Answer: "0In the elastic beanstalk console, I was able to go to Logs > Request last 100 lines and see the ELB pings under /var/log/eb-docker/containers/eb-current-app/eb-stdouterr.log:django_1  | ERROR 2021-09-13 13:55:37,146 exception 27 140410675223368 Invalid HTTP_HOST header: '172.31.XX.XX'. You may need to add '172.31.XX.XX' to ALLOWED_HOSTS.  Also, since I’m using Docker, I found some helpful notes on this page for mounting log directories within containers to the right place via "volumes" in the docker-compose file: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/docker-multicontainer-migration.htmlCommentSharepetersprstanswered 2 years ago"


Question: "Clarification on S3 Select with Parquet - indexing, range offsets, and pricing"

"Is there any indexing done on the data on S3's side? Or do they basically just execute the SQL in memory on the object via a scan?For Parquet files if you specify a range, what does that mean? Is it going to only scan the entire row group in that range? The column? Is the range a byte offset into the parquet object, and if so, is it the compressed size? ie: If I want to select from a column in the second rowgroup, I'd have my index by the byte size of the compressed row group + 1 ?Select charges you based on the data scanned - if you're using parquet and you index into a row group to scan a column, is that only going to charge for the column scan? Decompressed or compressed?FollowComment"

Answer: "0Thank you for reaching AWS repost, Please find my answers as below,Q1. Is there any indexing done on the data on S3's side? Or do they basically just execute the SQL in memory on the object via a scan?Answer - To understand the specific requirements driving this question, I request you to raise an S3 support case with AWS Support from your account to deep dive further and provide you relevant details.Q2.For Parquet files if you specify a range, what does that mean? Is it going to only scan the entire row group in that range? The column? Is the range a byte offset into the parquet object, and if so, is it the compressed size? ie: If I want to select from a column in the second rowgroup, I'd have my index by the byte size of the compressed row group + 1 ?Answer - For line-based CSV and JSON objects, when a scan range is specified as part of the Amazon S3 Select request, all records that start within the scan range are processed. For Parquet objects, all of the row groups that start within the scan range requested are processed. This is explained in documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.htmlQ3. Select charges you based on the data scanned - if you're using parquet and you index into a row group to scan a column, is that only going to charge for the column scan? Decompressed or compressed?Answer - Select will scan minimum data required to execute the query. The billing will be done using compressed size if data on S3 is already compressed.CommentShareSUPPORT ENGINEERSathyaanswered a year ago"


Question: What IAM Permissions do I need to consume an SQS que from Lambda?

"Hello,I'm trying to have an event driven solution where the SQS que sends messages to lambda as soon as it receives the message. I have a third party application that sends messages to SQS and assumes a role in which gives it permission to perform actions needed on the SQS.I am kind of confused what I need to consume an SQS que from lambda. In the documentation it indicates that to poll events from Lambda different permissions are needed, and then it gives directions for "event driven" triggers to lambda from the SQS Que.I am writing all of this in CDK so maybe that's where I may be missing something at.So far in my CDK I have the SQS message able to be consumed by lambda I believe, by usingsqsQue.grantConsumeMessages(Mylambda) which indicates here  that this will allow SQS messages to be consumed by grantee which is my lambda. I am not certain if this adds a resource based policy to lambda to SQS, I'm assuming that it adds it to SQS and I do not need to add it to the lambda.However, for the lambda in my CDK I just have the default execution policy and I do not believe I added a resource based policy. I'm not even sure if it is needed for my use caseSo do I need resource based policy for this? Or do I need anything particular in my execution role?Also is there a difference between event driven SQS triggering lambda vs lambda polling from SQS, aren't these two separate implementations?FollowComment"

Answer: 1You do not need a resource based policy to trigger Lambda functions from SQS. You need the Lambda execution role to include:sqs:ReceiveMessagesqs:DeleteMessagesqs:GetQueueAttributesYou can find all the information here.CommentShareEXPERTUrianswered 6 months ago


Question: Getting all users from AWS identitystore along with "status" property

"I have setup AWS SSO with MS Active Directory using the AD-Connector.I need to get all user that exist in AWS identitystore so that I can sync it with my on-prem DB. To do that I have tried below AWS CLI commandaws identitystore list-users --identity-store-id="d-XXXXXXXXX"The result of command above gives the listing of users with their details, but I doesn't mention the status of the user. I need the information whether the user is active/in-active, so that I can enable/disable the user in my DB too. I don't see such information returned from the above command.I tried AWS SDK for Nodejs too but I still get same json object with no "Status" field.Although the AWS console for Identitystore does show whether the user is active, but same is not returned in APIs or CLI response.Any help or alternative approach get user details with status is really appreciated.FollowCommentMichael-Lumos 3 days agoThis is a huge omission from the Identity Store API after the SSO transition. The UI and the SCIM API both provide User status of Enabled/Disabled. The replacement Identity Store list_users and describe_user APIs should also provide this data. This is significantly blocking the adoption of the Identity Store API.Share"

Answer: "1Accepted AnswerThere does not seem to be a way to check with the CLI at this time.https://awscli.amazonaws.com/v2/documentation/api/latest/reference/identitystore/list-users.htmlhttps://awscli.amazonaws.com/v2/documentation/api/latest/reference/identitystore/describe-user.htmlThe only other way I could think of was to create a script that would crawl the screen and check the status.CommentShareEXPERTRiku_Kobayashianswered a month agoAvinash-INDX a month agoThanks for sparing some time and effort. It seems AWS is yet to bring the "status" information in API or CLI. As you suggested, I am only left with option to crawl the page and get the status info. Thanks. Given current scenario with AWS I am going to accept the solution.Share"


Question: Is RDF-STAR supported

I'm in the middle of a POC datacentric application using Neptune with RDF and I'd like to use RDF-STAR syntax but it doesn't seem to allow me to when in a notebook with sparql magic. if it isn't supported at the moment is there a rough timeline of when it might be?thanksFollowComment

Answer: "0Accepted AnswerHi there,Amazon Neptune is a fully managed graph database service that supports the Resource Description Framework (RDF) data model and SPARQL query language. RDF-STAR is a proposed extension to the SPARQL query language that allows for the querying of RDF datasets that use the RDF* triplestore format. Unfortunately, Amazon Neptune does not currently support the RDF-STAR syntax in its SPARQL query language.I regret to say I cannot provide a timeline for when or if Amazon Neptune will support RDF-STAR syntax in the future. You can contact Amazon customer support for more information.CommentShareajmarcinanswered 6 months agoMathew Fitzsimon 6 months agothanks Ajmarcin, I'll see if I can get a steer on a timeline.Share"


Question: EKS Managed Node Groups - PodEvicitionFailure

"I receive an error when attempting to upgrade a managed node group. I don't see anything in the documentation highlighting this specific error.Currently the error reads:PodEvictionFailureReached max retries while trying to evict pods from node ip-10-50-20-101.us-east-2.compute.internal in node group GrowOps-NodeGroup-1istio-telemetry-55c8559456-86klxThanks for any assistance.RegardsEdited by: eaf-joseph on Mar 19, 2020 8:36 AMEdited by: eaf-joseph on Mar 19, 2020 8:37 AMFollowComment"

Answer: "1Hi Joseph, I know it has been a long time already, but in case you still wanted to know the root cause it is because by default Amazon EKS created AutoScaingGroups are using "Rolling Updates" strategy for upgrading the EKS cluster nodes. This is done to make sure that you don't have any interruptions in your workload while the upgrade is being done behind the scene. If you have deployed PodDisruptionBudgets that makes the pod rejects the eviction request then you will get this error because in that case this pod has to be forced to evict. You can do it with kubectl or just when you try to upgrade your EKS Cluster nodes make sure to select "Force Update" strategy, this will force the pods to evict because the EKS node will go away anyway and will not respect the PodDisruptionBudgets for you. But before doing so, please be aware that this will cause disruption to your workload i.e. a potential downtime.Edited by: ahmedelgamal on May 7, 2021 2:39 AMCommentShareahmedelgamalanswered 2 years ago"


Question: Workload discovery inside EC2 VM

what's the best way to detect what kind of service is hosted inside the VM? I'm talking at least about popular workloads like sql server/oracle/sap/mongo/ and if we can do more workload detection also that will be great :-)FollowComment

Answer: "0Take a look at AWS Systems Manager Inventory.CommentShareEXPERTkentradanswered 24 days ago0HelloI understand that you would like to know if there is a way to get a list of services/workloads running inside an EC2 instance.As suggested by kentrad@ in this thread, you could consider using AWS System Manager Inventory service which can be used to collect information like Applications inside VM, AWS components, Files (Name, size, version, installed date, modification and last accessed times, etc.) and more. Please refer:[1]However to know about the services running inside VM, there is an inventory parameter : service, but it can be used only with Windows OS and not linux. Here is the description of the parameter:(Optional, Windows OS only, requires SSM Agent version 2.2.64.0 and above) Collect data for service configurations.Parameter : file, this parameter can be used to get the metadata of the file, not the content of the file. Another AWS offering is AWS Application Discovery service, which aids in the capture of system configuration, system performance, running processes, and details of the network connections between systems. For more information, please refer:[2]Should you require further assistance regarding your specific situation/use-case, please feel free to create an AWS Support case to get in touch with an engineer to assist you further.[+] https://console.aws.amazon.com/support/home#/case/createReference:[1] https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-inventory.html[2] https://aws.amazon.com/application-discovery/faqs/CommentShareSUPPORT ENGINEERShreya_Wanswered 23 days ago"


Question: no automatic renewal on my plan

I need to talk to somebody at AWS my free tier expires and will not auto renew who do I talk to?FollowComment

Answer: "0When signing up for a new AWS account, the free tier for many services only lasts 12 months. There are some that last indefinitely that you always get for free.Unfortunately, there is no way to "refresh" the free tier for the services with the 12-month limitation. The only way to get that is to sign up for a completely new AWS account and linking it to a new email address.CommentShareLawrence Aielloanswered 2 months ago0If you don't want any charges after the free tier has expired take a look at these:Closing an accountGetting help with AWS BillingTracking your AWS Free Tier usageCommentShareEXPERTkentradanswered 2 months ago"


Question: Standalone Microsoft Office on EC2

"A customer requires a standalone version of Microsoft Office instead of Microsoft Office 365. They are using Windows Server 2019 Base (AMI ami-05bb2dae0b1de90b3) on their EC2 instances. From the Microsoft FAQ here (https://aws.amazon.com/windows/faq/) it seems like you can no longer BYOL for Microsoft Office for new licenses. Additionally, I checked the AWS Marketplace and only saw options for WorkSpaces, which they aren’t using. What options do they have for licensing a standalone copy of Microsoft Office on their EC2 instances?FollowComment"

Answer: "2Recently AWS announced Microsoft Office Amazon Machine Images (AMIs) on Amazon EC2 with AWS provided licenses, it means you can now provision EC2 machines with office licenses managed by AWS. The solution requires Managed AD as well as setting up license manager to track users and machines.There's with details on how to set it up - hereCommentShareDror Helperanswered 5 months agoEXPERTRob_Hreviewed 5 months ago1Accepted AnswerHi,Their options are limited. They can only bring MS Office licenses that were acquired before October 1, 2019 and they can only run on EC2 dedicated hosts or EC2 dedicated instances.CommentShareBradanswered 3 years agoEXPERTRob_Hreviewed a month ago"


Question: Move EMR Notebooks S3 directory structure between accounts

"Hi!So, we are currently working on migrating between 2 accounts.In the old account, we were working with EMR Notebooks directly, without EMR Studio or EMR WorkSpaces.In the new account, we made the necessary steps to start working with EMR Studio and EMR WorkSpaces.However, when we manually migrate the old notebooks from their S3 location to the new S3 location for the respective workspace, only the notebooks in root level directory appear, and not the whole directory structure. Even more surprisingly, when I create a folder through the Jupyter UI with the exact name as in the S3, the respective notebooks appear. This last behavior prompts me to eliminate any permissions issues.Of course, I have tried to attach/re-attach/restart the respective WorkSpaces.Have you any suggestion on what is happening here and how can I solve this?Thanks!FollowComment"

Answer: "0Hello Nikos,To answer your question, we require details that are non-public information. Could you please open a support case with AWS using the following link https://console.aws.amazon.com/support/home#/case/create ?CommentShareYiran_Wanswered 2 months ago0Hello Nikos,Please execute the following steps:In JupyterLab open a terminal session and execute the following command:find ~/work/  ! -readable  -maxdepth 1 2>/dev/null -print0 | xargs -0 mkdir || truePlease confirm if you can view the folders now. Please note you might need to adjust the "maxdepth" parameter from 1 to 2, etc. based on your directory depth level.Please let us know if the step work for you Nikos.CommentShareAWS-User-7917407answered 2 months ago"


Question: AWS bucket vs cloudfront for mp4 streaming

"Hello, I am new to AWS. I have created a s3 bucket and made it public. uploaded few mp4 files and I see that I can stream those videos comfortably without any issues. I just streamed a 600mb file (video of length 1 hour) in my angular app without any issues like downloading the whole video before playing, I can change the video on the video progress bar and skip to that part of the video without any issues (mp4 on azure blob storage is not allowing to skip the video). why should I use cloudfront? I am okay with viewers downloading the videos.FollowComment"

Answer: "1If I understand correctly, AWS will charge you for the amount of data leaving the bucket. If a lot of people begin downloading the file, your bill could go up more than you planned for. Something to be cautious about. On the other hand, if you switch to letting cloudfront serve the file, it won't be leaving your bucket as often, possibly saving you money. But it all depends on how much traffic you can handle.CommentShareZach Perlmananswered 7 months ago1Security - Making the bucket public will not allow you to track who access what objects in the bucket and per security guidelines you should always follow the security best practicesCost - Content served directly from S3 is charged at this time of writing $0.0004 per 1000 GET's requests, plus $0.09 per GB for the first 10 TB and you will have per month a free allowance of 100 GBScalability - Amazon S3 can perform 5,500 GET/HEAD requests per second per partitioned prefix , once that limit is reached the requests will start receiving 503 response errors. Further information at https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.htmlFronting the S3 Bucket with a CloudFront distribution endpoint will allow you to:Filter who can access the bucketStop direct access to the bucketCache content closer to the edge, reducing the number of S3 requestsReduce costs ($0.085 for the first 10 TB with a monthly free allowance of 1 TB)For further information on how to restrict access to S3 Buckets with CloudFront, please see https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.htmlCommentShareNuno_Qanswered 7 months ago0Two reasons :-Security :- once you make content of the bucket public or content link directly inside the application. its is wrong practice and can lead to catastrophic scenarios such as content is serving some other application without any notice to you. for that purpose also we prefer cloud Front service.Cost :- every time content leave bucket aws charge and even for the network traffic aws charge. so using to cloud front along with S3 can reduce your both cost.for On demand video broadcasting or streaming best possible solution is to use AWS elemental Media Services. these services take care of all possible scenarios ( Security ,cost, some other feature ) and help you achieve more.***cloud asterisk studioCommentShareutkarsh-awsanswered 7 months ago"


Question: SES - customer concerned that their emails will be marked as spam and disable the account's ability to send email

"Team,A customer wants to send automated emails [alerts, notifications, etc.] from their SaaS solution using SES.They are concerned that the emails sent from their system will be marked as spam and disable the account's ability to send email - thus affecting all of the system's tenants.Customer wording: "Our concern is for Application emails that configured by our end users and can be recognized as ‘spam’ by SES".The customer currently considers using another solution because they are concerned that their ability to send emails will be suddenly blocked without any prior notification. *[I will elaborate about the review process in my next meeting with them, but I also noticed that in 'serious' cases we can disable their ability to send email without any notice]A couple of questions regarding this situation:What is the probability of that the happen in the first place given this use case [automated emails]?Do we have references for proper guidance for mitigating this chance by using the right contentand pattern of sending emails?How this situation can be mitigated once it happens? (I'm planningto explain about the review period, the the Deliverability Dashboardand the ability to send notifications as preventive measures).Is there any suggested backup plan in case the account disable action happens? (like for example keeping a separate account with the ability to send email)?Thanks in advance.FollowComment"

Answer: "0Accepted AnswerAmazon SES uses in-house content filtering technologies to scan email content for spam and malware. If we determine that an account is sending spam or malicious content, we will pause that account's ability to send additional email.If their e-mail is going to be marked as spam by their recipients, it's going to affect their deliverability. If the rates at which they are marked as spam go above some thresholds they are going to be facing issues with us and eventually could lead to suspension in their ability to send e-mail with SES.SES requires customers to have a clean e-mail database. That means that they need implement a process to act upon  rejects/bounces/compaints as they happen. If they keep a clean database they should not face any issue with SES.A backup up plan can include from having a stand-by configuration on a different e-mail provider, to an MTA running in an EC2 instance. They should have in mind that those services usually will need some time to "warm up" before being able to handle all the e-mail trafficAmazon SES Best Practices: Top 5 Best Practices for List Management: https://aws.amazon.com/blogs/messaging-and-targeting/amazon-ses-best-practices-top-5-best-practices-for-list-management/Amazon SES Sending Review Process FAQs: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/e-faq.htmlCommentShareAWS-User-6833802answered 3 years ago"


Question: How to identify tls deprecation resource?

"I received a Security tls deprecation notification on my AWS Health Dashboard.This is the affected resource but I don't know how to identify it. I am using Lambda & Api Gateway to access Cognito API so I think those should be updated to TLS v1.2 automatically.What should I do to identify it? ThanksConnections details in the 'Affected resources' tab will be in the following format:Region | Endpoint | API Event Name | TLS Version | Connection Count | UserAgentus-east-1 | cognito-idp.us-east-1.amazonaws.com | GetUser | TLSv1 | 2 | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36From the UserAgent, that is a connection from a browser?FollowComment"

Answer: "1Hi,If I’m not mistaken, you’re environment is something similar to what described in https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-integrate-with-cognito.html. Maybe not exactly, but I’d say the point is the sources of the calls to cognito-idp (https://docs.aws.amazon.com/general/latest/gr/cognito_identity.html).As described in the first link “After the API is deployed, the client must first sign the user in to the user pool, obtain an identity or access token for the user, and then call the API method with one of the tokens, which are typically set to the request's Authorization header. The API call succeeds only if the required token is supplied and the supplied token is valid, otherwise, the client isn't authorized to make the call because the client did not have credentials that could be authorized.”Assuming your clients (probably an app) are also making calls to Cognito as part of the workflow, I’d look into it trying to find out if this particular AWS Health Dashboard entry is due to a client not able to use TLS v1.2.There are some more details in https://aws.amazon.com/blogs/security/tls-1-2-required-for-aws-endpoints/, in particular in the section “What should you do to prepare for this update?” that describes ways to find any connections using TLS 1.0 or 1.1.CommentShareSergioAanswered 3 months ago0Yea, I do use AWS Cognito to authorize my users and I use AWS Amplify in frontend to access cognito for signup, signin etcI have used CloudTrailI to check but I haven't seen any of the similar requests.From the AWS Health Dashboard entry, there are only 2 such requests.Could be an end-user who tried to use my app was unable to use TLS v1.2?CommentSharerePost-User-230101answered 3 months ago0Hi,It's highly probable it's an end-user using an old TLS version. In addition to the links in the previous message, you can start from https://docs.aws.amazon.com/cognito/latest/developerguide/monitoring.html to configure Amazon Cognito logging and maybe get more information about it, like the IP address. Important to note is that you can use the AWS CloudTrail console to view only the last 90 days of recorded API activity, no entries before 90 days. That's why it is recommended to send the logs to an Amazon S3 bucket, to keep the API activity (logs) as long as you want.Have a great day!!!CommentShareSergioAanswered 3 months ago"


Question: GGv2 Outbound only connection set for core device(inbound ports blocked by device): Is it possible to use this pattern for Sitewise ingestion?

"How does outbound only connection work when using Stream Manager to ingest data to a destination like Sitewise? Which port is used by GGv2 to send back data regarding API responses? As per this pattern(I don't have access to the full doc): https://www.internetofthingspatterns.com/patterns/security/outbound-only-connection/ Inbound communication happens only when an open session is established, is there a doc in AWS explaining this pattern for GGv2?FollowComment"

Answer: "0To ingest data into SiteWise, StreamManager establishes one or several outbound TLS connections to the SiteWise API endpoint, on which it sends the HTTP request for the API. It then receives the response for the call on the same connection.CommentShareEXPERTMassimilianoAWSanswered 18 days ago"


Question: Node.js: module not found using ES import with Lambda layers

"We're trying to deploy a Lambda function with AWS SDK v3 modules. The imports are standard ES syntax and work fine in a local setup, and work if we package the entire module into the function:import { SESv2Client, SendEmailCommand } from '@aws-sdk/client-sesv2';For efficiency and reuse, we moved the modules into a layer. When we run the function (tried with Node.js 14 and 16 runtimes), we continue to receive:Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@aws-sdk/client-sesv2' imported from /var/task/index.jsIf we strip out the ES pieces and use CommonJS syntax with require statements, the modules load correctly. Obviously we would prefer the modern syntax and the performance gains that go with it. According to AWS docs, node.js lambda functions support ES imports in Node.js 14.x and Node.js 16.x.Has anyone else run into this and found a solution?FollowComment"

Answer: "0Accepted AnswerUnfortunately AWS support has confirmed we can't utilize layers with ES:importing ESM modules from layers seems to not be supported when using a module type in Lambda. [...] this is a Node feature blocking us to import ES modules from a layer as "NODE_PATH" variable is not supported by the ESM loader in nodeDisappointing that AWS is making headway on supporting ES, but it means we lose the functionality of layers. Support provided a wrapper workaround that we'll be trying. It corresponds with what we found here. Hopefully this saves someone else the headache we've gone through the last week.CommentSharerePost-User-4160403answered a year ago0Is this still the case with node18.x???CommentSharerePost-User-4974576answered 4 months ago"


Question: Which AWS service(s) is cheaper to use to decode and view graphs corresponding to the payload emitted from Helium?

"Hello community, especially those more experienced in IoT via LoRaWAN:As you can see in the attached image, so far I have only been able to visualize the MQTT broadcast from Helium (label "red485-00)" in the "test client" by subscribing to all the topics of the object "DRAGINO-RS485LN-00" (type "Helium-Things").Said object is sending the "payload" corresponding to some sensors (temperatures and CO2 in only 9 bytes)Please, I need your help to learn how to program the necessary (and cheapest possible) AWS service/s to decode that payload and represent its values in graphs (historical, last value, etc.).To begin with, I would like to know which AWS services to use, in which language (JSON, JavaScripts, Python, etc.) and in which place to leave that written programming.Thank you very much to those who can give me a hand.220517-102036_red485FromHeliumToAWSFollowComment"

Answer: "1Hi,you can check this repo (https://github.com/aws-samples/aws-iot-core-lorawan) for examples on how to decode LoraWAN payloads. As for storing the decoded data, you might consider using Amazon Timestream. Alternatives to Timestream are also AWS IoT Analytics and  AWS IoT Sitewise. Key-value stores such as DynamoDB can be efficient if you only need to retrieve the latest value, but can also be used for historical data (see this project https://github.com/ruuvi/ruuvi-network-serverless)CommentShareEXPERTMassimilianoAWSanswered a year ago0Hi.I agree with Massi's answer.You can find more about decoders here in our LoRaWAN workshop: https://catalog.us-east-1.prod.workshops.aws/workshops/b95a6659-bd4f-4567-8307-bddb43a608c4/en-US/400-decoding-binaryAnd you can follow along on how to build-out a real-time dashboarding solution using Timestream and Grafana: https://catalog.us-east-1.prod.workshops.aws/workshops/b95a6659-bd4f-4567-8307-bddb43a608c4/en-US/500-grafana-timestreamIf you were to use SiteWise, you can achieve the dashboarding using either SiteWise Monitor or Grafana.One alternative to using DynamoDB for storing latest value, is instead to use AWS IoT Shadows. From the decoder Lambda, you could write the decoded (and possibly enriched) payload into the Thing's shadow document. This can be useful if you have interest in using fleet indexing with a LoRaWAN fleet.It's difficult to answer the cost question without more details. On the languages question; we support numerous options so you can choose whatever language you are most comfortable with. However, in terms of decoders, we currently have the most examples in Python.CommentShareEXPERTGreg_Banswered a year ago"


Question: ec2 Run Instance Request with Launch Template and a overriding network interfaces specification

"I am attempting to automate some instance creation using a Launch Template. As part of the RunInstanceRequest I sometimes need to specify that my instance be on a different network(s). While this works fine if the LT has the same number of devices (or less than), when the LT has more interfaces than what I specify in the RunInstanceRequest, the instance inherits the unspecified device specification from the LT.So for example if my LT has two devices and my RIR has one, my instance gets two interfaces.Is there a way for me to override this behavior so I can create my instance with a network interface spec which completely overrides what is defined in the LT?FollowComment"

Answer: "0To answer your question, we require details that are non-public information. Please open a support case with AWS using the following link - https://support.console.aws.amazon.com/support/home#/case/createCommentShareSUPPORT ENGINEERAWS-User-3805941answered 9 months ago"


Question: Alarm not triggering

"I`ve set up a lot of alarms of this kind without any issues, i don't know if there is a problem in cloudwatch or if I´m, missing something somehowBasically I have a filtered metric that grabs info from a log stream, then my alarm catches when there is new incoming data to the stream and the alarm is triggeredas stated I`ve 5 other alarms of this kind running without any issue, triggering as it should be,  but with this last one, it does not workso far I've tried, changing periods, threshold, deleted and re-created the alarm, deleted the metric and re-created again  and nothing worksAny advice is much appreciated :)Thanks "MetricAlarms": [        {            "Dimensions": [],             "Namespace": "AnaliticoMetricsNew",             "DatapointsToAlarm": 1,             "ActionsEnabled": true,             "MetricName": "ErrorInGaps",             "EvaluationPeriods": 1,             "StateValue": "OK",             "StateUpdatedTimestamp": "2019-11-15T18:32:37.577Z",             "AlarmConfigurationUpdatedTimestamp": "2019-11-15T21:55:31.440Z",             "AlarmActions": [                "arn:aws:sns:us-east-2:xxxxxxx:AnaliticoAWSNotification"            ],             "InsufficientDataActions": [],             "AlarmArn": "arn:aws:cloudwatch:us-east-2:xxxxxxxx:alarm:GapsFailing",             "StateReasonData": "{\"version\":\"1.0\",\"queryDate\":\"2019-11-15T18:32:37.571+0000\",\"statistic\":\"Sum\",\"period\":300,\"recentDatapoints\":[],\"threshold\":1.0}",             "Threshold": 1.0,             "StateReason": "Threshold Crossed: no datapoints were received for 1 period and 1 missing datapoint was treated as [NonBreaching].",             "OKActions": [],             "AlarmDescription": "when a gap fails",             "Period": 900,             "ComparisonOperator": "GreaterThanOrEqualToThreshold",             "AlarmName": "GapsFailing",             "Statistic": "Sum",             "TreatMissingData": "notBreaching"        }    ]FollowComment"

Answer: "0odd, deleted all and recreated logstream, metric and alarm and the issue was fixedCommentShareFranciscoHerreraanswered 4 years ago"


Question: EKS Anywhere\Connector - Register New Cluster Button Does Not React

"I've been able to create a local EKS Anywhere cluster in vCenter. I'd like to register it with EKS (using the webUI) but when I complete the form the 'Register' button doesn't react at all. There are no indications of missing fields, no error, and the button is not 'greyed out' it just doesn't react at all when clicked. We have created the AmazonEKSConnectorAgentRole role as defined in the prereqs and I have the required permissions.eks:RegisterClusterssm:CreateActivationssm:DeleteActivationiam:PassRoleFollowComment"

Answer: "0According to the documentation for Amazon EKS connector IAM role, you will need to create an inline policy and the trust-relationship for the AmazonEKSConnectorAgentRole as shown below.Please verify if your AmazonEKSConnectorAgentRole is configured as shown below.IAM Policy:{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "SsmControlChannel",            "Effect": "Allow",            "Action": [                "ssmmessages:CreateControlChannel"            ],            "Resource": "arn:aws:eks:*:*:cluster/*"        },        {            "Sid": "ssmDataplaneOperations",            "Effect": "Allow",            "Action": [                "ssmmessages:CreateDataChannel",                "ssmmessages:OpenDataChannel",                "ssmmessages:OpenControlChannel"            ],            "Resource": "*"        }    ]}Trust Relationship:{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "SSMAccess",            "Effect": "Allow",            "Principal": {                "Service": [                    "ssm.amazonaws.com"                ]            },            "Action": "sts:AssumeRole"        }    ]}I have followed the documentation for Amazon EKS connector IAM role and I was able to register the cluster.You can also create an issue in the AWS containers-roadmap project using this link - https://github.com/aws/containers-roadmap/issues/new/chooseCommentShareSUPPORT ENGINEERVenkat Penmetsaanswered a year agothomaseldredge a year agoThanks for your reply. The connector role and trust relationship were already created.Should there be any error generated when the cluster cannot be registered? It seems odd that the register button just does nothing.Share"


Question: Replace the pem file for an EC2 instance running Windows

"I'm looking for some guidance on replacing the pem file associated with an EC2 instance running Windows.The AWS documentation on key pairs for Windows doesn't seem to mention this, and the general advice seems more geared towards Linux.I have the old pem file and just want to replace it for security reasons. Any help would be appreciated!Resources I've read through so far are below:https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#replacing-key-pairhttps://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-key-pairs.htmlFollowComment"

Answer: "1Good day JamielairdAccording to my understanding you are trying to replace a .pem key file that is associated with an existing EC2 instance running Windows for security reasons.Based on my investigation and replication to your question, these are the steps I followed:Note: To give the new key pair the same name as the old key pair, you must first delete the old key pair1.Create a new key pair, and then save the private key file in a safe place. You can create a key pair using the console, the AWS CLI, or AWS Tools for Windows PowerShell.2. From the Amazon EC2 console, choose Instances from the navigation pane.3. Select your instance4. Stop your instance5. Select your instance. For Actions, choose Image, Create Image. For Image name, enter a name.6.  Create Image, and then choose Close.7. Launch an AMI - Launch an instance.8. Complete the wizard. Be sure to select the same Instance type, VPC ID, Subnet ID, Security groups, and IAM role as the instance that you are replacing For Select a key pair, choose the new key pair.9. Now that the private key file is replaced, you can reset the administrator password.To reset your administrator password:Use EC2Config for Windows Server 2012 R2 and earlier. https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ResettingAdminPassword_EC2Launch.htmlUse EC2Launch for Windows Server 2016 and later https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ResettingAdminPassword_EC2Config.htmlReferences[1] https://aws.amazon.com/premiumsupport/knowledge-center/ec2-windows-replace-lost-key-pair/CommentShareLihleanswered 2 years ago"


Question: SQS Queue Ring Buffer

"Hi, I'd like to know if there is a way to support ring buffer functionality to SQS queues?My infrastructure has multiple usecases of just requiring a latest data snapshot across a variety of apps. The ideal scenario is if I can set a maximum queue size to 1 message that all consumers have read access to, while the queue (or some other AWS piece) can manage the queue size to only contain the latest snapshot.Currently, the only way I have found to support my use case is to have one queue by consumer, which will become very difficult to manage properly. The ring buffer functionality would effectively solve this issue.Thanks in advance!FollowComment"

Answer: "0Sounds like you're fanning out from one producer to multiple consumers?  SNS or Kinesis Data Streams would probably suit.CommentShareEXPERTskinsmananswered 7 months agoMichael Hart 7 months agoYes, currently I am using SNS for the fanout to multiple SQS queues via a filter. The purpose for the queue in my use case is to store the latest state.My use case is a financial market data related application, which typically uses a messaging bus + ring buffer functionality because applications processing the data are not meant to process every type of event. So far, I found SNS + SQS to similarly mimic this, but it comes with some drawbacks.I am not too familiar with Kinesis - does it have this type of functionality?(Disclaimer, I'm new to AWS. My former firm managed this in house).Share0I am not sure what exactly you are trying to achieve. A ring buffer is a queue with a limited size. When it is full, no new elements can be added to the queue. There is a different mode in which new elements overwrite existing elements in the queue. In this form, the queue only holds the most recent N items. If I understand correctly, you want to the later implementation and you want the size to be 1, i.e., you want a queue that always holds only the most recent item.If this is the case, why use a queue? You could just update an item in DynamoDB. Every time a new item comes in, you update the item in the database to the latest value. When you want to consume elements, you just read from the database. If you need to know that you are handling a new item and not the one you already handled before, you can add an additional field which indicates that the item is new or not. The producer sets it to New. The consumer sets it to Old.CommentShareEXPERTUrianswered 7 months agoMichael Hart 7 months agoMarket data is known to overload databases, so a DynamoDB cannot be the approach unfortunately. There are companies that make specialized DB's for financial data and even those cannot be used for real time processing. For example, one financial product would post <10kb message every 25ms in normal conditions and consumers would need access to that data as low latency as possible.What I am looking for is a multi producer & multi consumer structure where consumers only require the most recent item that does not rely on a database. How are other trading applications on AWS doing this?Share0If I understand well you have multiple consumers who wait for same message  buy at a given time they need the last state of event,Is it the case?CommentShareOmid Eidivandianswered 7 months agoMichael Hart 7 months agoFor one producer, I will have multiple consumers of the data. The consumers will only care about the latest state if there are multiple messages produced during the time the consumer is processing the change in state.As I do more research, I am finding that AWS's MSK solution is most likely the best fit for my needs.Share0HiThanks for your replay , now it s clearIn my opinion you dont need a circular queue ad basically you are deduplicating the events at a given time,**Example : ***considering your event looks like *{      Identifier: "A",     State: "Created",     ....  }and other one{      Identifier: "A",     State: "Approved",     ....  }You dont care about the state and you need just At a given time the consumer get Id: "A" one time.In your design you need to use a systeme to deduplicate that events and avoid inserting the new ones if one exists alreadyYou have two choices based on level of complexity of your event structure:1- Using A FIFO SQS which let you grouping the messages and deduplicating them based on a provided DeduplicationId https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagededuplicationid-property.html2- Use a dynamodb table to deduplicate your event before getting to your SNS topic , i already used this pattern but as Outbox pattern but deduplicated as well is an state in that system, so i avoid generating events , simply using a simple TTL before Put Item and use conditional Expressions to avoid inserting if already existsbehind that the Dynamodb streams trigger a lambda which prepare the event and push it to the SNS topicWhich one to choose?If the deduplication is a concern in your consumer part and not in your own processing i propose using the SQS FIFO ,In the contrary if you care also about this problem in your processing as well i propose the Dynamodb Table solutionBTW its upon your need and design**A final word: In a fanout design avoid taking ownership of consumer Queues but if that part is in your Bounded context as well non problem **Having a look at following topics will help :Message Filtering PatternServerless Communication PatternsCommentShareOmid Eidivandianswered 7 months ago"


Question: EC2 instance unexpectedly reboots itself

"reboot   system boot  5.19.0-1024-aws  Mon May  1 20:06   still runningreboot   system boot  5.19.0-1022-aws  Mon Apr 17 18:22 - 20:00 (14+01:37)reboot   system boot  5.19.0-1022-aws  Wed Apr  5 21:22 - 20:00 (25+22:37)I'm checking root reason of the reboot, checked some system logs... didn't find any specific...Instance id : i-0dbafbb300887693dFollowComment"

Answer: "0Hi,As described on the following Knowledge Center page, an EC2 instance could be restarted due to multiple reasons, I recommend you take a look.If this is the first time it has happened, I would start checking if it is due to a scheduled event, in which AWS restart the instance when must perform tasks such as installing updates or maintaining the underlying host. In this case, AWS sends an email to the email associated with the AWS account prior to the scheduled event, providing details about the event, including the start and end date, but also sends an AWS Health event.You could also check the system logs at /var/log/messages or /var/log/syslog depending on your OS, to rule out that it is due to operating system-level issues.On the other hand, If the restarts are frequent, I would review if the health check are failing due to over-utilization of its resource, for which the following Knowledge Center page could be helpful.CommentShareMikel Del Tioanswered 15 days ago"


Question: IP access logs for LAMP instance with Load Balancer

"Hi all,I am using Lightsail with a LAMP instance and a load balancer.In the apache access logs I can only see the load balancer IP and not the original requester IP.Is there a way to include the origin / requester IP in the access logs of the instance behind the load balancer? How can I block certain bot/attacker IPs otherwise?Thanks!FollowComment"

Answer: 0Solution: https://www.techstacks.com/howto/log-client-ip-and-xforwardedfor-ip-in-apache.htmlCommentSharel4mp1answered 3 years ago


Question: Multi Thread / Process access to FPGA

"Hello.Is it possible to access the FPGA from multiple threads or processes ?Can I attach to the same FPGA from multiple processes at once?Can I open a DMA queue from two or more processes and transfer data to/from the same FPGA ?Can I use fpga_pci_peek() / fpga_pci_poke() from multiple threads in a process attached to the FPGA ?Example of utilization :I have 4 accelerators on the FPGA, each using a DDR4 memory and each having their own memory space (for control through OCL bus and for DDR4 access through PCIS bus) in the memory map.Can 4 different processes each use an accelerator through the provided interfaces safely ?Each attach to the same FPGA card on the same slot, each opening a DMA queue, or peeking/poking the OCL bus.I understand the PCIS and OCL bus will be shared and that a process may have to wait for it to be ready for a call to e.g., to fpga_pci_poke() to be completed.Is this situation possible / safe ?Thank you for your time.Regards.RickEdited by: rwe on Jun 26, 2019 5:55 AMAdded example.FollowComment"

Answer: "0Hi Rick,Yes, it is OK to use fpga_pci_peek/poke with multiple processes.  It is also OK to have different processes use different XDMA channels.Thanks,-AsifCommentShareawsasifanswered 4 years ago0Thank you for this information.Best Regards.RickCommentSharerweanswered 4 years ago"


Question: Updating a configuration

"Hello!I've managed to create a cluster configuration through the CLI (https://docs.aws.amazon.com/cli/latest/reference/kafka/create-configuration.html), and created a cluster that uses that configuration.How would I go about updating a configuration ("create a new revision?"), and what impact would that have on a cluster?I think I won't have to update the configuration often, and am mostly looking for the ability to react to operational changes (tune settings for performance, allow/disallow client behaviors such as deleting topics).I'm not "bound" to using the CLI, so if there is a REST API endpoint I'd be fine with that.FollowComment"

Answer: "0Accepted AnswerSorry for the delay. Have a look at this posting https://forums.aws.amazon.com/thread.jspa?threadID=304449&tstart=0 for updates.For deleting configuration, that feature is coming soon but is not available at the moment.Thank youCommentShareAWS-User-0635593answered 4 years ago0Incidentally: How would I delete a configuration? :)CommentShareankonanswered 4 years ago"


Question: CodeCommit - SNS Notification Beautify

"Is there anyway to beautify the CodeCommit Notifications instead of getting a large JSON?I was thinking of involving a Lambda function which receives that Large JSON and modify it, but i'm trying to find an easier way.This is an example of the JSON sent through the SNS Topic configured in CodeCommit Notifications (Replaced acct. number with ?):{"account":"??","detailType":"CodeCommit Repository State Change","region":"us-east-1","source":"aws.codecommit","time":"2022-02-10T14:18:25Z","notificationRuleArn":"arn:aws:codestar-notifications:us-east-1:??:notificationrule/a6d8f86a6feb72ca5b3c342a1c0e463e5394f5e1","detail":{"referenceFullName":"refs/heads/dev","repositoryId":"3f8a3a4f-eac4-4077-b13c-fd1ac749171e","referenceType":"branch","commitId":"822aef94a812d7d73c2382f3a68ca187ea67aec9","callerUserArn":"arn:aws:iam::??:user/athmfiuser","event":"referenceUpdated","repositoryName":"athm-dev-athmfi-onpremiseapi-pipeline","oldCommitId":"29a8687b679861dc257dba49b9440db85b0ced0a","referenceName":"dev"},"resources":["arn:aws:codecommit:us-east-1:??:athm-dev-athmfi-onpremiseapi-pipeline"],"additionalAttributes":{}}FollowComment"

Answer: 1I can think of two options:Create a lambda that that subscribes to the SNS topic and use the JSON to sendEmail or sendTemplatedEmail using SESUse InputTransformer to customize the message going into SNSCommentShareMODERATORMohamed_Banswered a year ago0An alternate solution is to use AWS Chatbot to forward pre-formatted CodeCommit notifications to your Slack channels.https://aws.amazon.com/blogs/devops/receive-aws-developer-tools-notifications-over-slack-using-aws-chatbot/CommentShareAbhijit_Banswered a year ago


Question: ExpiredCodeException in confirmPassword request with just recieved code.

"Hello. I have been implementing forgot password functionality for our js application working with this documentation - (https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_ForgotPassword.html). Verification code is properly sent to user email but when I try to finalize operation with next request(confirmPassword) I always get ExpiredCodeException with message: "Invalid code provided, please request a code again." Is there any way to check/set code lifetime or way to debug that feature? I'm not sure if I'm having some implementation issue or that is on Cognito side. Any help would be great.FollowCommentJaccoPK a year agoThe timeout is 24 hours and not configurable.https://github.com/amazon-archives/amazon-cognito-identity-js/issues/380Is the user email address confirmed before you try?Do you have a secret key on the userpool?ShareAWS-User-2789919 a year agoYes, email address that I was testing with is confirmed.There is no secret key set on user pool.ShareJaccoPK a year agoCan you give any details on what Lambda triggers you implemented?ShareAWS-User-2789919 a year agoI did not implement any lambda triggers  for this feature. Te only one trigger I see in project now launches after user is created.I used CognitoUser class with its .forgotPassword and .confirmPassword from https://www.npmjs.com/package/amazon-cognito-identity-js.Do I need any lambda functions for this to work?ShareJaccoPK a year agoNo I was just wondering if it didn't work because of triggers that were not properly configured. Now I'm thinking maybe your Cognito user pool is not configured properly. If you used CloudFormation or Terraform to created it would help seeing the properties.ShareShow 1 more comment"

Answer: "0The codes generated by the API ForgotPassword are valid for 1 hour[1].The error ExpiredCodeException normally occurs when you have already successfully used a valid code from a ForgotPassword API call with ConfirmForgotPassword and you attempt to make another ConfirmForgotPassword API call[2]. Once a code generated from ForgotPassword has been used once, it will be marked as no longer valid and you will begin to receive the ExpiredCodeException error. While a valid ForgotPassword Code is currently active for a Cognito User(e.g., a ForgotPassword API call has been made for that user in the past hour and the code has not been successfully used yet), the error you will get when an invalid code is provided to a ConfirmForgotPassword API Call is CodeMismatchException[3]. This error will continue to occur until either the code generated from ForgotPassword expires or is successfully used with a ConfirmForgotPassword API Call and is marked as no longer valid.Both ForgotPassword and ConfirmForgotPassword are logged in CloudTrail, so to debug this I would recommend that you go through your current ForgotPassword flow. Then check CloudTrail for the API Calls, first find the ForgotPassword API Call for the user you are testing, then look for any subsequent ConfirmForgotPassword API Calls. If there are multiple ConfirmForgotPassword API calls occurring after the ForgotPassword API Call and one of the ConfirmForgotPassword API calls is successful, then this would potentially be a possible cause of the ExpiredCodeException error. You can match Cognito users between the CloudTrail events by confirming the "sub" value in the "additionalEventData" is the same across all ForgotPassword/ConfirmForgotPassword CloudTrail events.[1] Quotas in Amazon Cognito - Non-adjustable resource quotas - https://docs.aws.amazon.com/cognito/latest/developerguide/limits.html#limits-hard[2]  ConfirmForgotPassword  - https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_ConfirmForgotPassword.html[3]  ConfirmForgotPassword  - Errors - https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_ConfirmForgotPassword.html#API_ConfirmForgotPassword_ErrorsCommentShareSUPPORT ENGINEERMichael_Hanswered a year ago"


Question: Is is safe to switch between addon and helm chart distribution of aws-ebs-csi-driver?

"Hi, I have multiple eks clusters and some of them are using addon, some helm chart. I want to unify the behavior by leaving only one of them. Is it considered safe just to remove the driver on fly and install different distribution of it?FollowComment"

Answer: "0Hello,Thank you for posting on AWS re:Post!I understand you are inquiring whether it is possible to switch between an installation of Amazon EBS CSI driver (aws-ebs-csi-driver) managed by Helm and an installation managed as an EKS addon. I am pleased to confirm that this is indeed possible!When an EBS-backed persistent volume claim is created, the Amazon EBS CSI driver automatically mounts the EBS volume to the node so it can be used by your pods. However, once the volume is mounted, the Amazon EBS CSI driver is not required to access data on the volume. In short, the Amazon EBS CSI driver is only required when making changes to persistent volume claims, not to access the data within them.Therefore, your pods will remain able to read and write to their persistent volume claims if you uninstall the Amazon EBS CSI driver and then re-install it in another way. However, you must bear in mind that if there are any worker node events (e.g. a pod needs to be replaced or a service is auto-scaled) then these operations could be delayed while the Amazon EBS CSI driver is being re-installed.I hope you find this information helpful. Please feel free to comment below if there is anything you would like me to elaborate on further!Regards,Marrick.CommentShareMarrick_Lanswered 5 days ago"


Question: Unused issued certificate issue

"HelloI have a "xyz.com" domain with a local domain provider (camo hosting)I'm using Route 53 from aws as DNS manager and ACM as certificate manager."xyz.com" and "www.xyz.com" point to a static IP address of aws and are dedicated to a showcase site and stored on Amazon S3.The "*xyz.com" subdomains point to a CloudFront link and are dedicated to a web application.Recently my certificates expired,Aws has automatically renewed a "*xyz.com" certificate for subdomains which works well, this certificate is valid used and eligible for renewal.However, the domains "www.xyz.com" and "xyz.com" were not taken into accountI manually issued a certificate for these two domains. This validated, it was issued with an ineligible status and is not currently used by the 2 domains in question.How to make these two domains use this certificate in question or a valid Other ❓Ps: The CNAMEs of the domains "*xyz.com", "xyz.com" and "www.xyz.com" are well informed and configured without errors on route 53 in the zone hosted by the domain "xyz.com" which has been verified since .FollowComment"


Question: num_repeats effect on costs (DWave's solvers)

The QBSolv solver has the attribute num_repeats. What does this attribute mean with regard to Braket's pricing structure of tasks and shots? Is every additional num_repeat an additional task or an additional shot (or "something else")?Thanks in advance.FollowComment

Answer: "0Accepted AnswerHello, and thank you for your interest in Braket! The num_repeats parameter on D-Wave's qbsolv does not correspond directly to Braket tasks or Braket shots. The qbsolv algorithm decomposes an annealing problem into smaller subproblems which are then sent to a sampling device. The num_repeats parameter controls how many times this happens, but the number of subproblems generated on each pass of the algorithm is dependent upon the input annealing problem which is to be solved. You can try using D-Wave's TrackingComposite (documentation) with one of the local samplers to test out how qbsolv may slice up your annealing problem.CommentShareStephen Fanswered a year ago"


Question: Does AWS Java SDK 2.0 Support Transcribe Services

"Hello Community,I'm trying to use the AWS SDK 2.0 for Java to programmatically kick off a transcription job.  Looking at the API specification for 2.x, it does look like the classes are available for me to use.BUT...I'm writing a Spring Boot app with Gradle and I cannot for the life of me find any GAV coordinates for importing the appropriate transcribe dependencies.Is it not supported yet in the 2.x SDK or am I just missing it?  I tried to guess it based on the S3 GAV coords (I tried 'software.amazon.awssdk:transcribe'), but that didn't work.Any help would be greatly appreciated!DavidEdited by: DavidAF on Jan 16, 2019 9:01 AMFollowComment"

Answer: "0Ah, I got it.  I'm using their BOM to manage dependency versions and I guess transcribe is not in the BOM yet.  Appending the version was all I needed.Silly me...CommentShareDavidAFanswered 4 years ago"


Question: Logging IoT Secure Tunnel Connections in Cloudwatch or Similar

"Hi,For audit trail/security purposes we would like to store a log of AWS IoT Secure Tunnel instances, including dates, durations, etc. If possible, even the originating IP address.What would be the easiest way to log IoT Secure Tunnel instances in Cloudwatch ? I can see that tunnels are displayed in the IoT dash, but these disappear if the tunnel is deleted.Current setup:Secure Tunnel Component deployed via GreengrassUsing the provided docker-run.sh / localproxy cli method to connectIf we wish to close tunnels before timeout we are using the "close/delete" tunnel method via the AWS dashFollowComment"

Answer: "0Hi. If you have AWS IoT logging enabled (INFO level), Publish-Out events will be logged in the AWSIotLogsV2 log group. When you create/open a tunnel, there will be a Publish-Out event on the $aws/things/thing-name/tunnels/notify topic.For broader audit trail and compliance scenarios you should probably consider CloudTrail: https://docs.aws.amazon.com/iot/latest/developerguide/iot-using-cloudtrail.htmlCommentShareEXPERTGreg_Banswered a year ago"


Question: Workspaces with trusted device by AWS Private CA

"Customer want to deploy a "secure" AWS workspaces by only trust "specific device" within their office environment. Based on the following link, we can use "certificate" to trust the device. While based on AWS CA hierarchy best practice, we should have at least two tier CA (root + subordinate).https://docs.aws.amazon.com/workspaces/latest/adminguide/trusted-devices.htmlhttps://docs.aws.amazon.com/acm-pca/latest/userguide/ca-hierarchy.htmlThus, we have two questions:Can we only use "single" root CA to issue certificate for Workspace users? (i.e. no subordinate CA in the design)Although "Root + subordinate" CA provide better security, each AWS private CA costs US$400. Under this situation, should we charge 2 x $400 for two CA even it is under the "same" hierarchy?FollowComment"

Answer: "0Accepted AnswerYes. Designing a CA hierarchy (as you mention) is following security best practices. but still, you can choose to have just one PCA in place for this project. As usual, customer is responsable for taking these decisions, knowing and accepting the risks.ACM-PCA pricing is based per PCA. So yes again, if you have a Root + Subordinate you will have to pay for both, regardless if they are or are not under same hierarchy.CommentShareEXPERTRodrigo@AWSanswered 3 years ago"


Question: AWS Lex seems to be broken for specific slots like Amazon.EmailAddress and Amazon.US_FIRST_NAME

"This is started happening few hours ago, it was working fine and suddenly stopped working, Amazon.EmailAddress is not working at all.Its on region=us-west-2 and on LexV1For Amazon.EmailAddress => none of the email addresses are detectedAlso for Amazon.US_FIRST_NAME behaviour is strange,John => not detected as first nameJonathan => working fineFollowComment"

Answer: "0Thanks for reaching out.We are looking into this issue, Can you please provide the answer of the below questions for us to debug furtherIs it only for Lex V1 ?Is the bot using advance option = false ?Is it in first turn or slot elicitation turn ?Can you please DM us with your AccountId, BotId and few requestIds or reach out through AWS customer support for us to debug further ?ThanksTaniaCommentSharetania_awsanswered 4 months agorePost-User-7146884 4 months agoIs it only for Lex V1 ?=> Yes I think and also i tried same intent on another v1 bot and its working fine, looks like issue is bot specific.Is the bot using advance option = false ?=> No, Advanced option is enabledis it in first turn or slot elicitation turn ?=> First turn onlyRegarding DM I'm not sure how to DM details,Thanks for your help.Share0Thanks for getting back to us.Can you also confirm if you have configured a dialogue or fulfillment lambda?Are you facing issues for audio inputs or text inputs or both?Can you please open a customer case with us with the accountId, botId, requestId details for us to help you further?CommentSharetania_awsanswered 4 months agorePost-User-7146884 4 months agoI tried by creating new bot of v1 and i thought i'll copy all my intents over there, but this issue started coming again, i only have one intent with slot which accepts email and nothing else,Can you also confirm if you have configured a dialogue or fulfillment lambda?=> No it just returns parameters.Are you facing issues for audio inputs or text inputs or both?=> facing issue for both.Can you please open a customer case with us with the accountId, botId, requestId details for us to help you further?=> Actually its a paid service so we cannot afford.RequestID: 325e2728-d075-43a8-92c7-eb2480f535c7Share0Checked the above requestId. Seems like the Advanced options are disabled for this bot. Can you please rebuild the bot with it enabled and try again?This blog for best practices on bot building can also support you in building a bot that will work well for your use case. - https://aws.amazon.com/blogs/machine-learning/best-practices-for-creating-amazon-lex-interaction-models/Hope this helps.CommentSharetania_awsanswered 4 months agorePost-User-7146884 4 months agoThanks Tania for the link I'll go through this linkShare0We further looked into the issue and identified an issue in our system which is not honoring the advanced options. We are actively working on a fix. Thank you for your patience while we remediate this.CommentSharetania_awsanswered 4 months agorePost-User-7146884 4 months agoThanks for the confirmation, I'll wait for the fixShare"


Question: Development machine for multiple developers

"Hi,I'm looking for a smart solution for offering my developers a shared environment and at the same time optimizing costs. Let me explain what we do today.I have 10 developers each of them with a personal EC2 instance. They use Ubuntu and then NICE DCV to remotely connect to the desktop and work from there. Usually, they use the EC2 for developing our internal application (coding Java, Terraform, Python, HTML, lambda, go), experimenting, and spinning up containers and when done they push the code to our centralized Git repository (Gitlab).Now, in this scenario, I have too many EC2 instances (we try to shut them down overnight and turn them on in the morning but quite often we incur into lack of on-demand resources and we have to wait).WHAT I WOULD LIKE TO ACHIEVE?Ideally, I believe I could still make use of EC2 instances but by sharing one or more with multiple users, I don't see the need of having a personal EC2 instance. I could still leverage NICE DCV configured for multiple sessions, and my developers will share the same EC2 instance. I could then turn off/on the EC2 instances when not used and benefit from on-demand capacity reservation for example to be sure (I'm not really convinced reserved instances is the right approach for my case).Do you have a smart idea for a better setup for the scenario described above?Thank you ;)FollowCommentKarstenNISP 4 months agoIndeed NICE DCV offers the very efficient sharing of a Linux instance with multiple users with e.g. 8-10 or more full-performance virtual DCV desktops depending on the resource usage. This works with GPU-sharing or without. It seems reserved instances is the right way to go in case of not getting hold of instances when needed - or you keep it running and add sessions when needed. What instance type are you looking at?Share"

Answer: "0Hi @Escape82,I have some thoughts to contribute to your design.First, for completeness, know that AWS Workspaces provides for managed cloud workstations (including Linux).  Although I do find it valuable at large scale, I am not sure it is necessary for your target of ~10 users nor am I sure it will be less expensive than an EC2 based solution.Yes, you could share EC2 instances among developers.  I think this reduces flexibility due to interference between developers whereas they would have otherwise been unimpeded by coordination with others (burst load conflict, reboots, system level packages installed).  But if you do not need to increase the size of the instances then running fewer of them will be less expensive.  In all cases but especially here you will probably need non-local user accounts - for example user auth from Active Directory or similar.You may also be able to reduce your cost even with per-developer EC2 instances by using the AWS Instance Scheduler, which automates stopping/starting of EC2 instances.  This prevents "forgetting" to stop the instance overnight, and you can schedule it to pre-start shortly before it is needed (avoiding waiting for an instance to start).  Maybe this small tweak is all you need to reduce your costs acceptably.You can also reduce costs by using Savings Plans.    This is an alternative to instance reservations that is much more flexible.  Consider looking at the Savings Plans recommendations in your AWS Cost Explorer to see if they are a fit for your usage patterns.Lastly, I have deployed Cendio ThinLinc with success.  This is not an AWS offering - Cendio is a third-party.  But I used this with a previous employer to create a group of Linux EC2 instances for them to remote into for work.  It has multiple users per Linux instance, and load balances among them.  The users do not have to pick a machine, they will get back into their same machine or another if their session ended, that machine recycled, or for load reasons.  So users always connect to the same endpoint and it "just works" for them even though there are multiple shared Linux instances on the back-end.  However note setting this all up is non-trivial, so it is an investment best for long-term use and at scale.  I am not sure if it would be worthwhile for your case.Hope this helps.Regards,RyanCommentShareRyan_Manswered 4 months agoEscape82 4 months agoThank you Ryan for your detailed answer. Actually, what would you recommend for solving the problem of the lack of ondemand resources? This is a problem when I need to start up 10 EC2 instance in the morning and in my region or AZ there not enough AWS resources for spinning up a the EC2 instance I need. Saving plan, based on my understanding, would not solve this case, unless I go for reserved instances (or I don’t switch off the EC2).Thank youShareRyan_M 4 months agoCorrect, Savings Plans do not reserve capacity whereas an instance reservation would.  If you otherwise would have only run these instances for 25% of a week (ex: 40 hours) then the instance reservation(s) would not be saving you money.I believe the instance type will have a significant effect on on-demand capacity availability.  Using a common instance type and modern generation (ex: M6g) may help.  What instance type and region are you using?  You might consider an AWS Support request for what instance types are plentiful in your region, in case they are able to share that information.You could also automate saving an instance to an AMI and terminating it rather than just stopping it, and then be able to re-launch it as a different but compatible instance type.  So if your first instance type choice has a capacity conflict you could retry with another instance type or a different AZ.  Or if there are multiple acceptable regions for you, try re-launching in another region.Hopefully finding and using a more plentiful instance type is all you need to do, rather than purchase reservations or retry using different instance types/zones/regions.Regards,RyanShare"


Question: AttributeError: 'DataFrame' object has no attribute '_get_object_id'

"Using the Zeppilin notebook server, I have written the following script. The initialization is taken from the template created in glue, but the rest of it is custom. I'm getting the error:AttributeError: 'DataFrame' object has no attribute '_get_object_id'when I run the script. I'm pretty confident the error is occurring during this line:datasink = glueContext.write_dynamic_frame.from_catalog(frame = source_dynamic_frame, database = target_database, table_name = target_table_name, transformation_ctx = "datasink")but I can't decipher what it's trying to tell me. Can anyone please help me out or point me in the right direction? Thanks!%pysparkimport sysfrom pyspark.context import SparkContextfrom pyspark.sql.functions import lit, current_timestampfrom pyspark.sql.window import Windowfrom pyspark.sql.types import IntegerType, StructType, StructField, LongTypefrom awsglue.context import GlueContextfrom awsglue.transforms import *from awsglue.utils import getResolvedOptionsfrom awsglue.dynamicframe import DynamicFramefrom awsglue.job import Jobdef dfZipWithIndex (df, offset=1, colName="RID"):    '''        Enumerates dataframe rows is native order, like rdd.ZipWithIndex(), but on a dataframe         and preserves a schema        :param df: source dataframe        :param offset: adjustment to zipWithIndex()'s index        :param colName: name of the index column    '''    new_schema = StructType(                    df.schema.fields +                            # previous schema                    [StructField(colName,LongType(),True)]        # new added field in front                )    zipped_rdd = df.rdd.zipWithIndex()    new_rdd = zipped_rdd.map(lambda (row,rowId): (list(row)+ [rowId +offset]))    return spark.createDataFrame(new_rdd, new_schema)    # if len(sys.argv) == 0:sys.argv.append('TEST_ETL_JOB')print sys.argv# args = getResolvedOptions(sys.argv, ['JOB_NAME'])args = {}args['JOB_NAME'] = 'TEST_ETL_JOB'print argssc = SparkContext.getOrCreate()glueContext = GlueContext(sc)spark = glueContext.spark_sessionjob = Job(glueContext)job.init(args['JOB_NAME'], args)# etl job configurationschema_mapping = [("name", "string", "NAME", "string"), ("carrier id", "bigint", "DESCRIPTION", "string")]source_file_name = "foo.csv"source_database = "test016"source_table_name = "5b3f6af0-9f34-4a6f-9c7d-c531b513f459-25a17d34_1f16_4013_9035_38783be0ba9b"target_database = "test016"target_table_name = "udf_udf_evt"# Create a DynamicFrame using the uploaded tablesource_dynamic_frame = glueContext.create_dynamic_frame.from_catalog(database=source_database, table_name=source_table_name)# the mapping from CDF to UDF should appear heresource_dynamic_frame = ApplyMapping.apply(frame=source_dynamic_frame, mappings=schema_mapping)# convert from dynamic frame to data frame to append columnssource_data_frame = source_dynamic_frame.toDF()source_data_frame = source_data_frame.withColumn('SOURCE_ID', lit(0))source_data_frame = source_data_frame.withColumn('TYPE_ID', lit(0))source_data_frame = source_data_frame.withColumn('DELIVERY_ID', lit(0))source_data_frame = source_data_frame.withColumn('SOURCE', lit('XXX'))source_data_frame = source_data_frame.withColumn('NOTES', lit('YYY'))# inject the source filename into the data framesource_data_frame = source_data_frame.withColumn('SOURCE_FILE', lit(source_file_name))# inject the row number from the source file into the data framesource_data_frame = dfZipWithIndex( source_data_frame, 1, 'SOURCE_ROW' )# inject the current date and time into the data frame for created and last modifiedsource_data_frame = source_data_frame.withColumn('DATE_CREATED', current_timestamp() )source_data_frame = source_data_frame.withColumn('DATE_LASTMODIFIED', current_timestamp() )# convert the data frame into a dynamic framesource_dynamic_frame = DynamicFrame(source_data_frame, glueContext)datasink = glueContext.write_dynamic_frame.from_catalog(frame = source_dynamic_frame, database = target_database, table_name = target_table_name, transformation_ctx = "datasink")#print out information about this data frame for testing and debug#print "Count: ", source_dynamic_frame.count()source_dynamic_frame.show()job.commit()Edited by: Kindle Customer on Oct 11, 2018 4:02 PMFollowComment"

Answer: "0Spark does lazy evaluation so , error occur when you call action (write). Could you try without dfZipWithIndex function ?source_data_frame = dfZipWithIndex( source_data_frame, 1, 'SOURCE_ROW' )CommentShareShivananswered 5 years ago0Thanks, but unfortunately that didn't change the error code. Any other suggestions?For what it's worth, when I uncomment the following line and comment out the write_dynamic_frame line, the table is show as expected without error.source_dynamic_frame.show()Error:Traceback (most recent call last):  File "/tmp/zeppelin_pyspark-444437833802934152.py", line 367, in <module>    raise Exception(traceback.format_exc())Exception: Traceback (most recent call last):  File "/tmp/zeppelin_pyspark-444437833802934152.py", line 355, in <module>    exec(code, _zcUserQueryNameSpace)  File "<stdin>", line 55, in <module>  File "/usr/share/aws/glue/etl/python/PyGlue.zip/awsglue/dynamicframe.py", line 591, in from_catalog    return self._glue_context.write_dynamic_frame_from_catalog(frame, db, table_name, redshift_tmp_dir, transformation_ctx, additional_options)  File "/usr/share/aws/glue/etl/python/PyGlue.zip/awsglue/context.py", line 247, in write_dynamic_frame_from_catalog    return DataSink(j_sink, self).write(frame)  File "/usr/share/aws/glue/etl/python/PyGlue.zip/awsglue/data_sink.py", line 32, in write    return self.writeFrame(dynamic_frame_or_dfc, info)  File "/usr/share/aws/glue/etl/python/PyGlue.zip/awsglue/data_sink.py", line 28, in writeFrame    return DynamicFrame(self._jsink.pyWriteDynamicFrame(dynamic_frame._jdf, callsite(), info), dynamic_frame.glue_ctx, dynamic_frame.name + "_errors")  File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1124, in __call__    args_command, temp_args = self._build_args(*args)  File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1094, in _build_args    [get_command_part(arg, self.pool) for arg in new_args])  File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 289, in get_command_part    command_part = REFERENCE_TYPE + parameter._get_object_id()  File "/usr/lib/spark/python/pyspark/sql/dataframe.py", line 1020, in __getattr__    "'%s' object has no attribute '%s'" % (self.__class__.__name__, name))AttributeError: 'DataFrame' object has no attribute '_get_object_id'Source Code:%pysparkimport sysfrom pyspark.context import SparkContextfrom pyspark.sql.functions import lit, current_timestampfrom pyspark.sql.window import Windowfrom pyspark.sql.types import IntegerType, StructType, StructField, LongTypefrom awsglue.context import GlueContextfrom awsglue.transforms import *from awsglue.utils import getResolvedOptionsfrom awsglue.dynamicframe import DynamicFramefrom awsglue.job import Jobdef dfZipWithIndex (df, offset=1, colName="RID"):    '''        Enumerates dataframe rows is native order, like rdd.ZipWithIndex(), but on a dataframe         and preserves a schema        :param df: source dataframe        :param offset: adjustment to zipWithIndex()'s index        :param colName: name of the index column    '''    new_schema = StructType(                    df.schema.fields +                            # previous schema                    [StructField(colName,LongType(),True)]        # new added field in front                )    zipped_rdd = df.rdd.zipWithIndex()    new_rdd = zipped_rdd.map(lambda (row,rowId): (list(row)+ [rowId +offset]))    return spark.createDataFrame(new_rdd, new_schema)    # if len(sys.argv) == 0:sys.argv.append('TEST_ETL_JOB')print sys.argv# args = getResolvedOptions(sys.argv, ['JOB_NAME'])args = {}args['JOB_NAME'] = 'TEST_ETL_JOB'print argssc = SparkContext.getOrCreate()glueContext = GlueContext(sc)spark = glueContext.spark_sessionjob = Job(glueContext)job.init(args['JOB_NAME'], args)# etl job configurationschema_mapping = [("name", "string", "NAME", "string"), ("carrier id", "bigint", "DESCRIPTION", "string")]source_file_name = "foo.csv"source_database = "test016"source_table_name = "5b3f6af0-9f34-4a6f-9c7d-c531b513f459-25a17d34_1f16_4013_9035_38783be0ba9b"target_database = "test016"target_table_name = "udf_udf_evt"# Create a DynamicFrame using the uploaded tablesource_dynamic_frame = glueContext.create_dynamic_frame.from_catalog(database=source_database, table_name=source_table_name)# the mapping from CDF to UDF should appear heresource_dynamic_frame = ApplyMapping.apply(frame=source_dynamic_frame, mappings=schema_mapping)# convert from dynamic frame to data frame to append columnssource_data_frame = source_dynamic_frame.toDF()source_data_frame = source_data_frame.withColumn('SOURCE_ID', lit(0))source_data_frame = source_data_frame.withColumn('TYPE_ID', lit(0))source_data_frame = source_data_frame.withColumn('DELIVERY_ID', lit(0))source_data_frame = source_data_frame.withColumn('SOURCE', lit('XXX'))source_data_frame = source_data_frame.withColumn('NOTES', lit('YYY'))# inject the source filename into the data framesource_data_frame = source_data_frame.withColumn('SOURCE_FILE', lit(source_file_name))# inject the row number from the source file into the data frame# source_data_frame = dfZipWithIndex( source_data_frame, 1, 'SOURCE_ROW' )source_data_frame = source_data_frame.withColumn('SOURCE_ROW', lit(0))# inject the current date and time into the data frame for created and last modifiedsource_data_frame = source_data_frame.withColumn('DATE_CREATED', current_timestamp() )source_data_frame = source_data_frame.withColumn('DATE_LASTMODIFIED', current_timestamp() )# convert the data frame into a dynamic framesource_dynamic_frame = DynamicFrame(source_data_frame, glueContext)datasink = glueContext.write_dynamic_frame.from_catalog(frame = source_dynamic_frame, database = target_database, table_name = target_table_name, transformation_ctx = "datasink")#print out information about this data frame for testing and debug#print "Count: ", source_dynamic_frame.count()source_dynamic_frame.show()job.commit()Edited by: Kindle Customer on Oct 12, 2018 10:34 AMCommentShareKindle Customeranswered 5 years ago0I was able to track down the issue. This line doesn't work:# convert the data frame into a dynamic framesource_dynamic_frame = DynamicFrame(source_data_frame, glueContext)It should be:# convert the data frame into a dynamic framesource_dynamic_frame = DynamicFrame.fromDF(source_data_frame, glueContext, "dynamic_frame")CommentShareKindle Customeranswered 5 years ago"


Question: Long running EC2s with Cloudformation and ServiceCatalog

"My organisation runs with a number of fairly tight controls for how we can build our environments. One of these is that infrastructure must be deployed via the service catalog using products that have been validated before being published for use. This means we rely on Cloudformation to build the EC2 servers we use.My use case for EC2 is as long running compute nodes within a cluster that store data. This requirement comes from the software in use. This means they are not sacrificial like a web server or app server might be.As such using autoscaling groups for the EC2s is not ideal, as I don't want the EC2s being terminated unexpectedly as this can cause data loss, and also new nodes need to be configured into the cluster which would mean a loss of capacity.What I'm looking for is suggestions on how to manage the EC2s using Cfn and ServiceCatalog without having to push new versions of the Service Catalog products when we want to make changes to the EC2s, such as volume sizes, or instance types. One option could be custom macros and various lambdas but this gets complex quickly.Terraform works well for this but is not an option due to the restrictions I mention around service catalog.TIA....FollowComment"

Answer: "0Hello,Service Catalog  basically operates similar to Cloudformation. If you don't want to create new versions for every new changes, you can parameterize the attributes  and just update the cloudformation stack with new values to update the EC2. An important point to keep in mind is  that update to some properties can trigger resource replacement and you mentioned you don't want your EC2 to be replaced and If you are looking to update any value without replacement, it won't be possible for properties that require replacement natively.If you need  granular review of your environment for the use case , we require details that are non-public information. Therefore, kindly open a support case with AWS using the following https://console.aws.amazon.com/support/home#/case/createCommentShareAditya_Tanswered 7 months ago"


Question: Lightsail returning self-signed certificate

"I am using a custom domain that I own and attached that to LightSail distribution.As part of the process, I also created a certificate, validated it in my DNS registrar(Namecheap) and then attached it.But when I make a call to my website:curl -v https://whatify.ioI see that it is returning a self signed certificate.Below is the relevant output:* ALPN, offering http/1.1* successfully set certificate verify locations:*  CAfile: /etc/ssl/certs/ca-certificates.crt*  CApath: /etc/ssl/certs* TLSv1.3 (OUT), TLS handshake, Client hello (1):* TLSv1.3 (IN), TLS handshake, Server hello (2):* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):* TLSv1.3 (IN), TLS handshake, Certificate (11):* TLSv1.3 (OUT), TLS alert, unknown CA (560):* SSL certificate problem: self signed certificate* Closing connection 0curl: (60) SSL certificate problem: self signed certificateMore details here: https://curl.se/docs/sslcerts.htmlI can see on the LightSail distribution tab in AWS console that the ceritifcate is valid and in-use. Why is this happening?FollowComment"


Question: Configure MQTT over Websockets [ActiveMQ}

"How does one configure ActiveMQ to use MQTT over Websockets ? I was unable to find proper documentation on how to set up the configuration file for this. However, I tried adding the following lines to the configuration file<transportConnectors><transportConnector name="mqtt+wss" uri="wss://myURLhere:61620"/></transportConnectors>This produced the following errorThe specified XML configuration data is invalid: cvc-attribute.3: The value 'mqtt_ws' of attribute 'name' on element 'transportConnector' is not valid with respect to its type, 'protocol'. and cvc-enumeration-valid: Value 'mqtt_ws' is not facet-valid with respect to enumeration '[openwire  ]'. It must be a value from the enumeration.Any help to resolve the issue would be appreciated. Thanks !Edited by: abhai on Feb 23, 2021 10:47 PMEdited by: abhai on Feb 23, 2021 10:47 PMEdited by: abhai on Feb 23, 2021 10:48 PMFollowComment"

Answer: "0Hi abhai,The websocket endpoint is already available to you. You can find details on our supported protocols here:https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/broker.htmlIt supports both STOMP over websocket as well as MQTT over websocket.Please note that we only support SSL endpoints so make sure you use that setting on the client side.Thanks,MithunCommentSharemithunm-awsanswered 2 years ago"


Question: Uniqueness of AWS resource ids

"Each resource has their own Unique ID in format specified here.How unique are these Unique ID, are they unique Globally for all AWS accounts or specific to account or region?And are there any chance for this ID to repeat (in case if resource is deleted).Could anyone clarify me on this?FollowComment"

Answer: "1Hi,Amazon Resource Names (ARNs) uniquely identify AWS resources. They are composed by partition, service, region, account-id, resource-id (and sometimes with resource-type).In some services the resource-id can be the name of the resource, so the only thing that can differentiate is the  partition or region, for example, two SQS with the same name, in the same account, but in different regions:arn:aws:sqs:us-east-1: 555555555555:teste-43523523543arn:aws:sqs:us-east-2: 555555555555:teste-43523523543Best regards,Ricardo MakinoCommentShareRicardo_Manswered 6 months agorePost-User-8310475 6 months agoI got how ARN is unique to a resource and cases where resource name can be resource-id. But I still have doubt on resources where resource-id is randomly generated like EC2, VPC. Does the resource-id for those resource has the chances to be same for different account.Share"


Question: Failure to detach EBS volume

The volume isn't attached to an instance but shows as 'in-use' under State (with no Attachment Information).The only Action I can do is 'Force Deatch Volume' and it fails with the following error:Error detaching volumevol-0fc7ea8bc6ebf5338: Volume 'vol-0fc7ea8bc6ebf5338'is in the 'available' state.region: us-west-2cVolume ID: vol-0fc7ea8bc6ebf5338Could you please detach and delete it for me?Thanks.FollowComment

Answer: "0Hello jonnyc,Thank you for letting us know about the problem. The issue seems to be related to our underlying AWS infrastructure and so I have created an internal ticket for our service team to deal with it. I will inform you when the issue is resolved.Thank you for your patience and please accept our apologies for the inconvenience caused.Regards,awstomasCommentShareAWS-User-3892169answered 4 years ago0Hello jonnyc,Thank you for your patience while awaiting my follow up response.Our service team has resolved the issue with your EBS volume and so you can now proceed to delete it.Once more lease accept our apologies for the inconvenience caused.Have a nice day,awstomasCommentShareAWS-User-3892169answered 4 years ago0Thanks!CommentSharejonnycanswered 4 years ago"


Question: When will there be a data api for Aurora Serverless V2 and V3?

"Please can you tell me when the data api is coming to Aurora V2 and V3? Also, why is this not supported yet? There are lots of people stuck on Aurora V1 because this feature is not available.Thanks,AdamFollowComment"

Answer: "0Unfortunately, it doesn't look like it is even on the roadmap; we'll have to keep waiting. You can read the AWS Support answer to this other question: Is DataAPI for Aurora v2 / v3 planned?CommentShareIvananswered 6 days ago"


Question: customization of the forgot password email

"Hi,I need to how ho to customize the email that Cognito sends when the user clicks on the "forgot password"ThanksFollowComment"

Answer: 1Accepted AnswerThe answer to your question is Custom message Lambda trigger.Amazon Cognito invokes  trigger before it sends an email or phone verification message or a multi-factor authentication (MFA) code. You can customize the message dynamically with your custom message trigger.The below doc describes the various ways you can use a custom message lambda.https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-custom-message.htmlCommentShareJoel_Ponukumatlaanswered 4 months ago