Question: Synschronous Glue Job in Step Function is slow to recognize completion of Glue Job

"I am using a Step Function to execute a Glue Job. The Step Function is set to run in synchronous mode, however, there is usually a 2-4 minute lag from Glue Job completion to the point at which the Step Function considers the Glue Job complete and moves to the next step. For example, the Glue Job's last run took 15 minutes but the Step Function spent 19 minutes on this step. Has anyone else experienced this? Is my only option to execution in async mode and poll more often for completion?FollowComment"

Answer: "1Accepted AnswerThe reason why your experiencing this delay is because Glue does not support cloudwatch event for notifying the step functions with the latest status. Same is the case with EMR as well. Currently, by default the polling schedule is every 1 minute for the first 10 minutes, then every 5 minutes thereafter. Therefore, if the job is taking more than 10 minutes to complete it's execution the you can expect a delay of an average of 2.5 minutes with 5 minutes being the worst case. The only way is we poll its status by making Describe* api call to EMR/Glue up to every 5 minutes. The step function team knows about this issue and are trying to implement a solution.The workaround that you can implement on your end is to make use of a Lambda function to make Describe API calls to describe EMR/Glue job status more often than Step Functions does.If you require in depth assistance about this issue then I would advice you to raise a support case with the Technical Support team of Step Functions.CommentShareSUPPORT ENGINEERChaituanswered 8 months agotjtoll 8 months agoThank you for responding. We'll implement a workaround for now.Share"


Question: adminitiateauth and refresh tokens

"We use the adminInitiateAuth API in our backend to authenticate our clients. We have deployed our Auth API endpoints using API gateway and AWS Lambda with User poolsA web app user authenticates with cognito via our api and the backend admininitiateauth call returns access, id & refresh tokens. Then when the user refreshes their tokens and passes the refresh token to our api we see that admininitiateauth only returns access & id token and not an new refresh token.How is the user expected to refresh the next time? Does a user only get one refresh?There is NOTHING in the documentation as you why refreshing tokens via admininitiateauth does not return a new refresh token as well.Anyone seen this and got any tips?FollowComment"

Answer: 0adminitiateauth never returns a new refreshtokenI misunderstood how the refreshtokens work.By increasing expiry time of refreshtoken we can extend the amount of time before the user needs to fully login again to obtain a new refresh tokenCommentShareMark Lloydanswered 4 years ago


Question: IoT Mobile App Architecture with AWS

"Hello,I want to build an IoT Mobile App for a students project. For this, I want to use AWS. I never used AWS before, so my question is maybe a little bit stupid.I want to send data from my IoT devices to AWS. For this, I am using the IoT core. This works fine. But I dont know what to do next.  I know that you can define rules for incomming MQTT topics. But I dont now how to use them. I need four things:Store data in a database (I think I can just use Dynamo DB in the rules section. But is there maybe another/better way / another DB?)Request the data from the Mobile AppSend data from the Mobile App to AWS and publish the data to my IoT devicesDelete Data from the database.How can I do this or what serviec/tools I need to use?Thanks for your help.FollowComment"

Answer: "0Rules give your devices the ability to interact with AWS services. Rules are analyzed and actions are performed based on the MQTT topic stream [1]. You can also see examples of all the services integration [2]. For dynamoDB , The DynamoDB (dynamoDB) action writes all or part of an MQTT message to an Amazon DynamoDB table [3].Storing your incoming data and how to access is thereafter depends upon the requirements. example is this more of analytics use case or transactional nature. If your mobile app wants to publish some data to IoT devices in real time, then best is to write it to MQTT topics which they can subscribe.There are some good blogs that you can refer as well [4]https://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.htmlhttps://docs.aws.amazon.com/iot/latest/developerguide/iot-rule-actions.htmlhttps://docs.aws.amazon.com/iot/latest/developerguide/dynamodb-rule-action.htmlhttps://aws.amazon.com/blogs/mobile/bluetooth-low-energy-ble-iot-mobile-application-with-aws-amplify-aws-iot-and-swift/CommentShareEXPERTAWS-User-Nitinanswered 6 months agoTTStudi 6 months agoHi, thanks for your answer. One more question: Is there any way to create a project where I can work together with another person? Or do we have to share one account?Share0Some additional useful links:https://aws.amazon.com/blogs/mobile/iot-with-aws-appsync/https://github.com/aws-samples/aws-alexa-workshop-smarthome-uihttps://github.com/aws-samples/aws-end-to-end-iot-amplify-demohttps://github.com/aws-samples/aws-appsync-iot-core-realtime-dashboardCommentShareEXPERTMassimilianoAWSanswered 6 months ago"


Question: Fargate profile all subnets disabled

"I'm trying to set up an EKS cluster and have successfully created one.But I couldn't set up a Fargate profile to be used by it. In the corresponding form, a drop-down field named Subnets lists all my subnets; but none is selectable/enabled.I've tried to create a new subnet, but no luck. What I've missed?FollowComment"

Answer: "0Hi,according to this,https://repost.aws/knowledge-center/fargate-troubleshoot-profile-creation I assume you are trying to associate private subnets, as they are the only one supported.Was profile associated with VPC?CommentShareEXPERTalatechanswered 19 days agoEmran 10 days agoSorry for my late response. I'm totally new to AWS and I've tried to create private subnets (and/or another vpc with private subnets) but no luck. I searched a bit and created a separate vpc with private subnets but they don't appear in Fargate profile configuration form at all.Share"


Question: Get Attributes from Amazon Connect to Amazon Lex

"Hi,I have in my contact flow this:Stored Customer Input ---> Set contact attributes: Key: numberInput, Type: System, Attribute: Stored customer input ---> play prompt ---> Get Customer Input: With my Lex's BOTand in the lambda of my Lex's BOTvar phone = event.Details.ContactData.Attributes.numberInputs;but when I try to get the customer's number I got nothing.what Im doing wrong? what Im missing?FollowComment"

Answer: 0Just I had to use Session attributesand call them by var xxxx = event.sessionAttributes.numberInputsCommentShareMiguelGanswered 4 years ago


Question: Neptune openCypher queries returning on wrong connection at high concurrency

"I've spent the last few days attempting to debug a pretty surprising issue with our use of Neptune in our API servers. The symptom was, if we tried to execute the same query on two connections, but with different parameters, sometimes the connections would get the responses meant for the other connection.We discovered this using a bolt client, with connection pooling. I stripped back our connection pooling, then worked all the way back to manually creating two separate connections and executing our queries. I set it up so each connection would only execute the same query over and over. Around 15% of the time, we still see this behavior.Finally, to isolate against any errors in the bolt client, I implemented the same behavior using the HTTPS endpoint instead of bolt and I am still able to produce this behavior. It's hard to fathom this could be true, but I'm struggling to see how this isn't an issue with Neptune itself.With bolt, I was doing something like:resetbeginrun <query> <parameters>pullcommitWith HTTP, I'm just simply doing:POST https://host:port/openCypher<http stuff>query=<query>parameters=<parameters>With both of these implementations, with a concurrency of 2, I start seeing corrupt responses in as few as 10 requests. The query itself is small and only returns a few records.What could I be doing wrong?FollowComment"


Question: Supporting mutual TLS on specific resource paths

"I need to setup a proxy service in AWS that provides mutual TLS termination and then forwards the call onto another service running in AWS using just regular TLS. I have followed the guidance given for configuring mutual TLS on an API Gateway and have that working with an Lambda function integration that can call the necessary service on the back end. The problem I have is that I don't want all incoming requests to require mTLS, only specific resource paths on the custom domain should enforce mTLS. Any thoughts on how I could go about configuring this ... it seems that mTLS gets enabled for the whole custom domain?https://mydomain/pathA  (require mTLS)https://mydomain/pathB  (don't require mTLS)If the API Gateway can't support this, any other thoughts on how to achieve the same mTLS => TLS proxy conversion in AWS?FollowCommentvivekk a year agoHello TAW, Did you find any feasible solution to this scenario?Share"

Answer: "0mTLS is configured on the Custom Domain level so you can decide that only some paths will require it and other will not. What you can do is create two APIs with two different domains, e.g., mtls.api.com and tls.api.com. The first will map to the API for all routes that require mTLS. The second will map to an API that does not require it.CommentShareEXPERTUrianswered a year agoTAW a year agoUnfortunately this solution needs to work with a legacy system where ALL resource paths need to be off of the same domain ... having two separate domains (one with and one without mTLS) is not an option. Your comment suggests what I was already thinking ... that it is not going to be possible to do what I need to do with the API Gateway mTLS support.Share0Hello,I recently came across a blog post within the Compute Blog, which addresses mutual Transport Layer Security (TLS) and how a customer's self-managed Kafka clusters can have a trust relationship established between AWS Lambda using a private certificate authority (CA), even using a Amazon Managed Streaming Kafka (MSK) certificate by default as the certificates are signed by Amazon Trust Services CAs.Also, without knowing many more of the customer requirements and/or dependencies of the applications, proxy configuration(s) involved, or that may need to be involved yet,You can activate any combination of authentication modes (mutual TLS, SASL SCRAM, or IAM access control) on new or existing clusters. This is useful if you are migrating to a new authentication mode or must run multiple authentication modes simultaneously. Lambda natively supports consuming messages from both self-managed Kafka and Amazon MSK through event source mapping.Also, I just saw this related topic within another re:Post, if the solution necessitates API Gateway.I hope this helps.GabrielCommentShareGabriel_Sanswered a year ago"


Question: reference resources between CDK stacks

"Hi Team,I have 2 CDK stacks one in the Front end project and the second in the Backend project.the first stack needs resources ref from the second stack (and vice versa can be true)I have a lambda function in the backend stack that needs to call another lambda in the front end stack,If I deploy the backend CDK stack first, how can I make a reference/call to the second lambda in the front end stack?If I use event bridge or SNS I still need to have the name of the target lambda which is the lambda created by the CDK front endor I need the SNS ref to be shared between the 2 stacks for pub/sub.any idea how to solve this :)Thank you.FollowComment"

Answer: "2CloudFormation supports the concept of exporting outputs from one stack and importing their values into another stack.In CDK, you can export a resource's output by creating a cfnOutput with an exportName property. You can also use Fn.ImportValue to import an exported value to a stack from a different stack.Here's a blog post that illustrates this concept in further detail.Note that neither CDK nor CloudFormation support circular stack dependencies, so you can't have stack A depend on a resource in stack B while also having stack B depend on a resource in stack A. The stack dependency graph must be acyclical.CommentShareEXPERTMichael_Fanswered a year ago0CDK can handle export/import of the resource variable automatically.If you make a resource a member variable of the stack instance, and reference that stack member variable from the other stack by passing the variable as a property of stack props parameter, CDK will export the resource reference, and import it from the other stack automatically.You can find more detailed information from https://docs.aws.amazon.com/cdk/v2/guide/resources.html#resource_stack.Be aware that this operation only supported in a same region stacks. If you need to deploy the two stacks in two different regions, you need other mechanism.CommentShareJunseong_Janswered a year ago"


Question: Query Id: ddfe7caf-be98-4c33-a2f1-b50502a22961

amazon athena takes long time to run my queryI want all s3 object lists in csv formatFollowComment

Answer: "1HiS3 Inventory (not real time) can be an option depending on what you are trying to achieve. You can use Athena to query this CSV also. Overall Amazon S3 Inventory is one of the tools Amazon S3 provides to help manage your storage. You can use it to audit and report on the replication and encryption status of your objects. You can also simplify and speed up business workflows and big data jobs using Amazon S3 Inventory, which provides a scheduled alternative to the Amazon S3 synchronous List API operation. Amazon S3 Inventory does not use the List API to audit your objects and does not affect the request rate of your bucket. Refer to S3 inventory pricing at https://aws.amazon.com/s3/pricing/.Amazon S3 Inventory provides comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or a shared prefix (that is, objects that have names that begin with a common string).For more informationhttps://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.htmlCommentSharerePostfanclubanswered a year agoEXPERTFabrizio@AWSreviewed a year agoFabrizio@AWS EXPERTa year agobased on the limited information on the question , the option from this answer seems the most appropriate.Share1You need to consider the volume of data you're trying to scan - not just the amount you expect to retrieve, but how much data will have to be scanned to gather the retrieved data.  If you have a lot of data you'll need to partition it and limit your query to target less partitions to avoid timeouts and long queries.  As an example, see this blog post about using Partition Projection for Athena queries on CloudTrail logs in S3 - https://www.linkedin.com/pulse/using-athena-partition-projection-query-cloudtrail-other-kinsman/?lipi=urn%3Ali%3Apage%3Ad_flagship3_profile_view_base_post_details%3Brq0yhJ20SQKlSN9blwev9g%3D%3DCommentShareEXPERTskinsmananswered a year agoEXPERTFabrizio@AWSreviewed a year ago"


Question: Very fast volume Snpashot

"I took a Snapshot of my volume, but it usually takes a few minutes to be ready, but I've noticed that some are ready 100% in secondsFollowComment"

Answer: "0Hi there!Please note snapshots are incremental backups, which means that only the blocks on the device that have changed after your most recent snapshot are saved.This minimizes the time required to create the snapshot and saves on storage costs by not duplicating data. Please refer to [1] on Amazon EBS snapshots.I hope you find the above information helpful.Have a great day ahead!References:[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSSnapshots.htmlCommentShareLettyanswered a year ago"


Question: AWS Config : Accessing AWS Config NonComplaint rules and resources

I am trying to import the noncompliant rules and noncompliant resources from the AWS Config Dashboard. It there a way to do it using CLI or the console.FollowComment

Answer: "0Accepted AnswerHi - if you are looking for a way to export information from AWS Config about your current rules and resources, I'd recommend taking a look at the Advanced Query feature - https://docs.aws.amazon.com/config/latest/developerguide/querying-AWS-resources.html.  This will let you query information from AWS Config using a SQL-like query language.  As an example, this is a sample query for retrieving a list of non-compliant resources:SELECT  configuration.targetResourceId,  configuration.targetResourceType,  configuration.complianceType,  configuration.configRuleListWHERE  configuration.complianceType = 'NON_COMPLIANT'This feature can be accessed via either the CLI, or through the AWS Console.CLI - https://docs.aws.amazon.com/config/latest/developerguide/query-using-sql-editor-cli.htmlConsole - https://docs.aws.amazon.com/config/latest/developerguide/query-using-sql-editor-console.htmlCommentShareDavid_Ranswered a year agoEXPERTJames_Sreviewed a year ago"


Question: DMS from Aurora to Redshift

"Customer has the following requirements.Getting new patient data file into S3 bucket.Loading patient data file into Aurora.Filtering rows based on criteria and storing them in separate file to place it in S3 bucket.Loading the selected records into Redshift which in turn will be visible through QuickSight.In order to automate this process, we are making use of Lambda functions where in –a.Client will place new file in to S3 bucket.b.This will trigger Lambda function which will load this file into Aurorac.Using filtration criteria selected rows will be copied into new file. This file will be placed into S3 bucketd.A new Lambda function will load this file into Redshift.I was thinking about using DMS to directly  replicate the data from Aurora into Redshift. Is this possible? is there any model or use case available to validate this?If no, is there any advise on how to achieve this from the DB perspective?FollowComment"

Answer: "0Accepted AnswerDMS supports both Aurora as a source and Redshift as a target.When using Redshift as a target, DMS uses an intermediate S3 bucket, where changes are stored before they are loaded into Redshift. The size of the file that triggers a load into Redshift can be customized as specified in http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.ConnectionAttributes.html#CHAP_Introduction.ConnectionAttributes.RedshiftCommentShareAWS-User-4665333answered 6 years ago"


Question: VPC with no Virtual Private Gateway. Is it possible?

Is it possible to have a VPC with no Virtual Private Gateway?Could a VPC with no Virtual Private Gateway connect directly to a Direct Connect Gateway?The reason would be in case there is possible reduced latency with this setup.FollowComment

Answer: "0Accepted AnswerYes, it is possible. See the option 3 in the below link. You can use a Transit Gateway (Transit VIF) along with DirectConnect Gateway, in this scenario you do not need VGW.https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/direct-connect.htmlCommentShareEXPERTTushar_Janswered 6 hours ago0It is possible to connect your VPC to Direct Connect via the following methods:Virtual Private GatewayDirect Connect Gateway and Virtual Private GatewayDirect Connect Gateway and Transit GatewayThis article provides information on how to pick which is best for your use case (including considering latency) - https://docs.aws.amazon.com/wellarchitected/latest/hybrid-networking-lens/performance-efficiency-pillar.html#:~:text=VPN%20connectivity%20options-,AWS%20Direct,-Connect%20options%20includeReferences:Building a Scalable and Secure Multi-VPC AWS Network Infrastructure - https://docs.aws.amazon.com/whitepapers/latest/building-scalable-secure-multi-vpc-network-infrastructure/direct-connect.htmlAWS DX – DXGW with AWS Transit Gateway, Multi-Regions, and AWS Public Peering - https://docs.aws.amazon.com/whitepapers/latest/hybrid-connectivity/aws-dx-dxgw-with-aws-transit-gateway-multi-regions-and-aws-public-peering.htmlConnect your network to AWS with hybrid connectivity solutions - https://www.youtube.com/watch?v=uq8C-DmrKXcHope this helps.CommentShareCarlos del Castilloanswered 2 days agoEXPERTkentradreviewed 2 days agorePost-User-3360667 13 hours agoThank you Carlos for these documentation references. I had read these before I posted my question. The documentation does not answer my question explicitly which is why I posted it.Share"


Question: Is it possible to manually create a bootable UEFI RHEL 8 AMI?

"We have a requirement to produce AMI built to CIS hardened standards from a Red Hat Enterprise Linux 8 OS including custom partitioning scheme with mount points of /home, /var, /var/log, etc. Our requirement dictates the AMI can only have a single root EBS backed device. Multiple EBS devices are not satisfactory in this situation. Also note, we are running the process via Packer in automated pipelines.Our first approach was attempting to modify a running EC2 RHEL 8 instance's root EBS device using the AWS CLI, but with the introduction of the 6 hour cool-down period from the time an EBS device is created to when it can be modified, we had to abandon this approach.When spinning up a new EC2 instance, the root partition will chew up all configured disk space. We have also tried shrinking the root volume partition to make room for additional partitions, but abandoned that approach.The closest we have made it is to use an auxiliary RHEL8 server and snapshot it's root EBS device. Then, using Packer, spin up a new 'source' EC2 instance that starts with the source's root EBS device, as well as a blank EBS device (we'll call myroot).At the beginning of the Packer build, the source devices look like the following:[root@ip-172-31-67-219 ec2-user]# lsblkNAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTnvme0n1     259:0    0   10G  0 disk├─nvme0n1p1 259:1    0  200M  0 part /boot/efi├─nvme0n1p2 259:2    0  512M  0 part /boot└─nvme0n1p3 259:3    0  9.3G  0 part /nvme1n1     259:4    0   10G  0 diskNote, source OS device is named /dev/nvme0n1, and the empty myroot device is /dev/nvme1n1. During the build, we reach out and create a new EBS device from the auxiliary server's snapshot, and attach it to the source server. At this point, we have the following configuration:[root@ip-172-31-67-219 ec2-user]# lsblkNAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTnvme0n1     259:0    0   10G  0 disk├─nvme0n1p1 259:1    0  200M  0 part /boot/efi├─nvme0n1p2 259:2    0  512M  0 part /boot└─nvme0n1p3 259:3    0  9.3G  0 part /nvme1n1     259:4    0   10G  0 disknvme2n1     259:5    0   10G  0 disk├─nvme2n1p1 259:6    0  200M  0 part├─nvme2n1p2 259:7    0  512M  0 part└─nvme2n1p3 259:8    0  9.3G  0 partNote the new /dev/nvme2n1 device as the auxiliary server's non-running OS drive.Our intent is to now use the aux server's OS files to make a running OS out of our blank myroot device.We can do this by partitioning and copying OS files to new temporary mount points on myroot device. The result of the partitioning and file movement process shows this configuration:[root@ip-172-31-67-219 ec2-user]# lsblkNAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTnvme0n1     259:0    0   10G  0 disk├─nvme0n1p1 259:1    0  200M  0 part├─nvme0n1p2 259:2    0  512M  0 part /boot└─nvme0n1p3 259:3    0  9.3G  0 part /nvme1n1     259:4    0   10G  0 disk├─nvme1n1p1 259:9    0  191M  0 part /boot/efi├─nvme1n1p2 259:10   0  488M  0 part /newboot├─nvme1n1p3 259:11   0  4.3G  0 part /mnt/myroot/root├─nvme1n1p4 259:12   0    1G  0 part /mnt/myroot/var├─nvme1n1p5 259:13   0    1G  0 part /mnt/myroot/varlog├─nvme1n1p6 259:14   0    1G  0 part /mnt/myroot/varlogaudit└─nvme1n1p7 259:15   0    2G  0 part /mnt/myroot/homeNote: we remove the temporary /dev/nvme2n1 disk prior to creating and modifying the /boot and /boot/efi partitions.  We have also modified myroot's /etc/fstab file with the appropriate partition identifiers in preparation for creating a new AMI off myroot device.[root@ip-172-31-67-219 ec2-user]# cat /mnt/myroot/root/etc/fstabUUID=DCE5-01EE  /boot/efi    vfat    defaults,uid=0,gid=0,umask=077,shortname=winnt 0    2UUID=2fe05fef-dab6-4ced-be2d-f7a9bd613269  /boot        xfs     defaults                                       0    0UUID=2ca0c201-48ae-482b-850c-d26dc26b9de0  /            xfs     defaults                                       0    0UUID=3830ef77-5f8d-4a73-bec8-c7f7d4085bba  /var            xfs  defaults,nofail,nodev                          0    0UUID=85b31b3b-f420-4adb-a221-204d79718626  /var/log        xfs  defaults,nofail,nodev,nosuid                   0    0UUID=26531d55-fa36-47f2-9778-5f035fb6fac7  /var/log/audit  xfs  defaults,nofail,nodev,nosuid                   0    0UUID=3506a1fd-a1b9-4748-8c07-a978eca2bc7f  /home           xfs  defaults,nofail,nodev,nosuid                   0    0When an AMI is created from myroot EBS device, we run into problems with launching an EC2 instance. EC2 status checks for reachability are not satisfied and fail after about 5 minutes, and viewing system boot logs in the console has brought us to a GRUB loader screen, but it doesn't seem to find the root RHEL8 OS on myroot device.I feel I am close to getting the instance to launch, but I'm missing something in either the /boot or /boot/efi configuration points.Has anyone successfully manipulated these directories and grub loader configurations and launched an EC2 instance?FollowComment"

Answer: "1I've done exactly this, you're basically doing the same as I did (slice up a second disk into all your mountpoints and rsync the first disk across to the second).You need to specifically pay attention to the character devices in /dev, making sure the ownership, perms, major & minor numbers are carried across. And do an install -Ddm 000755 on the bind devices on the new disks. Also remember to re-make your GRUB config at the end.I also found it useful to disable FIPS mode and SELinux just while I got it working, these can be re-enabled later.I found this very helpful indeed https://github.com/plus3it/amigen8 and looking at how it was done in here got me over a lot of hurdles.CommentShareRWCanswered a month agoadam gantt a month ago@RWC Appreciate the pointers, and thanks for pointing me to the amigen8 repo. Looks like I have some homework ahead of me tomorrow.ShareRWC a month agoYou're welcome. Much as I would like to be able to post my code here (or put it on a public repo in github) I believe my employer will consider it to be their IP, which is why I have to stick to vague, general advice. The amigen8 repo is great though - clone it and run it yourself and you'll find it works. Then pick through the code and see what it's doing that you've missed.Another couple of things - don't worry about a separate boot device, just fold it into the root filesystem. It will still work, and grub2-install & grub2-mkconfig will take care of pointing things at the right place. Also your new root disk can be any size you like, doesn't need to match the original.Shareadam gantt a month agoI understand being under the corporate IP; same here so I can't publish what I have thus far. While we were able to glean a few nuggets from the amigen8 repo, they built theirs to a CentOS system using LVM volumes. We are stuck using RHEL 8 system without the ability use LVM.Unfortunately the Grub2 loader configuration and installation process they use does not work for secure UEFI devices, which I've ran into past testing of my solution. So still stuck at trying to get the /boot and /boot/efi mounts configured correctly as the last step in order to rip off an AMI and get an instance.Share"


Question: MWAA CloudWatch metrics - Billing too high

"Hi, we're using MWAA and we're charged almost 300$/month for Cloudwatch metrics, for example, our last month:$0.30 per metric-month for the first 10,000 metrics - EU (Ireland) 907.597 Metrics  $272.28But when I go to Cloudwatch > Metrics > All Metrics I can only see a total of 11,056 metrics, so why is the metric number in billing so high?Also, we don't actually use metrics, they were automatically set up by MWAA, so can we disable them as we just need MWAA logs but we don't care about metrics?Thanks,FollowComment"

Answer: "3Hello,Please see https://docs.aws.amazon.com/mwaa/latest/userguide/access-metrics-cw-202.html#choosing-metrics regarding choosing which metrics you would like to Airflow/MWAA to report.CommentShareJohn_Janswered 6 months agoAWS-User-8775889 6 months agoThanks Jhon,After your reply we completely disabled metrics with these configuration options:metrics.statsd_allow_list = nonemetrics.statsd_on = FalseCloudwatch metrics spent already decreased by half yesterday, and we expect to reach almost 0 today.Since this metric collection can become a large extra cost ( in our case almost 300 USD/month), it should be disabled by default or it should be explicit in the MWAA setup that this will be an extra cost and give the option to disable it.Best!Share"


Question: How can I use AWS re:post for other than AWS questions ?

How can I use AWS re:post for other than AWS questions ?.  Can we use this for answering any Programming troubleshooting questions ?FollowComment

Answer: "2Hello rePost-User-9083111,Thank you for your interest in re:Post. AWS re:Post is designed to help users with their cloud journey using AWS Services.You can participate in several ways:Answering questions posted. They are related to AWS Services.Upvoting right answers that you read.Asking questions related to AWS services.Reading and upvoting articlesWhen a question you ask is answered and the answer is correct, marking it as accepted answers.We look forward to your contributions and welcome to re:Post!CommentShareSUPPORT ENGINEERMonica_Lluisanswered 8 months agorePost-User-9083111 8 months agoWell, I wants to ask a Question other than AWS ? is that possible ?ShareMonica_Lluis SUPPORT ENGINEER8 months agoCommunity members would not be able to help with questions not related to AWS Services. I recommend you use other forums that are more appropriate for your non AWS related questions.Share0Hello. Thank you for using re:Post for your AWS queries. Current re:Post guidelines encourages AWS related questions.Please see current 'Guidelines for asking questions' on re:Post:Do's:Ask your question as concisely as possible.Provide helpful details that could aid in others providing a good answer.Include prior research or steps you may have taken to attempt to resolve the issue.Select the tag(s) that best relate to your question.Don'ts:Include personal information such as your organization, a customer you're asking on behalf of, or vendor.Include confidential information such as your AWS Account ID or workload-specific configurations.Post questions that include inaccurate information.Select tags that are not relevant to the question.Post questions that are not related to AWS.CommentShareAWS-User-2870192answered 8 months ago"


Question: Inbuilt Video analytics module in AWS Greengrass

"Hello All,I am new to Amazon Web Services. I have a requirement to capture live video stream from an IP surveillance camera and perform object detection at real time. We already have the customized object detection model developed in python. We even have dockerized the same.I have the following questions :Is there any built -in module or function available in AWS Greengrass for capturing live video frames from IP surveiilance camera? For instance, Microsoft Azure already has an out of the box module called Live Video Analytics on IoT Edge. Kindly suggest.We want to deploy the customized AI module as discussed above in AWS Greengrass such that it can receive the incoming video frames per second and return the detected person counts. The output needs to be pushed to AWS IoT Core. Kindly suggest.Is it possible to achieve this using the architecture as discussed below ?https://aws.amazon.com/blogs/iot/improving-industrial-safety-with-video-analytics-aws-iot-core-and-aws-iot-greengrass/Thanks & RegardsAmitFollowComment"

Answer: "0Kindly suggest. It's a bit urgentCommentShareAm-400answered 3 years ago0Hi,To answer your questions:Greengrass have a list of connectors at https://docs.aws.amazon.com/greengrass/latest/developerguide/connectors-list.html . There is an MLImageCassification connector that you might be interested in, however it requires that the ML model is trained by the SageMaker image classification algorithm . As for " capture live video stream from an IP surveillance camera ", Greengrass doesn't have a connector for this. However you can experiment to create a pinned lambda for this function.2 & 3. I don't have the full context of your project so my answer might be inaccurate. From my understanding this should be achievable. You can write your lambda to include the customized object detection model , and have the lambda take camera input every second. The lambda can publish output to specific topics and Greengrass can configure the topic route.CommentShareShirley-awsanswered 3 years ago0Hi Shirley,Thanks for your valuable inputs. By pinned lambda, do you mean to write a custom lambda function that will be able to read the streaming video? Also, is it possible to push an existing docker image to AWS Container repository and pull the same into a lambda deployed in AWS Greengrass?Edited by: Am-400 on Sep 6, 2020 10:30 PMCommentShareAm-400answered 3 years ago0Hi,Pinned lambda is a long-lived lambda, detailed explanation is here https://docs.aws.amazon.com/greengrass/latest/developerguide/lambda-functions.html#lambda-lifecycle . You probably need to configure that lambda's containerization, so that it has access to camera/IO/etc. https://docs.aws.amazon.com/greengrass/latest/developerguide/lambda-group-config.html#lambda-containerization-considerationsYou can pull docker image into your Greengrass device using the docker connector https://docs.aws.amazon.com/greengrass/latest/developerguide/docker-app-connector.html.CommentShareShirley-awsanswered 3 years ago0ThanksCommentShareAm-400answered 3 years ago"


Question: Adding a User to Simple AD Directory in Workspaces - AWS CLI

"I have recently created a Simple AD directory for the purpose of  launching a bunch of Workspaces. Normally, I create the workspace through the console when a user requests one. First, I add the user to the Simple AD through the workspace management console, then I launch the workspace for that user. Now, I am looking for a way to add the user and create the workspace though the CLI. However, it seems that AWS only allows Simple AD user addition through the console. The only alternative I found was to add the user through WorkDocs, but this way, I have to set the user's password (required parameter) rather than having the user set their own password. Additionally, my intention is to create a workspace, not a WorkDocs site. I don't understand why would AWS let you programmatically add the user to Simple AD through WorkDocs, but not through Workspaces. Any feedback is greatly appreciated.Edited by: JayBachir on Apr 26, 2019 11:49 AMFollowComment"

Answer: "0Hi Jay,Thanks for the feedback. WorkSpaces currently doesn't support adding users via AWS CLI (or using the APIs). However, we have recorded this request and will evaluate for future delivery.Thanks,HassanCommentShareAWS-User-3798414answered 4 years ago"


Question: Executing hive create table in Spark.sql -- java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found

"The Issue:We have a Spark EMR cluster which connects to a remote hive metastore to use our emr hive data warehouse.When executing Pyspark statement in Zeppelin notebook: sc.sql("create table userdb_emr_search.test_table (id int, attr string)")Got this exception:java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not foundEMR Spark cluster configures:Release label:emr-6.3.0Hadoop distribution:Amazon 3.2.1Applications:Spark 3.1.1, JupyterHub 1.2.0, Ganglia 3.7.2, Zeppelin 0.9.0The class org.apache.hadoop.fs.s3a.S3AFileSystem has its ClassPath on spark class path correctly:'spark.executor.extraClassPath', '....:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/:....'spark.driver.extraClassPath', '....:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/:....Jar files are in the right places:Under /usr/lib/hadoop:-rw-r--r-- 1 root root  501704 Mar 30  2021 hadoop-aws-3.2.1-amzn-3.jarlrwxrwxrwx 1 root root      27 Sep  8 01:50 hadoop-aws.jar -> hadoop-aws-3.2.1-amzn-3.jar-rw-r--r-- 1 root root 4175105 Mar 30  2021 hadoop-common-3.2.1-amzn-3.jarlrwxrwxrwx 1 root root      30 Sep  8 01:50 hadoop-common.jar -> hadoop-common-3.2.1-amzn-3.jarUnder /usr/share/aws/aws-java-sdk/:-rw-r--r-- 1 root root 216879203 Apr  1  2021 aws-java-sdk-bundle-1.11.977.jarHadoop storage:Use Amazon S3 for Hadoop storage instead of HDFSError log when executing spark sql create table in Zeppelin notebook:WARN [2022-09-05 03:24:11,785] ({SchedulerFactory3} NotebookServer.java[onStatusChange]:1928) - Job paragraph_1662330571651_66787638 is finished, status: ERROR, exception: null, result: %text Fail to execute line 2: sc.sql("create table userdb_emr_search.test_table (id int, attr string)")Traceback (most recent call last):File "/tmp/1662348163304-0/zeppelin_python.py", line 158, in <module>exec(code, _zcUserQueryNameSpace)File "<stdin>", line 2, in <module>File "/usr/lib/spark/python/pyspark/sql/session.py", line 723, in sqlreturn DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in callanswer, self.gateway_client, self.target_id, self.name)File "/usr/lib/spark/python/pyspark/sql/utils.py", line 117, in decoraise converted from Nonepyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found)INFO [2022-09-05 03:24:11,785] ({SchedulerFactory3} VFSNotebookRepo.java[save]:144) - Saving note 2HDK22P2Z to Untitled Note 1_2HDK22P2Z.zplnPlease help investigate why spark sql cannot see the class org.apache.hadoop.fs.s3a.S3AFileSystem even its jar files are in right place and have correct ClassPath.FollowComment"

Answer: "0Hi,Thanks for writing to re:Post.As I understand you are facing an issue on your spark job that is failing with the exception " java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found".As stated in this document [1] the usage of s3a filesystem is not recommended, and the suggestion is to leverage the EMRFS (i.e. using the s3:// scheme in place the s3a:// scheme).The EMRFS will grant you best performance, security, and reliability.That being said, in order to solve your issue, I would suggest you to replace the uri scheme "s3a://" with "s3://". And let us know how it goes at your end.Thanks.[1] Work with storage and file systems - https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-file-systems.htmlCommentShareSUPPORT ENGINEERMudasser_Aanswered 9 months agorePost-User-9712417 9 months agoThanks for your response. The ClassNotFoundException only occurs when using spark.sql to create a hive table. We can insert data into an existing hive table without issues. Also tried s3:// and got the same error.Share"


Question: Lambda Authorizer Cookies as identity source

I am trying to send cookies IdToken but my authorizer is not receiving it. but If I use header.authorization it works.My use case is to validate access token by header.authorization and IdToken sent from browser via cookies.FollowComment

Answer: "0Have you tried providing a token source header as mentioned here - https://docs.aws.amazon.com/apigateway/latest/developerguide/configure-api-gateway-lambda-authorization-with-console.html under point 9a - "Type the name of a header in Token Source. The API client must include a header of this name to send the authorization token to the Lambda authorizer."CommentShareEXPERTIndranil Banerjee AWSanswered 8 months ago0yes, header.authorization works. but header.cookies does not work.CommentSharerePost-User-6968266answered 8 months ago0I have the same issue, I believe someone on the internet theorised that the implicit cloudfront in front of your rest api is blocking the cookie header. As far as I know there is no way to fix this for REST apis, the only option seems to be to setup a (regional?) HTTP api instead and use the v2 payload which includes cookie headers. Unfortunately the http api doesn't have some features of the rest api.CommentShareAdamanswered 6 months ago"


Question: Not able to access RDS console

"I am unable to modify the RDS aurora cluster, located in Ireland region,  due to unexpected error as in the screenshot attached. I tried to clear caching and cookies, using different browsers and devices and even different users but no luck.FollowComment"

Answer: 1I'd suggest clicking on the "contact us" link - it will most likely lead to the support team who can assist you with this particular problem. There's not much that we here on re:Post can do.CommentShareEXPERTBrettski-AWSanswered 2 months ago


Question: Seems that validation is wrong when creating a certificate for a load balancer in lightsail

The validation message says Domain names much be at least 2 characters which is correct. However it fails to validate .co.uk domains which are 2 chars or any other 2 char domain. It actually demands a 3 digit top level which is not correct.FollowComment

Answer: "0Thanks for using Lightsail!Thanks for reporting this issue to us! Lightsail engineer will work on this issue ASAP!YiranCommentShareYiran-AT-AWSanswered 9 months agonathan-l 9 months agoOkay. Thanks will it be updated on here when I can try and create it again.? ThanksSharerePost-User-3889567 9 months agoHello, I have same issue with my domain ( isdtech.com.pe ) is not possible create certificates.Share0CommentSharerePost-User-3889567answered 9 months ago0Hi,Sorry that you've been experiencing problems with inputting domains with 2 character top-level domains. I'm happy to provide the update that this issue has since been fixed and you should be able to create certificates with domain name like 'domains.com.au' again.LeslieCommentShareAWS-Leslieanswered 9 months ago"


Question: Cognito vs Identity Center (SSO)

"I am building a web application. Customers should have a valid AWS account to onboard. Each customer could be a whole corporate on their own with their own Identity provider.The application should authenticate users of each Customer's org and authorize their access to certain APIs within my application. The application should also be able to run automation in the customer's AWS account by assuming certain IAM role.Looking at Identity solutions from AWS, I see native IAM, Cognito, and SSO.Native IAM doesn't present the identity of the user and their group membership to my application.Cognito seems to fit my use case. I can provide the customer with Cloudformation template to run in their account to prepare things: Cognito user pool, certain group name that my application looks for, certain IAM roles for my application to assume, and Cognito Identity pool to exchange the user's authenticated Identity with IAM temp creds to run automation in their account within certain permissions scope.The customer can integrate Cognito with their own IDP to have centralized user and group management.Does this solution look sane? Does SSO provide better integration for my application? If yes, does SSO allow me to provide the customer with a Cloudformation template to configure their SSO before they can onboard to my application?Related Q prior to rePost era: https://stackoverflow.com/questions/48767172/whats-the-difference-between-aws-sso-and-aws-cognito - but it is not answered yet.FollowComment"

Answer: "1In regard to your use-case, cognito user pool would indeed be a better option, as you would have access to certain security feature such as Advance security feature, integration with WAF, customizing the login page, which would provide you an edge with your web application.That been said, If you have a user-base in AWS SSO and would like to use that, you may create AWS SSO as SAML provider in AWS CognitoCommentShareSUPPORT ENGINEERVarunanswered 7 months ago"


Question: 403 Error on endpoint in sagemaker

We updated our endpoint on Sagemaker and we get 403 errors that lasts a couple of hours and then resolves themselves without us doing anything. Does anyone know why this is happening or how we could solve it?FollowComment

Answer: "0Hello, could you provide the detailed error message please ?CommentShareAWS-User-8651363answered a year ago"


Question: EC2 to RDS write down speed

"we have migrated our current db from a dedicated server, on OVH, to an postgres RDS DB on AWS. In our old set up, we had the postgres db running on ubuntu os. The same server also ran the scripts that were writing data to the db. It was lightning fast because it was writing locally. I was hoping having a RDS db in the same VPC as a separate EC2 instance would be fast as writing it locally. (I know that was silly) What are my solutions here? Is this even possible? We are paying 10x the amount for the RDS as compared to the dedicated server. We are new to AWS so help is needed.FollowComment"

Answer: "1Hello,Thanks for reaching out to AWS re:Post - In this scenario I would recommend to check "why" the slowness is seen. Is it due to network bottleneck - is it due to RDS resource bottleneck or something other than that.Now - the solution would be either to troubleshoot the bottlenecks or revert back to EC2 via Database Migration Service or by native dump methods - I would also recommend you to talk to your account manager (in-case you have a support plan) and request for a Solutions Architect to be on your project to do a well architected reviewCommentShareEXPERTNIRUPAM_danswered 10 months agoChris_Mainline 10 months agoThanks for your reply. If we had the RDS on a private subnet and the web app on a public subnet, both with in the same group, would the write down be as fast as if the web app was just writing down locally to a db with in the instance? The current set up is web app and postgres db are on the same server so write downs take 10 seconds for our script. When running the same script on a EC2 and writing down to the RDS db it takes 1:30 minutes. Is it even possible to get it as fast as the old set up?Share0Dear Customer - Thank You for writing back - Well, in this case network speeds needs to be measured - please feel free to reach out to our AWS Support Team and they will be able to assist you by diving deep on your network and confirming if they see any bottleneckHope I was-able to assist in your-taskCommentShareEXPERTNIRUPAM_danswered 10 months ago"


Question: Not able to send or receive mails

"Hey,I did configure SES correctly, but i am still enable to send or receive mails from : example@domain.com.from the verifying identity, i did verify my domain name and one of my mails address, the other is pending because i am enable to receive the verification link.SMTP is configure correctly. and is out of sandbox.Also : i can receive emails from amazon only.Note : i can send mails normaly from FluentSMTP linked to AWSFollowComment"

Answer: "0Hello,For sending mail, please review the following knowledge base article.This can be caused by:Template Rendering Failures.Account being in the Amazon SES sandbox. (you already ruled this out)Issue with the email recipient's email address.Issue with the email recipient's server.Your email getting labeled as spam.Sending delays.Ongoing issues with SES.You can check the above link for the reasons and resolutions to the above. As for receiving mail please review the following knowledge base article in addition to the setup docs here.While you have verified your domain name, be sure to publish the necessary MX record and give permission to SES for email receiving per instructions from the above docs.Finally you can also determine if your account is under review per the following knowledge base article and take steps to lift restrictions.CommentShareKameron_Banswered 3 months ago0If you have a verified domain identity then you do not need to verify email address identities within the domain. Furthermore, you should not verify email address identities within a verified domain identity because: a) the more specific email address identity supersedes the domain identity when you send messages, and b) email address identities cannot be authenticated which can lead to deliverability issues. Once your domain identity is verified, you can send as any email address within the domain.To test sending email using the AWS console, use the "Send test email" button that is available within the domain identity configuration, then choose "Custom" as the scenario.CommentShareJesse_Tanswered 2 months ago"


Question: AWS S3 / EC2: Downloading file with file-saver giving CORS error

"I am using AWS S3 to host my website and set my bucket with the follow CORS settings:<?xml version="1.0" encoding="UTF-8"?>  <CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">  <CORSRule>      <AllowedOrigin>*</AllowedOrigin>      <AllowedMethod>GET</AllowedMethod>      <AllowedMethod>PUT</AllowedMethod>      <AllowedMethod>POST</AllowedMethod>      <AllowedMethod>DELETE</AllowedMethod>      <AllowedMethod>HEAD</AllowedMethod>      <MaxAgeSeconds>3000</MaxAgeSeconds>      <AllowedHeader>*</AllowedHeader>  </CORSRule>  </CORSConfiguration>  When I try to download a file saved on my EC2 instance with the Javascript Library File-Saver, I get the following error:  Access to XMLHttpRequest at 'EC2-Instance' from origin 'S3-Bucket' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.The line that is causing the error from File-Saver is:function d(a) {var b = new XMLHttpRequest();return b.open("HEAD", a, !1), b.send(), 200 <= b.status && 299 >= b.status;  <-- This line}When I simulate the request in curl with verbose on, I get the following info:< Access-Control-Allow-Origin: *< Access-Control-Allow-Methods: GET, PUT, POST, DELETE, HEAD< Access-Control-Max-Age: 3000< Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-MethodAny idea why I am still getting this error?EDIT:I tried the following request to see header details:curl -H "Access-Control-Request-Method: GET" -H "Origin: S3 Bucket" --head -IXHEAD EC2_InstanceAnd am getting the following info without CORS info for some reason:HTTP/1.1 200 OKServer: nginx/1.15.8Date: Sun, 24 Feb 2019 04:38:20 GMTContent-Type: application/octet-streamContent-Length: 6950Last-Modified: Sat, 23 Feb 2019 22:35:01 GMTConnection: keep-aliveETag: "5c71ca95-1b26"Expires: Thu, 31 Dec 2037 23:55:55 GMTCache-Control: max-age=315360000Cache-Control: privateAccept-Ranges: bytesFollowComment"

Answer: "0Accepted AnswerS3 not send the 'Access-Control-Allow-Origin' header if the wildcard * like there:<AllowedOrigin>*</AllowedOrigin>Force s3 to send AllowedOrigin header and your content be loaded from any site, try this:<AllowedOrigin>http://</AllowedOrigin><AllowedOrigin>https://</AllowedOrigin>i think one will work ..!CommentShareharshmanvaranswered 4 years ago"


Question: Feature Request Increase DynamoDB transaction limit count from 25

"Over time AWS is doing a fantastic job of continually offering improved compute, memory and storage options and resources.  However, the current service limits in DynamoDB, in particular the limit of 25 items per transaction is I would say very low.  If you need good transactionality above this level you are limited in your options.  For that reason I am proposing an increase to something in the realm of 100.FollowComment"

Answer: "3Accepted AnswerHi,Thanks for reaching out with your feature  request. There is already a feature request in place for this feature and I will add you as a +1 to it. However, please be mindful we cannot provide you with an ETA on when or if the feature will be released. Please keep an eye on our blog https://aws.amazon.com/new/ for all new release announcements.Update:DynamoDB transactions now support 100 items for both read and write transactions:https://aws.amazon.com/about-aws/whats-new/2022/09/amazon-dynamodb-supports-100-actions-per-transaction/CommentShareEXPERTLeeroy Hannigananswered a year ago0Many thanks and I fully understand in regards to the ETA.  Is the feature request internally viewable only? If it is public could you provide a link so I can track and so others can +1.CommentShareTomTwoanswered a year agoLeeroy Hannigan EXPERTa year agoYes, the feature request is only viewable internally I'm afraid. Rest assured I have added your voice.Share"


Question: Not able to get the data in query result in the Athena for the AWS config from S3 bucket

"Hi,I have been trying to implement a monitoring solution to monitor the resources for AWS accounts in the organizations with AWS config, AWS Athena and Quicksight. I have set up all the services however Athena is not able to query all the data from the S3 bucket where the config data for all the accounts are stored. It is able to only query the data for the current account from where I am running the query. I can see the config data for all accounts in the S3 bucket as well.Athena table creation queryCREATE EXTERNAL TABLE aws_config_configuration_snapshot (    fileversion STRING,    configSnapshotId STRING,    configurationitems ARRAY < STRUCT <        configurationItemVersion : STRING,        configurationItemCaptureTime : STRING,        configurationStateId : BIGINT,        awsAccountId : STRING,        configurationItemStatus : STRING,        resourceType : STRING,        resourceId : STRING,        resourceName : STRING,        ARN : STRING,        awsRegion : STRING,        availabilityZone : STRING,        configurationStateMd5Hash : STRING,        configuration : STRING,        supplementaryConfiguration : MAP < STRING, STRING >,        tags: MAP < STRING, STRING >,        resourceCreationTime : STRING    > >) PARTITIONED BY (accountid STRING, dt STRING, region STRING) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'LOCATION 's3://<S3_BUCKET_NAME>/AWSLogs/';The lambda function used for data partitioning as per accountsimport datetimeimport reimport boto3import osTABLE_NAME = 'aws_config_configuration_snapshot'DATABASE_NAME = 'sampledb'ACCOUNT_ID = None # Determined at runtimeLATEST_PARTITION_VALUE = 'latest'athena = boto3.client('athena')def lambda_handler(event, context):    global ACCOUNT_ID    object_key = event['Records'][0]['s3']['object']['key']    match = get_configuration_snapshot_object_key_match(object_key)    if match is None:        print('Ignoring event for non-configuration snapshot object key', object_key)        return    print('Adding partitions for configuration snapshot object key', object_key)        ACCOUNT_ID = context.invoked_function_arn.split(':')[4]    object_key_parent = 's3://{bucket_name}/{object_key_parent}/'.format(        bucket_name=event['Records'][0]['s3']['bucket']['name'],        object_key_parent=os.path.dirname(object_key))    configuration_snapshot_accountid = get_configuration_snapshot_accountid(match)    configuration_snapshot_region = get_configuration_snapshot_region(match)    configuration_snapshot_date = get_configuration_snapshot_date(match)        drop_partition(configuration_snapshot_accountid, configuration_snapshot_region, LATEST_PARTITION_VALUE)    add_partition(configuration_snapshot_accountid, configuration_snapshot_region, LATEST_PARTITION_VALUE, object_key_parent)    add_partition(configuration_snapshot_accountid, configuration_snapshot_region, get_configuration_snapshot_date(match).strftime('%Y-%m-%d'), object_key_parent)    def get_configuration_snapshot_object_key_match(object_key):    # Matches object keys like AWSLogs/123456789012/Config/us-east-1/2018/4/11/ConfigSnapshot/123456789012_Config_us-east-1_ConfigSnapshot_20180411T054711Z_a970aeff-cb3d-4c4e-806b-88fa14702hdb.json.gz    return re.match('^AWSLogs/(\d+)/Config/([\w-]+)/(\d+)/(\d+)/(\d+)/ConfigSnapshot/[^\\\]+$', object_key)def get_configuration_snapshot_accountid(match):    print('AccountId:', match.group(1))    return match.group(1)def get_configuration_snapshot_region(match):    return match.group(2)def get_configuration_snapshot_date(match):    return datetime.date(int(match.group(3)), int(match.group(4)), int(match.group(5)))    def add_partition(accountid_partition_value, region_partition_value, dt_partition_value, partition_location):    execute_query('ALTER TABLE {table_name} ADD PARTITION {partition} location \'{partition_location}\''.format(        table_name=TABLE_NAME,        partition=build_partition_string(accountid_partition_value, region_partition_value, dt_partition_value),        partition_location=partition_location))        def drop_partition(accountid_partition_value, region_partition_value, dt_partition_value):    execute_query('ALTER TABLE {table_name} DROP PARTITION {partition}'.format(        table_name=TABLE_NAME,        partition=build_partition_string(accountid_partition_value, region_partition_value, dt_partition_value)))        def build_partition_string(accountid_partition_value, region_partition_value, dt_partition_value):    return "(accountid='{accountid_partition_value}', dt='{dt_partition_value}', region='{region_partition_value}')".format(    accountid_partition_value=accountid_partition_value,        dt_partition_value=dt_partition_value,        region_partition_value=region_partition_value)def execute_query(query):    print('Executing query:', query)    query_output_location = 's3://aws-athena-query-results-{account_id}-{region}'.format(        account_id=ACCOUNT_ID,        region=os.environ['AWS_REGION'])    start_query_response = athena.start_query_execution(        QueryString=query,        QueryExecutionContext={            'Database': DATABASE_NAME        },        ResultConfiguration={            'OutputLocation': query_output_location,        }    )    print('Query started')        is_query_running = True    while is_query_running:        get_query_execution_response = athena.get_query_execution(            QueryExecutionId=start_query_response['QueryExecutionId']        )        query_state = get_query_execution_response['QueryExecution']['Status']['State']        is_query_running = query_state in ('RUNNING','QUEUED')                if not is_query_running and query_state != 'SUCCEEDED':            raise Exception('Query failed')    print('Query completed')sample query tried:CREATE OR REPLACE VIEW v_config_ec2_vpcs ASSELECT DISTINCT"accountId" "633328536665", "region" "us-east-1", "configurationItem"."resourceid" "ResourceId", "configurationItem"."tags"['name'] "TagName", "json_extract_scalar"("configurationItem"."configuration", '$.isdefault') "IsDefault", "json_extract_scalar"("configurationItem"."configuration", '$.cidrblockassociationset[0].cidrblock') "CidrBlock0", "json_extract_scalar"("configurationItem"."configuration", '$.cidrblockassociationset[1].cidrblock') "CidrBlock1", "json_extract_scalar"("configurationItem"."configuration", '$.cidrblockassociationset[2].cidrblock') "CidrBlock2", "json_extract_scalar"("configurationItem"."configuration", '$.cidrblockassociationset[3].cidrblock') "CidrBlock3", "json_extract_scalar"("configurationItem"."configuration", '$.cidrblockassociationset[4].cidrblock') "CidrBlock4"FROM default.aws_config_configuration_snapshotCROSS JOIN UNNEST("configurationitems") t (configurationItem)WHERE (("dt" = 'latest') AND ("configurationItem"."resourcetype" = 'AWS::EC2::VPC'))It is not able to get the data from the S3 bucket for all the AWS account for some reason(only the data for the current account the data is queried.). I have checked the s3 bucket policy and it is set up as per the given below solution.Solution referred:https://aws.amazon.com/blogs/mt/visualizing-aws-config-data-using-amazon-athena-and-amazon-quicksight/https://aws.amazon.com/blogs/mt/how-to-query-your-aws-resource-configuration-states-using-aws-config-and-amazon-athena/Thanks and Regards,Mahesh B.FollowComment"


Question: QuickSight Migration to another AWS Account

"Hello,At the state of the art what is the simplest and fastest way to migrate QuickSight Datasets, Analysis and Dashboards to another AWS Account?Thanks for any contributions.Zee9 | AlessandroFollowComment"

Answer: "0The range of possible data sources for QuickSights is very wide: see https://docs.aws.amazon.com/quicksight/latest/user/supported-data-sources.htmlSo, can you detail which ones you need to migrate?CommentShareDidier_AWSanswered 2 months ago0Hi,Thanks for your reply.I'm not talking about data sources, I need to migrate existing QuickSight datasets, analyses and dashboards to another AWS account in the fastest way. I know you can use AWS CLI, prepare Json file etc, but it's not an easy way.CommentShareZee9 Ltd - Alessandroanswered 2 months ago"


Question: IP that resolves to other IP

"Hello,I hope you can help me or point me in the right direction.We have several IoT devices on the field which comunicate via LTE to a given server IP, changing the server IP on the devices comes with the risk of losing remote access to the IoT devices, so we rather keep this IP fixed, but at the same time we want to have the ability to change which server gets this communication. This looks like a Domain DNS, but the IoT devices only accept IPs.So... I need to get a fixed IP address  that resolves to another IP address (EC2 instance) which I can change on the fly.Thanks guysFollowComment"

Answer: 2It sounds like you want an Elastic IP Address.This is a fixed IPv4 address that you can reassign from one EC2 instance to another. This address stays persistent through instance stops and starts.https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.htmlCommentSharetedtrentanswered 6 months agoEXPERTBrettski-AWSreviewed 6 months ago0Depending on your use case and how many region you IOT devices are connecting from AWS Global Accelerator Might also be able to reduce you latencyCommentSharekaveerhanswered 6 months ago


Question: I can't commit changes to DynamoDB

"Using NoSQL Borkbench. I create a new data model with two tables, and then I commit to Amazon DynamoDB. It worked.But oops, I left a field out in one table, so I add it, and then go and hit the Commit to Amazon DynamoDB button again. Now I get this error:ErrorCould not commit to DynamoDB as table creation failed: ResourceInUseException: Table already exists: Devices.Ok, so the table already exists, but shouldn’t I be able to make changes to it?So let’s skip that datamodel change. Let me just start adding some data to those tables (using NoSQL Workbench). Maybe it won’t let me change the model once it is created, but I must surely be able to add data to the DynamoDB, no? No. Same error when I try to commit.Can someone tell me where my misunderstanding is? I am assuming that NoSQL Workbench has created a database in memory, that it then should be able to send to the cloud.FollowComment"

Answer: "1After three months I would have expected Amazon to have addressed this issue.  As you said it's a KNOWN issue.  Come on guys, appending data is vital to any tool.CommentShareDWPalmeanswered a year ago0Accepted AnswerThank you for using NoSQL Workbench. Currently we don’t support this functionality. Its a known issue and also have added in our backlog and will do our best to address it . Please stay tuned for more updates.quick fix: you can delete the entire table and commit the updated table.Thanks,Amazon DynamoDB TeamCommentShareAWS-User-4973242answered a year agomyAWSAdventure a year agoThank you. Good to know it isn't me. I will follow your advice.ShareDWPalme a year agoDeleting the entire table and committing the update burns a lot of resources that we have to pay for.  That is a useless answer.Share"


Question: Document DB $muliply query not working

"We have created a documentDb instance having a 4.0.0 version. When running the query using $muliply it throws an error as Bad Query.**Query:  'test': { $multiply: ["$price", 8 ] }**When checking the documentation it says the $muliply is supported. Not sure why it throws a Bad QueryFollowComment"

Answer: "0What is the error message you are getting? From the syntax of what you provided, it looks like you are missing brackets around your multiply aggregation expressions.'test': { $multiply: "$price", 8 }...should be...'test': { $multiply: ["$price", 8]}See more here: https://docs.mongodb.com/manual/reference/operator/aggregation/multiply/CommentShareCody_Aanswered a year ago"


Question: Cluster stopped publishing metrics

"Hi,my cluster has 3 broker and Prometheus monitoring is enabled. Everyhing was fine and this afternoon 2 of 3 brokers stopped publishing metrics.curl -s b-1.XXX.kafka.eu-central-1.amazonaws.com:11001/metrics | grep kafka_consumer_group_ConsumerLagMetrics_ValueAny ideas why this metric for example is missing on 2 brokers?FollowComment"

Answer: "0This sounds like a temporary problem specific to these brokers, that may occur during maintenance. The best way to address issues like this is usually though a support ticket where we can access and discuss more specific details of you cluster. Please do raise a ticket for us with the details of your cluster, and we will take a look.CommentShareAWS-User-7376622answered 2 years ago0After 3 day the metrics are back.CommentSharecmoetzinganswered 2 years ago0It is worth noting that if you have very high partition counts, or are using smaller broker instance types, especially the t3 nodes which are intended primarily to development use, then there can be resource constraints that affect the publication of metrics like this. It is worth raising a support ticket to look into the specific details of the case, but the most common cause of these events is brokers that are small for their workload.CommentShareAWS-User-7376622answered 2 years ago"


Question: How to filter routers on a TG for Private IP VPN

"I am implementing a new feature from AWS called Private IP VPN using Direct connect, My question is that how do i filter routes entering my P2P IPSec tunnel from the transit gateway towards onprem as i would like to receive all the routes that exist in the TG.FollowComment"

Answer: "1You can use 'Allowed prefix' feature on DirectConnectGateway for the routes that get advertised from AWS -> On-premises, more information on this topic at the below link:https://docs.aws.amazon.com/directconnect/latest/UserGuide/allowed-to-prefixes.htmlAlso check out this blogCommentShareEXPERTTushar_Janswered 9 months ago"


Question: BasicStation AWS IoTCore disconnect after "Repeated excesive clock drift"

"Hello,I’m using a RAK2287 with a RPi4 along with BasicStation to create a gateway device to deliver sensor data through to AWS IoTCore with CUPS authentication.After starting my gateway it seems as if things are going well, but after a few minutes I get a number of ‘time sync rejected’ messages and sometimes some Repeated excessive clock drifts between MCU/SX130X#0 messages before the station disconnects and restarts.I'm not sure what's causing this: whether it's on the AWS side, or the gateway side; triggered by AWS or triggered by the gateway; if it relates to time sync or if it's just coincidental.Some logs below..  2021-10-08 12:55:48.920 \[SYS:INFO] Logging     : stderr (maxsize=10000000, rotate=3)  2021-10-08 12:55:48.920 \[SYS:INFO] Station Ver : 2.0.5(corecell/std) 2021-10-05 15:02:16  2021-10-08 12:55:48.921 \[SYS:INFO] Package Ver : (null)  2021-10-08 12:55:48.921 \[SYS:INFO] proto EUI   : 0:dca6:3205:f1a        (/sys/class/net/eth0/address)  2021-10-08 12:55:48.921 \[SYS:INFO] prefix EUI  : ::1    (builtin)  2021-10-08 12:55:48.921 \[SYS:INFO] Station EUI : dca6:32ff:fe05:f1a  2021-10-08 12:55:48.921 \[SYS:INFO] Station home: ./     (builtin)  2021-10-08 12:55:48.921 \[SYS:INFO] Station temp: /var/tmp/      (builtin)  2021-10-08 12:55:49.123 \[TCE:INFO] Starting TC engine  2021-10-08 12:55:49.124 \[any:INFO] ./tc.trust:  cert. version     : 3  serial number     : A7:0E:4A:4C:34:82:B7:7F  issuer name       : C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority  subject name      : C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2  issued  on        : 2009-09-02 00:00:00  expires on        : 2034-06-28 17:39:16  signed using      : RSA with SHA-256  RSA key size      : 2048 bits  basic con2021-10-08 12:55:49.132 \[any:INFO] ./tc.crt:  cert. version     : 3  serial number     : 70:B9:E0:68:0E:62:85:AC:A4:7F:0D:0C:81:3D:6D:06:D7:95:C0:AE  issuer name       : OU=Amazon Web Services O=Amazon.com Inc. L=Seattle ST=Washington C=US  subject name      : CN=AWS IoT Certificate  issued  on        : 2021-10-05 13:52:18  expires on        : 2049-12-31 23:59:59  signed using      : RSA with SHA-256  RSA key size      : 2048 bits  basic constraints : CA=false  key usage         : Digital Signature  2021-10-08 12:55:49.132 \[AIO:INFO]  2021-10-08 12:55:51.807 \[TCE:INFO] Connecting to INFOS: wss://A3OQ74V8656RDU.lns.lorawan.us-west-2.amazonaws.com:443  2021-10-08 12:55:51.807 \[CUP:INFO] Starting a CUPS session in 0 seconds.  2021-10-08 12:55:51.807 \[CUP:INFO] Starting a CUPS session now.  2021-10-08 12:55:51.807 \[CUP:INFO] Connecting to CUPS ... https://A3OQ74V8656RDU.cups.lorawan.us-west-2.amazonaws.com:443 (try #1)  2021-10-08 12:55:51.808 \[any:INFO] ./cups.trust:  cert. version     : 3  serial number     : A7:0E:4A:4C:34:82:B7:7F  issuer name       : C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority  subject name      : C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2  issued  on        : 2009-09-02 00:00:00  expires on        : 2034-06-28 17:39:16  signed using      : RSA with SHA-256  RSA key size      : 2048 bits  basic c2021-10-08 12:55:51.816 \[any:INFO] ./cups.crt:  cert. version     : 3  serial number     : 2F:B2:92:89:7F:3C:5C:D7:89:CB:36:E6:5A:1B:82:93:A7:9D:AF:B6  issuer name       : OU=Amazon Web Services O=Amazon.com Inc. L=Seattle ST=Washington C=US  subject name      : CN=AWS IoT Certificate  issued  on        : 2021-09-27 10:25:44  expires on        : 2049-12-31 23:59:59  signed using      : RSA with SHA-256  RSA key size      : 2048 bits  basic constraints : CA=false  key usage         : Digital Signature  2021-10-08 12:55:51.816 \[AIO:INFO]  2021-10-08 12:55:51.928 \[CUP:VERB] Retrieving update-info from CUPS https://A3OQ74V8656RDU.cups.lorawan.us-west-2.amazonaws.com:443...  2021-10-08 12:55:53.746 \[AIO:DEBU] \[4] HTTP connection shutdown...  2021-10-08 12:55:53.746 \[CUP:INFO] Interaction with CUPS done (no updates) - next regular check in 1d  2021-10-08 12:55:53.832 \[TCE:INFO] Infos: dca6:32ff:fe05:0f1a dca6:32ff:fe05:0f1a wss://A3OQ74V8656RDU.lns.lorawan.us-west-2.amazonaws.com:443/gateway/dca632fffe050f1a  2021-10-08 12:55:53.832 \[AIO:DEBU] \[3] ws_close reason=1000  2021-10-08 12:55:53.832 \[AIO:DEBU] \[3|WS] Server sent close: reason=1006  2021-10-08 12:55:53.832 \[AIO:DEBU] \[3] WS connection shutdown...  2021-10-08 12:55:53.832 \[any:INFO] ./tc.trust:  cert. version     : 3  serial number     : A7:0E:4A:4C:34:82:B7:7F  issuer name       : C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority  subject name      : C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2  issued  on        : 2009-09-02 00:00:00  expires on        : 2034-06-28 17:39:16  signed using      : RSA with SHA-256  RSA key size      : 2048 bits  basic con2021-10-08 12:55:53.835 \[any:INFO] ./tc.crt:  cert. version     : 3  serial number     : 70:B9:E0:68:0E:62:85:AC:A4:7F:0D:0C:81:3D:6D:06:D7:95:C0:AE  issuer name       : OU=Amazon Web Services O=Amazon.com Inc. L=Seattle ST=Washington C=US  subject name      : CN=AWS IoT Certificate  issued  on        : 2021-10-05 13:52:18  expires on        : 2049-12-31 23:59:59  signed using      : RSA with SHA-256  RSA key size      : 2048 bits  basic constraints : CA=false  key usage         : Digital Signature  2021-10-08 12:55:53.835 \[AIO:INFO]  2021-10-08 12:55:53.878 \[TCE:VERB] Connecting to MUXS...  2021-10-08 12:55:54.964 \[TCE:VERB] Connected to MUXS.  2021-10-08 12:55:55.169 \[S2E:WARN] Unknown field in router_config - ignored: protocol (0xFD309030)  2021-10-08 12:55:55.169 \[S2E:WARN] Unknown field in router_config - ignored: regionid (0xE6FFB211)  2021-10-08 12:55:55.171 \[SYS:VERB] rinit.sh: Forked, waiting...  CoreCell reset through GPIO17...  CoreCell power enable through GPIO18...  2021-10-08 12:55:56.140 \[SYS:INFO] Process rinit.sh (pid=886) completed  ERROR: Failed to stop TX trigger  ERROR: Failed to stop TX trigger  ERROR: failed to close I2C device (err=-1)  2021-10-08 12:55:56.140 \[RAL:INFO] Lora gateway library version: Version: 1.0.5;  2021-10-08 12:55:56.146 \[RAL:VERB] Connecting to device: /dev/spidev0.0  2021-10-08 12:55:56.146 \[RAL:DEBU] SX130x txlut table (16 entries)  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut  0:  rf_power=12 pa_gain=0 pwr_idx=15  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut  1:  rf_power=13 pa_gain=0 pwr_idx=16  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut  2:  rf_power=14 pa_gain=0 pwr_idx=17  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut  3:  rf_power=15 pa_gain=0 pwr_idx=19  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut  4:  rf_power=16 pa_gain=0 pwr_idx=20  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut  5:  rf_power=17 pa_gain=0 pwr_idx=22  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut  6:  rf_power=18 pa_gain=1 pwr_idx=1  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut  7:  rf_power=19 pa_gain=1 pwr_idx=2  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut  8:  rf_power=20 pa_gain=1 pwr_idx=3  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut  9:  rf_power=21 pa_gain=1 pwr_idx=4  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut 10:  rf_power=22 pa_gain=1 pwr_idx=5  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut 11:  rf_power=23 pa_gain=1 pwr_idx=6  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut 12:  rf_power=24 pa_gain=1 pwr_idx=7  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut 13:  rf_power=25 pa_gain=1 pwr_idx=9  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut 14:  rf_power=26 pa_gain=1 pwr_idx=11  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 txlut 15:  rf_power=27 pa_gain=1 pwr_idx=14  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 rssi_tcomp: coeff_a=0.000 coeff_b=0.000 coeff_c=20.410 coeff_d=2162.560 coeff_e=0.000  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 rxrfchain 0: enable=1 freq=867.5MHz rssi_offset=-215.399994 type=5 tx_enable=1  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 rxrfchain 1: enable=1 freq=868.5MHz rssi_offset=-215.399994 type=5 tx_enable=0  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 ifchain  0: enable=1 rf_chain=0 freq=-400000 bw=0 SF=0 sync_word=0/0  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 ifchain  1: enable=1 rf_chain=0 freq=-200000 bw=0 SF=0 sync_word=0/0  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 ifchain  2: enable=1 rf_chain=0 freq=0 bw=0 SF=0 sync_word=0/0  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 ifchain  3: enable=1 rf_chain=0 freq=200000 bw=0 SF=0 sync_word=0/0  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 ifchain  4: enable=1 rf_chain=0 freq=400000 bw=0 SF=0 sync_word=0/0  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 ifchain  5: enable=1 rf_chain=1 freq=-400000 bw=0 SF=0 sync_word=0/0  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 ifchain  6: enable=1 rf_chain=1 freq=-200000 bw=0 SF=0 sync_word=0/0  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 ifchain  7: enable=1 rf_chain=1 freq=0 bw=0 SF=0 sync_word=0/0  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 ifchain  8: enable=1 rf_chain=1 freq=-200000 bw=5 SF=7 sync_word=0/0 \[STD] Explicit header  2021-10-08 12:55:56.146 \[RAL:VERB] SX1302 ifchain  9: enable=1 rf_chain=0 freq=0 bw=0 SF=0 sync_word=0/0  2021-10-08 12:55:56.146 \[RAL:VERB] Station device: /dev/spidev0.0 (PPS capture disabled)  2021-10-08 12:55:58.575 \[RAL:VERB] Concentrator started (2s428ms)  2021-10-08 12:55:58.575 \[S2E:INFO] Configuring for region: EU868 -- 863.0MHz..870.0MHz  2021-10-08 12:55:58.575 \[S2E:VERB]   DR0  SF12/BW125  2021-10-08 12:55:58.575 \[S2E:VERB]   DR1  SF11/BW125  2021-10-08 12:55:58.575 \[S2E:VERB]   DR2  SF10/BW125  2021-10-08 12:55:58.575 \[S2E:VERB]   DR3  SF9/BW125  2021-10-08 12:55:58.575 \[S2E:VERB]   DR4  SF8/BW125  2021-10-08 12:55:58.575 \[S2E:VERB]   DR5  SF7/BW125  2021-10-08 12:55:58.575 \[S2E:VERB]   DR6  SF7/BW250  2021-10-08 12:55:58.575 \[S2E:VERB]   DR7  FSK  2021-10-08 12:55:58.575 \[S2E:VERB]   DR8  undefined  2021-10-08 12:55:58.575 \[S2E:VERB]   DR9  undefined  2021-10-08 12:55:58.575 \[S2E:VERB]   DR10 undefined  2021-10-08 12:55:58.575 \[S2E:VERB]   DR11 undefined  2021-10-08 12:55:58.575 \[S2E:VERB]   DR12 undefined  2021-10-08 12:55:58.575 \[S2E:VERB]   DR13 undefined  2021-10-08 12:55:58.575 \[S2E:VERB]   DR14 undefined  2021-10-08 12:55:58.575 \[S2E:VERB]   DR15 undefined  2021-10-08 12:55:58.575 \[S2E:VERB]   TX power: 0.0 dBm EIRP  2021-10-08 12:55:58.575 \[S2E:VERB]   JoinEUI list: 0 entries  2021-10-08 12:55:58.575 \[S2E:VERB]   NetID filter: FFFFFFFF-FFFFFFFF-FFFFFFFF-FFFFFFFF  2021-10-08 12:55:58.575 \[S2E:VERB]   Dev/test settings: nocca=0 nodc=0 nodwell=0  2021-10-08 12:56:25.206 \[S2E:VERB] RX 867.5MHz DR0 SF12/BW125 snr=-19.2 rssi=-114 xtime=0xB0000001A2303B - updf mhdr=40 DevAddr=0100DA30 FCtrl=D8 FCnt=21758 FOpts=\[BD0069F0009E437B] 8E8D31E6..0626 mic=1125777498 (52 bytes)  2021-10-08 12:56:40.586 \[SYN:INFO] MCU/SX130X drift stats: min: _27.1ppm  q50: _30.9ppm  q80: _33.3ppm  max: _34.3ppm - threshold q90: +34.3ppm  2021-10-08 12:56:40.586 \[SYN:INFO] Mean MCU drift vs SX130X#0: 30.8ppm  2021-10-08 12:56:59.489 \[SYN:INFO] Time sync qualities: min=185 q90=213 max=213 (previous q90=2147483647)  2021-10-08 12:57:22.599 \[SYN:VERB] Time sync rejected: quality=6398 threshold=213  2021-10-08 12:57:24.699 \[SYN:INFO] MCU/SX130X drift stats: min: _22.4ppm  q50: _26.2ppm  q80: _29.0ppm  max: _30.5ppm - threshold q90: +30.4ppm  2021-10-08 12:57:24.699 \[SYN:INFO] Mean MCU drift vs SX130X#0: 26.3ppm  2021-10-08 12:57:45.705 \[SYN:VERB] Time sync rejected: quality=3486 threshold=213  2021-10-08 12:58:02.508 \[SYN:INFO] Time sync qualities: min=188 q90=213 max=6398 (previous q90=213)  2021-10-08 12:58:08.811 \[SYN:INFO] MCU/SX130X drift stats: min: _16.2ppm  q50: _22.4ppm  q80: _23.8ppm  max: _26.2ppm - threshold q90: +26.2ppm  2021-10-08 12:58:08.811 \[SYN:INFO] Mean MCU drift vs SX130X#0: 22.1ppm  2021-10-08 12:58:40.320 \[SYN:VERB] Time sync rejected: quality=6319 threshold=213  2021-10-08 12:58:52.922 \[SYN:INFO] MCU/SX130X drift stats: min: _12.4ppm  q50: _19.0ppm  q80: _20.5ppm  max: _22.4ppm - threshold q90: +21.4ppm  2021-10-08 12:58:52.922 \[SYN:INFO] Mean MCU drift vs SX130X#0: 18.7ppm  2021-10-08 12:59:05.523 \[SYN:INFO] Time sync qualities: min=185 q90=212 max=6319 (previous q90=213)  2021-10-08 12:59:20.229 \[SYN:VERB] Time sync rejected: quality=3559 threshold=212  2021-10-08 12:59:25.586 \[AIO:DEBU] \[3] Connection closed unexpectedly  2021-10-08 12:59:25.586 \[AIO:DEBU] \[3] WS connection shutdown...  2021-10-08 12:59:25.587 \[TCE:VERB] Connection to MUXS closed in state 4  2021-10-08 12:59:25.587 \[TCE:INFO] MUXS reconnect backoff 1s (retry 0)  FollowComment"

Answer: "0Turns out I just had to add a "pps": true line to the station.conf file.CommentShareCraigEanswered 2 years ago0Glad that the issue has been solved. The time sync issue does exist. But it should not disconnect the gateway. We will be working to fix this time sync issue very soon.CommentShareAWS-User-Russanswered 2 years ago0Hi Russ,I've had to revisit this problem since we're seeing a repeated loss of connection to AWS when using a USB cellular modem which emulated as an eth interface.Every ~3 minutes, we get this error:2021-10-25 12:29:08.820 \\[AIO:DEBU] \[3] Connection closed unexpectedly  2021-10-25 12:29:08.820 \\[AIO:DEBU] \[3] WS connection shutdown...  2021-10-25 12:29:08.821 \\[TCE:VERB] Connection to MUXS closed in state 4  I'm not sure if this is a problem on the AWS side, maybe something we have to configure, or if this is a problem with using a cellular modem. Do you have any insight here?Thanks,CraigEdited by: CraigE on Oct 25, 2021 6:09 AMCommentShareCraigEanswered 2 years ago0Hi Craig,Is that all in the gateway's log? How do we know why the gateway is disconnecting so frequently?Does this happen only recently or always?If it relates to the AWS side, any error message from the gateway would be helpful, such as what caused the disconnect, etc.But it looks like all the handshakes with AWS have passed. If the gateway initiates all the disconnections, checking with the gateway vendor should be the next step to go.Thanks,RussCommentShareAWS-User-Russanswered 2 years ago0Hi Russ,This happens reliably when we are on cellular connection, with multiple cellular modules. The gateway appears to only disconnect if there has not been any LoRaWAN packets come through for about 3 minutes. If we have regular LoRaWAN packets on the gateway this seems to fend off the disconnect.This is the only log information that shows at the time of disconnect that relates to the web socket. We get time sync messages, but we also get these on wifi and do not disconnect:2021-10-25 12:31:33.493 \[SYN:INFO] Mean MCU drift vs SX130X#0: 7.8ppm2021-10-25 12:31:39.796 \[SYN:INFO] Time sync qualities: min=237 q90=261 max=261 (previous q90=261)2021-10-25 12:31:49.253 \[SYN:VERB] Time sync rejected: quality=262 threshold=2612021-10-25 12:31:51.354 \[SYN:ERRO] No PPS pulse for ~84 secs2021-10-25 12:31:53.456 \[SYN:VERB] Time sync rejected: quality=284 threshold=2612021-10-25 12:32:10.266 \[SYN:VERB] Time sync rejected: quality=284 threshold=2612021-10-25 12:32:20.772 \[SYN:INFO] MCU/SX130X drift stats: min: _1.0ppm  q50: _6.7ppm  q80: _10.5ppm  max: _12.8ppm - threshold q90: +11.9ppm2021-10-25 12:32:20.772 \[SYN:INFO] Mean MCU drift vs SX130X#0: 7.2ppm2021-10-25 12:32:26.069 \[AIO:DEBU] [3] Connection closed unexpectedly2021-10-25 12:32:26.069 \[AIO:DEBU] [3] WS connection shutdown...2021-10-25 12:32:26.069 \[TCE:VERB] Connection to MUXS closed in state 42021-10-25 12:32:26.069 \[TCE:INFO] MUXS reconnect backoff 16s (retry 4)2021-10-25 12:32:41.784 \[SYN:INFO] Time sync qualities: min=236 q90=262 max=284 (previous q90=261)2021-10-25 12:32:42.070 \[any:INFO] ./tc.trust:We weren't sure if there was something on the AWS side that may be kicking us off the web socket. I'm aware that basicstation has a keepalive TCP packet but despite me changing the regularity of these it doesn't seem to have an effect.Edited by: CraigE on Oct 26, 2021 3:08 AMCommentShareCraigEanswered 2 years ago0Hi Craig,Thanks for the input. I will try to look into any possible cause for the disconnections. If you have a way to check with the gateway vendor, they probably can provide further investigation that lead to the potential final fix. I will get back to you after some investigation.thanks,RussCommentShareAWS-User-Russanswered 2 years ago0Hi Craig,I've looked into the backend logs. I do not see any cloud-initiated disconnect logs. It will be very likely the disconnection is triggered by the gateway's firmware. One noticeable thing I see in the logs is that there were very frequent (every 20~30 seconds) TLS handshake requests coming from the gateways. AWS side showed no error.Can you check more on that perspective?Thanks,CommentShareAWS-User-Russanswered 2 years ago"


Question: How to perform scroll / scroll not working on Amazon Elasticsearch

"We used this guide: https://opensearch.org/docs/latest/opensearch/rest-api/scroll/, and the amazon elasticsearch docs: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-operations.html to try and find out how to perform scroll search on amazon elasticsearch. We recently switched from elasticsearch on one of our local servers to amazon elasticsearch. We came to realize that scroll performs differently. See below a snippet on how we now try to perform scroll (in python3):params = {"size": 5000, "q": query, "scroll": "5m"}resp = requests.get(ELASTICSEARCH_URL + "/_search", params=params, auth=auth)resp.status code:200scroll_id = resp.json().get("_scroll_id")In [23]: resp = requests.get(ELASTICSEARCH_URL + "_search/scroll", params={"scro...: ll": "5m", "scroll_id": scroll_id}, auth=auth)In [24]: respOut[24]: <Response [400]>In [25]: resp._contentOut[25]: b'{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/users/_search/scroll] contains unrecognized parameters: [scroll], [scroll_id]"}],"type":"illegal_argument_exception","reason":"request [/users/_search/scroll] contains unrecognized parameters: [scroll], [scroll_id]"},"status":400}'We've tried about all the variations, remove "/"'s or tweaking params, but getting this same error.Thank you in advance.FollowComment"

Answer: "0Syntax for scroll request:GET _search/scroll/<scroll-id>https://opensearch.org/docs/latest/opensearch/rest-api/scroll/When running the scroll request for the first time, we use the index name. Below is an example:GET shakespeare/_search?scroll=10m{"size": 10000}This will return scroll id:"_scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAUWdmpUZDhnRFBUcWFtV21nMmFwUGJEQQ=="From the second time use only  _search/scroll/<scrollid>. we don't pass index name the second timesample:  GET _search/scroll{"scroll": "10m","scroll_id": "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAUWdmpUZDhnRFBUcWFtV21nMmFwUGJEQQ=="}Note: Once the scroll id expires, scroll request will start failing. Please consider starting a new scroll.params = {"size": 5000, "q": query, "scroll": "5m"} resp = requests.get(ELASTICSEARCH_URL + "/_search", params=params, auth=auth) resp.status code: 200 scroll_id = resp.json().get("_scroll_id")Above Query works fine on versions 7.10, 7.9, 7.7 and 7.4. Since python says illegal argument exception, version of ES/OS is required to replicate the issue using the same python client.CommentShareSUPPORT ENGINEERVarun_Sanswered a year ago"


Question: Can you run Microsoft Intune on Amazon WorkSpaces?

Is it possible to run Microsoft Intune on Amazon WorkSpaces?FollowComment

Answer: "0Accepted AnswerMicrosoft Intune is used to manage Microsoft client operating systems, and some selected Android devices. Standard Amazon Windows WorkSpaces use Windows Server with desktop experience, which isn't an operating system supported by Microsoft Intune.However, if your licensing agreement with Microsoft allows it, you can use your Windows 10 Enterprise or Windows 10 Pro desktop licenses for your WorkSpaces and then use Intune to manage those WorkSpaces.For more information, see Bring your own Windows desktop licenses in the Amazon WorkSpaces administration guide.CommentShareMODERATORAWS-User-7744098answered 3 years ago"


Question: Python deployment failing on CodeDeploy

"Hello..I have an issue with my CodeDeploy that Im using to deploy a Python application.. Im using Django which is stored in CodeCommit and Im deploying to EC2 instances. It is failing and Im getting this error when trying to deploy.."Script at specified location: scripts/install_dependencies.sh run as user root failed with exit code 5"Note that when I try to run the script manually (without CodeDeploy), it works without issues.I would appreciate if you guide me on how to get the following:-- Clear the error and get the deployment working.-- Make the application run as a service so I can execute stop_service.sh script when performing in-place software upgrade.Thanks!!FollowComment"

Answer: "1Accepted AnswerHi Ivan, based on the information you provided, the appspec.yml' looks fine - we can still improve it a bit as shown in this thread - but it's unclear how you're creating the application service definition as I mentioned in my previous reply because the error indicates that the service definition doesn't exist specifically the file 'app.service' in the directory '/lib/systemd/system'.Can you follow the below steps and let me know the outcome?Create a file in your code repo in the 'scripts' folder with the following content and name it 'app.service':[Unit]Description=web_appAfter=network.target[Service]Type=idleRestart=on-failureUser=rootWorkingDirectory=/home/app/ExecStart=/bin/bash -c 'python3 /home/app/webapp.py runserver 0.0.0.0:8080'[Install]WantedBy=multi-user.targetI assumed your application entry point is the file 'webapp.py', this is just an example and you can substitute the actual one you have. I also assumed the TCP port that you'll bind to the application is 8080, you can change it, but remember allowing that port in the instance's security group or ACL.Modify your 'install_dependencies.sh' to look like this:#!/bin/bashpip3 install -r requirements.txtcp /home/app/scripts/app.service /lib/systemd/systemCreate a script in the folder 'scripts' and name if 'start_service.sh' and paste these commands:#!/bin/bashsystemctl daemon-reloadsystemctl enable app.servicesystemctl start app.serviceModify your 'appspec.yml' to look like this:version: 0.0os: linuxfiles:  - source: /    destination: /home/app/   overwrite: truefile_exists_behavior: OVERWRITEhooks:  AfterInstall:    - location: scripts/install_dependencies.sh      timeout: 180      runas: root    ApplicationStart:    - location: scripts/start_service.sh      timeout: 120      runas: rootLet me know how it goes.CommentSharemmlanswered 5 months agoIvan Hristovski 5 months agoI missed creating the service definition file (app.service). Followed your instruction and it worked like a charm!!! Many thanks!Share0Hi, error code 5 refers to I/O error in your script.Are you able to run your application manually? Typically you would use in Django something like this to run your python app:python3 /app_code_path/app.py runserver 0.0.0.0:port_numberwhere app_code_path: represents the path of your code repo in the EC2 instance and port_number is the TCP port that you would bind to the application in the instance OS.Also can you share with me how your appspec.yml looks like and the content of 'install_dependencies.sh' script or any other scripts that you're referencing in the appspec.yml file?As for your question on how to make the application run as a service, in Linux, you would typically need to create a file in this path '/lib/systemd/system' , name it like 'test_web' and provide this sample content:[Unit]Description=sample_web_appAfter=network.target[Service]Type=idleRestart=on-failureUser=rootWorkingDirectory=/app_code_path/ExecStart=/bin/bash -c 'python3 /app_code_path/app.py runserver 0.0.0.0:8080'[Install]WantedBy=multi-user.targetNow when you want to run the app as a service, you would need to execute these commands:sudo systemctl daemon-reloadsudo systemctl enable test_websudo systemctl start test_webCommentSharemmlanswered 5 months agoIvan Hristovski 5 months agoThanks for the reply.. I tried following your instructions and modify the script a little bit but Im getting this error now:[stderr]Failed to execute operation: No such file or directory[stderr]Failed to start app.service: Unit not found.The 'appspec.yml' content is:version: 0.0os: linuxfiles:source: /destination: /home/app/hooks:AfterInstall:- location: scripts/install_dependencies.shtimeout: 180And the "install_dependencies.sh' is:#!/bin/bashpip3 install -r requirements.txtsystemctl daemon-reloadsystemctl enable app.servicesystemctl start app.serviceAm I missing anything here??Share"


Question: How to enable SSL on my LightSail instance ?

"I have a domain at GoDaddy thats pointing tons-352.awsdns-44.comns-1222.awsdns-24.orgns-1908.awsdns-46.co.ukns-887.awsdns-46.netI created a LightSail instance and it's live on http://x.x.x.x and added the A record at Route 53 - and my-subdomain.my-domain.com also works. But now, how do I get https://my-subdomain.my-domain.com to work ?https://i.imgur.com/Ou9HMik.pngFollowComment"

Answer: "0To provide HTTPS for your web site free of charge, I recommend you try to use Let's Encrypt.CommentShareLipianswered a year agoallcareerguru a year agoWhat do I have to do to get AWS Certificates working ?https://imgur.com/a/wDkZDVoShare"


Question: "I would like to build simple solution S3 - AMAZON Glue - AMAZON Redshift - Tableaue, but can't do it for different reasons..."

"how to fix this to execute Jobs from Amazon Glue to Redshift?  Error: "JobName:s3-redshift and JobRunId:jr_30d8ac0c6b35d44f641b7d5b55819365897b18e6244c8a5559bff53efc8e23c1 failed to execute with exception At least one security group must open all egress ports.To limit traffic, the source security group in your outbound rule can be restricted to the same security group (Service: AWSGlueJobExecutor; Status Code: 400; Error Code: InvalidInputException; Request ID: eafb6d33-21d0-438c-96b9-c7b7ea6162f9; Proxy: null)", I updated my security group inbound rules - but nothing happens. What I need to do in this case?FollowComment"

Answer: "2Accepted AnswerYou need to update the inbound AND outbound rules of the security group. Important: You probably do not need to open it to world (ie do NOT open it to 0.0.0.0/0), but have as a source always the security group itself (ie all compute that is assigned to the security group can talk with each other).Make sure also that the security group is assigned to a Glue connection (if you use Glue) of type network and that the connection is assigned to the Glue job.CommentShareJ Frankeanswered 18 days ago2As mentioned in the previous answer there might be additional steps that need to be done:the security group are configured on all components , refer to [this page in the documentation[(https://docs.aws.amazon.com/glue/latest/dg/setup-vpc-for-glue-access.html)check also this blog for additional configurationcheck the Redshift connection is properly setup (when you test the connection does it complete succesfully or it still fails?hope this helpsCommentShareEXPERTFabrizio@AWSanswered 16 days ago"


Question: When will java-17-amazon-corretto be patched in the Amazon Linux 2 AMI?

"The latest version of Corretto has a critical security fix for CVE-2022-21449:Corretto version: 17.0.3.6.1Release Date: April 17, 2022Yet this latest version is not available from the package repo on the very latest Amazon Linux 2 AMI. When can we expect to see the patched version?[ec2-user@ip-XXX ~]$ sudo yum install -y java-17-amazon-corretto-headlessLoaded plugins: extras_suggestions, langpacks, priorities, update-motdamzn2-core                                                                                | 3.7 kB  00:00:004 packages excluded due to repository priority protectionsPackage 1:java-17-amazon-corretto-headless-17.0.2+8-1.amzn2.1.x86_64 already installed and latest versionNothing to do[ec2-user@ip-XXX ~]$ cat /etc/image-idimage_name="amzn2-ami-hvm"image_version="2"image_arch="x86_64"image_file="amzn2-ami-hvm-2.0.20220419.0-x86_64.xfs.gpt"image_stamp="8e61-6310"image_date="20220419161702"recipe_name="amzn2 ami"recipe_id="fd8fd9c3-72c3-a01f-c3a3-db8b-8481-685a-78a2e8af"FollowComment"

Answer: "0This has been addressed by AWS, the latest version Corretto has the fix for CVE-2022-21449CommentSharepianowitchanswered a year ago"


Question: Progress of migrating C4/M4 instances to nitro hypervisor.

"In this article: https://perspectives.mvdirona.com/2021/11/xen-on-nitro-aws-nitro-for-legacy-instances/ it mentioned that nitro can support legacy instances. But I can't find an official notice of this. Can you confirm this article is correct and if so, can you tell if M4/C4 instances are now supported or a time frame. Also, when they are supported, does this mean that the bidirectional serial console will also be supported for this legacy instances?FollowComment"

Answer: 0Here is the announcement regarding Nitro support for previous generation instances: https://aws.amazon.com/about-aws/whats-new/2022/11/aws-nitro-system-supports-previous-generation-instances/CommentShareChris Pollardanswered 6 months ago


Question: "Cognito's API ResendConfirmationCode/ConfirmSignUp param input by email or phone number, instead of username attribute only"

how can i achieve it? any workaround or custom auth flow can do that? thank you!!!FollowComment

Answer: "0Hello,As you rightfully noted, the ConfirmSignUp[1] and ResendConfirmationCode[2] API operation only supports including the username at present and does not support including aliases such as the email address/phone number for the user. To be able to call the above API operations, you would need to include the actual username of the Cognito User, which would look more like a GUID/UUID instead of a name.Having noted the above, I can confirm that an existing feature request is in place with the Service Team, to add support for including the aliases, such as the email address/phone number, in the ConfirmSignUp and ResendConfirmationCode API operation instead of the username and I have added your use-case as a +1 to the request, to help with the roll-out of the feature.While I am unable to comment on if/when this feature may get released, I request you to keep an eye on our What's New[3] and Blog[4] pages for any new feature announcements.As a suggested work-around, you may consider caching the UUID/Username of the user in the browser for an email in some form of cookie, as noted in the 3rd party link here[5]. However, please note that the link provided here[5], is done so on a best-effort basis, as the work-around of caching the UUID would be dependent on the application that is integrated with the Cognito Userpool and would generally involve code development.[1] ConfirmSignUp - https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_ConfirmSignUp.html[2] ResendConfirmationCode - https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_ResendConfirmationCode.html[3] https://aws.amazon.com/new/[4] https://aws.amazon.com/blogs/aws/[5] https://github.com/amazon-archives/amazon-cognito-identity-js/issues/147#issuecomment-248402163I sincerely hope this helps!CommentShareSUPPORT ENGINEERBradley_Aanswered a month ago"