Question: Cloudwatch Input Transformer returning no data

"We have a CloudWatch event monitoring for changes in Route 53 and sends to an SNS topic for email notification. All of the Route53 API calls are being logged to an S3Bucket.When setting up the "Input Transformer" to clean up the mess of JSON it spits out we are not getting any results.The message section in the SNS email is empty. I was able to take the Full Event from CloudTrail and parse it with JQ and return the proper information.Input Transformer;{"zonename":"$.responseElements.hostedZone.name"}Input Template;"Hosted zone is <zonename>."From the email notification;{"Type" : "Notification","MessageId" : "27346a5e-7e4a-5f4a-bda9-3e831a084b7c","TopicArn" : "arn:aws:sns:us-east-1:432254711293:blaketest","Message" : ""Hosted zone is ."","Timestamp" : "2019-02-13T00:17:06.434Z","SignatureVersion" : "1","Signature" : "YgCr24cPcMNoBzV6RpjDjQi74zDHcRKIFnKkKhsIPdeUienu2W5Lww7vhSJqTF7oz2D9aQ_M53voodt1VVdOO8T2Ot44qHlDKf1MibkT4/1exZGPYBLBGfgn/cg8QtjURq0Qb6/dSjRvUWLr9xk/lUSyYvBlm1hgq9bXVGvD6toP2qCuudTzgdvZ1FJ9lAEzDi8YI_xZEnATZoa6jABMBkgca56p/Ix9NZ3gkFa4CY2WoAyr2MEAqY1RrBS/DHlVT1yQDc6WpW1mWy2bDirtNwsdVR02UY788rLoTMiTsMf1z0R8osBpNMeMd7f/Pd3+4A9R9K6W6lW1qzkooPa1PQ==","SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-ac565b8b1a6c5d002d285f959****.pem","UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:********:erictest:89551bd8-76d3-458a-a93c-8377486dde19"}From CloudTrail Event;{"eventVersion": "1.05","userIdentity": {"type": "IAMUser","principalId": "","arn": "arn:aws:iam:::user/blakes","accountId": "","accessKeyId": "*","userName": "blakes","sessionContext": {"attributes": {"mfaAuthenticated": "true","creationDate": "2019-02-12T18:23:55Z"}},"invokedBy": "signin.amazonaws.com"},"eventTime": "2019-02-13T00:16:25Z","eventSource": "route53.amazonaws.com","eventName": "CreateHostedZone","awsRegion": "us-east-1","sourceIPAddress": "****","userAgent": "signin.amazonaws.com","requestParameters": {"hostedZoneConfig": {"privateZone": false,"comment": "ERIC testing"},"callerReference": "AB354EGG09E-FC5E-046A-A6C1-8B1815375B0C","name": "testeric.com"},"responseElements": {"changeInfo": {"status": "PENDING","id": "/change/C3TJKPOO14VP6B","submittedAt": "Feb 13, 2019 12:16:25 AM"},"location": "https://route53.amazonaws.com/2013-04-01/hostedzone/Z3JK3REDKJVTQZ","hostedZone": {"resourceRecordSetCount": 2,"config": {"privateZone": false,"comment": "eric testing"},"name": "testeric.com.","id": "/hostedzone/Z3JKGRGERGDKJVTQZ","callerReference": "AB35409E-FC5EHERH-046A-A6C1-8B1815375B0C"},"delegationSet": {"nameServers": ["ns-942.awsdns-53.net","ns-121.awsdns-15.com","ns-1417.awsdns-49.org","ns-1927.awsdns-48.co.uk"]}},"requestID": "99c9fb95-2f24-11e9-a419-4fc3574ffweewqb5f52","eventID": "32bbf0ffw0e-1a95-416e-9dd5-ae2eec971b41","eventType": "AwsApiCall","apiVersion": "2013-04-01","recipientAccountId": "*****"}Thank you,Let me know if any additional information is needed.FollowComment"

Answer: "0Can anyone provide any updates to this issue? Still having the same problem.Even following the AWS docs and using the JSON code provided to write input transformer queries it still returns no results.CommentShareBlakesanswered 4 years ago0/CommentShareBlakesanswered 3 years ago0Hello,An event in EventBridge has the format as defined in [1]. When you monitor an CloudTrail API call, it has the following format.{"id": "<required>","account": "<required>","source": "aws.<service>, <--- the source service"time": "<required>","region": "<required>","resources": "<required>","detail-type": "AWS API Call via CloudTrail""detail" : {// Contains the complete cloudtrail event}}Therefore for your use case, the actual event received by EventBridge would be:{"version": "0","id": "b03f6e6c-1a74-4f69-e219-84e2498d7f20","detail-type": "AWS API Call via CloudTrail","source": "aws.route53","account": "123456789012","time": "2022-04-01T00:00:00Z","region": "us-east-1","resources": [],"detail": {"eventVersion": "1.05","userIdentity": {"type": "IAMUser","principalId": "","arn": "arn:aws:iam:::user/blakes","accountId": "","accessKeyId": "","userName": "blakes","sessionContext": {"attributes": {"mfaAuthenticated": "true","creationDate": "2019-02-12T18:23:55Z"}},"invokedBy": "signin.amazonaws.com"},"eventTime": "2019-02-13T00:16:25Z","eventSource": "route53.amazonaws.com","eventName": "CreateHostedZone","awsRegion": "us-east-1","sourceIPAddress": "","userAgent": "signin.amazonaws.com","requestParameters": {"hostedZoneConfig": {"privateZone": false,"comment": "ERIC testing"},"callerReference": "AB354EGG09E-FC5E-046A-A6C1-8B1815375B0C","name": "testeric.com"},"responseElements": {"changeInfo": {"status": "PENDING","id": "/change/C3TJKPOO14VP6B","submittedAt": "Feb 13, 2019 12:16:25 AM"},"location": "https://route53.amazonaws.com/2013-04-01/hostedzone/Z3JK3REDKJVTQZ","hostedZone": {"resourceRecordSetCount": 2,"config": {"privateZone": false,"comment": "eric testing"},"name": "testeric.com.","id": "/hostedzone/Z3JKGRGERGDKJVTQZ","callerReference": "AB35409E-FC5EHERH-046A-A6C1-8B1815375B0C"},"delegationSet": {"nameServers": ["ns-942.awsdns-53.net","ns-121.awsdns-15.com","ns-1417.awsdns-49.org","ns-1927.awsdns-48.co.uk"]}},"requestID": "99c9fb95-2f24-11e9-a419-4fc3574ffweewqb5f52","eventID": "32bbf0ffw0e-1a95-416e-9dd5-ae2eec971b41","eventType": "AwsApiCall","apiVersion": "2013-04-01","recipientAccountId": "**"}}Based on the above, to extract the hosted zone name, you can use the following input transformer.{"hzname":"$.detail.responseElements.hostedZone.name"}"HZ name is <hzname>"The output:HZ name is testeric.com.Similarly you can extract other fields as required.[1] https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.htmlCommentShareSUPPORT ENGINEERAshish_Kanswered 4 months ago"


Question: Using an Intermediate CA with IAM Roles Anywhere

"I've created a private CA with an intermediate CA signed by the root CA and a user signed by the intermediate. The certs are valid, if I sign a user cert with the root CA, it works and AWS creds are returned, following the exact same process with the user cert signed by the intermediate CA yields a 403 AccessDeniedException.The intermediate Trust Anchor was created with the cert chain, I've tried with the root above and below the intermediate cert. I've tried with and without a "root" Trust Anchor containing just the root cert.The documentation doesn't describe the use case I'm attempting to make work, but the aws_signing_helper has an option --intermediates that I've supplied with the full chain, just the intermediate and just the root... all result in the 403.I am exploring this feature so I can be confident about discussing it with colleagues & customers, the fact that there are no references to intermediate CAs in the docs may well be because this isn't supported, but it'd be nice if the docs reflected that - it doesn't seem an reasonable use of the IAM Roles Anywhere service?Thanks in advance for thoughts or guidance from the community.FollowComment"

Answer: "0Intermediate CAs are supported by Roles Anywhere. I have successfully tested this with a private CA setup with a Root and Subordinate CA. A chain certificate comprising the Subordinate CA and Root CA was imported into Roles Anywhere as Trust Anchor. Then, an end entity certificate issued by the Subordinate CA was then used to authenticate successfully with Roles Anywhere without the  --intermediates option.CommentShareodeniqueanswered 10 months ago"


Question: AWSWickr vs WickrMe

"Dear colleagues,I am communicating in AWSWickr not only with my network, but t=with outside of the network! but today it stopped working and I cannot send messages anymore to the users outside of the networkFollowComment"

Answer: 0Are you communicating to the users outside network via federated access? Administrators have the ability to restrict access using network-ids or deactivate federation. Please check the FAQ link below and search for "External federation" for more details.I would also recommend opening a technical support case in case problem still persists.https://aws.amazon.com/wickr/faqs/CommentShareDhilipanswered 7 days ago


Question: How to enable access to a set of users to linux EC2 instance

"I have a Linux EC2 instance running in an AWS account. I have this requirementBuild an IAM group that enables ABC Organization's users to login into the EC2 instance, which contains following users :a_user,b_user,c_user,d_user,e_user,f_userAlso make the following users sudoers on the instancea_user (sudo)b_user (sudo)Which approach is best, Since I am blank on how to approach this. Any ideas?Everone has their own home directory and we don't want it to handle it manually by configuring SSH access for every user.FollowComment"

Answer: "0Create a group in IAM and give it a name,give the group permissions relating to EC2 and the level you require,then add the users to the group.Don't worry about making the users sudo. You can just type that in when you get to the command line. Example: sudo yum update -yhttps://docs.aws.amazon.com/IAM/latest/UserGuide/id_groups_create.htmlCommentShareKevin Foulgeranswered 8 months ago"


Question: user data commands not running

"Hi,I'm passing a number of powershell commands to an instance and the commands are been skipped, here's the log entry which is a warning:Skipping task postReadyUserData-executeScript-0I've searched for the error and there's nothing, has anybody got an idea of what's wrong?FollowComment"

Answer: "0Found the problem, the file C:\ProgramData\Amazon\EC2Launch\state.run-once already exists, so user data is not run.The image I'm using is a copy of a another instance, I'm having to reverse engineer a three year old system and it's all a bit messy.CommentSharecleggisimoanswered 3 years ago0Found the problem, the file C:\ProgramData\Amazon\EC2Launch\state.run-once already exists, so user data is not run.The image I'm using is a copy of another instance, I'm having to reverse engineer a three year old system and it's all a bit messy.Edited by: cleggisimo on Sep 14, 2020 3:30 AMCommentSharecleggisimoanswered 3 years ago"


Question: Significant delay in cloudwatch metrics for aurora db instances

We are experiencing a strange delay in CPU Utilization and DB Connections metrics for any new created aurora mysql db instances. Once the instance is created the metrics data is available as usual but within a time the delay is increasing. After a couple of days the delay is already about 2 hours.Engine version: 5.7.mysql_aurora.2.11.1Instance type: db.r5.xlargeRegion: eu-west-1This is how it looks like comparing to the old instances. The orange line is the new instance.All other metrics are up to date.CPU Total for enhanced monitoring is also working normally.Due to such delay it is not possible to create an alarm and CPU column does not show value on the database page.We have tried to create several new instances in different availability zones. The same issue for any of them.Has anyone experienced something like that? What can we do to resolve that?FollowComment


Question: MediaPackage corrupts input when passing in Tags

"aws-sdk javascript 2.419.0When I create a MediaPackage Channel in the Console, I get the correct inputs showing on MediaPackage, which are two inputs with an ID, URL, Username, and Password fields all filled out.When I use the SDK and use MediaPackage, the channel builds, but the MediaPackage channel has two inputs but the first one has no ID and the URL is has /None/channel in it (eg. https://555555ca212fa91fb.mediapackage.us-east-1.amazonaws.com/in/v2/zzzzza44a01634ee6bfa5af15cc13zzzzz/None/channel ).input data is normal:             var params = {                 Id: 'testchannel6',                 Description: 'standard description',                 Tags: {                     'OrgID': 'testorg'                 }             }This is exactly how it shows in the documentation:https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/MediaPackage.html#createChannel-propertyAlso, the tag is added... its just that when I add Tags to the params I get the /None/channel issue.to clarify. If I remove the "Tags" it works as expected - just without the tags obviously.I suppose this is a bug?Edited by: rbalhorn on Mar 22, 2019 2:58 PMFollowComment"

Answer: "0Accepted AnswerHi rballhorn,Thanks again for bringing this to our attention. New channels should now no longer show this behaviour. Please recreate any channels you still have in this state.I hope this helps, please let us know if you have any other questions!Kind regards,ThijsCommentShareThijsBanswered 4 years ago0Hi rbalhorn. Thanks for bringing this up. The service team will look into this and respond with an update as soon as possible.CommentShareAWS-User-6573863answered 4 years ago0works nowCommentSharerbalhornanswered 4 years ago"


Question: Does anyone having problem with AWS billing for ubuntu pro?

"We are looking to switch to ubuntu pro and ran into problem with billing. For t3a.nano running 18.04 lts, AWS provides pricing guideline on website (https://aws.amazon.com/marketplace/pp/prodview-rhaqbfd6lepbw#pdp-pricing)Software License - $0.001/hr + EC2 Instance - $0.005/hr = Total - $0.006/hrWe are charged - $0.0082 / hr on demand cost and then our savings plan for EC2 instance was deducted as - $0.0037 / hr (for half the time) and a net charge of $0.0045 / hrThere are few issues here -On demand price should have $0.006 / hr and not $0.0082 / hr as chargedThe credit should have been $0.005 / hr for EC2 instance and not $0.0037 / hrThe net charge for software license should have been $0.001 / hr instead of $0.0045 /hrI had a long and frustrating chat with AWS support center. Just checking if I am missing something here.FollowComment"

Answer: "0The figure in the screengrab from AWS Billing on your account matches exactly https://aws.amazon.com/ec2/pricing/on-demand/So the question to ask is why that set of figures doesn't align with those on the Ubuntu page on AWS Marketplace. Albeit the "Pricing Information" section includes a disclaimer twice calling it an "estimate". And the set of figures in that section are more like those on https://ubuntu.com/aws/pro and in the datasheet at https://assets.ubuntu.com/v1/f8087897-Ubuntu_Pro_AWS_07.12.21.1.pdf both of which are obviously Ubuntu-maintained and not AWS-maintained.It would be interesting to know if the figure on Marketplace constitutes an offer by AWS to sell the product/service at the indicated price, or whether it's just an indicative "best efforts" guess by Canonical and not enforceable on AWS (note that the EULA link on the marketplace page opens a link to an agreement between the customer and Canonical, not AWS).CommentShareRWCanswered 2 days ago0Thanks for your response. I missed the EC2 pricing information. I agree, AWS needs to clarify why they have two pricing information on their website, whoever maintains it. The difference is material enough to require an explanation.The second question is why the ec2 charge was for 10.83 hours while the savings plan credit was for 5.05 hour as provided in the screen capture. These two should match as well, unless I am missing something. If the credit was given for 10.83 hours, the difference would have washed out.CommentSharerePost-User-6419936answered 2 days ago"


Question: CIDR ranges for Red Hat OpenShift Service on AWS (ROSA) cluster

"Hi team,I want to create a Red Hat OpenShift Service on AWS (ROSA) clusterand I'm not sure if I should giveMachine CIDR:Pod CIDR:the same IP CIDR range which is the subnet CIDR, since machines and pods are running inside the same VPC, inside the same subnets.or those values should be different?my understanding is that machines and pods are running inside the same subnet soMachine CIDR:Pod CIDR:logically have the same IP range which is the IP range of the subnet in which they are running.for service CIDR can use always this value?Service CIDR:            172.30.0.0/16is there a specific workshop that walkthrough detailed steps on how to create a ROSA cluster inside private VPC with a  private linkthank you!FollowComment"

Answer: "0The Machine CIDR should include all the CIDR ranges for the subnets in your VPC. The Service CIDR and Pod CIDR ranges should be different from the Machine CIDR and each other, should not overlap, and should be the same among multiple clusters. Perhaps you will find this documentation helpful.CommentShareEXPERTbwhaleyanswered 5 months ago"


Question: Can you tell me about character setup.

"Special nodes es locomotion locator I don't know if I got what it is needed for, and I didn't get able to l use it in any way(as it doesn't seem to make any difference)Documentation doesn't tell so much about character details so it is really a try and miss.Is there a way to override motion extracted from root?(as root isn't really the best node to capture movement from as it generally traslate the entire rig.where do I have to place things in hierarchy es why to put bip bone in the hierarchy if it does just add complexity?Is there any kind of source files for max , maya and fbx with properly set up character to see how specific things can be implemented(lods , collision and ragdoll shapes, root motion, all kind of runtime ik?Some documentation looks criptic or just obsolete.What's the best pratices for rigging it would be very appreciated specific best pratices for common rigs es Biped, CAT , humanIK etc...It looks you refer to biped , but I found very uncomfortable using biped as the root of the biped isn't a great bone to capture motion.(It also goes up and down while walking) and you also need to move it on ground from the pelvis or character get absorbed in the floor.Geppetto also crash a lot especially when you create a new character.It also looks like fbx in far behind max exporter (compression makes feet slide a lot)proxies seems to be disabled,also had problem with skeleton scale different from animation one and so traslation get scaled differently than the rest of motion.FollowComment"

Answer: "0Great question! I've created a ticket to get some answers.CommentSharerePost-User-8353690answered 6 years ago0Hi @REDACTEDUSERThe legacy CryAnimation locomotion locator node was used for varying the speed of root motion of a character for particular animations - you wouldn't use it for animations where the speed of the character is constant. There is some more information about it here:http://docs.aws.amazon.com/lumberyard/latest/userguide/anim-char-locator.htmlIf you are using the Biped rig in 3D Max, you might actually need to add an extra joint and name it Locomotion Locator, since the root node of the Biped rig is at the pelvis, and not at the ground. In this case, it will need to be keyframed at the start and end frames of each animation in the position you desire for the speed of the character. Regarding root motion, in general most character rigs for video games separate the control of the entire rig with the control of root joint, so that root motion of the character can be controlled independantly of the entire rest of the character. This is how many games key the root motion relative to the character speed to prevent foot sliding, by tuning this in the animation. But if you actually want to override root motion, just disable root motion entirely and drive the speed of the character with your game code.Apologies that some of the documentation may need updating, it is constantly being worked on by our documentation team to provide the latest information. We appreciate your feedback and are working to improve it, so if you have any specific information you wish to see, please let us know.I am not sure what you mean by "bip" bone in regards to placing joints in the hierarchy, but there are many types of special joints that the game engine uses for information. Identifying the limbs for limb IK, ground contact IK, weapon aim joints, head look joints, and IK target joints are all part of IK systems designed to do specific things in the engine.Limb IK:http://docs.aws.amazon.com/lumberyard/latest/userguide/char-ik-limb.htmlGround Contact IK:http://docs.aws.amazon.com/lumberyard/latest/userguide/char-ik-leg-foot.htmlAim IK:http://docs.aws.amazon.com/lumberyard/latest/userguide/char-ik-aimpose.htmlLook IK:http://docs.aws.amazon.com/lumberyard/latest/userguide/char-ik-look.htmlIK Target:http://docs.aws.amazon.com/lumberyard/latest/userguide/char-ik-anim.htmlIn regards to a good example of character setup, I would point you to the Jack character in our new Starter Game. You can download the Starter Game pack if you haven't already here:https://aws.amazon.com/lumberyard/downloads/Inside the Starter Game, navigate to ...\dev\StarterGame\Objects\Characters\Jack, and you can look at how the Jack player character has been set up on the exported character side (not the source file). Thanks for your feedback regarding the source file, we have not released this character's source file, but we are working on releasing the FBX file in the future, which can help in examining the setup. However, you can see that the Jack character has root motion, physics proxies for ragdoll, and the runtime IK limbs, IK aim, and IK ground contact setups.LOD:http://docs.aws.amazon.com/lumberyard/latest/userguide/char-model-export-max-lod.html http://docs.aws.amazon.com/lumberyard/latest/userguide/char-export-maya-lumberyard-tools-lod.html http://docs.aws.amazon.com/lumberyard/latest/userguide/char-fbx-import-lod.htmlRagdoll:http://docs.aws.amazon.com/lumberyard/latest/userguide/char-phys-intro.htmlPhysics colliders:http://docs.aws.amazon.com/lumberyard/latest/userguide/char-attach-collisions.html http://docs.aws.amazon.com/lumberyard/latest/userguide/char-attach-proxy.htmlRoot motion ("Animate root" component property) on simple animation component: http://docs.aws.amazon.com/lumberyard/latest/userguide/component-simpleanimation.htmlFor character rigging best practices, you can look at our page here: http://docs.aws.amazon.com/lumberyard/latest/userguide/char-rigging-best-practices.htmlI think that any reference to "biped" in our documentation may either be a leftover reference from the old documentation, or a generic reference to a bipedal human rig (not the 3ds Max Biped system). In any event, the root joint of character should be at the ground level, not at the pelvis level, and oriented properly with a scene root node as needed.Geppetto crashes on new character creation are a known issue and being fixed in an upcoming release.As far as the new FBX importer versus the Max exporter, we are moving everything in our systems towards the FBX importer, and are improving it with every release. But regarding the foot sliding you are seeing, you can actually adjust the compression of the animation that is imported, and this should help.Scale is not currently used by the engine - make sure all joints are scaled to 1.0/1.0/1.0 in your base skeleton and in all animations.Thanks for your interest, please let us know if you have any further questions.CommentSharerePost-User-0479726answered 6 years ago0Thanks for the infos.As for the 3dsmax biped you can move the root at feet(using rubber band mode), but it doesn't work out as I previously stated, so the solution for biped was to move root out of the hierarchy(including the "bip" node it in the export isn't indeed required as it seems from the documentation) and use a different root bone in its place.Root seems to work even if randomly named.So yes the "bip" thing is possibly an old naming convention...indeed I achieved to export properly the character and the anim with this alternative setup with root motion and all.Also Cat worked quite well not without some modifications and always substituting the root motion extraction bone(the big arrow at character feets) with a custom one that follows the salient movement of the real one(the real one is too sensitive to rotation keeping the character turn at every step).It was a lot of trial and error however just to sort it out to export a clean root motion.Keying root motion at start and end of the animation is too simplistic for varing footstep timings key it manually sampling the good parts from the old root bone is far better with feet completely planted.fbx and compression is very bad as even small levels make feet slide and I don't know the actual compression as I didn't found the final compression ratio so I had to totally disable compression.This need to be verified imho the final result of fbx should be the equivalent of the other export system and sport the same exact parameters and final compression ratio and quality.Naturally ik driven feet may solve any sliding and compression issue as ik single targets compress far better than the hierarchical fk chain in any scenario.Look forward for the fbx sources.CommentSharerePost-User-7621041answered 6 years ago"


Question: Unload command added more decimal places to float4 field

"I have a table with a float4 field, and one of the values is120.12When I unload the data to a file in S3, and look at the file in a text editor, the related field now has the value120.120003I.e. it has made the value more precise by adding 4 more decimal places, and then (randomly) making the last decimal place a non-zero value. This is happening for many database records that I am unloading/exporting.Why is this happening, and what can I do to prevent it? Is it because I'm using float4 and it has "6 significant digits of precision"? Is redshift just adding in extra precision because I used float4? If so, should I be using DECIMAL(8,2) instead? I have used float4 previously without this issue when exporting/unloading.An example of the create table query:CREATE TABLE exampleTableName (  id       int identity(1,1),  accId    varchar(16) encode zstd,  amount   float4,  primary key(id))distkey(accId)interleaved sortkey(accId);The unload command:UNLOAD ('YYY')TO 's3://bucket/prefixFile_'IAM_ROLE 'XXX'HEADERADDQUOTESPARALLEL OFFMAXFILESIZE AS 5gbDELIMITER AS ',' GZIP;FollowComment"

Answer: "0Accepted AnswerYes, this happens when you use FLOAT for the decimal data because it cannot store values of arbitrary precision. https://en.wikipedia.org/wiki/IEEE_754I would generally always recommend using the DECIMAL datatype unless you have an existing application that has an unchangeable requirement for FLOAT, e.g., calculation expects FLOAT and output cannot change.Additionally, by using DECIMAL you are able to use our new AZ64 compression encoding which will reduce the amount of storage needed and improve your query performance. https://aws.amazon.com/about-aws/whats-new/2019/10/amazon-redshift-introduces-az64-a-new-compression-encoding-for-optimized-storage-and-high-query-performance/CommentShareAWS-User-2965899answered 4 years ago"


Question: Installing Python modules in EMR Cluster using EMR Notebook

"Folks:I am running some code that uses a mix of PySpark (for data manipulation) and Python (for visualization).  Very similar to this blog:https://aws.amazon.com/blogs/big-data/install-python-libraries-on-a-running-cluster-with-emr-notebooks/.  The cluster I am using has all of the defaults:Release label: emr-5.36.0Hadoop distribution: Amazon 2.10.1Applications:Spark 2.4.8, Livy 0.7.1, Hive 2.3.9, JupyterEnterpriseGateway 2.1.0The command:  sc.install_pypi_package("pandas")  seems to work successfully.However the command: sc.install_pypi_package("matplotlib") fails with an error on the Pillow dependency. The specific error is:Building wheels for collected packages: unknown, unknown  Running setup.py bdist_wheel for unknown: started  Running setup.py bdist_wheel for unknown: finished with status 'error'  Complete output from command /tmp/1669916958616-0/bin/python -u -c "import setuptools, tokenize;__file__='/mnt/tmp/pip-build-e28wksxd/pillow/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmprh77de26pip-wheel- --python-tag cp37:  running bdist_wheel  running build  running build_ext      The headers or library files could not be found for jpeg,  a required dependency when compiling Pillow from source.I logged into the Master node on the EMR cluster and attempted to install some of the libraries and Python compiler support using:sudo yum install python3-devel redhat-rpm-config libtiff-devel libjpeg-devel openjpeg2-devel zlib-devel \    freetype-devel lcms2-devel libwebp-devel tcl-devel tk-devel \    harfbuzz-devel fribidi-devel libraqm-devel libimagequant-devel libxcb-develWhile the installation of pillow gets a bit further, there are still errors such as:Collecting pillow  Using cached https://files.pythonhosted.org/packages/16/11/da8d395299ca166aa56d9436e26fe8440e5443471de16ccd9a1d06f5993a/Pillow-9.3.0.tar.gzBuilding wheels for collected packages: unknown, unknown  Running setup.py bdist_wheel for unknown: started  Running setup.py bdist_wheel for unknown: finished with status 'done'  Stored in directory: /var/lib/livy/.cache/pip/wheels/55/5a/ad/9f708fd6d1500e9ff680e17b1c2f436e8439477a5a226611c6  Running setup.py bdist_wheel for unknown: started  Running setup.py bdist_wheel for unknown: finished with status 'error'  Complete output from command /tmp/1669918064446-0/bin/python -u -c "import setuptools, tokenize;__file__='/mnt/tmp/pip-build-jnpkka_0/unknown/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpn9k3ctzopip-wheel- --python-tag cp37:  Traceback (most recent call last):    File "<string>", line 1, in <module>    File "/tmp/1669918064446-0/lib64/python3.7/tokenize.py", line 447, in open      buffer = _builtin_open(filename, 'rb')  FileNotFoundError: [Errno 2] No such file or directory: '/mnt/tmp/pip-build-jnpkka_0/unknown/setup.py'I feel as if I am missing something very obvious. It can not possibly be this difficult to get a commonly used package like matplotlib to work.Any suggestions?ThanksRich H.FollowComment"


Question: "24x7 access to Cloud Support Engineers via phone, chat, and web.  -  HOW?"

"I just purchase a Business plan , it states 24x7 access to Cloud Support Engineers via phone, chat, and web. but I can't even find a phone number to ask for support! help!FollowComment"

Answer: "0Hello,For security purposes, AWS Support doesn’t have a phone number. When you open a support case through the case console, select Phone. You're prompted for a callback number. You'll receive a call from AWS Support as soon as you create the case.Please see this article for full instructions on opening a support case by phone. Hope this helps. Please select accept answer if it did.Reference: https://aws.amazon.com/premiumsupport/knowledge-center/aws-phone-support/CommentShareSUPPORT ENGINEERAWS-User-Chiraganswered 9 months ago"


Question: Codedeploy to server farm - high level understanding and confirmation

"Morning all, I want to confirm I am on the right track and just logically trying to put things in order.   I have a group of servers that are quite static (no need for autoscaling).  It's a java app, and Beanstalk doesn't support the app.   So right now the developer is going to each server via custom ports to undeploy and deploy new apps.  The farm has grown and its time consuming.    Some of the reading is a bit confusing, so for a quick high level, I had these questions/help items.  Using the left side nav from codecommit,Under Sourceits always just one file (a .war file). so regardless, I believe I will need to use either GIT/CodeCommit and can't just have the developer upload a file to an S3 bucket right?   That would enable the version control, revert, etc.Under DeployApplications -  I have created the application, service role, type and configuration (by key/value).  Also have the enable load balancing checked with a test group (one server right now).I don't think I really need to worry about the artifacts or the build, all I want is for the developer to finish his WAR file, commit.   So not sure what is the magic that after I he commits, it takes that code, then push's to the servers.  The agent is installed and just not sure if I need a pipeline setup, or there is something between the deploy and the agent and a simple answer here may prove much faster than other resources.So thank you again for the feedback and/or suggestions if there is a better way but think once setup this really is quite simple.FollowComment"

Answer: "0I recommend you to use AWS CodePipeline ( a service similar to Jenkins), which cost $1 monthly per active pipeline.In theory, with CodePipeline you don't need CodeCommit you can use S3 as source.Ideally you should only checkin your source code, then use CodeBuild to build an artifact (WAR for example).However if you decide to directly push WAR for now, S3 might be a better choice.After Checkin / upload to S3, the CodePipeline can be configured to either going through a build process (in case of a source file) or directly deploy and configure the artifact to target environment (EC2 or otherwise) through a deployment provider, such as CodeDeploy.CommentShareJason_Sanswered a year ago0Thanks @Jason -Edited this after more reading and understanding ...  The basic flow works like this and this is my current status;code is pushed   (I am testing with 2 files, the WAR and the appspec.yaml file).    From the CLI;aws deploy push —application-name name —description “iniital test” —ignore-hidden-files —s3-location s3://bucket/folder/myapp.zip —source .After that, I had the single file in that S3 bucket myapp.zip.Npw step 2 is to use codedeploy (or another method) to get that down to the server.  I like CodeDeploy as I can do a one at a time (not sure if I can with pipeline), but from the console, when I tried, it just kept spinning.  I had the code-deploy agent installed (manually from an earlier test), so removed it and said allow SSM to install.  That machine should have the permissions via IAM-Roles.   But after the timeout, the error is "CodeDeploy agent was not able to receive the lifecycle event. Check the CodeDeploy agent logs on your host and make sure the agent is running and can connect to the CodeDeploy server."However code-deploy is not running, so I assume its an issue with SSM permissions not being allowed to run there?   So, I can continue this, OR as you suggested code pipeline (if it can do the same one at a time), however the bigger picture is the apppec file as that has the where to put and what commands to run after.Remember this is a single file and source control is not needed as we keep the olders in a sub folder, so really would be perfect if he dropped a file in a folder (new), something like codepipeline fired, which put the file in the right place and ran the scripts to do (which are on an EFS).   So I was even looking at eventbridge, but really need that one at a time feature.Thank you again.CommentSharelraymondanswered a year ago"


Question: Do emails to addresses on your account-level suppression list count towards your Complaints Rate?

The SES documentation states:SES doesn't count the messages that you send to addresses on the account-level suppression list toward the bounce rate for your account.Is the same true for Complaints where the complaint sub type is "OnAccountSuppressionList"? I saw an increase to our Complaint Rate this morning that appeared to be due to two emails sent to addresses that were already on our account-level suppression list. Do we need to prevent our application from sending to email addresses on our account-level suppression list for which we have previously received a Complaint to prevent further hits to our Complaint Rate? My understanding and expectation was that the suppression list prevents hits to your Bounce and Complaint rates even though the documentation only explicitly mentions Bounces.FollowComment


Question: Restore Redshift Snapshot to different account

"We are trying to restore a snapshot of a redshift cluster from one account into a new account. Accounts are unlinked.  Snapshot process and copy works fine.  When we try to restore the snapshot on the new account it is requiring access to to the Redshift KMS key from the source account.The source cluster for this snapshot is encrypted. When you restore, the target database will be encrypted as well. You can't unencrypt the data as part of the restore operation. After you restore, you can change encryption settings.Here it is prompting for access to the source key.   I am not sure how, or if it is possible to  give the new/destination account or my user access to the key from the old source account.    The source Redshift cluster uses a AWS managed  aws/redshift key from the source account.   We want to bring the whole snapshot, redshift users and all, not just the data.FollowComment"

Answer: "0Accepted AnswerYou can't use built-on KMS key for Redshift on another account.CommentSharerePost-User-1579460answered 10 days ago0Thank you Anusha.  Since the source key is an AWS managed key, is it still possible to do what you described?  I can edit the key policy for customer managed keys but don't appear to have the same option to edit key policy for AWS managed keys.   It could be my access level but maybe the key policy can't be edited for AWS managed keys?CommentSharerePost-User-1579460answered 5 months agoAnusha_C 5 months agoKey policy on AWS managed keys cannot be updated. However, you can change they KMS key during restore process. Under Additional configurations, provide the key you want to use for Choose an AWS KMS key. This can be the AWS managed redshift key from the account.Please note that restore process takes longer when the key is changed during restore process.Share0In the source account, provide access on the KMS Key used for encryption to the target account.In order to do it, you can follow the steps below:In AWS Management console of the source account, navigate to KMSClick on the KMS key that was used to encrypt the clusterClick on the Edit button in the Key Policy tabUpdate the Key policy to add the principal for destination account  as shown in the sample policy below{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Principal": {                "AWS": [                    "arn:aws:iam::{SourceAccountID}:root",                    "arn:aws:iam::{DestinationAccountID}:root"                ]            },            "Action": "kms:*",            "Resource": "*"        }    ]}Once you provide access on KMS key to destination account's principal, re-try snapshot restore.If you are using an AWS managed, Key policy cannot be updated. In such case, you can change they KMS key during restore process. In the restore screen, under Additional configurations, provide the key you want to use for Choose an AWS KMS key. This can be the AWS managed redshift key from the target account.Please note that restore process takes longer when the key is changed during restore process.The restore process moves users, groups along with data.CommentShareAnusha_Canswered 5 months agorePost-User-1579460 5 months agoOn more research we found that we can't share or edit the policy of an AWS managed key.  We were able to unecrypt and re-encrypt a test cluster to a customer managed key but we do not want the state of the production cluster to ever be unencrypted - even during this process. We tried changing they aws key to our key directly in the console but got the error below.InvalidParameterValueYou can't transform a cluster with KMS encryption type to KMS encryption type.Share"


Question: Using AWS Tools for Powershell in a Powershell lambda

"Hello,I am encountering a huge roadblock performing the most basic AWS tasks within a Powershell Lambda.I have created a role with secretsmanager and basic lambda execute permissions. Created a test secret. Created a one-liner Powershell Lambda with Get-SECSecretValue to retrieve the test key value.No VPC is assigned to the lambda, from what I understand this is not needed for a simple internal service call.The command executes, and then hangs. Both the lambda and secret are in the same region, same account.I have tried to get the lambda to access various other services, all unsuccessfully.To add further datapoints, I created a Python lambda that does the same thing, accessing the same secret. Used the same role and did not select a VPC. This lambda has no issues whatsoever retrieving the key.Could really use some help here, being able to retrieve a secret is mission critical to this project.Unfortunately I can't convert the scripts to Python (Thanks Microsoft).I could use another python lambda to retrieve the secrets and pass them to the initiating lambda, but that seems a bit excessive, and I don't even know if I can trigger a lambda from the PS one, since I can't seem to connect to any AWS services.FollowCommentAWS-User-5291433 a year agodid you check the CloudWatch log? Did you try the CLI with debug option? You can run PowerShell from AWS SSM as well.Share"

Answer: "1This should be working without any issues, if configured correctly.You could leverage the AWS Lambda Developer Guide - there is an example of setting up a basic infrastructure for a PowerShell Lambda (the link above points directly to that example).Following the steps in the guide, you get an AWS Lambda with the .NET Core 3.1 (C#/PowerShell) runtime and some basic CmdLets as an example.To test your use case, I manually created a simple secret in the AWS Secrets Manager with a secret ID test-secret. I used the DefaultEncryptionKey and no additional options. I placed the secret in the same region where my Lambda function resides (eu-west-1 in my case).In the Lambda function, I call the AWS API as follows:$secret = Get-SECSecretValue -SecretId test-secretWrite-Host $secret.SecretStringTo allow AWS Lambda to access my newly created secret, I extended the already pre-configured Lambda execution role with an inline policy (here, I anonymized some values with *):{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "MySecretAccess",            "Effect": "Allow",            "Action": "secretsmanager:GetSecretValue",            "Resource": "arn:aws:secretsmanager:eu-west-1:************:secret:test-secret-******"        }    ]}So this immediately worked out-of-the-box, here is my CloudWatch log for this function call:Importing module ./Modules/AWSPowerShell.NetCore/3.3.618.0/AWSPowerShell.NetCore.psd1[Information] - {"test-secret":"TEST_SECRET_VALUE"}Please try the steps from the guide I mentioned above and see if this works for you. If it still doesn't, I'm happy to help, but please take a look on your CloudWatch logs and also make sure your deployment works.CommentShareDmitry Balabanovanswered a year agoDerekBotbyl a year agoFollowing the Developer Guide, I created a basic lambda and modified the code as such:#Requires -Modules @{ModuleName='AWS.Tools.SecretsManager';ModuleVersion='4.1.24.0'}write-host(Get-SECSecretValue -SecretId "TestKeys" -verbose)Execution role has been extended with"Effect": "Allow","Action": "secretsmanager:","Resource": "arn:aws:secretsmanager:us-east-1:*****************:secret:TestKeys-*********"ShareDerekBotbyl a year agoCloudwatch log:2022-02-22T13:39:29.043-06:00START RequestId: {task guid} Version: $LATEST2022-02-22T13:39:29.047-06:00Importing module ./Modules/AWS.Tools.Common/4.1.24.0/AWS.Tools.Common.psd1 Importing module ./Modules/AWS.Tools.SecretsManager/4.1.24.0/AWS.Tools.SecretsManager.psd12022-02-22T13:39:29.964-06:00[Verbose] - Invoking AWS Secrets Manager operation 'GetSecretValue' in region 'us-east-1'2022-02-22T13:39:59.077-06:00END RequestId: {task guid}2022-02-22T13:39:59.077-06:00REPORT RequestId: {task guid} Duration: 30030.73 ms Billed Duration: 30000 ms Memory Size: 512 MB Max Memory Used: 161 MB Init Duration: 2328.03 ms2022-02-22T13:39:59.077-06:002022-02-22T19:39:59.075Z {task guid} Task timed out after 30.03 secondsShareDerekBotbyl a year agoI had been following the documentation here: https://docs.aws.amazon.com/lambda/latest/dg/powershell-devenv.htmlIn changing from the module import "AWS.Tools.Common" to the older "AWSPowerShell.NetCore" I now get an SSL error "The remote certificate is invalid according to the validation procedure." How am I getting SSL cert errors when I'm not even using external services? I am initiating the lambda via the Test function, and I have used the CLI to invoke it with the same results.Share"


Question: "Can Redshift copy function take partial CDC 'Change data capture' data ? Namely, only primary key(s) and any fields that have changed"

"We have a need to start streaming oracle golden gate changed data capture data into S3 then Redshift. However, the cdc data we received will be partial, namely, only primary key(s) and any fields that have changed. It will not a full record.  Is it possible to stream these data into S3 and Redshift? can Redshift copy function take partially cdc data?FollowComment"

Answer: "0You can specify a comma-separated list of column names to load source data fields into specific target columns. The columns can be in any order in the COPY statement, but when loading from flat files, such as in an Amazon S3 bucket, their order must match the order of the source data.When loading from an Amazon DynamoDB table, order doesn't matter. The COPY command matches attribute names in the items retrieved from the DynamoDB table to column names in the Amazon Redshift table. For more information, see Loading data from an Amazon DynamoDB tableThe format for a column list is as follows.COPY tablename (column1 [,column2, ...])If a column in the target table is omitted from the column list, then COPY loads the target column's DEFAULT expression.If the target column doesn't have a default, then COPY attempts to load NULL.If COPY attempts to assign NULL to a column that is defined as NOT NULL, the COPY command fails.If an IDENTITY column is included in the column list, then EXPLICIT_IDS must also be specified; if an IDENTITY column is omitted, then EXPLICIT_IDS can't be specified. If no column list is specified, the command behaves as if a complete, in-order column list was specified, with IDENTITY columns omitted if EXPLICIT_IDS was also not specified.If a column is defined with GENERATED BY DEFAULT AS IDENTITY, then it can be copied. Values are generated or updated with values that you supply. The EXPLICIT_IDS option isn't required. COPY doesn't update the identity high watermark. For more information, see GENERATED BY DEFAULT AS IDENTITY.https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-column-mapping.html#copy-column-listCommentShareJaganswered a year ago"


Question: Can't Create Cluster - The placement group for EFA-enabled compute resource

"Hello Folks,I'm following along with one of the HPC PCluster Gromacs workshops.  I've made the modifications to get the example to work with PClusterv3 (I think).  However, when I issue the command to create the cluster, I get"    {"level": "ERROR","type": "EfaPlacementGroupValidator","message": "The placement group for EFA-enabled compute resources must be explicit. You may see better performance using a placement group, but if you don't wish to use one please add 'Enabled: false' to the compute resource's configuration section."},"The code that I  think is the problem looks like this:Scheduling:Scheduler: slurmSlurmQueues:- Name: g4dnComputeSettings:LocalStorage:RootVolume:Size: '100'Encrypted: 'false'VolumeType: 'gp2'ComputeResources:- Name: g4dnInstanceType: g4dn.8xlargeMaxCount: 3MinCount: 0DisableSimultaneousMultithreading: 'false'CustomActions:OnNodeStart:Script: s3://pcluster-2021-10-07-XXXXX/post.install.shNetworking:SubnetIds:- subnet-098eaXXXXXPlacementGroup:Enabled: 'true'Id: 'TestPlacementGroup'I've tried using an Id of an existing PlacementGroup in the targeted AWS Region, and I've tried not having an Id as the docs say that one will be created if not specified.  Both attempts fail.How do I get the "PlacementGroup" to not throw an error and to create the cluster?Any help is greatly appreciated.FollowComment"

Answer: "0Hello darkdiesel,the EfaPlacementGroupValidator throws an error when EFA is enabled through the EFA option (https://docs.aws.amazon.com/parallelcluster/latest/ug/Scheduling-v3.html#yaml-Scheduling-SlurmQueues-ComputeResources-Efa) but placement group is not explicitly set with the option PlacementGroup (https://docs.aws.amazon.com/parallelcluster/latest/ug/Scheduling-v3.html#yaml-Scheduling-SlurmQueues-Networking-PlacementGroup-Enabled).In your configuration I don't see any EFA setting and unfortunately indentation got lost. Can you please open a ticket on https://github.com/aws/aws-parallelcluster/issues reporting again the configuration used so we can better follow you on this? ThanksCommentSharelukeseawalkeranswered 2 years ago0Hello Luca,I had messed up my config, and was looking at the wrong EFa location.  It was user error all along.CommentShareAWS-User-0014169answered 2 years ago"


Question: Direct Connect Location with different associated AWS Region

I have a question from a customer:We are planning on installing 2 new DX for the ap-southeast-1(Singapore) region.  We would prefer to be in Equinix facilities as wealready have a relationship with them.  I have earmarked HK1 and SG2as possibilities but I notice that they have a different AssociatedAWS region but are in the same Direct Connect Geographical Region.Please can you confirm that these are a valid choice?Is it ok to have a Direct Connect Location such as HK1 connecting to a non "Associated AWS Region" like Singapore?FollowComment

Answer: "0Accepted AnswerYes, it's OK.With direct connect gateway, you can connect to all AWS region (except China) from any Direct Connect location. You can check the deep dive session from last reinvent "NET403_DirectConnect_DeepDive":https://youtu.be/DXFooR95BYcCommentShareAWS-User-2823488answered 4 years ago"


Question: Can I implement Sender Rewrite Scheme for forwarded emails

"I have a mail rule set to redirect all incoming emails to a different email address which is provided by a 3rd party email provider.  This works for most emails but for any incoming email where the originating domain uses strict SPF, the email servers for the domain I forward to do not accept the email as the forwarded email is seen as coming from Amazon servers and not the original domain servers as defined in the SPF record.  Is it possible to implement the Sender Rewriting Scheme so that these emails meet the SPF requirements?  Does it need to be done in SES? Or is this something that needs to be done by AWS at a higher level?FollowComment"


Question: unable to run mysql once it is downloaded

"I have successfully reached the point where MySQL is downloaded but udo yum install -y mysql-serverLoaded plugins: extras_suggestions, langpacks, priorities, update-motd235 packages excluded due to repository priority protectionsPackage 3:mariadb-server-10.2.38-1.amzn2.0.1.x86_64 already installed and latest versionNothing to dowhen trying to runsudo service mysqld start && sudo service mysqld statusI receive this errorFailed to start mysqld.service: Unit not found.Anyone else had this problem and resolved it ?FollowComment"

Answer: 0Try thissystemctl start mariadb--SydCommentShareSydanswered a year ago


Question: How can I reset a member account's 2FA from the root account?

We no longer have access to the 2FA device for a couple of our organization's member accounts. We do have the id's and passwords. How do we reset the 2FA for those accounts from the root account?FollowComment

Answer: "3https://repost.aws/knowledge-center/reset-root-user-mfa ResolutionTo reset your MFA device, you must have access to the AWS root user account email address and phone number associated with the account.Note: If you are an AWS Identity and Access Management (IAM) user, you can't reset MFA by yourself. You must contact your administrator to deactivate the device. For more information, see Recovering an IAM user MFA device.Sign in using your AWS account root user email address.On the Root user sign in page, enter the password of your root account.On the Amazon Web Services Sign In With Authentication Device page, choose Troubleshoot MFA? Click here.On the Troubleshoot Your Authentication Device page, choose Sign In using alternative factors.On Step 1: Email address verification, validate that the email address is correct and choose Send verification email.In the email from AWS with the subject line, AWS Email Verification, choose Verify your email address. The Step 2 page in the verification process appears.On Step 2: Phone number verification, confirm the phone number listed is correct, and then choose Call me now.Note: If you didn't receive the automated telephone call or you need to update the phone number, see How do I update my telephone number to reset my lost MFA device?Answer the phone call from AWS and use your phone’s keypad to submit the six-digit verification code that appears on your device's screen.On Step 3: Sign In, choose Sign in to the console. You are automatically redirected to your Security Credentials.Choose Deactivate, next to the MFA device that you want to reset.Choose Activate MFA to add an MFA device.Note: It's a best practice to enable a new MFA device on your root account as soon as possible to make sure that your root account is protected by MFA.If you lost your MFA device, you can still protect your root account with MFA by adding a virtual device, see Enabling a virtual multi-factor authentication (MFA) device.If you need additional assistance, see Lost or unusable multi-factor authentication (MFA) device . Choose the button I'm still having problems and would like to contact AWS Support. Then, complete the Request assistance with lost or unusable MFA device form and choose Submit.CommentShareabemusaanswered 2 months ago0Hi, I think this guide will make It more clear: https://repost.aws/knowledge-center/reset-root-user-mfaHope it helps ;)CommentShareEXPERTalatechanswered 2 months ago"


Question: Error in connecting to document DB intermittently

"Hi,I am facing the below exception intermittently.  I am using the connection as below.“errorMessage”:“MongoTimeoutError: Server selection timed out after 30000 ms”,“reason”:{“errorType”:“MongoTimeoutError”,“errorMessage”:“Server selection timed out after 30000 ms”,“name”:“MongoTimeoutError”,“stack”:[“MongoTimeoutError: Server selection timed out after 30000 ms”," at Timeout.setTimeout [as _onTimeout] (/var/task/node_modules/mongodb/lib/core/sdam/server_selection.js:308:9)"," at ontimeout (timers.js:436:11)"," at tryOnTimeout (timers.js:300:5)"," at listOnTimeout (timers.js:263:5)"," at Timer.processTimers (timers.js:223:10)"]},“promise”:{},“stack”:[“Runtime.UnhandledPromiseRejection: MongoTimeoutError: Server selection timed out after 30000 ms”," at process.on (/var/runtime/index.js:37:15)"," at process.emit (events.js:198:13)"," at process.EventEmitter.emit (domain.js:448:20)"," at emitPromiseRejectionWarnings (internal/process/promises.js:140:18)"," at process._tickCallback (internal/process/next_tick.js:69:34)"]}let connectionString = mongodb://${data.username}:${data.password}@${clusterEndpoint}:27017/?ssl=true&ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0;return new Promise((resolve, reject) => {mongodb.connect(connectionString, { useUnifiedTopology: true }, (err, res) => {if (err) {reject(err);} else {resolve(res);}})});FollowComment"

Answer: "0Looking at the errors below, seems like your connection is timing out. Timeouts could happen if 1/ The client where your Node.js app is running is in a different VPC/different region than your Amazon DocumentDB cluster 2/ The security group associated with your cluster does not allow inbound connections on port 27017. You mentioned you see the error intermittently. Are there any patterns, triggers that typically cause this? Are there any changes in your environment that would cause this intermittently?Edited by: AWSMeetB on Sep 10, 2020 11:19 PMCommentShareAWS-User-8311923answered 3 years ago0I am also facing this issue intermittently without any pattern. I am using pymongo python client to connect with docdb.CommentSharesumit0900answered 2 years ago"


Question: Migration of wordpress woocommerce website from Bluehost to AWS

"As mentioned in the subject, I need to migrate my wordpress woocommerce website which is currently hosted on Bluehost server to be migrated on AWS server. Please help me in guiding the same step by step .FollowComment"

Answer: "0Hi @Trenfort,Luckily there are lots of blogs and tutorials available on the internet.  The first thing though is to decide on your non-functional requirements, for example do you need high availability, or auto-recovery with a brief outage, or just a single server that needs manual fixing if it goes down?  Here's some links that might help:Basic single server - https://aws.amazon.com/getting-started/hands-on/launch-a-wordpress-website/, https://aws.amazon.com/lightsail/projects/wordpress/HA-capable - https://aws.amazon.com/solutions/partners/wordpress-high-availability-bitnami/, https://aws.amazon.com/getting-started/hands-on/build-wordpress-website/CommentShareEXPERTskinsmananswered 4 months ago"


Question: (Boto3) How can I call the VPC Route tables to return an indication that a Network Firewall is being used?

I'm currently writing a config rule. I was able to write a config rule to see if a Network Firewall is attached to the VPC. I am not revieiwing the boto3 documentation here to determine if a networkfirewall is being used.Is there a method or certain Filters I can call to determine if the subnet is going to a Network Firewall?FollowComment

Answer: "3There's nothing that is going to directly indicate that a Network Firewall is being used. However, what you can do is get a list of Network Firewall endpoints that are deployed into the VPC; then check the route entries in each route table and see if the destination is one of those endpoints.CommentShareEXPERTBrettski-AWSanswered 7 months agoEXPERTTushar_Jreviewed 7 months agorePost-User-8984971 7 months agoDo I use filter to get a list of the route entries?The filters in boto3 docuemntation for describe_route tables?ShareBrettski-AWS EXPERT7 months agoYou may be able to filter on route.gateway-id.Share"


Question: Redshift Serverless Charging

"Redshift Serverless ChargingThe Charge of Compute usage on Frankfurt region  for Redshift Serverless is not matching the log on the sys_serverless_usage .I have charged by 758 RPU -Hour on the month of sep till 28th , while the below query for the same period returned 76 RPUHour ?select date_trunc('day',start_time) usage_date,sum(compute_seconds) com_sec,sum(((compute_seconds) ) )/ (60 * 60 ) total_consumer_hour,(sum(((compute_seconds) ) )/ (60 * 60 ) ) *  (0.45) compute_costfromsys_serverless_usagegroup by date_trunc('day',start_time)I believe the issue on the charging is dividing the compute seconds by (60 times 6) rather than (60 times60), this is also happen when setting alert on RPU-USAGE is also alert on the number i post (example 20 RPU-H daily) it alert on 2 RPU-H.FollowCommentrePost-User-7883820 7 months agoCurious if you figured this out, we are facing similar problems!Share"

Answer: "0Hi ,After one and half month with the support team , it is noticed that using the query on the documentation is wrong to be able to monitor cost. Because the serverless charge with min of 60 sec.So if you run on each min for 10 mins  a query that take 1 sec on base rpu 32The view will have a record each min (132) =32    to i assume that to get the cost for this min this will be 32/3600 cost of rpuhowever actual charging will be (6032) to charge will be (6032)/3600* costSo the conclusion serverless version shouldn't be used for less small running queriesCommentSharerePost-User-2747121answered 7 months ago-1Hi,I found this article about Redshift billing that you may find helpful: https://docs.aws.amazon.com/redshift/latest/mgmt/serverless-billing.html. If this doesn't clear up your concerns, I suggest opening a case with our Billing and Account Support team via Support Center for further assistance: go.aws/support-center.— Sage A.CommentShareEXPERTAWS Support - Sageanswered 8 months ago"


Question: Conversion to Datatype Results in Error

"SQL Server TIMESTAMP fields are being converted to a Postgres BIGINT.When I run a Data Migration Task with SQL Server as the Source and Postgres as the target, I receive the error below in the RDS logs of the target Postgres instance.[23476]:ERROR:  invalid input syntax for type bigint: "000000000064B0AD" 2023-01-23 21:43:18 UTC:10.220.246.94(54136):administrator@staging:[23476]:CONTEXT:  COPY tblDocument, line 1, column thetimestamp: "000000000064B0AD" 2023-01-23 21:43:18 UTC:10.220.246.94(54136):administrator@staging:[23476]:STATEMENT:  COPY  "dbo"."tblDocument" FROM STDIN WITH DELIMITER ',' CSV NULL 'attNULL' ESCAPE '\'When I change the datatype of the field tyo BYTEA, no errors.FollowComment"

Answer: "1Hello Ross,Did you have a chance to take a look here?https://docs.aws.amazon.com/dms/latest/sql-server-to-aurora-postgresql-migration-playbook/chap-sql-server-aurora-pg.sql.datatypes.htmlNot sure if this would work for you, but there is an option to use SCT as well.https://aws.amazon.com/dms/schema-conversion-tool/CommentShareTodor Todorovanswered 4 months agoRoss Bush 4 months agoThanks for the answer. I used the SCT to migrate and the conversion chosen was bigint. I have skimmed the documentation you posted and will read it in its entirety tomorrow.Share"


Question: AWS Neptune returning InternalFailureException

"I was trying to do iterate over AWS Neptune graph and do some manipulation from my Lambda function but I am getting an error of2023-05-22T01:34:38.352Zf8bfa54b-8230-47eb-a5ab-be56e8ae6190ERRORError: ResponseError: Server error: {"requestId": "5d69c1d4-5db0-4632-8604-50eab55b9c16","code": "InternalFailureException"} (599)at Connection._handleMessage (/var/task/node_modules/gremlin/lib/driver/connection.js:341:9)at WebSocket.<anonymous> (/var/task/node_modules/gremlin/lib/driver/connection.js:144:43)at WebSocket.emit (node:events:513:28)at Receiver.receiverOnMessage (/var/task/node_modules/ws/lib/websocket.js:1184:20)at Receiver.emit (node:events:513:28)at Receiver.dataMessage (/var/task/node_modules/ws/lib/receiver.js:526:14)at /var/task/node_modules/ws/lib/receiver.js:492:23at /var/task/node_modules/ws/lib/permessage-deflate.js:309:9at /var/task/node_modules/ws/lib/permessage-deflate.js:392:7at afterWrite (node:internal/streams/writable:500:5) {statusCode: 599,statusMessage: '{"rquestId":"5d69c1d4-5db0-4632-8604-50eab55b9c16","code":"InternalFailureException"}',statusAttributes: Map(0) {}}I have no idea what the problem is , the code is correct and clean .FollowCommentAWS-KRL 8 days agoIt's hard to know just from the exception what may have happened there. Are you able to share a code snippet showing what was being attempted at the time of the failure? Is this something that happens every time, or just happened once/happens occasionally?Share"


Question: Removing Stack Instances During Account Decommission

"I'm looking for best practice guidance on removing accounts from Control Tower and dealing with stack instances.In my management account I have CfCT setup. If I understand correctly, I need to delete the Service Catalog product, move the account to a Suspended OU outside the scope of CfCT, and close the account.At that point, all stack instances from CT and CfCT will still remain and I should eventually remove them.My question is: do you need to remove these stack instances immediately to prevent errors in Control Tower or CfCT? Or can I remove these over an extended period (for example 7 days) of time without risk of errors as I continue to use CT/CfCT?FollowComment"

Answer: "1Accepted AnswerIt would be better to remove them immediately during the account decommissioning process. CfCT may throw errors if an account is listed in stack instances and it can't access the account (suspended or had the AWSControlTowerExecution role removed)CommentShareRoguenanswered 3 months ago0So it sounds like the best order of operations is to remove all stack sets from CT/CfCT prior to account closure. Or all together:-Remove Service Catalog Product-Move to suspended OU-Delete any remaining Stack Instances-Close account.CommentShareTSquaredanswered 3 months agoRoguen 3 months agoA few other steps that I think would be relevant with some added detail and a little re-ordering. For the most part I think you’ve got the idea though:-Move account to “Transitional” OU - or some OU that is outside of manifest OUs but within Control Tower governance. Do this by doing an update to the provisioned product in Service Catalog.-Rerun the CfCT pipeline, this action will delete StackSet instances deployed by CfCT from the account.-Terminate the provisioned Service Catalog product associated with the account to unmanage account from Control Tower. This action will also delete StackSet instances deployed by Control Tower from the account and also removes the Control Tower admin role.-Ensure all resources are shut down/deleted on the account (EC2, RDS, etc…).-Move to “Suspended” OU which is outside of both Control Tower control and CfCT manifest and has a deny * SCP attached--Leave in Suspended OU. Verify CfCT and StackSets are working properly.--Delete the account following this process:  https://aws.amazon.com/premiumsupport/knowledge-center/close-aws-account/--The account will be in suspend mode for 90 days before deletion.ShareTSquared 3 months agoThank you for the very thorough response to this!Share0And removing them is just a manual process (or could be scripted)?CommentShareTSquaredanswered 3 months agoRoguen 3 months agoIt could be manual, though it's just removing the stack instances from the StackSets, so could be scripted via CLI calls or other tooling.Share"


Question: "MGN for RDM, direct-attach disks(ISCSI,NBD)"

"I need to migrate multiple VMs with RDM and direct-attach disks(ISCSI, NBD). After reading, MGN agentless replication doesn't support migrating those VMs with independent disks, Raw Device Mappings (RDM), or direct-attach disks (iSCSI, NBD).Could the agent-based replication support migrating those VMs with RDM or direct-attach disks(ISCSI, NBD)?FollowComment"


Question: AppStream 2.0 with Database Users

"I have a working AppStream 2.0 instance, I am curious can the user login be linked to a SQL database of existing users for an Application?FollowComment"

Answer: "0There are three methods for authentication 1) User Pool, 2) SAML 2.0 IdP, and 3) Programmatic. You can build an authentication service that utilizes your existing set of services and then programmatically creates a streaming URL, then the answer is yes. I highly recommend reviewing ISV documentation for more information.CommentShareEXPERTStevieStetsanswered 7 months ago"


Question: When URL with www does not resolve

"I had set up my site url as sitedomain.com in route 53 with all the associated records.  The site resolved fine when accessed via sitedomain.com, but did not resolve when accessed via www.sitedomain.com.  In other words it did not resolve when adding the 'www' prefix to the URL.  I tried setting up an entirely separate domain in route 53 which included the 'www' but that was also not correct.The solution as provided was ->'In order to get www.yourdomain.com  working you need to create a CNAME record in your hosted zone pointing to yourdomain.com  record'.I'm sure there are google entries for this but I couldn't find any so thought I would share this in the hopes of helping anyone with a similar problem.FollowComment"

Answer: "0this is a resolved issue, just posting things to shareCommentShareK. Milleranswered 4 years ago"


Question: IoT Core OTA Job FAILED retry techniques

"Hi there IoT people,I am looking into the best practice approach to retrying an OTA update to failed devices.Scenario:"Let's say we ran an OTA update for 1000 devices, and 42 of them FAILED for some reason.We corrected the "reason" and now need to rerun the job for 42 devices only.So, I looked thru the API's and I got as far asAttach a lambda to a IoT Rule that filters for Completed events, as perhttps://docs.aws.amazon.com/iot/latest/developerguide/events-jobs.htmlhttps://docs.aws.amazon.com/iot/latest/developerguide/iot-lambda-rule.htmlThe next logical step is call ListJobExecutionsForJob, passing in the job Id, filtering for FAILED jobs only.But I can't see a way to marry up the Execution to a Thing, because you need the thing name to call DescribeJobExecution.Do you need to use Fleet Indexing, or is there a better way to programmatically get the list of failed Things for an OTA Job, so we can alert / report / build retry tooling?Thanks!FollowComment"

Answer: 0Accepted AnswerListJobExecutionsForJob returns the ThingArn which is arn:aws:iot:<region><account-id>:thing/<thing-name>. You can extract the ThingName from that.CommentShareAWS-User-1695858answered 3 years ago


Question: AWS Elemental MediaLive – HEVC/H265 Input not working

"Hi,I am trying to send an HEVC encoded SRT stream from OBS via AWS Elemental MediaConnect to AWS Elemental MediaLive. The pipeline works fine with an H264 encoded SRT stream but when I change the codec to HEVC, Elemental MediaLive doesn't process the stream. I changed the input secifications for the channel to HEVC. Why is it not working? How can I fix it?FollowCommentsaltarob 2 days agoUpdate: It suddenly worked two times but then it stopped working again. And now the same behaviour also shows up with H264 imputs. Looks like the actual problem is not the H265 ingest. It seems there is something wrong with the pipeline but I can't figure out what. I created the channel several times. I also tried the workflow and some templates. Any ideas how to narrow down that problem? How can I see and analyse some logs?Sharesaltarob a day agoThe logs show following failure message with H265 and H264:"Failed to set follow modes: Invalid input video frame rate: 0/1"Share"

Answer: "0Have you tried stopping the MediaLive channel, performing the change from H.264 to H.265 source, then starting the MediaLive Channel. Keeping the ML channel running when a change to the source coded occurs may cause decode pipeline issues.Also, check the MediaConnect output status and MediaLive Input stream status to determine that a signal is being delivered to MediaLive.CommentShareMike-MEanswered 11 days agosaltarob 11 days agoHi MIke. Thank you for your answer. As far as I know you can't change anything on the channel without stopping it. So yes I stopped and started the channel several times. MediaConnect shows the right source bitrate of 5Mbit/s. But the MediaLive Input shows 16 Mbit/s with the H.265 codec and 0.16 Mbit/s for the output. There is a Video Not Detected alert right at the beginning. But this alert also shows up at the beginning of the working H.264 stream.Share0One last thing and very important. Define everything. Avoid "same as source".Also avoid anything variable in your encoder. Constant frame rate, constant data rate.CommentShareBruce Wilkinsonanswered 11 days agosaltarob 11 days agoThank you so much Bruce. Yes I think that's the problem. I have only Apple devices and they offer only ABR as an option for the bitrate on their HEVC codec. Just made a test with Larix Broadcaster from an iPhone. Same result.Share0This is an interesting problem and I would like to help. I have done the same workflow using something other that OBS. I think it is an excellent to use HEVC SRT to MC to ML and exploit the benefit of HEVC for the first mile.Your experience/understanding or not it should be SOP for ML to be stopped and restarted for any significant input change.Believe it or not I have sent HEVC to ML with the input selected AVC and it still worked. Odd I know. Either way good you have done both.My experience with many encoders is that once you change to HEVC lots of other things change too. ML has limits on what Frame Rate it can accept as an example. I'd check in OBS once you switch to HEVC if all the other setting you were using with AVC are close to the same.My guess is this is more OBS than Medialive. You have those bases covered. I'm not a fan of Open Source encoders. DM me if you would like me to connect to your MC SRT input with a Hardware HEVC encoder. That way at least you will know where to look.CommentShareBruce Wilkinsonanswered 11 days ago"


Question: AWS Public Prices in EUR

"Hi,I need to get all AWS Public prices in EUR and not in USD.I used the Bulk Price API https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/using-ppslong.html but the prices are all in USD.Any way to get them in EUR ?Thanks for your helpFollowComment"

Answer: "0Hi FabienG,At the moment all billing reporting tools (Cost Explorer, Cost and Usage Report, Budgets and other billing reports), as well as Public Pricing bulk API only show prices and bills in USD, because the exchange rate will change on a daily basis. The final rate that's used to generate your bill is only displayed on your finalized AWS bill (invoice) that you would normally use for payment.I shared more info on this question here: https://repost.aws/questions/QUzwFKJB6iSuKNlGeBfKVtRQ/how-can-i-get-the-change-rate-between-usd-and-eur-used-by-awsI hope this info helps!CommentShareEXPERTNataliya_Ganswered a year ago-1Hi Fabien,As per the article, "Can I pay my AWS bill in a currency other than US dollars (USD)?" https://aws.amazon.com/premiumsupport/knowledge-center/non-us-currency/When using a non-USD currency, consider the following:All preferred currency amounts shown in the AWS Billing and Cost Management console are estimated in USD.Your finalized monthly bill is generated in the local currency you choose.Rates can change daily. The exchange rate applied to your invoice is the current rate when your invoice is created.Note: Currency conversion is provided by Amazon Services LLC.You can change your local currency at any time, but after an invoice is finalized, the invoice must be paid in that currency.You can pay invoices in non-USD currencies by using Visa or MasterCard. All other cards are charged only in USD.Note: If you choose a payment currency that the card issuer doesn't support, you might be charged foreign currency conversion charges or fees.-— Brian D.CommentShareEXPERTAWS Support - Briananswered a year agoFabienG a year agoHI Brain,Thanks for your answer,How can we get the Currency conversion rate provided by Amazon Services LLC.Is there any API to get this rate ? So I can convert the USD public prices to EUR.ThanksShare"


Question: Inconsitent Tone for Mathew with Neural Engine

"I find a weird issue with the second sentence while using Mathew in Neural voice. The entire Tone of Mathew changes when starting the second sentence starting "In the code...SSML Text below:<speak><amazon:domain name="conversational"><prosody rate="85%"><break time="1s" />
Process all gives the advantage in avoiding the mistakes that may occur due to accidental missing of any signal listing, renaming or any addition of new signal.

In the code on the slide, process all implies process of A, B, SEL as A , B and SEL are the inputs red by the process MUX.
<break time="1s" /></prosody></amazon:domain></speak>FollowComment"

Answer:
"-1Hi, in my opinion, the length of the sentence affects the final result. I tried to break it with some pauses (<s/>) instead of commas and the pronunciation style changes a little (see example 1)If you want a stronger result, you can switch from one style to another (see example 2)<speak><amazon:domain name="conversational"><prosody rate="85%"><break time="1s" />
Process all gives the advantage in avoiding the mistakes that may occur due to accidental missing of any signal listing, renaming or any addition of new signal.

In the code on the slide<s/> process all implies process of A, B, SEL as A <s/>B and SEL are the inputs red by the process MUX.
 <break time="1s" /></prosody></amazon:domain></speak><speak><break time="1s"/><amazon:domain name="conversational"><prosody rate="85%">
Process all gives the advantage in avoiding the mistakes that may occur due to accidental missing of any signal listing, renaming or any addition of new signal.</prosody></amazon:domain>
<amazon:domain name="news"><prosody rate="85%">In the code on the slide<s/> process all implies process of A, B, SEL as A , B <s/>and SEL are the inputs red by the process MUX.</prosody></amazon:domain><break time="1s"/></speak>CommentShareLBajardanswered a year agoAWS-User-7410164 a year agoIt didn't worked.Share"


Question: Instance upgraded from t2 to t3 is not booting up

"Hi,The instance in question is t3.medium instance in eu-west-1b region.The instance has been upgraded from T2 to T3 a couple weeks ago.It is set to start and stop daily but in the last 2 weeks - it happened the 2nd time already it is not able to boot.This problem is not daily but I would say weekly.The instance was fine yesterday but today it got stuck.It passes the network check but instance status check fails, it is showing as Running but I cannot remote to it.It's a RHEL8.4 instance created from AWS AMI RHEL_HA-8.4.0_HVM-20210504-x86_64-2-Hourly2-GP2 (ami-0d81e223398b63904)It must be a rather random issue because as I said the instance was starting and stopping fine in the previous days but from time to time a glitch happens.Is this something to do with ENA or NVME or both?Please adviseFollowCommentAWS-User-4707325 a year agoaws ec2 describe-instances --instance-ids i-xxxxxxx --query "Reservations[].Instances[].EnaSupport"comes TRUEShare"

Answer: "0Hi,Based on the fact that t2 and t3 instance types use different virtualisation (t2 uses Xen, t3 uses Nitro) I suspect the issue may be the ENA driver.If you changed the instance type to an instance built on the Nitro System, status checks fail if you migrated from an instance that does not have the required ENA and NVMe drivers. For more information, see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/resize-limitations.html. However, you mentioned that is was working fine until in the last 2 weeks but it is not the problem that you encounter daily.For more information on the ENA driver on Red Hat, see [this](How do I install and activate the latest ENA driver for enhanced network support on an Amazon EC2 instance running Red Hat 6/7?) on how to install and activate the latest ENA driver.The following documents will help you to troubleshoot the instance status check fails. [1]https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-status-check-failure-os-errors/ [2] https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-status-check-failure/I hope the above information is helpful. Kindly let me know if you have additional concerns.Thank you.CommentSharePhindianswered a year ago"


Question: Feedback forwarding/bounces via SMTP issue

"Hi,I verified a domain (example.com) for use with SES. Software we're using needs to receive bounces via SMTP to automatically handle bounces. As a test I wrote an email from myaddress@example.com to bounce@simulator.amazonses.com. The expected bounce did not reach my inbox at Office 365. (Checked junk folder, too.) Note: myaddress@example.com had been used as envelope from address (MAIL FROM).Next try was setting MAIL FROM to myaddress@example.com and manually setting the header "Return-Path: myaddress@example.com" but no bounce has been received neither.I also checked in the O365 Exchange admin center if the bounce had been rejected but there's no sign of a bounce message.So how can I receive bounces via SMTP?Thank youFollowComment"

Answer: "0Finally got a reply from AWS support. Office365 blocks emails from SES at the moment:550 5.7.606 Access denied, banned sending IP [69.169.224.240]. To request removal from this list please visit https://sender.office.com/ and follow the directions. For more information please go to  http://go.microsoft.com/fwlink/?LinkID=526655 AS(1430) [VE1EUR01FT026.eop-EUR01.prod.protection.outlook.com]CommentSharebengelsanswered 3 years ago"


Question: Custom entity recognition supports a maximum of 120000 samples.

"Hello,When trying to create a custom entity recognizer I get the following message when attempting to train:Custom entity recognition supports a maximum of 120000 samples.The "samples" call out is vague as the data points of interest are documents (3800) and annotations (35539) neither of which are close to 120000.  I had prior errors that I was able to fix so I know it's looking at the correct S3 locations.Any direction about how to resolve this error would be appreciated, thanks!FollowComment"

Answer: "0Hi,The number of samples include both positive and negative samples.For example, consider a document 'doc1.txt':Bob lives in Seattle, Washington.George Washington (February 22, 1732 – December 14, 1799) was an American political leader, military general, statesman, and Founding Father who served as the first president of the United States (1789–1797).Corresponding annotation:File, Line, Begin Offset, End Offset, Typedoc1.txt, 0, 22, 32, STATESo, in the above example: we have 2 samples, as opposed to 1 sample.Comprehend considers all sentences in the document which are not annotated as negative samples, which helps to improve the model precision.In the above example, "Washington" in the first sentence referes to "STATE" while second sentence refers to "PERSON".So, in your training corpus, there are 35539 positive samples which are annotated. All other sentences are taken as negative samples from 3800 documents, which is exceeding our limit 120,000.Please remove negative samples which is not helpful as training data.Please also note, Comprehend reads all the input files present in the input-data-config, including sub directory as training data.example:aws comprehend start-entities-detection-job \     --entity-recognizer-arn "entity recognizer arn" \     --job-name job name \     --data-access-role-arn "data access role arn" \     --language-code en \     --input-data-config "S3Uri=s3://my_training_data/" \     --output-data-config "S3Uri=s3://my_model_output/" \     --region regionaws s3 ls s3://my_training_data2013-07-11 17:08:50 train.txt2013-07-24 14:55:44 folder2So, my_training_data has subdirectory called "folder2" which may or may not contain training data.All the files under folder2 will be considered as part of training data along with train.txtPlease let us know if you have more questions.CommentShareAWS-User-3165305answered 4 years ago0Thanks that explains the error!CommentSharebackwardsEanswered 4 years ago0It would be so much better if the recognizer just stopped processing more sample at 120 000 without crashing the job. It's really hard to remove all negative samples on production data.CommentShareCedanswered 4 years ago0Hi,We did consider this option and decided it is best to leave the user, who is more familiar with data for discarding the negative samples than us selecting the negative samples. We are working on downsampling the negative samples and keep you posted once this improvement is added.Thanks,Ravindra MCommentShareAWS-User-3165305answered 4 years ago"


Question: Is it possible to check video on corruption witouth encoding?

"I am using AWS Elastic Transcoder to encode videos from different formats to mp4. Also I have videos with mp4 extension as input at my application. I need to check all the videos on corruption before displaying them for user. Transcoder does it automatically for encoding jobs. I don't want to encode mp4 files and want to leave them as is, but I want to check them on corruption.Is it achivable by AWS Elastic Transcoder or any other AWS service?FollowComment"

Answer: "0Depending on what type of corruption you wish to check for, there are different tools that can be used.MediaInfo, an industry popular application, can provide metadata details on resolution, framerate, bitrate and other generic parameters. There are other 3rd party tools that can validate the actual video and audio within a file. These can be run manually or triggered automatically, yet take longer to run.Using Lambda functions, an file upload to an S3 bucket can trigger a Lambda function (or Step Function - a sequence of Lambdas) that calls MediaInfo to determine if/how to transcode the file.Perhaps you could consider using MediaConvert instead of Elastic Transcode. MediaConvert is more powerful, provides better video quality and is less expensive. The above mentioned Lambda functions could be used to call one of several pre-defined MediaConvert Transcode Templates, based on metadata details that were discovered.CommentShareMike-MEanswered a year ago0There's are quality control tools like this one you can buy.  https://www.interrasystems.com/file-based-qc.phpCommentSharerePost-User-6242190answered 10 months ago0here two more QC tools:Aurora and Vidchecker from Telestream - https://www.telestream.net/video/aurora.htmPulsar from Venera Technologies -  https://www.veneratech.com/pulsar-automated-file-qc/CommentSharerePost-User-6242190answered 10 months ago"


Question: Make single AWS XRay Trace Id by propagating it end to end

"I have use-case where the Producer Java lambda publishes file messages to SQS and Consumer Nodejs consumes them. Producer side of X-ray is enabled in serverless.yml; displays a Service map with all AWS components with a new Trace Id. The Consumer side of Nodejs lambda; is enabled with another Trace Id. I see a single service map, but with two clients. I guess it is because of the 2 separate Trace Ids. Is it possible to make single Trace Id by propagating it end to end?FollowComment"

Answer: "1Hi,it is not possible at the moment. If you are using SQS than Lambda can get more than one Message per invocation. Each Messages has it´s own TracId associated and so it is not possible to assign on single traceId to the receiver Lambda.CommentShareMarcoanswered a year ago0Thanks for asking this question. Unfortunately this is not yet supported. Please see this open issue on the X-Ray Node JS SDK which explains why this is a difficult problem to solve:https://github.com/aws/aws-xray-sdk-node/issues/208CommentShareaws-nathanielanswered a year ago0The Producer side sends the messages to SQS using sqs.sendMessageBatch(new SendMessageBatchRequest(queueUrl)). The SQS can be seen in the Service graph, with its Trace Id.The Consumer side is consuming messages from SQS. Yet the service graph does not show SQS with its Trace Id.Is there a way/workaround to show SQS on the Consumer side of Trace?CommentShareAshrufanswered a year ago"


Question: Amazon CloudWatch Metric ModelSetupTime not available

The ModelSetupTime metric  is not available for monitoring the serverless endpoint(https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html) in Amazon CloudWatch.Able to see only the below mentioned metricsInvocationsInvocation5XXErrorsInvocation4XXErrorsModelLatencyOverheadLatencyIs there any role/configuration changes I have to do to view the ModelSetupTime ?FollowComment

Answer: 0Currently the metric is not published to CW.CommentShareAWS-Raghuanswered 7 months ago


Question: Bare Metal instances with KVM

"My customer is trying to set up a customized version of KVM that allows the virtualization layer to monitor in depth the activities performed on the VMs hosted. The VMs have a specialized kernel module that KVM can use to collect the data. Their assumption (and mine) is that to run this, they would need to run on bare metal instances. They are asking two questions that I am not sure of.Can they create an AMI of a bare metal instance and use it to create new bare metal instances?Is there any limit on the number of bare metal instances they can procure? Are there any availability limits or issues issues with on-demand provisioning of bare-metal instances?Can customers launch bare metal instances on demand?FollowComment"

Answer: 0Accepted AnswerYes .metal instances expose Intel VT-x to the OS which the customer can use to run KVMYes AMIs can be launched on .metal instances.Yes .metal instances are available for provisioning on-demand.CommentShareEXPERTJDBanswered 2 years ago


Question: "Something went wrong" Error when Attempting to Navigate to a DMS S3 Target Endpoint

"During a recent release of DMS, Links to the details of Target Endpoints of Type S3 stopped working.  The S3 buckets that are associated with the endpoints can be accessed via the S3 console and the Database migration tasks associated run without error. Also, all other endpoints work, even Source Endpoints of type S3. It is just the S3 Target endpoints that have this issue.Here is the error message that is displayed when a link to the endpoint detail is clicked.EndpointsNOTE: After calling the cli command dms:describe-endpoints and comparing the output with the output from a test environment, there are no difference in the endpoint configurations.However, the same error is now occurring in the test environment which has not been touched since January 26, at which point the error above was not present in both environments.Below is the dsm:describe-endpoint output from a S3 Target Endpoint that is failing in the console.{"EndpointIdentifier": "aaaa-bbbbbbbb-data-dev-blue-dms-target-endpoint-bronze-s3-cccccccccccccc","EndpointType": "TARGET","EngineName": "s3","EngineDisplayName": "Amazon S3","ExtraConnectionAttributes": "bucketFolder=CCCCCCCCCCCCCCC;bucketName=aaaa-us-east-1-bbbbbbbb-blue-data-bronze-dev;CdcPath=cdc-files;compressionType=NONE;csvDelimiter=,;csvRowDelimiter=\\n;datePartitionEnabled=false;","Status": "active","EndpointArn": "arn:aws:dms:us-east-1:111111111111:endpoint:VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV","SslMode": "none","ServiceAccessRoleArn": "arn:aws:iam::111111111111:role/aaaa-bbbbbbbb-data-useast1-dev-blue-dms-s3-role","S3Settings": {"ServiceAccessRoleArn": "arn:aws:iam::111111111111:role/aaaa-bbbbbbbb-data-useast1-dev-blue-dms-s3-role","CsvRowDelimiter": "\\n","CsvDelimiter": ",","BucketFolder": "CCCCCCCCCCCCCCC","BucketName": "aaaa-us-east-1-bbbbbbbb-blue-data-bronze-dev","CompressionType": "NONE","EnableStatistics": true,"IncludeOpForFullLoad": false,"CdcInsertsOnly": false,"ParquetTimestampInMillisecond": false,"CdcInsertsAndUpdates": false,"DatePartitionEnabled": false,"UseCsvNoSupValue": false,"PreserveTransactions": true,"CdcPath": "cdc-files","UseTaskStartTimeForFullLoadTimestamp": false,"AddColumnName": true,"CsvNullValue": "''","IgnoreHeaderRows": 1,"MaxFileSize": 50000,"Rfc4180": true,"AddTrailingPaddingCharacter": false}}FollowComment"

Answer: "1Accepted AnswerHi,Sorry that you've been impacted by this. Our team has identified the issue while accessing DMS S3 target endpoint via AWS Console and a fix for the same is actively being worked on.While this is being fixed, as a workaround, you can use the AWS CLI/API commands to describe the endpoint and get the details of the endpoints.> aws dms describe-endpoints --filter="Name=endpoint-id,Values=s3endpointname"[+] https://docs.aws.amazon.com/cli/latest/reference/dms/describe-endpoints.htmlCommentShareSUPPORT ENGINEERdeepamanswered 3 months ago0Thanks for the quick reply.CommentShareRoss Bushanswered 3 months agodeepam SUPPORT ENGINEER2 months agoThis issue has now been resolved.Share"


Question: Amazon Connect reports not showing date range on export

"It appears that some of the amazon connect report were changed.  I have a daily agent report that I run on a schedule and we save a copy of that report.  The problem the report now does not have a start interval and end interval field.  The only way to get an idea of the date is by the file name, but that is not ideal when we combine multiple extracted reports into a single dashboard.  Is anyone else seeing this issue, or is there an option to include the date range when the report is saved to the S3 bucketFollowComment"

Answer: "0I just checked mine and it still exports the startInterval and endinterval what settings do you have on the report?  And if you make a new one picking something like Interval to be "Daily".  And try to download csv from Action, do you see the start/end interval?CommentShareClarenceChoianswered 3 months ago"


Question: Forwarding log messages to log file using step function

"Hi there,We are migrating from datapipeline to step function, and on datapipeline I am able to forward the log messages for individual steps to a log file, but this doesn't work on step function. Am I missing something in the configuration?On datapipeline I have/home/hadoop/general/ProfileData.sh $1  > >(tee -a /home/hadoop/myLogFile.log) -- WorkedOn step function I have"Args.$": "States.Array('bash', '/home/hadoop/general/ProfileData.sh', $.argument1, ' > >(tee -a /home/hadoop/myLogFile.log)' " --Not worksFollowComment"

Answer: "1What integration are you calling from Step Functions?Usually, in distributed systems like this, it is best to send the logs outside the host that generated them because you may not have access to it later. In AWS, we send logs to CloudWatch Logs.You can then create a CloudWatch Logs subscription that will send the logs to a file in S3.CommentShareEXPERTUrianswered 20 days agoJustin Callison 20 days agoYou can find documentation to enable CloudWatch Logs for Step Functions below. When you enable this, each History Event for your workflow executions will be logged to the log group you have specified.https://docs.aws.amazon.com/step-functions/latest/dg/cw-logs.htmlShare"


Question: Enable MFA on AWS Workspaces

"How do  i enable MFA on AWS workspaces, i see Radius server is required , do we have any step by step process to setup Radius server and enable MFA on Workspaces. mostly users commented Radius server wont work for MFA on AWS workspace , so do we have any radius server on AWS market place that integrate with Workspaces and implement MFA.OR any working document that has covered all details , Thanks in advance and highly appreciate if some one can help with verified document for  MFA on AWS workspacesFollowComment"

Answer: 1Enabling RADIUS on the Directory Service (AWS Managed Microsoft AD) first is key: https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ms_ad_mfa.html.Here is a blog that has a detailed walkthrough of the process for Workspaces.CommentShareRajarshi_Danswered a year ago


Question: CodePipeline file generation

"So I have a react with SSR project that I am trying to create a CI/CD pipeline. My project needs to deploy the following artifacts:dist folder created by Webpackappspec.yml file needed by Code Deployscript folder for scripts that are references in appspec.ymlWhen I tried to get the files one by one using this buildspec.yml:version: 0.2phases:install:runtime-versions:nodejs: 10commands:- npm installbuild:commands:- npm run buildartifacts:files:- 'dist/*'- 'appspec.yml'- 'deploy-scripts'I got dist with only part of it's content and the appspec.yml and the deploy scripts folder.Then I tried a different approach:version: 0.2phases:install:runtime-versions:nodejs: 10commands:- npm installbuild:commands:- npm run buildartifacts:files:- '**/*'base-directory: "dist"discard-paths: yesThe dist folder has the scripts and the appspec file inside it. Now it does deploy, but I lose the folder structure of dist which is necessary for my deploy scripts.I need to get all the contents of dist in their folder structure. As well as the scripts and the appspec.yml file. The scripts and the appspec.yml can't be inside dist, but dist needs to have all of it's content.Can anyone help with this?FollowComment"

Answer: "0Hi,Have you tried the following?- 'dist/**/*'CommentShareRandyTakeshitaanswered 4 years ago0The suggestion did the trick. Thanks!CommentSharecjimenezberanswered 4 years ago"


Question: Billing Because of RDS Backup Services

"I'm newbie to AWS. When I looked at my Bill today, the RDS Backup services were the cause of the rise in the bill.There isn't a database,nor snapshots or anything operating at the RDS Dashboard, only the standard mysql:80.How can I terminate the RDS Backup Services?FollowComment"

Answer: "0RDS Backup services can be disabled by going into the RDS console and selecting Modify. Then identifying the location where automated backups are enabled. Change the setting to 0 and this will disable the automated backups.To disable automated backups immediatelySign in to the AWS Management Console and open the Amazon RDS console at https://console.aws.amazon.com/rds/.In the navigation pane, choose Databases, and then choose the DB instance or Multi-AZ DB cluster that you want to modify.Choose Modify.For Backup retention period, choose 0 days.Choose Continue.Choose Apply immediately.Choose Modify DB instance or Modify cluster to save your changes and disable automated backups.Detailed link provided below:https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.htmlCommentSharerePost-User-venkyanswered 2 months ago"


Question: Index list of bucket

I want to get a bucket/directory///name of fileListing to search and find content easily. Can I generate a directory/index? I am not a programmer just trying to document what is in s3 buckets.FollowComment

Answer: "0You can Filtering and retrieving data using Amazon S3 Select https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html or Analyzing Data in S3 using Amazon Athena https://aws.amazon.com/blogs/big-data/analyzing-data-in-s3-using-amazon-athena/ easily.Please let me know if there is anything else you are looking and we can guide.CommentShareEXPERTAWS-User-Nitinanswered 2 months agoEXPERTkentradreviewed 2 months ago0https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.htmluse aws cli >List buckets and objectsTo list your buckets, folders, or objects, use the s3 ls command. Using the command without a target or options lists all buckets.Syntax$ aws s3 ls <target> [--options]CommentShareHemant Chughanswered 2 months ago"