Question: "creating kernel" step took more than 36 hours in hardware system build

"Hello!I'm doing a hardware system build using Vitis on z1d.2xlarge instance. The goal is to accelerate an algorithm with the f1.2xlarge instance. I am using the latest FPGA AMI for this instance.My application has one kernel in which the core computation-intensive part is two completely unrolled matrix-vector multiplication, each one looks like the following:#define MAX_ROWS #a number##define MAX_COLUMNS #a number#typedef ap_fixed<2, 2, AP_TRN, AP_WRAP> fixed_point_2t;typedef ap_fixed<18, 6, AP_TRN, AP_WRAP> fixed_point_16t;typedef ap_fixed<32, 16, AP_TRN, AP_WRAP> fixed_point_32t;fixed_point_2t matrix[MAX_ROWS][MAX_COLUMNS] = {0};fixed_point_32t vector[MAX_COLUMNS] = {0};fixed_point_32t product[MAX_ROWS] = {0};for (int i =0; i < NUM_COLUMNS; i ++){#pragma HLS PIPELINE    for (int j = 0; j < MAX_ROWS; j++)    {        product[j] += fixed_point_16t(matrix[j][i]) * fixed_point_16t(vector[i]);    }}NUM_COLUMNS is a variable in the kernel function, and MAX_ROWS is a hardcoded constant. The goal here is to have the kernel support matrix-vector multiplication for N_ROWS, N_COLUMNS < MAX_ROWS, MAX_COLUMNS. All the arrays are partitioned to support the corresponding unrolling. I omitted the memory transferring part from the above code.Currently, I am compiling a version for MAX_ROWS = 2000 and MAX_COLUMNS = 2000, it has stuck on the "creating kernel" step for 36 hours now. I can see that it is always using 100% of one CPU core, and the memory (RAM) usage on the z1d instance slowly increases over time.With some back-of-envelope calculations, I am pretty confident that the resources (DSPs, BRAMs) on the FPGA are able to support what the application requires. However, this long building time has made me very worried.Here are my questions:Is this not the correct way to do large size matrix-vector multiplication (for N >= 2000)? If so, what could be a better way to do it?Is there any place where I can check if the synthesis progress is stuck? It is taking a fairly long time now and I am not sure if it is ever going to finish.I can share my source code if needed, but I would prefer not posting it on the forum (sorry about that). Thank you for reading through my post, and I am open to any opinion/help on this issue. Please let me know if any part of my question is not clear, and any suggestion is greatly appreciated!Thank you,OwenFollowComment"

Answer: "0Hi,Looks like either your design is running into congestion issues, or you are running out of memory on host machine. Does the build run successfully if you reduce the number of multiplications in your design?Alternatively, you can try running builds on instances with larger memory.Vivado tool outputs vivado.log to check the progress of builds.Thanks!ChakraCommentShareawschakraanswered 3 years ago0Hi Chakra,Thank you very much for your reply!1. Possibility of running out of memory on host machineI've looked at the the RAM usage on the host machine. It uses at most 24.7 GB RAM with total of 60 GB RAM available on the host machine. Therefore, I suppose memory is not a critical issue here?2. Does  the build run successfully if you reduce the number of multiplications in your design?Yes, the build does run successfully if the number if multiplications is reduced. Please see  https://forums.xilinx.com/t5/Vitis-Acceleration-SDAccel-SDSoC/Vitis-hardware-emulation-cannot-finish-scheduling/td-p/1131555 for my vivado hls reports for matrix size of 100 and 1000. When compiling with matrix size of 1000, seems like vivado hls is stuck on scheduling. Do you have any insights on why this happens? Any suggestions you have is greatly appreciated!According to my post in Xilinx (the link above), seems like they also recommend me to use the latest release of Vitis, which is not officially supported by AWS. I am wondering if that is worth trying?Thank you,OwenCommentShareOwenooolanswered 3 years ago0Hello,Thanks for the feedback. Please stay tuned for availability of the latest Vitis version for F1.-ChakraCommentShareawschakraanswered 3 years ago0Turns out the problem is due to a large array partition factor (500). If any is caught into the problem as I did please try a smaller partition factor.CommentShareOwenooolanswered 3 years ago"


Question: EKS Node with 2 network interface

"Is it possible to set up a node with 2 network interface? if it is how?When I try to configure node group with launch template witch is configure to use 2 network interface, I can't select the template version because it is grayed out and it shows "the launch template version can only have a max of one network interface"FollowComment"

Answer: "1It is possible to have a node that has 2 ENIs without having to manually configure it in Launch templates. Amazon EC2 provides a wide selection of instance types for worker nodes. Each instance type offers its own maximum number of ENI. For example, t3.micro has 2 maximum network interfaces.Refer to doc for a table that lists the maximum number of network interfaces per instance type.CommentShareOlawale-AWSanswered 4 months agoNenad 4 months agoI see that document , my main question is how to setup node to have 2 network interface in initial setup , if I use create node group wizard you choose instance witch support 2 and more interface , but you can't  select number of interface for deploy of those nodes.ShareBrad Werner 4 months agoNenad, if you are using the Amazon VPC CNI then it will proactively manage the ENIs actually used. With the same numbers as the doc Olawale noted, this file https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt describes the maximum number of pods per node. What is your use case for wanting to manage the number of ENIs yourself instead of letting the aws-node agent (DaemonSet pod for Amazon VPC CNI) manage them for you?You could alternatively use self-managed nodes if that is what you really need/want.ShareNenad 4 months agoHi Brad ,thx for answers. The thing is a app witch we plan to deploy on EKS need to received traffic A to one NIC and traffic B on another NIC (this is a requirement) , trust me it would be easier for me to use EKS CNI and setup two NICs on a pod. It is possibility that I will need to deploy the app with hostNetwork: true so app pickup host network.Share"


Question: There's no any available versions when trying to restore spanshot of aurora serverless v1 mysql-compatible

"As title, my snapshot is made from 5.6.I know the support of the 5.6 one is ended, however, I can't see any higher version to restore it.FollowComment"


Question: Manage FSx Storage Gateway Cache

We are thinking about moving to FSx FileStorage+Gateway (on-prem/ESXi).What I would like to know: Is it possible to manage the cache?Looking at the docs all i see is the ability to set the refresh interval/set auto vs manual. I would like to be able to define certain items/folders that are ALWAYS available locally on the cache server.FollowComment

Answer: "3Accepted AnswerYes, For FSx type storage gateway you can set refresh cache based on TTL but  you can't run manually that is only for S3 file type storage gateway.Coming to your 2nd question, it is not possible to pin specific folders or files to cache. Storage gateway has built-in intelligence algorithm which manages what files to store in or evict from cache. However, all Files metadata only is always stored within storage gateway root disk.CommentShareRRP_AWSanswered 7 months agoEXPERTAWS-User-Nitinreviewed 7 months agoEXPERTTushar_Jreviewed 7 months agojeffp2016 7 months agoThank you!Share"


Question: Is it possible for Jobs in the same EMR Serverless Application to share workers?

"I understand there are application limits for EMR serverless. But when there are multiple jobs running at the same time on the same application, is it possible for them to share workers' available capacity? OR if one job is assigned a work they are assigned the entire worker capacity?FollowComment"

Answer: 1Accepted AnswerEMR Serverless workers are dedicated to jobs and not shared among jobs even if jobs are running under same serverless application.CommentShareKashifanswered 4 months ago


Question: S3 bucket creation with encryption is failing because of AWSSamples::S3BucketEncrypt::Hook

"Hi,I have activated AWSSamples::S3BucketEncrypt::Hook with the following configuration but S3 bucket creation with encryption enabled seems to be failing because of the hook. It works when I disable the hook. Could this be an issue?{  "CloudFormationConfiguration": {    "HookConfiguration": {      "TargetStacks": "ALL",      "FailureMode": "FAIL",      "Properties": {        "minBuckets": "1",        "encryptionAlgorithm": "AES256"      }    }  }}{  "CloudFormationConfiguration": {    "HookConfiguration": {      "TargetStacks": "ALL",      "FailureMode": "FAIL",      "Properties": {        "minBuckets": "1",        "encryptionAlgorithm": "aws:kms"      }    }  }}AWSSamples::S3BucketEncrypt::Hook configurationAWSSamples::S3BucketEncrypt::HookCloudFormation for S3 bucket with AES256 encryption - Expected to PassAWSTemplateFormatVersion: 2010-09-09Description: S3 bucket with default encryptionResources:  EncryptedS3Bucket:    Type: 'AWS::S3::Bucket'    Properties:      BucketName: !Sub 'encryptedbucket-${AWS::Region}-${AWS::AccountId}'      BucketEncryption:        ServerSideEncryptionConfiguration:          - ServerSideEncryptionByDefault:              SSEAlgorithm: 'AES256'    DeletionPolicy: DeleteCloudFormation for S3 bucket with KMS encryption - Expected to PassAWSTemplateFormatVersion: "2010-09-09"Description: This CloudFormation template provisions an encrypted S3 BucketResources:  EncryptedS3Bucket:    Type: 'AWS::S3::Bucket'    Properties:      BucketName: !Sub 'encryptedbucket-${AWS::Region}-${AWS::AccountId}'      BucketEncryption:        ServerSideEncryptionConfiguration:          - ServerSideEncryptionByDefault:              SSEAlgorithm: 'aws:kms'              KMSMasterKeyID: !Ref EncryptionKey            BucketKeyEnabled: true      Tags:         - Key: "keyname1"          Value: "value1"  EncryptionKey:      Type: AWS::KMS::Key    Properties:     Description: KMS key used to encrypt the resource type artifacts     EnableKeyRotation: true     KeyPolicy:      Version: "2012-10-17"      Statement:      - Sid: Enable full access for owning account        Effect: Allow        Principal:           AWS: !Ref "AWS::AccountId"        Action: kms:*        Resource: "*"Outputs:  EncryptedBucketName:    Value: !Ref EncryptedS3BucketFollowComment"

Answer: "0Accepted AnswerThere were multiple issues when I attempted this. Here are the list of things to do in order to resolve.The role need to be created using https://github.com/aws-cloudformation/aws-cloudformation-samples/blob/main/hooks/python-hooks/s3-bucket-encryption/hook-role.yamlCF Hook Configuration need to be the followingKMS{  "CloudFormationConfiguration": {    "HookConfiguration": {      "TargetStacks": "ALL",      "FailureMode": "FAIL",      "Properties": {        "minBuckets": "1",        "encryptionAlgorithm": "aws:kms"      }    }  }}3.1 Use the following CF to create SSE-S3 bucket for point no 3AWSTemplateFormatVersion: "2010-09-09"Description: This CloudFormation template provisions an encrypted S3 BucketResources:  EncryptedS3Bucket:    Type: 'AWS::S3::Bucket'    Properties:      BucketName: !Sub 'encryptedbucket-${AWS::Region}-${AWS::AccountId}'      BucketEncryption:        ServerSideEncryptionConfiguration:          - ServerSideEncryptionByDefault:              SSEAlgorithm: 'aws:kms'              KMSMasterKeyID: !Ref EncryptionKey            BucketKeyEnabled: true      Tags:         - Key: "keyname1"          Value: "value1"  EncryptionKey:      Type: AWS::KMS::Key    Properties:     Description: KMS key used to encrypt the resource type artifacts     EnableKeyRotation: true     KeyPolicy:      Version: "2012-10-17"      Statement:      - Sid: Enable full access for owning account        Effect: Allow        Principal:           AWS: !Ref "AWS::AccountId"        Action: kms:*        Resource: "*"Outputs:  EncryptedBucketName:    Value: !Ref EncryptedS3BucketORSSE-S3AWSTemplateFormatVersion: 2010-09-09Description: S3 bucket with default encryptionResources: EncryptedS3Bucket:   Type: 'AWS::S3::Bucket'   Properties:     BucketName: !Sub 'aesencryptedbucket-${AWS::Region}-${AWS::AccountId}'     BucketEncryption:       ServerSideEncryptionConfiguration:         - ServerSideEncryptionByDefault:             SSEAlgorithm: AES256   DeletionPolicy: Delete4.1 Use the following CF to create SSE-S3 bucket for point no 4AWSTemplateFormatVersion: "2010-09-09"Description: This CloudFormation template provisions an encrypted S3 BucketResources:  EncryptedS3Bucket:    Type: 'AWS::S3::Bucket'    Properties:      BucketName: !Sub 'encryptedbucket-${AWS::Region}-${AWS::AccountId}'      BucketEncryption:        ServerSideEncryptionConfiguration:          - ServerSideEncryptionByDefault:              SSEAlgorithm: 'AES256'            BucketKeyEnabled: true      Tags:         - Key: "keyname1"          Value: "value1"CommentShareSrianswered a year ago"


Question: AWS DataPipeline Is Failed for EMrClusterBackup with Error  Resource Not Healthy:- JobFlow retired

AWS DataPipeline Is Failed for EMrClusterBackup with Error  Resource Not Healthy:- JobFlow retiredFollowComment

Answer: "1For this, you will need to inspect your EMR logs and will need some knowledge on how logging works on EMR: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-manage-view-web-log-files.htmlI'm wondering why you are using DataPipeline? Are you exporting to CSV to use with some other application? Or are you trying to copy DynamoDB tables cross-account/region?CommentShareEXPERTLeeroy Hannigananswered a year ago"


Question: Policy IAM user Appstream2.0

"Is it possible to give access only to a certain image, stack, or fleet with IAM policies?Do you have any examples?I tried with a policy but it returns this error:User: arn:aws:iam::xxxxxxxxx:user/xxxxxxxx is not authorized to perform: appstream:DescribeFleets on resource: arn:aws:appstream:eu-central-1:xxxxxxxxxxx:fleet/* because no boundary policy allows the appstream:DescribeFleets actionMy need is:in an AWS account, an IAM user must only see some image/fleet/stack.thanksFollowComment"

Answer: "1The AppStream 2.0 console doesn't currently support restricting what a user can see based on tags or other resource boundary. For example, the Stacks page calls the AppStream 2.0 API "DescribeStacks" without specifying any boundaries or tags. You can use tags to prevent a user from updating an AppStream 2.0 resource without specify a specific tag - for example, if a user/role should only be able to modify resources with a tag key of "Stage" and value of "NonProd", you can specify that as a condition.CommentShareEXPERTMuraliAtAWSanswered a year agoAWS-User-6631477 a year agothanks for your answer MuraliAtAWS.So a policy with these controls (see example) is not possible?"Resource": ["arn:aws:appstream:eu-central-1:123.....:image-builder/imagebuilder01""arn:aws:appstream:eu-central-1:123.....:stack/stack01","arn:aws:appstream:eu-central-1:123.....:app-block/","arn:aws:appstream:eu-central-1:123.....:fleet/fleet1","arn:aws:appstream:eu-central-1:123.....:application/","arn:aws:appstream:eu-central-1:123.....:image/image01"],"Condition": {"StringEqualsIfExists": {"aws:username": "user01"}}Share"


Question: MFA setup and login for new iOS device

I have MFA setup on my AWS account and use Google authenticator. I recently got a new iphone and had to re-install Google authenticator but the AWS MFA setup is not on there. How do I re-add AWS on Google authenticator? All the documentation I've found tells me to login to the AWS console first but I can't do that with MFA. If I had a QR code or setup key for AWS I can do it. ThanksFollowComment

Answer: "0If this is the root account, have you tried to follow our guide on how to recover it? We also have a blog post that cover the details of recovering root account.CommentShareEXPERTToni_Sanswered a year ago"


Question: Error on browser to open "redis create cluster"

When opening the "redis create cluster" (https://sa-east-1.console.aws.amazon.com/elasticache/home?region=sa-east-1#create-cluster:ct=redis:) I get the following error:elasticacheconsole-0.js:11058 Mon Mar 28 11:42:08 GMT-300 2022 SEVERE: Uncaught exceptioncom.google.gwt.core.client.JavaScriptException: (TypeError) : Cannot read properties of undefined (reading 'e')at Unknown.TypeError: Cannot read properties of undefined(Unknown)at Unknown.n$b(elasticacheconsole-0.js)at Unknown.MS(elasticacheconsole-0.js)at Unknown.sY(elasticacheconsole-0.js)at Unknown.dkh(elasticacheconsole-0.js)at Unknown.zY(elasticacheconsole-0.js)at Unknown.dTg(elasticacheconsole-0.js)at Unknown.QSg(elasticacheconsole-0.js)at Unknown.TSg(elasticacheconsole-0.js)at Unknown.Dmh(elasticacheconsole-0.js)at Unknown.kYl(elasticacheconsole-0.js)at Unknown.Pmh(elasticacheconsole-0.js)at Unknown.orj(elasticacheconsole-0.js)at Unknown.Erj(elasticacheconsole-0.js)at Unknown.eval(elasticacheconsole-0.js)at Unknown.Ggj(elasticacheconsole-0.js)at Unknown.Jgj(elasticacheconsole-0.js)at Unknown.eval(elasticacheconsole-0.js)FollowComment

Answer: "0Hello,I have been trying to reproduce the issue with no luck. For me, the URL provided successfully opens the "Create your Amazon ElastiCache cluster" page, with no errors on the screen or Java Script console.If the error persists for you, please try the following tests:Try accessing the same URL from a different browser and/or different device.Try acessing https://sa-east-1.console.aws.amazon.com/elasticache/home?region=sa-east-1#redis: and then click on the "Create" button.Thanks and let us know if you still have any trouble.CommentShareSUPPORT ENGINEERTulio_Manswered a year ago"


Question: AWS RDS Aurora latency baseline

"Hi,I'm looking to some informations about common latency between EC2 instances and AWS RDS Aurora cluster.For reference, I have multiple instances running on the same VPC than a cluster and from what we see (tcptraceroute on both writer and reader endpoints) cluster latency is about ~100/200ms if on the same AZ (with some spike at 450ms ... is this normal ?).Results can raised to +800ms if not on the same AZ.It seems pretty slow to me. What can explain this ? do you have some latency baseline or maybe best practices to share to limit this results.RegardsFollowCommentDaniel Craigie a year ago"multiple instances running on the same VPC than a cluster" - sorry, but I'm a little confused by your description, are your Instances running in the same VPC as the RDS cluster?  Or are you routing DB traffic between two VPCs?Share"

Answer: "0Hello, In the documentation it says that the latency between different AZ is 1 to 2 millisecondshttps://docs.aws.amazon.com/sap/latest/general/arch-guide-architecture-guidelines-and-decisions.html (See Latency across Availability Zones)There's a caveat. If you're only using the low-end instances like t2, you will likely have a lot of jitter and can sometimes encounter more latency.If the instances are ENA (or SR-IOV) compatible, and using it, you'll get a big reduction in actual jitter and can realistically expect very low latency so long as you stay within the same VPC and region.RDS specifically seems to be well optimized for networking, in the few years I've been using it (us-west-2) I've never had a single problem related to latency with the RDS instances themselves, but this is a N=1 data point.Also, If you're using many instances connecting to one postgres server, make sure you're using pgbouncer or a similar connection pooler or you'll hit connection issues way before even a few ms of latency will cause problems.CommentShareMaxCloudanswered a year agoAWS-User-1650441 a year agoHello,To answer you, we're using t3 instances (ENA enabled) and proxysql.So what's acceptable speaking of WriteLatency and ReadLatency ? In our case getting x4 latency just by fetching data from another AZ is questionable (but maybe it's just that Aurora is not the good fit for us)Share"


Question: Codepipeline deploy fail

"While using cloud9 to implement changes in template and when codepipeline executing it, it keeps on failing stating that cloudformation met with rollbackcomplete while running change set action. What could be the possible mistakeFollowComment"

Answer: "0Hi,The best way to debug this is going to the AWS CloudFormation stack and check the Events messages. Can you provide those outputs here?RegardsCommentShareAWS-User-3082121answered 8 months agorePost-User-4553484 8 months agomy output was just rollback. And even after trying lot of times it was unable to change setShare"


Question: White layer and wrong animation problem

"Some problems with animation and material. I have made model in Blender But when I import this model in Lumberyard and assign material, it seems the editor adds an additional white texture or something. I tried different options in material editor, but no result. Here the video how it looks and works in Blender, and then in Lamberyard. Any ways to remove this white layer? https://youtu.be/6qwxXj8AqqoThere are also an animation problem, I tried different expot FBX settings and Lumbermixalot plugin also, but animation still works wrong. The most strange thing is that knight also was edited in Blender(its from Mixamo) and his animations works perfectly well (I have exported his animations with the same settings).Follow"

Answer: "0Accepted AnswerProblem was in the metric system, I set units scale to 0.01 (found decision in other topic). Now animation works fine)SharerePost-User-9113359answered 3 years ago0After some experiments with diffuse and specular colors the texture looks fine in game, but still a big problem with animation.SharerePost-User-9113359answered 3 years ago0Hi Phantomny,Which version of Lumberyard are you using?Thanks.SharerePost-User-8164888answered 3 years ago0Hello, its 1.24.1.0.SharerePost-User-9113359answered 3 years ago0Gotcha, I've found a previous post on a similar issue with importing FBX from Blender, https://forums.awsgametech.com/t/import-fbx-to-ly-from-blender-2-8/7774It offers some possible solutions, and let me know if it works.SharerePost-User-8164888answered 3 years ago0Awesome, glad you found a solution for this.SharerePost-User-8164888answered 3 years ago"


Question: Serverless Gps tracking Server

"Hi all!i wish a great year with health!Currently i am looking to implement a gps server that i have created with c# to aws and i would appreciate if i had any hint from you about the following case:I have created in an on Premise windows server, a gps server with c# that accept HEX messages from gps devices.The server run on specific port that i have declared and the devices needs and IP:PORT in which they could send the messages.Also the server can send back to the device commands.I am struggling on  how i could create a similar aws serverless setup .I have find aws API gateway with socket but cant setup a port there and also accept messages only in json format as i have seen.Also i looked around aws iot but works with mqtt protocol but the majority of the devices cant support it now.Is there any way to create the same setup that i have on-premise to aws with  serveless setup  and not to need to create ec2 windows instance and upload there my c# gps server?My ideal setup is to create a serverless flow that the device can send messages to IP:PORT with a bi-directional communicationThank you all for your timeFollowComment"

Answer: "0You can use Lambda function for your code and host an Application load balancer with required port listener and target group to run your lambda function.CommentShareSumit Kumaranswered 5 months agoUri EXPERT5 months agoALB only supports HTTP traffic, not arbitrary TCP connections. An NLB supports TCP traffic, but it does not support Lambda in the taaget group.Share0API Gateway supports Web Sockets, which are based on HTTP. It does not support generic TCP connections.You will need to use an NLB that front maybe some containers that run on Fargate, so you do not need to manage the infrastructure. This is not a serverless solution though.CommentShareEXPERTUrianswered 5 months ago"


Question: Can no longer rename partitions in a glue table?

"Hi, it seems that sometime in the past couple of weeks, the ability to rename a partition column in a Glue table created by a crawler has been removed.The documentation still claims you can do this, but if I edit a Glue table in the console, change a partition column name (which the UI allows,) and then save the new table version, I'm met with the error:InvalidInputException (status: 400): Trying to change partitionColumn name from : partition_0to new partitionColumn name : year. Change of partitionColumn names is not allowed.In my case, I am streaming in log data in S3 partitioned by hour, which is an important optimization for querying it efficiently. But I'm forced to use these partitions by referring to "year" in my Athena queries as "partiton_0", month as "partiton_1", etc. This seems silly, and I can't seem to find a workaround. I tried setting up an ETL job in Glue to do this in a more modern way, but it doesn't see the partition columns at all.Is this a bug that will be fixed, is there some new way to transform the partition column names, or am I just out of luck now?FollowComment"

Answer: "-1Partition columns is something very sensitive that should not be changed since normally you'll break the table.  If you can it's always better to follow the conventions, creating tables where the directory structure doesn't match the table, is often problems down the line.If your files follow the convention, you can recreate the table and run REPAIR (see https://docs.aws.amazon.com/athena/latest/ug/msck-repair-table.html), but if the crawler didn't do that, probably means your directories don't have the partition names.BTW, I don't see where the documentation says you can do that.CommentShareGonzalo Herrerosanswered a month agoSundog a month agoThanks, but I guess I didn't explain the issue clearly. I'm not looking to repartition the data, I'm just looking to assign meaningful names to the partitions discovered by the Glue crawler. The documentation I'm referring to linked above is:"To change the default names on the console, navigate to the table, choose Edit Schema, and modify the names of the partition columns there."That doesn't work anymore.ShareGonzalo Herreros 25 days agoI see your point about the documentation, thanks for raising that. Maybe you could try to rename using ALTER TABLE https://docs.aws.amazon.com/athena/latest/ug/alter-table-replace-columns.htmlAnyway, it's not ideal when the column names don't match the directories, if you don't follow the standards you will find some limitations and will have to add partitions yourself in the futureShareSundog 24 days agoI don't really understand how the column name can match the directories in the case of a time-based partition. What's happening is Kinesis Firehose is streaming data into S3 into a directory structure of the year / month / date / hour. The directory names are just the numbers associated with these (i.e., 2023/05/10/12)- all Glue gives us upon crawling that structure is default partition names of partition_0 for the year level, partition_1 for month, etc.I read that Glue crawlers can interpret directory names in the style of "year=2023" for example in order to attach the desired column names for the partition that way, but if there's a way to configure Firehose to do that then I'm missing it.FYI I'm told that ALTER TABLE results in an error in this case.ShareGonzalo Herreros 23 days agoWhen the naming conventions are not followed, everything is harder. You could live with the generic column names, or define the partitions with the meaningful names and then load the partitions using ALTER TABLE ADD PARTITION commands using the meaningful names to their corresponding  paths. It might be possible to update an existing table modifying the metadata directly using the command line API but there is a good change the table gets broken and unusableShareSundog 22 days agoSorry if I'm being clueless here. How can I configure Firehose to "follow the naming conventions" when writing data into S3?ShareShow 2 more comments"


Question: Amplify service role / PassRole / Wild Rydes tutorial

"Hello,On region=ap-southeast-2I'm following the AWS tutorial to create an Amplify app:https://aws.amazon.com/getting-started/hands-on/build-serverless-web-app-lambda-apigateway-s3-dynamodb-cognito/module-1/I get error:"Failed to create service role, please create a service role your self and attach to this app."And:Houston, there's a problemUser: arn:aws:iam::YYYYY:user/XXXX is not authorized to perform: iam:PassRole on resource: arn:aws:iam::YYYYY:role/AmplifyConsoleServiceRole-AmplifyRoleWhen I check the AmplifyConsoleServiceRole-AmplifyRole Trust Relationships it says:"The following trusted entities can assume this role:The identity provider(s) amplify.amazonaws.com"I understand there's a concept of PassRole to Amplify based on the User's policy per:https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_passrole.htmlAccording to this the PassRole policy should be associated with a User but I don't know how to attach PassRole to a specific User.(this also isn't documented in the tutorial so not sure if that's bad advice for setting up an Amplify app).I'm also new to AWS so working with policies etc is new to me.Appreciate any help you may have.Note: After removing the AmplifyConsoleServiceRole-AmplifyRole and trying again to deploy I still get the error:Failed to create service role, please create a service role your self and attach to this app.And the app fails to build (just hangs there).Update again: I've progressed it further per thread here:https://github.com/aws-amplify/amplify-console/issues/1426Closing out to reflect the progress.MarkEdited by: MarkJC on Jan 5, 2021 10:36 PMEdited by: MarkJC on Jan 5, 2021 10:51 PMEdited by: MarkJC on Jan 6, 2021 1:46 AMFollowComment"

Answer: "0Answered per post update.CommentShareMarkJCanswered 2 years ago0Hello,When configuring the Amplify CLI, you can create a user with AdministratorAccess to your account to provision AWS resources for you like AppSync, Cognito etc. If not that, the please make sure that Amplify CLI requires the IAM policies mentioned in below document to function properly:https://docs.amplify.aws/cli/usage/iamYou can grant or restrict category permissions by including or removing items from the Action section as appropriate.CommentShareAWS-User-9460217answered 2 years ago"


Question: The screen does not show up in AppStream 2.0

"OverviewThe AppStream 2.0 screen (Windows) does not show up in a browser even after signing into the AppStream's UserPool account. I encountered this bug in us-west-2 a few weeks ago, so I set up a new stack and fleet in us-east-1 to make it work.Today (2022-06-24 PST), my us-west-2's session was expired, so I logged in to the account and encountered the same issue again. From this point, I cannot get into the AppStream's Window screen.Repro stepsI do not know the reproducible steps, but it seems that AppStream is stuck in this situation after session gets expired.Open the link from the AppStream invitation email.Sign in to the AppStream using the UserPool account.The Windows screen does not show up.##* Expected behavior:*The application selection page (or the previous session's Windows screen) should appear in a browser.Actual behavior:The browser shows a plain dark-navy screen (AppStream's background color) and outputs the following errors in the browser console:Error: Template (/appstream/views/directives/notification.html) could not be loaded. TypeError: Cannot read properties of undefined (reading 'ownerDocument'){code: 2, message: 'Display channel is not available.'}NotesOS: macOS Big Sur 11.6.5Browsers: Chrome 102.0.5005.115 and Firefox 101.0.1Reproducible in an Incognito/Private window. I also removed the cookie/local storage and tried again.FollowComment"

Answer: "0Please submit a support request to figure out what's going on to get it resolved. This is definitely not a standard error.CommentShareEXPERTMuraliAtAWSanswered a year ago0WorkaroundCreating a new stack with an existing fleet mitigated the issue.I'll submit a support request if the issue happens again. The only information is the following error in the browser console:Error: Template (/appstream/views/directives/notification.html) could not be loaded. TypeError: Cannot read properties of undefined (reading 'ownerDocument'){code: 2, message: 'Display channel is not available.'}CommentSharerePost-User-3359017answered a year ago"


Question: Error when calling aws cloudfront create-invalidation

A deploy script running in a codebuild job part of a codepipeline pipeline is failing. The script has been running fine for several months until now.The command called is: aws cloudfront create-invalidation --distribution-id $DISTRIBUTION_ID --paths '/*'The exception we get is: An error occurred (ServiceUnavailable) when calling the CreateInvalidation operation (reached max retries: 2): CloudFront encountered an internal error. Please try again.Running the command locally works fine.FollowCommentcrazyoptimist a year agoI experienced the same issue today in our GHA workflow which uses this command:aws cloudfront create-invalidation --distribution-id ${{ secrets.AWS_CLOUDFRONT_ID_STAGING }} --paths /\*Re-ran the failed workflow and it's passed. But not sure why and when it happens. AWS status page tells me that Cloudfront service stays healthy. :-(Share

Answer: "2Accepted AnswerTo find out the error cause, an investigation of the internal logs might be required. The best way for resolving such kind of issues is probably just contacting AWS Support, so the support team can look into this. Please open a support request.CommentShareDmitry Balabanovanswered a year agoFredrik a year agoThe problem with AWS support is that for questions outside of billing, the support is paid. This means bug reports like this have no where to go. It seems like it was a bug btw, because now it's working without any changes from our side. Thanks for the answer!Share"


Question: FRUSTRATION / Exam passed / Grade : pass

"Hello,I want to report an extremely bad experience giving the AWS Cloud practitionner exam on 07/21/2022. Apologies if this is not the right place to post this experience as I did not find any other forum to post exam-related experiences. I appeared for AWS cloud practitionner exam on 07/21/2022. After completing the exam, I submitted the exam for the result and was very happy to see the screen "Grade: PASS. Congratulation! You have successfully passed the AWS  .... Within 5 business days, you will receive an email stating your exam result ......"After that, I thought that the exam is now over, as I sat for another 30 seconds and did not see any notification from Proctor or any other dialog on the PASS result screen. I took the photo of the screen from my mobile to keep it for my records so that I don't lose my result and have proof(Already made it with Personvue exam center without any problem).Right then I saw a screen stating your exam is terminated due to a violation of taking photos and within 10 seconds screen closed.I chatted PSI (Jessica) and explained the matter that I completed the exam and saw the result on the screen as "PASS". Only after that, I take the picture. She told me to wait email. I also opened a ticket on AWS side because This is a very frustrating situation. It's not acceptable for AWS/PSI to make candidates suffer and lose belief in the certification process.I found on internet people which have encountered such bad experience (As my english is not fluent, I took same explanation.):https://repost.aws/questions/QUCV5L0SLUSFWBSGTi413jEg/exam-revoked-after-passing-bad-experience-with-psi-online-aws-solution-architect-associate-examIf it happens to many candidate, it 's maybe time for PSI/AWS to bring some clarification in process.FollowCommentrePost-User-0918389 10 months agoHello, could the situation be resolved? Do you have your certification now?Share"

Answer: "0I am sorry you had a frustrating experience. Until you completely close/exit the exam delivery application/browser, you are still being proctored. The proctors have a strict set of rules to follow and properly revoked your exam as you used your cell phone which is against our policies.Your exam result and case information will be reviewed by the AWS Certification Exam Security team. They will make the final determination on your result and will post that to your AWS Certification portal within 5 business days.KimAWS CertificationCommentShareKimWD-AWSanswered 10 months ago0Hello KimMany thanks for your feedback. Please let me provide you full statement about how situation can be frustrating.I'm IT manager and technical support manager for a International company which is software editor.As main IT manager in my company, I'm interested with cloud adoption for our internal needs. I also accompany my dev team with moving forward with delivery our solution as saas in our software model distribution. Moreover, as techsupport manager, our customers asks us to be on their side for implementing Cloud environment (for them) on which we implement our software solution. So, for some years now, I build lot of cloud environement for test, dev and prod. Until now, I was mainly trained on Microsoft Azure solution. I've passed lot of certification (az900,az303, az305, sc900, dp900) with personvue exam center. (Each time, I passed an exam with personvue, I made a photo of result without any problem)It's important for us to have some alternative to Microsoft and as our customer can ask for it, we were also interested by AWS. As my CTO ask me, I follow this AWS course and pass this certification. I spend time (a lot of), money for it. So, I pass this certification; I press on button "End my exam". In my language, End means Finish (I'm French). I've got message saying ""Grade: PASS. Congratulation! You have successfully passed the AWS  .... Within 5 business days, you will receive an email stating your exam result ......" "So, at that stage, I made a photo in order to have a proof for me and my boss. I'm not a cheater. I do not break rules in my life.Certification should be for a company like yours a source of motivation and adoption of your solution. It was the plan for me. My objective was to start with practitionner certification and then, AWS Certified Solutions Architect. Because of this bad process, it causes a negative experience about AWS. I'm sure that AWS managers can find such situation ridiculous.I hope that you can change this negative optinion and turn situation on a positive way in a timely manner.Best Regards,CommentSharerePost-User-5765227answered 10 months ago0Hellothanks for your follow upI ve got certificationfrustration remainsbest regardsCommentSharerePost-User-5765227answered 10 months ago"


Question: Port Forwarding in AWS Lightsail

"I've set up a WAMP server in my AWS Windows Lightsail instance, running on my private IP address, port 80. I'd like to make it so that the public IP has a port that redirects traffic to my server. On my home machine, I can do this by accessing router settings from my internet provider. How would I do it on my VPC?FollowComment"

Answer: "0To redirect traffic from a public IP and port to your WAMP server running on a private IP and port within your AWS Lightsail instance, you'll need to configure a Network Address Translation (NAT) rule. AWS Lightsail instances use an automatically assigned static public IP address, and you can create a custom firewall rule to allow incoming traffic to your desired port.Follow these steps:Log in to the AWS Management Console and navigate to the Lightsail service.Click on the "Instances" tab in the left navigation pane.Locate your Windows instance and click on its name.In the instance details page, click on the "Networking" tab.In the "Firewall" section, click "Add rule."For the "Type" field, select "HTTP" if you want to allow HTTP traffic, or "Custom" for other specific ports. If you choose "Custom," provide the port number you'd like to open in the "Port range" field.For the "Source" field, you can choose "Anywhere" to allow traffic from any IP address or "Custom" to restrict access to specific IP addresses or CIDR blocks.Click "Save" to apply the new firewall rule.Now, your WAMP server should be accessible from the public IP address of your Lightsail instance. However, since your WAMP server is running on your private IP, you need to ensure that it is configured to listen on all available network interfaces, not just the private IP address. To do this, you may need to edit your Apache configuration file (httpd.conf) and change the "Listen" directive to "Listen 80" (without quotes) or "Listen 0.0.0.0:80". Then, restart the Apache service for the changes to take effect.CommentShareEXPERTsedat_salmananswered 2 months ago"


Question: CloudShell not connecting

How to connect AWS Cloudshell even i refresh the browser?FollowComment

Answer: "0There might be several reasons that causes issues when trying to launch your Cloudshell environment.One of the most common scenarios is having some permission restrictions with your IAM identity for accessing Cloudshell:1. Unable to start the environment. You don't have required permissions. Ask your IAM administrator to be granted access to AWS CloudShellIssue: When you attempt to launch AWS CloudShell from the AWS Management Console, you're denied access and notified that you don't have required permissions.Cause: The IAM identity that you're using to access AWS CloudShell lacks the necessary IAM permissions.Solution: Request your IAM administrator to provide you with the necessary permissions. They can do this either through adding an attached AWS managed policy (AWSCloudShellFullAccess) or an embedded inline policy. For more information, see Managing AWS CloudShell access and usage with IAM policies.For more troubleshooting scenarios please check the Cloudshell troubleshooting guideAs alternative to those scenarios I leave you some other potential issues that can cause this behaviour:Make sure you launch the CloudShell in a supported region.Ensure there are no policies that could be denying CloudShell, e.g. SCP (AWS Organizations), managed policies, permission boundaries.Make sure your machine's time is accurate (e.g. synchronize your time using a time server).Make sure your account is fully verified (if it's a brand new account) - contact AWS Support for that.Try restarting CloudShell via Actions --> Restart CloudShell.Try deleting the home directory via Actions --> Delete AWS CloudShell home directory.CommentShareigsalvaranswered a month ago0I'd advise to read the troubleshooting guide to get all hints on possible errors: https://docs.aws.amazon.com/cloudshell/latest/userguide/troubleshooting.htmlAdditional hints : https://stackoverflow.com/questions/70719732/aws-cloudshell-error-unable-to-start-the-environmentCommentShareDidier_AWSanswered a month ago"


Question: Whitelisting specific IPs for NLB and the effect on LCU cost?

"A customer wants to whitelist specific IPs to access the NLB (and the EC2 instances).Is it better to use NACLs on the Subnets or Security Groups on the EC2 instances?Also, how does it implies in terms of cost, meaning, if someone else tries to connect to the NLB and is refused due to the NACL or SG, does the connection count in the LCU?FollowComment"

Answer: "0Accepted AnswerUsing NACLs and/or SGs is not an either/or proposition. You could use both.NACLs require more careful configuration to make sure you are allowing the ephemeral range outbound (so traffic can get back to the source), Also make sure to allow for the ephemeral range coming INTO the NLB from the targets.  One way this can be done by just blanket allowing the targets (ex: 10.0.0.0/24, all traffic) in the NACL. The payoff for this higher amount of configuration is that you can DENY as well as ALLOW and that entire subnets are protected (guarding against Jr Admin launching an instance and not applying the special Security Group to it). Be aware that the NACL for the NLB subnet and the NACL for the target's subnet both come into play here.Since Security Groups are stateful they take care of the ephemeral ports for you. However with an SG you cannot DENY, only ALLOW. In your case that sounds like it would work fine since your guy wants to only allow certain IPs. Not as powerful as NACLs in my mind, but it is simpler, particularly if they don't have people launching instances manually (and thus risking missing applying the special SG).One option is that he could control the flow mostly using Security Groups and use the Network ACL for blocking certain traffic. Example - if he wanted to allow x.x.x.x/24 but x.x.x.30/32 was abusive he could allow the /24 in the SG and put a specific /32 DENY rule in the NACL (ahead of a general ALLOW rule).Both NACLs and SGs have limits: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html that you need to be aware of.As for LCUs - since blocked traffic never reaches the NLB, there would be no connection and thus nothing to meter for an LCU.CommentShareAWS-User-2996111answered 6 years ago"


Question: Assigning Required Permission to EKS Role When Enabling Secret Encryption

"I want to enable secret encryption in EKS. Base on this page : Enabling secret encryption on an existing cluster, permission kms:DescribeKey and kms:CreateGrant are required.My question is which one is the preferable way to assign these permission? Is it assign the permission manually or giving key usage permission to the eks-role ?FollowComment"

Answer: "0Hi,As the document states, you have to make sure that the kms:DescribeKey and kms:CreateGrant actions are permitted on the policy for the principal that calls the create-cluster API. You can do it either editing the KMS key policy directly (manually), or giving key usage permission to the eks-role.For example, you can find below the policy of a brand new KMS key without any “usage permissions / Key users” selected.{    "Id": "TestKey",    "Version": "2012-10-17",    "Statement": [        {            "Sid": "Enable IAM User Permissions",            "Effect": "Allow",            "Principal": {                "AWS": "arn:aws:iam::AWSAccountID:root"            },            "Action": "kms:*",            "Resource": "*"        }    ]}This policy above is similar to what the documentation refers with “By default, the create-key command creates a symmetric encryption KMS key with a key policy that gives the account root admin access on AWS KMS actions and resources.” It is called Default Key Policy (https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-default.html)You can then go to the policy and edit it manually, adding the required permissions to the right role (eks-role in your case), or you can do the same through the “usage permissions / Key users” section.Going back to the same policy above, this one below is exactly the same KMS key policy after adding the role TestRole in the “usage permissions / Key user” section.{    "Id": " TestKey ",    "Version": "2012-10-17",    "Statement": [        {            "Sid": "Enable IAM User Permissions",            "Effect": "Allow",            "Principal": {                "AWS": "arn:aws:iam:: AWSAccountID:root"            },            "Action": "kms:*",            "Resource": "*"        },        {            "Sid": "Allow use of the key",            "Effect": "Allow",            "Principal": {                "AWS": "arn:aws:iam:: AWSAccountID:role/TestRole"            },            "Action": [                "kms:Encrypt",                "kms:Decrypt",                "kms:ReEncrypt*",                "kms:GenerateDataKey*",                "kms:DescribeKey"            ],            "Resource": "*"        },        {            "Sid": "Allow attachment of persistent resources",            "Effect": "Allow",            "Principal": {                "AWS": "arn:aws:iam:: AWSAccountID:role/TestRole"            },            "Action": [                "kms:CreateGrant",                "kms:ListGrants",                "kms:RevokeGrant"            ],            "Resource": "*",            "Condition": {                "Bool": {                    "kms:GrantIsForAWSResource": "true"                }            }        }    ]}Another option you have is to leave the Default key policy the way it is (first example above) and use IAM policies to grant access to KMS keys. You can find some information https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-default.html and https://docs.aws.amazon.com/kms/latest/developerguide/customer-managed-policies.htmlCommentShareSergioAanswered 3 months ago"


Question: Cloudwatch issue - ModuleNotFoundError: No module named 'cwlogs'

"Hello,I'm facing some complicated and un-thinking issue with Cloudwatch,I believe Cloudwatch are easy to use and install because i used to install it at others EC2 instance before.Unfortunately,but it doesn't happen this time.Might need your guys help for this issue.The issue is:-Cloudwatch unable to get the metric information from my EC2.Troubleshooting i do as per below:Check config file:/etc/awslogs/awslogs.conf point to correct server - MyServer1 &/etc/awslogs/awscli.conf  point to correct region = ap-southeast-12)Following the step to re-install cloudwatch:https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.htmlStuck at because there's an error at the log."To install and configure CloudWatch Logs on an existing Amazon Linux instance" section,(Optional) Check the /var/log/awslogs.log file for errors logged when starting the service.The error as per below - /var/log/awslog.log:File "/usr/lib/python3.7/site-packages/awscli/plugin.py", line 58, in _import_pluginsplugins.append(import(path))ModuleNotFoundError: No module named 'cwlogs'What i do is install =pip install awscli-cwlogsBut the issue still persist.Any steps that i missing?Please advice.FollowComment"

Answer: "0Accepted AnswerI notice the error happen because of python are unable to communicate with aws api.What i do to settle this issue is:1)Python version conflict - got 2 version at server which is 3.7 & 2.7-AWS API so far not support python 3-Solution: Remove python "yum remove python3".2)Re-install cloudwatch agent services at server "sudo yum install -y awslogs".3)Reconfigure cloudwatch config files stepshttps://youtu.be/22lzkpra_WUIssue solved...thanks everyone...CommentShareZeeeanswered 6 months ago0Hi,sounds like the module is still missing. Did you try running the above command with a sudo (not sure if you need the sudo or not, might depend on how you access your EC2)? sudo pip install awscli-cwlogsWhat does that command says?if the command succeeds, did you also try running pip install --upgrade awscli-cwlogs (again, not sure whether you need sudo or not)CommentShareJscanswered 6 months agoZeee 6 months agoHi mate.. Thanks for your reply...Just to inform u that I'm in root account,but i will run the command as per suggest and give you the results... Thanks again for the reply..Share0Hello...I run the command as per suggested.This is the output:.sudo pip install awscli-cwlogsAnother oneCommentShareZeeeanswered 6 months ago0I'm sorry I don't know what's causing your issue, I hope someone else jumps in with another suggestionCommentShareJscanswered 6 months ago"


Question: Does billing alarm itself cost money?

"Sorry if this is a dumb question. I have an AWS account that I do not have intention to use any time soon. So I cleaned all resources I can think of, delete all IAM users except a readonly one and both root/readonly user is guarded with strong password through password manager + MFA. But I am still a little paranoid over the tragedies of hacked AWS account so I want to set up billing alarms for this "frozen" account.However upon careful examination it looks likeI need to enable Billing Alerts at first, which asks Billing to send data to CloudWatch as a metric, and this is not reversible (you cannot deactivate it anymore)I need then a cloudwatch alarm on this metric.In that case, am I right that billing alarm itself will cost a small amount each month since it is a cloudwatch tool, and there is no way to turn it off since I cannot deactivate Billing alert?Thanks!FollowComment"

Answer: "0As per https://aws.amazon.com/aws-cost-management/aws-budgets/pricing/ the first two budgets are free.CommentShareSiTheAWSGuyanswered 8 months agoNataliya_G EXPERT8 months agoIt's only when you use Budget Actions. If it's just for cost alerting and notifications, then it's free.Share0We highly recommend using AWS Budgets instead. If you use Budgets for notifications and alerting only https://aws.amazon.com/aws-cost-management/aws-budgets/pricing/ - they are free. You don't pay anything.If you want to use some extra features (in particular, Budget Actions) - then only first 2 actions-enabled budgets are free, and then you pay a fee for additional ones.However, using Budgets for billing alerts and notifications only, will not cost you anything.CommentShareEXPERTNataliya_Ganswered 8 months ago"


Question: Component Broken error while deploying Postgressql community Component

"Able to build and publish as given in https://github.com/awslabs/aws-greengrass-labs-database-postgresql .While deploying from aws console getting component broken errorBelow is the logfile content aws.greengrass.labs.database.PostgreSQL.log2023-05-05T08:28:12.074Z [WARN] (Copier) aws.greengrass.labs.database.PostgreSQL: stderr. File "/usr/local/lib/python3.7/site-packages/docker/init.py", line 2, in <module>. {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Run.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.074Z [WARN] (Copier) aws.greengrass.labs.database.PostgreSQL: stderr. from .api import APIClient. {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Run.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.074Z [WARN] (Copier) aws.greengrass.labs.database.PostgreSQL: stderr. File "/usr/local/lib/python3.7/site-packages/docker/api/init.py", line 2, in <module>. {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Run.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.074Z [WARN] (Copier) aws.greengrass.labs.database.PostgreSQL: stderr. from .client import APIClient. {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Run.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.074Z [WARN] (Copier) aws.greengrass.labs.database.PostgreSQL: stderr. File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 6, in <module>. {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Run.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.074Z [WARN] (Copier) aws.greengrass.labs.database.PostgreSQL: stderr. import requests. {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Run.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.074Z [WARN] (Copier) aws.greengrass.labs.database.PostgreSQL: stderr. File "/usr/local/lib/python3.7/site-packages/requests/init.py", line 43, in <module>. {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Run.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.074Z [WARN] (Copier) aws.greengrass.labs.database.PostgreSQL: stderr. import urllib3. {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Run.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.075Z [WARN] (Copier) aws.greengrass.labs.database.PostgreSQL: stderr. File "/usr/local/lib/python3.7/site-packages/urllib3/init.py", line 39, in <module>. {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Run.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.075Z [WARN] (Copier) aws.greengrass.labs.database.PostgreSQL: stderr. "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently ". {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Run.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.075Z [WARN] (Copier) aws.greengrass.labs.database.PostgreSQL: stderr. ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips  26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168. {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Run.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.087Z [INFO] (Copier) aws.greengrass.labs.database.PostgreSQL: Run script exited. {exitCode=1, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=RUNNING}2023-05-05T08:28:12.093Z [INFO] (pool-2-thread-54) aws.greengrass.labs.database.PostgreSQL: shell-runner-start. {scriptName=services.aws.greengrass.labs.database.PostgreSQL.lifecycle.Shutdown.script, serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=STOPPING, command=["export PYTHONPATH=$PYTHONPATH:/greengrass/v2/packages/artifacts-unarchived/aws..."]}serviceName=aws.greengrass.labs.database.PostgreSQL, currentState=STOPPING}FollowComment"

Answer: "2Hi, it looks like the urllib3 issue here is coming from the requests library.  Please check the requests library version installed on your core device by using pip3 show requests command. If it is greater than or equal to 2.30.0, please remove it and install it again using pip3 install "requests<2.30.0"  (or you can also use pip3 upgrade). This should install urllib3 version < 2.0 which is compatible with your openssl version.If the urllib3>=2.0 is coming from some other library, you can choose to uninstall and install again the urllib3 < 2.0 library only.Doc for migrating to urllib3 v2.0 and potential remediation steps for the issue with openssl- https://urllib3.readthedocs.io/en/latest/v2-migration-guide.html#ssl-module-is-compiled-with-openssl-1-0-2-k-fipsCommentShareSaranyaanswered 24 days ago"


Question: Docker Image for Greengrass V2

Is a greengrass v2 core docker image going to be made available soon? Are there are any challenges to creating one from scratch at this time?FollowComment

Answer: "0Accepted AnswerHi,We will be releasing Dockerfiles and Docker images for v2 by the end of April, with Dockerfiles being released before the Docker images.We recommend using the official Dockerfile/Docker images once they are released, since they will be fully validated.Thanks for the interest!AVCommentShareavipinkuanswered 2 years ago0Greengrass V2 images for Docker are available now: https://docs.aws.amazon.com/greengrass/v2/developerguide/run-greengrass-docker.htmlCommentShareEXPERTMichaelDombrowski-AWSanswered 2 years agottnickb 6 months agoReplying a bit delayed...but I see the latest V2 version is ~2.5.3 presently.  Are there plans to continue releasing updated versions of the V2 Docker image?Also, will any aarch64 versions for V2 be released?https://hub.docker.com/r/amazon/aws-iot-greengrass/tags?page=1Share"


Question: .env not available in Elastic Beanstalk with Docker

"Hello,I am trying to use the Elastic Beanstalk environment variables in a container on the Docker platform for EB with docker compose.Aaccording to the documentation, the EB environment variables can be passed into the container by including .env as an environment file (https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.container.console.html#docker-env-cfg.env-variables).However, this does not work for me as the .env file can't be found (and EB fails).I logged into the EC2 instance and could in fact not find the file.Upon researching this issue, I found the suggestion to use /opt/elasticbeanstalk/deployment/env.list instead.However, this file only seems to be created after the first launch of EB (EB fails with an error message that the file cannot be found but file exists when logging into the EC2 instance).This is my docker-compose.yml:version: '3.7'services:  gui-app:    image: xxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/gui-app:latest    ports:      - 80:8080    depends_on:      - xray-daemon    environment:      - AWS_XRAY_DAEMON_ADDRESS=xray-daemon:2000    env_file:      - .env    links:      - xray-daemon  xray-daemon:    image: xxxxxxxxxx.dkr.ecr.eu-central-1.amazonaws.com/aws-xray-daemon:latest    ports:      - 2000/udpAny help is appreciated.Thank you!FollowComment"

Answer: "-1Hello,From your question it seems that the Beanstalk is unable to find the .env file in your application bundle.The document specified mentions it is a must to follow one or both of these configuration approaches.Add the .env file generated by Elastic Beanstalk to the env_file configuration option in the docker-compose.yml file.Directly define the environment variables in the docker-compose.yml file.However I can see that you have you have chosen to follow the first approach. In that case I would refer to please check the file where it has been placed as the file ".env" must be placed in the same directory from where you are calling the "docker-compose.yml" file. This issue mainly occurs due to ".env" file. not being recognised/found as you are including a .env file in your application bundle, hence, not allowing Elastic Beanstalk to generate an ".env" file. Please note to be careful to not leave spaces between the equal sign (=) and the value assigned to your variable in order to prevent spaces from being added to the string. This can also result in issues while creating the environment.I would suggest you to directly define the environment variables in the "docker-compose.yml file" as mentioned in the Second approach.Please refer to the following document for any help in performing the above mentioned task of defining environment variables.[+] https://docs.docker.com/compose/environment-variables/CommentShareSUPPORT ENGINEERrePost-User-0470248answered 9 months agorePost-User-8390464 18 days agoSo, what should I put into docker-compose.yaml to point to ".env file generated by Elastic Beanstalk"? Where does beanstalk generate .env file?The problem is that beanstalk runs"Running command /bin/sh -c docker-compose -f /opt/elasticbeanstalk/deployment/app_source_bundle config -q" before generating environment file.which fails with "Couldn't find env file: /opt/elasticbeanstalk/deployment/.envand then proceeds "Executing instruction: Generate environment file", which does not generate .env file, but does generate /opt/elasticbeanstalk/deployment/env.list.SharerePost-User-8390464 18 days agodocker-compose config fails because environment file is not generated yet and because of that failure the file is not recognized as docker-compose file, so beanstalk tries to use it as Dockerfile and fails. It looks like a bug and there is no way to use docker-compose with the environment file generated by Beanstalk.Share"


Question: "fargate service for which public ip disabled in public subnet , not able to connect to internet"

"i have  a fargate servcice whose public ip is disabled , and its inside a public subnet , in order for the service to work, i need my fargate service to access the internet and talk to the other service which is on the internet , but the faragte container is draining because of connection timeout issue , i have an internet gateway for my service as well and all the secuirty group rules and cofigurations have been thoroughly checked by team , but still the same issue persists.also there is a load balancer that i have created specifically to access the service . when i am enablic the public ip for the service, things are working great , but as soon as the public ip is disabled thing breaks, by any means can i connect my fargate service to connect to the internet service . and if possibe refer a doc or provide me with some steps that i should carry out,FollowComment"

Answer: "1You can connect your tasks to internet using either public subnet and internet gateway (with public IP)  or private subnet and NAT gateway. The public IP is required to be able to reach out public internet endpoints.If the selected subnet is a private subnet, the NAT gateway (which has a public IP on its interface attached on public subnet) will allow your task running on this private subnet to reach out to internet endpoints.To ilustrate:Using a public subnet and internet gatewayUsing a private subnet and NAT gatewayPlease refer this Best Practices documentation to read more about Connecting your ECS tasks to the internet.CommentShareHenrique Santanaanswered 4 months ago0I assume the LB is in the public subnet for the services configured in fargate. You don't want the fargate pods to have public IP but also want those pods to connect outbound on the internet.In that case launch the fargate pods in private subnet where the outbound traffic is routed via NAT. That should suffice for your requirement.Any instance without public IP but launched in public subnet wont have outbound connection to internet. So the behaviour you are observing is normal--SydCommentShareSydanswered 4 months ago"


Question: Add a second network interface for a vlan/exclusive network between two EC2 instances in same VPC

"Hi, I was asked to provide 2 instances for a development project. One instance will use Windows Server and the other CentOS.They need each of the instances to have two network cards, one for internet connection and the other to be privately connected to the other instance. For example, the second network card of the windows server should be 10.88.8.1 and the second card of the centOS should be 10.88.8.10 and it should be an exclusive connection, but those IPs are out of the range of the primary private IP from each aditional network interface. When adding the extra network cards, I see that they come with a specific private IP but from the instances I can change the IP of each instance: in Windows through CPL and in CentOS from the command line, but I can't ping between them . What would be the simplest way to be able to configure those two extra network cards that the client needs?Thanks in advance.Regards.FollowComment"

Answer: "0First: While you can do what you want in your VPC you don't have to do it that way. Instances within the VPC already have private IP addresses and can communicate with each other. The public IP addresses that they have (assuming they are on a public subnet) are not assigned directly to the instance network interface; instead that access is provided using 1:1 NAT on your Internet Gateway. The instances can transparently (and without further configuration) communicate privately on their private IP addresses with no further action by you.But let's assume that you absolutely need separate interfaces. You should do the following:Create a new subnet in your VPC with a new IP range that is within the VPC IP range but different to your existing subnet(s).Create two network interfaces on that subnet. The interfaces will automatically get IP addresses.Attach the network interfaces to the instances.Create a security group and in the inbound rules, allow the security group to communicate with itself. This means that any instances that has been assigned to the security group can communicate with any other instance that has been assigned that security group.Attach the security group to the two network interfaces.CommentShareEXPERTBrettski-AWSanswered 4 months agoLeonardo 3 months agoHi, Brettski-AWS.Thanks for your help. In the end, I tried to attach a second network interface to each instace, each secondary network interface belongs to the same security group of the corresponding primary network interface. I didn't have to configure nothing, I see the Centos instance and the Windows instance have assigned the corresponding private IP, however I can only ping from Windows instance to the primary private Centos IP, I can't ping to the secondary network interface private IP. And neither can ping from Centos instance to any Windows instance. What extra configuration do you think I need to make? Thanks in advance, and sorry for the bad English.ShareBrettski-AWS EXPERT3 months agoI think you need to look at the built-in Windows firewall - it will allow communications that are initiated by Windows but block anything that is initiated by another host.Share"


Question: How to set-up my EC2 server and find the right model for me?

"Hello,I am building out a casino game that will not involve multiplayer online gameplay. Rather, it is a simple slots game with 6 different rooms. I anticipate having some users, perhaps optimistically around 1000 per month.Eventually, I'd like to move into the multiplayer realm and incorporate poker/blackjack live games; however, for now, I'd like to stick to the slots game concept. How much data/ what service do you recommend for me? My programmer recommended between 4-8 GB RAM, 2-4 core CPU, and 165-240 GB SSD. What do you guys think?Also, could anyone provide a step-by-step guide to how to set-up such a server with the recommended specs? Many thanks.FollowCommentTolis 8 months agoit depends on the kind of slot games you are building, the state they need to keep in memory, your payout calculations and any additional bonus features you're developing, if players will have sticky sessions to the game sessions etcShare"

Answer: "1Here are some good resources that you can refer to architect your solution. Use Amazon EC2 for cost-efficient cloud gaming with pay-as-you-go pricingHow do I choose the appropriate EC2 instance type for my workload?Choosing the Right EC2 Instance Type for Your ApplicationAWS re:Invent 2020: Selecting the right Amazon EC2 instance for your workloadsAs one of the above references says that because you can launch and terminate instances as desired, profiling and load testing across a variety of instance types is simple and cost effective. You can easily change instance types as your needs change. You can even profile multiple instance types as part of your Continuous Integration process and use a different set of instance types for each minor release.CommentShareEXPERTAWS-User-Nitinanswered 8 months agoEXPERTIndranil Banerjee AWSreviewed 8 months ago0Auto Scaling Groups (ASG) can help with provisioning EC2 easily and it can help you manage the scaling easily so that you don't have pay for EC2 when you don't need them. Also, ASG makes it easy to pick and choose the instances by offering a new feature called Attribute based instance selection. If you want to further cost optimize and if your use-case can tolerate interruptions, then you can leverage EC2 Spot instances. You could also use ASG to mix and match On-Demand and Spot instances as well. There is a lab which explains how to deploy ASG with different options (last link in the reference) which you may find useful.References:https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.htmlhttps://aws.amazon.com/blogs/aws/new-attribute-based-instance-type-selection-for-ec2-auto-scaling-and-ec2-fleet/https://aws.amazon.com/ec2/spot/https://ec2spotworkshops.com/ec2-auto-scaling-with-multiple-instance-types-and-purchase-options.htmlCommentShareSudhi_Banswered 7 months ago0There is actually an entire learning plan on skillbuilder dedicated to this:https://explore.skillbuilder.aws/learn/public/learning_plan/view/1569/aws-for-games-learning-plan-game-serversCommentShare_Jen_answered 5 months ago"


Question: EC2 instance suddenly unreachable by SSH

"For some reason, my EC2 instance, running Ubuntu, has suddenly become unreachable through my usual SSH instruction, although I have changed nothing.By running the ssh connection instruction with verbosity, i can read in the response:include /etc/ssh/ssh_config.d/*.conf matched no filesCan it happen that the ssh configuration on the EC2 instance changes for some reason, without any action on my part??Thank you very much for your help.FollowComment"

Answer: "0Hi,I would recommend to follow this guidance to troubleshoot this further:https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.htmlhttps://repost.aws/knowledge-center/ec2-linux-resolve-ssh-connection-errorsIf this does not solve the problem, you can also try to connect using Session Manager: https://docs.aws.amazon.com/ja_jp/systems-manager/latest/userguide/session-manager.htmlCommentShareEXPERTJonathan_Danswered 16 days ago"


Question: EC2 Instance ssh

"Hello,I am trying to ssh EC2 instance from GIT bash but getting the error"$ ssh -i "webserver-key.pem" ec2-user@ec2-18-216-146-49.us-east-2.compute.amazonaws.comssh: connect to host ec2-18-216-146-49.us-east-2.compute.amazonaws.com port 22: Connection timed out"Also I tried EC2 instance connect and got below error:-Failed to connect to your instanceEC2 Instance Connect is unable to connect to your instance. Ensure your instance network settings are configured correctly for EC2 Instance Connect. For more information, see Set up EC2 Instance Connect at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html.Thanks,MonicaFollowComment"

Answer: "0I had to set up a security group that gave inbound and outbound TCP access to all IP addresses and attach it to the Internet Gateway.  TBH, I googled the answer.Also, make sure that the permissions on the SSH Certificate that you copy to your local workstation are set to your account only. In Linux, that's chmod 400 keyfile.pem. In Windows, you have to add your user with Full Access and then disable the inheritance to remove the inherited permissions.CommentSharedsudds1963answered 4 months agoMonica 4 months agoNSG i used default one which allow all traffic. Also user has full access and inheritance disabled. but its giving same error.Share0I found my mistake so i followed the below steps:-Download putty and puttygen, or the putty package and install it.Get the .pem file from your AWS EC2 instance.Use puttygen to convert the .pem file so that you'll have a private key --- mistake happened here. I chose "Conversions" tab from PuttyGen, and load my .pem file. After loading pem file, here DO NOT hit "Generate", instead directly "Save private key". That's the key you need. If you click Generate, you'll have a totally different pair of keys.In putty, use ec2-user@your.public.dns.that.you.get.from.aws.ec2.instance, and load the private key at SSH/AuthThanks,CommentShareMonicaanswered 4 months agoHalal 4 months agoThat's how I usually connect, but today suddenly I'm unable to connect using Putty with my preset saved connectionShare0I tried that myself, but ultimately had to define an explicit one and assign to the VPC for my instance. Also, for SSH, you will need to have an outbound rule on your default Security Group that explicitly allows connectivity to the Ephemeral Ports (1024-65535), as outbound SSH will connect on one of those ports.In windows, this is viewable using the netstat command, but in Linux netstat is deprecated in favor of 'ss'.  Most distributions still install it with the networking tools, but ss is the preferred command.CommentSharedsudds1963answered 4 months ago0I'm having the exact same issue. Not sure why this is happening.CommentShareHalalanswered 4 months ago"


Question: How to retreive GreengrassV2 Core Device's AWS IoT Thing name

"When working with already-configured GreengrassV2 core devices (and therefore the associated AWS IoT Thing), how should one go about the local identification of a physical device that's been registered? The Greengrass CLI (https://docs.aws.amazon.com/greengrass/v2/developerguide/gg-cli-reference.html) has no method to describe how the local device is registered to the cloud, and the AWS CLI greengrassv2 command does not help identify "which physical device corresponds to the registered core-device".Consider a scenario where someone has 5 unmarked Raspberry Pi's set up as core devices. Each is registered as a Greengrass core device / IoT Thing. How can one best cross-reference which Raspberry Pi corresponds to which core device definition?I see in a recent documentation update, that in a component lifecycle script, a number of environment variables are automatically set including AWS_IOT_THING_NAME.  However, these don't seem to be available outisde of a lifecycle script. Is there a configuration file that stores that information which could easily be accessed, outside of the context of a lifecycle script? Is there a local utility installed that can print out this information?I feel like there must be some simple way to SSH into a physical device and quickly identify which IoT Thing it is. Or am I wrong?FollowComment"

Answer: "0Accepted AnswerHi,The correct and the recommended way of knowing the thing name within the Greengrass scope is by using the AWS_IOT_THING_NAME environment variable. However, as you mentioned it already - you can read the effective config file at <greengrass-root>/config/effectiveConfig.yaml to know the thingName under system property outside of Greengrass.Thanks,SaranyaCommentShareSaranyaanswered 2 years ago0As a follow-up, I now see that there is an "effective config" file in the Greengrass root directory (/greengrass/v2/config/effectiveConfig.yaml on my system after following the current quick-start installation instructions).In that YAML file, there is a section system.thingName which shows the currently registered name.Is this the appropriate place to find this information?CommentSharetraviprossanswered 2 years agoflycast 9 months agoJust wondering if you explored this idea of reading the config file?Share"


Question: RDS mariadb performance on update queries

"Hi, I've migrated a workload from mariadb on ec2 to rds and I'm facing a performance issue.This job make massive update queries on some tables (69465).Using mariadb service on Ec2 common instance (t3a.small) it takes 1m31,148sUsing RDS (db.m6g.xlarge - storage io1 200GB ): 5m46,239sAfter many tries I realized that the only way to have the same performance on Ec2 and RDS is to disable "automated backups" on RDSCan anybody help me to understand why? Backup is a requirementFollowComment"

Answer: "0Enabling backups in RDS requires enabling binlogs. So, with/without binlogging you will see different io pattern.Also, you don't state the IOPS for 'RDS1' or your self managed ec2. These are important details for your question.'RDS2' is a 100g gp2 volume, so you need to look at the BurstBalance to be sure that you are not running out of Burst IOPS.Finally, IOPS < 0.5 is essentially an idle instance. It is unlikely with any configuration that your massive update workload is active and the IOPS is near zero.Hope that is helpful.CommentShareMODERATORphilawsanswered a year agoFdalmolin a year agoHi Phil, thanks for your answer but the problem is not in what I see in logs but the performance of my workload. So I rewrite my question to make it clearerShare"


Question: EC2 p2.xlarge instance losing GPU support after reboot

"Since Tuesday, I face the odd problem that some of my p2.xlarge instances lost GPU support with no apparent cause. Today, I was able to recreate the problem with a minimal example that should be reproducible for everyone.Basically, after shutdown and reboot, the instance no longer has the nvidia module loaded in the kernel. Furthermore, according to dmesg, there seems to be a different kernel loaded. All of this happens without me actively causing it.Here are the steps to reproduce the problem using a fresh instance and no custom code. I am working in Ireland (eu-west-1), the instance was launched in the Availability Zone eu-west-1a:Launched an instance with the "Deep Learning AMI (Ubuntu) Version 21.2 (ami-0e9085a8d461c2d01)Instance type: p2.xlarge, all defaultsLogged into instance, only ran the following four commands:ubuntu@...:~$ lsmod | grep nvidianvidia              16592896  0ipmi_msghandler        49152  1 nvidiadmesg | less...[    0.000000] Linux version 4.4.0-1075-aws (buildd@lgw01-amd64-035) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) ) #85-Ubuntu SMP Thu Jan 17 17:15:12 UTC 2019 (Ubuntu 4.4.0-1075.85-aws 4.4.167)[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-1075-aws root=UUID=96950bba-70e8-4a4b-9d78-d2bc1c767e04 ro console=tty1 console=ttyS0 nvme.io_timeout=4294967295...ubuntu@...:~$ nvidia-smiTue Mar 19 16:41:53 2019+-----------------------------------------------------------------------------+| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     ||-------------------------------+----------------------+----------------------+| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC || Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. ||===============================+======================+======================||   0  Tesla K80           On   | 00000000:00:1E.0 Off |                    0 || N/A   42C    P8    32W / 149W |      0MiB / 11441MiB |      0%      Default |+-------------------------------+----------------------+----------------------++-----------------------------------------------------------------------------+| Processes:                                                       GPU Memory ||  GPU       PID   Type   Process name                             Usage      ||=============================================================================||  No running processes found                                                 |+-----------------------------------------------------------------------------+ubuntu@...:~$ sudo shutdown nowThe instance does not shut down right away, maybe it is running updates that however I have NOT actively triggered.After the state showed "stopped", started the instance again via the AWS Management ConsoleRan the first three commands:ubuntu@...:~$ lsmod | grep nvidia(no output)dmesg | less...[    0.000000] Linux version 4.4.0-1077-aws (buildd@lcy01-amd64-021) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) ) #87-Ubuntu SMP Wed Mar 6 00:03:05 UTC 2019 (Ubuntu 4.4.0-1077.87-aws 4.4.170)[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-1077-aws root=UUID=96950bba-70e8-4a4b-9d78-d2bc1c767e04 ro console=tty1 console=ttyS0 nvme.io_timeout=4294967295...ubuntu@...:~$ nvidia-smiNVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.Apparently, the kernel seems to change from version 4.4.0-1075-aws to version 4.4.0-1077-aws. What is causing this? How can I prevent it?FollowComment"

Answer: "0By now, I know about three other people having the same problem. But nobody found a solution yet. Can we get a statement from AWS on this?CommentSharesgasseanswered 4 years ago0Same issue here :/nvidia-smiNVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.lsmod | grep nvidia(no output)dmesg | grep "Linux version"0.000000 Linux version 4.4.0-1077-aws (buildd@lcy01-amd64-021) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) ) #87-Ubuntu SMP Wed Mar 6 00:03:05 UTC 2019 (Ubuntu 4.4.0-1077.87-aws 4.4.170)Edit:Same after updating packages with:sudo apt-get updatesudo apt-get upgradeEdited by: aureliencluzeau on Mar 26, 2019 5:51 AMCommentShareaureliencluzeauanswered 4 years ago0I'm having the same issue :'''NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver'''The solution seems to manually download and update the kernel as well as the  Nvidia driver, but I dont want to mess with my setup unless I absolutely have to.CommentSharekunalguptaanswered 4 years ago0I'm having the same issue on p3.2xlarge instances.Related: https://stackoverflow.com/questions/40201165/nvidia-drivers-not-running-on-aws-after-restarting-the-amiCommentSharepapjulianswered 4 years ago0Hello sgasse,As you have correctly observed, the Deep Learning Ubuntu AMI family you are using autonomously performs updates in the background.You can see the systemd timers/services responsible for this by running:systemctl list-units --all | grep aptThis happens independently of the instance type used.In the case you and others in this thread described, the unattended upgrade must have included a newer kernel which became active when the instance was rebooted. Normally, the Dynamic Kernel Module Support (DKMS) framework would automatically ensure that external kernel modules are recompiled to work with the updated kernel. However, with the NVIDIA driver's DKMS configuration used in older versions of the Deep Learning Ubuntu AMI, the recompilation would fail if gcc was also updated.This has been addressed in "Deep Learning AMI (Ubuntu 18.04) Version 28.0" (and later) by including the --no-cc-version-check option when installing the NVIDIA driver as part of the image build process, e.g.:sudo sh ./nvidia_driver.run --no-drm --disable-nouveau --dkms --silent --no-cc-version-check --install-libglvndOn an older AMI, you can apply a workaround without having to re-install the driver (replace "440.33.01" with your actual version):Open /usr/src/nvidia-440.33.01/dkms.confAdd IGNORE_CC_MISMATCH=1 to the MAKE commandRun dkms install -m nvidia/440.33.01Note: We have made NVIDIA aware that this option may also feasibly lead to recompilation failures in the future if an updated gcc has a change that is not backwards-compatible with an older driver's build process. There is not yet a perfect solution here that is guaranteed always fully transparent to customers, while keeping instances as secure as possible.StanislavCommentShareStanislav-at-awsanswered 3 years ago"


Question: SQS long polling: messages delayed

"Hi,I use SQS standard queues with long polling via a Python client. Recently I've been seeing delays between when the message is successfully sent to the queue and when the message is received by the consumer, where the delay is approximately equal to the visibility timeout. Please see the simple example below, keeping in mind the following:•  There are no other consumers for this queue in this example•  When a message is ultimately received with delay, ApproximateReceiveCount=2 and ApproximateFirstReceiveTimestamp ~=SentTimestamp•  Debug-level logs in boto3 do not indicate that the delayed message was ever received in any way other than in the ultimate delayed receipt observed by my code•  A delay will be observed in ~30% of runs, and is always for one of the first few messages sent (have never seen it later than the 5th message after running for a long time)#!/usr/bin/env python3import osimport sysimport timeimport boto3import loggingimport threading# Logging setup omittedtimes = {}prefix = os.urandom(4).hex() + "_"def sender(client, url):    # Send a message every 2 seconds, recording the send time per message    global times    i = 0    while True:        body = prefix + str(i)        times[body] = time.monotonic()        client.send_message(QueueUrl=url, MessageBody=body)        i += 1        time.sleep(2)def recver(client ,url):    # Loop over receive_message forever on a long poll    global times, total    while True:        resp = client.receive_message(QueueUrl=url,                                      MaxNumberOfMessages=10,                                      VisibilityTimeout=5,                                      WaitTimeSeconds=20,                                      AttributeNames=["All"])        print("Got resp")        for msg in resp["Messages"]:            # Delete the received messages            print(f"Msg attrs: {msg.get('Attributes')}")            handle = msg["ReceiptHandle"]            client.delete_message(QueueUrl=url, ReceiptHandle=handle)            # See how long it took using the time set in the sender            body = msg["Body"]            try:                start = times.pop(body)            except KeyError:                print(f"ERROR: key missing: {body}")                continue            delta = time.monotonic() - start            print(f"{body} took {delta}")            if delta > 1:                print(f"LONG DELAY: body={body} delay={delta}")                os._exit(1)if __name__ == "__main__":    # Start up a sender and receiver thread    cli = boto3.client("sqs", region_name='us-east-1')    url = cli.get_queue_url(QueueName="test_py")["QueueUrl"]    threading.Thread(target=recver, args=(cli, url), daemon=True).start()    threading.Thread(target=sender, args=(cli, url), daemon=True).start()    time.sleep(24 * 60 * 60)Please let me know if I can provide any more info. Thanks in advance for your help!FollowComment"

Answer: "0How much do you wait before executing two runs of this same test code?Your test code is using the same queue name for each run. On shutdown it does not seem to gracefully wait for the receive calls to finish. This means that when your test completes and you immediately start another copy of it, the first few messages you send into the queue are returned to the long-poll receive requests started by your previous test run, that you aborted waiting for, but were still issued to SQS.To avoid the old long-poll receives from returning messages from new sends, consider either using a unique queue name for each test run or waiting at least 20s between runs so that all old long polls complete as empty.CommentShareKubaanswered 4 years ago0You're right, that was the exact problem. Thanks for the help.CommentSharechrisdotkamaanswered 4 years ago"


Question: Is there a programmatic way to download the EC2 Instance Recommendations from Migration Hub?

"I am able to import server discovery through the Application Discovery Service API, however I cannot download the EC2 instance recommendations as I cannot find any API that supports this. Is there a way to do it programmatically?Thank you.FollowComment"


Question: Welcome to the Amazon DevOps Guru Forum!

"Hello!We welcome you to leave questions and comments about our service. The service team will be monitoring the forum and we hope to build a community of users who can help others who have questions about the service.Since this is the first post on the forum, we'll start things off by asking what is your use case and what do you plan on using Amazon DevOps Guru to analyze?Thank you!Amazon DevOps Guru teamEdited by: Jarrod@aws on Dec 4, 2020 8:36 AMEdited by: Jarrod@aws on Dec 4, 2020 8:39 AMFollowComment"

Answer: "0Hi,Can we use/integrate aws code guru to analyze repository in azure DevOps repo? Can you let us know how?CommentShareJim28answered 2 years ago0Hi Jim,I asked this question to the CodeGuru Reviewer team (this is a forum for the DevOps Guru service) and they told me that while CodeGuru Reviewer doesn't support Azure DevOps repo, they do support CodeCommit, GitHub, CitHub Enterprise, and BitBucket. If you have your repos on any of those providers, you can easily associate your repos with CodeGuru Reviewer.If you have further questions about the CodeGuru Reviewer service, you can visit their forum at here.JarrodEdited by: Jarrod@aws on Jan 19, 2021 4:17 PMEdited by: Jarrod@aws on Jan 19, 2021 4:17 PMCommentShareJarrodAWSanswered 2 years ago"


Question: Test IOS error "Failed to find matching arch for 64-bit Mach-O input file"

hiWe are doing a trial of the Device Farm and are having issues running against IOS devices.We are running a Built-InFuzz and devices return the error "Install failed. Got error "IncorrectArchitecture" with code 0x00000000: Failed to find matching arch for 64-bit Mach-O input file".Any suggestions?We use AWS Device Farm Sample App for iOS.https://github.com/aws-samples/aws-device-farm-sample-app-for-iosRun URL:https://us-west-2.console.aws.amazon.com/devicefarm/home?region=us-east-2#/projects/9c1b5592-3116-43ec-8f3c-67a8f685db65/runs/051c2bdb-13e7-4288-a58b-4654ece510ceFollowComment

Answer: "0Hello,I was unable to install your build on either a 32 bit phone nor a 64 bit phone. Could your try to build with build for testing?To do this From xcode, using the top menuProduct > Build For > TestingOr from CLIxcodebuild build-for-testing -workspace <PATH>.xcworkspace -scheme <YOUR_SCHEME>Then repackaging and trying your app again.CommentShareAWS-User-1182145answered 4 years ago0HiI tried to build with build for testing, but device returned same error.Run URL:https://us-west-2.console.aws.amazon.com/devicefarm/home?region=us-east-2#/projects/9c1b5592-3116-43ec-8f3c-67a8f685db65/runs/15096bc4-8248-4fa5-8039-f39ede68bfebthank you.CommentSharenaruakianswered 4 years ago0hi,sorry, I didn't select an "iOS Device" , before build for Running.now, runs successfully.Thanks!!CommentSharenaruakianswered 4 years ago"


Question: Your instance encountered an error and has closed the connection. Try again or contact customer support.  UPSTREAM_ERROR [515]

"I have magento 2 installed, for a few days I have this problem I don't know how to solve it, I need urgent help if someone has mercy, I can't advance in my project help, Amazon LightsailFollowComment"

Answer: 0Please check out the following links where your issue is addressed.https://aws.amazon.com/premiumsupport/knowledge-center/lightsail-resolve-ssh-console-errors/https://forums.aws.amazon.com/thread.jspa?messageID=965197&tstart=0CommentShareSandeepVudataanswered a year ago


Question: An error occurred (AccessDenied) when calling the PutBucketPolicy operation: Access Denied

"Hello,aws CLI : aws s3api put-bucket-policy --bucket compty08052023 --policy file://public-access-policy.jsonAn error occurred (AccessDenied) when calling the PutBucketPolicy operation: Access DeniedAmazonS3FullAccessAWS managedDirectly i also tried AdministratorAccess and even to create policy like : {"Version": "2012-10-17","Statement": [{"Sid": "S3BucketManagement","Effect": "Allow","Action": ["s3:CreateBucket","s3:DeleteBucket","s3:ListAllMyBuckets"],"Resource": ""},{"Sid": "S3BucketPolicyManagement","Effect": "Allow","Action": ["s3:PutBucketPolicy","s3:GetBucketPolicy","s3:DeleteBucketPolicy"],"Resource": "arn:aws:s3:::"}]}but nothing worked. Don t know what to do.FollowComment"

Answer: "0Hello,When I have seen Access Denied like this, it's the credentials that you are using to perform the operation and not what the policy looks like.  Based on the example above you are using the default user credentials for your CLI on the host that you are using.  I say that because there is no --profile option to the CLI statement above. So what access does the account that you are using have?   In the documentation https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketPolicy.html. it states.If you don't have PutBucketPolicy permissions, Amazon S3 returns a 403 Access Denied error.Which is what you are receiving when making the call.  If that's the issue, you only need to grant the account that you are using PutBucketPolicy right, or something more general like PowerUserCommentShareCraig Simonanswered a month ago"


Question: Receiving consistent AccessDenied errors

"I am trying to use SageMaker Notebook Instances, but consistently receive AccessDenied errors for commands that my IAM role should have access to (and for commands that worked the last time I tried several weeks ago).  For example:aws s3 ls results in An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied despite my role having the AmazonS3FullAccess policy attached.Also aws ecr describe-repositories --repository-names "sagemaker-decision-trees" results in An error occurred (AccessDeniedException) when calling the DescribeRepositories operation: User: arn:aws:sts::XXXXXXXXXX:assumed-role/AmazonSageMaker-ExecutionRole-20201123T151452/SageMaker is not authorized to perform: ecr:DescribeRepositories on resource: arn:aws:ecr:us-east-2:XXXXXXXXXX:repository/sagemaker-decision-trees with an explicit deny despite my role having the AmazonEC2ContainerRegistryFullAccess policy attached.One thing that seems new is that "SageMaker" is appended to my user ARN.  I can't remember seeing errors with this appended before.Note: I've replicated these errors with several combinations of configurations:a new IAM role (which I created in the SageMaker console to have AmazonSageMakerFullAccess to any S3 bucket)fresh notebook instancewith (and without) a VPCAlso, these commands all work when run outside of a notebook instance (i.e. when run locally from my laptop).I'm guessing there's some problem with my account setup, but not sure what to try next.Thanks.Edited by: DJAIndeed on Nov 24, 2020 8:35 AMFollowComment"

Answer: "0I am sorry to hear that you are not able to access required services from  Sagemaker.You can run below command to check the execution role that is getting used and then verify that required permission are present.import sagemaker  sagemaker.get_execution_role()  Other useful links -https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.htmlhttps://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/Edited by: amitsur on Dec 22, 2020 2:41 PMCommentShareamitsuranswered 2 years ago0Thanks, @amitsur.  We had confirmed that the SageMaker Notebook Instance was using the desired execution role and that that role had the required permissions.  The issue appears to have resolved itself though, since we're no longer receiving these errors.  So it must have been a configuration elsewhere?  We appreciate the help anyway.CommentShareDJAIndeedanswered 2 years ago"


Question: Unable to use IAM permissions to access MSK Brokers

"I have been experimenting with a cluster that has IAM Authentication, and I cannot seem to get it working.-I have a security group in the cluster that allows in-bound traffic from the ec2 instance I am testing from. I can even do zookeeper interactions like list topics just fine.-My ec2 instance has an IAM role with a policy that specifically allows for all kafka interactions on all resources-I also tried an aws local profile that has the same attached policy.-I am using the following command to attempt a consumer interactionbin/kafka-console-consumer.sh --bootstrap-server b-1.examplename.kafka.us-east-1.amazonaws.com:9098 --topic exampleTopic --consumer.config config/consumer.propertiesconsumer.properties has the below propertiessecurity.protocol=SASL_SSLsasl.mechanism=AWS_MSK_IAMsasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandlerAm I missing anything?FollowComment"

Answer: "0Hey there,I had same issue.Follow this guide: https://aws.amazon.com/blogs/big-data/securing-apache-kafka-is-easy-and-familiar-with-iam-access-control-for-amazon-msk/Best RegardsCommentSharedbartashevichanswered 2 years ago"