Question: Aurora MySQL crashes after update from 2.10.1 to 2.11.0 with many hanging "unauthenticated user"

"Hi,After updating Aurora MySQL from 2.10.1 to 2.11.0 on Sunday the following problem occurred immediately. Suddenly users (all internal from one IP) can no longer log in. Existing connections are not affected. Later I was able to see in the process list a lot of hanging "unauthenticated user" with a console connection created before the problem. If I try to kill such a process, the status changes to 'killed' but it keeps hanging unlimited. A reboot fixes the issue for a few minutes. Then again no users are able to login. It's a production system (db.r5.xlarge) with rather high traffic - only one internal application (no public web site). I created a clone Sunday night and it was fine for 48 hours. Then again. I suspect that it's caused by a certain pattern of many users at a time because a reboot helped only for a minute. I created another clone, stopped the application before, started it and it worked for 1 hour. Then again. There are no errors in the log, CPU is low and memory is plenty available. I created another clone with a larger machine and switched off some clients that are mainly reading. So far good since 2 hours. It seems like a bug in version 2.11.0. All recommended OS updates are applied.This is what I found on the web - similar symptoms:https://dba.stackexchange.com/questions/291167/many-unauthenticated-user-on-mysql-processlistUnfortunately no solution.Many thanks for any suggestions,AndreasFollowCommentrePost-User-3374891 4 months agoI'm experiencing the exact same problem with an instance of Aurora MySQL 2.11.0 / 5.7.12-log after an update from 2.10.2 and applying the recommended OS update associated with this release.When an application tries to open multiple connections concurrently, they may get stuck in state 'login'.Show full processlist shows status like this:"187"     "unauthenticated user" "10.90.100.5:57551"        \N           "Connect"           "89"       "login"  \N"188"     "unauthenticated user" "10.90.100.5:57550"        \N           "Connect"           "89"       "login"  \N"189"     "unauthenticated user" "10.90.100.5:57549"        \N           "Connect"           "89"       "login"  \N"190"     "unauthenticated user" "10.90.100.5:57553"        \N           "Connect"           "89"       "login"  \N"191"     "unauthenticated user" "10.90.100.5:57552"        \N           "Connect"           "89"       "login"  \N"192"     "unauthenticated user" "10.90.100.5:57554"        \N           "Connect"           "89"       "login"  \NSharerePost-User-3374891 4 months agoAn additional observation: When this happens, calls like "SELECT * FROM INFORMATION_SCHEMA.GLOBAL_STATUS" and "SELECT VARIABLE_NAME, VARIABLE_VALUE FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME IN ( 'AURORA_VOLUME_BYTES_LEFT_TOTAL')" (the later is called by rdsadmin) will also block, and not complete till the application initiating the stuck connections is stopped.Share"

Answer: "2Updating to 2.11.1 have fixed the problem for me.CommentSharerePost-User-3374891answered 3 months agorePost-User-6479663 3 months agoThank you very much. We are going to try it. In our case the problem was caused by special BI/Reporting clients, only. So we moved them temporarily on a green replication that we updated to Aurora 3.Share"


Question: G5 EC2 instances do not have a proper driver to talk to GPU

"I started a G5.xlarge and a 2xlarge instance on ec2.However, when I tried to check on the GPU status via the nvidia-smi command, it errored with:NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.Why is this happening? G4 instances worked fine though. Seems to be just g5?FollowComment"


Question: Email still marked as bounced on Amazon SES

We use Amazon SES to send weekly newsletter to all of our employees email address. One of the employee left the company but we did not remove this person email address from the newsletter mailing list and it bounced. The employee returned back to the company several months later and the employee was assigned with same email as before.I removed the bounce tag off the mailing list that we use to send out the newsletter and I also went to the suppression removal list on Amazon and entered the email address to remove the email but Amazon is still marking it as bounce on each test we tried to send it out.Is there a way for us to see the suppression list to verify that our employee email address is not on it and why is Amazon still marking it bounce after we submitted the email address to be removed from the suppression list?BillFollowComment

Answer: 0Accepted AnswerI had to enable the new console to be able to see the new account suppression list which fixes my issue I was having.CommentShareTechBillanswered a year ago0Did you review both the global suppression list and the account-level suppression list?   see https://docs.aws.amazon.com/ses/latest/dg/lists-and-subscriptions.html.CommentSharemdibellaanswered a year agoTechBill a year agoI read the document again and now I am able to find the suppression list to be able to take the that email off the list.  What I missed was that I had to set the console to the new console then the suppression list option became available for me to use.Thank you for your prompt reply and answer!Share


Question: Unreal -- unable to package for linux on windows.

"In AttributeValue.h, line 309 and line 320,m_key[MAX_STRING_LENGTH] = '\0';m_key is a value with Type AttributeStringType. AttributeStringTypeis defined astypedef char AttributeStringType[MAX_STRING_LENGTH];When I compile developer Server for linux. there is an errorSeverityCodeDescriptionProjectFileLineSuppression StateErrorarray index 101 is past the end of the array (which contains 101 elements) [-Werror,-Warray-bounds]projectname projectname\Plugins\GameLiftServerSDK\Source\GameLiftServerSDK\Public\aws\gamelift\server\model\AttributeValue.h309How can I solve this problem? Plz help me. Sorry, my English is very bad.FollowComment"

Answer: "0Hey there, sorry that you are running into trouble here. This looks like a bug we are currently aware of on our end - looking to include the fix for it in our next SDK release. In the meantime, could you try adding "-1" after "MAX_STRING_LENGTH" to see if that helps? For example, code for m_key should look like:m_key[MAX_STRING_LENGTH - 1] = '\0';CommentSharerePost-User-5697045answered 5 years ago0Do you know when the next SDK release is?CommentSharerePost-User-5448528answered 5 years ago0I'm getting the same issue which is preventing me from building my UE4 game's dedicated server for LinuxServer while on Windows.C:/Genesis/Krieg 4.20/Plugins/GameLiftServerSDK/Source/GameLiftServerSDK/Public\aws/gamelift/server/model/AttributeValue.h(309,13): error: array index 101 is past the end of the array (which contains 101 elements) [-Werror,-Warray-bounds]Also getting these errors:    C:/Genesis/Krieg 4.20/Plugins/GameLiftClientSDK/Source/GameLiftClientSDK/Public\aws/gamelift/GameLiftErrors.h(74,3): error: '__declspec' attributes are not enabled; use '-fdeclspec' or '-fms-extensions' to enable support for __declspec attributesC:/Genesis/Krieg 4.20/Plugins/GameLiftClientSDK/Source/GameLiftClientSDK/Public\aws/gamelift/model/AcceptMatchResult.h(35,25): error: expected ';' after top level declaratorC:/Genesis/Krieg 4.20/Plugins/GameLiftClientSDK/Source/GameLiftClientSDK/Public\aws/gamelift/model/Alias.h(46,9): error: redefinition of 'dllimport'class AWS_GAMELIFT_API Alias^C:/Genesis/Krieg 4.20/Plugins/GameLiftClientSDK/Source/GameLiftClientSDK/Public\aws/gamelift/GameLift_EXPORTS.h(38,41): note: expanded from macro 'AWS_GAMELIFT_API'#define AWS_GAMELIFT_API __declspec(dllimport)^C:/Genesis/Krieg 4.20/Plugins/GameLiftClientSDK/Source/GameLiftClientSDK/Public\aws/gamelift/model/RoutingStrategy.h(58,9): note: previous definition is hereclass AWS_GAMELIFT_API RoutingStrategy^C:/Genesis/Krieg 4.20/Plugins/GameLiftClientSDK/Source/GameLiftClientSDK/Public\aws/gamelift/GameLift_EXPORTS.h(38,41): note: expanded from macro 'AWS_GAMELIFT_API'#define AWS_GAMELIFT_API __declspec(dllimport)CommentSharerePost-User-5448528answered 5 years ago"


Question: Why does the EC2 Auto Scaling Group automatically generate Scheduled Actions?

"For some reason the EC2 Auto Scaling Group automatically generates Scheduled Actions and so limiting the maximum EC2 instances launched. Everytime I have to delete the scheduled actions to prevent limiting the maximum EC2 instances launched.FollowCommentben-from-aws a year agoHi, could you please explain how you created or provisioned the Auto Scaling Group?ShareAWS-User-9256134 a year agoHey Ben,I thought by simply creating a new auto scaling group from an existing launch template.ShareA-Shevchenko a year agoWhen you say "everytime I have to delete the scheduled actions" - what everytime actually means? Do oyu have to delete it for each new ASG or you have to do it from time to time for the same ASG?Shareben-from-aws a year agoCould you share the configuration of your launch template?Sharekennyk a year agoScheduled actions are something deliberately created (https://docs.aws.amazon.com/autoscaling/ec2/userguide/schedule_time.html).  It sounds as if you have created a recurring scheduled action and are wondering why it is recurring.  Back to ben-from-aws's comment, can you give us more information on how you created the ASG?ShareShow 1 more comment"

Answer: "0Accepted AnswerStupid me, I had an Auto Scaling Plan active, that generates the Scheduled Actions.@Amazon: Maybe there can be a new feature that shows active Auto Scaling Plans for a specific Auto Scaling Group.Thanks all for your support.CommentShareAWS-User-9256134answered a year ago"


Question: How to avoid charges for Simple Storage Service in S3

"Hi,All my files are saved in Glacier Deep Archive. How come I am still getting charged monthly for Simple Storage Service?Thank you.FollowComment"

Answer: "0Sorry to hear you're seeing these charges.You can follow the steps in this article to track down the resources at fault: http://go.aws/resources-unexpected-chargesYou can also each out to our Accounts and Billing experts via your Support Center: http://go.aws/tech-support.Regards,- Randi S.CommentShareEXPERTAWS Support - Randianswered 3 months ago0Hello,Sorry to hear you're seeing charges you were not expecting.First, to help you find where these charges are coming from, you can use AWS Cost Explorer.For your scenario, you can filter to services "S3 (Simple Storage Service)" and "Glacier"; also set your Group by Dimension to "Usage type". This will show all Glacier and S3 costs and group them by type of charge (for example, storage volume per class, API requests, etc).Second, to clarify S3 and Glacier charges, it's important to understand that there is a difference between Amazon S3 Glacier (formerly known as Amazon Glacier) and the Glacier storage classes in Amazon S3.Although the names are similar, Amazon S3 Glacier is a separate service from Amazon S3, with distinct APIs and individual billing. When you use Glacier storage classes in an S3 bucket, you will be billed through S3 - not through Glacier.As a result, when you use the Cost Explorer as directed above (filter to S3 & Glacier, group by Usage type), you will see line items for "TimedStorage-ByteHours", where '' reflects the S3 storage class. For example:GDA = Glacier Deep ArchiveGIR = Glacier Instant RetrievalGLACIER = Glacier Flexible Retrieval (fka. Glacier)etc.If you have your objects in S3 Glacier storage classes, you'll see that they are being billed under Amazon S3 at the appropriate Glacier storage class pricing rate.One additional caveat is that Amazon S3 Glacier storage classes have a minimum object size of 128 KB. Objects smaller than 128 KB will be charged for a minimum of 128 KB; there is also a small additional overhead for index and metadata. See the annotations on the S3 Storage pricing table.CommentShareMichael_Ranswered 3 months ago"


Question: I need to decommission multiple workspace in multiple aws account. Is there is any boto3 script available to perform this operation.

I need to decommission multiple workspace in multiple AWS account. Currently i have a boto3 script which will terminate the workspace in only one account. I want to terminate multiple worksapce on multiple account. Could anyone help me with the boto3 script to perform this operation.FollowComment

Answer: "1Accepted AnswerI have used the botocove decorator to run a function in multiple accounts.CommentShareJeremy_Sanswered 7 months agoAWS-User-2038220 7 months agothanks for sharing it.. Very useful..by anychance, could you please share the model code\script for the same. it will be super helpfulShare"


Question: Problem with Kinesis Video Stream PutMedia OPTIONS request.

"Hello,I got 404 on POST https://s-######a.kinesisvideo.eu-central-1.amazonaws.com/putMedia request:OPTIONS request:https://user-images.githubusercontent.com/86299738/145280743-18ab38a7-0981-44e1-b3ca-fab52bcca98b.pngMain request(which induces the OPTIONS request):https://user-images.githubusercontent.com/86299738/145280462-9bbca7b5-a68d-41fd-aa27-6e2827250a6a.pngThis documentationhttps://docs.aws.amazon.com/kinesisvideostreams/latest/dg/API_dataplane_PutMedia.htmldoesn't state whether the endpoint should handle the OPTIONS request or not.Should I expect a response on the KVS <DataEndpoint>/putMedia OPTIONS request?Detailed info about my attempts in this issue https://github.com/aws-samples/amazon-kinesis-video-streams-demos/issues/143#issuecomment-988313055Is it still possible to stream to KVS from a browser?Thank youFollowComment"


Question: Connect to AWS GreenGrass without aws device SDK

"I have a device, who can connect to MQTT broker. I just need to provide an endpoint of the MQTT broker. I cannot change code on that device. and I have greengrass running, on IP 192.168.1.12 on port 8883 (windows machine). i downloaded Group CA using basic discovery api:https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-discover-api.html so i used this mqtt settings for device:broker endpoint (192.168.1.12:8883)topic (same as registered in subscription in aws iot GG)device certificate and private key (the device is registered in aws iot GG)Group CA (downloaded by basic discovery sample application)but i am still not able to connect to aws gg?my understanding is aws GG also has a MQTT broker. so by providing proper certificates and topic, i should be able to connect to it, whether or not I have aws device SDKs. Am I wrong?FollowComment"

Answer: "0Hi omki12,Correct me if I understand wrong. You have 2 devices in your setup. One is a windows machine running Greengrass (core device). The core device is able to connect to AWS IoT successfully. The second device is set up to connect to the core device, but it failed to.I have a few questions based on this assumption.Were you able to add the second device to the Greengrass group and successfully deploy the group to the core device?Can you provide the logs from Greengrass core device? You can find the Greengrass logs here https://docs.aws.amazon.com/greengrass/latest/developerguide/greengrass-logs-overview.html#gg-logs-localHow did you initiate the connection from the second device to core device? Can you provide the output of it? Have you tried to use device SDK and what version did you use?I'm assuming you're running Greengrass in a docker container, because windows is not currently supported. Which version of Greengrass are you running?CommentShareAWS-User-5233880answered 3 years ago0Hi there,thank you for your reply.I added the second device in GG group and deployed the group successfully.If I use AWS IoT device SDK , I am able to connect to core device (GG).I tested the GG using basic Discovery Example provided by amazon: https://github.com/aws/aws-iot-device-sdk-python/blob/master/samples/greengrass/basicDiscovery.pyBut the issue is I cannot use AWS IoT Device sdk on the actual device. let's suppose I am using the MQTT-paho library on actual device. is it possible to connect to AWS GG using an mqtt-paho library, given all the certificates and topic is valid as mentioned in the GG group?yes, I am using docker on windows and using the latest version of Greengrass. GG is up and running. tested using "Basic Discovery Example" provided by amazon.CommentShareomki12answered 3 years ago0Maybe something likehttps://vixtera.com/software-appliance/will translate your payloads to something that AWS GG can understand.CommentShareuwegambitcommanswered 3 years ago0Thanks for explaining your use case.The mqtt settings you mentioned in the post seem find to me. Just to confirm one thing, did you also pass in the device thing name as client ID?Also what error messages do you get?Thanks,HuiCommentShareAWS-User-5233880answered 3 years ago0yes.. that was the issue. I was passing a random client id.instead, I passed "Device name" as client id and it worked.thank you.. :DCommentShareomki12answered 3 years ago"


Question: EFS metrics not showing and EC2 cannot write on EFS

I was moving many files from S3 to EFS using my website software on EC2 and after many files were moved EFS was not accepting more writes.EFS metrics are not available on AWS console.Throughput utilization is RED and shows a 75 warning. The other metrics have no data.What can I do?FollowComment

Answer: "0Possible reason for slowness in files copy is EFS throughput utilization might have reached to its max limit..any specific error you are getting ? However, You should be able to see the metrics for your EFS as described in [1] using AWS CloudWatch console.Please note that if your EFS is bursting type throughput mode then the baseline throughput value is depends on capacity stored in standard class and if its provisioned type then you can set the throughput number irrespective of EFS storage used. Please refer [2] for more on this..If you still have any further issues, I would suggest to reach-out to support for further troubleshooting.[1]: https://docs.aws.amazon.com/efs/latest/ug/efs-metrics.html[2]: https://docs.aws.amazon.com/efs/latest/ug/performance.htmlCommentShareRRP_AWSanswered 9 months ago"


Question: Call to Textract API issue - UnknownOperationException

"Hello,I want to call textract DetectDocumentText API from Salesforce using a base64 string that represents a black png.I tried using the following http request:Authorization headers using aws signature with secret and keyEndpoint: https://textract.us-east-2.amazonaws.com/DetectDocumentTextMethod: Tried GET/POST/PUT but I am not sure which one to use.JSON:{"Document": {"Bytes": "iVBORw0KGgoAAAANSUhEUgAAAGQAAAAKCAQAAADoFTP1AAAAIklEQVR42mNk+MkwLADjqEdGPTLqkVGPjHpk1COjHmFgAAAiwgnFqUltyAAAAABJRU5ErkJggg=="}}also tried with S3 bucket file and JSON:{"Document": {"S3Object": {"Bucket": "mybucket","Name": "filename.jpeg"}}}Result: 200 with POST method => response: {"Output":{"__type":"com.amazon.coral.service#UnknownOperationException"},"Version":"1.0"}Apart from this, I tried calling S3 API from Salesforce and successfully uploaded a file. There, I am using the same authorization method so that seems to be fine I guess.Not sure what I am doing wrong or if something is missing in my http request.Thanks for your help.Edited by: AgustinB on Aug 31, 2021 3:12 PMEdited by: AgustinB on Aug 31, 2021 3:18 PMFollowComment"

Answer: "0Thank you for using Amazon Textract. Can you share the code that you're using to make the HTTP request so that we can help you better? The endpoint shouldn't contain any path appended for different APIs. The API you'd want to use must be set with x-amz-target: Textract.DetectDocumentText. Could you try with this and let us know how that goes?Also, we encourage to use the AWS SDKs. Are you facing any issues with SDK or is there a use case that you're not able to achieve with the SDK?CommentShareraghuAtAWSanswered 2 years ago0Thanks for your reply raghuataws.I tried appending x-amz-target: Textract.DetectDocumentText to the headers but same error.HTTP request from Salesforce:HttpRequest req = new HttpRequest();req.setEndpoint('callout:TEXTRACT');req.setMethod('POST');req.setHeader('Content-Type', 'application/json');req.setHeader('X-Amz-Target', 'Textract.DetectDocumentText');req.setBody('{ "Document": { "Bytes": "", "S3Object": { "Bucket": "", "Name": "", "Version": "" } } }');Note that the endpoint is set as a Named Credential in Salesforce with the following details:URL:https://textract.us-east-2.amazonaws.comIdentity Type: Named PrincipalAuthentication Protocol: AWS Signature Version 4AWS Access and Secret key from my user in AWSAWS Region: us-east-2AWS Service: TextractThis automatically generates an authorization header when making the callout.thank you!CommentShareAgustinBanswered 2 years ago0Can you please share the request ID and region info so we can take a look at our log? or the complete code snippet so that we can run and reproduce the issue.CommentShareAWS-User-4977386answered 2 years ago0Thanks for your reply, awscaesar.Request ID: 311e933c-e658-4773-8a66-8486e65ec812Region: us-east-2To clarify, I am not doing this through AWS SDK. I want to call Amazon Textract API directly from Salesforce with an HTTP request callout.As explained previously, I was able to call the S3 API from Salesforce directly and was able to upload a file. So, unless you tell me it is mandatory to use the AWS SDK for Textract, then something is wrong with the request I am sending.Thanks again.Regards,CommentShareAgustinBanswered 2 years ago0Hi AgustinB, from our log, seems like the request doesn't contain access key and account id, probably need to fix the syntax. But it's also a miss from service side, Textract should not return 200, but return 400/4xx instead with helpful debugging message.I'll report the issue to the right service team to fix it.CommentShareAWS-User-4977386answered 2 years ago0Hello awscaesar,Thanks for taking a look into this.I don't understand why the access key and account id are not in my request.For the authorization I am using AWS signature method with the following parameters:AccessKey = ******SecretKey = ******AWS Region = us-east-2Service Name = textractCan the textract service be authorized via this method? Cause it is working for s3 service.Thanks also for reporting the 200 status issue from the service side.Best Regards,CommentShareAgustinBanswered 2 years ago0Hi AgustinB,Can you try content type as application/x-amz-json-1.1 as instructed from https://docs.aws.amazon.com/translate/latest/dg/API_Reference.html ? S3 service may support more protocol types but generally AWS API requires content headers as specified in https://docs.aws.amazon.com/translate/latest/dg/API_Reference.html, otherwise you'd got UnknownOperationException in POST request.Additionally since you are not using AWS SDK, please make sure you sign the request with sigv4 as instructed in https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html.Another way to get the proper headers needed is to use AWS CLI with --debug mode. You can use AWS CLI to call the API, which should print the header information on screen. For example:pic=$(base64 $YOUR_IMAGE_PATH)  doc_string=$( jq -n --arg p "$pic" '{Bytes:$p}')  aws textract detect-document-text --region us-east-2 --document="${doc_string}" --debug  Thanks,SiqiCommentShareAWS-Siqfananswered 2 years ago0Hi Siqi-AWS,Changing the Content-Type to application/x-amz-json-1.1, as you suggested did the trick. It is now working when sending an S3 object to the DetectDocumentText API from Salesforce. Just to clarify, the request was already using sigv4.Really appreciate your help. Many thanks.CommentShareAgustinBanswered 2 years ago"


Question: Invalid SSL Certificate

"Hi, my domain is registered in godaddy and my hosting server is aws, to get ssl certified i have set up my dns server in cloudflare, but it's not working. somehow i get to know that my hostname (mydomain.com) does not match the common name (example.com) so that's why the certificate is invalid for this host.Now, can anyone guide me how can i change this common name?FollowComment"

Answer: "0Accepted AnswerThe SSL Certificate bears the hostname whatever you keyed in as you CN in your CSR (request file). If you want to change it, then you need to rekey you CSR and regenerate the whole certificate and import it again.CommentShareAWS-User-2036647answered a year agoSaibal a year agoThanks for your response,Yes i started all over again, but while generating the domain its showing a notice in cloudflare dashboard that -This zone is in a pending state and contains proxied records. Until the zone is activated, DNS queries to proxied records will expose the origin IP / hostname and traffic cannot be proxied through Cloudflare. For more information refer to Limitations for pending domains.Now, what's that mean, how to activated the zone??ShareSaibal a year agoPending domainsEvery zone onboarded onto Cloudflare will initially be in pending state until we can verify ownership. This means that DNS records are not proxied until your zone has been activated and any requests to your DNS records will return your origin server’s IP address.For enhanced security, we recommend rolling your origin IP addresses at your hosting provider after your zone has been activated. This action prevents your origin IPs from being leaked during onboarding.Share"


Question: Aws Transfer family vpc end point not working

"Hi,Can anyone please help me to take the below yaml template for aws transfer family with internet facing vpc endpoint? Here I vpc endpoint was not working.Please any one run the below template as a stack with your testing account and give me the correct template for me.AWSTemplateFormatVersion: '2010-09-09'Parameters:  EnvironmentName:    Description: An environment name that is prefixed to resource names    Type: String  VpcCIDR:    Description: Please enter the IP range (CIDR notation) for this VPC    Type: String    Default: 10.192.0.0/16  PublicSubnetCIDR:    Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone    Type: String    Default: 10.192.10.0/24  PrivateSubnetCIDR:    Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone    Type: String    Default: 10.192.20.0/24  CreateServer:    AllowedValues:      - 'true'      - 'false'    Type: String    Description: >-      Whether this stack creates a server internally or not. If a server is      created internally, the customer identity provider is automatically      associated with it.    Default: 'true'    Endpointtype:    AllowedValues:      - 'Internal'      - 'Internet facing'    Type: String    Default: 'Internet facing'Conditions:  CreateServer:    'Fn::Equals':      - Ref: CreateServer      - 'true'    Resources:  VPC:    Type: AWS::EC2::VPC    Properties:      CidrBlock: !Ref VpcCIDR      EnableDnsSupport: true      EnableDnsHostnames: true      Tags:        - Key: Name          Value: !Sub ${EnvironmentName} Resources  CloudWatchLoggingRole:    Description: IAM role used by Transfer  to log API requests to CloudWatch    Type: 'AWS::IAM::Role'    Condition: CreateServer    Properties:      AssumeRolePolicyDocument:        Version: '2012-10-17'        Statement:          - Effect: Allow            Principal:              Service:                - transfer.amazonaws.com            Action:              - 'sts:AssumeRole'       GoldcoastTvodUser:    Type: 'AWS::Transfer::User'    Properties:      HomeDirectory: "/goldcoast-tvod"      HomeDirectoryType: "PATH"      Policy:        'Fn::Sub': |          {            "Version": "2012-10-17",            "Statement": {              "Sid": "AllowFullAccessToBucket",              "Action": "s3:*",              "Effect": "Allow",               "Resource": [                "arn:aws:s3:::goldcoast-tvod",                "arn:aws:s3:::goldcoast-tvod/*"              ]            }          }      Role:        'Fn::Sub': 'arn:aws:iam::${AWS::AccountId}:role/TransferManagementRole'      ServerId:        'Fn::GetAtt': TransferServer.ServerId      SshPublicKeys:        - >-          ssh-rsa      UserName: GoldcoastTvodUser  etcsvoduser:    Type: 'AWS::Transfer::User'    Properties:      HomeDirectory: "/etc-svod"      HomeDirectoryType: "PATH"      Policy:        'Fn::Sub': |          {            "Version": "2012-10-17",            "Statement": {              "Sid": "AllowFullAccessToBucket",              "Action": "s3:*",              "Effect": "Allow",               "Resource": [                "arn:aws:s3:::my-bucket",                "arn:aws:s3:::my-bucket/*"              ]            }          }      Role:        'Fn::Sub': 'arn:aws:iam::${AWS::AccountId}:role/TransferManagementRole'      ServerId:        'Fn::GetAtt': TransferServer.ServerId      SshPublicKeys:        - >-          ssh-rsa       UserName: etc-svod-user  etctvoduser:    Type: 'AWS::Transfer::User'    Properties:      HomeDirectory: "/tvn-tvod"      HomeDirectoryType: "PATH"      Policy:        'Fn::Sub': |          {            "Version": "2012-10-17",            "Statement": {              "Sid": "AllowFullAccessToBucket",              "Action": "s3:*",              "Effect": "Allow",               "Resource": [                "arn:aws:s3:::my-bucket",                "arn:aws:s3:::my-bucket/*"              ]            }          }      Role:        'Fn::Sub': 'arn:aws:iam::${AWS::AccountId}:role/TransferManagementRole'      ServerId:        'Fn::GetAtt': TransferServer.ServerId      SshPublicKeys:        - >-      UserName: etc-tvod-user  lhtcsvoduser:    Type: 'AWS::Transfer::User'    Properties:      HomeDirectory: "/lhtc-svod"      HomeDirectoryType: "PATH"      Policy:        'Fn::Sub': |          {            "Version": "2012-10-17",            "Statement": {              "Sid": "AllowFullAccessToBucket",              "Action": "s3:*",              "Effect": "Allow",               "Resource": [                "arn:aws:s3:::my-bucket",                "arn:aws:s3:::my-bucket/*"              ]            }          }      Role:        'Fn::Sub': 'arn:aws:iam::${AWS::AccountId}:role/TransferManagementRole'      ServerId:        'Fn::GetAtt': TransferServer.ServerId      SshPublicKeys:        - >-          ssh-rsa       UserName: lhtc-svod-user  lhtctvoduser:    Type: 'AWS::Transfer::User'    Properties:      HomeDirectory: "/tvn-tvod"      HomeDirectoryType: "PATH"      Policy:        'Fn::Sub': |          {            "Version": "2012-10-17",            "Statement": {              "Sid": "AllowFullAccessToBucket",              "Action": "s3:*",              "Effect": "Allow",               "Resource": [                "arn:aws:s3:::my-bucket",                "arn:aws:s3:::my-bucket/*"              ]            }          }      Role:        'Fn::Sub': 'arn:aws:iam::${AWS::AccountId}:role/TransferManagementRole'      ServerId:        'Fn::GetAtt': TransferServer.ServerId      SshPublicKeys:        - >-          ssh-rsa       UserName: lhtc-tvod-user  mastercopyfoleuser:    Type: 'AWS::Transfer::User'    Properties:      HomeDirectory: "/mastercopyfiles"      HomeDirectoryType: "PATH"      Policy:        'Fn::Sub': |          {            "Version": "2012-10-17",            "Statement": {              "Sid": "AllowFullAccessToBucket",              "Action": "s3:*",              "Effect": "Allow",               "Resource": [                "arn:aws:s3:::my-bucket",                "arn:aws:s3:::my-bucket/*"              ]            }          }      Role:        'Fn::Sub': 'arn:aws:iam::${AWS::AccountId}:role/TransferManagementRole'      ServerId:        'Fn::GetAtt': TransferServer.ServerId      SshPublicKeys:        - >-          ssh-rsa       UserName: mastercopyfole-user    InternetGateway:    Type: AWS::EC2::InternetGateway    Properties:      Tags:        - Key: Name          Value: !Ref EnvironmentName  InternetGatewayAttachment:    Type: AWS::EC2::VPCGatewayAttachment    Properties:      InternetGatewayId: !Ref InternetGateway      VpcId: !Ref VPC  PublicSubnet:    Type: AWS::EC2::Subnet    Properties:      VpcId: !Ref VPC      AvailabilityZone: !Select [ 0, !GetAZs '' ]      CidrBlock: !Ref PublicSubnetCIDR      MapPublicIpOnLaunch: true      Tags:        - Key: Name          Value: !Sub ${EnvironmentName} Public Subnet  PrivateSubnet:    Type: AWS::EC2::Subnet    Properties:      VpcId: !Ref VPC      AvailabilityZone: !Select [ 0, !GetAZs  '' ]      CidrBlock: !Ref PrivateSubnetCIDR      MapPublicIpOnLaunch: false      Tags:        - Key: Name          Value: !Sub ${EnvironmentName} Private Subnet  NatGatewayEIP:    Type: AWS::EC2::EIP    DependsOn: InternetGatewayAttachment    Properties:      Domain: vpc      Tags:        - Key: Name          Value: !Sub ${EnvironmentName} Elsatic Ip  PublicRouteTable:    Type: AWS::EC2::RouteTable    Properties:      VpcId: !Ref VPC    #  Tags:    #    - Key: Name    #      Value: !Sub ${EnvironmentName} Public Routes  PublicSubnetRouteTableAssociation:    Type: AWS::EC2::SubnetRouteTableAssociation    Properties:      RouteTableId: !Ref PublicRouteTable      SubnetId: !Ref PublicSubnet  PrivateSubnetRouteTableAssociation:    Type: AWS::EC2::SubnetRouteTableAssociation    Properties:      RouteTableId: !Ref PublicRouteTable      SubnetId: !Ref PrivateSubnet  myCarrierRoute:    Type: AWS::EC2::Route    #DependsOn: GatewayToInternetAndCarrierNetwork    Properties:       RouteTableId:         Ref: PublicRouteTable       DestinationCidrBlock: 0.0.0.0/0       GatewayId:         Ref: InternetGateway      SecurityGroup:    Type: AWS::EC2::SecurityGroup    Properties:      GroupName: "Production Security Group"      GroupDescription: "Security Group with inbound and outbound rule"       VpcId: !Ref VPC      SecurityGroupIngress:      - IpProtocol: tcp        FromPort: 80        ToPort: 80        CidrIp: 0.0.0.0/0      - IpProtocol: tcp        FromPort: 22        ToPort: 22        CidrIp: 0.0.0.0/0      - IpProtocol: udp        FromPort: 69        ToPort: 69        CidrIp: 96.47.148.171/32      - IpProtocol: tcp        FromPort: 22        ToPort: 22        CidrIp: 3.16.146.0/29      SecurityGroupEgress:      - IpProtocol: tcp        FromPort: 80        ToPort: 80        CidrIp: 0.0.0.0/0      Tags:        - Key: Name          Value: !Sub ${EnvironmentName}  TfVPCInterfaceEndpoint:    Type: 'AWS::EC2::VPCEndpoint'    Properties:      VpcEndpointType: Interface      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.logs'      VpcId: !Ref VPC      SubnetIds:         - !Ref PublicSubnet      SecurityGroupIds:        - !Ref SecurityGroup  TransferServer:    Type: 'AWS::Transfer::Server'    Condition: CreateServer    Properties:      EndpointType: 'VPC'      SecurityPolicyName: TransferSecurityPolicy-FIPS-2020-06      LoggingRole:        'Fn::GetAtt': CloudWatchLoggingRole.Arn      Tags:        - Key: Name          Value: !Sub ${EnvironmentName} Transferserver      EndpointDetails:        VpcId: !Ref VPC        SubnetIds:           - !Ref PublicSubnet        AddressAllocationIds:          - !GetAtt NatGatewayEIP.AllocationId            Tags:        - Key: Name          Value: !Sub ${EnvironmentName} TransferserverFollowComment"

Answer: "1Accepted AnswerIf the security group is not set in "EndpointDetails" of "TransferServer", the default security group of the VPC will be attached.If so, it should result in an error because the inbound rule is something that cannot be connected with the default.You can specify a security group by setting the following.  TransferServer:    Type: 'AWS::Transfer::Server'    Condition: CreateServer    Properties:      EndpointType: 'VPC'      SecurityPolicyName: TransferSecurityPolicy-FIPS-2020-06      LoggingRole:        'Fn::GetAtt': CloudWatchLoggingRole.Arn      Tags:        - Key: Name          Value: !Sub ${EnvironmentName} Transferserver      EndpointDetails:        VpcId: !Ref VPC        SubnetIds:           - !Ref PublicSubnet        SecurityGroupIds:          - !Ref SecurityGroup        AddressAllocationIds:          - !GetAtt NatGatewayEIP.AllocationIdI also think your security group settings are a bit wrong.Modify as follows.Since we are using this security group to access the VPC endpoints in CloudWatch Logs, we needed to allow HTTPS in an inbound rule.  SecurityGroup:    Type: AWS::EC2::SecurityGroup    Properties:      GroupName: "Production Security Group"      GroupDescription: "Security Group with inbound and outbound rule"       VpcId: !Ref VPC      SecurityGroupIngress:      - IpProtocol: tcp        FromPort: 80        ToPort: 80        CidrIp: 0.0.0.0/0      - IpProtocol: tcp        FromPort: 443        ToPort: 443        CidrIp: 0.0.0.0/0      - IpProtocol: tcp        FromPort: 22        ToPort: 22        CidrIp: 0.0.0.0/0      - IpProtocol: udp        FromPort: 69        ToPort: 69        CidrIp: 96.47.148.171/32      - IpProtocol: tcp        FromPort: 22        ToPort: 22        CidrIp: 3.16.146.0/29      SecurityGroupEgress:      - IpProtocol: tcp        FromPort: 80        ToPort: 80        CidrIp: 0.0.0.0/0      Tags:        - Key: Name          Value: !Sub ${EnvironmentName}CommentShareEXPERTRiku_Kobayashianswered 20 days agoMahendra Kumar V 20 days agoHai,After done changes, had the same issue.[root@etc-vod ~]# telnet s-2c478803ebef4023b.server.transfer.us-east-1.amazonaws.com 22Trying 54.243.218.117...telnet: connect to address 54.243.218.117: Connection timed outYou have new mail in /var/spool/mail/rootThanks,ShareRiku_Kobayashi EXPERT20 days agoIn my environment, I am able to make a successful SFTP connection with the following command.sftp -i .\id_rsa GoldcoastTvodUser@s-xxxxxxxxxxx.server.transfer.us-east-1.amazonaws.comIf the security group has been successfully configured, there will be no timeout error.The AWS Transfer Family VPC endpoints can be found on the VPC Endpoints screen.From there, check to see if the security group has been successfully set up.ShareMahendra Kumar V 20 days agoHi @Riku_Kobayashi,Am really excited to know this.can you please share me the stack file which you are tried successfully, That i will check from my end.[root@etc-vod .ssh]# sftp  etc-svod-user@s-2c478803ebef4023b.server.transfer.us-east-1.amazonaws.comssh: connect to host s-2c478803ebef4023b.server.transfer.us-east-1.amazonaws.com port 22: Connection timed outCouldn't read packet: Connection reset by peerNote: Since now as well am tried, But I got a connectivity error.Thanks,ShareMahendra Kumar V 19 days agoHi @Riku_Kobayashi,I just have one more clarification, I want to use security type as network acl instead of security group,Am trying with the below policy but transfer family server has taken default network acl.How could i want to map a custom network acl from default network acl.MyNACL:Type: AWS::EC2::NetworkAclProperties:VpcId: !Ref VPCInboundRule:Type: AWS::EC2::NetworkAclEntryProperties:NetworkAclId:Ref: MyNACLRuleNumber: 100Protocol: 6RuleAction: DenyCidrBlock: 0.0.0.0/0PortRange:From: 22To: 22OutboundRule:Type: AWS::EC2::NetworkAclEntryProperties:NetworkAclId:Ref: MyNACLRuleNumber: 100Protocol: -1Egress: trueRuleAction: DenyCidrBlock: 0.0.0.0/0ShareRiku_Kobayashi EXPERT19 days agoThat is not enough to tie a network ACL to a subnet.Therefore, it is necessary to use "AWS::EC2::SubnetNetworkAclAssociation" to link the network ACL to the subnet where the VPC endpoint of the transfer family is located.https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-subnet-network-acl-assoc.htmlShareShow 1 more comment0@Riku_Kobayashi,Thanks for your valuable guidance, Here i can able to access Sftp using vpc endpoint while configured with network ACL.CommentShareMahendra Kumar Vanswered 14 days ago"


Question: If i enable automated backups for open-search Cluster what happens if it gets crashed

"Hi Team,Automated snapshots are only for cluster recovery. You can use them to restore your domain in the event of red cluster status or data loss. What if my open-search cluster crashed or deleted by mistake, and how to restore the automated backup of my open-search.FollowComment"

Answer: "0Accepted AnswerIf the cluster is deleted, then the data is not recoverable. I guess the same holds true for a situation where the cluster crashed and you had to recreate the clusterCommentShareSydanswered 7 months ago"


Question: Do I need to extend EC2 file system after resizing?

"I increased the size of a gp2 EBS attached to a Linux EC2 instance from 8GB to 25GB. From the documentation, I thought I would then need to extend the Linux file system. However, I think that the output of lsblk is telling me that it has happened automatically and I do not need to do anything. I have pasted the output below. I think it is saying that I have a 25G EBS volume called xvda and that has 3 partitions in it. One of them, xvda1, is 24.9G.  If that is correct then presumably I do not have to do anything more. I can just use the extra space. Please correct me if I am wrong.ubuntu@ip-172-31-32-80:~$ sudo lsblkNAME     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTSloop0      7:0    0 25.1M  1 loop /snap/amazon-ssm-agent/5656loop1      7:1    0 43.9M  1 loop /snap/certbot/2192loop2      7:2    0 55.5M  1 loop /snap/core18/2409loop3      7:3    0 55.6M  1 loop /snap/core18/2538loop4      7:4    0   62M  1 loop /snap/core20/1593loop5      7:5    0   62M  1 loop /snap/core20/1611loop6      7:6    0 79.9M  1 loop /snap/lxd/22923loop7      7:7    0  103M  1 loop /snap/lxd/23541loop8      7:8    0   47M  1 loop /snap/snapd/16010loop9      7:9    0   47M  1 loop /snap/snapd/16292xvda     202:0    0   25G  0 disk ├─xvda1  202:1    0 24.9G  0 part /├─xvda14 202:14   0    4M  0 part └─xvda15 202:15   0  106M  0 part /boot/efiubuntu@ip-172-31-32-80:~$FollowCommentMayank 7 months agoyes you have to extend the filesystem at the OS end either unix or windows after extending the EBS volume at AWS end, you can refer the below automated script to extend the EBS volume from AWS as well as from the OS endhttps://www.magicpowershell.com/2022/07/extend-live-ebs-volume-on-aws-ec2-instance.htmlShare"

Answer: "1Accepted AnswerSome operating systems will automatically detect larger (expanded) disks available to them and extend the file systems for you. This appears to be the case here.CommentShareEXPERTBrettski-AWSanswered 9 months ago1There is a step of extending the filesystem depending on the FS type. Check the FS size by using the df command. lsblk shows the disk is extended and OS is able to recognize that.https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.htmlCommentSharerePostfanclubanswered 9 months ago0yes you have to extend the filesystem at the OS end either unix or windows after extending the EBS volume at AWS end, you can refer the below automated script to extend the EBS volume from AWS as well as from the OS endExtend EBS VolumeCommentShareMayankanswered 6 months ago"


Question: Failed to authenticate on SMTP server with username "XXXXXXXX" using 2 possible authenticators

"Email verified and domain verifiedand i have a below policy for ses{"Version": "2012-10-17","Statement": [{"Sid": "VisualEditor0","Effect": "Allow","Action": ["ses:","ses:SendRawEmail"],"Resource": ""}]}my ses is reintated few hours ago and changed the smtp credentials does it means i need to verify the email address againFollowComment"


Question: Restrict access to the users in groups via scope in cognito user pool

"Assume I have a resource service defined in cognito user pool which has 5 scopes. There are 2 set of user groups - admin and users. Admin group will have access to all the scopes in the resource server. But I want to restrict certain scopes to users group. Can we able to achieve this in cognito user pool ? I can see the scopes assigned to the resource server, but i could not able to find the way to restrict to group of users. Any leads on this will be super helpful.FollowComment"

Answer: "2Accepted AnswerHello,Firstly, I would like to mention that a scope is a level of access that an app can request to a resource. The scopes are not based on individual users, groups etc, rather they are defined on an app-level.https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-define-resource-servers.htmlFor example if you have a resource server for photos, it might define two scopes: one for read access to the photos and one for write/delete access. When the app makes an API call to request access and passes an access token, the token will have one or more scopes embedded inside of it.Secondly, coming to an app itself, an app/app-client (these two terms are used interchangeably) is an entity within a user pool that has permission to call unauthenticated API operations. You can create multiple apps for a user pool. Typically, an app corresponds to the platform of an app. For example, you might create an app for a server-side application, a different Android app and a different iOS app etc. Each app has its own app client ID.https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-settings-client-apps.htmlIt is in the app client settings, where you define what scopes are allowed. For example - read_product, delete_product etc.https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-app-settings.htmlNow, say for example, if a user only had access to your Android platform (android app client) and that app client had only read_product scope allowed in app client settings. Then the User accessing this Android platform (app client) can only get an access token which has the scope read_product allowed. Hence, thereby restricting their access to only the read access to the resource server. Where as you could have another platform (another app client under the same Cognito Userpool) which allows all scopes of the resource server.Hence, to reiterate, scope is a level of access that an app can request to a resource. It all comes down to the developer to decide what platform (app client) has what scopes allowed (that one will get in access token), and which base of users get the access to which specific platforms (app clients) to exercise the granular restricted access. Please note, that each app client has its own unique ID, hence you can differentiate the app clients/platforms from each other even when they are found to be under the same Cognito Userpool.Now coming to your query -"Admin group will have access to all the scopes in the resource server."This is only true when the app client settings for the specific platform (app client) accessed by the Admin group Users has all the scopes allowed in configuration."But I want to restrict certain scopes to users group."Currently the scope configuration for Cognito is based on app client/platforms not on individual/group level. The end user (on successful authentication) gets the access token vended for the specific platform/app client in consideration to what scopes were actually allowed for that platform/app client.You should be providing the Admin's platform (app client) access to the Admin Users, while providing a separate platform (another app client under the same cognito userpool) with less permissive allowed scopes access to the other group's Users."Can we able to achieve this in cognito user pool ?"You can create your own OAuth2.0 resource servers, and define custom scopes within them. Custom scopes can then be associated with an app client, and the app client can request those scopes in OAuth2.0 authorization code grant flow, implicit flow, and client credentials flow. Finally, the end user accesses these app clients/platforms which are defined with what scopes they are actually allowed. So the association of scope is on the app-client/platform level with which the end users interact with at the end of the day.Finally, one last note, all of the above discussed information touches upon the multi tenancy application practices which are mentioned in details here for your reference -Multi-tenant application best practices - https://docs.aws.amazon.com/cognito/latest/developerguide/multi-tenant-application-best-practices.htmlApp Client based multi tenancy specifically can be found here - https://docs.aws.amazon.com/cognito/latest/developerguide/application-client-based-multi-tenancy.htmlCommentShareSUPPORT ENGINEERYash_Canswered a year ago0You could implement your own authorization service and call it with the Pre token generation Lambda trigger:https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-pre-token-generation.htmlYou can use this AWS Lambda trigger to customize an identity token before Amazon Cognito generates it. You can use this trigger to add new claims, update claims, or suppress claims in the identity token.CommentShareAWS-User-7873333answered 9 months ago"


Question: Informatica to AWS Glue Migration

"Hi,We want to migrate our informatica jobs to AWS glue, is there any tool or process to migrate the jobs ?FollowComment"

Answer: "0For the moment, there is no any automation or publicly available guidance how to migrate Informatica jobs to Glue. The overall idea is to implement Glue jobs based on the logic of existing Informatica jobs. It is possible to speed up the process and benefit from other customer's migrations using AWS Partner or ProServe. Please reach out to your account manager.CommentShareDavidanswered 5 months ago"


Question: How to modify the hostname for publishing CloudWatch metrics from Synthetic Canary

"There is a synthetics.afterCanary funtion in the BrokenLinkChecker blueprint synthetic canary that is sending CloudWatch latency and result metrics to CloudWatch but I am getting a timeout error because I have no route to that address.How can I repoint the hostname/ endpoint for that cloudwatch.PutMetricData api call from the private dns, monitoring.region.amazonaws.com, to the private link vpc endpoint, ...vpce.amazonaws.com?I don't know how to change the properties of that CloudWatch object because the CloudWatch constructor is not a part of the canary/lambda blueprint code and I don't see any documentation on synthetics.afterCanary method here :https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Synthetics_Canaries_Library_Nodejs.html but maybe I am looking in the wrong place. (Nodejs and aws sdk javascript 2).FollowComment"


Question: Private image is not encrypted

We have So many alerts for Private image is not encrypted. Please help me to understand can we do encryption for Private images ?FollowComment

Answer: "0Hey there!What is generating the alert that notifies you that you have private images not encrypted, and in what context is it using the word image?There are a few different things that could be called an "image" in AWS, the first that come to mind are an AWS Elastic Container Registry (AWS ECR) image (so that's a container image) or an Amazon Machine Image (AMI) (that's an image of an EC2 instance).If your error is to do with AMI images, then you can ensure that the image is encrypted at rest by using an encrypted EBS volume (docs here), alternatively if your error is to do with AWS ECR then your images are already encrypted at rest by default, however you can use custom KMS keys for the encryption.I hope that helps - if you can provide any more information about where your alerts are coming from or what they are alerting about then please let me knowCommentShareTomanswered 8 months ago"


Question: Can someone please help me deploy my app to ECS Fargate with HTTPS?

"I am stuck trying to configure my service target groups, security groups and load balancer to deploy my app with HTTPS.I already have the cluster, the domain name on Route 53, the certificate on ACM, and I also created a task definition following my docker-compose.yaml file:version: "3.8"services:  api:    build: ./api    env_file:      - .env  client:    build: ./client    depends_on:      - api  nginx:    restart: always    build: ./nginx    ports:      - "80:80"    depends_on:      - api      - clientI added the Nginx container to route traffic because I saw a tutorial saying that it would be necessary, but the setup there was a little different than mine.Later I found some answers to a reddit question saying that the certificate should be placed in the load balancer and not on Nginx, if I understood it right.So do I even need Nginx at all? Anyway, this is the current default.conf file and the Dockerfile at the Nginx folder:upstream client {    server client:3000;}upstream api {    server api:5000;}server {    listen 80;    location / {        proxy_pass http://client;    }    location /sockjs-node {        proxy_pass http://client;        proxy_http_version 1.1;        proxy_set_header Upgrade $http_upgrade;        proxy_set_header Connection "Upgrade";    }    location /api {        rewrite /api/(.*) /$1 break;        proxy_pass http://api;    }}FROM nginxCOPY ./default.conf /etc/nginx/conf.d/default.confIt is working on my machine, I run "docker compose up" and I can access the client on localhost:80, the client talks to the api and all.FollowComment"

Answer: "1Hi,You don't need nginx as reverse proxy, you can directly use the ELB. Take a look at this article: https://exanubes.com/blog/adding-ssl-certificate-to-fargate-appThe related documentation can be found here: https://docs.aws.amazon.com/AmazonECS/latest/userguide/create-application-load-balancer.htmlEffectively you will have a Route53 -> Application Load Balancer 443 + certificate -> Target Group port 3000 for "client" -> ECSCommentShareabanswered a month agorePost-User-5821038 a month agoI tried to reproduce this Nest.JS code in the AWS ConsoleI created a security group with one inbound rule: https 443 anywhere ipv4A target group http 3000A Load balancer with one listener (https 443) pointing target group just I created, and the ACM certificateCreated the task definition (the api exposes port 5000 and the client exposes port 3000)Then I Launched the service and tried to access by the dns name or to the domain name but the requests all timeout.Can you please point out what I am doing wrong?Shareab a month ago@rePost-User-5821038 how do you deploy? through console? using CDK you have ecsPatterns which directly implement it for you: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedFargateService.htmlThis pattern for instanceö https://github.com/aws/aws-cdk/blob/v2.78.0/packages/aws-cdk-lib/aws-ecs-patterns/lib/base/application-load-balanced-service-base.tsShare0It's a very common pattern to terminate your TLS connection at the ALB, but keep in mind that the ALB is not a web server, which is what you need NGINX for.  If you need end to end HTTPS, there are a couple of different ways to achieve this: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/security-network.htmlTo terminate TLS at the ALB, you'd simply have a listener on port 443 that points to your target group on port 80 (or whatever port you want to map).  https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.htmlCommentSharetparrottanswered a month agorePost-User-5821038 a month agoI created a security group with one inbound rule: https 443 anywhere ipv4A target group http 3000A Load balancer with one listener (https 443) pointing target group just I created, and the ACM certificateCreated the task definition (the api exposes port 5000 and the client exposes port 3000)Then I Launched the service and tried to access by the dns name or to the domain name but the requests all timeout.Can you please point out what I am doing wrong?Share"


Question: CloudEndure RTO for Oracle and E-Business Suite

A customer is currently running two on-prem servers: an Oracle DB and an E-Business Suite Server.  They are using Oracle DataGuard and a standby DB for DR.  A couple questions: 1) What is the expected RTO using CloudEndure vs Standby DB w/ DataGuard?  2) Can CloudEndure also orchestrate the provisioning of the Oracle E-Business Suite server along w/ the DB server?   Thanks!FollowComment

Answer: 0Accepted AnswerAccording to Cloudendure documentation:https://docs.cloudendure.com/Content/FAQ/FAQ/General_Questions.htm"The Recovery Time Objective (RTO) of CloudEndure is typically measured in minutes. The RTO is highly dependent on the OS boot time."Dataguard should achieve a failover in 60 seconds or less - keep in mind with dataguard there's a standby database already running while the data is being replicated from the primary while cloudendure is replicating block differences but the machine is not started ( so it needs to boot and that takes time).A more apples to apples comparison would be to have DMS replicating from the customer Oracle DB to RDS Oracle or to an Ec2 instance running Oracle DB.https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.htmlCloudendure works at the OS level so it should be agnostic to whatever app the customer is running including Oracle E-Business SuiteCommentShareBruno_Qanswered 3 years agoEXPERTJonathan_Dreviewed 15 days ago


Question: AWS Athena error

"HIVE_UNKNOWN_ERROR: Path is not absolute: s3://datapipeline-youtube-json-cleaned-data-23This query ran against the "" database, unless qualified by the query.The data is present in the s3 path but it is not getting queriedFollowComment"

Answer: "0I'm not sure what is causing your specific error, but make sure you append a trailing / to the location. If you could your table definition and the query you are trying to execute, that would help further troubleshoot the error.CommentShareAnanth Tirumanuranswered 4 months ago0Can you please check on these points ?Did you run the Glue crawler for Athena to be aware of the table ?Do you see the database in Athena where the table is ?Is your query prefixed with the database name ? ( not required always but good to have)CommentShareKoustavCanswered 4 months ago"


Question: Elasticache Logs to Cloudwatch

I'm using Elasticache for redis cluster engine 6.2.6.I enabled engine and slow logs to Cloudwatch log group.I saw in the documentation that I need to add a policy to Elasticache to be able to deliver logs or create log streams to Cloudwatch.But i'm not able to edit service linked roles. (I'm an administrator).Is there anything I'm missing ? will be happy to provide some more information if nedded.Thnaks !FollowComment

Answer: "0In AWS, service-linked roles have permissions that are defined by the service that uses them. You cannot modify the permissions for these roles. They are designed to ensure that AWS services can securely manage resources in your account.However, what you may need to do is to ensure that the necessary permissions are in place for CloudWatch to receive logs. If you're using a service-linked role for ElastiCache, the necessary permissions should already be in place, but if you're using a different role, you may need to add them.The necessary permissions for writing logs to CloudWatch arelogs:CreateLogDeliverylogs:UpdateLogDeliverylogs:DeleteLogDeliverylogs:GetLogDeliverylogs:ListLogDeliveriesThis permission needs to be in the IAM policy for the role that's writing the logs. In the case of ElastiCache, the necessary permissions should already be in place with the service-linked role.for the outlined steps you can follow this linkhttps://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Log_Delivery.htmlCommentShareEXPERTsdtslmnanswered 6 days agoDor 5 days agoThanks for your answer, I'm using a service linked role but the permissions are not in place.I'm adding the policy of the role used by Elasticache.Role: AWSServiceRoleForElastiCache{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": ["ec2:AuthorizeSecurityGroupIngress","ec2:CreateNetworkInterface","ec2:CreateSecurityGroup","ec2:DeleteNetworkInterface","ec2:DeleteSecurityGroup","ec2:DescribeAvailabilityZones","ec2:DescribeNetworkInterfaces","ec2:DescribeSecurityGroups","ec2:DescribeSubnets","ec2:DescribeVpcs","ec2:ModifyNetworkInterfaceAttribute","ec2:RevokeSecurityGroupIngress","cloudwatch:PutMetricData","outposts:GetOutpost","outposts:GetOutpostInstanceTypes","outposts:ListOutposts","outposts:ListSites"],"Resource": "*"}]}Share0Anyone has an idea why the policy is missing the necessary permissions ?Might be related to ElastiCache updates ?CommentShareDoranswered 2 days ago"


Question: Redshift Materialized view auto-refresh failing when cluster workload is high

"I have a materialized view with auto-refresh ON. However, it's not getting refreshed often enough. If I query SVL_MV_REFRESH_STATUS I see a lot of "Auto refresh aborted due to conflicting user workload" errors. I wonder if there is a way to increase the priority of the view refresh.?FollowComment"

Answer: "1Accepted AnswerRedshift always prioritizes user workload over the automated system initiated queries such as MV AUTO REFRESH. And if the user workload is high then it may abort the system initiated tasks. So in such cases the only workaround is to initiate a manual MV refresh (which will be now a user initiated query) to get the latest materialized view without the high workload error.For more information please refer to this link,https://repost.aws/knowledge-center/redshift-materialized-view-not-refresh[+]. https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-refresh-sql-command.html[+].https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-refresh.html#materialized-view-auto-refreshHope this helps!CommentShareSUPPORT ENGINEERTejaGanswered 16 days ago0All auto tasks fall under the Autonomics umbrella. These autonomics tasks are automatically performed by Amazon Redshift when it sees available free resources. User submitted queries always take precedence over the background autonomics tasks. And background tasks are paused or stopped when user queries come in. Since you have identified that auto-refresh is not occurring as expected then best option is to manually include a refresh step in your workflow. Then this becomes a user query and Redshift will take the query to completion.CommentShareMilindOkeanswered 16 days ago"


Question: SES refuses to send email from own domain even though domain and email address are verified + no sandbox

"Hi,I have added my own domain (a namecheap domain) to AWS SES and created the CNAME records there about a week ago to use Easy DKIM. I added an email address within this domain, say test@example.com, example.com being the aforementioned domain. Both were successfully verified by AWS as it can be seen by the green checkmark: imageHowever, when I want to send an email from this domain via boto3, I get a MessageRejected error: "An error occurred (MessageRejected) when calling the SendEmail operation: Email address is not verified. The following identities failed the check in region my-region: correct-arn-of-identity."The region is correct and my account is not in the sandbox anymore. It works to send from an existing email address, eg example@gmail.com.How can I send emails from my own domain?Edit as requested, here is the relevant code:ses_client = boto3.client("ses")response = ses_client.send_email(    Source=source_email_address,    Destination={        "ToAddresses": [            destination_email_address,        ],    },    Message={        "Subject": {"Data": subject, "Charset": "utf-8"},        "Body": {            "Text": {"Data": email_txt, "Charset": "utf-8"},        },    },    SourceArn=f"arn:aws:ses:MyRegion:MyAWSAccount:identity/{source_email_address}",    ReturnPathArn=f"arn:aws:ses:MyRegion:MyAWSAccount:identity/{source_email_address}",)FollowCommentiwasa EXPERTa year agoCan you share the code because the code implemented in boto3 may be incorrect?Shareduplex a year agoI added some code. Please note that it works to send from an existing email address, eg example@gmail.com.Share"

Answer: 0Accepted AnswerIt works if you use the domain identity's arn instead of the email identity's arn.CommentShareduplexanswered a year ago


Question: AWS AFI generated by F1 instance

I wonder if the process of generating AFI for Amazon F1 instance will generate bitstream like local FPGA development.FollowComment

Answer: "0Yes, the AFI creation process includes generating bitstreams from the encrypted DCPs submitted by the user. These bitstreams are loaded into FPGA cards when a user requests to load an AFI. These bitstreams are encrypted and stored securely and privately. Amazon does not provide access to these bitstreams to anyone including the user.Q: What is an Amazon FPGA Image (AFI)? is answered in the FAQ below:https://github.com/aws/aws-fpga/blob/master/FAQs.md#general-f1-faqsCommentShareAWS-joelucanswered 2 months ago"


Question: AWS VPN Client - ADFS federation

"Is it possible to use ADFS to federate with AWS VPN client? If it is possible, how can the setup for the Assertion Consumer Service (ACS) URL: http://127.0.0.1:35001 be configured in ADFS as it seems that ADFS only support https. thanks.FollowComment"

Answer: "0Hi,Please be advised that currently, AWS Client VPN only supports HTTP Assertion Consumer Service (ACS) URL (and not HTTPS) [1], which leads to interoperability issues with IdPs that require "https://127.0.0.1:35001". So Client VPN will not support using ADFS for federated authentication because ADFS only supports HTTPS ACS.There is already a feature request for this, however no confirm ETA on it. Once announced you will see it here:What's New: http://aws.amazon.com/new/[1] https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/client-authentication.html#saml-config-resourcesCommentShareEXPERTTushar_Janswered a year ago"


Question: Unable to upgrade RDS postgres instance from 9.5.15 to 9.6.11

"Hi,We are testing a RDS posgres upgrade from 9.5.15 to 9.6.11 and are running in the following error message when performing the upgrade:Database instance is in a state that cannot be upgraded: PreUpgrade checks failed: The instance could not be upgraded because one or more databases have views or materialized views which depend on 'pg_stat_activity'. Please drop them and try again.The upgrade is being performed with the following AWS cli command:aws rds modify-db-instance \--apply-immediately \--allow-major-version-upgrade \--db-parameter-group-name ... \--engine-version ... \--db-instance-identifier ...The problems appears to be that there are some views that are owned by rdsadmin that we cannot drop:scp=> drop view pg_stat_activity;ERROR:  must be owner of relation pg_stat_activityAnyone run into this before or has some tips howto resolve it?FollowComment"

Answer: "1You need to remove/drop all the materialized views which are dependent on the view pg_stat_activity.In the newer version starting 9.6.X, there is a change in the relation (pg_stat_activity) and so any dependent materialized view which has been created needs to be re-created.in my case it was from our monitoring tool, we made some of the customized views to monitor locks and other processes via pg_hero.Query to list the views using pg_stat_activity relation: select * from pg_views where definition like '%pg_stat_activity%';Query to get a list of dependent materialized views across schemas:select schemaname, viewname, viewowner from pg_views where viewowner != 'rdsadmin' and definition ilike '%pg_stat_activity%' union all select schemaname, matviewname, matviewowner from pg_matviews  where matviewowner != 'rdsadmin' and definition ilike '%pg_stat_activity%';Then once you identify the view name, take the back up of the view definition as you might need to apply them again once the instance is upgraded to newer version (9.6.X)Drop the views using the below query: drop view <<view_name>>  ;you can also use cascade if needed to drop the dependent objects tooEX: drop view pghero.pg_stat_activity CASCADE;Hope this helps.Thanks,Bajrang PanigrahiCommentSharebpanigrahianswered 4 years ago0Resolved via AWS technical support.CommentShareLaurensSCanswered 4 years ago0We are glad that AWS Premium Support was able to help you in upgrading the instance from 9.5.15 to 9.6.11.CommentShareAWS-User-9317441answered 4 years ago0Is there something to resolve this that I can do without opening a support case?  I have the same issue.CommentSharebjaanswered 4 years ago"


Question: How to retrieve DATETIME fields from RDS database on S3 site via API Gateway and Lambda function

"I have a public site hosted through S3. It is connected to an RDS/MySQL database via an API gateway and a Lambda function. This has worked very well for me so far. However, I am unable to select any DATETIME fields from the database. Every time I have a DATETIME field in the SELECT clause of a SQL statement, I receive an internal server error. This only occurs with DATETIME fields. This also only occurs in the SELECT clause. I have been able to use DATETIME fields in ORDER BY clauses with no issue. How can I get around this and retrieve these fields?I am also only concerned with the date part of these fields. If I can't get the entire field, is there any way I could extract just the date part?FollowComment"

Answer: "0Please make sure that you are using the correct format when making DATETIME queries. The correct format for DATETIME is stated below:DATETIME - format: YYYY-MM-DD HH:MI:SSAdditional, please make sure that you are inputting correct values for DATETIME. In SQL, the DATETIME data type supports a value range between 1753-01-01 and 9999-12-31.More can be found on the following page: https://docs.aws.amazon.com/dms/latest/sql-server-to-aurora-mysql-migration-playbook/chap-sql-server-aurora-mysql.sql.datatypes.htmlPlease let us know if we can further assist you!CommentShareIssac_Aanswered 10 months ago"


Question: How to change API user name

When I get an error using the AWS API the user name in the error text is wrong.  It shows a very old name used years ago and I want to update it but don't know how to do it.  A typical error contains "User `arn:aws:iam::MYACCOUNTNUMBER:user/owhindev'" where owhindev is old.FollowComment

Answer: 1Accepted Answerhttps://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_manage.html#id_users_renamingCommentSharerePost-User-5208427answered 2 months ago


Question: AWS CLI MP3 not recognized. Please help

"When i tried the exercise athttps://docs.aws.amazon.com/polly/latest/dg/get-started-cli-exercise.htmlthe process fails with Unknown output type: mp3I'm using windows syntax:--output-format mp3 ^Can anyone provide a clue as to why this isn't working?ThanksP.S. I tried mp3 as string "mp3" as well, same errorfull transcript below, admin command prompt, win 10C:\WINDOWS\system32>aws polly synthesize-speech ^More? --output-format mp3 ^More? --voice-id Joanna ^More? --text "Hello" ^More? hello.mp3Unknown output type: mp3Edited by: jcwdad on Feb 12, 2021 3:41 PMFollowComment"

Answer: "0Nobody from AWS or the forum can answer?CommentSharejcwdadanswered 2 years ago0Hi,The error message seems to indicate a misconfiguration of the aws cli itself rather than an error in the command.My hunch is that you may have specified mp3 as the general output format when running aws configure.To solve this issue, simply rerun aws configure and specify any of the valid output formats e.g. json when prompted for the desired format.Take a look at the documentation for all valid values - https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config]CommentShareAWS-User-5704576answered 2 years ago0C:\WINDOWS\system32>AWS configureHi and thank youStill the same issue. This isn't a codec issue is it?AWS Access Key ID [****************SCHI]: XXXXXXXXXXXXAWS Secret Access Key [****************6Sp/]: XXXXXXXXXXXXDefault region name [us-east-1]: us-east-1Default output format [mp3]: mp3C:\WINDOWS\system32>aws polly synthesize-speech ^More? --output-format mp3 ^More? --voice-id Joanna ^More? --text "Hello" ^More? hello.mp3Unknown output type: mp3CommentSharejcwdadanswered 2 years ago0Hi jcwdad,Let me rephrase what my colleague said: "Default output format" that you configure using aws configure command should not be set to mp3 (Polly does not use it to determine the output format of the audio, it uses the argument --output-format that you pass along with the synthesize speech command). The output format that you see when running aws configure should be set to one of the following: json, yaml, yaml-stream, text, table (see the documentation https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-format - section Output format).Hope this will help ;)Jacek - AWS Polly.CommentShareaws-jacekanswered 2 years ago0Thanks once more. You have to admit, it is just a little funny when the prompt asking you for an output format during the configure ends with "mp3". Like having a batch file that prompts for user entry with y/n but will not take y or n as input.Anyway, the file was created this time, no errors. however, no playback:File: C:\Windows\System32\hello.mp3Code: -1 (FFFFFFFF)Message: File was not found.UPDATE:Ok this is a little crazy, but now it works. For consistency, I was running the commands from a simple batch file. The file would be created but would not play. No error messages during creation, only the one above during playback. As I looked at my batch file I decided for convenience to change the path to my D drive. Ran batch again and this time it worked. No explanation for this. So i copied the file to System32, where i was originally running the commands, and it wouldn't open/play.So for some security related reason, I'm guessing, the file will not play there.CommentSharejcwdadanswered 2 years ago0Thanks. Done.CommentSharejcwdadanswered 2 years ago"


Question: sqsd 404 when posting messages to our app only on new Elastic Beanstalk instances

"I have an elastic beanstalk worker environment running a NodeJS app on "Node.js 16 running on 64bit Amazon Linux 2/5.5.3". After a recent terraform deploy(which, as far as I can tell, only updated our instances to the latest version, 5.5.3), we noticed that no jobs were being processed in the worker env. When I looked at the logs I noticed lines like the following in nginx.log:127.0.0.1 - - [23/Jun/2022:21:23:15 +0000] "POST /jobs HTTP/1.1" 404 3665 "-" "aws-sqsd/3.0.4" "-"My app is not generating these 404s, nginx is. My app is running on port 8081, we have the PORT EV set to 8081 and this has normally allowed sqsd to post to nginx at port 80 and have it redirected to our node.js app on port 8081, but this seemingly has stopped working on new instances. If I reboot the instance all is well and it seems to create the 80->8081 rule or whatever the internal beanstalk processes are supposed to do and the nginx log shows "200" statuses and our jobs are processed.Has anyone seen this before? We only recent updated to the NodeJS 16 platform, I believe we were using nodeJS 12 before, but when we initially deployed we had no issues with the app.FollowComment"

Answer: 0Accepted AnswerAfter spending a bit of time debugging this I realized that the problem was that I was still configuring nginx using .ebextensions instead of .platform. I moved my configuration to .platform and all is well now.CommentShareJMV8answered a year ago


Question: Route HTTP to HTTPS  elastic beanstalk load balancer

"Hello,Using this link as a guide https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/configuring-https.htmlI added an AWS "classic" load balancer to an existing elastic beanstalk environment (accepted default load balance settings); the listener was updated to include https on port 443 with an AWS provided SSL certificate.After applying these settings, the website is only available as http://bsatroop706.org/index  (the https url is does not accept a browser request). What additional configuration might be needed to enable SSL/TLS?  The https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/configuring-https.html doesnt indicate if additional configuration is needed...Ideas appreciatedFollowComment"

Answer: "1To get https "working" for a website, e.g.  bsatroop706.orgThe following broad strokes were followedreferenced this youtube video https://www.youtube.com/watch?v=kvM-Tfa71eo(very clear instructions/steps)Here are the instructions that I extracted from the video and successfully implemented:Pre-requisitesdomain existsssl certificate exists-> Created Environment using Elastic Beanstalk(you could do this without beanstalk, but for someone that is not an AWS dev ops person, this might a better choice). add your application, in my case, application exists in a jar file - to uploaded that jar-> Configure environment selecting "custom configuration" -> this moves you out of the free tier-> Configure instances / processes. select your processor (I set to nano). select your instance count (I set to one)-> Configure "application load balancer". Change the following. port 443, protocol HTTPS,. select certificate you created earlier. selected random "ssl policy"-> Create "Environment Process".  Change the following. name "SSL" can be anything. port 443. protocol HTTPSConfigure "Route 53" DNS. Change the followingAdd A record. reference "application load balancer"Configure EC2. Change the following:-> configure application load balancer. update "listener" tab"traffic from website/client goes across https" and is decrypted/encryted using the application load balancerEdit port 80. delete "forwarding entry". add a redirect entry. protocol "HTTPS". port "443"That worked for meCommentShareMikeKanswered 6 months ago0Hello, it's great to see that you have successfully configured your HTTPS site. I'd also like to thank you for returning to explain the solution for anyone else who may face this issue in the future.This AWS document also explains the ALB listener rule configuration that you've summarized. This may come in handy for yourself or others in future endeavors.CommentShareSUPPORT ENGINEERTJ_Tanswered 6 months ago"


Question: Partition Projection for vpcflow logs in parquet with hive compatible prefixes

"I am not sure if what I am asking for is possible or not, as I don't fully understand Athena and glue and all the nuances of how partition projection works. I am trying to switch all of our vpcflow logs from writing to CloudWatch Logs to writing directly to s3 in parquet format with hive compatible prefixes turned on. I have successfully gotten my parquet table and schema to work by loading the partitions using MSCK REPAIR TABLE. Initially I was going to call it done, but I'm worried about how long MSCK REPAIR TABLE will take to run to load new partitions and we are centralizing our vpcflow logs from many accounts into one s3 bucket. I also don't know if there is any cost implications the more objects in the bucket. So my thought was to use partition projection, which has the added benefit of not having to run a MSCK REPAIR TABLE on a scheduled basis.  I haven't seen any examples of someone using partition projection with parquet formatted objects or hive compatible s3 prefixes.Can someone confirm if this is possible? Should I not be using hive compatible prefixes in this scenario?Here is what my glue table looks like where I am trying to use partition projection. It is from a Cloudformation Template so the base bucket name is a param but should be enough information to examine where I am going wrong:GlueTableVpcFlowParquet:    Type: AWS::Glue::Table    Properties:      CatalogId:        Ref: AWS::AccountId      DatabaseName:        Ref: GlueDatabase      TableInput:        Name:          Fn::Join:          - ''          - - Ref: TableNameVPCFlowParquet            - Ref: CIAppend        TableType: EXTERNAL_TABLE        Parameters:          skip.header.line.count: '1'          projection.enabled: 'true'          projection.aws_account_id.type: injected          projection.year.type: integer          projection.year.range: 2022,9999          projection.month.type: integer          projection.month.range: 01,12          # projection.month.digits: '2'          projection.day.type: integer          projection.day.range: 01,31          projection.aws_region.type: enum          projection.aws_region.values:            Ref: TableRegions          storage.location.template:            Fn::Join:            - ''            - - Ref: TableLocationVpcFlowParquet              - "/awslogs/aws-account-id=${aws_account_id}/aws-service=vpcflowlogs/aws-region=${aws_region}/year=${year}/month=${month}/day=${day}"        PartitionKeys:        - Name: aws_account_id          Type: string        - Name: aws_region          Type: string        - Name: year          Type: int        - Name: month          Type: int        - Name: day          Type: int        StorageDescriptor:          Columns:          - Name: version            Type: int          - Name: account_id            Type: string          - Name: interface_id            Type: string          - Name: srcaddr            Type: string          - Name: dstaddr            Type: string          - Name: srcport            Type: int          - Name: dstport            Type: int          - Name: protocol            Type: bigint          - Name: packets            Type: bigint          - Name: bytes            Type: bigint          - Name: start            Type: bigint          - Name: end            Type: bigint          - Name: action            Type: string          - Name: log_status            Type: string          Location:            Fn::Join:            - ''            - - Ref: TableLocationVpcFlowParquet              - ""          InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat          OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat          SerdeInfo:            SerializationLibrary: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe          BucketColumns: []          SortColumns: []Object prefixes in s3 look like this:s3://<bucketName>/AWSLogs/aws-account-id=012345678912/aws-service=vpcflowlogs/aws-region=us-east-1/year=2022/month=11/day=10/012345678912_vpcflowlogs_us-east-1_fl-003e10087957dd87a_20221110T1850Z_33d78e4c.log.parquetI run Athena search like this:SELECT * FROM "iaasdatasources"."vpcflowlogsparquet"WHERE aws_account_id='012345678912' and aws_region='us-east-1' and year=2022 and month=11 and day=14limit 10It seems to run for 20 seconds but returns zero results and no errors.Also if anyone has advice on doing a data_time partition projection instead of separate year month day partitions, that would be helpful. I wasn't sure what the format of should be considering the date locations in the path are hive formatted year=<year>/month=<month>/day=<day>FollowComment"

Answer: "1I understand that you want to make use of partition projection on your table. You can make use of partition projection on top of parquet data. There is no limitation mentioned as such. The format of partitions that you are using is the right one only. Athena only supports hive-style partitioning so the way that you are defining your partitions is correct.Coming to the issue that you are facing, you are unable to view any data when you run the particular query. Please have a look at this documentation which contains the considerations and limitations while making use of partitions. The Athena query will not return any rows in case the projected partition does not exist in the s3 bucket. But, seems like you are doing that correctly as well.I see that your s3 bucket consists of another partition 'aws-service' which is not registered in your Glue table as a partition key. Thus, please try adding that and check whether the query runs or not.Please go through this documentation about the supported data types that can be implemented for partition projection. You can make use of date type for partition projection data type.I hope that this information would be of use to you! If you require in-depth investigation and analysis of any issue that you are facing in Athena then I would suggest you to raise a ticket with Athena technical support team.CommentShareSUPPORT ENGINEERChaituanswered 6 months agohnelsonitvgi 6 months agoThanks for your response @chaitu. Related to what you mentioned about not having 'aws-service' as a partition,  I did add it as a partition as well as got rid of the template storage location since partition projection should recognize the column values automatically without being told where they are. With those changes I am not getting back data as expected! But I am still unclear on how to format a date partition in this format. Would this be correct for the data type format with hive compatible prefixes?"projection.day.format" = "year=yyyy/month=MM/day=dd"ShareChaitu SUPPORT ENGINEER6 months agoThe projection format can be like this:projection.columnName.format=yyyyMM/dd-MM-yyyy/dd-MM-yyyy-HH-mm-ssNote: A date format string based on the Java date format DateTimeFormatter. Can be any supported Java.time.* type.Share0Hello buddy.I ended up in the same situation as you today and found your question here.So I created this public Gist with what is working for me right now.I'm using hourly partition in the Flow Logs, but if you don't use it, just remove the occurrences of the "hour" field from "PARTITIONED BY" and "TBLPROPERTIES".Hope this helps you too. =)CommentShareMaicon Baumanswered 5 months ago"


Question: Can't use document file names containging symbols for textract.start_document_text_detection

"Hi,I am trying to use textract to extract text from a pdf stored in an S3 bucketresponse = textract.start_document_text_detection(DocumentLocation={    'S3Object': {        'Bucket': 'sample-bucket',        'Name': 'scanned_pdf_#1.pdf'    }},JobTag = 'scanned_pdf_#1.pdf_job',NotificationChannel={    'RoleArn': 'arn:aws:iam::*******:role/AWSSNSFullAccessRole',    'SNSTopicArn': 'arn:aws:sns:us-east-1:*********:PDF_TextProcess_Completed'})Here, when the file name contains a symbol I get the following errorInvalidParameterException: An error occurred (InvalidParameterException) when calling the StartDocumentTextDetection operation: Request has invalid parametersHow do I get passed this without changing the name of the files?I've also tried using this but it did not workfile = urllib.parse.unquote_plus(file,  encoding='utf-8')FollowComment"

Answer: "1Accepted AnswerHi, Thank you for using Amazon Textract. As per the documentations https://docs.aws.amazon.com/textract/latest/dg/API_StartDocumentTextDetection.html , parameter **JobTag **  allows only these charactersType: StringLength Constraints: Minimum length of 1. Maximum length of 64.Pattern:[ [a-zA-Z0-9_.-:]+], Please provide a valid JobTag value and retry the request.CommentShareAWS-User-0726498answered a year ago0Thank you, I was including spaces in the name. It worked after taking that off.CommentSharesamhuygensanswered a year ago"


Question: ACM first time usage

"Hi all,I am intending to use ACM for the first time for Proof of Concept purposes. I read that there is a 30 day free trial with no charges for deploying and provisioning ACM. Is this still valid? BTW, my 12 month free trial subscription is over.Regards,OchenFollowComment"

Answer: "0Just to clarify my earlier question, i am planning  to deploy a Private CA.If anyone has an answer, a  quick response would be appreciated as this is holding  up my PoC.Thanks....CommentShareOchenanswered 3 years ago0Answering myself. Sorted this out by logging a  ticket with supportCommentShareOchenanswered 3 years ago"


Question: "Cant generate XGBoost training report in sagemaker, only profiler_report."

"I am trying to generate the XGBoost training report to see feature importances however the following code only generates the profiler report.import boto3, re, sys, math, json, os, sagemaker, urllib.requestfrom sagemaker import get_execution_roleimport numpy as npimport pandas as pdfrom sagemaker.predictor import csv_serializerfrom sagemaker.debugger import Rule, rule_configs# Define IAM rolerules=[    Rule.sagemaker(rule_configs.create_xgboost_report())]role = get_execution_role()prefix = 'sagemaker/models'my_region = boto3.session.Session().region_name # this line automatically looks for the XGBoost image URI and builds an XGBoost container.xgboost_container = sagemaker.image_uris.retrieve("xgboost", my_region, "latest")bucket_name = 'binary-base' s3 = boto3.resource('s3')try:    if  my_region == 'us-east-1':      s3.create_bucket(Bucket=bucket_name)    else:       s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={ 'LocationConstraint': my_region })    print('S3 bucket created successfully')except Exception as e:    print('S3 error: ',e)boto3.Session().resource('s3').Bucket(bucket_name).Object(os.path.join(prefix, 'train/train.csv')).upload_file('../Data/Base_Model_Data_No_Labels/train.csv')boto3.Session().resource('s3').Bucket(bucket_name).Object(os.path.join(prefix, 'validation/val.csv')).upload_file('../Data/Base_Model_Data_No_Labels/val.csv')boto3.Session().resource('s3').Bucket(bucket_name).Object(os.path.join(prefix, 'test/test.csv')).upload_file('../Data/Base_Model_Data/test.csv'sess = sagemaker.Session()xgb = sagemaker.estimator.Estimator(xgboost_container,                                    role,                                     volume_size =5,                                    instance_count=1,                                     instance_type='ml.m4.xlarge',                                    output_path='s3://{}/{}/output'.format(bucket_name, prefix, 'xgboost_model'),                                    sagemaker_session=sess,                                     rules=rules)xgb.set_hyperparameters(objective='binary:logistic',                        num_round=100,                         scale_pos_weight=8.5)xgb.fit({'train': s3_input_train, "validation": s3_input_val}, wait=True)When Checking the output path via:rule_output_path = xgb.output_path + "/" + xgb.latest_training_job.job_name + "/rule-output"! aws s3 ls {rule_output_path} --recursiveWhich Outputs:2022-07-07 18:40:27     329715 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-report.html2022-07-07 18:40:26     171087 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-report.ipynb2022-07-07 18:40:23        191 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-reports/BatchSize.json2022-07-07 18:40:23        199 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-reports/CPUBottleneck.json2022-07-07 18:40:23        126 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-reports/Dataloader.json2022-07-07 18:40:23        127 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-reports/GPUMemoryIncrease.json2022-07-07 18:40:23        198 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-reports/IOBottleneck.json2022-07-07 18:40:23        119 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-reports/LoadBalancing.json2022-07-07 18:40:23        151 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-reports/LowGPUUtilization.json2022-07-07 18:40:23        179 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-reports/MaxInitializationTime.json2022-07-07 18:40:23        133 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-reports/OverallFrameworkMetrics.json2022-07-07 18:40:23        465 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-reports/OverallSystemUsage.json2022-07-07 18:40:23        156 sagemaker/models/output/xgboost-2022-07-07-18-35-55-436/rule-output/ProfilerReport-1657218955/profiler-output/profiler-reports/StepOutlier.jsonAs you can see only the profiler report in created which does not interest me. Why isn't there a CreateXGBoostReport folder generated with the training report? How do I generate this/what am I missing?FollowComment"


Question: Nameserver on Cloudflare registered domain name

My domain name is registered with Cloudflare.How do I add my AWS Lightsail nameservers to Cloudflare?FollowComment

Answer: "1Accepted AnswerThe following documents are available.The name servers of the DNS zone created by AWS Lightsail can be used by registering them with Cloudflare.https://lightsail.aws.amazon.com/ls/docs/en_us/articles/lightsail-how-to-create-dns-entryIf this answer leads to a resolution, please approve the answer for the betterment of the community.CommentShareEXPERTRiku_Kobayashianswered a month agosamuelOlacom a month agoThanks for your answer. Cloudflare is my domain registrar and AWS is my server. Since it’s not possible to add nameserver on Cloudflare, What do I do to point AWS to Cloudflare or point my domain to AWS?ShareRiku_Kobayashi EXPERTa month agoNS records for DNS zones created by AWS must be registered with Cloudflare.Share"


Question: Overlapping IP with GWLB

"Hi,I understand from this article that GWLB supports overlapping CIDRs through the GENEVE protocol. However, does this also work with TGW?ThanksFollowComment"

Answer: 0You can connect VPCs with the same (or overlapping) IP ranges to Transit Gateway but you cannot route between them. Transit Gateway doesn't provide any functionality to NAT from one VPC to another with the same range.This also means that if you have VPC A and B with the same range and VPC C with a different range: If they're all connected to Transit Gateway then A can communicate with C; or B can communicate with C but not both at the same time; nor can A communicate with B - because they have the same IP range.You could (in some circumstances) use NAT Gateway to provide functionality.What I think you're actually asking is: If you're using the centralised inspection pattern (where inspection is performed in a VPC connected to Transit Gateway) how does the overlapping IP range work? Answer: It doesn't - for the reasons above. You would need to inspect in each VPC separately using Gateway Load Balancer endpoints in each VPC.CommentShareEXPERTBrettski-AWSanswered a year ago0Aviatrix is an AWS partner and has a solution for the Overlapping IP challenge. Take a look at this videohttps://youtu.be/KSrt3htsGkAYou should reach out to info@aviatrix.comhttps://aviatrix.com/learn-center/cloud-networking/handling-overlapping-ips/CommentSharenetJointsanswered a year ago


Question: You cannot add accounts to your organization while it is initializing.

Can someone help us resolve this issue so that we can start using organisations?FollowComment

Answer: "0Contacted AWS support who looked into this issueCommentSharetushar00jainanswered 4 years ago0Was it possible to keep adding account during initialisation? Thanks,CommentSharee287d672284cadfbc074ab749b3e75d8answered 4 years ago0We are having the same issue. "You cannot add accounts to your organization while it is initializing. Try again later." Can someone take a look so we can begin invitingCommentSharebluespireDevanswered 4 years ago"


Question: PowerBi automatic update with AWS Datasources

"Hello everyone,In my company we have a powerbi file that is daily updated manually (using the refresh button), this file is connected using odbc to a couple datasources, both are in AWS, one is a linux server running PostgreSQL and the other one is a RDS. After the refresh I upload the new data and, in some cases, the new pages of the report into the PowerBi Service where all other employees can see the reports.Now, the case is that it takes a lot of time to do the refresh in my powerbi desktop file and sometimes I am not physically able to do it so I was wondering how can I link the datasources directly with AWS, similarly with this diagram (https://docs.aws.amazon.com/whitepapers/latest/using-power-bi-with-aws-cloud/connecting-the-microsoft-power-bi-service-to-aws-data-sources.html), we do not know how to properly connect the datasource, we installed a gateway on a VM inside the VPC but we are not able to connect it to my powerbi desktop file.How we should have done it with the gateway? Are we forced to use Amazon Athena to update the PBI Service without supervision?Thank youFollowComment"

Answer: "1Accepted AnswerHi,I understand you want to connect Microsoft Power BI directly to data sources in AWS VPC. According to the documentation you've mentioned [1], this is possible and it requires following components and configurations:An Amazon EC2 instance which has Microsoft on-premises data gateway in the same private VPC with your RDS and PostgreSQL serverThe private VPC subnet must have a route to a NAT gateway [2][3]The security group associated with your RDS and PostgreSQL server must allow the traffic from the EC2 instance that launched in step#1Amazon Athena is another supported data source by the Microsoft Power BI gateway, and it is not required for such an use case.I'd suggest you to kindly double check the VPC network and security groups configuration according to the above-mentioned steps. If the issue still persists, please don't hesitate to cut a technical support ticket from your AWS account with network or RDS topic, our support team will be happy to help you further troubleshoot and resolve the issue![1] https://docs.aws.amazon.com/whitepapers/latest/using-power-bi-with-aws-cloud/connecting-the-microsoft-power-bi-service-to-aws-data-sources.html[2] https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-scenarios.html#public-nat-internet-access[3] https://aws.amazon.com/premiumsupport/knowledge-center/nat-gateway-vpc-private-subneCommentShareEthan_Hanswered 6 months agoMartinRios 6 months agoThanks Ethan_H, as you said I tried again the installation of the gateway and the EC2 instance and I noticed that there was some problems in the configuration of the ODBC. The EC2 had all the required settings such as same VPC, NAT routes, DB access, ODBC configs. But the issue was that the name of the ODBC driver was different from the one that I use in my computer. I did what they say in this reference 1 to match the driver name, I checked that the version and the type were the same and after that I corrected the name.After that I restarted the gateway on the EC2 then suddenly it worked.Now I can succesfully refresh the data on the PowerBi Service using the AWS gateway.Share"