Question: Restricting IAM CreateRole to disallow trust policies with external AWS accounts

"Let's say an organization has a policy that denies the CreateRole action for most users. However, this restriction hinders rapid development as it blocks users from using the console wizard's create role features for various AWS services.My goal is to allow users to create IAM roles using the console wizard, but restrict them from creating roles with trust policies that include external AWS account numbers or IAM users/roles from different accounts. The trust policies should only allow resources within the current AWS account. (Any other restrictions I should consider?)I have come up with an IAM policy that uses the StringNotLike condition operator on the aws:PrincipalARN condition key. Here's the IAM policy I have in mind:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": "iam:CreateRole",      "Resource": "*",      "Condition": {        "StringNotLike": {          "aws:PrincipalARN": "arn:aws:iam::*:root"        }      }    }  ]}And then I would add an SCP in AWS Organizations to only allow CreateRole (with the above restrictions) in development accounts.I am concerned that this approach might not cover all cases, especially when the trust policy includes an AWS IAM user ARN from a different account. Is this the best approach for achieving my goal, or is there a better way to restrict CreateRole action to trust policies containing only the current AWS account's resources?FollowComment"

Answer: "3Accepted AnswerHere may be one possible solutions that may work. Granting a user permissions to pass a role to an AWS service in the AWS IAM Documentation.This grants Users permissions to create service roles in order to launch as EC2 instance. I am curious if it will be effected differently for services launched through the Launch Wizard. Please reply and follow up if you continue to have issues. I'm certain there is more than one way to accomplish this task, using a combination of permission boundaries, SCP's, or Control Tower Guardrails. This was the first solution that came to mind.Granting a user permissions to pass a role to an AWS serviceExample 1Suppose you want to grant a user the ability to pass any of an approved set of roles to the Amazon EC2 service upon launching an instance. You need three elements:An IAM permissions policy attached to the role that determines what the role can do. Scope permissions to only the actions that the role must perform, and to only the resources that the role needs for those actions. You can use an AWS managed or customer-created IAM permissions policy.{    "Version": "2012-10-17",    "Statement": {        "Effect": "Allow",        "Action": [ "A list of the permissions the role is allowed to use" ],        "Resource": [ "A list of the resources the role is allowed to access" ]    }}      A trust policy for the role that allows the service to assume the role. For example, you could attach the following trust policy to the role with the UpdateAssumeRolePolicy action. This trust policy allows Amazon EC2 to use the role and the permissions attached to the role.{    "Version": "2012-10-17",    "Statement": {        "Sid": "TrustPolicyStatementThatAllowsEC2ServiceToAssumeTheAttachedRole",        "Effect": "Allow",        "Principal": { "Service": "ec2.amazonaws.com" },       "Action": "sts:AssumeRole"    }}                    An IAM permissions policy attached to the IAM user that allows the user to pass only those approved roles. You usually add iam:GetRole to iam:PassRole so the user can get the details of the role to be passed. In this example, the user can pass only roles that exist in the specified account with names beginning with EC2-roles-for-XYZ-:{    "Version": "2012-10-17",    "Statement": [{        "Effect": "Allow",        "Action": [            "iam:GetRole",            "iam:PassRole"        ],        "Resource": "arn:aws:iam::account-id:role/EC2-roles-for-XYZ-*"    }]}Now the user can start an Amazon EC2 instance with an assigned role. Applications running on the instance can access temporary credentials for the role through the instance profile metadata. The permissions policies attached to the role determine what the instance can do.CommentShareabemusaanswered 2 months ago0Service Control Policies cannot be used to control the contents of trust policy documents or any other kind of resource-based policy. At this time there is no AWS-native method for preventing a user from creating a trust policy based on its content. See the link below to learn more about the difference between identity-based and resource-based policies.https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_identity-vs-resource.htmlCustomers who want to perform custom safety checks on policies often use a deployment pipeline that tests the policy with IAM Access Analyzer Policy Validator in addition to their custom safety checks instead of allowing users to make policy changes directly.https://docs.aws.amazon.com/IAM/latest/UserGuide/access-analyzer-policy-validation.htmlCommentShareDon Edwardsanswered 2 months agoEXPERTkentradreviewed 2 months ago"


Question: Unable to create Cognito User Pool

"There appears to be a persistent issue with the creation of a new Cognito User Pool. While just a guess, perhaps the TLD ".pt" is unsupported as an email address specification for verification emails and such.Steps to reproduce:Cognito > Manage User Pools > Create a user pool > Step through settings > Selected options "Email address or phone number", "Allow both...", "Email" attribute required > Next step > Next step > Create role (can't select previously created roles?) > Next step > SES Region: US West / FROM email address ARN: hello@hello.pt  (verified) / FROM email address and REPLY-TO email address: "Hello hello@hello.pt" > Next step > Next step > Next step > Next step > Create pool > The following error has been copied and pasted as is and the HTML entities are what I see also:1 validation error detected: Value 'Hello <hello@hello.pt>' at 'emailConfiguration.replyToEmailAddress' failed to satisfy constraint: Member must satisfy regular expression pattern: [\p{L}\p{M}\p{S}\p{N}\p{P}]+@[\p{L}\p{M}\p{S}\p{N}\p{P}]+ (Service: AWSCognitoIdentityProviderService; Status Code: 400; Error Code: InvalidParameterException; Request ID: f1c2d27c-e1ce-4fca-b813-f64051ba32eb)Based on the errant value mentioned above, "Hello hello@hello.pt" appears to be an undesirable value for "FROM email address" and/or "REPLY-TO email address" inputs on the Manage Customizations page (e.g. https://us-west-2.console.aws.amazon.com/cognito/users/?region=us-west-2#/pool/new/message-customizations). The placeholder value of the "FROM email address" input suggests this format. When I change these values to simply "hello@hello.pt", I receive the following error instead:Cannot configure From email address for default email configuration (Service: AWSCognitoIdentityProviderService; Status Code: 400; Error Code: InvalidParameterException; Request ID: 3119206a-323d-4bc1-905e-3d865cf7a6b2)Any help would be much appreciated.FollowComment"

Answer: "0kevireillyI was facing the same issue. What worked for me was, since I provided the FROM Email ARN, just remove the entry from the "FROM Email address". It seems that if you provide the SES ARN, it doesn't take the EmailCommentSharegojhaanswered 3 years ago0Thanks SO much!!!  This worked like a charm!CommentShareAmyBraswellanswered 3 years ago0Thanks SO much!!! This worked like a charm!CommentShareAmyBraswellanswered 3 years ago"


Question: Deploy RDS migrations from GitHub with Actions

"I have a stack configured in Cloudformation with RDS (Postgres) on a private subnet, to be accessed from Elastic Beanstalk which has ec2 instances on the public subnet.I'd like to use sqitch to manage my database migrations (updates.) I have a GitHub repository specifically for my Postgres database, and each time there is a commit to the main branch I'd like to have a GitHub Action deploy the migration to my RDS instance using the sqitch command.How can I do this with RDS on the private subnet? Is there some way I could use eb ssh in a GitHub Action to create an ssh tunnel to RDS, such that sqitch can connect directly from GitHub to the DBMS on RDS and deploy the migration?Should I setup a bastion host? Or should I have a GitHub Action that somehow creates an ephemeral ec2 instance to retrieve my database repository and deploy the migration to RDS on the private subnet?Or are there alternatives I haven't thought of?Apologies if I've mixed up some AWS/Cloudformation terminology, I'm pretty new to this.FollowComment"

Answer: "1Hello!You should create a bastion host to access your RDS instance. With AWS Systems Manager it's not necessary anymore to create the bastion host in a public subnet. You can connect to the host directly with the AWS CLI. You can then set up port forwarding to bind the remote database port to a local port for the migration.Please see instructions for setting up Systems Manager here: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started.htmlHere are instructions for Systems Manager port forwarding: https://aws.amazon.com/blogs/aws/new-port-forwarding-using-aws-system-manager-sessions-manager/CommentSharemasseanswered 7 months agomattdeluco 6 months agoIf the bastion host isn't on the public subnet, how would I connect to the forwarded port?Share"


Question: HIVE_BAD_DATA: Error Parsing a column in the table: empty String

"I have an error in athena when select double item.(only a specific field)at some date, there's no error but most of dates I got the error.HIVE_BAD_DATA: Error Parsing a column in the table: empty StringI migrated data using appflow and cralwed as table using aws glue.please let me know the reason of this error and how to solve itFollowComment"

Answer: "0as an example if you have a column defined as a "double" in the schema but there's a record in your data that's an empty string, you'll get this error.to solve thisCheck Your DataCheck Your Schemaor you can try another SQL server to remove spaces or other irrelevant data from the imported dataSELECT CASE     WHEN trim(column_name)='' THEN null     ELSE cast(column_name as double)     END as column_nameFROM table_nameCommentShareEXPERTsedat_salmananswered 23 days agorePost-User-4840232 21 days agoThank you for the answer:) I tried this but I got an error because trim cannot be applied to double.Share0Such error can occur when the table column with "double" data type is having an empty record value in it. Validate your data of the impacted column and replace the empty records with 0 or NULL using SQL statement syntax supported by Presto. Else, if possible you can also try changing the data type of the columns.Similar issues are highlighted in AWS knowledge center article 1 and article 2 also.CommentShareSUPPORT ENGINEERJaishu_Aanswered 19 days ago"


Question: CloudFront | default cache key | wildcard domain

"Hi, according to https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/understanding-the-cache-key.html the default cache key is the CF distribution name, but the example shows that it's the host header.What is right?I'm asking because I have wildcard domain (*.example.com) that resolves to the CF distribution (CNAME) and if the cache key is based on the host header it won't cache objects between different domains. (for example test.example.com and test2.example.com won't share cached objects)Is there a way to overcome this? i.e. I want the cache key to include the URI and the actual CF distribution name, not the host header.Thanks,RoiFollowComment"

Answer: "1Accepted AnswerHi There,I understand that as per the document you have read [1] you analyzed that the default cache key is the CF distribution name, but the example shows that it's the host header.I understand the confusion, but no, the Host header is not included in the cache key by default. For that, one must select Host under "Cache Based on Selected Request Headers". What the documentation you linked refers to is that the CloudFront default domain is included in the cache key, by default, but not any custom Alternate Domain Names you may have.With that said, to answer your question, the scenario you provided will only result in two different cache keys if Host is whitelisted. Otherwise the cache key will be the same irrespective of the CNAME you use to access the content in CloudFront.I hope you find this helpful[1] https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/understanding-the-cache-key.htmlPlease note that I personally value your feedback and would appreciate it if you accept and thumbs up this response if it satisfies your questionCommentShareMfaneloanswered a year ago"


Question: Test sharding in aws

"Hello,I have a test suite that takes more than 150 minutes to complete both in android and ios,and I wanted to run my tests in parallel (running different tests from the same test suite in multiple devices)I bought two slots (one for android and one for ios), so would it be possible for me to implement test sharding?or is there any other strategy I'm not aware of? because a lot of my tests aren't run since they exceed the time limit of 150 minutes per device.Thank you very muchEdited by: aguspeq on Jan 16, 2021 10:40 AMFollowComment"

Answer: "0Accepted AnswerHi aguspeq,We don't support conventional sharding as of today. The way I can think of achieving it would be building sharding logic yourself (by using test targets) and run different sections of my suite per run on different device.CommentShareGuneesh-awsanswered 2 years ago"


Question: Lambda+ALB vs Lambda+API GW

"Lambda can be invoked by http endpoint via ALB as per the links pasted below. On the other hand, API GW can also be used to invoke the lambda (this approach is recommended widely over online). When to use ALB and when to use API GW ? Are there any recommendation on this ? Are there any key differentiating factors to pick one over the other ?Please find the few below differences that could figure out. (Pls correct me if i am wrong)API GW aggregates multiple service end points or multiple lambdas whereas one ALB can point to only one lambda.All requests through that ALB will go the same lambda and the code inside the lambda need to identify which functionality to execute.ALB Linkshttps://docs.aws.amazon.com/lambda/latest/dg/services-alb.htmlhttps://aws.amazon.com/blogs/networking-and-content-delivery/lambda-functions-as-targets-for-application-load-balancers/FollowComment"

Answer: "4Accepted AnswerIn case of relative low traffic (around 5 million requests per month), I would discourage the use of ALB + Lambda for services that are more normal REST API and when you don't need any of functionalities offered by ALB (like the openID authentication, etc), as the cost of having ALB+Lambda is much higher for .Anyway, you could use ALB + Lambda, when you are already using ALB for your EC2 instances.For example you have a domain for your application "www.myapp.com" pointing to your ALB, and you want to extend some of the functionalities of your domain using lambda.You can create new ALB targets pointing to lambda.For example you can create the target "/billing" to point to lambda, and the rest of the request to go to EC2, doing this you spread the traffic of your domain between different backends.Major differences between API GW and ALBAPI GWALBRequest/secondsLimited to 10000virtually unlimitedIntegrationLambda and many others AWS servicesEC2, ECS, private IP addresses, LambdaRoutingOnly on path basedPath base and others like Requesters IP, Http Headers, Http method, etcCostPay per requestPay by time + Request (LCU)Access LoggingCloudwatchS3 bucketCommentShareMikianswered a year agomreferre EXPERTa year agoI'd argue that, from a cost optimization perspective, there is a break-even point when one will cost more than the other based on # of requests.ShareMiki a year agoThanks mreferre, you are right, I updated the responseShare"


Question: Can customers opt out of Redshift's serverless compute engine?

In the recent Redshift announcement I read:"query compilations are now scaled to a serverless compute engine beyond the compute resources of the leader node of your cluster".The queries can contain PII information in the text and having this information processed in what looks like a multi-tenant compiling environment might be an issue in some industries.I have had customers skipping even local logging of query logs to ensure such information can't get exposed.Can customers opt out of Redshift's serverless compute engine?FollowComment

Answer: "0Accepted AnswerThe query parsing, the optimization and the plan creation (the parts that potentially use the PII data in the query text and/or in the statistic) are carried over inside the leader node.Only the C++ code created at the end of the optimization (that doesn't contain any PII data) is sent to the serverless infrastructure to be compiled.This means that there is no reason to need an opt-out option.CommentShareMarco Ullascianswered 3 years ago"


Question: How do I specify a launch path to be used for a specific game session on Gamelift when using multiple launch paths or launch parameters?

"Hi, when I'm hosting a Gamelift Game Session it defaults to the first launch path and launch parameter I've set while creating a fleet. How can a game session be specified to link to a specific launch path that's running inside the fleet? Can the property Game properties for the game session be used for this and if so, how should it be formatted?Best regards,AlexFollowComment"

Answer: "0Hey Alex,As part of your fleet's runtime configuration you can specify multiple server process configurations, where each represents one game server executable. As part of this you can additionally also specify  the number of processes to run concurrently (ConcurrentExecutions). Each one of these processes will use the associated configuration.Here's the relevant documentation on how to manage multiple server processes with different configurations: https://docs.aws.amazon.com/gamelift/latest/developerguide/fleets-multiprocess.htmlThanks!CommentShareShashankJ_AWSanswered 3 months ago"


Question: Interactive multi-execution shell

"Hello All,I am looking for a way to interactively run shell commands in multiple instances simultaneously.I know I can run (shell) commands interactively with a single instance through SSM console, and non-interactive scripts, document, with multiple instances through Run Command.I am using the SSH client tool with 'multi-execution mode' to perform this kind of interactive work. However, I have to open the port to do the work.If Systems Manager can add 'multi-execution mode' to SSM or interactivity to Run Command, or just a new tool. Then I do not have to poke a hole in my VPC to do the work.Best regards,TigerFollowComment"

Answer: "0Hello! SSM has the AWS-StartInteractiveCommand document. For more information you can read the following links:https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-restrict-command-access.htmlhttps://aws.amazon.com/blogs/mt/limit-ssm-interactive-session-commands-by-users/Please let me know if the above helps!CommentShareSUPPORT ENGINEERFrancisco_Lanswered a year ago0Systems Manager allows you to use your SSH client to interactively access instances without requiring network ingress to your EC2 instances.You will need:An SSH client that can be configured with ProxyCommand, such as OpenSSH; andA recent version of the AWS CLI installed.Once you have that, you can set up your SSH configuration file to access your instances, like so:# ~/.ssh/config file# SSH over Session Managerhost i-* mi-*    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"Then, when you run something like ssh i-01234567890, the session will be transported via SSM.If your multi-session manager uses the standard ssh program under the hood, your experience should be identical to that of using ssh directly.See Enabling SSH connections for Session Manager for details.CommentShareEXPERTMichael_Fanswered a year ago"


Question: Glue Data Catalog storage charge

"I do not understand this charge.$1 per 100,000 objects per month for AWS Glue Data Catalog storage 1,797,692.518 Obj-Month $17.98What should I do in order to avoid it?FollowComment"

Answer: "0Hi Shantanu,I understand you're looking to get recommendations to avoid unwanted charges with AWS Glue. While the first 1 million objects stored are free per month. You're correct that the cost will be $1.00 per 100,000 objects stored that are above the initial 1 million.In terms of your ask, a clearer understanding or your use case and service config would be needed to best be able to assist.Please consider reaching out to our Sales & Business Support team to via this contact form.Alternatively, you can also reach out via a support case to discuss this with one of our Support Engineers. Check out this Knowledge Center article  for more details on reaching out via a support case.Regards,Kraig E.CommentShareEXPERTAWS Support - Kraiganswered 6 months ago"


Question: EC2 machines under ALB went unreachable

While running ECS application behind all the applications behind ALB went unreachable.FollowComment

Answer: 0Hello there.What do you mean by "went unreachable"? Are ALB healthchecks not passing? or is it about the instances being network isolated?Can you add a bit more details please?Thank you.CommentSharemarianoanswered 3 months ago


Question: How to create a Glue Workflow programmatically?

"Is there a way to create a Glue workflow programmatically?I looked at CloudFormation  but the only one I found is to create an empty Workflow (just Workflow name, Description and properties).https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-glue-workflow.htmlI tried to look at the APIs as well (https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-workflow.html), and even if there are all the data type for all the structures the only create API is again only adding the blank box.Am I missing something? How do we create the workflow from the blueprint in Lake Formation?is some sort of pre-assembled JSON file that we just link to the workflow in glue?can you do something similar, or need to wait for the customizable blueprints?Thank youUPDATE:As it can be derived from the snippet of code from the Accepted Answer, the key is to consider that it is actually the :AWS::Glue::Triggerconstruct that helps you build the Workflow.Specifically,  you need to:create the Workflow with AWS::Glue::WorkflowIf you need create Database and connection as well ( AWS::Glue::Database , AWS::Glue::Connection)Create any Crawler and any Job you want to add to the workflow using : AWS::Glue::Crawler or AWS::Glue::JobCreate a first Trigger (AWS::Glue::Trigger ) with Type : ON-DEMAND , and Actions = to the firs Crawler or job your Workflow need to launch and Workflowname referencing the Workflow created at point 1Create any other Trigger with Type : CONDITIONALBelow an Example (to create a Workflow that launch a Crawler on an S3 Bucket (cloudtraillogs) , if successfull launch a python script to change the table and partition schema to make them work with Athena )).hope this helps---AWSTemplateFormatVersion: '2010-09-09'Description: Creates cloudtrail crwaler and catalog for Athena and a job to transform to ParquetParameters:   CloudtrailS3:     Type: String    Description: Enter the unique bucket name where the cloud trails log are stored  CloudtrailS3Path:     Type: String    Description: Enter the path/prefix that you want to crawl   CloudtrailDataLakeS3:     Type: String    Description: Enter the unique bucket name for the data lake in which to store the logs in Parquet FormatResources:      CloudTrailGlueExecutionRole:    Type: 'AWS::IAM::Role'    Properties:      AssumeRolePolicyDocument:        Version: "2012-10-17"        Statement:          - Effect: Allow            Principal:              Service:                - glue.amazonaws.com            Action:              - 'sts:AssumeRole'      Path: /      ManagedPolicyArns:        - arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole    GluePolicy:    Properties:      PolicyDocument:        Version: '2012-10-17'        Statement:        - Action:          - s3:GetBucketLocation          - s3:GetObject          - s3:PutObject          - s3:ListBucket          Effect: Allow          Resource:          - !Join ['', ['arn:aws:s3:::', !Ref CloudtrailS3] ]          - !Join ['', ['arn:aws:s3:::', !Ref CloudtrailS3, '/*'] ]          - !Join ['', ['arn:aws:s3:::', !Ref CloudtrailDataLakeS3] ]          - !Join ['', ['arn:aws:s3:::', !Ref CloudtrailDataLakeS3, '/*'] ]        - Action:          - s3:DeleteObject          Effect: Allow          Resource:          - !Join ['', ['arn:aws:s3:::', !Ref CloudtrailDataLakeS3] ]          - !Join ['', ['arn:aws:s3:::', !Ref CloudtrailDataLakeS3, '/*'] ]      PolicyName: glue_cloudtrail_S3_policy      Roles:      - Ref: CloudTrailGlueExecutionRole    Type: AWS::IAM::Policy  GlueWorkflow:    Type: AWS::Glue::Workflow    Properties:       Description: Workflow to crawl the cloudtrail logs      Name: cloudtrail_discovery_workflow  GlueDatabaseCloudTrail:    Type: AWS::Glue::Database    Properties:      # The database is created in the Data Catalog for your account      CatalogId: !Ref AWS::AccountId         DatabaseInput:        # The name of the database is defined in the Parameters section above        Name: cloudtrail_db        Description: Database to hold tables for NY Philarmonica data        LocationUri: !Ref CloudtrailDataLakeS3    GlueCrawlerCTSource:    Type: AWS::Glue::Crawler    Properties:      Name: cloudtrail_source_crawler      Role: !GetAtt CloudTrailGlueExecutionRole.Arn      #Classifiers: none, use the default classifier      Description: AWS Glue crawler to crawl cloudtrail logs      Schedule:         ScheduleExpression: 'cron(0 9 * * ? *)'      DatabaseName: !Ref GlueDatabaseCloudTrail      Targets:        S3Targets:          - Path: !Sub            - s3://${bucket}/${path}            - {              bucket: !Ref CloudtrailS3,              path : !Ref CloudtrailS3Path              }            Exclusions:               - '*/CloudTrail-Digest/**'              - '*/Config/**'                  #TablePrefix: ''      SchemaChangePolicy:        UpdateBehavior: "UPDATE_IN_DATABASE"        DeleteBehavior: "LOG"      Configuration: "{\"Version\":1.0,\"CrawlerOutput\":{\"Partitions\":{\"AddOrUpdateBehavior\":\"InheritFromTable\"},\"Tables\":{\"AddOrUpdateBehavior\":\"MergeNewColumns\"}}}"  GlueJobConvertTable:    Type: AWS::Glue::Job    Properties:      Name: ct_change_table_schema      Role:        Fn::GetAtt: [CloudTrailGlueExecutionRole, Arn]      ExecutionProperty:        MaxConcurrentRuns: 1      GlueVersion: 1.0      Command:        Name: pythonshell        PythonVersion: 3        ScriptLocation: !Sub          - s3://${bucket}/python/ct_change_table_schema.py          - {bucket: !Ref CloudtrailDataLakeS3}      DefaultArguments:        '--TempDir': !Sub          - s3://${bucket}/glue_tmp/          - {bucket: !Ref CloudtrailDataLakeS3}        "--job-bookmark-option" : "job-bookmark-disable"        "--enable-metrics" : ""    DependsOn:      - CloudTrailGlueExecutionRole  GlueSourceCrawlerTrigger:    Type: AWS::Glue::Trigger    Properties:      Name: ct_start_source_crawl_Trigger      Type: ON_DEMAND      Description: Source Crawler trigger      WorkflowName: !Ref GlueWorkflow      Actions:      - CrawlerName:          Ref: GlueCrawlerCTSource    DependsOn:      - GlueCrawlerCTSource  GlueJobTrigger:    Type: AWS::Glue::Trigger    Properties:      Name: ct_change_schema_Job_Trigger      Type: CONDITIONAL      Description: Job trigger      WorkflowName: !Ref GlueWorkflow      StartOnCreation: 'true'      Actions:      - JobName: !Ref GlueJobConvertTable      Predicate:        Conditions:        - LogicalOperator: EQUALS          CrawlerName: !Ref GlueCrawlerCTSource          CrawlState: SUCCEEDED        Logical: ANY    DependsOn:    - GlueJobConvertTableFollowComment"

Answer: "0Accepted AnswerFor a workflow, you'll want a mixture of trigger, crawler and jobs. There's a fairly good coverage in CloudFormation but you might still need some Custom Resources and/or to kick off something like a Step Function.Example (from the internet, just trying to minimize traffic to the post):---Parameters:  OutputPathLocation:    Description: Output path of the transformation file    Type: String  WorkFlowName:    Description: Name of the workflow    Type: String    Default: test-workflow  MyScriptLocation:    Description: Location of ETL script    Type: StringResources:  MyGlueWorkFlow:    Type: AWS::Glue::Workflow    Properties:      Description: test cfn workflow      Name:        Ref: WorkFlowName  MyGlueCrawler:    Type: AWS::Glue::Crawler    Properties:      DatabaseName: cfndb      Description: My crawler      Name: MyGlueCrawler      Role: AWSGlueServiceRole      TablePrefix: cfn_      Targets:        S3Targets:        - Path: s3://crawler-public-us-east-1/flight/2016/csv    DependsOn:    - MyGlueWorkFlow  MyGlueCrawlerTrigger:    Type: AWS::Glue::Trigger    Properties:      Name: MyCrawlerTrigger      Type: ON_DEMAND      Description: Crawler trigger      WorkflowName:        Ref: MyGlueWorkFlow      Actions:      - CrawlerName:          Ref: MyGlueCrawler    DependsOn:    - MyGlueCrawler  MyGlueJob:    Type: AWS::Glue::Job    Properties:      Command:        Name: glueetl        ScriptLocation:          Ref: MyScriptLocation      Description: My workflow job      GlueVersion: '1.0'      Name: MyGlueJob      Role: AWSGlueServiceRole      DefaultArguments:        "--outputpath":          Ref: OutputPathLocation    DependsOn:    - MyGlueCrawler  MyGlueJobTrigger:    Type: AWS::Glue::Trigger    Properties:      Name: MyGlueJobTrigger      Type: CONDITIONAL      Description: Job trigger      WorkflowName:        Ref: MyGlueWorkFlow      StartOnCreation: 'true'      Actions:      - JobName:          Ref: MyGlueJob      Predicate:        Conditions:        - LogicalOperator: EQUALS          CrawlerName:            Ref: MyGlueCrawler          CrawlState: SUCCEEDED        Logical: ANY    DependsOn:    - MyGlueJobCommentShareEXPERTRaphaelanswered 3 years agorePost-User-4238514 5 months agoWhere do we run this code ?. Which script should I use in order to utilize this script ?. I tried creating with boto3 only I was able to create the workflow , wasn't able to add triggers/jobsShare"


Question: Want to create g5g instance for EC2

"I want to create a g5g instance for my EC2 deep learning.However when I choose Ubuntu Server 20.04 LTS (HVM), SSD Volume Type 64bit (x86) g5g familly is not supported.I have switched to Ubuntu Server 20.04 LTS (HVM), SSD Volume Type Arm, g5g family supports it, but when I recreate it, I get the following error message: "The requested configuaration is currently not supported. Please check the documentation for supported configuration".I have switched to AWS Deep Learning AMI (Ubuntu 18.04) but g5g series is not supported.So I don't know how I can create a g5g family instance for EC2.Can anyone help me for this?FollowComment"

Answer: "2EC2 g5g instances use the arm64 CPU architecture, not the x86 CPU architecture. So you will need to launch your instance with an arm64-compatible AMI.CommentShareEXPERTMichael_Fanswered a year ago0Accepted AnswerI have found that to create EC2 g5g I need to use Marketplace Deep Learning Nvidia. I tried creating under this AMI. Thank Michael_F for your supportCommentShareTrungPNanswered a year ago0There are a few Deep Learning AMIs that can support g5g instance type, below are their release notes:https://aws.amazon.com/releasenotes/deep-learning-ami-graviton-gpu-pytorch-1-10-ubuntu-20-04/https://aws.amazon.com/releasenotes/aws-deep-learning-ami-graviton-gpu-tensorflow-2-6-ubuntu-20-04/https://aws.amazon.com/releasenotes/aws-deep-learning-ami-graviton-gpu-cuda-11-4-ubuntu-20-04/You could probably use one of the above AMIs for your g5g work.CommentShareJunpuanswered a year ago"


Question: error in creating  lambda function

"I am unable to create a lambda function, I don't know the error exactlyFollowComment"

Answer: "1Hi,this is due to your account being considered "suspect" and being flagged by AWS Support.I suggest you to reach out to them so they can unblock it.You can see similar issues here: https://repost.aws/questions/QU1smotENFRF2kobsHIQMhmQ/lambda-wont-load, https://repost.aws/questions/QU-uyxOY7aSWeBVbVCkbxRmg/aws-lambda-function-issueHope it helps ;)CommentShareEXPERTalatechanswered a month ago"


Question: AWS API Gatway to upload File To S3 without AWS Lambda

"Hii have create 1 aws api gateway which post file to s3 using aws service integration , api works fine from backend code or postman , but when i am trying to post the api from vue js or any java script  , its thorn the CORS error , i have enabled the cors setting , and tested using curl its return CORS header in response .As per Cloud watch preflight called is failing with 500 .`2022-05-25T13:38:14.343+05:30(18e87194-3f4d-4ecf-b8fc-e6b1069eb878) Method request body before transformations: [Binary Data]2022-05-25T13:38:14.343+05:30(18e87194-3f4d-4ecf-b8fc-e6b1069eb878) Execution failed due to configuration error: Unable to transform request`my vue js codeconst article =  "Vue POST Request Example" ;const headers = {"Content-Type":'text/plain'};axios.put("https://ivdfgjilc5.execute-api.ap-southeast-1.amazonaws.com/devv1/vxpoc-es-appsearch/tests3/tryfinal44.txt", article,headers).then((response) => {console.log(response)}).catch(error => {this.errorMessage = error.message;console.error("There was an error!", error);});any suggestion on this will be helpfulFollowComment"

Answer: "0Accepted Answerhi we able to solve this issue , after discussing with aws support , as problem was not with cors. actual problem was with preflight request was giving 500 which i was mentioned in question  .with this errorerror Execution failed due to configuration error: Unable to transform requestwe figure out Binary media in api setting / was causing the problem in my case , i have changed to image/png and its started working . Thanks for all help.CommentShareAWS-User-3783034answered a year ago0I am not sure what is causing the error, but why are you uploading through API Gateway? Why not upload directly to S3? Just generate a pre-signed upload URL using a Lambda function and upload directly using that URL. It will reduce the latency and also will bypass API Gateway's 10MB payload limit.CommentShareEXPERTUrianswered a year agoAWS-User-3783034 a year agohi i have requirement where i wanted to send event data direct to s3  as text file . so i do not have any file to upload. its actually text data which will store as file . but still i will look ur suggestion .Share0Did you enable CORS for both API Gateway and S3?CORS docs:https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-cors.htmlhttps://docs.aws.amazon.com/AmazonS3/latest/userguide/cors.htmlCommentSharematthew_danswered a year agoAWS-User-3783034 a year agoyes i enabled the CORS for both , i able to solve it . posting the answer below .Share"


Question: How to download entire sql query on RDS

"I am using RDS and there are some queries causing load issues. How to download the entire query?When I use the download button on the specific query, it only downloads a partial query. Is there a way to inform the AWS team that their SQL download button is broken?FollowComment"


Question: Frontend fails but still update

"Hello,The frontend build fails even though there are no errors, only warnings for the npm install. I confirmed that build is working via CLI (using create react app). Also, confirmed my front end changes are deployed.It seems like Amplify Console is reporting a failed build when in reality it worked.FollowComment"

Answer: "0Also, builds sometimes take up to 30 mins.CommentSharekay33answered 4 years ago0Now my app is not updating at all.CommentSharekay33answered 4 years ago0cache size issue. see here for workaround:https://github.com/aws-amplify/amplify-cli/issues/1415CommentSharekay33answered 4 years ago"


Question: How can i check files and data stored in EBS volume

"Hello members, I am running ec2 instance with 20 GB ebs volume.Actually still i have not stored any files in ebs volume using cpanel but when i am visiting billing dahsboard it showing me 14 GB usage of EBS volume so can anyone tell me where this storage would be used and how can i see its data?FollowComment"

Answer: "0Hi! With EBS volumes you are not charged for the space you use inside the volume, - you are charged for the space you provision. You mentioned you provisioned 20GB volume. It means that you will be charged for 20GB of EBS storage for the full month (pricing unit = Gb/month).In your example, I believe you are looking at the billing console of current month (bill for Nov-2022), which is not a finalized bill yet for November. Let's assume that you had your volume available at the very beginning of the month, and let's say you looked at the bill yesterday (23th Nov), which means that you likely saw the bill for only first 22 days of the month. In that case, for full November you would get charged for all 20GB of your provisioned volume. However, for 22 days of the month, you would only get charges for (20GB / 30days * 22days) = approx. 14GBYou can find more info about EBS pricing and how you are charged here: https://aws.amazon.com/ebs/pricing/CommentShareEXPERTNataliya_Ganswered 6 months ago"


Question: RHEL AMI questions

"A customer is asking the questions below:Are ec2-utils package supported on RHEL images?do RHEL based AMIs not have ENI drivers? If that is true, then who is on point to resolving the issues? AWS support or RHEL as the AMI are provided by AWS.ThanksFollowComment"

Answer: "0Accepted AnswerFrom what I can tell, the ec2-net-utils package doesn't contain drivers, rather it's scripts and udev rules to support the adding of additional ENIs. You can look at them in detail on an Amazon Linux instance:ec2-user@ip-172-31-0-39 ~]$ rpm -ql ec2-net-utils/etc/dhcp/dhclient.d/ec2dhcp.sh/etc/init/elastic-network-interfaces.conf/etc/modprobe.d/ixgbevf.conf/etc/sysconfig/modules/acpiphp.modules/etc/sysconfig/network-scripts/ec2net-functions/etc/sysconfig/network-scripts/ec2net.hotplug/etc/udev/rules.d/53-ec2-network-interfaces.rules/etc/udev/rules.d/75-persistent-net-generator.rules/sbin/ec2ifdown/sbin/ec2ifscan/sbin/ec2ifup/usr/share/man/man8/ec2ifdown.8.gz/usr/share/man/man8/ec2ifscan.8.gz/usr/share/man/man8/ec2ifup.8.gzYou can still at multiple ENIs in RHEL, you just don't have those scripts to help with it.On the support question there's good info on this in the RHEL Partner FAQ:"Q: How does support work for Red Hat Enterprise Linux on Amazon EC2?All instances of Red Hat Enterprise Linux on Amazon EC2 will receive updates at the same time that updates are made available from Red Hat. To be notified when updates are available, subscribe to Red Hat Security Announcements.  Updates can be installed using standard Red Hat systems management tools.Additionally, Red Hat Enterprise Linux on Amazon EC2 is covered under AWS Premium Support. AWS Premium Support customers can contact AWS for help to resolve issues that are related to Red Hat Enterprise Linux on Amazon EC2. AWS Premium support will facilitate resolution with Red Hat support for any issues that require escalation. AWS customers using RHEL will not contact Red Hat directly unless they have purchased EC2 through Red Hat Cloud Access and have a direct support agreement with Red Hat."CommentShareEXPERTAdam_Wanswered 6 years ago"


Question: Amazon Products API Help: The Access Key ID or security token included in the request is invalid.

"Hi, I have been trying to get started with the amazon products API, and I am currently an affiliate, which means I should already have access.Upon setting up a new project with some of Amazon's sample product search code, and inputting my access key/security token, it gives me the error "The Access Key ID or security token included in the request is invalid."I've looked at the documentation here https://webservices.amazon.com/paapi5/documentation/quick-start.html.I tried running a search on https://webservices.amazon.com/paapi5/scratchpad/index.html with the store ID and key/token and it said "Error! InvalidSignatureThe request has not been correctly signed. If you are using an AWS SDK, requests are signed for you automatically; otherwise, go to https://webservices.amazon.com/paapi5/documentation/sending-request.html#signing."I have emailed about this several times, being told to wait 72 hours and I have done that. I generated new keys, and got the same result. Finally I was told to post here for help, so I hope someone here can assist me. Judging by the tags offered on this post, this seems like it's not the right place for this question... but I'm sure someone here may still know better.If you need more information let me know. Thank you!FollowComment"

Answer: "0Does anyone have a solution for this?@gredenko Were you able to fix it?CommentShareFaulkanswered 5 months ago0@Faulk No, nobody here answered and even Amazon didn't help when I emailed. They sent generic responses and no one is taking a look at it I guess.CommentSharegredenkoanswered 5 months ago"


Question: RDS modify instance not working

"I can't modify my RDS instance, drop-downs are not working, getting invalid values on a disk size and also getting 400 error in browser console, here's couple screenshots for exampleFollowComment"

Answer: "0Check out this previous post, is it possible for you to try a different browser?https://repost.aws/questions/QU-IvTEuCPSxunWr_87NU5Bw/cant-modify-rds-instance-classYou can also try using the CLI to modify the instance class.https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/modify-db-instance.htmlCommentShareEXPERTMatt-Banswered a year agoydehtyaryov a year agoI've tried three different browsers, incognito tabs - none of these options work.Yeah, I know that I can try to use AWS CLI, but that's not what I wanted you know...I want to be able to change DB engine version, I guess I have to try to use modify-db-instance with --engine-version param, but as for the value for this param I need to view it from the AWS console and then type it into the CLI...Share"


Question: Mutual TLS with Commercial Client Certificates

"I have setup and configured an API Gateway for using mutual TLS using the AWS documentation and it is working fine. I am trying to understand if it is possible/practical to use commercial/public CA's for generating the client certificates that are used in this context. While I understand that it would be technically possible, given that only the CA certificate goes into the truststore that is used to determine which client certificates are considered valid, it would seem that ANY client certificate issued by that commercial/public CA would pass that check ... hence, does that mean that in practice only self signed client certificates should be used with this solution or is there a 'secure' way for client certs from commercial/public CA's to work as well?FollowComment"

Answer: "0If I'm reading this correctly, you're asking if the client certificate can be generated by a CA, not by AWS?From the documentation:  "You can use API Gateway to generate an SSL certificate and then use its public key in the backend to verify that HTTP requests to your backend system are from API Gateway."So Client Cert is downloaded to your system, to authenticate a request from AWS API Gateway... can you dig into why you'd want a separate trusted authority?If you're asking about a non-AWS cert, you can upload that into AWS Certificate ManagerCommentShareDan_Hanswered a year agoTAW a year agoIf I understand the documentation link that you provided, this is for having the API Gateway itself provide a client certificate to some backend service though http (presumably using mTLS) ... what I am asking about is when external systems are talking to the API Gateway through mutual TLS. In this situation, it is these external systems that need to provide a client certificate to the API Gateway as documented here: https://docs.aws.amazon.com/apigateway/latest/developerguide/rest-api-mutual-tls.html. As per this link, validation of the incoming client certs is via a truststore containing the appropriate CA(s) and it is these client certs that I am asking about here.ShareDan_H a year agoI think I'm with you now - thanks.If you want to restrict the custom API gateway domain to only your resources, make it a very tiny truststore (i.e. just your certs).If you want to let anyone ping your API gateway (think of some kind of a service provider), then you can open the truststore to any CA that you'd like. You'll want to evaluate your business case for API gateway (and all resources), in the lens of least privileges.ShareTAW a year agoCan you clarify what exactly you mean by "make it a very tiny truststore (i.e. just your certs)"? As per the documentation I referenced above for setting up mTLS, you just put the relevant CA certs in the truststore ... so I can have a very 'tiny' truststore that only contains the single CA cert for some commercial CA, however, now any incoming request containing a client cert signed by that commercial CA would be allowed to pass through to my service? Can I somehow restrict it to only client certs that I have handed out from this commercial CA ... OR as I state in my original question, is the only way to do that by using self signed client certs?Share"


Question: Does AWS Kinesis auto scaling support setting different level of parallelism for different operators?

"We are trying to run a Query on AWS Kinesis, and use autoscaling.We found that if user specified the Parallelism option in AWS Kinesis configuration, AWS Kinesis will apply the specified parallelism to all the operators.We also tried run the query without specifing any parallelism (set ConfigurationType to DEFAULT). And in this case, Kinesis will only start 1 task manager with 1 slots, even if that's not enough to run our Query.So we are wondering does AWS Kinesis auto scaling supports setting different level of parallelism for different operators?ThanksFollowComment"


Question: How to correctly download an audio file from S3 bucket using java sdk?

"I uploaded a sample audio.mp3 file to a S3 bucket. I am trying to download that file using java sdk. The file is getting downloaded, however when I play it there are some weird sound glitches which are not present in the original audio. What am I missing?String key = "audio.mp3";    InputStream s3ObjectInputStream = s3Client.getObject("audio-bucket", key).getObjectContent();    byte[] buffer = new byte[1024000];    try {        Path currentRelativePath = Paths.get("");        String path = currentRelativePath.toAbsolutePath() + "/temp.mp3";        File targetFile = new File(path);        OutputStream outStream = new FileOutputStream(targetFile);        while ((s3ObjectInputStream.read(buffer)) != -1) {            outStream.write(buffer);        }        return buffer;    } catch (IOException e) {        throw new RuntimeException(e);    }FollowComment"


Question: Confusion around SCP and IAM policy intersection ?

"Hi forum,I've recently started to use organizations (in a test account).I have a user, let's call them NotOwner, that has the following IAM policy, via group policy only (no boundaries), full S3 access:{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": "s3:*",            "Resource": "*"        }    ]}The user is invited, and accepted, into an OU with the following policy attached:{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "Statement1",            "Effect": "Deny",            "Action": [                "s3:PutBucketPolicy"            ],            "Resource": [                "*"            ]        }    ]}However, I can still affect s3:PutBucketPolicy from that user (i.e. edit the bucket policy).I'm I missing what SCP does? I thought if this policy intersected with the Not Owner IAM policy I would be denied s3:PutBucketPolicy ? I also deleted the user and created the user after setting up organizations. It also works if I set the policy as a boundary on the user directly.Regards--- Extra Info ---The owner of the bucket is another account, let's call them Owner, and that account has explicitly given permissions to this account via these policies.{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Deny",            "NotPrincipal": {                "AWS": [                    "arn:aws:iam::111111111111:root",                    "arn:aws:iam::111111111111:user/NotOwner",                    "arn:aws:iam::111111111111:user/Owner"                ]            },            "Action": "s3:*",            "Resource": [                "arn:aws:s3:::bucketname",                "arn:aws:s3:::bucketname/*"            ]        }    ]}Edited by: davporte on Jan 27, 2020 10:42 PMAh ok, I'm telling S3 ok, go for it so IAM don't matter !FollowComment"

Answer: "0I think I answered myself, IAM is not relevant S3 says the user can do this to this bucket so they can.And considering, I was in an IAM account under the Organizations root, that is also not surprising!Once I logged out/in into an account under the correct OU, it all worked fine!Edited by: davporte on Jan 27, 2020 10:45 PMEdited by: davporte on Jan 28, 2020 3:58 AMCommentSharedavporteanswered 3 years ago"


Question: Cannot Update Glue Partition Names Populating table overwrites Partition names

"I have a database that I have been working with and I am trying to add a new table through Glue.I have been able to create the table and populate data, but I cannot update the partition names in the table and are stuck with partition_0.You should be able to edit the table by using the actions button, but when I click "edit table" nothing happens in this or any other table that I have previously setup.Things I have tried:edit the table in the glue consolemanually update the partition name ALTER TABLE missing_trips partition (partition_0) RENAME TO PARTITION (agency) produces an unknown errorCreate the table with Athena and then populate. I am able to create the table, and then populate that table with a crawler, but when I do so it reverts the partition name to partition_0#2 error: [ErrorCategory:USER_ERROR, ErrorCode:UNKNOWN_ERROR] Query Id: 61c7b446-16c7-4459-94fd-f097dc15835aFollowComment"


Question: EKS - How to communication from managed node group to self-managed node groups and vice versa

"So I have created a EKS Cluster with 1 EKS managed node group. I also created a in-region Self-managed node group.I was trying to install some workloads into the cluster using Helm, by simply invoking helm install method.The first chart is installed into self-managed node group and everything works fine. But when I tried to install the second helm chart, it went to managed node group. However, the second helm chart has some dependency on the first helm chart and right now second helm chart is stuck trying to find the first helm chart. But it is in the other node group.I am wondering what kind of Security Group rule changes I need to make so that I could make this cross node group communication work ?Currently these two node groups are within the same VPC and SubnetThanksFollowComment"

Answer: "0I think you should be more intentional in your helm chart targeting nodes if these workloads are meant to run (Pod or Node Affinity) OR making sure you are exposing things via services the right way, possibly using service mesh.  If you are using default CNI, with VPC IP's, things should work.  If you use an overlay CNI, where different nodes groups have different IP ranges, etc.....  then you may need some network config.CommentShareRolyanswered 2 months ago"


Question: How to extend backup Retention for FSx

I am trying to extend the retention of already made backups in FSx for Windows. AWS Backup can have unlimited retention but in the console for FSx you can only do a maximum of 90 days. For existing backups can they be copied and then categorized as a user-initiated backup and then use the Custom Backup schedule [1] extend the retention based on whats defined? Importing the Automatic backups defined in the FSx console would be ideal but I have not found a way to do this.[1] - https://docs.aws.amazon.com/fsx/latest/WindowsGuide/additional-info.htmlFollowComment

Answer: "0Automatic backups cannot be recategorized to user-initiated backups by copying. However, you can create a custom backup plan in AWS Backup and assign a specific resource ID to include the copied backup in the new plan. Going forward, you can create custom backup schedules with retention periods of longer than 90 days.CommentShareNikhil Srivastavaanswered a year agoAWS-User-2960459 a year agoWe tried this out in the AWS Backup console and while you can assign specific resource IDs to the backup plan those are limited to existing filesystems and not backups. Is the suggestion here to restore the file system from the older automatic backup and then have AWS backup make one automatically with the older point? I think that can work but will not scale well as we have a few older backups that are aging out that we need to keep for compliance.Share"


Question: How to see the Egress Cost

"Hi,My company is migrating to AWS from On-perm. I'm doing a POC to see what will be my egress cost when all the accounts move from On-perm to AWS.I have created the existing data model in AWS Redshift and accessing the data through Azure Power BI. I want to see cost while using  the import model vs direct query. I have couple of reports in Power BI service pointing to AWS Redshift tables. I couldn't get any decent information from AWS team about how much data is being transferred from AWS to Public internet. I tried cost explorer, Bill sections and didn't find any useful information.I really appreciate if someone can show me where i can get either data transfer or egress informationThanks,MikyFollowComment"

Answer: "2Hello,Check out this blog post, I think it will help answer your questions around Cost Explorer/Data Transfer: Using AWS Cost Explorer to analyze data transfer costs.  Essentially to find Data Transfer, you will need to look under "Usage Type" and "Usage Type Group".CommentShareEXPERTChris_Ganswered 9 months agoNataliya_G EXPERT9 months ago+1 here; we also have this video that can help learn more about data transfer charges https://www.youtube.com/watch?v=UjliYz25_qoSharerePost-User-5797488 9 months agoThank you for responding to my question. I tried the cost explorer and couldn't see any charges. I can see the data flow through VPC flow logs but not in the cost explorer.ShareChris_G EXPERT9 months agoSo you're seeing egress traffic on the VPC Flow logs, but no data transfer usage in Cost Explorer?  Can you check under Billing Portal > Bills and see if there's any charges under the Data Transfer section?SharerePost-User-5797488 9 months agoThat's correct. The VPC Flow logs shows the 2 gb as egress and it shows my IP as destination address but I don't see that activity or charge in the cost explorer. There no charge in the Bills section either.Share"


Question: .no TLD Norwegian zones

"Hei,Could you please implement Norweigan .no TLDs to Route53?It would make my life so much easier :)FollowComment"

Answer: "0Unfortunately the Norwegian ccTLD .no is not currently supported on Route 53 for domain registration.  However, you can still purchase the domain from another registrar and point the domain name servers to Route 53.For a list of currently supported ccTLDs, you can find the list here: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/registrar-tld-list-geographic.htmlCommentShareEXPERTChris_Ganswered a year ago"


Question: Cannot include localhost in cloudfront CORS policy

"Hi all,I want to include a CORS policy in Cloudfront, but the UI will not allow me to add https://localhost.  I get the following error:"The parameter Access-Control-Allow-Origin contains https://localhost which is not a valid URL."I can set this in other CORS policies (such as S3) so think this should be allowed.Does anyone know if this is considered a bug? Is there a workaround?Many thanksFollowComment"

Answer: "0Hi there,Thank you for reporting this behavior when attempting to add "https://localhost" to the "Access-Control-Allow-Origin" section of a Response Header Policy. I have communicated this behavior to the CloudFront service team and they are investigating it further.There are a few possible workarounds. If you can customize the response from your origin, you can configure CloudFront to forward the "Origin" header in the origin request and have your origin return this header in its response with the desired "https://localhost" value. As noted, S3 can be configured to return this value in its CORS configuration if you're using S3 as your origin.Another approach is utilizing CloudFront Functions to add the "Access-Control-Allow-Origin: https://localhost" header to the response. An example function that can be adapted to accomplish this can be found here: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/example-function-add-cors-header-response.htmlCommentShareSUPPORT ENGINEERMichael_Lanswered a year ago"


Question: Lambda Run Remote  - Runtime.InvalidEntrypoint

"I've created a new project from a template HelloWorld. I use Rider and macOS. When I build the image with local configuration, the image is created successfully.When I run the remote configuration I receive Runtime.InvalidEntrypointThis is how my default template.yaml file looks like.AWSTemplateFormatVersion: '2010-09-09'Transform: AWS::Serverless-2016-10-31Description: >  Sample SAM Template for HelloWorld# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rstGlobals:  Function:    Timeout: 10    MemorySize: 128Resources:  HelloWorldFunction:    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction    Properties:      PackageType: Image      Events:        HelloWorld:          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api          Properties:            Path: /hello            Method: get    Metadata:      DockerTag: dotnet6-v1      DockerContext: ./src/HelloWorld      Dockerfile: Dockerfile      DockerBuildArgs:        SAM_BUILD_MODE: runOutputs:  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function  # Find out more about other implicit resources you can reference within SAM  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api  HelloWorldApi:    Description: "API Gateway endpoint URL for Prod stage for Hello World function"    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"  HelloWorldFunction:    Description: "Hello World Lambda Function ARN"    Value: !GetAtt HelloWorldFunction.Arn  HelloWorldFunctionIamRole:    Description: "Implicit IAM Role created for Hello World function"    Value: !GetAtt HelloWorldFunctionRole.ArnThis is how my Remote configuration looks like.I use macOS and I read that Runtime.InvalidEntrypoint is thrown when the image is build with a different architecture type than the expected which make sense. macOS default configuration is arm64 and I guess lambda is expecting x86/64.If that's the case, how to setup the remote configuration so that the output is x64 image?FollowComment"

Answer: "0Hi,I hope that you are doing well. From checking our documentation, it seems that this error is related to the ENTRYPOINT Container setting -> https://docs.aws.amazon.com/lambda/latest/dg/troubleshooting-images.html#troubleshooting-images-entryIssue: You receive a Runtime.ExitError error message, or an error message with "errorType": "Runtime.InvalidEntrypoint".Verify that the ENTRYPOINT to your container image includes the absolute path as the location. Also verify that the image does not contain a symlink as the ENTRYPOINT.Can you kindly check this so it can be ruled out?CommentShareSUPPORT ENGINEERRyan_Aanswered 3 months ago"


Question: Limit Increase: SES Service Limits

"Hi all,We just want to ask why our email sending are limited to 45 email only per batch.Our email size is only 780KB, but looks like Amazon SES limit our sending to 45 emails only per batch and our campaign has been dragged for 6 days to complete the sending.Can you imagine we blast out to 800+ subscribers and only 45 emails send out for every 8 hours? Is there any setting to adjust this limit?FollowComment"

Answer: "0As per the documentation, there are SES service quotas that you need to look at:If you're in using SES Sandbox environment, then you only get to send 200 emails per 24 hours period plus you can add maximum of 50 recipients per message. (Note: Recipients number is not adjustable). Please refer link {1 & 2}Note: A recipient is any "To", "CC", or "BCC" address.If you are using Sandbox environment then you need to upgrade to the production and can always initiate "request for production access" from SES console.How to move from Sandbox to Production, please refer link {3}There is another option of "Bulk email sending" where in programmatically you can send an email to several customers at the same time. (Note: Recipients number is not adjustable)To know more:{1} https://docs.aws.amazon.com/ses/latest/dg/quotas.html#quotas-email-sending{2} https://docs.aws.amazon.com/ses/latest/dg/manage-sending-quotas.html{3} https://docs.aws.amazon.com/ses/latest/dg/request-production-access.htmlLet me know If this helps.CommentShareSwwapnilanswered a year agoKent7 a year agoHello, I want to make a response to this claim in reference to an issue I am dealing with:"If you are using Sandbox environment then you need to upgrade to the production and can always initiate "request for production access" from SES console."This isn't necessarily true as I have made multiple request production cases in the past and now I am no longer able to request production access due to dealing with the difficulty of the AWS team."Amazon SES has denied your request for production access. You may no longer request access from the Amazon SES console. For details about why your request was denied or to file an appeal, see case ID 9790969121  in the AWS Support Center."Share0Hi,Actually we have upgrade our limit to 100k daily. And even if AWS limit our sending to 200 daily, why did our sending only 136 emails per day? I mean our campaign only send out to 45 emails after 8 hours for 3 stages/daily. And its repeated for 6 days until all 800 emails finish sent out.Example : 1am - send to 45 emails,9am - send to another 45 emails,5pm - send to another 45 emailsCommentShareAWS-User-1826316answered a year ago"


Question: JUMBO FRAMES ON RDS & ORACLE RDS

HiIs there something needed to be done to enable jumbo packets for RDS instances? Is this feature enabled by default?Thanks!FollowComment

Answer: "1You can increase them on some instances, but there are some limitations. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#jumbo_frame_instancesCommentShareRodney Lesteranswered 6 months ago"


Question: What does the error "VALUE_PRINT_NOT_SUPPORTED" mean and how do I fix it?

"I'm using DMS to move a MySQL 5.7 database from one AWS account to another.  I'm not changing versions or schemas or anything, just migrating it to a new RDS instance.  Everything was going pretty well until I ran into the "VALUE_PRINT_NOT_SUPPORTED" error.  This error only seems to happen on MySQL "TEXT" fields, and only on some of them.Looking at my table statistics I can see that there are 10 tables that have a Validation State of "Mismatched records".  When I query the "awsdms_validation_failures_v1" table in the TARGET database I can see these errors are listed as "VALUE_PRINT_NOT_SUPPORTED".{'contact_notes': '<VALUE_PRINT_NOT_SUPPORTED>'}, {'contact_notes': '<VALUE_PRINT_NOT_SUPPORTED>'}What is this error trying to tell me?  Are there characters in those records that can't be migrated?  Is the content on the fields too large?  Is there some other error?FollowCommentawsharmeet 10 months agoScenario where source DB is Oracle and destination DB is PostgreSQL. Below scenario could be one of the cases.This happens if column in Oracle table is of CLOB type and while inserting into this column, EMPTY_CLOB() function is used. It will be easy to understand if the value is space/null/blank depending on which IDE your are using. When using AWS DMS, these values are migrated as blanks/NULLs. In Oracle, you may see either nothing or if you use DUMP function in Oracle then you will see NULL as its value.Use function dbms_lob.compare(<CLOBColumnName>,empty_clob()) to determine the value for that column in a row. If empty_clob() is used, then you will get 0 as an output for this function.Share"

Answer: "1if we are moving db instance across VPC. i would suggest create a read replica from main instance using a precreated dbsubnet group of the required vpc subnets.once ready to cutover just promote the read replica.CommentShareAWS-User-8174525-subhashanswered a year agoRNHurt a year agoThat's a good idea and I didn't realize that RDS supported read replicas for MySQL.  Unfortunately, I am moving my DB to a VPN in a different account and according to the docs, read replicas are only supported in the same account.However, I will probably use MySQL RR for some upcoming data warehousing situations.  So, thanx for the tip!ShareAWS-User-8174525-subhash a year agoIn case of cross account you can use rds snapshot to restore to that account.lastly if downtime windows is smaller you can setup native mysql replication between them post snapshot restore.ShareAWS-User-8174525-subhash a year agoalso you can refer to this article to create read replica in another accounthttps://aws.amazon.com/premiumsupport/knowledge-center/rds-mysql-cross-region-replica/Share0I'm also getting this error on a field in one of my tables.  I've tried reloading and re-verifying the data but it still doesn't work.  I also can't find the row that is having problems because the "key" doesn't seem to match up with any of the indexes on that table.+--------------------------+----------+--------------------------+--------------+-----------------------------------------------------------------------------------------------------------+| KEY_TYPE | KEY                      | FAILURE_TYPE | DETAILS                                                                                |+--------------------------+----------+--------------------------+--------------+-----------------------------------------------------------------------------------------------------------+| Row      | {"key":  ["11111111"] }  | RECORD_DIFF  | [[{'object_changes': '<VALUE_PRINT_NOT_SUPPORTED>'}, {'object_changes': '<VALUE_PRINT_NOT_SUPPORTED>'}],] || Row      | {"key":  ["22222222"] }  | RECORD_DIFF  | [[{'object_changes': '<VALUE_PRINT_NOT_SUPPORTED>'}, {'object_changes': '<VALUE_PRINT_NOT_SUPPORTED>'}],] |CommentShareRNHurtanswered a year agoawsharmeet 10 months agoScenario where source DB is Oracle and destination DB is PostgreSQL. Below scenario could be one of the cases.This happens if column in Oracle table is of CLOB type and while inserting into this column, EMPTY_CLOB() function is used. It will be easy to understand if the value is space/null/blank depending on which IDE your are using. When using AWS DMS, these values are migrated as blanks/NULLs. In Oracle, you may see either nothing or if you use DUMP function in Oracle then you will see NULL as its value.Use function dbms_lob.compare(<CLOBColumnName>,empty_clob()) to determine the value for that column in a row. If empty_clob() is used, then you will get 0 as an output for this function.Share0Hello,Within an AWS Region, we strongly recommend that you create all read replicas in the same virtual private cloud (VPC) based on Amazon VPC as the source DB instance. If you create a read replica in a different VPC from the source DB instance, classless inter-domain routing (CIDR) ranges can overlap between the replica and the RDS system. CIDR overlap makes the replica unstable, which can negatively impact applications connecting to it. If you receive an error when creating the read replica, choose a different destination DB subnet group.That is correct, you cannot create a RR in a different AWS account from the source DB instance.CommentShareSUPPORT ENGINEERAWS-User-Chiraganswered a year ago"


Question: "Not able to do one time load, from postgres to opensearch using DMS"

"Trying to migrate existing data from AWS RDS Postgres to AWS managed OpenSearch, but it is not working, no rows were migrated to opensearch, When checking the Cloudwatch log getting below errorBulk request failed. no retry. TotalRecordCount 4080, FailedRecordCount 4080 [1026400] (elasticsearch_bulk_utils.c:181)DMS has the following configuration:{  "TargetMetadata": {    "TargetSchema": "",    "SupportLobs": false,    "FullLobMode": false,    "LobChunkSize": 0,    "LimitedSizeLobMode": false,    "LobMaxSize": 0,    "InlineLobMaxSize": 0,    "LoadMaxFileSize": 0,    "ParallelLoadThreads": 5,    "ParallelLoadBufferSize": 100,    "BatchApplyEnabled": false,    "TaskRecoveryTableEnabled": false,    "ParallelLoadQueuesPerThread": 0,    "ParallelApplyThreads": 0,    "ParallelApplyBufferSize": 100,    "ParallelApplyQueuesPerThread": 0  },  "FullLoadSettings": {    "TargetTablePrepMode": "DO_NOTHING",    "CreatePkAfterFullLoad": false,    "StopTaskCachedChangesApplied": false,    "StopTaskCachedChangesNotApplied": false,    "MaxFullLoadSubTasks": 8,    "TransactionConsistencyTimeout": 600,    "CommitRate": 50000  },  "Logging": {    "EnableLogging": true,    "LogComponents": [      {        "Id": "TRANSFORMATION",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "SOURCE_UNLOAD",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "IO",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "TARGET_LOAD",        "Severity": "LOGGER_SEVERITY_DETAILED_DEBUG"      },      {        "Id": "PERFORMANCE",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "SOURCE_CAPTURE",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "SORTER",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "REST_SERVER",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "VALIDATOR_EXT",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "TARGET_APPLY",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "TASK_MANAGER",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "TABLES_MANAGER",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "METADATA_MANAGER",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "FILE_FACTORY",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "COMMON",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "ADDONS",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "DATA_STRUCTURE",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "COMMUNICATION",        "Severity": "LOGGER_SEVERITY_DEFAULT"      },      {        "Id": "FILE_TRANSFER",        "Severity": "LOGGER_SEVERITY_DEFAULT"      }    ],    "CloudWatchLogGroup": null,    "CloudWatchLogStream": null  },  "ControlTablesSettings": {    "historyTimeslotInMinutes": 5,    "ControlSchema": "",    "HistoryTimeslotInMinutes": 5,    "HistoryTableEnabled": true,    "SuspendedTablesTableEnabled": false,    "StatusTableEnabled": true,    "FullLoadExceptionTableEnabled": false  },  "StreamBufferSettings": {    "StreamBufferCount": 3,    "StreamBufferSizeInMB": 8,    "CtrlStreamBufferSizeInMB": 5  },  "ChangeProcessingDdlHandlingPolicy": {    "HandleSourceTableDropped": true,    "HandleSourceTableTruncated": true,    "HandleSourceTableAltered": true  },  "ErrorBehavior": {    "DataErrorPolicy": "LOG_ERROR",    "EventErrorPolicy": null,    "DataTruncationErrorPolicy": "LOG_ERROR",    "DataErrorEscalationPolicy": "SUSPEND_TABLE",    "DataErrorEscalationCount": 0,    "TableErrorPolicy": "SUSPEND_TABLE",    "TableErrorEscalationPolicy": "STOP_TASK",    "TableErrorEscalationCount": 0,    "RecoverableErrorCount": -1,    "RecoverableErrorInterval": 5,    "RecoverableErrorThrottling": true,    "RecoverableErrorThrottlingMax": 1800,    "RecoverableErrorStopRetryAfterThrottlingMax": true,    "ApplyErrorDeletePolicy": "IGNORE_RECORD",    "ApplyErrorInsertPolicy": "LOG_ERROR",    "ApplyErrorUpdatePolicy": "LOG_ERROR",    "ApplyErrorEscalationPolicy": "LOG_ERROR",    "ApplyErrorEscalationCount": 0,    "ApplyErrorFailOnTruncationDdl": false,    "FullLoadIgnoreConflicts": true,    "FailOnTransactionConsistencyBreached": false,    "FailOnNoTablesCaptured": true  },  "ChangeProcessingTuning": {    "BatchApplyPreserveTransaction": true,    "BatchApplyTimeoutMin": 1,    "BatchApplyTimeoutMax": 30,    "BatchApplyMemoryLimit": 500,    "BatchSplitSize": 0,    "MinTransactionSize": 1000,    "CommitTimeout": 1,    "MemoryLimitTotal": 1024,    "MemoryKeepTime": 60,    "StatementCacheSize": 50  },  "PostProcessingRules": null,  "CharacterSetSettings": null,  "LoopbackPreventionSettings": null,  "BeforeImageSettings": null,  "FailTaskWhenCleanTaskResourceFailed": false,  "TTSettings": null}Opensearch have index with following settings{     "settings": {      "index.max_ngram_diff" :8,      "analysis": {        "analyzer": {                    "my_ngram_analyzer": {            "type": "custom",            "tokenizer": "standard",            "filter": [              "lowercase",              "mynGram"            ]          }        },        "filter": {          "mynGram": {            "type": "nGram",            "min_gram": 6,            "max_gram": 14,            "token_chars": [                "letter", "digit", "whitespace", "symbol"              ]          }        }      },         "number_of_shards": 6,        "number_of_replicas": 1    },    "mappings" : {      "properties" : {        "created_at" : {          "type" : "date"        },        "id" : {          "type" : "long"        },        "name" : {          "type" : "text",          "analyzer":"my_ngram_analyzer"          , "search_analyzer": "my_ngram_analyzer"        },        "phone" : {          "type" : "text",          "analyzer":"my_ngram_analyzer"          , "search_analyzer": "my_ngram_analyzer"        },        "updated_at" : {          "type" : "date"        }      }    }}I have tried to insert a sample document using _bulk API on opensearch console and it worked, below is the thing I had tried over opensearch, which workedPOST _bulk{"index":{"_index":"contacts"}}{"name": "name","phone" : "11111111","created_at" : "2021-12-21T12:12:59","updated_at" : "2021-12-21T12:12:59","id": 101}FollowCommentLeonardo Rifeli 7 months agoDo you fix that problema? I'm getting the same problem here.Share"

Answer: "0To answer your question, we require details that are non-public information. Please open a support case with AWS using the following link[-] https://console.aws.amazon.com/support/home#/case/createCommentShareSUPPORT ENGINEERarpmehraanswered a year ago"


Question: invoke_endpoint error in Lambda: StreamingBody is not JSON serializable

"I'm writing a Lambda function that invokes an endpoint:runtime= boto3.Session().client('runtime.sagemaker')payload = {"data": ["McDonalds"]}response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME,                                       ContentType='application/json',                                       Body=json.dumps(payload))It returns this errorAn error occurred during JSON serialization of response: <botocore.response.StreamingBody object at 0x7f59e40acc50> is not JSON serializableI tried this exact function in SageMaker notebook and it works but it doesn't work in Lambda. Can someone please help me?Edited by: aurelius on Feb 19, 2019 10:38 PMFollowComment"

Answer: "0Was this because I didn't attach the necessary SageMaker policies to the IAM role? I only added this policy:{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "VisualEditor0",            "Effect": "Allow",            "Action": "sagemaker:InvokeEndpoint",            "Resource": "*"        }    ]}CommentShareaureliusanswered 4 years ago0Hi aurelius,Do you have further code after the response = runtime.invoke_endpoint(..) line? the error message says problem with serialization of the response object.Your role permission looks fine.Thank you,ArunCommentShareAWS-User-1911581answered 4 years ago0I am facing similar issue, in sagemaker jupyter notebook instance the endpoint is invoked successfully and I am able to get back the inference results.Below is my Lambda function that i used to invoke the endpoint, but I am facing the following error,import json import ioimport boto3 client = boto3.client('runtime.sagemaker')def lambda_handler(event, context):    print("Received event: " + json.dumps(event, indent=2))        #data = json.loads(json.dumps(event))    #payload = data['data']    print(json.dumps(event))        response = client.invoke_endpoint(EndpointName='linear-learner-2019-12-12-16-20-56-788',                                  ContentType='application/json',                                  Body=(json.dumps(event)))    return responseOutput:Function Logs:START RequestId: 7f4c7589-b70f-4af8-834c-89a1a1fbe5e5 Version: $LATESTReceived event: {  "instances": [    {      "features": [        0.1,        0.2      ]    }  ]}{"instances": [{"features": [0.1, 0.2]}]}An error occurred during JSON serialization of response: <botocore.response.StreamingBody object at 0x7f53918e2828> is not JSON serializablePlease help me resolve the issue, you can see the JSON input passed to the function. Not sure what is going wrong here. I even checked the cloudwatch logs not able to identify the origin of the issue.Thanks in advance,ArunEdited by: NMAK on Dec 18, 2019 5:54 AMCommentShareNMAKanswered 3 years ago0I call Sagemaker from Lambda using a slightly different approach in terms of data structures:final_data = ','.join(ordered_data.iloc[0].astype(str).values.tolist())runtime = boto3.client('runtime.sagemaker')response = runtime.invoke_endpoint(EndpointName='whatever-endpoint',                                            ContentType='text/csv',                                           Body=final_data)result = json.loads(response['Body'].read().decode())I start with a dataframe of one row containing all the data.CommentShareJavierLopezanswered 3 years ago0hello Javierlopez,Thanks for your response, I really appreciate it.I tried the below code and still it gives me same error    payload = ','.join(str(item) for item in data['instances'][0]['data']['features'])    #payload=bytearray(payload)    print(payload)        response = client.invoke_endpoint(EndpointName='linear-learner-2019-12-12-16-20-56-788',                                  ContentType='text/csv',                                  Body=payload)    return responsebelow is output/error for the above code.Function Logs:START RequestId: 73daa03e-dec2-4d37-b779-72c1f70a7142 Version: $LATESTReceived event: {  "instances": [    {      "data": {        "features": [          0.1,          0.2        ]      }    }  ]}0.1,0.2 # this is the payload sent for inference.An error occurred during JSON serialization of response: <botocore.response.StreamingBody object at 0x7f77555e59e8> is not JSON serializableI really don't understand the logic behind this on what format the function expects the input, my colleague ran a different model with same JSON format I used and it works fine. I believe you used this is inference format for XGBoost. I am not able to find any documentation on linear-learner sample inference requests using lambda.Could you please point me in the right direction.Regards,ArunEdited by: NMAK on Dec 20, 2019 3:04 AMCommentShareNMAKanswered 3 years ago"


Question: Date_Parse INVALID_FUNCTION_ARGUMENT: Invalid format

"I am trying to convert string into timestamp however getting INVALID_FUNCTION_ARGUMENT: Invalid format: "2010-12-23 00:00:00" is malformed at " 00:00:00" error.The source data is in CSV format. Sample output below..110100001,48,F,5182210000,1,2011-01-03 09:34:00,2010-12-23 00:00:00,CRDINV,CLOSED,280,0,0,,,,,,,,,,,,,,CCCOMP,,<Error>,89,,E:\OPTIMDATA\ARCHIVE\ABC\ABAFThe query I am using is :SELECT date_parse(aws_athena_div99.DATE_CLM_RPTD, '%Y-%m-%dT%H:%i:%s') as DateConvertedToTimestamp FROM athenadb.aws_athena_div99;I am storing data as TEXTFILE in Athena.DATE_CLM_RPTD is column #7.Please help.FollowComment"

Answer: "0Hi,Please check on the below working solution:sample string is “2010-12-23 00:00:00”query:SELECT date_parse(‘2010-12-23 00:00:00’,'%Y-%m-%d %H:%i:%s');Result:   _col0  2010-12-23 00:00:00.000Note : Format of the date parse should match the sample date string.I hope above helps.have a good day :)CommentShareAWS-User-4025440answered 4 years ago0Unfortunately, it did not help. Got following error:Your query has the following error(s):INVALID_FUNCTION_ARGUMENT: Invalid format: ""I understand when you say format should match the input and it is matching however its not working.. Not sure why.I wonder if the format is getting changed while Athena is reading from S3 and hence when I query it does not match. One reason why explicitly specified TEXTFILE as store format but still the same result.CommentShareDiv99answered 4 years ago0Hi! Did you manage to solve this?I have the same issue. I have a the column 'date_x' that contains the date of purchase but it appears in Athena as VARCHAR.And I want to retrieve from the table the lines that have the purchase date between 1st of September and 3rd of November. I tried to use the following condition:WHERE date_parse(date_x, 'YYYY-MM-DD') BETWEEN date_parse('2020-09-01', 'yyyy-mm-dd') AND date_parse('2020-11-03', 'yyyy-mm-dd')and i get the same error as you:INVALID_FUNCTION_ARGUMENT: Invalid format: "2020-09-01"Edited by: GabrielaS on Nov 3, 2020 9:04 AMEdited by: GabrielaS on Nov 3, 2020 9:05 AMCommentShareGabrielaSanswered 3 years ago0HI man I have been bumping my head against the wall with this one. however, in my case, similar to your case I was using data_format(string_column_name, 'YYYY-MM-dd HH:mm:ss') but I was getting the same error and after looking everywhere I found that since athena uses presto under the hood they both have a similar catch, which is that  there is confusing naming between the MySQL data functions and the Java  data functions, MySQL functions are data_parse and date_format; the java functions are format_date and parse_datetime. details here https://github.com/prestodb/presto/issues/1019long story short, for date_parse you shouldn't be using the YYYY-MM-DD  format (Java simple date format), instead you should use the %Y-%m-%d format, so the expressions in your query should look like thisdate_parse ( date_x, '%Y-%m-%d') assuming date_x follows something similar to the othersdate_parse ( '2020-09-01', '%Y-%m-%d')date_parse ( '2020-11-03', '%Y-%m-%d')in my specific scenario I was trying to parse a column where the values where strings that look like "2020-01-20 00:00:00"  and I was getting the invalid format error; once I changed the format to the %Y etc form, I still go the error, but, it was because the function was evaluating the double quotes as part of the string so I had to account for those on my format and my expression ended up looking like this:date_parse(string_date_column,'"%Y-%m-%d %H:%i:%s"')hope this helpsEdited by: Elruncho35 on Nov 24, 2020 6:03 PMEdited by: Elruncho35 on Nov 24, 2020 6:07 PMCommentShareElruncho35answered 3 years ago"


Question: Amplify datastore deleting specific list of data

My app is a social media clone app. I want a logic like below for my home feed:amplify api will fetch list of 10 items through pagination -> this 10 items will be showing up to user in  feed -> when user reaches the 7th item on the screen I will fetch again 10 items through api call and save it to local datastore-> when user will reach the end item/10th item i will show the  next 10 items present in local datastore and I will clear that 10 items in local storage-> and fetch next 10 items-> if user closes app immediately the 10 items will be there in local storage-> next time when user will open app I will show the data without buffer form local storageMy issue is how if user likes a post or makes a comment I will save that to data store for short time and when sync event occurs I will update that like and comment to DynamoDB.1. My question is how to delete some specific data when some event occurs like list of post?2. Can we upload new post from datastore? how?FollowComment


Question: Cannot create a gateway.

"Hi,I am not able to create a gateway in IoT SiteWise. When I am creating a new gateway and assigning a Greengrass group it says: Greengrass group with ID xxxxxxxxxxxxxxx does not have an associated role.I have a service role and a SiteWise policy is associated with it and the trust policy contains Greengrass service.Thanks,SantoshFollowComment"

Answer: "0Hi Santosh,Looks like the only step you are missing to make this work is attaching the role to the green grass group.Refer to "Attach an IAM Role to an AWS IoT Greengrass Group" section within this link https://docs.aws.amazon.com/iot-sitewise/latest/userguide/configure-gateway.html#create-iam-resourcesHope that helps,SantoshCommentShareAWS-User-4947585answered 3 years ago0Hi,I have already attached the role to the Greengrass group. Still, I got the same error.PFB the same.Current service roleGreenGrass RolePolicies attached to this roleAWSGreengrassResourceAccessRolePolicyAWSLambdaBasicExecutionRole-d56f5b4d-b1bf-4be5-94e8-e20380ce18f9GreengrassUpdateSiteWise -- This is the policy for SiteWise.Thanks,SantoshEdited by: SantoshPanda on Jan 15, 2020 1:00 AMEdited by: SantoshPanda on Jan 15, 2020 1:03 AMCommentShareSantoshPandaanswered 3 years ago0Hi,However when I used CLI to check the associated role I got an error that there is no associated role to the deployment group. So, I associated it to the group via CLI and now I am successful.But, I would like to know why the CLI gave me an error when in the console I can see that the role is associated to the group.Thanks,SantoshCommentShareSantoshPandaanswered 3 years ago0Hi SantoshPanda,There are 2 concepts here:Greengrass Service RoleThis is the role you grant AWS Greengrass to assume to access resources in your AWS account. It allows Greengrass group deployments to succeed. Service role association is done at the account level per region.Greengrass Group RoleThis is the role you associate for each of your Greengrass group. It is used and assumed by lambdas within that group to access other AWS services. This association is done at the group level.The error "Greengrass is not authorized to assume the service role" indicates that your Greengrass Service Role might not be correctly configured. Can you try and confirm the following?With AWS CLI, run the following:aws greengrass get-service-role-for-account --region <YOUR_REGION>Check if you have associated a service role for your account in that region. If yes, you should be able to see something like "arn:aws:iam::<ACCOUNT_ID>:role/service-role/<ROLE_NAME>". Note down the role arn/name.2. With AWS CLI, run the following:aws iam get-role --role-name <ROLE_NAME> --region <YOUR_REGION>You should be able to see an output similar to the following:{    "Role": {        "Path": "/",        "RoleName": "ROLE_NAME",        "RoleId": "ROLE_ID",        "Arn": "arn:aws:iam::<ACCOUNT>:role/<ROLE_NAME>",        "CreateDate": "2019-12-27T18:38:54Z",        "AssumeRolePolicyDocument": {            "Version": "2012-10-17",            "Statement": [                {                    "Effect": "Allow",                    "Principal": {                        "Service": "greengrass.amazonaws.com"  <--- Check here                    },                    "Action": "sts:AssumeRole"                }            ]        },        "MaxSessionDuration": 3600    }}Verify if the statement allows Greengrass service principal to assume role.Thanks,KR-AWSCommentShareKR-AWSanswered 3 years ago0Thanks for the help. I checked everything was done as mentioned.CommentShareSantoshPandaanswered 3 years ago"


Question: complex scheduler solution ideas please

"What is a scalable solution for running lambda at specific times in the future?We are building a SaaS platform, in which our users can request a task to happen according a recurring schedule (simple parameter of {"frequency": "NNN minutes"}. They can also edit/delete this schedule, to affect that future events. There will be K's of users, requesting M's of tasks over time (from 1 min to 1 year). The tasks will not be created in the order they should execute. I need high integrity on completion.I've discounted just using sqs. I've looked at cloudwatch events, but have concerns about scaling. I've considered putting the tasks into a DB table then polling. Is there something else I should look at?FollowComment"

Answer: "2This is what you need .... https://aws.amazon.com/pt/blogs/compute/introducing-amazon-eventbridge-scheduler/CommentShareEXPERTDaniel ABIBanswered 5 months agorePost-User-8564696 5 months agoyep, looks like thats it. We're giving it a run now. Thanks for your helpShare0Accepted AnswerWhat are you scaling concerns with cloudwatch? Have you also considered EventBridge or Step Functions to create a workflow? Also why have you discontinued sqs?CommentShareCloud_Ganswered 5 months agorePost-User-8564696 5 months agodiscounted SQS on its own, because there's no time/schedule attribute, all tasks are just FIFO... previously thought that cloudwatch events (where each future job is a scheduled task) would become expensive (millions of jobs). Now thinking that the recurring schedule of jobs is the item i create (likely in eventbridge), and that handles the creation of individual jobs/lambda callsShare"


Question: Delete Images in Custom Labels Datasets

Hello!Is it possible to delete images that have been uploaded to a custom labels dataset?If I delete all label annotations from an image will it be removed from training or will it still be used in the training process?Thanks for the help!MattFollowComment

Answer: "1Is there a reason removing images is not supported?  Removing annotations and leaving unused images in datasets creates noise when managing large datasets.  Hoping this feature will be added soon.CommentSharegrajameyanswered 3 years ago0Hey MattIf you remove all label annotations from an image, it will be ignored during the training.Note that your training might fail in some cases if removing label annotations caused validation rules to fail.E.g. if you have only 2 labels and you removed all annotations corresponding to 1 label, your training will fail since at least 2 labels are required.Validation list can be found here (see section "Semantics") - https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/cd-manifest-files-validation-rules.htmlHope this helps!CommentShareawsrakeshanswered 3 years ago0Thank you, that helps a lot!If I wanted to remove images that I mistakenly uploaded from a dataset so my model is not trained on them, would you suggest just removing all annotations as they are then essentially ignored from training?Appreciate the help!MattCommentShareMattSolarVoxanswered 3 years ago0MattYes you can remove the annotations to avoid them being used for training.If you have access to the manifest file, you can manually delete the offending rows.CommentShareawsrakeshanswered 3 years ago0Hey grajamey,Thank you for your feedback. Deleting images from datasets is on our roadmap. I have also passed on your feedback to the product team to help in the prioritization of this feature.CommentShareAWS-User-9970485answered 2 years ago0Hi,I am unable to delete images from Amazon Rekognition dataset.Is there any option to do it myself or would you be able to delete it?Thanks,PrathyushaCommentSharePrathyushaanswered 2 years ago0Hi Prathyusha,Custom labels dataset is managed by a dataset manifest file. If you have access to the manifest file, you can manually delete the offending rows. After deletion you can create a new dataset by importing the updated manifest file.The manifest file can be accessed through a bucket which name starts with custom-labels-console-*. Within the bucket, you can locate the file with key datasets/<data set name>/manifests/output/output.manifest.We have an item on our roadmap to support deleting images from dataset and I have also passed on your feedback to the team.ThanksCommentShareaws-renjunanswered 2 years ago"


Question: Limit of 50 Custom Scopes per App Client in Cognito

"In my microservices authentication model, I'm viewing each microservice as a Cognito Resource Server. And each microservice has multiple custom scopes associated with it (eg. payment:view, payment:create). When an App Client retrieves an access token for a guest user for example, I want to return all the custom scopes for all the microservices (Resource Servers) the client may interact with.My first question is, there is a Cognito limit of 50 scopes per App Client. If I have many microservices with multiple scopes in each, how do I get around this limit? It seems I may be approaching this in the wrong way but not sure.And second question is, as scopes are associated with an App Client, if I want my "guest" users to have one set of scopes and my "registered" users to have another set of scopes, should I create two separate App Clients for the same client entity? For example if the client is an iOS app, I would create App Clients "ios-client-guest-user" and "ios-client-registered-user". Is this the correct approach in having multiple groups of users associated with different sets of scopes using the same client?Thanks!FollowComment"

Answer: "0Hi,Yes, there is a limit of 50 custom attributes per user pool and cannot be increased.However, you can store this information on a DynamoDB and add to the identity token on the login workflow through the Amazon Cognito’s Pre token generation Lambda trigger, which is specifically designed to add new claims, update claims, or suppress claims.This also could help you to solve other Cognito limitations, such as not being able to remove or change attributes to the user pool once it has been created, or or not being able to backup this information, although it requires you to add (and mantain) additional infrastructure services to your solution.I hope this helps you.CommentShareMikel Del Tioanswered 20 days agorePost-User-2336927 12 days agoThanks for the response! You mentioned the limit of 50 custom attributes per user pool, which I'm not concerned about. My question was more about the 50 custom scope limit per App Client. After reading up on the Pre token generation Lambda trigger, it seems like I can't update the access token (ID token only) and scopes in the access token. I could potentially add a scope claim to the ID token but typically only the access token is used to authorize against resource servers. Also the Pre Token Lambda won't work for the Client Credentials grant as it doesn't send back an ID token.ShareMikel Del Tio 9 days agoHi,First of all I want to apologize, since as you say, my answer is incorrect: you asked about scopes limits per app client and I answered you with custom attributes limits per user pool, which is not the same. I'm sorry.As you say, the access token cannot be updated using the Amazon Cognito’s Pre token generation Lambda trigger, this is just for the ID token, and although it can also be used to authenticate users to your resource servers or server applications, it is not compatible with Client Credentials grant.SharerePost-User-2336927 4 days agoNot a problem at all. Thanks for taking the time to read and reply to my question.Share"


Question: how to install node js 18 in amazon linux 2 AMI ?

"Can someone please advise me on how to upgrade the gcc version when trying to install nodejs 18? Despite attempting to update to version 11, it remains stuck at version 7. Additionally, I am curious if it's possible to install yum groupinstall "Development Tools" version 10 or 11 as an alternative solution. Thank you for any assistance you can provide.FollowComment"

Answer: "1Accepted AnswerNode.js 18 is not supported on Amazon Linux2 as described in the following documentPlease install Node.js 16 insteadhttps://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-up-node-on-ec2-instance.htmlCommentShareEXPERTRiku_Kobayashianswered 2 months agoEXPERTalatechreviewed 2 months agomtariqsajid 2 months agobut if want to install node js 18 can i just update the GCC version ?ShareRiku_Kobayashi EXPERT2 months agoI can't update to glibc 2.27...Amazon Linux 2023 can use up to glibc 2.34, so please consider that.https://docs.aws.amazon.com/ja_jp/linux/al2023/ug/core-glibc.htmlShare0Hi,yes, I have read the mentioned article, I also read https://github.com/nodesource/distributions, and I understand that there seems to be some dependency issue between the glibc version used in amazon linux 2 and nodejs 18.However, such a beast exists, and is provided by amazon themselves: the nodejs 18 aws lambda runtime -- which exactly includes nodejs 18 in an amazonlinux2 image. That obviously means that there must be a way to run both of them together. Maybe AWS could share their knowledge about how to build that ...ThanksCommentShareb_nutzeranswered 19 days ago"