Question: Ubuntu Managed Nodes creation failure in Fully-Private cluster

"Hi,For some reason I am not able to create Ubuntu managed nodes in fully private cluster. Though, managed Amazon-Linux nodes and all other self-managed nodes are joining the cluster successfully. I have followed all the guides and troubleshooting aws websites already still I am not successful. I have also run the troubleshooting script. Below is the resultHERE IS A SUMMARY OF THE ITEMS THAT REQUIRE YOUR ATTENTION: [WARNING]: Worker node's AMI ami-0ebb49de26355a371 differs from the public EKS Optimized AMIs. Ensure that the Kubelet daemon is at the same version as your cluster's version 1.23 or only one minor version behind. Please review this URL for further details: https://kubernetes.io/releases/version-skew-policy/ .[WARNING]: No secondary private IP addresses are assigned to worker node i-0e775ca75fe57bb70, ensure that the CNI plugin is running properly. Please review this URL for further details: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html[WARNING]: As SSM agent is not reachable on worker node, this document did not check the status of Containerd, Docker and Kubelet daemons. Ensure that required daemons (containerd, docker, kubelet) are running on the worker node using command "systemctl status <daemon-name>".============================================================================================================================================Here are the detailed steps of the document execution: [X] Checking EKS cluster test-cluster: EKS Cluster: test-cluster is in Active state.    1. Checking if the cluster Security Group is allowing traffic from the worker node:       Passed: The cluster Security Group sg-068d83e5a68aff9c7 is allowing traffic from the worker node.    2. Checking DHCP options of the cluster VPC:       Passed: AmazonProvidedDNS is enabled    3. Checking cluster IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-cluster-ServiceRole-9WPNFE2Q3N2T for the required permissions:       Passed: IAM role for cluster test-cluster has the required IAM policies attached.       Passed: The cluster IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-cluster-ServiceRole-9WPNFE2Q3N2T has the required trust relationship for the EKS service.    4. Checking control plane Elastic Network Interfaces(ENIs) in the cluster VPC:       Passed: The cluster Elastic Network Interfaces(ENIs) exist.    5. Cluster Endpoint Private access is disabled for your cluster, checking if the Public CIDR ranges include worker node i-0e775ca75fe57bb70 outbound IP:       Passed: The cluster allows public access from 0.0.0.0/0     6. Checking cluster VPC for required DNS attributes:       Passed: Cluster VPC vpc-0cbb4879c588fb52a has the required DNS attributes correctly set.--------------------------------------------------------------------------------------------------------------------------------------------[X] Checking worker node i-0e775ca75fe57bb70 state: The instance is Running.    1. Checking if the EC2 instance family is supported:       Passed: EC2 instance family m5.xlarge is supported.    2. Checking the worker node network configuration:       Passed: Worker node is created in a private subnet without a NAT Gateway so VPC endpoints need to be used.       Passed: Checking VPC Endpoints setup:        Passed: The VPC Endpoint com.amazonaws.eu-west-2.ec2 exists. Checking its configuration:           Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.ec2 is allowing the worker node to reach the endpoint.           Passed: The default VPC Endpoint Policy is being used.        Passed: The VPC Endpoint com.amazonaws.eu-west-2.ecr.api exists. Checking its configuration:           Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.ecr.api is allowing the worker node to reach the endpoint.           Passed: The default VPC Endpoint Policy is being used.        Passed: The VPC Endpoint com.amazonaws.eu-west-2.ecr.dkr exists. Checking its configuration:           Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.ecr.dkr is allowing the worker node to reach the endpoint.           Passed: The default VPC Endpoint Policy is being used.        Passed: The VPC Endpoint com.amazonaws.eu-west-2.sts exists. Checking its configuration:           Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.sts is allowing the worker node to reach the endpoint.           Passed: The default VPC Endpoint Policy is being used.        Passed: S3 gateway endpoint ['vpce-0e01febb999cf1735', 'vpce-08ef7ffac589c3d01'] is added to the worker's VPC.        Passed: Worker node's route table rtb-0d26fefe5a613e64f has the required route for the S3 endpoint vpce-0e01febb999cf1735.        Passed: Worker node's route table rtb-0d26fefe5a613e64f has the required route for the S3 endpoint vpce-08ef7ffac589c3d01.    3. Checking the IAM instance Profile of the worker node:       Passed: The instance profile arn:aws:iam::782534010321:instance-profile/eks-d0c1cede-94e5-c6bc-75fd-2f86043a90eb is used with the worker node i-0e775ca75fe57bb70 .        Passed: IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-nodegroup-ser-NodeInstanceRole-12LZWOL4EJAJX is attached to Instance Profile.        Passed: IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-nodegroup-ser-NodeInstanceRole-12LZWOL4EJAJX has the required IAM policies attached.        Passed: No issues detected with the trust relationship policy of the arn:aws:iam::782534010321:role/eksctl-test-cluster-nodegroup-ser-NodeInstanceRole-12LZWOL4EJAJX role.    4. Checking worker node's UserData bootstrap script:       Passed: The UserData of the worker node contains the required bootstrap script.    5. Checking the worker node i-0e775ca75fe57bb70 tags:       Passed: Worker node i-0e775ca75fe57bb70 has the required cluster tags.    6. Checking the AMI version for EC2 instance i-0e775ca75fe57bb70:       [WARNING]: Worker node's AMI ami-0ebb49de26355a371 differs from the public EKS Optimized AMIs. Ensure that the Kubelet daemon is at the same version as your cluster's version 1.23 or only one minor version behind. Please review this URL for further details: https://kubernetes.io/releases/version-skew-policy/ .    7. Checking worker node i-0e775ca75fe57bb70 Elastic Network Interfaces(ENIs) and Private IP addresses to check if CNI is running:       [WARNING]: No secondary private IP addresses are assigned to worker node i-0e775ca75fe57bb70, ensure that the CNI plugin is running properly. Please review this URL for further details: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html    8. Checking the outbound SG rules for worker node i-0e775ca75fe57bb70       Passed: The Outbound security group rules for worker node i-0e775ca75fe57bb70 are sufficient to allow traffic to the EKS cluster endpoint    9. Checking if the worker node is running in AWS Outposts subnet       Passed: Worker node's subnet subnet-092b2d53a74ac655f is not running in AWS Outposts    10. Checking basic NACL rules       Passed: NACL acl-0507434ac9745b3dc has sufficient rules to allow cluster traffic.    11. Checking STS regional endpoint availability:       Passed: STS endpoint is activated within region eu-west-2.    12. Checking if Instance Metadata http endpoint is enabled on the worker node:       Passed: Instance metadata endpoint is enabled on the worker node.    13. Checking if SSM agent is running and reachable on worker node:       [WARNING]: As SSM agent is not reachable on worker node, this document did not check the status of Containerd, Docker and Kubelet daemons. Ensure that required daemons (containerd, docker, kubelet) are running on the worker node using command "systemctl status <daemon-name>".============================================================================================================================================Here is a list of other possible causes that were NOT checked by this document: [-] Ensure that Instance IAM role is added to aws-auth configmap, please check: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html. [-] If your account is a part of AWS Organizations Service, confirm that no Service Control Policy (SCP) is denying required permissions, please check: https://aws.amazon.com/premiumsupport/knowledge-center/eks-node-status-ready/. One strange thing is that in point 5 it says private endpoint access is disabled but I have already created the cluster fully private.{    "update": {        "id": "4c429db1-80bb-48d2-bf98-3f84524c0b83",        "status": "Successful",        "type": "EndpointAccessUpdate",        "params": [            {                "type": "EndpointPublicAccess",                "value": "false"            },            {                "type": "EndpointPrivateAccess",                "value": "true"            },            {                "type": "PublicAccessCidrs",                "value": "[\"0.0.0.0/0\"]"            }        ],        "createdAt": "2022-10-03T09:35:42.092000+00:00",        "errors": []    }}Also, when I run the command sudo systemctl status kubelet I get the result.Unit kubelet.service could not be found.My cluster config is as belowapiVersion: eksctl.io/v1alpha5kind: ClusterConfigmetadata:  name: test-cluster  region: eu-west-2  version: "1.23"privateCluster:  enabled: truevpc:  id: vpc-ID  subnets:    private:      hscn-private1-subnet:        id: subnet-1      hscn-private2-subnet:        id: subnet-2managedNodeGroups:  - name: serv-test-1    ami: ami-0ebb49de26355a371    instanceType: m5.xlarge    desiredCapacity: 1    volumeType: gp2    volumeSize: 50    privateNetworking: true    disableIMDSv1: true    subnets:      - hscn-private2-subnet    ssh:      allow: true    tags:      kubernetes.io/cluster/test-cluster: owned    overrideBootstrapCommand: |      #!/bin/bash      /etc/eks/bootstrap.sh test-cluster --kubelet-extra-args '--node-labels=eks.amazonaws.com/nodegroup=serv-test-1,eks.amazonaws.com/nodegroup-image=ami-0ebb49de26355a371' --dns-cluster-ip 10.100.0.10 --apiserver-endpoint {My endpoint} --b64-cluster-ca {My-CA}FollowComment"


Question: OpenSearch upgrade Stuck at "Check eligibility" for multiple hours

"Hello,I am currently upgrading my OpenSearch instance and it has been stuck at "Check eligibility" with status " In progress (28.57%)" for multiple hours.Is it expected to take so long? Blue-green deployments on that cluster usually complete within an hour.I have looked at the OpenSearch FAQ and there is a question that mentions the following:For detailed list of the tests we run to validate upgrade eligibility, please refer to our documentation.However, the documentation mentions no useful information about what happens during that eligibility check.Not sure what to do here. Wait for a full day? Does it eventually timeout?FollowComment"

Answer: "0Hi ,hope this situation has been resolved by now.for the next time , or otherwise if you still have the issue. This is a case in which AWS Support should be able to help you, because no general answer may apply and something contextual to your account might be causing the issue.Your best next steps would be to open a Support case, in case you have not yet a Support subscription, consider to upgrade to Developer Support or Business support  depending on the urgency of your case.Without looking at the internal logs or having more detailed information on the situation it is not possible to provide guidance, sorry.hope this helpsCommentShareEXPERTFabrizio@AWSanswered a year ago"


Question: Performance issue on GIN index with gin_trgm_ops index column

"Hi all,We are investigating a performance issue with searching on a GIN with gin_trgm_ops indexed column. This specific (organization_id,aggregate_type) has in total 19K records and the (organization_id) has in total 30K records. The search record table has in total 38M records.The table and index are defined as follows:       Column      |       Type        | Collation | Nullable |                               Default  -----------------+-------------------+-----------+----------+---------------------------------------------------------------------   id              | bigint            |           | not null | nextval('search_records_id_seq'::regclass)   organization_id | uuid              |           | not null |   aggregate_id    | uuid              |           | not null |   aggregate_type  | character varying |           | not null |   document        | text              |           |          |  Indexes:      "search_records_pkey" PRIMARY KEY, btree (id)      "search_records_keys" UNIQUE, btree (organization_id, aggregate_id, aggregate_type)      "search_records_btree_gin" gin ((organization_id::character varying), aggregate_type, document gin_trgm_ops)The query we execute is:select aggregate_id from search_records where organization_id::varchar = '975097c5-e760-4603-9236-fcf2e8580a7c' and aggregate_type = 'FooRecord' and document ilike '%user.name@gmail.com%';Resulting in the following plan:   Bitmap Heap Scan on search_records  (cost=2184.00..2188.02 rows=1 width=104) (actual time=4332.007..4332.008 rows=1 loops=1)     Recheck Cond: ((((organization_id)::character varying)::text = '975097c5-e760-4603-9236-fcf2e8580a7c'::text) AND ((aggregate_type)::text = 'FooRecord'::text) AND (document ~~* '%user.name@gmail.com%'::text))     Heap Blocks: exact=1     Buffers: shared hit=23920 read=9752     I/O Timings: read=4017.360     ->  Bitmap Index Scan on search_records_btree_gin  (cost=0.00..2184.00 rows=1 width=0) (actual time=4331.987..4331.987 rows=1 loops=1)           Index Cond: ((((organization_id)::character varying)::text = '975097c5-e760-4603-9236-fcf2e8580a7c'::text) AND ((aggregate_type)::text = 'FooRecord'::text) AND (document ~~* '%user.name@gmail.com%'::text))           Buffers: shared hit=23920 read=9751           I/O Timings: read=4017.355   Planning Time: 0.268 ms   Execution Time: 4332.030 ms  (11 rows)We are running on Postgres RDS with engine version 12.8 with 32GB memory and 8GB shared_buffer. We have 442GB of 2000GB diskspace left.Sometimes we also have queries for this particular customer which take more than 20 seconds. The content of the document in those cases are similar to:   User Name Kees postgresstreet Amsterdam 1000 AA user.name@gmail.com 1234Are we doing something wrong? I find the I/O timings quite high, does this mean that it took 4000MS to read the 9752 blocks from the disk?Any other tips and or suggestions are welcome.LarsFollowComment"

Answer: "1Hello Lars,Thank you for reaching us and I understand that you are running into a query performance slowness issue after using a PG GIN index for a full text search:->  Bitmap Index Scan on search_records_btree_gin  (cost=0.00..2184.00 rows=1 width=0) (actual time=4331.987..4331.987 rows=1 loops=1)While query tuning is beyond the scope of this forum there are certainly a few things which I can recommend for additional deep dive on the issue:a) I see you have used trigrams for the index but did you check on the bitmap scan performance without the trigrams to understand if that's adding any benefitb) I will also go ahead and check if the index blocks are being cached by the DB:[+] https://www.postgresguide.com/performance/cache/[+] https://dba.stackexchange.com/questions/25513/postgresql-index-cachingc) Parameter track_io_timing might help show what component of that time is coming from IO and if it's indeed the IO fetch time from the disk. This parameter is off by default, as it will repeatedly query the operating system for the current time, which may cause significant overhead on some platforms:[+] https://www.postgresql.org/docs/current/runtime-config-statistics.htmld) Is the table 'search_records' frequently written upon? If yes then I will also recommend you to check on the pending list size as it could lead to the slower performance:[+] https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-GIN-PENDING-LIST-LIMITI sincerely hope that you will find these helpful.CommentShareSUPPORT ENGINEERMano_Sanswered 10 months agoAWS-User-5988924 9 months agoSorry I was offline for a while. Thanks for all the pointers, I will look into this.Share"


Question: How to allow access to EFS from SFTP using lambda as Identity Provider?

"'m trying to build an SFTP server for an EFS that uses a lambda function to check username and password provided against a Secret in AWS.I followed this article but changed it a bit, I'm not using an API Gateway, I use the lambda function directly as identity provider which fetches the following data from secret Manager : "Role" : "arn:aws:iam::xxxxxxxxxxx:role/my-transfer-role","PosixProfile": {    "Uid": 1001,    "Gid": 1001,    "SecondaryGids": [] }, "HomeDirectory": "/"so far I can only connect to the SFTP server, but can't read or write what's on the EFS Message="Unable to list directory: permission denied for /"I created a role and a policy attached to Transfer with permissions on my EFS as explained in this guideIs there something I'm missing in this configuration please? ThanksFollowComment"

Answer: "0Accepted AnswerHichamZouarhi@,Considering you are able to connect to the server and authenticate, the overall Lambda IDP configuration seems to be working. In regards to your error, I believe you need to provide the EFS id within the HomeDirectory to be able to access. Right now with HomeDirectory: / , it assumes root of EFS and probably attempts to list available EFS resources. Unless you have permissions for the same, the error would be expected behavior.Could you update the HomeDirectory to the EFS resource that you are trying to access? Example: HomeDirectory: /fs-000000/user. Further, you need to ensure that the folder you are trying to access is accessible using the POSIX configuration you specified: (1001, 1001) in your case.References:EFS GuideTransfer GuideLet me know if you have follow up questions.-- SagarCommentShareEXPERTsagarb-awsanswered 6 months agoHichamZouarhi 6 months agoThanks Sagar, I thought at first that my SFTP would be linked directly to the efs for which I gave permission on the policy. by adding the file system id to HomeDirectory it worked. Have a nice day :)Share"


Question: "Can't remove EC2 instance from Placement Group, Maybe bug?"

"I followed instructions from herehttps://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#change-instance-placement-groupAnd used commandaws ec2 modify-instance-placement --instance-id i-0123a456700123456 --group-name ""But I'm getting errorerror: argument --group-name: expected one argumentAlso would be good idea to set "if else" or whatever to prevent users to add "t" instances to placement group.I can't start instance, and command not works, the only way is to export it to AMI and run like new instance? :)FollowComment"

Answer: "0It is possible to do with PowerShell cmdletEdit-EC2InstancePlacement -InstanceId i-0123a456700123456 -GroupName "" -Region eu-central-1and empty string here worksEdited by: CoyoteKG on Feb 23, 2019 2:16 PMCommentShareCoyoteKGanswered 4 years ago"


Question: AWS Glue - Map against a template

"Hello, I am trying to use Glue to take an input file, do my required transformations, then output the columns in a specific order. I also want to output columns that may not be present in the input and have those columns just be null as placeholders.For example, I ingest columns CustomerID, Customer Name, DoB in this order. I want to output a file that goes: DoB, Customer ID, Customer Phone Number, Customer Address, Customer Name.Essentially I have a template that I have to match my output's format to but my input may not have every column, which is fine, but I still need at least placeholder columns there in order to match my template and for all my columns to be in a specific order.Is there a better way to do this then a custom SQL statement node? I would ideally like to be able to view the input schema against the template output schema and just point the relevant inputs to mapFollowComment"


Question: I cannot seem to create schema-level permissions in Babelfish

Why can't I grant permissions on a schema in Babelfish 2.1.1 or earlier?FollowComment

Answer: "0Accepted AnswerPlease check updates for the latest version of Babelfish. As of July 2022, Babelfish version 2.1.1 does not have a way to grant permissions to the schema level.For now, the best practice is to create a database role like db_reader and then add your users to that role. Then, you can do the grant select statements for each table in your schema.CommentSharelightanswered a year ago"


Question: "API gateway, S3 integration and compression"

"Hello,has anyone been able to figure out gzip-compression with API Gateway and S3 integration?If I have a uncompressed binary file in S3 and enable compression in API Gateway then requesting with "accept-encoding: gzip" works and returns proper gzip data (as long as payload is under 10MB).What doesn't seem to work is that when file in S3 is already gzipped and has content-encoding metadata defined. API gateway re-compresses the data. So I will then receive twice gzipped data, which then breaks on all http clients.This document here says that API gateway should not re-compress the data when integration returns content-encoding header. If the content-encoding metadata for the file has been defined, S3 will actually return that header. So is this a bug in API gateway? It seems to ignore the header and re-compresses already compressed data.Anyone has figured out how to get this to work?FollowCommentrePost-User-8157978 2 months agoAnd if I disable the compression in API gateway, then API gateway won't do any gzipping but it will also omit the "content-encoding" header, although S3 returns it (as it has been added as metadata for the file)Share"


Question: Amazon Linux 2 and Xilinx Runtime Issue

"Installation of the Xilinx Runtime on a clear Amazon Linux 2 instance (which is RedHat based) leads to OpenCL runtime errors.Xilinx Runtime (XRT) 2018.2_XDF.RC5 Installation instructions:https://github.com/aws/aws-fpga/blob/master/hdk/docs/SDxPatch_AR71715_and_XRT_installation_instructions.md#installing-xilinx-runtime-xrt-20182_xdfrc5Prerequisites:Extra Packages for Enterprise Linux (EPEL)sudo amazon-linux-extras install -y epelcurl -s https://s3.amazonaws.com/aws-fpga-developer-ami/1.5.0/Patches/XRT_2018_2_XDF_RC5/xrt_201802.2.1.0_7.5.1804-xrt.rpm -o xrt_201802.2.1.0_7.5.1804-xrt.rpmcurl -s https://s3.amazonaws.com/aws-fpga-developer-ami/1.5.0/Patches/XRT_2018_2_XDF_RC5/xrt_201802.2.1.0_7.5.1804-aws.rpm -o xrt_201802.2.1.0_7.5.1804-aws.rpmsudo yum install -y xrt_201802.2.1.0_7.5.1804-xrt.rpmsudo yum install -y xrt_201802.2.1.0_7.5.1804-aws.rpmExample application:#include <stdio.h>#include <CL/opencl.h>int main(){        cl_uint num_platforms;        cl_int errcode_ret = clGetPlatformIDs(0, NULL, &num_platforms);        if (errcode_ret != CL_SUCCESS){                fprintf(stderr, "Error: clGetPlatformIDs (%d)\n", errcode_ret);                return EXIT_FAILURE;        }        printf("Info: %d opencl platform(s) found!", num_platforms);        return EXIT_SUCCESS;}Makefile:SRCS = test.cppOBJCTS = $(SRCS:.cpp=.o)EXE=testCC = g++ -WallCFLAGS = -I${XILINX_XRT}/includeLFLAGS = -L${XILINX_XRT}/lib -lxilinxopenclall: ${OBJCTS}        ${CC} ${OBJCTS} ${LFLAGS} -o ${EXE}        ${RM} ${OBJCTS}%.o: %.cpp        ${CC} ${CFLAGS} -c ${<} -o ${@}clean:        ${RM} ${OBJCTS} ${EXE}3 sequential runs:$ ./testSegmentation fault$ ./testSegmentation fault$ ./testterminate called after throwing an instance of 'std::bad_alloc'what():  std::bad_allocAbortedThe problem must be related to boost library dependency!(Our JVM application stated libboost.so as the problematic frame.)Note that on a CentOS machine the same XRT installation works smoothly.FollowComment"

Answer: "0Hello,The XRT RPM's are built using Centos 7 and would yield indeterminate results when used on Amazon Linux 2.We currently do not support using the provided RPM's on Amazon Linux 2 but are working towards that goal with Xilinx and should be able to provide you a path to using it in the future.We will update here once we have a path forward.Thanks,DeepCommentShareDeep_Panswered 4 years ago0Fixed with Xilinx Runtime (XRT) 2019.1Download and install the InAccel-vended aws-xrt packages for Amazon Linux 2:sudo yum install -y \     https://inaccel-amazon.s3.amazonaws.com/eks/xrt_201910.2.2.0_2-xrt.rpm \     https://inaccel-amazon.s3.amazonaws.com/eks/xrt_201910.2.2.0_2-aws.rpmCommentShareInAccelanswered 4 years ago"


Question: Firehose to S3 with One Record Per Line

"Hey all,On this post there is a solution to have a target rule on a Firehose to add a newline char to every JSON event. However, the solution is for the JS CDK version and doesn't work for the Python version (1.134.0). We tried to find a way to have this solution on Python but seems that the CDK doesn't map all the needed properties from JS to Python.For now, we have a very ugly workaround that manipulates the JSON template before sending it to CloudFormation.To create the target firehose we use the code below, where the problem is the RuleTargetInput that have just a few options and doesn't enable a custom InputTransformerProperty.        firehose_target = KinesisFirehoseStream(            stream=self.delivery_stream,            # Python CDK is not allowing Custom CfnRule.InputTransformerProperty            # Makefile will make the workaround            message=RuleTargetInput.from_text(f'{EventField.from_path("$")}'),        )Piece of the JSON template generated by the CDK:        "Targets": [          {            "Arn": {              "Fn::GetAtt": [                "firehose",                "Arn"              ]            },            "Id": "Target0",            "InputTransformer": {              "InputPathsMap": {"f1":"$"},              "InputTemplate": "\\"<f1>\\""            },            "RoleArn": {              "Fn::GetAtt": [                "firehoseEventsRole1814C701",                "Arn"              ]            }          }        ]To manipulate the InputTransformer, we run the code below before sending it to CloudFormation:jq -c . cdk.out/robotic-accounting-firehose.template.json \| sed -e 's/"InputTransformer":{"InputPathsMap":{"f1":"$$"},"InputTemplate":"\\"<f1>\\""}/"InputTransformer":{"InputPathsMap":{},"InputTemplate":"<aws.events.event>\\n"}/g' \| jq '.' > cdk.out/robotic-accounting-firehose.template.json.tmprm cdk.out/robotic-accounting-firehose.template.jsonmv cdk.out/robotic-accounting-firehose.template.json.tmp cdk.out/robotic-accounting-firehose.template.jsonThat gives us the InputTransformer that we need and works:        "Targets": [          {            "Arn": {              "Fn::GetAtt": [                "firehose",                "Arn"              ]            },            "Id": "Target0",            "InputTransformer": {              "InputPathsMap": {},              "InputTemplate": "<aws.events.event>\n"            },            "RoleArn": {              "Fn::GetAtt": [                "firehoseEventsRole1814C701",                "Arn"              ]            }          }        ]We know, it's horrible, but it works.Does someone else have this problem and a better solution?Does the CDK v2 solve this?Tks,DanielFollowComment"

Answer: "1Hi there, you can implement the lambda transformation function to add new line to the records. Here the reference  article - Append Newline to Amazon Kinesis Firehose JSON Formatted Records with Python and AWS Lambda - https://medium.com/analytics-vidhya/append-newline-to-amazon-kinesis-firehose-json-formatted-records-with-python-f58498d0177aAnother option is to use dynamic partitioning, but if you would like to have only a new line and does not require the partitioning in s3, this wouldn't be an option and Dynamic partitioning is expensive than lambda transformation. Considering the expense of dynamic partitioning and limitation of Python version of CDK, recommended approach would be to use Lambda transformation.CommentShareSUPPORT ENGINEERHarshith_Manswered a year ago0Hey Harshith,I know that our current solution isn't the best one but, at least, we avoid having lambdas and paying for them. It's really unfortunate that the CDK Python version has such a limitation but your workaround is a cheaper option.Tks, DanielCommentShareDaniel Ferrarianswered a year ago"


Question: Create a presigned URL to upload objects to an Amazon S3 bucket that does not expire

how to create a file url saved in a bucket in s3 without expiration token and have access to the file whenever you wantFollowComment

Answer: "0Presigned URLs with very long expiration were possible with Sigv2 signing but not anymore.  From a blog article about Sigv2 deprecation: "While using long-lived pre-signed URLs was easy and convenient for developers, using SigV4 with URLs that have a finite expiration is a much better security practice."So now the longest you can go is 7 days.  Note that a presigned URL will expire when the creating IAM principal's credentials expire, so the only way to get expiry over 12 hours is to create it with IAM User credentials.CommentShareEXPERTskinsmananswered 3 months ago0Why not recreate the functionality in a bucket policy:{  "Id": "Policy1677592146473",  "Version": "2012-10-17",  "Statement": [    {      "Sid": "Stmt1677592144861",      "Action": [        "s3:PutObject"      ],      "Effect": "Allow",      "Resource": "arn:aws:s3:::mybucket/publicput.doc",      "Principal": "*"    }  ]}CommentShareEXPERTkentradanswered 3 months ago"


Question: WE1T105410002 Instance 1/2 Checks failed

"Hi Team,We are unable to start Instance WE1T105410002 and getting 1/2 Status check failed. We are also unable to check system log as they are empty. Could you please check from your end.RegardsSaud SalmanFollowComment"

Answer: "0Hello,I understand that you are unable to start Instance WE1T105410002 and getting 1/2 Status check failed. You checked the system logs and they were empty.The instance status check failure indicates an issue with the reachability of the instance. This issue occurs due to operating system-level errors such as the following:Failure to boot the operating systemFailure to mount the volumes correctlyExhausted CPU and memoryKernel panicNetwork failed to come upFor more information regarding instance status checks you can review the attached document.[+] For Linux Instances: https://repost.aws/knowledge-center/ec2-linux-status-check-failure[+] For Windows Instances: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/troubleshooting-windows-instances.htmlPlease follow the below steps to check the logs.Launch a new Amazon EC2 instance in your virtual private cloud (VPC) using the same Amazon Machine Image (AMI) and in the same Availability Zone as the impaired instance.Detach the Amazon Elastic Block Store (Amazon EBS) root volume (/dev/xvda or /dev/sda1) from your impaired instance. Make a note of the device name to be sure it is the same when you re-attach it laterAttach the EBS volume as a secondary device (/dev/sdf) to the rescue instance.Connect to your rescue instance using SSH or RDP.Mount the secondary volume. You should be able to see the data and log file.Additionally, you can review the instance screenshot to see whether there are any boot errors causing your instance to fail instance status checks, this can be reviewed using the steps mentioned here: https://aws.amazon.com/blogs/aws/ec2-instance-console-screenshot/Having said this, in case you are still not able to resolve the issue, we would need to check the instance specific details in order to further troubleshoot the issue. Therefore, I would recommend you to raise a support case with us. You can use below link for the same : https://support.console.aws.amazon.com/support/home#/case/createThank you!Resources:-[1] EC2 status check failure: https://repost.aws/knowledge-center/ec2-linux-status-check-failure[2] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html[3] https://repost.aws/knowledge-center/ec2-instance-boot-issuesCommentShareSUPPORT ENGINEERrePost-User-940793answered 2 months ago"


Question: VPC peering between Atlas cluster and AWS

"I have set up VPC Peering between my AWS VPC and Atlas cluster. My problem is that when I go to change my “IP access list” to remove the “0.0.0.0/0” IP, I am no longer able to connect to the cluster. I have included the VPC’s CIDR block in the allowed IPs and have also tried adding the Security group ID that is associated with the VPC.On the AWS side, I have a lambda that is being triggered and which then communicates with Mongo. The VPC that the lambda sits within, has no “internet gateway” so as I understand it, it should not be able to connect to the open internet. The VPC’s Route table diverts all traffic (from the lambda) to the peering connection (which connects Atlas to my VPC).From my setup, it seems that no traffic should be leaving my AWS VPC from any other IP address than the ones included in my VPC CIDR block. So why is it that removing the “0.0.0.0/0” IP from the list of allowed IP’s on mongo stops me from being able to connect?FollowComment"

Answer: "1Assuming routing and everything else is setup properly I would suggest enabling VPC Flow logs with additional Meta-Data to find out what is being rejected in both the Source and Destination VPCs.https://aws.amazon.com/blogs/aws/learn-from-your-vpc-flow-logs-with-additional-meta-data/Knowledge center article for troubleshooting connectivity over VPC Peering.https://aws.amazon.com/premiumsupport/knowledge-center/vpc-peering-connectivity/CommentShareEXPERTTushar_Janswered 8 months agoedouard_f 8 months agoI have added flow logs and it seems that none of the traffic goes through the VPC (I'm guessing this is the case since the logs seem very empty, don't show any source or destination IP or anything). Is it possible for the lambda that is connected to the VPC to not send traffic through it?ShareTushar_J EXPERT8 months agoBest way for you to isolate this is to create vanilla EC2 instance in the Source VPC and in the destination VPC and test a simple connectivity such as SSH over the Peering if that works fine then it is something with the Lambda or MongoDB that needs fixed.Share"


Question: obsolote

"deletedEdited by: PeterS99084 on Aug 3, 2021 12:53 AMFollowComment"

Answer: "0duplicate because of AWS POST error, can be deletedCommentSharePeterS99084answered 2 years ago"


Question: about session attributes and authentication of user

"Actually i am not clear yet. In my scenario I have created an authentication  intent . for other intent like transaction related intent if the user directly hit those transaction related intent it should ask first authentication like username , mobile and email. also i want that if the authentication happened for one time then it should not ask for other intent .FollowComment"

Answer: "0You can use a session attribute to see if the customer is authenticated already or not.Use conditional branching to branch the conversation based on if the customer is authenticated. (https://docs.aws.amazon.com/lexv2/latest/dg/paths-branching.html)If the customer is not authenticated. Save the name of the current intent in an attribute (so you know where to go back to later).You can switch to the authentication intent (https://docs.aws.amazon.com/lexv2/latest/dg/paths-nextstep.html)When the user is authenticated. Update the attribute you use for authentication (Now that it is set to true, other intents won't have to transfer to the authentication intent).6.. When the user has been authenticated, look at the attribute where you saved the intent name you came from and switch back to it.CommentShareThomas_Ranswered 4 months ago"


Question: Do we have to Spice duplicate data?

"When joining 2 pre-existing datasets where at least 1 is spiced, the resulting 3rd dataset automatically defaults to be spiced also, with no apparent way of changing this.  is this the expected behaviour?The datasets are all from the same data source. This results in the data in the parent dataset that is spiced being also spiced in the child dataset, resulting in a double charge for this data.If this is correct, are there any plans to change this, so that the customer only pays for the same data once?FollowComment"

Answer: "0Accepted AnswerHi, @PhillB.This is a designed behavior.Your pattern does not meet the following, so it is considered a cross-source join and will be loaded into SPICE.https://docs.aws.amazon.com/quicksight/latest/user/joining-data.htmlIf any of the logical tables refer to a QuickSight dataset that is a parent dataset:The parent dataset must use direct query.CommentShareEXPERTiwasaanswered a year ago"


Question: DynamoDB Tables Quota Increase Limits

"For any AWS account, there is an initial DynamoDB table quota of 2,500 tables per AWS Region. One can request a service quota increase using  the support page.Can one assume that requests will always be approved? Can a service design assume an unlimited number of tables? Otherwise, is there a limit to the quota increase for DynamoDB tables?FollowComment"

Answer: "1One should not assume it will always be approved. Justification must be provided. The table count limit is there as a nudge toward best practices, which usually doesn't involve multiple thousands of tables unless there's been careful consideration of alternative designs first.CommentSharejzhunteranswered a year ago"


Question: Cpu to shader parameter passing

"Hi Guys,I am trying to pass in a vec3 to the shaders from c++I can see SRenderObjData has a couple of arrays which might do the job.m_fTempVars and m_Constants look possible, even m_pShaderParams looks possible, but when I look into the actual shaders, I cannot see where any of them end up.I can see the obvious ones likefloat4 PS_ScreenSize : PB_ScreenSize;And all the things like that I would expect to see, but nothing that maps to the data in SRenderObjData.Anyone point me in the right direction ?CheersPaulFollow"

Answer: "0Accepted AnswerHi @REDACTEDUSERThe way to go here is to declare the parameter by name and enumeration, fill it, and then match it in the shader – here is a pre-existing example for sending a camera's front vector to a shader:The enum declaration (ShaderComponents.h):ECGP_PB_CameraFront,The mapping to the shader (ShaderComponents.cpp):SParamDB(PARAM(PB_CameraFront, ECGP_PB_CameraFront), 0),The data structure mapped to the buffer (ShaderComponents.h):Vec3 pCameraFront; // ECGP_PB_CameraFrontThe setting of the data into the buffer (D3DHWShader.cpp):        case ECGP_PB_CameraFront:sCameraFront(sData, r);break;And finally, the shader declaration / matching (in Common.cfx) to be used by the shaders:float4 CameraFrontVector: PB_CameraFrontSharerePost-User-7158712answered 6 years ago0I'm looking into an answer on this one for you. I'll get back to you soon.SharerePost-User-8353690answered 6 years ago0Hi Binky,Any progress?The whole shader system seems to be designed to be obfuscated.I have written a tool to parse the source code and handle all the conditional flags so I can look at what actually gets passed to the shader compiler, but it's not helped me at all.For example I can see global constants all over the place. ( I hate global shader constants, but that's another issue). When I look for those values in the C++ code, no sign of them.float4 g_PS_NearestScaled : PF_NearestScaled < psregister = PS_PROJ_RATIO_NEAREST_SCALED; >;If you do a find in files of the source code  Find all "g_PS_NearestScaled", Match case, Subfolders, Find Results 1, Entire Solution, ""Matching lines: 0 Matching files: 0 Total files searched: 11083Or  Find all "PF_NearestScaled ", Match case, Subfolders, Find Results 1, Entire Solution, ""Matching lines: 0 Matching files: 0 Total files searched: 11083Nothing,I can see it is mapped to registers ..#define PS_PROJ_RATIO_NEAREST_SCALED c221But that's all.This is frustrating as hell, I just need to do some really simple changes to the shaders, but if I cannot pass in a vec3, I am screwed.Time is running short and there is talk of me flying out to LA to work on a UE4 project, so I really want to get this working ASAP.Anything you can do to speed things up would be worth beer.I am going to be at the Lumberyard event in London on Monday, but I would like to be in the position to show something to the guys , rather than turn up and batter them with technical questions.CheersPaulSharerePost-User-9173357answered 6 years ago0Ah yes, these can be tricky to find. g_PS_NearestScaled is getting used inside of Engine/Shaders/HWScripts/CryFx/ShadowMaskGen.cfx (line 361)float2 ProjRatio = (fSceneDepth<fNearestScaledLinear)?g_PS_NearestScaled.xy:g_PS_ProjRatio.xy;SharerePost-User-7158712answered 6 years ago0Thanks Gene,That sorted me out. Now I can actually get on with some real work.CheersPaulSharerePost-User-9173357answered 6 years ago"


Question: DocumentDB slow in LIMIT_SKIP stage.

"Hi.I was used documentDB 4.0.0 version.It is my code. (This function make Numeric ID sequence)  private async generatePushMessageSequence(): Promise<number> {    const result = await this.sequenceModel      .findOneAndUpdate(        { _id: 'XXX' },        { $inc: { sequence: 1 } },        { new: true, upsert: true },      )      .exec();    return result.sequence;  }But sometimes this operation is too slow{"op":"command","ts":1644926941441,"ns":"test.xxx","command":{"findAndModify":"xxx","query":{"_id":"XXX"},"remove":false,"new":true,"upsert":true,"fields":{},"update":{"$inc":{"sequence":1}},"lsid":{"id":{"$binary":"XXX","$type":"4"}},"$db":"test"},"nMatched":1,"nModified":1,"protocol":"op_query","millis":1472,"planSummary":"IXSCAN","execStats":{"stage":"UPDATE","nReturned":1,"executionTimeMillisEstimate":0,"inputStage":{"stage":"LIMIT_SKIP","nReturned":"1","executionTimeMillisEstimate":"1441.544","inputStage":{"stage":"IXSCAN","nReturned":"1","executionTimeMillisEstimate":"2.736","indexName":"_id_","direction":"forward"}}},}In "LIMIT_SKIP" stage, 1441ms needed.I suspecting mognoose 'findOneAndUpdate' function called documentDB 'findAndModify' && limit(1)' query.But I cannot understand, 'Why too slow in LIMIT_SKIP stage.'. Bcz, nReturn is already only 1 doc.Can I get some guide for this slow situation?FollowComment"


Question: "CodeBuild runtime hours is upto 8 hrs, can i keep it running for 24hrs?"

"I am working with CodeBuild and its runtime by default is upto 8hrs. i want my build to be running 24 hrs, is there any way to achieve it?FollowComment"

Answer: "0Hi,No, according to AWS CoudeBuild service quotas, the maximum build timeout is 480 minutes (8h) and cannot be increased.CommentShareMikel Del Tioanswered 19 days ago"


Question: Deeplens ; Face Detection; send MQTT output locally

"Hello,I would like to send in parallel to the cloud inference output MQTT messages locally from deeplens to my raspberry by. In the Deeplens recipes (trash sorting) there is an example of it which I took as guide to modify the Face Detection Lambda. I am struggling to get it work. Could you please help me out here.  The version code posted here below does not send any messages anymore.Here the code I'm trying to use:#******************************************************Copyright 2018 Amazon.com, Inc. or its affiliates. *All Rights Reserved.                               **#*****************************************************""" A sample lambda for face detection"""from threading import Thread, Eventimport osimport jsonimport numpy as npimport awscamimport cv2import greengrasssdkimport timeimport moclass LocalDisplay(Thread):""" Class for facilitating the local display of inference results(as images). The class is designed to run on its own thread. Inparticular the class dumps the inference results into a FIFOlocated in the tmp directory (which lambda has access to). Theresults can be rendered using mplayer by typing:mplayer -demuxer lavf -lavfdopts format=mjpeg:probesize=32 /tmp/results.mjpeg"""def init(self, resolution):""" resolution - Desired resolution of the project stream """# Initialize the base class, so that the object can run on its own# thread.super(LocalDisplay, self).init()# List of valid resolutionsRESOLUTION = {'1080p' : (1920, 1080), '720p' : (1280, 720), '480p' : (858, 480)}if resolution not in RESOLUTION:raise Exception("Invalid resolution")self.resolution = RESOLUTION[resolution]# Initialize the default image to be a white canvas. Clients# will update the image when ready.self.frame = cv2.imencode('.jpg', 255*np.ones([640, 480, 3]))[1]self.stop_request = Event()def run(self):    """ Overridden method that continually dumps images to the desired        FIFO file.    """    # Path to the FIFO file. The lambda only has permissions to the tmp    # directory. Pointing to a FIFO file in another directory    # will cause the lambda to crash.    result_path = '/tmp/results.mjpeg'    # Create the FIFO file if it doesn't exist.    if not os.path.exists(result_path):        os.mkfifo(result_path)    # This call will block until a consumer is available    with open(result_path, 'wb') as fifo_file:        while not self.stop_request.isSet():            try:                # Write the data to the FIFO file. This call will block                # meaning the code will come to a halt here until a consumer                # is available.                fifo_file.write(self.frame.tobytes())            except IOError:                continuedef set_frame_data(self, frame):    """ Method updates the image data. This currently encodes the        numpy array to jpg but can be modified to support other encodings.        frame - Numpy array containing the image data tof the next frame                in the project stream.    """    ret, jpeg = cv2.imencode('.jpg', cv2.resize(frame, self.resolution))    if not ret:        raise Exception('Failed to set frame data')    self.frame = jpegdef join(self):    self.stop_request.set()def infinite_infer_run():""" Entry point of the lambda function"""try:# This face detection model is implemented as single shot detector (ssd).model_type = 'ssd'output_map = {1: 'face'}# Create an IoT client for sending to messages to the cloud.client = greengrasssdk.client('iot-data')iot_topic = '$aws/things/{}/infer'.format(os.environ['AWS_IOT_THING_NAME'])pi_topic = 'deeplens/infer'# Create a local display instance that will dump the image bytes to a FIFO# file that the image can be rendered locally.local_display = LocalDisplay('480p')local_display.start()# The sample projects come with optimized artifacts, hence only the artifact# path is required.model_path = '/opt/awscam/artifacts/mxnet_deploy_ssd_FP16_FUSED.xml'# Load the model onto the GPU.client.publish(topic=iot_topic, payload='Loading face detection model')model = awscam.Model(model_path, {'GPU': 1})client.publish(topic=iot_topic, payload='Face detection model loaded')# Set the threshold for detectiondetection_threshold = 0.25# The height and width of the training set imagesinput_height = 300input_width = 300# Do inference until the lambda is killed.while True:# Get a frame from the video streamret, frame = awscam.getLastFrame()if not ret:raise Exception('Failed to get frame from the stream')# Resize frame to the same size as the training set.frame_resize = cv2.resize(frame, (input_height, input_width))# Run the images through the inference engine and parse the results using# the parser API, note it is possible to get the output of doInference# and do the parsing manually, but since it is a ssd model,# a simple API is provided.parsed_inference_results = model.parseResult(model_type,model.doInference(frame_resize))# Compute the scale in order to draw bounding boxes on the full resolution# image.yscale = float(frame.shape[0]) / float(input_height)xscale = float(frame.shape[1]) / float(input_width)# Dictionary to be filled with labels and probabilities for MQTTcloud_output = {}# Get the detected faces and probabilitiesfor obj in parsed_inference_results[model_type]:if obj['prob'] > detection_threshold:# Add bounding boxes to full resolution framexmin = int(xscale * obj['xmin'])ymin = int(yscale * obj['ymin'])xmax = int(xscale * obj['xmax'])ymax = int(yscale * obj['ymax'])# See https://docs.opencv.org/3.4.1/d6/d6e/group__imgproc__draw.html# for more information about the cv2.rectangle method.# Method signature: image, point1, point2, color, and tickness.cv2.rectangle(frame, (xmin, ymin), (xmax, ymax), (255, 165, 20), 10)# Amount to offset the label/probability text above the bounding box.text_offset = 15# See https://docs.opencv.org/3.4.1/d6/d6e/group__imgproc__draw.html# for more information about the cv2.putText method.# Method signature: image, text, origin, font face, font scale, color,# and ticknesscv2.putText(frame, '{:.2f}%'.format(obj['prob'] * 100),(xmin, ymin-text_offset),cv2.FONT_HERSHEY_SIMPLEX, 2.5, (255, 165, 20), 6)# Store label and probability to send to cloudcloud_output[output_map[obj['label']]] = obj['prob']# Set the next frame in the local display stream.local_display.set_frame_data(frame)# Send results to the cloudclient.publish(topic=iot_topic, payload=json.dumps(cloud_output))# Send the top k results to the Raspberry Pi via MQTTpi_output = {}pi_output[output_map[obj['label']]] = obj['prob']client.publish(topic=pi_topic, payload=json.dumps(pi_output))except Exception as ex:client.publish(topic=iot_topic, payload='Error in face detection lambda: {}'.format(ex))infinite_infer_run()Essentially I added the few last lines and also setup the IoT part of it with certificates and policies.Thank you for any help or hintFollowComment"


Question: Lambda Selectolax Import partially initialized module 'selectolax'

"I tryed to fix this problem for hours now but i can't solve it. I did read through some similiar questions but they coudnt help me.I want to use the Selectolax HTMLParser Module inside my AWS Lambda Function. I Import the module like this from an Layer like this:from selectolax.parser import HTMLParserI always get the error:** "errorMessage": "cannot import name 'parser' from partially initialized module 'selectolax' (most likely due to a circular import)**The Problem lays not in the Name of My Function/File, i called it "Test123". As Selectolax is a public Module, i was afraid to change something after installing it with pip. I reinstalled the package at least 3 times and uploaded it again as a layer.FollowComment"


Question: Support Ordinal Position Mismatches in cdc Replication - Prefer Column Name Instead.

"Greetings,There is a Many to one database replication from many on-prem databases to one RDS database instance (Full-Load, Ongoing). In order to support a blue/green type of replication, table mapping transformations rules have been added to "include-column". This allows fields added to the source databases to be ignored from replication and that works great.Another problem is that there are over 200 source databases, with most of them having been generated from the same model database. Some of the older databases created 20+ years ago have the same columns but minor changes in some of the columns ordinal positioning in relation to the target RDS database.I have been looking at transformation rules to add-column and remove-column, however I can't seem to find a way to put priority on the column name and not ordinal position.For example, with the following Source Databases:DB_Coke.dbo.TableX.CreatedBy intCreatedDate datetime...DB_Pepsi.dbo.TableXCreatedDate datetime.CreatedBy intDoes there exist a built-in transformation strategy that will allow full-load and ongoing replication tasks to feed both source tables into the single target table below?RDS_Database.dbo.TableX.CreatedBy intCreatedDate datetimeFollowComment"

Answer: "0Hello There,Based on the requirement shared, I have tried the replicate the same at my end and was able to achieve the below.Created a database (employee) at the source instance with two tables X & Y.Create table tableX(createdby int, createddate datetime);Create table tableY(createddate datetime, createdby int);Setup DMS RI & endpoints.Configured the task with below mapping rule.* Selection rule: where schema name is like 'employee' and Source table name is like 'table%', includeTransformation rules: where schema name is like 'employee' and Source table name is like 'table%', renameAction:Rename:FinalDataThis would create a new table 'finaldata', in the target database 'employee'.To test this further, I have executed below insert queries.   * Insert into tableY values(CURRENT_TIME(),20);       * Insert into tableY values(CURRENT_TIME(),20);       * Insert into tableX values('12',CURRENT_TIME());       * Insert into tableX values('12',CURRENT_TIME());On the destination, the misplaced columns are handled, as indicated in the below result.    select * from finaldata;createdbycreateddate202023-02-18 07:43:31202023-02-18 07:43:35122023-02-18 07:43:46122023-02-18 07:43:52Hence, I would suggest you to try the above setup at your end and test it further.CommentShareSUPPORT ENGINEERSanketh Saliananswered 3 months agoRoss Bush 3 months agoThanks for the post it is very helpful. This might work for full load. However, when the CDC files land in the cdc-file folder there is no column name. I am not seeing where, in the transformation above, the column order has been reversed for TableY to match the expected structure in the target, so no errors occur during cdc ongoing replication.ShareRoss Bush 3 months agoJust to be clear all databases have a “Full-Load and Ongoing” Task that pulls data from on-prem to S3 targets. So, in the example above there would be four S3 target locations:S3:Bucket/Coke/dbo/TableX,S3:Bucket/Pepsi/dbo/TableX,S3:Bucket/Coke/cdc-filesS3:Bucket/Pepsi/cdc-filesThese buckets above are also the Source of an S3->Single RDS instance where a SourceID column is added to every table with the value of “Coke” or “Pepsi” or whatever other database is in the mix.Share"


Question: What does the X-Forwarded-For note in the docs mean?

"The docs:https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.htmlsay that:$context.identity.sourceIp — The source IP address of the TCP connection making the request to API Gateway. Warning. You should not trust this value if there is any chance that the X-Forwarded-For header could be forged.What does that even mean?Anyone could "forge" the X-Forwarded-For header easily.So what? Does it mean, that $context.identity.sourceIp shouldn't be trusted at all? Why are the docs so elusive about it? The whole internet is using $context.identity.sourceIp, and you're saying that it doesn't actually guarantee anything? Why not remove this property entirely then, in order to stop confusing people.FollowComment"

Answer: "0I think the note is from this thread https://forums.aws.amazon.com/message.jspa?messageID=669697.At that time, the value of $context.identity.sourceIp is came from X-Forwarded-For header but it is fixed now.AWS team should remove that note.CommentSharetdkimanswered 2 years ago0As stated in documentation, $context.identity.sourceIp is the source IP address of the TCP connection making the request to API Gateway.Whenever X-Forwarded-For header is available, the source IP would come from it, but we don't trust all entries in the header. We only trust the entry who made a call directly to endpoint.If the call chain is like this,ClientIp, \[proxyServer1, ..., proxyServerN], CloudFront/Regional  proxyServerN would be used for the sourceIp address, when X-Forwarded-For header is available.CloudFront adds itself to the X-Forwarded-For header but Regional endpoint API/Regional custom domain name does not add itself to the X-Forwarded-For header. Therefore, In case of CloudFront, IP address appended by CloudFront would not be used as sourceIp, but the IP of the last proxy prior to CloudFront IP in X-Forwarded-For header. In case of Regional,  the last IP address in X-Forwarded-For header would be used as sourceIp.Please ignore the warning in the documentation. The team is aware of it and the documentation would be updated.CommentShareAWS-User-1790716answered 2 years ago"


Question: Why does import fail with Jira data to QuickSight?

"I've generated an API token for our Jira board on our Atlassian account. After going on [New Dataset] -> [Jira] in QuickSight, the connection is validated after entering the API token. But this message appears even though I am an admin:"The SaaS isn’t available. The URL might not be accessible, or the instance might not be available. You can check the service and try again. Contact your data source administrator for assistance."FollowComment"

Answer: 0This is possibly due to JIRA residing in a private network and QuickSight in a public platform.In order to allow access to QuickSight to connect JIRA in private network - Please try to allow QuickSight IP address in your JIRA application network based on listings here https://docs.aws.amazon.com/quicksight/latest/user/regions.htmlCommentShareEXPERTGokulanswered a year agorePost-User-7135452 2 months agoand how can we do this?Share


Question: Blue Green Deployment

"I am using Free Tier accountLast year this time, I could do the ECS bluegreen deployment, as there is such a deployment type option for Blue Green deployment type.However, I found it recently, there is change of ECS, the UI is totally different.After I have created cluster, then goto create service, I have done the following[Capacity provider strategy]-> [user custom(advanced)-> [Fargate]->[Latest]For the Deployment Configuration[Service]-> [nginxTask1]->[1(Latest)] -> key in service name[Replica] -> [1] for Desired taskHowever for the Deployment Options, the Deployment type is disabled, the 'Rolling update' and 'Blue green' is supposed to be here, but I could not selectI could not proceed from herecould advise the problem, thanksFollowComment"

Answer: "0Quoting from the blogWhile we work to bring parity between the two consoles, both will exist in parallel. If you rely on one of the few features that have not yet transitioned to the new console, then please continue to use the previous console experience and send us feedback with what features are missingLooks like not all the options are available in the new UI and you can share the feedback while you continue using the old classic UI.--SydCommentShareSydanswered 4 months ago0Thanks for your replySo far, I could not proceed with the ECS as the deployment type is not enabled, yet to try and feedback to you as I will continue to use the old ECSJust found out, the first-time user that start with ECS, will encounter missing role issue when they first create an ECS. The error was gone when they cancel and go in to ECS again (with the old UI)CommentSharerePost-User-5288036answered 4 months ago"


Question: Aurora Multi-AZ versus reader instances - paid instances and available zones?

"We are moving from mysql to aurora on a production environment, so having multi-az/high availability is required. At this moment a replica cluster is begin setup - see screenshot.My question is about multi-az, "high availability" and the reader instances (not storage)During creation, multi-az was selected. The expectation was to see 1 reader instance marked as multi-az - but we ended up with 2 reader instances.So I am confused, do we now have 2 instances that are both multi-az (as in, 4 paid instances in total) or does the interface points out that the cluster contains 2 readers (2 paid instances in total) which are considered as multi-az? (as stated in the multi-az columns as "2 zones", meaning eu-west-1c and eu-west-1b - adding another reader in zone -1a will then display 3 zones?)Next to that, the terms "high availability" in combination with multi-az are throwing me off...Thanks for any details!-- RobFollowComment"

Answer: "0Accepted AnswerOnce it is created in a multi-az, you will have one reader and one writer as shown belowWhen you create one additional reader on top of it then it will be like belowHowever when you create a replica and select multi-az on the regional cluster it will have a replica cluster with one writer and one reader and if you do no multi-az then it will have a replica cluster with writer as shown once created. Then in case fail over you can move application from current primary to the cross-region replica by promoting your cross-region replica to be the new primary from the Amazon RDS console.when using multi-az and once it is done creatingwhen using no multi-az and once it is done creatingCommentShareEXPERTAWS-User-Nitinanswered 6 months agoEXPERTTushar_Jreviewed 6 months agoRob 6 months agoNice, that clears up the "Multi-AZ" colum with "x zones". So all instances, writer or reader, are counted into the nr of zones, acting as "Multi-AZ".Thanks!Share0You have an RDS MySQL instance that is replicating to an RDS Aurora Cluster. The cluster has two reader nodes. When you promote that cluster one of the reader nodes will flip to a writer node.CommentShareMODERATORphilawsanswered 6 months agoRob 6 months ago"When you promote that cluster one of the reader nodes will flip to a writer node."Super clear - that is indeed the case after the cluster was done setting up.ThanksShare"


Question: AWS IoT Greengrass ECR Docker deployment: GET_ECR_CREDENTIAL_ERROR

"Hey there,I tried to deploy a simple Docker image that I previously upoaded to a private repository on ECR on my Raspberry Pi. I uploaded the image using a different user than the one I have saved on through access keys on the RPI. However, both users have full access to all ECR services. Now the following error occurred as I tried to deploy the docker image:GET_ECR_CREDENTIAL_ERROR: FAILED_NO_STATE_CHANGE: Failed to download artifact name: 'docker:242944196659.dkr.ecr.eu-central-1.amazonaws.com/test_repo:latest' for component com.example.MyPrivateDockerComponent-1.0.0, reason: Failed to get auth token for docker login. Failed to get credentials for ECR registry - 242944196659. User: arn:aws:sts::242944196659:assumed-role/GreengrassV2TokenExchangeRole/82ddfef99dfb0585b238481427e354b015fa33c72fd5cf52a6b5595df294438a is not authorized to perform: ecr:GetAuthorizationToken on resource: * because no identity-based policy allows the ecr:GetAuthorizationToken action (Service: Ecr, Status Code: 400, Request ID: 60278c5f-3049-4b01-b9b8-ac4b54e6cb0c)It seems to me that somehow my RPI is not authorized to dowload the private docker image. Any suggestions how I could solve this issue?Thanks a lot in beforehand!FollowComment"

Answer: "0Hi. You added those to your user or the Greengrass device role?https://docs.aws.amazon.com/greengrass/v2/developerguide/run-docker-container.html#run-docker-container-private-ecrCommentShareEXPERTGreg_Banswered 3 months agorePost-User-9168686 3 months agodo you mean the permissions I just posted in my answer?Share0In the IAM console, find the role GreengrassV2TokenExchangeRole and then add an inline policy with the required permissions.CommentShareEXPERTMassimilianoAWSanswered 3 months agorePost-User-9168686 3 months agoThank you!Share0just to clarify: I know that I need to add the following permissions to the user on my RPI, but I already granted it ecr:* so all of these should be included.{  "Version": "2012-10-17",  "Statement": [    {      "Action": [        "ecr:GetAuthorizationToken",        "ecr:BatchGetImage",        "ecr:GetDownloadUrlForLayer"      ],      "Resource": [        "*"      ],      "Effect": "Allow"    }  ]}CommentSharerePost-User-9168686answered 3 months ago0okay so apparently I need to add these permissions to my thing group that the IoT Greengrass Core device belongs to. Is that correct? and how do I do this?CommentSharerePost-User-9168686answered 3 months ago0There are two things that need to do,Add permission for the role to do ecr permissionsAlso do the permissions in the ecr repository level accessCheck and post here again if still issueCommentShareSaravana Kumar Damodharananswered 3 months ago"


Question: Aws Ampify Cognito Session Persistance

"Hi the cognito session is not getting persisted below is my auth configurationAmplify.configure({Auth: {mandatorySignIn: true,userPoolId: 'adasdadasdada',userPoolWebClientId: 'tesstt',storage:localStorage,oauth: {domain: 'spring-auth-oro.auth.ap-south-1.amazoncognito.com',scope: ['email', 'aws.cognito.signin.user.admin', 'openid'],redirectSignIn: 'http://localhost:8080',redirectSignOut: 'http://localhost:8080',region: 'ap-south-1',responseType: 'code'}}});FollowComment"

Answer: "0Hello,I understand that you want to achieve persistence with Amplify Cognito session. As per the shared Amplify.configure(), there are few configuration issues such as the region is being passed into the incorrect location, and by default, Amplify uses the local storage already, so there is no need to define that.We need additional information to troubleshoot this issue, and some information are account-specific, which cannot be shared publicly. If you have AWS support plan, kindly open a support case with AWS using the following link.In support case, please include following additional information:What does lack of "persistence" mean here? Is the user receiving tokens? If yes, do they see the tokens stored within the local storage of the browser?How long are the tokens set to live for, when configuring the Auth resource?Do you see any errors, after setting set window.LOG_LEVEL = 'DEBUG';, as described here, around the time the user signs in, as well as when the tokens are expected to be refreshed.How are you defining the localStorage object?How is the user signing into the application? Is it the Authenticator module, or the SDK methods? What function is used from the Amplify SDK?What  is the version of  SDK?CommentShareSUPPORT ENGINEERIsha_Kanswered a year ago"


Question: Need help to get the correct CA certificate for Amazon s3 endpoints

"Hello,Need help to find which CA certificate to use for the Amazon s3 endpoints?I see the list of certificates here - https://www.amazontrust.com/repository/Tried https://www.amazontrust.com/repository/AmazonRootCA1.pem with "ap-southeast-1" but it failed.Please can someone help me to understand which certificate to use and if we need to use separate certificates for every AWS region (such as ap-south-1, us-east-1, etc.) then how to identify them? Thanks.FollowComment"

Answer: "0The CA Certificates required to work with s3 are covered in the FAQ in following blog post[1] which goes over the s3/CloudFront migration to Amazon Trust Services. See the question "What do I need to do?" which mentions that you need to "update your client certificate trust store to include all of Amazon Trust Services’ root certificates".So to answer your question, you will need to trust all of the root CA certificates that are available on the Amazon Trust Services Repository[2].[1] Reminder: Amazon S3 and Amazon CloudFront service certificates migrating to Amazon Trust Services starting March 23, 2021 - https://aws.amazon.com/blogs/storage/reminder-amazon-s3-and-amazon-cloudfront-migrating-service-certificates-to-amazon-trust-services-starting-march-23-2021/[2] Amazon Trust Services Repository - https://www.amazontrust.com/repository/CommentShareSUPPORT ENGINEERMichael_Hanswered 4 months ago"


Question: ERROR: NotAuthorizedError - Operation Denied with EB CREATE

"Hi, I'm trying to connect to Elastic Beanstalk to deploy my Flask App but I keep getting this error = ERROR: NotAuthorizedError - Operation Denied. Access Denied, whenever I type eb create in my EB CLI. Can please urgently help with us?The permissions I have given to the user are - AdministratorAccess-AWSElasticBeanstalk, AmazonEC2FullAccess and AWSCodeCommitFullAccess.I am trying to deploy my local code for the app. The app gets successfully created but the env does not and I get the abovementioned error.FollowComment"

Answer: 0It looks like the user you are using to deploy the Flask app to Elastic Beanstalk does not have the necessary permissions to complete the deployment.Here are a few steps you can try:Check the IAM policy for the user. Some common policies for Elastic Beanstalk include "AWSElasticBeanstalkFullAccess" and "AWSElasticBeanstalkServiceRole".Verify the region: Ensure that you are deploying the app to the correct AWS region.Use an IAM role with Elastic Beanstalk: You can use an IAM role to deploy your app to Elastic Beanstalk.CommentShareEXPERTcloudgeek7answered 4 months ago


Question: Internal error when EMR  start_notebook_execution

"Hi All,I am creating an EMR cluster programmatically and calling the  start_notebook_execution and attaching that cluster to the notebook. This works fine when I do manually. But programmatically the start_notebook_execution is failing with below response. I am unable to locate the logs for this and debug further. Please help'StartTime': datetime.datetime(2023, 1, 8, 21, 19, 21, 610000, tzinfo=tzlocal()),'Arn': 'arn:aws:elasticmapreduce:us-east-1:account-number:notebook-execution/ex-J05JCMWBMEEUD1AE2C3MIB3CBSG0E','LastStateChangeReason': 'Execution has failed for cluster j-1ZYESBPM8DZMZ. Internal error',FollowComment"


Question: Processing cost of a Python script

I'm running some python scripts that take many hours... How will I be charged for this? ThanksFollowComment

Answer: "0Where do you run Python?If you are running on EC2, you will incur EC2 startup time and data communication charges.In the case of Lambda, the fee is based on the time the script is executed.https://aws.amazon.com/ec2/pricing/?nc1=h_lshttps://aws.amazon.com/lambda/pricing/?nc1=h_lsCommentShareEXPERTRiku_Kobayashianswered 9 days ago0In addition to @RikuEC2 Instances: If you are running your Python script on Amazon EC2 instances, you will be charged for the usage of those instances. The cost will depend on the instance type, the duration for which the instances are running, and any additional resources you use, such as storage or data transfer. EC2 pricing is based on an hourly rate for the instance type you choose.AWS Lambda: If you use AWS Lambda to run your Python script, you are billed based on the number of requests made to your function and the duration of the function's execution. AWS Lambda pricing is based on the number of invocations, the total duration (calculated in milliseconds), and the amount of memory allocated to your function.AWS Batch: If you are using AWS Batch to run your Python script, you will be billed for the compute resources used by your batch jobs. The cost is calculated based on the number and type of instances used, the duration of the instances, and any additional resources consumed.AWS Glue: If you are using AWS Glue to run your Python script for data processing or ETL (extract, transform, load) tasks, you will be billed based on the number of Data Processing Units (DPUs) consumed during the job execution. AWS Glue pricing is based on the number of DPUs used per hour.CommentShareEXPERTsdtslmnanswered 9 days ago"


Question: Limited number of in-flight DDR requests on SDAccel shell

"Hello,I am developing an RTL kernel for SDAccel 2019.1, which performs single read requests (no bursts) to all DDR4 channels. Since the bandwidth I was getting was significantly lower than expected (only about 0.7 memory requests per cycle across all four channels in aggregate), I developed a simple kernel that performs single sequential accesses to benchmark the ideal performance of the four memory channels. To my surprise, it appears that channel 0 performs differently from channel 1, 2 and 3, which contradicts what is said in the specifications: https://github.com/aws/aws-fpga/blob/master/hdk/docs/AWS_Shell_Interface_Specification.md#ddr4axi "NOTE: There is no performance or frequency difference between the four DRAM controllers regardless whether they resides in the CL or the Shell logic"In particular:channel 0 has a latency of 80-140 cycles at 250 MHz, vs 10-20 cycles for the other channelschannel 0 only supports 18 outstanding memory requests, while the other channels are limited to 3Because the maximum number of outstanding memory requests is, in both cases, lower than the latency, the maximum bandwidth that can be achieved with single requests is very low, about 19% of the ideal bandwidth on channel 0 and 17% on the other channels.Considering that the specs say that "The DRAM interface uses the Xilinx DDR-4 Interface controller" I was expecting the interfaces would behave similarly to the MIG DDR3 controller, where accesses can be fully pipelined and it is possible to achieve almost 100% of the ideal bandwidth even with single requests.If this is the case also for the DDR4 controller, then I believe that some of the blocks along the path between the kernel and the MIG may have a buffering capacity that is lower than the memory latency. I tried editing the Vivado block diagram generated by SDAccel and I could see there are some AXI SmartConnects between the kernel and the memory port inside a block called "SDx Memory Subsystem", but it is flagged as Read-Only. Is there a way to edit those block, or to configure SDAccel in such a way that they are generated without buffers that limit the number of in-flight memory requests?Thanks,MikhailEdited by: mikhai on Apr 22, 2020 1:21 AMEdited by: mikhai on Apr 22, 2020 5:10 PMFollowComment"

Answer: "0Accepted AnswerSDAccel platforms are optimized for larger block transfers into and out of DDR memories. Any time a kernel or host requests very short (1-4 data-beat) accesses, we expect a drop in memory bandwidth. As you pointed out, this is a direct result of the relationship between round-trip latency and command pipelining. That is why Xilinx recommends memory accesses that are close to the AXI protocol limit of 4kB per access, as described in this documentation link: https://www.xilinx.com/html_docs/xilinx2019_2/vitis_doc/Chunk2020182740.html#ghz1504034325224 . The best path forward would be to increase your kernel’s burst-length and add internal BRAM buffering, as needed, to support non-sequential data access patterns. Trying to increase command pipelining might help to a limited extent, but for accesses as short as singles or 4-beat bursts, but will not efficiently utilize the AXI bandwidth therefore limiting in overall bandwidth.Also, the results posted indicate a significant difference in latency between the SDAccel and HDK design methodologies. Please keep in mind that SDAccel solutions always provide pathways from the host PCIe bridge to each of the DDR memories, causing kernel-to-memory traffic to also pass through AXI interconnect switches. Some of those latency cycles are consumed in the clock domain crossing which is present, by default, in the kernel-to-memory pathways of SDAccel solutions. You have control over your kernel clock to possibly eliminate those CDC cycles.Please let us know if you are still seeing issues or have additional questions.CommentShareAWS-kristopkanswered 3 years ago0A small follow up for other developers who may be interested in maximizing bandwidth with short requests.I benchmarked the four DDR4 channels using a simple accelerator that sends out read requests with arbitrary stride and burst length, both as an RTL kernel in SDAccel and using the HDK flow. Here are the main findings:Achieved bandwidth in GB/s, per channel, with 2^20 sequential reads as a function of burst length: https://pastebin.com/05JNMd00 (the ideal bandwidth per channel is 64 bytes/cycle * 250 MHz = 16 GB/s)Bandwidth in GB/s per channel with burst length = 1 (ARLEN=0) as a function of the stride (in 64 byte words): https://pastebin.com/2uE42E8uWith burst length = 1 (ARLEN=0), stride = 1:Latency in cycles:HDK, DDRA, B, D: 39-170, avg. 64.1HDK, DDRC: 43-177, avg. 74.9SDAccel, bank1, 2, 3: 70-208, avg. 84.2SDAccel, bank0: 74-191, avg. 91.8Average number of in-flight requests:HDK, DDRA, B, D: 32.4HDK, DDRC: 35.8SDAccel, bank1, 2, 3: 17.8SDAccel, bank0: 17.8TL;DRFrom the accelerator perspective, the DDR4 on SDAccel have slightly longer latency and sustain about half the number of in-flight requests compared to HDK.The minimum burst length that saturates the bandwidth is 2 on HDK and 8 on SDAccel.bank0 or DDRC (the controller implemented in the shell) has a 5-6% lower bandwidth than the other ones.Bandwidth as a function of the stride is consistent with the address mapping specified here: https://forums.aws.amazon.com/message.jspa?messageID=897304Edited by: mikhai on May 4, 2020 8:54 PMCommentSharemikhaianswered 3 years ago0Hi mikhaiI have sent you a  PM requesting additional information, in order to help resolve this.RegardsamrxilinxCommentShareamrxilinxanswered 3 years ago"


Question: How to query Codebuild/Cloudwatch Logs from Buildspec.yml in CodeBuild

I have an automated Codebuild project that is triggered by a Github Enterprise Pull Request Webhook. The problem is that I cannot see the information related inside the PR comments.  Maybe there is a way from buildspec.yml to push certain logging information back to the PR comments so certain non-privileged users can see the Codebuild results.FollowComment


Question: How to create BarCode scanner application in AWS

How to create BarCode scanner application in AWS?I need to make a small demo that allows me to scan barcode using my mobile phone then to retrieve saved details about it.How can I do this In AWS..if there is no ready made solution..how to architect this solution?FollowComment

Answer: "0You should probably do the scanning and decoding on the device and then use an API call to something like API Gateway -> Lambda to get the related data. An alternative could be to take the image on the device, upload it to the cloud, either to S3 or directly to API GW -> Lambda for processing, depending on the side of the images. I would probably choose the first option.CommentShareEXPERTUrianswered a year agoMaha a year agoCould you please elaborate what do you mean by the decoding on the device? how?...I would like to use my mobile camera to read/scan the barcode live without taking image and save it in my mobile ..Share"


Question: Auto Scaling Group SNS Notifications not being sent

"I've setup SNS notifications for my auto scaling group for all events. The SNS topic is configured to the following endpoints 1) Email in JSON format to me 2) A lambda function that sends the message to Microsoft Teams Webhook.When the notification was initially setup from Auto Scaling Group's activity notifications page, I've received the test notification on both Email and Microsoft Teams, proving that the Lambda function had executed properly. However, when I tried increasing the desired capacity of the auto scaling group to test the notification as per the guidance of this AWS article, I did not receive any notifications even after verifying that the instance is running and passing all health checks. Consequently, when I triggered instance termination by decreasing the auto scaling group's desired capacity, I did not receive a notification on both email or Microsoft Teams.Does anyone know why this is happening and how to troubleshoot?FollowComment"

Answer: "0Hello,I understand that you have setup Autoscaling Group (ASG) event notification, to monitor scaling activity, using SNS which has email and Lambda function subscription. Initial ASG SNS notification was working, but later tests done by increasing and decreasing ASG desired capacity did not work as expected, since you neither received email not Microsoft Teams (via Lambda) notification.In order to troubleshoot this issue, first make sure that ASG notification is including event types Launch and Terminate. Please make sure from ASG Activity history that instance is created/terminated successfully. As per your description, I believe you have already verified up to this point.Moving further, we can check whether ASG scaling event is actually triggering SNS notification. SNS CloudWatch metrics will provide information on NumberOfMessagesPublished, NumberOfNotificationsDelivered and NumberOfNotificationsFailed. If you do not see any data in NumberOfMessagesPublished, it indicates no message was published to SNS topic from ASG side. If you find NumberOfNotificationsFailed, that indicates possible issue with SNS delivery. For Lambda endpoint, you can also enable Delivery Status Logging, which provides further details on whether message delivery and possible reasons for delivery failure.If you find that the issue is not identified with general troubleshooting discussed above, we will require details that are account-specific and non-public information. In that case, please open a support case with AWS using the link: https://docs.aws.amazon.com/awssupport/latest/user/case-management.htmlCommentShareSUPPORT ENGINEERIsha_Kanswered a year ago"


Question: Using AWS backup service for DB backup

"Hi,My question is regarding the execution of backup job in AWS backup service.Description of the use case:We have lots of DBs from different kind on EC2 (Oracle, PostgreSQL, MySQL) + PostgreSQL RDS and Aurora cluster.To backup the DB we currently use EBS snapshots, we have a bash script that:•put the DB into backup mode•create snapshots on 2 EBS that contains the data and redo/wal files•end backup mode•delete old archives and snapshots according to retention policy.For RDS/Aurora we have a script to create snapshot, and delete old onesIn both cases the backups are executed several times per day and it can also be triggered on demand from our tools.We want to check the option to use AWS backup service to replace the EBS/RDS snapshots creation, mainly to simplify the copy of the snapshots to DR account.I have read the documentation about the service and my questions are:What would be the suitable solution in AWS backup for our use case?Can we call a backup job from our script using cli and make sure the job will be executed immediately?In our use case, do we need to create a job per DB or there is an option to create something generic to all but still have flexibility to trigger the backup from the script?Thanks,EldadFollowComment"

Answer: "0What would be the suitable solution in AWS backup for our use case?-> AWS Backup will completely fit your use case. You can add a particular tag (key-value pairs) to your RDS and EBS resources.Can we call a backup job from our script using CLI and make sure the job will be executed immediately?-> There is no difference between the backup job executed by the backup plan or on-demand backup using CLI. The backup job will be in running state with start within the window mentioned in the backup plan or CLI command.[+] https://docs.aws.amazon.com/cli/latest/reference/backup/start-backup-job.htmlIn our use case, do we need to create a job per DB or there is an option to create something generic to all but still have flexibility to trigger the backup from the script?->  You can create a single backup plan for all the RDS and EBS resources with tag based selection and associate same tag to each resource if the retention, start time, backup vault is same.  You can also create different backup plan if the start time, backup vault, retention or tags associated with resources are different.-> So, the reason you should switch from script to AWS backup is that this will eliminate the use of scripts, and it's maintenance + there is no extra charge for AWS backup service as you pay for snapshot, and you have benefits of AWS backup like Vault lock, backup protection etc.[+] https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html#features-for-all-resourcesCommentSharerePost-User-5605882answered 10 months ago"


Question: How does it work differently per target group?

"Hello? ExpertsAs I know, NLB has two different target group as below;instance The targets are specified by instance ID.ip The targets are specified by IP address.Q1) Does backend instances will establish connection with client ip address when target group is instance?Q2) Does backed instances will establish connection with NLB private ip address when target group is ip?Q3) If Q1 & Q2 are right, why it connects differently base on target group?FollowComment"

Answer: "0Accepted AnswerQ1: If you specify targets using an instance ID, the source IP addresses of the clients are preserved and provided to your applications.  Q2) If you specify targets by IP address, the source IP addresses are the private IP addresses of the NLB nodes. If you need the IP addresses of the clients, enable Proxy Protocol and get the client IP addresses from the Proxy Protocol header. Q3) different target groups for different types of requestsCommentShareEXPERTRizwan_Manswered 4 years ago"


Question: RDS Oracle database: ORA-01031 while granting privilege

"Hi,Logged on as the m aster user on a Oracle 19 RDS database, I would like to execute the following:GRANT EXECUTE ON XDB.DBMS_XMLSCHEMA TO appl_owner;This fails with ORA-01031: insufficient privileges.I tried this with RDSADMIN package:EXECUTE rdsadmin.rdsadmin_util.grant_sys_object( p_obj_name=>'DBMS_XMLSCHEMA', p_grantee=>'FWD_OWNER', p_privilege=>'EXECUTE');But that want to grant privileges on SYS objects, so it doesn't find objects in schema XDB.How can I grant privileges on objects in different schema (xdb) to o different user/schema?ThanksFollowComment"

Answer: "0When you have Database instance in AWS i.e. RDS, you will never be granted the FULL blown Admin as this could be used to elevate access to the underlying operating system.Please see the table in the link of the Maximum privilege that your master user will be grated. You may NOT grant any access higher than what the master user is granted. You can see in your error message that you cant alter SYSTEM access.. This is because AWS does not allow SYSTEM to be altered.https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.MasterAccounts.htmlIf this answers your question, please be sure to accept so that this helps others and me.CommentShareGary Mcleananswered 2 months ago0DBMS_XMLSCHEMA does not look like owned by sys but XDB.  XDB should able to give grant to respective owner.CommentShareJitenderanswered 2 months ago0XDB has no active login and connecting to XDB via proxy (alter user xdb grant connect through <<master user>>) is not allowed.But... the problem is no longer an issue really: EXECUTE privilege on XDB.DBMS_XMLSCHEMA was granted to PUBLIC; a detail I overlooked. The query I had to check that showed I had an issue, did not check on privileges granted to PUBLIC. So the application can call this package successfully.Case closed, thank you for your responses!CommentSharerePost-User-4966726answered 2 months ago"


Question: "Connect failed" when trying to connect to the DocumentDB cluster from AWS C9

"In order to insert data to a cluster with a python script, I try to connect to my cluster DocumentDB on AWS from AWS C9 with this tutorial. But every time I try to connect I get connection failed after 6 attempts:    (scr_env) me:~/environment/sephora $ mongo --ssl --host xxxxxxxxxxxxx:xxxxx --sslCAFile rds-combined-ca-bundle.pem --username username --password mypassword    MongoDB shell version v3.6.3    connecting to: mongodb://xxxxxxxxxxxxx:xxxxx/    2022-03-22T23:12:38.725+0000 W NETWORK  [thread1] Failed to connect to xxx.xx.xx.xxx:xxxxx after 5000ms milliseconds, giving up.    2022-03-22T23:12:38.726+0000 E QUERY    [thread1] Error: couldn't connect to server xxxxxxxxxxxxx:xxxxx, connection attempt failed :    connect@src/mongo/shell/mongo.js:251:13    @(connect):1:6    exception: connect failedI tried to remove the lock file to repair the instance as they do in this answer:    (scr_env) me:~/environment/sephora $ sudo rm /var/lib/mongodb/mongod.lock    rm: cannot remove '/var/lib/mongodb/mongod.lock': No such file or directoryI know that the "fail to connect" error can be caused by a lot of reasons, but usually they are:the service (mongo) might not be running on the destination server;the service (mongo) might be listening on a different port number;the service (mongo) might be protected by a firewall somewhere on the destination.So how can I ensure that:mongo is running on the destination host?mongo is listening on the defined port?I've allowed to connect to the destination host on the defined port from the IP address or network range you're running the command to connect?FollowComment"

Answer: "1Hi. Make sure the Cloud9 instance is in the same VPC as Amazon DocumentDB cluster. Then double check the security group attached with DocumentDB, it should have an inbound rule to allow connections with source the C9 security group.I recommend following the getting started guide from the DocumentDB documentation.CommentShareMihai Aanswered a year agoAWS-alphacharlie a year agoThank you very much for your reference @Mihai yet I struggle with installing the mongo shell in my AWS Cloud9.Indeed, it seems I can't create the repository file with the echo -e "[mongodb-org-4.0] \nname=MongoDB Repository\nbaseurl=...it  returns: mongodb-org-4.0.repo: No such file or directoryFurthermore, when I tried to install the mongo shell with sudo yum install -y mongodb-org-shell which I didn't had, and which I installed, it returns repolist: 0ShareMihai A a year agoWhat's the full 'echo' command you're running ?Share"


Question: QR Code not showing on EC2 Linux

"I have a message trigger project on my Windows machine. When I run the code through an IDE, it works fine, displaying my web page along with the QR Code image generated by WhatsApp. I transferred the same code to EC2, and the web page works fine, but the QR Code image doesn't appear. I have already installed all the necessary dependencies, but the QR Code doesn't show up when I run the command "node bot1". The code runs and opens a port, but the QR Code doesn't appear on that port.FollowComment"


Question: Image recognition/comparision

I have to make a real project on image recognition where I have some metadata images and if I shows new image to system then it will recognize image on based on images I have in data.AWS comparision service or AWS rekognition service.Like I have 2 images for testing and 1 image is real  if I give 2 images to for 1st image then it has to compare them and give insight according to it.How can we do this?FollowComment

Answer: "0Accepted AnswerHi,If you are looking to compare faces, you can use Amazon Rekognition, which provides a CompareFaces operation to compare faces in images. Pls refer: https://docs.aws.amazon.com/rekognition/latest/dg/faces-comparefaces.htmlIf you are looking to compare the image exif data, you can use S3, SDK & AWS Rekognition. Pls refer: https://docs.aws.amazon.com/rekognition/latest/dg/example_cross_DetectLabels_section.htmlFor other use cases, you could try the Image Similarity AI Solution Kit, which calculates the similarity of two images to determine how similar they are. Pls refer: https://awslabs.github.io/aws-ai-solution-kit/en/deploy-image-similarity/Hope this gives you a fair idea on how to implement your solution.ThanksCommentShareRamaanswered a month ago"


Question: How to create a sagemaker serverless endpoint via cloudformation?

"based on the documentation, i am creating a model and trying to create an sagemaker endpoint configuration for a serverless endpoint (sample below) , I added a ServerlessConfig attribute in my endpoint configuration resource below,  but i'm get an error = "Property validation failure: [Encountered unsupported properties in {/} [ServerlessConfig]]. any ideas?SageMakerModel:    Type: AWS::SageMaker::Model    Properties:       Containers:         -          Image: !Ref ImageURI          ModelDataUrl: !Ref ModelData          ExecutionRoleArn: !Ref RoleArnSageMakerEndpointConfig:    Type: "AWS::SageMaker::EndpointConfig"    Properties:      ServerlessConfig:         -          MaxConcurrency: 5FollowComment"

Answer: "0ServerlessConfig is a property of the ProductionVariant object. So, something similar to this would work -Type: "AWS::SageMaker::EndpointConfig"Properties:  ProductionVariants:    -        VariantName: serverless-variant      ServerlessConfig:         MaxConcurrency: 5        ...CommentShareDurga_Sanswered 8 months ago"


Question: Custom Identity Provider blank Message in response

"I have a problem using the Custom Identity Provider, I am trying to setup the SFTP Transfer service using this page: https://docs.aws.amazon.com/transfer/latest/userguide/authenticating-users.htmlI used this template  https://s3.amazonaws.com/aws-transfer-resources/custom-idp-templates/aws-transfer-custom-idp-basic-apig.template.yml to create my CF stack and when I test authentication using the SFTP Transfer console I get the following:{"Response": "{"Role": "arn:aws:iam::0000000000:role/s3-sftp-transfer","HomeDirectory": "/"}","StatusCode": 200,"Message": "","Url": "https://xxxxxxxxx.xxxxxxxx.amazonaws.com/prod/servers/s-xxxxxxxxxxxxxxxx/users/myuser/config"}According to the documentation, what I am getting in the "Response" field, should be in the "Message" field, and when I test with the AWS CLI test-identity-provider, I get the URL and statusCode, but again the "Message" field is blank.Does anyone have any idea what I am doing wrong, or what I could check?I manually tested the Lambda and the API Gateway and they both seem to work. I am not sure where to look next.Thank you!FollowComment"

Answer: "0kio14:Assuming you are using a valid Role ARN, you are seeing "message" as blank because your integration is working as expected. Have you tested with an SFTP client? Are you seeing any other indications that tell you it's not working?Thanks,SmithaCommentShareAWS-User-9826533answered 4 years ago0Thanks for confirming Smitha, I was under the impression that the Message needed to contain the Role as per the documentation on the AWS site.I will continue testing on my end with an SFTP client to see if I can get it to work.Thanks again!CommentSharekio14answered 4 years ago"


Question: An error occurred while calling o352.pyWriteDynamicFrame. Job 1 cancelled because SparkContext was shut down caused by threshold for consecutive task creation reached

"Hi,I have a Glue job script that ingests tables from a Postgres database to AWS Catalog database. Here are the steps of the ingestion:Read postgres tables using SparkDFConvert SparkDF to DynamicDFWrite the DF directly to a table using DynamicDF's sink.writeFrame()We set the "Maximum concurrency" to 8 for this job. We have another Glue job running as a workflow which triggers this job 8 times to ingest 8 tables simultaneously with different parameters. The total number of DPUs of the 8 concurrent job runs is around 100. Sometimes, the jobs ran successfully. But sometimes, some of the jobs succeeded but some failed with the following error:An error occurred while calling o352.pyWriteDynamicFrame. Job 1 cancelled because SparkContext was shut down caused by threshold for consecutive task creation reachedThe above error message indicating the job failed while calling o352.pyWriteDynamicFrame. But it also happened while  calling o93.purgeS3Path. So, I don't think it's related to a specific function in the job and I think it's more likely related to the job configs. I couldn't find any answer on this online. I also checked our service quota and don't think the jobs exceed any limitations, like the maximum number of concurrent running DPUs, maximum number of concurrent job runs, etc. Do you have any suggestions on why this happens and how to fix it? Should I set the "Maximum concurrency" to a higher number, like 16, for the job?FollowComment"


Question: AWS Glue crawler

"AWS Glue crawler is not pulling all tables names from s3 bucket. Files are present in S3 bucket, if there any setting we have to change for crawler so it pull all the tables from s3  bucket. Is there is know issue for glue crawler to pull data from s3 bucket.FollowComment"

Answer: "0Hi,how the Crawler is extracting the metadata from the files in the buckets depends on a few factors that you can review in this knowledge base article.For example if the files are in subfolders/prefixes and are very similar they could be detected as partitions. More information can be found here.To have more precise feedback for your specific situation could you please provide an example of the structure in S3 and the tables that are and are not created by the crawler?Thank you, and hope this helpsCommentShareEXPERTFabrizio@AWSanswered a year ago"


Question: Backend and frontend communication In ECS

"I have 2 containers in ECR. I have formed an ECS cluster with 2 task definitions, 2 services. The frontend is an angular container, backend is springboot container.I have a .com dns. I would like to connect them. How can I go about it?FollowComment"

Answer: "0Hey Joash,There are many options and without some more business context it's hard to prescribe any solution.On top of the solutions proposed on FRONTEND AND BACKEND CONNECTION IN AWS ECS by Venkat Penmetsa, I would recommend you to consider ECS Service Connect.CommentSharefercunhaanswered 5 months ago"


Question: Client VPN doesn't work suddently

I have a client VPN endpoint in ap-southeast-1 region. I was able to connect to the VPN using OpenVPN Connect on 29 Nov 2022. But now I cannot connect any more. No configuration changes to the VPN endpoint. The ssl cert is still valid. OpenVPN only shows timeout. What could go wrong? Any hint is appreciated.FollowComment

Answer: "0It would be good, if you could have posted the actuall error. I assume that you might have mutual authentication . Please refer to the below articles for troubleshooting.https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/troubleshooting.htmlhttps://docs.aws.amazon.com/vpn/latest/clientvpn-user/common-troubleshooting.htmlIf you still face issues, please post with more logs .CommentShareAWS-User-7653869answered 6 months agoblacktulip11 6 months agoHere's the snippet of the OpenVPN Connect log. Seems the DNS server was resolved but the ssl handshake failed.Connecting to [ascfd.cvpn-endpoint-xxxxxxxxxxxx.prod.clientvpn.ap-southeast-1.amazonaws.com]:443 (xx.xxx.xxx.xxx) via UDPv4EVENT: CONNECTINGTunnel Options:V4,dev-type tun,link-mtu 1521,tun-mtu 1500,proto UDPv4,cipher AES-256-GCM,auth [null-digest],keysize 256,key-method 2,tls-client...Session invalidated: KEEPALIVE_TIMEOUTClient terminated, restarting in..Shareblacktulip11 6 months agoI re-created a new server certificate and a few client certificates. Still doesn't work.SharerePost-User-0521889 6 months agoIm facing a similar issue, everything worked great, no changes were made at all, I can connect but now nothing resolves. I cannot get online or hit internal resources.Share0The culprit is the client certificate revocation list (crl), which expired on 30 Nov 2022. The resolution is to generate a new crl and upload to the client vpn endpoint.CommentShareblacktulip11answered 6 months ago"