Question: AWS WAF for HTTP APIs

Are there any plans to add WAF support for the new HTTP API endpoints?FollowComment

Answer: "1Currently, we do not support WAF for HTTP APIs.  We have a backlog item for this request. Thank you for reaching out.regards,SenthilCommentShareAWS-User-0795098answered 3 years ago0Hello there,Since AWS is not providing the mentioned HTTP API end-point protection you are welcome to try Wallarm's API protection service - https://wallarm.com/solutions#api-protectionIt is easy to try the service - just open a trial account at https://us1.my.wallarm.com/signupThanks.CommentShareWallarmExpertanswered 3 years ago0It seems like most WAF providers out there don't provide support for HTTP APIs. Even saw one which had a support page saying to just turn off the WAF when it comes to API requests. Too prone to false positives lol.I checked out Wallarm too as per the other suggestion and looks like they actually accurately handle API requests without any manual tuningCommentShareWokeDotaanswered 3 years ago0I'm glad to hear that you liked the Wallarm API protection product - the platform also protects gRPC, GraphQL and Websocket protocols. Feel free to reach out to sales@wallarm.com for a detailed demo.CommentShareWallarmExpertanswered 3 years ago0Is there an ETA on this feature, or is the backlog visible?CommentSharePJRanswered 3 years ago0Is there an ETA on this feature?CommentSharekkosowskanswered 2 years ago0Seconding an interest in what the ETA is, or transparency into the backlog progress.CommentSharemwalleanswered 2 years ago0I'm also in need for waf support for apigatewayv2CommentShareMyaDevSecOpsanswered 2 years ago0Much needs for me, WAF for HTTP APIsCommentShareravindrathakuranswered 2 years ago0We are also really in need of this.CommentSharecmanzianswered 2 years ago0Another upvote for this backlog item.  ThanksCommentSharemjvananswered 2 years ago0+1 feature needed !ThanksCommentSharesplissonanswered 2 years ago0+1 for either WAF support or private HTTP API.We would love to switch to HTTP API gateways and make use of the built-in JWT authorizers, but many of our API should only be accessible internally and the lack of support on HTTP API's currently pushes us to continue to use REST APIs.Thank you!CommentSharescorgatellianswered 2 years ago0Hi,We'd very much like this feature processed on your backlog. WAF protection for HTTP APIs would greatly help us as we're figuring out how to protect us from harmful intents.Regards,CommentSharetomanswered 6 months ago02023 still waiting :/CommentShareJoananswered 2 months ago"


Question: File Gateway AD Authentication

"Does storage gateway in File mode support any AD domain to join/or is it dependent on MAD/AD Connector like fSX and other solutions?i,e, over direct connect - can they just auth against On-prem DC, or a controller they manage in AWS vs one of our managed services?FollowComment"

Answer: "0Accepted AnswerYes, Storage Gateway in File mode (SMB) supports use of on-premises Active Directory directly, or customer can choose to use Active Directory options in AWS (for e.g.  EC2 based AD or AWS Managed AD)CommentShareAWS-User-9826533answered 4 years ago"


Question: Segregate logs in Opensearch based on accounts

"Hello, I have implemented a central logging solution with Opensearch. There are multiple AWS accounts for different projects. All logs are sent to Single S3 and then to opensearch. How can I segregate logs (cloudtrail, vpc flow logs, ec3 logs, web server logs) based on accounts, since one project doesnt want to see another project logs.FollowComment"

Answer: "0If you follow this process, it will work well and will save you time.  We also did the same as you, but kept running into obstacles.  So enjoy the process.https://github.com/aws-samples/siem-on-amazon-opensearch-serviceCommentShareDwayneWhiteanswered a year ago"


Question: AWS Comprehend (Flywheel) Internal Server Error

"Hello,I am building a CER model using Flywheel. First Iteration works fine, but after adding more datasets the next iterations fail with "Internal Server Error".Are there any restrictions about how the new datasets must be added (order, size...) to make the iteration works? (all datasets I used got status: "Complete")Is it possible to get more info about what is going on?Is there anyway to check the Comprehend logs?BRFollowComment"


Question: Access External AWS GovCloud and AWS China accounts via SSO CLI

"I have setup 2 external AWS accounts to be accessed via SSO. One is GovCloud and the other one is AWS-CN China. They are not showing up when I log in using the CLI. If I log in using the SSO Dashboard, I can get to them via the Management Console but I'm not presented with the temporary STS credentials. Is there a way to make this work for China and GovCloud AWS accounts?FollowComment"

Answer: "0In regard to your use-case of external accounts, it is an expected behaviour.  One can access AWS accounts outside their organization by configuring an application to access the 'External Account' (through IAM federation to the external account AWS console).Although there are options to configure AWS SSO-authenticated CLI sessions and retrieve programmatic credentials for accounts within the organization, there is no option to programmatically access the 'External Account' provided by the SSO user portal as Applications.As an alternative, you can either utilize the Chrome extension "SAML to AWS STS Keys Conversion" to obtain the temporary credentials via AWS STS service.Alternatively, you can use "assume-role-with-saml" AWS CLI command to obtain the temporary credentials.Further, obtained credentials can either be fed to the credentials file or could be set as environment variables.Hope above shared information was useful. Thank you.CommentShareSUPPORT ENGINEERVarunanswered 7 months ago"


Question: Sharing AGFIs with user groups (modify-fpga-image-attribute)

"I'd like to share FPGA images with just accounts in my company's AWS organization.https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/modify-fpga-image-attribute.html suggests appending user-groups to user groups might work, but the aws-fpga FAQ. implies this can only be done a global scope.Is there any plan support user-defined groups, or should I share with a list of IDs?FollowComment"


Question: How long does it take a Reserved Instance to change to Active State

"Hi,I purchased a Reserved Instance with the payment option as "No upfront". The State is shown as "Payment pending". How long does it usually take to change to "Active" state?ThanksFollowComment"

Answer: 0It is ACTIVE now. Just found online that it takes around 6 hours to change the state.CommentSharedan-effyanswered 2 years ago


Question: issue enabling SASL/SCRAM authentication on MSK

"Hi,I've setup a Kakfa cluster on MSK, and it has IAM role-based authentication and Public Access enabled. I'm able to connect from my local environment so no issues there.However, I can no longer update the security settings on the cluster. I'm trying to enable SASL/SCRAM authentication, however I get the following error in the console:Error updating security settingsAmazon MSK wasn’t able to update the security settings. Wait a few minutes and then try again.API responseConfiguration ARN does not exist.I've re-tried the request a number of times, trying a few different settings, however, it keeps returning this error.Is there anything I could be missing here?FollowCommentRichard Lau 8 months agoHi Nrafp,Are you able to enable the SASL authentication? I'm doing the same thing as what you are doing. But I'm not able to make the MSK and IAM role-base authentication working, let alone the SASL. Public access is enabled. Also same error I got when I tried to enable SASL: API response Configuration ARN does not exist.Can you please show me how to make IAM role-base authentication to work? And then, the SASL? Many thanks.Share"

Answer: "0Hi there, please check for 'UpdateSecurity' event in cloudtrail and check the response or error field there. If the error doesn't give any clear indication please open a support case with AWS Support Engineering so that this can be investigated as there could several reasons this might be happening.CommentShareSUPPORT ENGINEERHarshith_Manswered a year ago"


Question: XDMA register layout in BAR2 of AppPF

Hi ThereI am looking for the next level of detail on how the XDMA registers are mapped into BAR2 of the AppPF in the AWS F1 shell. I know I could read the XDMA driver code and try and work it out from there but I was hoping there was documentation for this and I was missing it. Any pointers would be appreciated! I am looking for specific offsets in BAR2 and what XDMA registers they map too. I can then use the XDMA IP user guide to work out exactly what those registers can do.StephenFollowComment

Answer: "0Hi,The AppPF BAR2 maps directly to the 64KiB of address space for registers inside the XDMA IP.This address space exposes the registers described in "PCIe to DMA (BAR1) Address Map" section in PG195 (https://www.xilinx.com/support/documentation/ip_documentation/xdma/v4_0/pg195-pcie-dma.pdf) user guide.Hope this helps. Please contact us if you need anything else.Thanks!ChakraCommentShareawschakraanswered 3 years ago0ChakraPERFECT! That was exactly what I was looking for.StephenCommentSharesbates-nvmeanswered 3 years ago"


Question: Triggering A Scheduled Task On Fargate

"Hello All!So I'm on my first foray in dockerized applications and would like to deploy a cron task on fargate to run the docker image.Some background:The docker image is published to ECR via CI/CD pipeline from my gitlab account. The image is a slim buster python, and I utilize docker in docker to build the image. The push to ECR is successful.The basics of the image are:Create a python env with the libraries specified in the requirements.txtImport libraries into main.py along with all the user-defined funcs on a func.py filemain.py runs a series of DDL sqls and DML sqls passed to our snowflake via the snowflake python SDK which stages data. Then the staged data is looped through in batches with each batch sent to an external api (smarty streets) and the GET results sent back to snowflake (basically standardizing free text address fields is the point of the application).My GoalHave this run on a chron via a scheduled Fargate task once per day.My ProblemI have a fargate cluster defined, I have a task definition, and a schedule rule (Event Bridge) created, but nothing happens. When I try to manually run the task definition nothing happens. The task shows that its in a pending status then disappears, and there are no logs given either (telling me something is failing) and nothing happens on the scheduled time from the event bridge rule I created. On the SQL DMLs I have a start|finish time for this process being logged to a snowflake table, which is showing that there is nothing being sent to snowflake. I'm sure there is something very stupid that I'm missing on getting a task to run the docker container but am not finding it. Looked through a lot of tutorials on youtube, but they all seem to be web app driven for a Fargate *service *(typically a flask), and not a simple burst ETL job on a schedule like I"m trying to do.Any examples on how to tie this all together???Added NotesI'm pretty sure the subnets I chosen have their IPs whitelisted on our snowflake, but I am expecting the API to fail as its external. But at this point I just want to trigger the docker container/FollowCommentHenrique Santana 3 months agoThank you for sharing all this information. Could you please check the task logs? Also these two below links can help you to troubleshoot what is going on:https://docs.aws.amazon.com/AmazonECS/latest/developerguide/stopped-task-errors.htmlhttps://aws.amazon.com/premiumsupport/knowledge-center/ecs-task-stopped/It is likely that the task is failing to start due to a missing configuration. Checking it would help to go deep on this and figure out what is happening.Share"

Answer:
"0Accepted AnswerHello,Please note that, you can check the "Triggered Rules", "Invocations" and "Failed Invocations" for the event rule in the EventBridge metrics[1] and you can observe the rule whether it is triggered once in a day as expected from the EventBridge side or not.Moreover, you can check the api call “RunTask” in the cloudtrail logs, to check whether the task was logged into the logs section at the scheduled time. If, the RunTask api call logs are present, then you would be able to see more details on why the tasks got stopped[2] or whether it started or not that would help you in narrowing down the issue.Thank you and have a nice day.
REFERENCE:[1] https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-monitoring.html[2]https://aws.amazon.com/premiumsupport/knowledge-center/ecs-task-stopped/CommentShareMudit Jainanswered 3 months ago"


Question: Allow a user to create an IAM profile only for specific kind of resources

"Hey guys,My question might be funky but it has a real purpose on our end.I'm trying to create an IAM profile that allow another user to create IAM profile but only for specific kind of resource. I will take the example of SQS service.We need a news SQS and for that, we will trigger a new deployment on our deployment tool. Our deployment tool must have limited access to avoid any troubles, so that deployment tool needs to have a profile that allow the creation of an SQS, a user and an IAM profile that will allow that user to use that SQS.The deployment tool should only be able to create an IAM profile that interact with SQS, nothing else. Is this even possible ? I have been able to allow a user to create IAM profile but for any type of service, which is not what I'm looking forThank you in advance for you help.Regards,VincentFollowComment"

Answer: "0This sounds like the perfect use case for IAM Permissions Boundaries: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_boundaries.htmlAlso see the blog post here explaining a use case that sounds very similar to what you're describing: https://aws.amazon.com/blogs/security/when-and-where-to-use-iam-permissions-boundaries/CommentShareGreg_Hanswered 5 months agoVincent_G 5 months agoHey Greg,Despite reading the documentation, I still don't see how to create that policy.Creating the policy to allow the user to create a policy is fine but how to allow that user to create policies that only set permissions for SQS ?Because it's 2 separated item no ?Regards,VincentShare"


Question: AWS Registered Domain Not Updating

"Hello,I purchased a domain from AWS. I verified it by email. For the last month I can't get it to host static site nor can I get it share MX records. Originally the domain was with Linode. I probably transferred it incorrectly. However, it has had a month to shake out.dig deusdatsolutions.com ANY; <<>> DiG 9.10.6 <<>> deusdatsolutions.com ANY;; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 1407;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1;; OPT PSEUDOSECTION:; EDNS: version: 0, flags:; udp: 512;; QUESTION SECTION:;deusdatsolutions.com.INANY;; Query time: 249 msec;; SERVER: 8.8.8.8#53(8.8.8.8);; WHEN: Thu Jan 09 22:42:51 EST 2020;; MSG SIZE  rcvd: 49If I use one of AWS's NS servers, I get pretty much the same dig result.Here's what my hosted zone looks like: https://postimg.cc/c6506XbzIn the Registered domains > deusdatsolutions.com the name servers arens-1168.awsdns-18.orgns-1688.awsdns-19.co.ukns-93.awsdns-11.comns-982.awsdns-58.netCloudFront makes the site accessible at d2m10d7xvwvico.cloudfront.net, which is pointed to properly in the A record. It also uses the cert from Cert Manager.As far as I know, I have no logs to share. Finally, I created all of this following the example from CDK static site sample code. https://github.com/aws-samples/aws-cdk-examples/tree/master/typescript/static-siteI appreciate the help. Thanks.FollowComment"

Answer: "0Hi,Name servers appear in three places, and they should all be the same:~ The name servers that Route 53 assigned to a hosted zone when you created it. These are the "master" name servers, and the other two places should match these.~ The name servers in the NS record in that hosted zone.~ The name servers that are assigned to your domain registration.I'd guess that you deleted a hosted zone, created a new one, and then changed the name servers in the NS record to match the name servers for the domain registration. To make them all match again, perform the following procedures:Get the name servers that Route 53 assigned to the hosted zone. See "Getting the Name Servers for a Public Hosted Zone" in the Route 53 Developer Guide:https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/GetInfoAboutHostedZone.htmlUpdate the NS record to use the name servers that you got in step 1. See "Editing Records":https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-editing.htmlUpdate the name servers for your domain registration to use the name servers that you got in step 1. See "Adding or Changing Name Servers and Glue Records for a Domain":https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/domain-name-servers-glue-records.htmlScottCommentShareEXPERTAWS-User-6179575answered 3 years ago0That worked perfectly. Thank you for the succinct steps. I understand what I did wrong. I also would not have figured this out on my own.CommentShareJPDDSanswered 3 years ago"


Question: How can I use the Timestream Query Client - AWS SDK for JavaScript v3 > Class QueryCommand to query all table rows?

"Hello,I have successfully configured Class QueryCommand on a ReactJS project and it works well when I use a query that returns less than 2,000 records.The problem I have is when I use a broader query term like to return all the rows inside the table then the QueryCommand just loops with a ProgressPercentage of 0, it only returns the metadata of the query but no data (the same query works well through the AWS Timestream Query Editor). This is what returns on my console log:Current query status: {"CumulativeBytesMetered":10000000,"CumulativeBytesScanned":0,"ProgressPercentage":0}I read in the QueryCommand class documentation (link shared above) the following:If the size of the row (including the query metadata) exceeds 1 MB, then the query will fail with the following error message: Query aborted as max page response size has been exceeded by the output result rowSo perhaps this is the reason for my problem but I am not getting that error either.My build is the same as the AWS timestream sample found here: https://github.com/awslabs/amazon-timestream-tools/tree/f3e2a6037a158eed41ed285deec4562d5b3c73fa/sample_apps/js with @aws-sdk/client-timestream-queryMy only question is:Is it possible to use the Class QueryCommand to query and paginate all the records in a table?Update:I've found out that there is something strange going on the iteration seems to be happening but not showing, the following picture is taken from the console log:If I start a page refresh then I suddenly start getting all the query rows while the page refresh last:Best regards.Gianpiero BenvenutoFollowComment"

Answer: "0Yes it is possible. To paginate a TimeStream query command call, you can use the "MaxResults" and "NextToken" parameters in your API request.https://docs.aws.amazon.com/timestream/latest/developerguide/API_query_Query.htmlYou could iterate until the response contains a NextToken and provising MaxResults as query parameters in the request.Hope it helps!CommentShareEXPERTalatechanswered 4 months agocompanyglue 3 months agoHello alatech, thank you for your comment, I've implemented what you said, I am using the exact code as shown on https://github.com/awslabs/amazon-timestream-tools/tree/f3e2a6037a158eed41ed285deec4562d5b3c73fa/sample_apps/js with @aws-sdk/client-timestream-query but it is not working. The strange thing is that if I refresh the page with the console log open I suddenly start seeing all the query results coming up fast. I have updated my question with some screenshots of what I am experiencing.Share"


Question: Retrieving DNS Name and Port Before the onGameSessionStart Callback

"I am currently writing a server using the C# SDK. I would ideally like to retrieve the DNS Name and Port of the process before calling ProcessReady(), as there is some initialization I would to do with that information. Is this possible? Am I misunderstanding a part of the process?Thanks for the help!FollowComment"

Answer: "0Accepted AnswerHi @REDACTEDUSERThose are certainly possible.For DNS name of the instance, you'd need to use EC2 metadata, see "HostName" in https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-categories.html. Here is how you can retrieve the data: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.htmlPort, on the other hand, is under your control. The recommendation is to pass them in via command line argument when executing your game server process, e.g.aws gamelift create-fleet --region $REGION \--name "TestFleet" --ec2-instance-type c5.large  \--ec2-inbound-permissions FromPort=6252,ToPort=6254,IpRange="0.0.0.0/0",Protocol=TCP FromPort=22,ToPort=22,IpRange="0.0.0.0/0",Protocol=TCP \--build-id $BUILD_ID \--runtime-configuration ServerProcesses="[{LaunchPath="/local/game/gamelift-test-app",Parameters='port:6251 gameSessionLengthSeconds:10',ConcurrentExecutions=1},\{LaunchPath="/local/game/gamelift-test-app",Parameters='port:6252 gameSessionLengthSeconds:10',ConcurrentExecutions=1},\{LaunchPath="/local/game/gamelift-test-app",Parameters='port:6253 gameSessionLengthSeconds:10',ConcurrentExecutions=1},\{LaunchPath="/local/game/gamelift-test-app",Parameters='port:6254 gameSessionLengthSeconds:10',ConcurrentExecutions=1}]"Then, in my gamelift-test-app, I retrieve the port parameter from the CLI and use that for ProcessReady.CommentSharerePost-User-7108006answered 2 years ago0That is exactly what I was looking for. Thank you for the help!CommentSharerePost-User-5642628answered 2 years ago"


Question: Lorawan Fuota Task error message - wrong time

"Once the device has joined the network, we create a multicast group that FUOTA will target. For our tests, we're adding just a device in the group.  When the Multicast Setup is ready, we schedule the FUOTA task. Here, we're getting issues on the Start date (i.e. "The date entered must be greater than or equal to the current calendar date and no greater than 48 hour"). We are using the "calendar" control present on the web page, and we are scheduling it for at least 30 minutes into the future. It seems related to international datetime settingsFollowComment"

Answer: "0Hi OroN. The dialog says the time is in UTC but this is incorrect. It's local time. Per the screenshot below, it was 17:28 where I am when I captured it. If I change the scheduled time from 17:57 to 17:59, it works.CommentShareEXPERTGreg_Banswered 7 months agoOroN 6 months agoThank you Greb, for your quick reply. I've added more details on this threadShareGreg_B EXPERT6 months agoHi OroN. Does the dialog accept any date or time from you? If so, can you identify a pattern or time range?ShareOroN 6 months agoHi Greg_B, we just select the day on calendar control, nothing more.ShareGreg_B EXPERT6 months agoOK. I do the same and don't have a problem. Can you please try some different dates and times (forward and back) to see if it will let you select any without error? What browser and OS do you use?ShareOroN 6 months agoThanks Greg_B. We tryied with different dates and different browsers (Edge, Chrome), with no luck. We are using Win 10 Enterprise 22h2ShareShow 1 more comment0Thank you Greg, for your quick reply!We have the error message on calendar control, see my picture below. (My current time is UTC + 1)CommentShareOroNanswered 7 months ago"


Question: Redshift Data Share External Schema

"Hello,I have created a data share in our redshift cluster and added schema/tables to it. To enable  dot notation I have created external schema in consumer cluster from Redshift DB  as: create external schema "external_schema_name" from redshift database 'db_ds' schema "schema_name"; to query shared tables asexternal_schema_name.table_nameinstead ofshared_db_name.schema_name.table_namewill cost of querying this external schema be same as querying spectrum ?FollowComment"

Answer:
"1Hi Yasmeen,Creating an external schema in Redshift does not always mean that Redshift Spectrum will be used. It depends on the FROM parameter; Redshift Spectrum is used to query data directly from files on an Amazon S3.In your case, when you execute the  create external schema ... from redshift ... command, you are just mapping a data share schema to a local schema. This means both ways, with or without an external schema, will have the same cost which is, the producer cluster is charged for the data they shared and consumer clusters get charged for the compute they use in accessing the shared data.Thanks,CommentShareZiadanswered 8 months ago0Hello,In this context I would like to inform you that with respect to Redshift spectrum pricing the main parameters that can incur costs are:Volume of data scanned.S3 storage.S3 access requests made.

In case of data sharing, no additional cost is incurred to use the feature because no data is being scanned from S3 and no S3 storage is involved. You can securely share live data with Amazon Redshift clusters in the same or different AWS accounts, and across regions.There will also be no additional cost for creating EXTERNAL SCHEMA, cost is incurred only when you scan data from AWS Glue data catalog, S3, KMS or any such resources.Refer https://aws.amazon.com/redshift/pricing/  for more details on pricing.CommentShareSmrithi_Ranswered 8 months ago"


Question: DynamoDB is not responding to lambda

"The dynamodb is getting connected and the data is also stored via console, but while doing CRUD operations in lambda there is no response sent and the code exits with 1. I have tried changing boto3 versions as well but there is no error thrown nor is the response coming.Can someone please help us sort this?FollowComment"

Answer: "2Since you said you are able to save to data to dynamodb there might not be an issue while saving, You can try to add your whole lambda code in try-except block and catch the exception and maybe add some traceback that will give you more idea of what error your logic gottendef lambda_handler(event, context):try:# your awsm logicresp = save_to_db()helper()except Exception as e:logger.exception(e)Hope that will help else you can put your code snippet error snapshots, cheers!CommentShareKhushalanswered 7 months agoEXPERTIndranil Banerjee AWSreviewed 7 months agoscottgrossworks 2 months agoI have the same problem, and neither the except block nor the finally get triggered.  I obtain a boto3.session --> (session.resource) db --> table but when I call scan() on the table it stalls and I never re-enter the code.  The function times out, the container is not created, and I get 'No response from invoke container for <function name>' and an invalid lambda response, invalid json -- because I never re-enter my code to format the response.Share1Another option is if you attached the Lambda function to a VPC, you need to create a DynamoDB endpoint in that VPC so that the function can access it.CommentShareEXPERTUrianswered 7 months agoEXPERTIndranil Banerjee AWSreviewed 7 months ago0Have you ensured that the Lambda function role has the right policies to access DynamoDB. Here is a blog that you can take a look at - https://aws.amazon.com/blogs/security/how-to-create-an-aws-iam-policy-to-grant-aws-lambda-access-to-an-amazon-dynamodb-table/Also look at this document - https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_lambda-access-dynamodb.htmlOnce you ensure that and if it still fails, then please follow Khushal's recommendation.CommentShareEXPERTIndranil Banerjee AWSanswered 7 months ago"


Question: What infrastructure should I use to persist messages from a public websocket?

"I want to connect to a publicly available websocket, create a request and persist the subsequent, long running stream of events for later processing.I will likely use node.js or Java to connect to the websocket and raise the request.I want to avoid missing any messages that the service sends, so always on, re-connection and redundancy are important so that I can build an optimal data set at a later stage.What AWS architecture would you suggest?FollowComment"

Answer: 0AWS lambda and DynamoDB.and to call through Websockets you can use wscat tool https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-how-to-call-websocket-api-wscat.htmlCommentSharezouzanswered 6 months ago


Question: Is it possible and OK to migrate Windows Server 2008 and SQL 2008 R2 as is

"Hi All,Greetings, I would appreciate if you can share some advise, and your experience if its possible and OK to migrate Windows Server 2008 and SQL 2008 R2 as is (Lift and Shift) without migrating to higher version ? Most of these are running on Physical servers. Or what can be the best approach to migrate and upgrade during migration to higher version of OS and SQL ?Would really appreciate if you can provide some suggestions / guidance.Regards,DilawarEdited by: Dilawar on Sep 25, 2019 5:10 AMFollowComment"

Answer: "0HiI recommend using AWS CloudEndure to migrate your on-premise windows 2008 server and SQL 2008 r2 VMs.Please note that Support for Microsoft 2008 R2 Workloads in AWS ended in July 2019 and Windows Server 2008 R2 will end as of January 2020.When migrating to AWS it would be best for you to upgrade to newer version of Windows and SQL Server due to the end of support on both Windows Server 2008 and SQL Server 2008 R2.I would recommend you to look at the following link for AWS CloudEndure Migration:https://aws.amazon.com/cloudendure/Here is a link that discusses “Upgrade Your End of Support Microsoft 2008 R2 Workloads in AWS”:https://aws.amazon.com/blogs/database/upgrade-your-end-of-support-microsoft-2008-r2-workloads-in-aws-with-ease/MahlwempuCommentSharemahlwempuanswered 4 years ago0Thanks Mahlwempu, however if we choose to upgrade windows and SQL during migration from these physical servers 1. I think we can not use cloud endure as it does block level as-is migration, right?2. If so, what are my best options to upgrade as we migrate to AWS ?3. Any other suggestions on this migration as it need to be completed ASAP and we are not sure if applications will work on higher versions of windows and SQL4. Most importantly, there is one application TFS (Team Foundation Server 2013) running on 2 VMs..How that can be migrated to AWS ? What are all options if migrating to AWS as comparison to migrate TFS to Azure DevOps ?CommentShareDilawaranswered 4 years ago0Hello,1.Yes CloudEndure can not be used to upgrade it creates a clone of what you have on premise and move it to AWS Cloud.2.you can upgrade on-premise or launch ec2 instance using AWS provided AMI with Windows OS and MYSQL version of your choice. Do a backup of your on-premise database, copy it to S3, download SQL Backup file on S3 and restore on AWS EC2 instance.If you are not sure of the compatibility of the application with higher windows version. Then use CLoudEndure to perform the migration to AWS. This will create a clone of your server on-premise in the AWS cloud.Same as question 3. CLoudEndure will create a clone of your servers on-premise in the AWS cloud. Azure Devops service is a managed service provided by Microsoft. While Azure DevOps Server is the latest Microsoft Visual Studio Team Foundation Server renamed.Thank youMahlwempuCommentSharemahlwempuanswered 4 years ago0Thanks Mahlwempu. I'll post for further queries if any.CommentShareDilawaranswered 4 years ago0Hi DilawarI am happy i could help. Should you need any further information, please do not hesitate to ask.Thank youMahlwempuCommentSharemahlwempuanswered 4 years ago"


Question: "I forgot to close the instance after using it and it runned all night, is possible to cancel the fees for the night?"

"Yesterday night I was using my server. After I have done all my things I forgot to stop it and it continued to run all night long. Those are a lot of money, so I’m asking if it’s possible to cancel the fees for the time I didn’t use. ThanksFollowComment"

Answer: "0Basically, refunds are difficult to obtain.We recommend that you contact AWS Support to discuss the fees.Although they will not be able to respond with certainty, they may be able to provide a partial refund or other support if you tell them exactly what caused the high bill.CommentShareEXPERTRiku_Kobayashianswered a month ago"


Question: Logical Directories not visible if empty

"If I define 2 logical directories, such as "/in", "/out" and the S3 location that is mapped to "/in" contains no files, then the virtual folder is not displayed.  Only the "/out" folder is displayed, I assume because it has files in it.It is not permissions related, because if I remove the files from the S3 location mapped to "/out" then that folder also disappears in the SFTP client.How can we get virtual paths that map to empty S3 locations to appear?The use case is, the user uploads to "/in" and a lambda function moves the file from there to a hidden location for processing.  Results are then deposited in the "/out" folder.  If things are working properly, there will never be a collection of files in the "/in" folder, but I'd still like the SFTP user to be able to see that folder and drop files in there.How can we do this?FollowComment"

Answer: "0I heard back from an aws ticket which indicated:As you may not be aware of this, there is no such thing as "folder" in S3. When you upload a file to S3, such as 'prefix/key', the console interprets that as if the file 'key' was inside the 'prefix' folder. However, in reality it's just an object named 'prefix/key', and as soon as you delete the file, the "folder" is gone as well. You can workaround this by clicking the button "Create Folder" within your S3 bucket, which creates a 0 byte object with a forward slash at the end (e.g prefix/), thus it displays the "folder" irrespective of whether there are files inside of it, since you now have:prefix/  prefix/object1  prefix/object2  If object1 and object2 are deleted, prefix/ still remains to act as the "folder".That said, from the information provided, it seems to me that the issue you're experiencing might be a result of that behavior. Therefore, I'd suggest trying and creating the 0 byte object in S3 in the location your logical directory '/in' is mapped to, and see if that resolves the issue.CommentShareTTF2019answered 4 years ago0I understand the concept of prefix and key, however, this detail seems to conflict with the intended use of logical directories as defined within the aws SFTP transfer service.  The documentation suggests that logical directories can created with a mapping to an S3 location.  It does not say anything about there having to be objects at that location for it to be visible.I understand I can use the aws console to create "folders".  Can the 0 byte files with names that end with '/' be created using the API?  My initial attempts to do so using the CLI have failed.  These folders are dynamically provisioned for each user when created and having to use the console to create a "folder" is not really a viable solution.A workaround could also include writing a dummy file of some kind with the desired prefix so that the mapped folder is not empty and therefore is displayed by the SFTP properly.  However, having to create dummy files all over the place for folders that might otherwise be empty seems like a cumbersome solution, not to mention that the file would be visible to the user, possibly causing confusion inside a folder that is expected to be empty.Can you inquire with the SFTP team to find out if this is the intended user experience for the logical directories feature?Thanks.CommentShareTTF2019answered 4 years ago0The ticket response from support was:Since Transfer for SFTP is using S3 as the back end storage, we are limited to the rules and constraints that S3 forces on all bucket owners unfortunately.  You are correct, that the user experience is not going to be the same as if the storage were traditional "file and folder" storage.  The workarounds you describe are the exact workarounds we give to customer in circumstances similar to yours.  Today, there is no option but to deal with the workarounds.What I have done, however, is turned your case into a feature request for the Transfer team.  We can't change the rules of S3, but we may be able to make the 0 byte solution more manageable or automated.  I have no timeline for when this feature may be engineered or introduced, but at least your request is logged with any other customer looking for a similar solution.CommentShareTTF2019answered 4 years ago0I'm aware of the limitations of S3 and I don't think it is necessary to change its' behaviour, but rather, change the behaviour of the SFTP service.I think the important part of logical or virtual directories is that they are "pretend" directories defined for display in the SFTP client.  I think for this feature to be correct, I think they should implement the virtual folders in my SFTP client, regardless of what exists (or doesn't) in S3.   If the virtual folder is mapped to a valid S3 location that contains objects, then display them, otherwise, just display an empty folder.  They have control over the SFTP service and I imagine could do this without changing S3 at all.Thanks.CommentShareTTF2019answered 4 years ago0My name is Brian and I am with the Transfer for SFTP team.What you are running into at the moment with the directory not showing up if nothing exists would be the current expected behavior. Thank you for your suggestions on this which I have relayed to our Product Management team to be added to our back log of improvements.As a workaround in the mean time, you can use the S3 api to create 0 byte objects as place holders for your directory. If using the cli you would want to use the s3api call instead of s3 so you can utilize the put-object operation.EX:aws s3api put-object --bucket bucketname --key path/to/folder/You'll want to make sure that the end of the key name ends in a / for it to be considered a folder.https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object.htmlCommentShareBrian Canswered 4 years ago0Was this ever changed?  We are creating folders using an SFTP client and occasionally, the folders are disappearing.  We have logging turned on and do not see any DELETEs pointed at the folders (only the files within them).The behavior seems inconsistent.CommentShareBrianSGlickanswered 3 years ago0Can someone provide an update on this?  Has anything changed since 2019, or are the 0 byte workarounds still required for logically defined folder paths to be displayed in an SFTP client if the folder contains no files?CommentShareTTF2019answered 2 years ago0Hello,For the logical mapping to appear the Target mush exist on the destination. That means that the folders (0 byte object) for the users is still required.AWS Transfer Family also supports EFS and you can also configure logical directories. The difference with S3 (regarding the folders) is that if you delete the content of the folder it will not disappear and you will not lose the logical mapping since its a file system and the directories exist (but again the folders of the target are required the same way with the 0 byte object on s3).For example if I have the following mapping with my EFS both folder1 and folder2 should exist in EFS and the same applies if I use S3 (with s3 if the zero byte object does not exist but you have an object with that prefix the mapping will show, however, if you delete that object and the prefix no longer exists the mapping will not show). If we use EFS and we delete the content of the folder1 it will not remove the folder and the mapping will still appear when we do a list operation.[{"Entry": "/logical1","Target": "/fs-XXXXXXXX/folder1"},{"Entry": "/logical2","Target": "/fs-XXXXXXXX/folder2"}]Please check the links below for more information regarding AWS Transfer Family with EFS:https://docs.aws.amazon.com/efs/latest/ug/using-aws-transfer-integration.htmlhttps://aws.amazon.com/blogs/aws/new-aws-transfer-family-support-for-amazon-elastic-file-system/Kind Regards,PanagiotisCommentShareAWS-User-0349305answered 2 years ago0I know this question is old, but since someone recently asked about this again, I figured I would share the solution that worked for me, given that I had similar requirements - I wanted every SFTP mailbox to have predefined set of folders available to the user. It might help others head in the right direction.My SFTP transfer family set up uses an API gateway/Lambda function combo to authenticate users. As part of the logic in the authenticating Lambda function, I added logic so that once a user is deemed valid, any required mailbox/S3 folders are created before returning a successful authentication response. This way, the user always has the expected folder structure upon connecting.The way I created the empty folders is like other responses have suggested - creating an empty object with the key ending in slash, and the key being the full path to the folder (i.e. for expected folder structure "mytest/myfiles", create an empty S3 object with key "mytest/myfiles/".CommentSharefazcc73answered 2 years ago0As fazcc73 suggested, we did the same thing, within the custom identity-provider, we create folders using the above methods.One thing to be aware of:  If you are using S3:ObjectCreated events or similar to do any work when files are uploaded, don't forget that the zero byte files you create to imitate folders will also generate those ObjectCreated events, so your handler needs to be able to deal with those events gracefully, as well as legitimate file uploads.CommentShareTTF2019answered 2 years ago0Hello,I know this question is too old, but I'm facing the same problem in my current implementation, so just want to know, is there a permanent solution in place for this? Or we still need to follow the workarounds like 0 byte object or create folder using lambda etc.?CommentShareUser-7683793answered 6 months ago"


Question: OPTIMIZATION DATA COLLECTION - ComputeOptimizerModule not working in the OPTIMIZATION DATA COLLECTION (https://wellarchitectedlabs.com/cost/300_labs/300_optimization_data_collection)

I've been following the steps given in the cloud intelligence dashboard & LEVEL 300 - OPTIMIZATION DATA COLLECTION LAB to deploy dashboards.i test getting list of account id & data of compute optimizer. i confirmed updating account id and updating on a glue  service and on an athena service. but compute optimizer data didn't create on glue & athena.so i figured out cloud formation files that can be reasons.in Optimization_Data_Collector.yaml....      TemplateURL: !Sub "https://${CFNTemplateSourceBucket}.s3.amazonaws.com/Cost/Labs/300_Optimization_Data_Collection/module-compute-optimizer.yaml"....in module-compute-optimizer.yaml...  S3CrawlerQue:    Type: String   ** Description: SQS that will be used to trigger the crawler lambda to pull data in (Not used)**...Please let me know what am I missing and how can I fix it to get it working.FollowComment


Question: "Solution to delete new ECR images, from PutImage actions, that contain CRITICAL vulnerabilites"

"I'm trying to create a solution to capture new ECR image push, check ECR scan result to see if there is any CRITICAL finding, if it does then delete the image (based on image tag).There are existing 'bad' images that we can't cleanup yet, that's why I need this solution to only delete newly created images.Workflow will be like this:Docker push -> EventBridge captures PutImage API and trigger lambda -> Lambda function parses the input data, get the image ID, calls DescribeImageScanFindings API to get CRITICAL finding count, if > 1 then proceed to delete that image.Now here's the problem: When new image gets pushed up and lambda checks for scan result, it is either not available yet or still in progress (CRITICAL finding not yet found), and there is no way to tell if scan is still running. In the response from DescribeImageScanFindings API, there is a property called imageScanStatus, but it's the status of scan configuration. The value is always 'ACTIVE' for me. If I keep calling DescribeImageScanFindings API, the results change each time and imageScanStatus is still ACTIVE.We're using continuous scanning and StartImageScan API is disabled, so there is no way to scan on demand and check status.Is there a way to get the scan status after new image gets pushed up?FollowComment"

Answer: "0Hi,Since you have continuous scanning enabled, you can change the approach a little bit. Rather than trigger an Event Bridge -> Lambda on Image push you can trigger the same logic on completion of scan. Details about Event Bridge event on completion of scan can be found in the section - "Event for an initial image scan (enhanced scanning)" in this document. It gives a nice summary of findings by severity.--SydCommentShareSydanswered 10 months agorePost-User-2063218 10 months agoThanks for your suggestion, I tried to use this event pattern from inspector2 "Inspector2 Scan" and it looks usable for this scan-status "INITIAL_SCAN_COMPLETE".Do you also happen to know if "INITIAL_SCAN_COMPLETE" only on newly pushed images? I'm worried that this event might get triggered on existing images that could cause Lambda to delete them (old images), which is not ideal for me right now. I only want to delete new images with findings.ShareSyd 10 months agoI'm not sure if the INITIAL_SCAN_COMPLETE is applicable only for new images or also applies to images already in repo but being scanned for first time. But you can always lookup the image creation date (imagePushedAt value in describe images) and delete only images / tags pushed to repo after specific date. Not the perfect answer but should help achieve the desired result.ShareSyd 10 months agoUpdate - Did some basic testing on a ECR repo which had two images. One manually scanned and other not. After switching to Continuous scanning both the images were scanned by Inspector and I had INITIAL_SCAN_COMPLETE events for both. So I infer that you can expect one time INITIAL_SCAN_COMPLETE for already existing images which were not scanned by AWS Inspector.SharerePost-User-2063218 10 months agoHi Syd,This 'almost' works for my use case, there's still 1 problem with the functionality: When I push a new image with 2 different tags (latest and a random name, usually commit sha), only 1 of them received INITIAL_SCAN_COMPLETE treatment, that means the other image is left unchecked by my Lambda. The goal is to prevent images with Critical findings from being stored in ECS.ShareSyd 10 months agoTo test, I pushed three images with diff tags to the same repo and received INITIAL_SCAN_COMPLETE events for all three. I can only think of two reasons 1) delay in initiation of scan by AWS but that should happen eventually. 2) The image that was pushed to the repo is only a new tag of the existing image with some other tag. ie the SHA matches that of an existing image. Other than that even I'm clueless ;)ShareShow 4 more comments0About 2/3 months before continuous scanning was a thing, I had written ECR Scan Reporter | docs which I use mostly with the ECS definitions scanning (so I didn't scan all the images, only the ones I used).I specifically used SQS as the middle man here so that I can perform retries when there are too many API calls (which in repositories with 100s of images, definitely did get). But also that way I am truly waiting on events, there is no direct / sync dependencies between the trigger of the scan, the execution of the scan, and the evaluation of the scan.CommentShareJohnPrestonanswered 10 months ago"


Question: How to invoke the SageMaker model endpoint directly from PostMan without Lambda and API gateway setup

"Hi All,Good day!!How to invoke the SageMaker model endpoint directly from Postman without Lambda and API gateway setup?We know that we can invoke SM endpoint using SM boto3 SDK like below, but we would like to invoke the endpoint directly via Postman.import boto3smr = boto3.client('sagemaker-runtime')resp = smr.invoke_endpoint(EndpointName=endpoint_name, Body=b'.345,0.224414,.131102,0.042329,.279923,-0.110329,-0.099358,0.0',ContentType='text/csv')Thanks,VinayakFollowComment"

Answer: "0As I would frame it, there are 2 main things your client needs to directly invoke a SageMaker endpoint via AWS APIs (that it wouldn't need via API Gateway):Credentials for some kind of AWS IAM identity (for example an IAM User or Role) which is authorized to call that API (via IAM policies)To sign the request via the process expected via AWS APIs, which is more complex than simply passing an auth tokenHow you achieve (1) will depend on your actual use case... For example, is this just for an individual developer (who maybe already has an AWS IAM user) testing on their local machine? Are you building an application which will run somewhere you could store AWS credentials? Or are you trying to expose the model out to untrusted clients (such as users' web browsers) where directly sharing AWS credentials to the client would be inappropriate?Specifically for individual developer testing, you might refer to the guide on using Amazon Pinpoint with Postman - which goes via an IAM user.As you may see from the doc linked above, building (2) from scratch is also non-trivial, which is one good reason why it's usually best to use the AWS SDK for your target language/environment of choice (many more are available besides Python's boto3) rather than building raw API requests. There are other good reasons too, like simplified configuration for loading your credentials and setting up best practices like retries.However, assuming you validate that you really do have a good use case for Postman here:The specific API you want is SageMaker's InvokeEndpoint. On that page you'll find details about the path structure and different supported parameters... And also note that it's under the SageMakerRuntime service which is separate from the SageMaker administrative APIsEach service's base endpoint (host/domain name) is documented online, and SageMaker's are listed here. Since you're looking for the runtime, you'll be targeting something like  (https://) runtime.sagemaker.ap-southeast-1.amazonaws.comPostman actually provides native support for AWS request signing: Look for AWS Signature under the Authorization options... So no need to try and manually sign the request content yourself.Putting it all together, I believe you'll be making a POST request to something like https://runtime.sagemaker.ap-southeast-1.amazonaws.com/endpoints/my-endpoint-name/invocations. You'll see the other options documented for InvokeEndpoint correspond closely to boto3's invoke_endpoint() method.CommentShareEXPERTAlex_Tanswered a year ago0Hi @Alext_T,I have deployed the real-time endpoint on SM and data scientist (has AWS account) would like to invoke that endpoint via Postman from their local machine?Method: PostURL: https://runtime.sagemaker.ap-southeast-1.amazonaws.com/endpoints/my-endpoint-name/invocationsHow can we achieve that? Please put it in simple words.Thanks,VinayakCommentShareVinayak Shanawadanswered a year ago"


Question: "Why is my S3 Object metadata Content-Type application/pdf, refreshing pdf link until failure?"

Metadata: Content-Type; application/pdfAll my "block public access" settings are off. I can serve the file on desktop.Mobile access endlessly refreshes the webpage until failure.How do I ensure the public file is served correctly or view error logs?FollowComment


Question: Problem with due past payments

"Hi there, On February 7, 2023 I made a transfer for AWS Service Charges (for January) and also for Linked Account Allocation  (debt for November 2022).As a result, our debt for November 2022 is shown as ”payment past due” but it was paid.Could you help me to handle this?Thank you in advance!FollowComment"

Answer: 1You should submit an Account and billing case with AWS Support.CommentShareEXPERTkentradanswered 3 months ago


Question: [API Gateway] Why is the actual response time so large

"Hi everyone,I just have a test by setting up API Gateway + Lambda to check the response time.After finished i found a strange point after using X-Ray to trace the response time of APIMy curious is:1/ Why the response time so different ?2/ How can i decrease the response time of browser or postman under 100ms even just with very simple testing infrastructure (AGW + Lambda) ?p/s: i've enable API Gateway caching and thanks for your timeThe photos below of X-RayResponse time : 6~8msThe photos below of Browser & PostmanResponse time: ~ 200msFollowComment"

Answer: "0Accepted AnswerI am guessing that what you are seeing is a Lambda cold start, i.e., you are invoking the function for the first time after some idle time. To verify, go to the Lambda logs and check if re REPORT line includes "Initialization Duration".CommentShareEXPERTUrianswered 9 months agoAWS-User-8719096 9 months agoHi Uri, i found my problem. It's because of my region & network issues. Thanks for replyingShare"


Question: Regenerating CUR report for last month when S3 bucket was deleted accidentally

"Hi There, I have a scenario where the Cost and Usage report has been used for a while, however the corresponding S3 bucket was accidentally deleted last month. The bucket is now restored however there is a need to  regenerate the April report in CSV, is there any way to recreate this or is it lost due to the bucket deletion?ThanksFollowComment"

Answer: 1Accepted AnswerHi!  Please open a support ticket asking for the April CUR to be regenerated.  The support team should be able to assist with this.CommentShareLisa_Hanswered a year ago


Question: Cloudformation rule doesn't prevent stack creation

"Hi,Our cloudformation template requires user to enter the email, password and then confirm the password before creating the stack. We have cloudformation rule to check whether the password field and confirm password field are the same:Rules:  ConfirmPasswordsMatch:    Assertions:      - Assert:          Fn::Equals:            - !Ref AdminPassword            - !Ref ConfirmAdminPassword        AssertDescription: Passwords do not matchWhile it does render an error message below the password input field when two passwords are different, but it doesn't actually prevent user to create the stack. Even the password and confirm password are different, user can still successfully create the stack.This is a blocker for us, because we want to use similar mechanism to validate other template parameters.Using rule is the official validation mechanism , and we are doing in the same way.FollowComment"

Answer: "0In the console, I just tested the below template, and it worked as expected. If the fields, didn't match, I was unable to proceed, if they did I was able to create a stack from the template.AWSTemplateFormatVersion: 2010-09-09Parameters:    AdminPassword:        Type: String    ConfirmAdminPassword:        Type: StringRules:  ConfirmPasswordsMatch:    Assertions:      - Assert:          Fn::Equals:            - !Ref AdminPassword            - !Ref ConfirmAdminPassword        AssertDescription: Passwords do not match        Resources:    myVPC:        Type: AWS::EC2::VPC        Properties:            CidrBlock: 10.0.0.0/16            EnableDnsSupport: 'true'            EnableDnsHostnames: 'true'            Tags:                - Key: stack                  Value: testCommentShareAWS-Harrisonanswered a month ago"


Question: Can send but not receive emails

"Hello,I have some emails set up in MS office 365 and some are on AWS workmail for a register domain. So, to receive emails in MS office 365 and AWS webmail I set two MX records under Route 53 records. The current MX recrods value is https://tinyurl.com/ybf2ylkg . But after adding Outlook mail server in MX records the AWS webmail stop receiving emails and sender gets bounce notification.Please check it and let know if you need more details on it.Thanks & regards!FollowComment"

Answer: "0Thanks for the reply!I tried to setup a replay in MS office exchange but didn't worked for me.I have a query, Do I need to keep both MX records in AWS Route 53 or remove or only keep the primary mail server?CommentShareRavianswered a year agorobinkaws MODERATORa year agoHi,The MX records should contain the service that is the primary mail handler. Backup servers for that service can be added with a lower priority.Kind regards,RobinShare0Hi,I'm sorry to hear you're experiencing problems with receiving mail. In the MX record dns record you can only have 1 primary server to receive mail, the server with the lowest priority will receive the mail. Others will only be tried when the first server is unavailable.To achieve mail delivery on two different services, your primary service will need to redirect mails destined for the other service.Kind regards,RobinCommentShareMODERATORrobinkawsanswered a year ago"


Question: Expected max PIOPS for db.r5b instances

"Hi,I'm seeking clarification on the maximum provisioned IOPS performance for a db.r5b.8xlarge.According to https://aws.amazon.com/blogs/aws/new-amazon-ec2-r5b-instances-providing-3x-higher-ebs-performance/, this instance type is capable of 86,667 IOPS. With provisioned IOPS of 80k, we are observing a ceiling at 32k, with a significant IO queue.I noticed that on https://aws.amazon.com/rds/instance-types/, it does not mention that r5b is Nitro enabled, while r5 types are. However, https://aws.amazon.com/blogs/aws/new-amazon-ec2-r5b-instances-providing-3x-higher-ebs-performance/ mentions that r5b types are all nitro enabled.Is there a reason that an db.r5b.8xlarge instance would not achieve more than 32k IOPS?FollowComment"

Answer: "0Hi,r5b.8xlarge does support baseline IOPS of 86k. See the output below.There are often workload related factors why your experience might not be able to drive this level of IOPS. I dont see any active instances in your account with this setup so it is difficult to comment on why you are not observing higher IO rate.-Phil$ aws ec2 describe-instance-types --instance-types r5b.8xlarge --query "InstanceTypes[*].[InstanceType, EbsInfo.EbsOptimizedSupport, EbsInfo.EbsOptimizedInfo]" --output json[["r5b.8xlarge","default",{"BaselineIops": 86667,"MaximumThroughputInMBps": 2500.0,"MaximumIops": 86667,"MaximumBandwidthInMbps": 20000,"BaselineThroughputInMBps": 2500.0,"BaselineBandwidthInMbps": 20000}]]CommentShareMODERATORphilawsanswered 2 years ago"


Question: Install Microsoft ODBC Driver for SQL Server on Elastic Beanstalk

"I'm new to this cloud services and recently I'm trying to deploy my PHP project using Microsoft SQL Server on elastic beanstalk. When I run it, it says :This extension requires the Microsoft ODBC Driver for SQL Server. Access the following URL to download the ODBC Driver for SQL Server for x64: https://go.microsoft.com/fwlink/?LinkId=163712I already tried to install ODBC with config that I put at .ebextensions. I've been searching and trying any command to install the ODBC, but still not working. Is there anyway to install ODBC for SQL Server on elastic beanstalk?FollowComment"


Question: GWLB: TG Sends traffic to unhealthy instances

"I have the following scenario:1 Security VPC and 1 Transit Gateway. Inside the security VPC, there are 3 AZs and 1 Gateway Load Balancer. In each AZ there is an endpoint GWLBE to redirect the traffic to the GWLB. The transit gateway attachment is configured in appliance mode. The target group of the load balancer has 3 instances: 1 VM series Palo Alto NGFW and 2 Linux machines to simulate unhealthy VM machines in each one of the AZs. The Palo Alto machine responds to the TCP 80 probes, while the other Linux machines time-out.The target group is configured with the new failover feature to redirect existing flows to healthy instances.Testing:I am doing pings between 2 machines in 2 different VPCs. (I have also tested ssh traffic). I have found out that I have to send 2 to 4 times the ping command for the target machine to respond. After checking the flow logs, I found that the GWLB sends traffic to machines that are unhealthy, thus I only get the pings responses only when the GWLB send the traffic to the healthy Palo Alto.Why is the TG sending traffic to unhealthy machines?FollowComment"

Answer: "0Do you have Cross-Zone Load Balancing enabled?--When to use Cross-Zone Load Balancing--By default, the load balancer distributes traffic evenly across registered appliances within the same AZ. In this configuration, customers typically register more than one target within a single AZ behind the GWLB for firewall service availability and to distribute the traffic. In the event of a single target appliance failing health checks, the GWLB will route traffic to other healthy instances within the same AZ. This provides a cost-effective solution because the traffic does not cross AZ boundaries. While this setup is cost-effective, customers lose both the high availability and traffic distribution aspects in the event that all the targets in a specific AZ fail.In order to achieve high availability and balanced traffic distribution, some customers choose another approach by enabling a feature called “cross-zone load balancing”. This feature makes it easier for you to deploy and manage your applications across multiple AZs. When you enable cross-zone load balancing, GWLB distributes traffic across all registered and healthy targets regardless of which AZs these targets are in. Enabling cross-zone load balancing incurs standard inter-AZ charges when the traffic crosses an AZ.CommentShareAWS-User-6317007answered 7 months agorePost-User-0583042 7 months agoI didn't. Somehow I think the Cross-Zone feature should be mentioned more in the documentation. It was hard to find.I tried it and now I see in the VPC flow of the 3 GWLB interfaces traffic to the Palo Alto. The problem now is that the default gateway of the Palo Alto is the GWLB interface of the AZ 1. How do I configure the Palo Alto so it forwards back the packet to where originally came from?Share"


Question: Elemental Live xml to MediaLive json

"Hi All,I am trying to create a MediaLive channel that ingests into an existing MediaPackage channel. This ingest is meant to serve as a backup input to an existing workflow, where the MediaPackage is already receiving ingest from an (on-prem) Elemental Live encoder.I'd like the output configurations of MediaLive fitting the Elemental Live's so that the MediaPackage does not differentiate the two inputs.If anyone can point me to any existing tool that can do this, or any article that describe the process, that would be great.Can anyone advise how I can convert an Elemental Live (on-prem) encoder's event XML into a compatible MediaLive JSON, please?Thanks in advancemsirajFollowComment"

Answer: "0msirajThere is no tool currently available that can do the conversion from AWS Elemental Live event XML to AWS Elemental MediaLive channel json. You should be able to create a MediaLive channel config by using the Live console and MediaLive console side-by-side.Having said that, note that the workflow you mentioned in this post, i.e. Live and MediaLive both feeding the same MediaPackage channel as redundant inputs, is not a workflow that has undergone any level of rigorous testing. The Live and MediaLive outputs won't necessarily be synched. As an example the start and end of a given segment may vary between the two, or the segment numbering won't match between the two sources. There may be other issues as well, and it is not clear which of these difference would negatively impact MediaPackage.Elemental Live can be configured to have 2 events output locked so that they both send synched content to MediaPackage. And MediaLive can run in Standard Channel class, which consists of 2 pipelines that can be pipeline locked if the source contains timecode. For both of these use cases the MediaPackage would then received content that is synched.CommentShareAWS-User-5437070answered 2 years ago0thanks for response.Yes, we are aware of a good chance of out-of sync for the two feeds to MediaPackage, and are ok with that.The bigger concern was breaking the MP's endpoints, as we had experienced in the past.Having said that, we have created a workable configuration that meets our immediate needs, but:Wrt to using Elemental Live & MediaLive:we observed that the MediaLive always ingests to the first inputwhen we stop one input, the playback always breaks (because MP generates different child playlist/representation & so unable to seamlessly switch)That's for another day :)Best regardsSirajCommentSharemsirajanswered 2 years ago0Msirja,You can export your XML and open it with any text app and them copy and paste it next to a copy paste of the JSON that Media Live exports or displays in the Details > Advanced directory. If you are not worried about sync there are only about 5 or 6 settings that have to be identical so having and exact export to Media Live should not be needed.Edited by: B-Wilkinson on Sep 25, 2021 11:18 AMCommentShareB-Wilkinsonanswered 2 years ago"


Question: RtMessage payload exceeded maximum size of 4096 bytes

Hiii everyone. I have an issue when i send payload to server using Rtmessage i have exception : exceeded maximum size of 4096 bytes. I'm using Unity client and Realtime server script.Can i increase capacity of message payload size ?. Because i want to sync many data with other players.Thanks guys so much.FollowComment

Answer: "0The max message size I don't believe is configurable currently. I will check with the GameLift service team to see if they have any advice or can help you out here.If you want to "sync many data with other players"  you may be pushing the limits of Realtime. Its not designed to handle large volumes of network replication/sync.  If you need accurate, timely replication of large volumes of data you may need to consider custom servers or get clever with data compression, data splitting etc.If possible, could you provide your use case or examples of the type of data you are trying to synch?CommentSharerePost-User-5332620answered 3 years ago0I've sync many data example : Player location , Object location , Player status , ... . I splited data.I wonder why gamelift realtime server can send much large size than client does.When player connect server send all data realtime to players.CommentSharerePost-User-8219895answered 3 years ago0Hi!Unfortunately GameLift does not support this capability at the moment, however I can open a feature request on our end for this. To get some more data on what you would potentially looking for;What is the use case/flow that your server is doing such that it needs to send that much data?What is the maximum message size you are looking to support?I'd echo Pip's point above that if large scale replication across servers Realtime servers may not be solution you are looking forCommentSharerePost-User-1112179answered 3 years ago"


Question: GameSparks().DynamoDB().QueryIndex example please

"Can someone provide a working example of using QueryIndex in GameSparks cloud code for handler in an Event.I'm attempting to return a list of players in a lobby querying my table named BattleRoyalePlayerLobbyTable, with lobbyId, playerId as complex key, and index named lobbyId-index with primary key as lobbyId, using message.lobbyId passed in the request message.try{  const tableName = "BattleRoyalePlayerLobbyTable";  const indexName = 'lobbyId-index';  const keyConditionExpression = "lobbyId = :lobbyid";  const expressionAttributeNames = null;  const expressionAttributeValues = { ":lobbyid": {"S": message.lobbyId} };  const exclusiveStartKey = null;    let queryResult = GameSparks().DynamoDB().QueryIndex(    tableName,    indexName,    keyConditionExpression,    expressionAttributeNames,    expressionAttributeValues,    exclusiveStartKey);    GameSparks().Logging().Info(queryResult.ToString());    return GameSparks().Messaging.Response({ "players": queryResult.Items });}catch (ex){  return GameSparks().Messaging().Response({ "players": [], "error": ex.message });}Response receivedId: 56f05f7b-65ed-43b7-a8e4-8390dc9faf0e Type: Custom.Game.GetPlayersInLobby Category: Response ResponseTo: 92f4562c-cf9e-4f7a-ac13-444e31b1c93b  {"players":[],"error":"javaException.getCustomerFacingExceptionDetails is not a function"}Request sentId: 92f4562c-cf9e-4f7a-ac13-444e31b1c93b Type: Custom.Game.GetPlayersInLobby Category: Request  {"lobbyId":"fb3578b6797848448d5facca5555d12c"}Thank youFollowComment"

Answer: "0Accepted AnswerI figured it out.Attribute Values are type MAP.try{  const tableName = 'BattleRoyalePlayerLobbyTable';  const indexName = 'lobbyId-index';  const keyConditionExpression = '#lobbyId = :lobbyid';  const expressionAttributeNames = {'#lobbyId': 'lobbyId'};  const expressionAttributeValues = { ':lobbyid': message.lobbyId };  const exclusiveStartKey = {};    let queryResult = GameSparks().DynamoDB().QueryIndex(    tableName,    indexName,    keyConditionExpression,    expressionAttributeNames,    expressionAttributeValues,    exclusiveStartKey);    return GameSparks().Messaging().Response({ 'players': queryResult.Items });}catch (ex){  return GameSparks().Messaging().Response({ "players": [], "error": ex.message });}CommentShareKirkulesanswered a year agoHamza 9 months ago@Kirkuleshican you help me?I made a query just like yours but its gives me this error "Content does not match constraints applied by annotation [PatternConstraint]"Share"


Question: Controlling access to KMS keys using aliases fails

"I'm trying to control access to KMS keys based on their alias with a policy, following the guidance here.I want the policy to:Allow general access to kms operations for unrestricted keysDeny all access to keys that are restricted (have an alias matching restricted)Here is my attempt:{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "AllowAll",      "Effect": "Allow",      "Action": ["kms:*"],      "Resource": "*",    },    {      "Sid": "DenyKMSForProduction",      "Effect": "Deny",      "Action": ["kms:*"],      "Resource": "*",      "Condition": {        "ForAnyValue:StringLike": {          "kms:RequestAlias": [            "alias/*restricted*",            "alias/*RESTRICTED*"          ],          "kms:ResourceAliases": [            "alias/*restricted*",            "alias/*RESTRICTED*"          ]        }      }    }  ]}However, when I test this policy with the IAM policy simulator, it fails.Access to kms keys that have an alias with restricted are allowed, even if I pass in a request alias or resource alias in this simulatorIn my understanding of evaluation logic, any deny statement will be evaluated first, so operations to the key with the restricted alias should be deniedAnd even if I review the Allow * sid, it still only fails with an implicit deny, not an explicit denyIs this a limitation with the simulator, or an issue with my policy?FollowComment"

Answer: "0The following example IAM policy statement allows the principal to enable and disable KMS keys but only when all aliases of the KMS keys include "Test." This policy statement uses two conditions. The condition with the ForAllValues set operator requires that all aliases associated with the KMS key include "Test". The condition with the ForAnyValue set operator requires that the KMS key have at least one alias with "Test." Without the ForAnyValue condition, this policy statement would have allowed the principal to use KMS keys that had no aliases.{  "Sid": "AliasBasedIAMPolicy",  "Effect": "Allow",  "Action": [    "kms:EnableKey",    "kms:DisableKey"  ],  "Resource": "arn:aws:kms:*:111122223333:key/*",  "Condition": {    "ForAllValues:StringLike": {      "kms:ResourceAliases": [        "alias/*Test*"      ]    },    "ForAnyValue:StringLike": {      "kms:ResourceAliases": [        "alias/*Test*"      ]    }  }}CommentShareRishianswered 5 months ago"


Question: Why i am getting CustomerEndpoint.Address as "anonymous" in amazon connect?

I am using amazon connect streams to build my application most of the times it works fine but there are some cases when i got the phone number as "Anonymous". I don't why this is happening!Any idea??FollowComment

Answer: "1Hi,Thanks for reaching out. There are certain methods where customers can hide their phone number when making outbound calls e.g. to Amazon Connect.As an engineer based on Australia, I followed the steps here in this page under "Change your caller ID on an iPhone" and I can confirm that my calls to Amazon Connect now show the "Customer phone number" as "anonymous". Similar methods will exist for other carriers in other countries. There will most likely be other methods as well.A call will go through other intermediary carriers before arriving at Amazon Connect. If the customer phone number has been masked or is not sent to Amazon Connect then we will see this situation where the caller is marked as "anonymous".CommentShareSUPPORT ENGINEERRyan_Aanswered a year ago"


Question: Filtering dynamodb events to trigger Lambda

"Hi,I am trying to trigger a lambda function from a DynamoDB stream, and can't seem to get filtering working correctly. Without filtering the event looks like this:{'Records': [{'eventID': '8aa2e4c1d77d2596c741c378247ca43d', 'eventName': 'INSERT', 'eventVersion': '1.1', 'eventSource': 'aws:dynamodb', 'awsRegion': 'us-west-2', 'dynamodb': {'ApproximateCreationDateTime': 1682107857.0, 'Keys': {'IngestionID': {'S': 'INGEST_20230421131056'}, 'UserID': {'S': 'us-west-2:7bc59622-687b-41f8-91aa-dd63b566b3a4'}}, 'NewImage': {'IngestionComplete': {'N': '0'}, 'IngestionID': {'S': 'INGEST_20230421131056'}, 'SuccessfulUpload': {'N': '0'}, 'FailedItems': {'N': '0'}, 'FailedUpload': {'N': '0'}, 'UserID': {'S': 'us-west-2:7bc59622-687b-41f8-91aa-dd63b566b3a4'}, 'AbortIngest': {'N': '0'}, 'FailedTranscode': {'N': '0'}, 'TotalItems': {'N': '1'}, 'ItemsProcessed': {'N': '1'}}, 'SequenceNumber': '20422200000000005593358615', 'SizeBytes': 284, 'StreamViewType': 'NEW_AND_OLD_IMAGES'}, 'eventSourceARN': 'arn:aws:dynamodb:us-west-2:241649497082:table/IngestionData/stream/2023-04-21T18:31:57.157'}]}I would like to trigger my lambda when ItemsProcessed=0... I am using the AWS console (web) to configure the trigger. I have tried the following:{  "NewImage": {      "ItemsProcessed": {"S":["0"]}     }  }{  "NewImage": {      "ItemsProcessed": {"N":["0"]}     }  }{  "dynamodb": {    "NewImage": {      "ItemsProcessed": [{ "numeric": ["=", 0] }]     }  }}but none of these seem to generate triggers... What do I type in the "FilterCriteria" box in the console, to properly filter events?Thx.FollowComment"

Answer: "0Hi, can you try below?{ "dynamodb": { "NewImage": { "ItemsProcessed": { "N": [“0”] } } }}Haven’t tried directly on console, but that or slight modification of it may do the trick.And you may add the eventName as part of field.Hope it helps ;)CommentShareEXPERTalatechanswered a month ago"


Question: How do I backup my Macbook onto my S3 storage account?

"Hello,I am a basic user of Amazon S3. I do know what a "bucket" is and how to use the privacy settings. But that's it. I prefer not to know the advanced uses of it. I won't be using it for anything other than personal use.What are the steps to backup my MacBook (which the file size is about 600 GB) to my Amazon S3 account?I could just pay Apple for one of their backup plans, but I already have an Amazon S3 account (and it seems more secure and cheaper). So why not use S3?I might backup my computer twice monthly. I would like a simple process, but not have to buy a 3rd party software/plugin to do it.Any helpful link to a help file with steps, or advice is much appreciated.TFollowComment"

Answer: "0Download and install AWS CLI Create a bucket, say 'my-laptop-backup'Create an IAM User that has permissions to objects in that bucketCreate access keys for the IAM user  and install them on your laptop via 'aws configure'Twice a month run this command:aws s3 sync . s3://my-laptop-backup --deleteCommentShareEXPERTkentradanswered 3 months agorePost-User-8702646 3 months agoThank you for the helpful response.Would I run that command you gave me in "terminal"?And, if possible, could you provide a link to a help file on how to install my IAM access keys on my laptop via 'aws configure'?thanks again,TSharekentrad EXPERT3 months agoYes, the commands are run in the terminal. Here is a link that walks to setting up the access keys. https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.htmlShare0I'm afraid I'm stuck.I followed all the steps as best I could, but was only 30% confident that I was doing them right. All of the help files are beyond my understanding.Some of the errors I got along the way.This one when I tried to configure the access keys...aws: error: argument command: Invalid choice, valid choices are:accessanalyzer                           | accountacm                                      | acm-pcaalexaforbusiness                         | ampand then this one when I tried to run your backup command.warning: Skipping file /Users/toddmccarty/.Trash. File/Directory is not readable.fatal error: Unable to locate credentialsI think it would take a professional to set this up on my machine. I thank you for your time, but the $9.99 p/month from Apple is sounding much better now, haha.I wish Amazon had a simple cloud backup product. It's surprising that they don't get into the mass market like Apple and Google.  Are you positive one doesn't exist?TCommentSharerePost-User-8702646answered 2 months agokentrad EXPERT2 months agoWhen you install the access keys, run 'aws configure'. It will prompt you for the values. Once that is setup correctly, you can run something like 'aws s3 ls' to list your buckets.Sharekentrad EXPERT2 months agoI have not checked, but I suspect there are many third party vendors that offer backup solutions and use S3 as the offsite storage. But then you are back to paying for and installing additional software.Share0Thanks again for trying to help.I wonder if instead of trying to automate it, I could just do a direct upload of my entire hard drive to my Amazon S3 bucket?  I might give that a try in the meantime.If it works, I could just set myself a reminder twice a month to do it.TCommentSharerePost-User-8702646answered 2 months ago"


Question: Route53 DNS Hosting Stopped working

"On January 18th, my domain Wampei.com - hosted on Route53 and with a valid domain registration there - stopped working. I noticed because my email stopped getting new emails, then I noticed my website stopped.Route53 doesn't seem to think anything is wrong and when I query in the admin tool there, its fine.When I do an nslookup anywhere else:nslookup wampei.comServer:192.168.1.1Address:192.168.1.1#53** server can't find wampei.com: SERVFAILThe whois information is correct, but any request to the zone fails.How can I get this fixed?ThanksRCFollowComment"

Answer: "1Accepted AnswerYou don't appear to have any Name Servers in Route53 for wampei.com. While whois may display the Name Servers, I wonder if you've actually removed those NS entries somehow from Route53, or the addresses for them have changed.https://mxtoolbox.com/SuperTool.aspx?action=mx%3awampei.com&run=toolpageCommentSharesmcstewartanswered 4 months agorePost-User-4069786 4 months agoIt turns out the as smcstewart pointed out the hosted domain and the dns records don't match. Why this only started this month is beyond me.I was able to update the domain registration with the hosted zone and BOOM! Its working.In summary -If you have a situation where you have a hosted zone and a domain registration on AWS and it stops working. Make sure the Domain Name Server entries match the hosted zone entires exactly.Share0It turns out the as smcstewart pointed out the hosted domain and the dns records don't match.  Why this only started this month is beyond me.I was able to update the domain registration with the hosted zone and BOOM! Its working.In summary -If you have a situation where you have a hosted zone and a domain registration on AWS and it stops working.  Make sure the Domain Name Server entries match the hosted zone entires exactly.CommentSharerePost-User-4069786answered 4 months ago"


Question: DMA throughput of cl_dram_dma example

"Hello everyone!I have a small question regarding to DMA throughput of cl_dram_dma example with pre-generated afi (aws v1.4.17). The question is in a way duplicate of this one(https://forums.aws.amazon.com/thread.jspa?messageID=851546﹚). Yet, since it has been a while since this thread is posted. I wanted to formulate another one. If it is against the forum guidelines, I can reply to that thread too (Sorry for inconvenience).I have been running some tests with the cl_dram_dma example. I measured that the read throughput of ~4 GB/s and write throughput of ~1.6 GB/s.  In that discussion, it is stated that the throughput can be increased upto 10 GB/s. I measure the throughput in c program, simply measuring the execution times of read and write.What am I configuring incorrectly in here? Is this the most I can achieve? I have other applications having the same configurations with the cl_dram_dma example. I am trying to obtain best I can achieve this example so that I can estimate it for my own kernels. Also, I have seen the file cl_test.sv and read that its a traffic generator. Is there a way to configure kernel and use that module for more reliable tests?Thanks in advance!Edited by: yonselyuksel on Mar 5, 2021 8:45 AMFollowComment"

Answer: "0Hi,We bench marked DMA performance using cl_dram_dma example and FIO tools. Do you see poor performance when using FIO tests?Please refer to the following guidelines to run benchmark test using the CL_DRAM_DMA AFI provided by AWS:https://github.com/aws/aws-fpga/blob/master/sdk/tests/fio_dma_tools/README.mdAdditional items to consider:Please ensure the CL_DRAM_DMA is running at 250 MHz. (command to load AFI @250MHz : fpga-load-local-image -S 0 -I agfi-02f141212beac0cfb -F -a 250)Please use F1.16xL Instance type to achieve best performance.Yes, https://github.com/aws/aws-fpga/blob/master/hdk/cl/examples/cl_dram_dma/design/cl_tst.sv is basically an AXI4 Traffic Generator, connected to the four DDRs and PCIM Interface in the CL_DRAM_DMA example design. Note, FIO tools cannot be used with this Traffic Generator for performance measurements.Please let us know if you have more questions.Thanks!ChakraCommentShareawschakraanswered 2 years ago0Hi,Indeed, I was able to get best results with fio. I have observed that best performance achievable is with options ioengine=psync, iomem=mmap ( as specified in the example setup). My target instance is f1.2xlarge. In case I want to achieve as good performance as fio, would it be enough to use this kind of configuration? Would you recommend any other extra step. (lets say my file sizes are 16GB and less). I have seen that fio dispatches multiple jobs in parallel. Is there any other optimizations, which is not visible to the developer, offered other than running multiple jobs?I see that my questions are more application specific. I am only trying to understand fio.Also, I could not see the example agfi, which you have sent in the reply thread. Is it the same with the one(cl_dram_dma) in the aws-fpga repo?Thanks a lot!Edited by: yonselyuksel on Mar 8, 2021 1:37 AMEdited by: yonselyuksel on Mar 8, 2021 5:38 AMCommentShareyonselyukselanswered 2 years ago0Hi,The XDMA (DMA Engine inside AWS Shell) supports 4 channels for Data Transfers between Host and FPGA. You will get better performance when all four channels are engaged. FIO tool also leverages this feature. Example :https://github.com/aws/aws-fpga/blob/master/sdk/tests/fio_dma_tools/README.md#what-are-the-fio-config-file-naming-conventionsTypically, larger block size helps in achieving max performance since there will be less overhead for data transfers.F1.2xL may give you max performance at times, but is not guaranteed. We therefore recommend using F1.16xL for performance critical applications.Sorry my bad, I think I shared older AGFI in my previous reply. The latest AFI for CL_DRAM_DMA is noted here:https://github.com/aws/aws-fpga/tree/master/hdk/cl/examples/cl_dram_dma#dram-dma-example-metadataPre-generated AFI ID :afi-063e6afe717a22158Pre-generated AGFI ID: agfi-0b5c35827af676702Thanks!ChakraCommentShareawschakraanswered 2 years ago0Thanks a lot!I am also trying to visualize the affect of memory alignment and have a small question regarding to it:The documentation of fio states that, using iodepth the alignment is given by blocksize. Now, I am running block sizes which are multiple of page size. In order to see the affect of mem_align option, should I delete iodepth option from my .fio file? (I declare io_memalign under [global]) Right now, even though I change the mem_align, I do not see any affect of it.CommentShareyonselyukselanswered 2 years ago0hello,Unfortunately, we do not have data on performance effects due to various combinations of parameters offered by FIO tool. We achieved best possible performance on CL_DRAM_DMA using the FIO script provided in:https://github.com/aws/aws-fpga/blob/master/sdk/tests/fio_dma_tools/scripts/xdma_4-ch_4-1M_read.fiohttps://github.com/aws/aws-fpga/blob/master/sdk/tests/fio_dma_tools/scripts/xdma_4-ch_4-1M_write.fioThe following link describes the parameters supported by FIO:https://github.com/axboe/fio/blob/master/HOWTO#L1762Please contact us if you need any other details.Thanks!ChakraCommentShareawschakraanswered 2 years ago0Thanks a lot,Indeed, fio gives enough coverage in my case so that I can analyze different configurations.My question is resolved.CommentShareyonselyukselanswered 2 years ago"


Question: AWS DMS - CDC - Unique Constraint Error

"How to resolve "ORA-00001: unique constraint" during AWS DMS CDC mode ?Source - On Prem - Oracle || Target - AWS RDS - OracleI exported source DB with Datapump with SCN, imported same on target and started migration task with same SCN but I am getting error "ORA-00001: unique constraint" , because of this migration task is slow and change mode to "one-by-one" from "Bulk Apply".I am getting above error for table which has more than 100 millions of records. How can I get rid of this error ? Do I need to change anything before I take export?FollowComment"

Answer: "1Accepted AnswerYou can divide the tables into multiple tasks to improve the CDC performance. You can review the "awsdms_apply_exceptions" in your target database schema for more information on the violations. Increasing the logging will give you more information on the why the violations are occurring (duplicated data in batch transactions before applying the commit vs real violations on the keys).https://aws.amazon.com/premiumsupport/knowledge-center/dms-enable-debug-logging/Further you can also enable BatchApplyEnabled and disable (BatchApplyPreserveTransaction) please refer below link for more information:https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Oracle.html#CHAP_Target.Oracle.ConnectionAttribAWS DMS uses the following methods to replicate data in the change data capture (CDC) phase:Transactional applyBatch applyThe AWS DMS CDC process is single threaded, by default (transactional apply). This is the same method used for SQL replication as by all other online transactional processing (OLTP) database engines. DMS CDC replication is dependent on the source database transaction logs. During the ongoing replication phase, DMS applies changes using a transactional apply method, as follows:DMS reads changes from the transaction log, from the source into the replication DB instance memory.DMS translates changes and then passes them on to a sorter component.The sorter component sorts transactions in commit order, and then forwards them to the target, sequentially.If the rate of change is high on the source DB, this process can take time. You might observe a spike in CDC target latency metrics when DMS receives high incoming workload from source DB.DMS uses a single threaded replication method to process the CDC changes. DMS provides the task level setting BatchApplyEnabled to quickly process changes on a target using batches. BatchApplyEnabled is useful if you have high workload on the source DB, and a task with high target CDC latency. By default, DMS disables BatchApplySetting. You can enable this using AWS Command Line Interface (AWS CLI).https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.ChangeProcessingTuning.htmlIf you run a task with BatchApplyEnabled, DMS processes changes in the following way:DMS collects the changes in batch from the source DB transaction logs.DMS creates a table called the net changes table, with all changes from the batch.This table resides in the memory of the replication DB instance, and is passed on to the target DB instance.DMS applies a net changes algorithm that nets out all changes from the net changes table to actual target table.For example, if you run a DMS task with BatchApplyEnabled, and you have a new row insert, ten updates to that row, and a delete for that row in a single batch, then DMS nets out all these transactions and doesn’t carry them over. It does this because the row is eventually deleted and no longer exists. This process reduces the number of actual transactions that are applied on the target.BatchApplyEnabled applies the net changes algorithm in row level of a table within a batch of a particular task. So, if the source database has frequent changes (update, delete, and insert) or a combination of those workloads on the same rows, you can then get optimal use from the BatchApplyEnabled. This minimizes the changes to be applied to the target. If the collected batch is unique in changes (update/delete/insert changes for different row records), then the net change table algorithm process can't filter any events. As a result, all batch events are applied on the target in batch mode. Tables must have either a primary key or a unique key for batch apply to work.DMS also provides the BatchApplyPreserveTransaction setting for change-processing tuning. If you enable BatchApplySetting, then BatchApplyPreserveTransaction turns on, by default. If you set it to true, then transactional integrity is preserved. A batch is guaranteed to contain all the changes within a transaction from the source. This setting applies only to Oracle target endpoints.When the BatchApplyPreserveTransaction setting is true, DMS captures the entire long-running transaction in the memory of the replication DB instance. It does this according to the task settings MemoryLimitTotal and MemoryKeepTime, and swaps as needed, before it sends changes to the net changes table. When the BatchApplyPreserveTransaction setting is false, changes from a single transaction can span across multiple batches. This can lead to data loss when partially applied, for example, due to target database unavailability.[+] https://aws.amazon.com/blogs/database/debugging-your-aws-dms-migrations-what-to-do-when-things-go-wrong-part-2/[+] https://aws.amazon.com/blogs/database/debugging-your-aws-dms-migrations-what-to-do-when-things-go-wrong-part-3/You can use batch apply in the following circumstances:The task has a high number of transactions captured from the source and this is causing target latency.The task has a workload from source that is a combination of insert, update, and delete on the same rows.No requirement to keep strict referential integrity on the target (disabled FKs).When BatchApplyEnabled is set to true, AWS DMS generates an error message if a target table has a unique constraint.As such, I would recommend to enable Batch apply to 'true' and increasing BatchApplyTimeoutMin to ~300 seconds (5 minutes) and BatchApplyTimeoutMax to 600 seconds (10 minutes) and increasing BatchApplyMemoryLimit (depending on your current free memory on your RI).This means that DMS will only send a batch to target every 10 minutes, or once BatchApplyMemoryLimit of transactions have been captured on the RI. This should positively influence the target latency, by allowing larger, less frequent commits to take place. In order to alter this setting, stop the task, modify the setting, save the changes and resume the task (can test on test environment first to monitor change in case you need to further increase BatchApplyMemoryLimit).Only once the amount of time in BatchApplyTimeoutMin has passed, will DMS assess BatchApplyTimeoutMax and BatchApplyMemoryLimit. Once one of the values for BatchApplyTimeoutMax or BatchApplyMemoryLimit is reached, the batch will be sent to the target. So to explain it differently, once BatchApplyTimeoutMin has passed, whichever comes first between BatchApplyTimeoutMax and BatchApplyMemoryLimit will trigger DMS to send the batch to the target.https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.ChangeProcessingTuning.htmlCommentShareSUPPORT ENGINEERsuveshan_panswered 7 months ago"