Question: Best way to Read multiple files from S3 in paralell with .net

"I have a lot of xml files in S3(more 1,2 Million). They are in 12 folders (1 for month) with about 100.000 files in each folder. Each file is little size, about 6k-7kI have to read each file, parse the fields in the xml and call an external API Rest with this information. (I dont need download the files to local, only read them)What is the best way to do it? I try it sequentially, but it is very slowly.I also tried to put the file in an array and use paralell.for, but is slow too.I don't count the time to call API. I see that in read and parse the file it about 10 files per second.(arrayficheros has 1000 elements at max, only get about 6-7Mb of ram) using (var stream = await (client.GetObjectStreamAsync(parametros.bucketName, responseMetadatos.S3Objects[x].Key, null)))                            {                                                             arrayficheros[x] = new StreamReader(stream).ReadToEnd();                            }         -------Parallel.For(0, arrayficheros.length,                                 new ParallelOptions { MaxDegreeOfParallelism = 6 },                                  x =>                                  {                                    DoWork( arrayficheros[x] );                                  });ThanksFollowCommentHelge Mahrt a year agoIs this supposed to be a one-time thing, or do you need a reusable solution?Where are you running this code? On your machine or on some cloud service?Where do you see the bottleneck? I.e. is your CPU at 100%, or is your bandwidth maxed out?How many requests per second can the target API handle?You can approach this from different angles. You can either try to optimize your code (there's room for improvement) or you can scale out (e.g. by leveraging Lambda)Share"

Answer: "1This is a good scenario for partitioning to have horizontal scale and parallel processing. Consider partitioning per folder that corresponds each month, or if it is possible you can even consider partitioning per month and day example files from: Jan 01-10, Jan 11-20, Jan 21-31, Feb 01-10 and so on… And each partition to be processed by one instance, You can run this on Amazon ECS as Fargate Task. In your C# logic consider implementing Error retries and exponential backoff since there will be a lot of external API calls, read the post Exponential Backoff And Jitter to learn more about the benefits of Exponential Backoff.CommentShareUlili Nhagaanswered a year ago1Note that you can send up to 5,500 GET requests per second per prefix in an Amazon S3 bucket. Considering you have approx. 100,000 files in each of the 12 folders and you can read from those folders in parallel, you should be able to send all those GET requests in approx. 20 seconds. However, the network latency as well as the time needed to transfer the actual file content will increase the overall time needed. This will also be affected by the actual processing duration of your files (calling external APIs etc.)To process the content of an object (file) stored in Amazon S3, you need to download that object. You could also read only a portion of the object data if you know the exact byte offsets of interest. However, for text-based file formats like XML, it is probably better to download the whole object, because the XML parsers could fail processing partial content.To improve overall performance, you could parallelize the file downloading and the data processing. Consider the producer-consumer pattern here. You would have multiple producers (downloaders) and multiple consumers (data processors). I assume you can parallelize the data processing, and the files are independent of each other and don't have any particular processing order requirements.If you are not limited on storage, you could use the Amazon.S3.Transfer.TransferUtility class to download your files in parallel. This utility will handle all the connectivity for you (like e.g. retrying on failure). You can specify the number of parallel requests for downloading:using Amazon.S3;using Amazon.S3.Transfer;using var client = new AmazonS3Client();// 100 requests in parallel - adjust as neededvar config = new TransferUtilityConfig { ConcurrentServiceRequests = 100 };using var utility = new TransferUtility(client, config);var request = new TransferUtilityDownloadDirectoryRequest{    DownloadFilesConcurrently = true,    BucketName = "YourBucket", // your bucket name here    LocalDirectory = "/your/path/", // the path where to store the downloaded files    S3Directory = "prefix-by-month" // the bucket prefix (folder) for a month};await utility.DownloadDirectoryAsync(request);You can construct 12 requests for your 12 prefixes and download them all in parallel (12 * 100 parallel requests). Something like this:var prefixes = new List<string>(); // 12 prefixes (folders) herevar requests = prefixes.Select(    p => new TransferUtilityDownloadDirectoryRequest { S3Directory = p, /* ... */});var downloadTasks = requests.Select(r => utility.DownloadDirectoryAsync(r));// wait until all 12 tasks complete in parallelawait Task.WhenAll(downloadTasks);Now you can either wait until all the files will be downloaded, or subscribe to the request.DownloadedDirectoryProgressEvent events and process the files as they are arriving.Note that you are limited in the number of thread pool threads, so you can only execute a limited number of requests in parallel. You can obtain these limits by calling ThreadPool.GetMaxThreads(out var workerThreads, out var portThreads).If you don't want or cannot use storage as a cache for the files, you can implement your producer-consumer data flow (e.g. via System.Threading.Tasks.Dataflow) using the AWS SDK GetObjectStreamAsync method. However, you should also implement all the necessary details:retrying when the call failsretrying when the request throttling occurschunk-wise reading from the stream obtained by GetObjectStreamAsync (the stream might contain only partial file, so you would need to read the stream multiple times as new data gets transferred)You can also use the source code of the Amazon.S3.Transfer.TransferUtility class as an inspiration and create your own utility that gets the objects from an S3 bucket in parallel but doesn't store them into files but rather pushes them into an in-memory queue for further processing. The source code can be found on GitHub.CommentShareDmitry Balabanovanswered a year ago0What you have with Parallel.For seems like the optimal way to do this to me. Could it simply be a bandwidth issue? One way to test this would be to spin up an EC2 machine in the same region as the bucket, add some console.writelines, and try the same code on it really quick.CommentShareStevenSanswered a year ago"


Question: Join Tables in AWS

"I want to join two tables.I have the tables in CSV format stored in S3 bucket1.Is Amazon Glue studio,the right option?2.What is the correct procedure?3.What are the IAM permissions required?4.Where to see the joined table output?Please throw some lightFollowCommentSanjeeb 3 months agoI believe you are looking for a solution to join 2 tables where the data files are in csv. You can follow the below solutions.Using Athena, you can create tables directly pointing to csv files. Athena supported CSV files. Then you can do the join and create a final table and point to S3 as well.Using Glue, you can create crawlers and then run the crawler and it will populate the meta data for those tables and you can see the tables in Athena as well and do the join in Athena.Using Glue crawler, create the meta data, using Glue studio, create a job which will do join and complex transformation ( if you have) and the final data set can be put it in S3 and crawled and see the data in Athena as well.Glue studio is having GUI based development environment and it is easy to use. Hope the above information helped you.Share"

Answer: "0Hello,You can use Amazon Athena to query the csv data from S3 buckets by creating external tables. Keep the csv files belongs to two tables in two separate folders in S3. Then you can create a database / reuse exiting in Athena console and create tables from these csv files.Alternatively, you can combine AWS Glue and Athena together to achieve this. You can create a crawler in Glue and add data source pointing to S3 buckets and run the crawler which will create the tables automatically for you. These tables will then appear in the Athena console and you can query them using SQLs and join.Please refer to below where similar use case explained.https://stackoverflow.com/questions/73864748/query-to-multiple-csv-fles-at-s3-through-athenahttps://stackoverflow.com/questions/74009011/access-s3-csv-file-in-amazon-athenaPlease refer here for IAM for policies needed for accessing data catalogs. Also, you can use the AmazonAthenaFullAccess AWS managed policy assigned.Additional resources for your reference :https://docs.aws.amazon.com/athena/latest/ug/data-sources-glue.htmlhttps://docs.aws.amazon.com/athena/latest/ug/glue-best-practices.htmlCommentSharegokul_Ganswered 3 months ago0Athena will store the query result on the temporary S3 path, you can also ask Athena to create a table with the result of the query.https://docs.aws.amazon.com/athena/latest/ug/ctas.htmlIf you use Glue Studio you would need to output the result of the join to some sink such an s3 path or table, but if you just want to do the join without needing extra transformations of connections, Athena is probably more practical if you are familiar with SQLCommentShareGonzalo Herrerosanswered 3 months ago"


Question: cognito pre-signup

HelloI try to put a condition to only allow email addresses from my own domain to sign up for the cognito user pool.  I tried the pre-signup trigger and a lambda function to check the information passed;however I was able to automatically confirm users with email addresses from my own domain but other domain email addresses were still accepted as unconfirmed.Is there a way to stop the processes if the email provided is from a different domain?thanksRTFollowComment

Answer: "0I found out what i needed.On the else section of my code when the domain does not match, I needed to return the callback with the error.CommentSharerttanswered 3 years ago"


Question: "Activity has unexpected exception, because: no implicit conversion of nil"

"Most times I deploy, I get this error and deployment fails. Yesterday when I usde the 'application versions' tab to deploy the deployment worked as expected.I'm using the latest golang platform version. I usually have to manually reboot and re-register these failed instances with the load balancers to try again (otherwise my entire environment is red). Yesterday 'application versions' deployments worked without this error ... I'm not sure why they'd break again today.Looks to be a bug in the platform.[2020-12-07T20:57:01.145Z] INFO  [29598] - [Application update app-3898-201207_124042@325/AppDeployStage0/EbExtensionPreBuild] : Starting activity...[2020-12-07T20:57:11.948Z] INFO  [29598] - [Application update app-3898-201207_124042@325/AppDeployStage0/EbExtensionPreBuild] : Activity has unexpected exception, because: no implicit conversion of nilinto String (TypeError)at /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/cfn-wrapper.rb:117:in +'           from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticb   eanstalk/cfn-wrapper.rb:117:in recompile_exception'from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/cfn-wrapper.rb:53:in run_config_sets'           from infra-embedded_prebuild.rb:20:in block in execute!'from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/executable.rb:61:in instance_eval'           from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticb   eanstalk/executable.rb:61:in block in execute!'from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/executable.rb:60:in open'           from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticb   eanstalk/executable.rb:60:in execute!'from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command.rb:211:in block (4 levels) in exec_stage'           from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticb   eanstalk/command.rb:211:in chdir'from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command.rb:211:in block (3 levels) in exec_stage'           from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticb   eanstalk/activity.rb:195:in call'from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/activity.rb:195:in exec'           from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/activity.rb:140:in timeout_exec'from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/activity.rb:123:in block in create'           from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/activity.rb:122:in instance_eval'from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/activity.rb:122:in create'           from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command.rb:208:in block (2 levels) in exec_stage'from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command.rb:206:in each'           from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command.rb:206:in each_with_index'from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/command.rb:206:in block in exec_stage'           from /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/activity.rb:195:in call'Edited by: fdsa on Dec 9, 2020 2:53 PMFollowComment"

Answer: "0I run into this several times a day at this point. I suspect there's some corner case in my configuration that triggers this bug but given the non-descript log and the lib not being open source I can't troubleshoot.CommentSharefdsaanswered 2 years ago0The stack trace references what looks like a private gem. It looks like the eb is calling some script during prebuild that expects an argument "$1" and it doesn't get it (so it's nil) but attempts to concat it to a string. After calling .to_s on $1 to fix, I uncovered a race condition on "failed_key." This block is entered so failed_key must initially be truthy, yet when it's referenced in the block it's nil as well. I locally patched the gem on my instances to check if failed_key is truthy before using it like so:            case failure                  when failed_key && @@cfn_cmd_failed_regex                      name = prebuild ? "command #{$1}" : "container_command #{$1}"                      ret = ExternalInvocationError.new(                          reason: ebext_file_name ? "#{name} in #{ebext_file_name} failed." : "EBExtension #{name} failed.",                          output: failed_key && 'Command ' + $1.to_s == failed_key.name ? failed_key.output : '',                          exit_code: !failed_key || failed_key.exit_code.nil? ? 1 : failed_key.exit_code                      )  It'd be lovely if I didn't have to do this. The only ebextensions I have are 2 yum installs (via "packages:"), a "sources:", "Resources" to setup s3 auth, and a "files" section for 2 files. None of those extensions error though. I also have some nginx conf but this seems like something in the yaml stuff.CommentSharefdsaanswered 2 years ago0The issue was memory was running out during some internal eb task as I was on micro instances but it was getting reported in this weird way. This also explains why rebooting would resolve the issue. I was able to find this in the cfn logs.CommentSharefdsaanswered 2 years ago"


Question: From Oracle on premise to S3 using DMS

"Hi Folks, is there a way to ingest  some tables from Oracle on premise to S3 passing by DMS?, like a migration but only some tables (maintaining the DMS change data capture   etc), thanksFollowComment"

Answer: "0It is possible to filter tables as described in the following documentation.Table mapping can be used to filter tables that need to be converted or migrated.https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.Console.htmlDatabase migration should be done carefully and should be verified many times. (e.g., whether the necessary data has been securely migrated, whether it works, etc.)CommentShareEXPERTRiku_Kobayashianswered 11 days ago"


Question: Account blocked from creating free-tier EC2 instances

"My account appears to be blocked from creating free-tier EC2 instances. I've tried this with several different settings and from the free, US-East zones; the same account block message persists. I emailed aws-verification@amazon.com but have yet to receive any feedback. In reviewing my account from admin and root, there are no indications my account is blocked. Any insight and advice to resolve is greatly appreciated.FollowComment"


Question: View live stream 404

"We have started a live stream to ivs and the stream is pushed success. But when we view the live stream, it shows 404. We check the live channels and there are no live channels. We retry to push the stream and the issue still exists.The issue appeared from about 5/30, it disappeared on 6/02 and appears 6/03 (CST).ivs region: us-east-1.What's possible reason?FollowComment"

Answer: "0The 404 error indicates that the playback URL is not available, which indicates that the stream is either offline or does not exist (https://aws.github.io/amazon-ivs-player-docs/1.3.1/web/enums/errortype.html#not_available). This can be caused by multiple different reasons, however, the two most likely are that the playback URL is being requested before the stream is available or the stream did not start successfully.There is a slight delay between when the “Start Stream” button is pressed by the broadcaster and when the content is available for viewing. One potential reason for the error being seen, could be that the playback URL is requested for the channel before the content is available, which will result in a 404 error. To ensure the content is available for playback, please confirm the Amazon EventBridge “Stream Start” event (https://docs.aws.amazon.com/ivs/latest/userguide/eventbridge.html#eventbridge-examples-stream-state-change) is produced before requesting the playback URL. To help reduce the stream-start time, please refer to the following recommended encoder configuration:https://docs.aws.amazon.com/ivs/latest/userguide/streaming-config.html#streaming-config-reducing-latencyWhile a stream may be started by the broadcaster, if the stream exceeds the channel input limits, the stream will be disconnected by IVS and an Amazon EventBridge limit breach event will be produced (https://docs.aws.amazon.com/ivs/latest/userguide/eventbridge.html#eventbridge-examples-limit-breach). This would mean that the stream did not successfully start, thus causing a 404 error to be produced as the stream would not be online.If further investigation is required, please reach out to AWS Support, as then we will be able to look into the specific channel details and what may be causing the 404 error to occur during playback.CommentShareriley-awsanswered 2 years ago0Thanks for your answer.According to your description, we have found our problem. We really failed to push the live stream. Now we can view it successfully.CommentShareakataanswered 2 years ago"


Question: install code deploy agent in disabled mode(pre-baking in AMI)

"I would like to install code-deploy-agent on windows in disabled mode but I couldn't find any argument in the MSI package. I tried to search on the registry for a specific flag regarding registration.We are trying to install code-deploy agent in an AMI but after the ec2 instance is created it complains about the instance-id.Set-Service : Service 'CodeDeploy Host Agent Service (codedeployagent)' cannot be started due to the following error: Cannot start service codedeployagent on computer '.'.At C:\Windows\system32\config\systemprofile\AppData\Local\Temp\Amazon\EC2-Windows\Launch\InvokeUserData\UserScript.ps1:35 char:1+ Set-Service -Name 'codedeployagent' -StartupType 'automatic' -Status  ...+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~    + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Set-Service], ServiceCommandException    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.SetServiceCommandSet-Service : Service 'CodeDeploy Host Agent Service (codedeployagent)' cannot be started due to the following error:Cannot start service codedeployagent on computer '.'.AtC:\Windows\system32\config\systemprofile\AppData\Local\Temp\Amazon\EC2-Windows\Launch\InvokeUserData\UserScript.ps1:35char:1+ Set-Service -Name 'codedeployagent' -StartupType 'automatic' -Status  ...+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~    + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Set-Service], Se   rviceCommandException    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.SetServiceCommandFollowComment"


Question: How do you install dnf on Amazon Linux 2?

"I am trying to run Ansible using python 3 on Amazon Linux 2 but dnf is required to install OS packages.$ ansible-playbook -v playbook.yml  No config file found; using defaults   \[WARNING]: No inventory was parsed, only implicit localhost is available     \[WARNING]: provided hosts list is empty, only localhost is available. Note that the  implicit localhost does not match 'all'      PLAY \[localhost] ***********************************************************************    TASK \[Gathering Facts] *****************************************************************  ok: \[localhost]    TASK \[Install prerequisites packages] **************************************************  fatal: \[localhost]: FAILED! => {"changed": false, "msg": "The Python 2 bindings for rpm are needed for this module. If you require Python 3 support use the `dnf` Ansible module instead.. The Python 2 yum module is needed for this module. If you require Python 3 support use the `dnf` Ansible module instead."}    PLAY RECAP *****************************************************************************  localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0  How does one install dnf in Amazon Linux 2?  Is there another workaround beside using python 2?FollowComment"

Answer: "0Right now Amazon Linux 2 does not support dnf. If you require support for rpm and yum in ansible, you will need to run ansible with python 2.7.-HeathCommentShareAWS-User-2400882answered 4 years ago"


Question: How implement Flexmatch in an event-based way?

"Hi,in the tutorials I found the matchmaking is always done with a long-polling lambda request and within the lambda function there is short-polling against the flexmatch-service, to retrieve the state of current player matchmaking.Since this should not be done in production, I tried to implement it in an event-based way. But how?In my case the game should run on different devices/platforms like Android/Windows. The matchmaking does not have to be confirmed by the player -also it should be fast, cheap & scalable.If I don't want to make short-polling on the player-client (like calling lambda every second, which would be very expensive I guess), I have only 2 options with aws afaik, to get the ip-address+port from flexmatch:Long polling with lambda (and an event-based strategy within lambda)Using push notification service, to make usage of e.g. SNS (which would be connected to flexmatch)I liked the 1. way, because it seems to be the simplest solution. But I have no idea how to get data from a queue.E.g. I cannot subscribe within lambda-function to SNS. I would still be fine with like a every-second short-polling, but it seems I only have write-access there (yes, you can let lambda functions getting executed by SNS, but this does not help me here)Maybe emitting SNS messages to SQS, since I can read SQS within lambda. But I cannot filter SQS, so every waiting player lambda-function would get "all" flex-match messages. Can't imagine that this would be the right way.Then there is also this EVENT-BRIDGE service, but it seems I also cannot not read from it within lambda.Then there would be only the 2nd solution.I could imagine the workflow, like player-client makes a lambda-call, which will start matchmaking and returns the ticket-id. Then the player-client would try to connect to SNS via push notification service and provides the ticket-id as filter for SNS.But this seems to be very complex, since you need first to catch a token (for app+device) to establish a connection.. 😕Is there maybe another way? Or how was it done by others? I would be really thankful for any help!FollowComment"

Answer: "0Accepted AnswerHey Terrek7,Please take a look at the GameLift developer's guide: https://aws.amazon.com/blogs/gametech/a-guide-to-amazon-gamelift-game-servers/, which has the answers that you need with reference architectures in sections:"Standalone Game Session Servers with a Serverless Backend": Flexmatch SNS topic triggers a lambda to store tickets into dynamodb, and separate client calls to fetch ticket status would poll this table for ticket status updates."Standalone Game Session Servers with WebSockets-based Backend": Flexmatch SNS topic triggers a lambda to call game clients directly via ApiGateway WebsocketCommentSharerePost-User-7108006answered 2 years ago0Thanks JamesM_Aws!Actual I could not find your 2 mentioned points in your link (not even in the ebook).But nonetheless, that helped me already a lot!But just one additional question regarding SNS:I created a SNS topic with a lambda target and logged the events in lambda.And when 2 players were requesting for matchmaking, I got actual "3" message of type "MatchmakingSucceeded" for the same match (same matchId).Is this on purpose? I mean one would be enough, and since I have to pay for each lambda-call this is a bit annoying.CommentSharerePost-User-5601185answered 2 years ago0Hmm, that's strange, is it consistently reproducible? If so, could you PM me more information (e.g. region, account id, matchmaking configuration name, timestamp, match id, ticket ids, etc.) for us to investigate?Alternatively, FlexMatch also emit messages to CloudWatch Event, perhaps try setting up a lambda to listen to CW events to see if the issue persists? If so, please let us know.Another approach is to put SQS in between your SNS and Lambda (i.e. SNS -> SQS -> Lambda), the benefits are:You are able to batch process multiple records with a single lambda execution (https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#events-sqs-eventsource), you could potentially do some deduping there.You are able to reprocess failed records (however, game sessions are short-lived so this feature may not be that useful)CommentSharerePost-User-7108006answered 2 years ago0Hey JammesM_Aws,thanks again and thanks for your offer to help!Actual the same happened also with a CloudWatch Event, 3 instead of 1 message.But I could figure out to resolve the problem. I had an error in my lambda function, which caused it to run ~3000ms. It seems when there is no return to the sns/event within a second, then it will trigger again a lambda function.So as soon as the lambda problem was solved and it was running <1000ms, the problem was gone.Then I came to see that I can not filter the messages of SNS coming from GameLift, because there were no "MessageAttributes" on it. The whole MatchmakingMessage was put stringified in "message". And as I understood, you can only filter for "MessageAttributes".But at least CloudWatch Event had the option to filter for the messages I needed (MatchmakingSucceeded, MatchmakingTimedOut, MatchmakingFailed), so I used this instead (higher latency of ~500ms, instead of ~30ms with sns, but it's still fine)So everything is working now, thanks again for the help!Terrek 7CommentSharerePost-User-5601185answered 2 years ago0Glad you fixed the issue! I have submitted a feature request to add MessageAttributes to Flexmatch/GameSessionQueue SNS messages.For other folks in the future, please +1 on this if you also want this feature, so you could help us prioritize, thanks!GLIFT-14918CommentSharerePost-User-7108006answered 2 years ago"


Question: DynamoDB - Indexing a list attribute

"I have a dataset of call recording information, with fields/attributes including: Incoming Number, Dialled Number, Call Start Date, Agent(s) name.  The end user will most likely want to search on Agent name, which is stored in a list attribute as there might be more than one. I can't add an index to a list attribute, so as a SQL dev I'm learning towards storing the agent ID in a different table and creating a many to many relationship between the two tables.  However, Dynamo's a bit different so what's my best approach if I want to index the Agents attribute?FollowComment"

Answer: "1You're correct that you would not normally create "join" scenarios for DynamoDB. Typically you would have an individual item for each Agent:PKSKotherother-101202333444C#012023334441/1/2023Call metadata01202333444Bob1/1/2023Data01202333444Michelle1/1/2023Data01202333444Lee1/1/2023DataIn DynamoDB and other NoSQL databases its common to de-normalize the data, which takes a little bit of getting used to coming from a relational database background.CommentShareEXPERTLeeroy Hannigananswered 5 months agoNick 5 months agoThanks LeeroyI could denormalise to the following:Incoming Number    StartTime    PrimaryAgent    SecondaryAgent01202333444           1/1/2023    Bob                     Michellebut then, what happens if there's another agent?ShareLeeroy Hannigan EXPERT5 months agoI edited my answer to show how I would model the dataShareNick 5 months agoso, if I do a search for 01202333444, I'd get one record for each of the agents involved in the call, rather than just one record for the call?ShareLeeroy Hannigan EXPERT5 months agoYes and you would also get the call metadata. The cost of reading does not change as you still pay up to the nearest 4KB.ShareNick 5 months agonot sure I like that, to be honest. A user would see two calls where there are actually only one, and this would confuse them. Thanks for your help though, LeeroyShareShow 1 more comment"


Question: Delete media convert job

"Hello. I'm a beginner in aws services.Please, how can I delete created jobs with aws media convert using the aws console? the jobs are created in the default queue. when I select a job, the cancel button is not activated.FollowComment"

Answer: "1Accepted AnswerHi there,There is no way to delete jobs that have ran and are in a completed, errored, or cancelled stated. The service will display these jobs for 90 days [1], and then remove them from the console view.Thanks,Jeremy== Resources ==[1] https://docs.aws.amazon.com/mediaconvert/latest/ug/setting-up-resources-for-catt.htmlCommentShareAWS-User-8506763answered 2 years ago0HiDid you follow this document to cancel the job? https://docs.aws.amazon.com/mediaconvert/latest/ug/canceling-a-job.html.If yes, and you still unable to cancel the job. Could you check the policy of your user in IAM? For detail, you can read this doc: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.htmlHere is the tool: https://policysim.aws.amazon.com/home/index.jspIf you have this policy only: AWSElementalMediaConvertReadOnly, then you have readyonly access. You need to have AWSElementalMediaConvertFullAccess in order to control the job.RegardsMichaelCommentShareawsmicmicanswered 2 years ago0Thank you for your answerI discover that I hadn't the policy to delete a job. I add AWSElementalMediaConvertFullAccess to the user but that doesn't give me the access to the delete button.May be There's a condition in the status of jobs to be deleted?CommentSharemeriemiaganswered 2 years ago0Thank you very muchCommentSharemeriemiaganswered 2 years ago"


Question: Setup for Second Life

I'm doing research for a customer with the following requirements and I was wondering if this is a use case that is supported by AppStream 2.0Customer has built a Second Life world where members can interact including via voice. This means that the client device should be able to use the microphone to send to the AppStream instance.  I guess a similar use case would be supporting Skype installed in AppStream and that a user accessing the virtual desktop in their browser could make an skype call.Customer wants members to have access to the Second Life experience from devices with lower resources such as tablets and Chromebooks. Second Life requires the installation of a desktop client application that is resource intensive and requires specific graphic cards.  See https://secondlife.com/my/support/system-requirements/?lang=en-USEach user should access to its own virtual desktop.Is the Second Life viewer supported in AppStream? It looks like one of the Graphic instances would be required but which one? My customer has approximately 50 users who could access Second Life throughout the day.  What AppStream setup would be recommended for supporting these 50 users?FollowComment

Answer: "0Hi OscarWe're already doing something like this at vComm (https://www.vcommsolutions.com) - AppStream graphics design large instances can run the client, although performance is better with xLarge and 2xLarge.  Depending upon what it is your customer is planning to do, maybe I can help, or you might even want to consider using our virtual environment AULA :)Kind RegardsNeil CanhamCTO vComm Solutions AGCommentSharefinemananswered 4 years ago0Hello  Oscar Cortes,Any application that is designed to be installed on Windows OS is supported in Appstream 2.0. You will need to follow software requirements to determine which instance to use and might need to install on different Graphics design instances to test which instance gives you best cost/performance value.AppStream 2.0 allows you to automatically scale the size of your fleet to meet user demand. Each running instance in a fleet can be used by only one user at a time, which means the size of your fleet determines the number of users who can stream concurrently.Hope this helps. Please let us know if you have any additional questions.Best regards,LidiyaCommentShareAWS-User-3708156answered 4 years ago0thank you for the explanation!CommentShareOscar Cortesanswered 4 years ago"


Question: "Does Lymberyard team have any plans to replace cryengine old, heavy, and slow renderer with something modern and newest?"

"...for instance replace with Diligent Graphics core?That's is my question.We see how with every new iteration of Ly's engine, ly's killing cryengine's features within itself , but why amazon still keeping support and use CryRenderer in Ly engine? When it's just can use free modern middleware for LyRenderer. What can support more platforms and more graphics api. And it's a lot faster than cryRenderer what was written in ages when dx9 was used, so it's core all based on old graphics paradigm.Thanks, with best regards FluffyFollow"

Answer: "0Hey @REDACTEDUSERSharerePost-User-7556027answered 3 years ago0hey @REDACTEDUSERthere not helped - just "tweak", there need rewrite all, bcoz the renderer's code is mess and it's heavy.They better take a new bare GAPI like Vulkan/DX12 and make own AAA-renderer, if they dont wants use free middleware for some reason.AZN have resources for this, this will helps for Ly a lot.at the beginning own Ly'sRenderer may be like a gem in gems, in future it's just replace cryengine mess at allSharerePost-User-2514050answered 3 years ago0Yes @REDACTEDUSERSharerePost-User-7556027answered 3 years ago0[quote="WashedUpStudios, post:4, topic:7912"]LumberRenderer[/quote]Indeed! haha LumberRenderer :rofl:you are better hurry to register this (c)(tm) signature first, till they thinking about all of this  xdSharerePost-User-2514050answered 3 years ago0just add here few useful links on new hypothetical middleware for the LumberRenderer:https://github.com/DiligentGraphics/DiligentCore#initializationhttps://github.com/azhirnov/FrameGraphSharerePost-User-2514050answered 3 years ago0@REDACTEDUSERSharerePost-User-7556027answered 3 years ago0Hi all, thanks for the feedback and constructive discussion about Lumberyard. These are great suggestions and this sort of feedback from you all, our community, are what helps to drive our plans and roadmap.Currently, we do not have a public roadmap that we can share, but please keep sharing your thoughts and ideas and having discussions like this.SharerePost-User-8353690answered 3 years ago0Will do @REDACTEDUSERSharerePost-User-7556027answered 3 years ago0Great, my first post, greeted by something called a Discobot. Gee that sounds fun but where's the Beegee music?  So, say I was looking to hire a team of guys to work on a game, would I simply want to pay for add ons or should I consider getting something coded for greater efficiency anyway? Thanks!SharerePost-User-7357299answered 3 years ago0Hi @REDACTEDUSERSharerePost-User-7556027answered 3 years ago0I think LY's render team definitely should look on this...This is a graphics framework based on a concept of a render graph: modern way of designing a realtime renderer where you first define what's going to happen in your entire frame and then the engine can figure out on how to execute it optimally, safely and doing as much synchronization heavylifting automatically as possible. For example, if you say that you're going to use a texture as a rendertarget first and then you're going to sample from it, rendergraph can create all needed barriers just from this information.https://github.com/Raikiri/LegitEngineSharerePost-User-2514050answered 3 years ago0Lumberer sounds even better do i think😂SharerePost-User-0769324answered 3 years ago0😂 I mean not bad, love how the topic still gets great people to respond to it! Thanks @REDACTEDUSERSharerePost-User-7556027answered 3 years ago0It seems that you don't know anything about Cryengine. The rendering capabilities of Cryengine is better than any other engine because it is always realtime. It doesn't need any baking to achieve a better lighting for a scene like Unreal and Unity does. That is the reason why Lumberyard will not change its renderer. And the same reason why no matter what happens Cryengine is always one of those game engine with sophisticated technology. Vulkan implementation will surely come in Lumberyard since most of hardware now are inclined to use it. Lumberyard goal is to make the Cryengine technology user-friendly. Because that is the only problem of Cryengine. Cryengine and Lumberyard is already a realtime renderer engine.SharerePost-User-2061097answered 3 years ago0[quote="ralph, post:15, topic:7912"]It seems that you don’t know anything about Cryengine[/quote]oh c'mon? I know this renderer a lot:https://forums.awsgametech.com/t/cel-shading/1239/11https://forums.awsgametech.com/t/upload-please-rim-light-shader-effect-file-for-lumberyard/2580/9Renderer's picture looks outdated, like it still comes to us from 2009 year. And LY's team doing nothing to make it's looks cool and newer.Did you ever work with cry's renderer code? Or shader code? It's just mess. It's seems what you even do not looked into the codeSharerePost-User-2514050answered 3 years ago0You are so funny. The looks of the game depends on the one that uses the engine. You are either an Unreal fanboy or Unity enthusiast. Anything you can do in Unreal or Unity is also possible in Lumberyard and Cryengine. Outdated? It only depends on Graphics API implementation. Sooner it will be Vulkan. You know study things deeply before you talk because it seems to me that you are just talking without a deep understanding of the topic. Because many developers who is knowledgeable with this engine will laugh at you. You think that Star Citizen developer will use outdated game engine in their game? Is that a look of outdated rendering? A little advice. Don't pretend you are genius. It has nothing to do with Lua and Script Canvas. Because the render code of Lumberyard and Cryengine is natively C++. Lua is just for some functionalities of the engine. It is realtime engine. No baking except using Environment Probe. Amazon will not waste money on something that has no cutting edge technology.SharerePost-User-2061097answered 3 years ago0All 3d games use realtime rendering, by definition.I think what you're getting at is that Cryengine uses only dynamic lighting, rather than the pre-baked static lightmaps. That has its pros and cons, depending on what you want to do. Other engines do also have the option of dynamic lighting. Fortnite is fully dynamic, for example.SharerePost-User-5431342answered 3 years ago0[quote="ralph, post:17, topic:7912"]Amazon will not waste money on something that has no cutting edge technology.[/quote]Oh you are just trolling me. if you was working with engine code and was trying wrote code to make new graphics pipeline or made  new shader you should understand what code base of renderer is just shit. Dude you'll need just digging into the renderer's code to understand this.the buying of cryenging was big misstake for AMZ, they can write own's renderer what will be better.SharerePost-User-2514050answered 3 years ago0Trolling? I am simply expanding your knowledge. LMAO. Big mistake? If you observe the evolution of graphics technology carefully. Its aim is ray tracing. Ray tracing requires realtime computation. Cryengine and Lumberyard has that because of its core engineering is realtime from the beginning. Other game engine like Unreal doesn't have that. They are trying to implement it. But if you watch their talks about ray tracing they warn their developer not to use global illumination in large forest scene. It is so funny because Crysis is already using that technology without a problem. Cryengine has SVOTI and SVOGI to achieve global illumination. It seems your knowledge is product of Unreal marketing. It is not that easy to create another engine. Just buy one with high quality and improvised it further. If there is another engine there that can match Cryengine. It will easily catch many developers attention. See their latest ray tracing demo on reflection. You will wonder why Unreal and Unity can't release a demo like that. Because their engine is limited by its core engineering.SharerePost-User-2061097answered 3 years ago0Yeah, I was giving the benefit of the doubt, but I'm gonna call trolling.SharerePost-User-5431342answered 3 years ago0Those are 32 bit and Lumberyard is 64, or am I missing something here?SharerePost-User-7357299answered 3 years ago0Thanks Ralph. Very useful info.SharerePost-User-7357299answered 3 years ago"


Question: SCTP - AWS diameter sigtran integration

"Does AWS support the SCTP protocol for Diameter and Sigtran integration, and if so, how?Br, BojanFollowComment"


Question: VPN Connection Public IP

"During the VPN setup, a customer provide us the form request without specifying their VPN router public IP.They will provide their VPN Public IP after we complete the form (including AWS tunnel public IP end point).As per my understanding when we create site to site VPN connection we do the following:Setup VGWCreate CGW (including customer VPN router public IP)Then create the VPN connection by pointing to the corresponding CGW. We can only know the AWS VPN connection public IP.This will be cyclic process as we don't have customer public IP in the first hand.Any idea how to overcome this situation?Can we reserve or allocate AWS VPN pubic IP first?FollowComment"

Answer: "0Accepted AnswerI think I will go with the following workaround:Create dummy CGW (with dummy IP publicCreate VPN Connection & get the tunnel IP Public to be shared to customerLater on when customer already share their VPN router IP public, we create 2nd CGW & modify the VPN Connection to pointing to the 2nd CGW.Just tested this approach and it can be done, we can retain our 2 tunnel IP PublicCommentShareEXPERTTedy_Tanswered 3 years ago"


Question: We want to use Amazon Connect for multiple clients.

"**Use Case: **We would like all calls routed to one agent queue. We don't have a lot of agents but our agents have the ability to service all clients.Use Case: We would like each inbound call to the agent queue to have the customer’s specific branding displayed to the agent. This will help us identify the client better.Use Case: When an agent answers the call, we would like to restrict the S3 data available to the agent. We would like the Agent to only access data records specific to that Customer, i.e. name, date of birth, phone number, cases, previous notations, previous call records etc…**Use Case **- We want to use Amazon Wisdom for these clients as well but don’t want our agents to have access to all Wisdom content. We would like to restrict the Wisdom search results to Wisdom content pertaining to the specific Customer.FollowComment"

Answer: "0Use Case: We would like each inbound call to the agent queue to have the customer’s specific branding displayed to the agent. This will help us identify the client better.Very easy to do with an attribute and a custom CCP.Use Case: When an agent answers the call, we would like to restrict the S3 data available to the agent. We would like the Agent to only access data records specific to that Customer, i.e. name, date of birth, phone number, cases, previous notations, previous call records etc…I'm not sure I understand the S3 and agent relationship, but the customer data you mention and the first item can be solved the same way.**Use Case **- We want to use Amazon Wisdom for these clients as well but don’t want our agents to have access to all Wisdom content. We would like to restrict the Wisdom search results to Wisdom content pertaining to the specific Customer.I'm not sure on this one. If an agent can handle any client, why the restriction? Aren't you better served by tagging your knowledge with the customer name to ensure the most accurate hit?davidCommentSharedmaciasanswered a day ago"


Question: Images cannot be displayed in my snapshot lightsail wordpress instance

"I took a snapshot and created the instance, but I found that the instance images in my snapshot didn't show up in the preview and slowed down the site loading, only the text didFollowComment"


Question: Small server with IPMI access

"Hello,I'm new to AWS, so apologies for a possibly basic question. I am looking for the smallest server possible that will allow access to IPMI.Any help would be appreciated.Many thanks,RFollowComment"


Question: dash app not working when deployed in AWS ECS

"Hi all. I deployed a python dash app in ECS. Everything works fine and the task seems to be running.https://i.stack.imgur.com/C8jZh.png?s=556https://i.stack.imgur.com/mywPI.pngs=556However when i try to access it with the public or private ip on the port that the app is running (10000), it is not working as you can see below. Any idea why it could be happening?https://i.stack.imgur.com/dhamC.png?s=556Apologies, not sure how to post images.FollowComment"

Answer: 0Was the task created through a Service or directly?A few things you can check:Did you configure the network bindings in your container to expose port 10000? (portMappings section in the Task Definition)Check the inbound rules of the security group attached to your task ENI to make sure port 10000 is allowedVerify you don't have any NACL on the subnet that could block some trafficCommentShareogaumondanswered a year ago


Question: How to access multiple roles from single IAM user simultaneously?

"Can we access and use multiple roles from a single user simultaneously(using STS assumeRole api right now)? My use case requires creation of different roles which have their specific policies assigned, and I need a different way other than creating one user per role sort of like a multiple account access. Instead of creating 1 user per role, is there any way that a single user can access/assume resources of multiple roles/accounts simultaneously?FollowComment"

Answer: "0Technically, you can assume multiple IAM roles at the same time but the permissions will not be aggregated.Assuming an IAM role doesn't change who you are or what permissions you have.When you assume a role, you are given a new set of temporary credentials to use, instead of "your" credentials -- the credentials you used to assume the role.When interacting with AWS resources each request can only be associated with a singular principal. So, if you assume role1 and role2 you can make requests as role1 OR role2 but not as both together. So, if you are trying to perform a single action that requires an aggregate of the permissions of multiple roles, that's a not possible.CommentShareAWS-User-6966711answered a year ago0Another approach could involve session policies. See: Create fine-grained session permissions using IAM managed policies.CommentShareEXPERTkentradanswered a year ago"


Question: SageMaker PIPE Mode vs FSx ?

"Hi,SageMaker supports training data streaming via PIPE mode, and also reading from FSx distributed file system.Those options seem to provide same value: low latency, high throughput.What are the reasons for using one or the other?Do we have any benchmark of PIPE vs FSx for SageMaker, in terms of costs and speed?FollowComment"

Answer: "0Accepted AnswerI can think of the following scenariosPipemode cons** UPDATED**Data Shuffling -  In pipe mode you are working with streaming data and hence you cannot perform data shuffle operations unless you are prepared to shuffle within batches (as in wait to read a batch of records and shuffle within the batch in Pipe mode). Of if your data is distributed across multiples files, then you could use Sagemaker data shuffle to perform file level shuffleData readers -   There are default data readers for pipemode that come with Tensorflow for formats like csv, tfrecord etc. But if you have custom data formats or using a  different deep leaning  framework, yYou would have to use custom data readers to deal with the raw bytes and understand the logical end of record. You could also use ml-io to see if any of the built-in pipe mode readers work for your usecasePIPE mode streams the data for each epoch from S3 and hence will be slower than FSX when you run a few epochsFSX:FSX works by lazy loading the  s3 file and hence it has a start up delay but gets faster during repeated training.There is no  dependency on the framework and your existing code will work as is..The only con of using FSX is the additional storage costs, but I would almost prefer FSX to pipe mode in most cases.CommentShareEXPERTAparna_Eanswered 3 years ago"


Question: Enabling/Disabling AWS Load Balancer Target Nodes for Maintenance

"Question:Is there any way to temporarily disable/re-enable Network Load Balancer Target Group nodes for maintenance like we usually do with other load balancers?Background:We have a distributed replicated LDAP service that serves our enterprise.  It is based in multiple datacenters of various architectures.  We have several representative LDAP nodes in AWS under a Network Load Balancer to provide service to our AWS-based applications.  These LDAP nodes each hold a replicated copy of the entire directory, which consists of 1.8 million accounts plus attendant groups and other structures.  The service is more akin to a database and loading the data can take a few hours, so the nodes aren't so amenable to auto-scaling processes.  These nodes run RHEL7 and 389 Directory Server.Attempted Solution:I've tried draining/re-registering each node for maintenance but it didn't work particularly well as the connections don’t seem to get killed reliably when drained and the whole drain/re-register operation takes a long time.I also don't want to have to rebuild my target group every time I do maintenance on the nodesThanks a lotFollowComment"

Answer: "0A set of target group options you could look in to improve the drain and deregistration time of your NLB targets would be the Deregistration Delay and Connection Termination on Deregistration.The Deregistration Delay can be modified from the default value of 300 seconds to decrease the time ELB will wait before changing the state of a target from "draining" to "unused". As well, modifying "Connection termination on deregistration" to "true" is a recommended option that will cause NLB to terminate any remaining active connections to that target once the Deregistration Delay time is reached, bringing it to a fully deregistered state.Hope this helps!CommentShareSUPPORT ENGINEERJustin_Banswered a year ago"


Question: debugging using cloudwatch logs from different

We have a build an application using mutiple aws services. The request from front end goes to API GW and then to Aggregator Lambda Function and then to Individual Data Based Lambda Function and then the request goes to Dynamo db or Elastic Search.The logs from Lambda goes to Cloudwatch.  We have to open 2 different cloudwatch logs to debug the issues.We want to understand the ways to get all logs in a single log view. Are there any way to get these kind of logs ?Can some one share any insights on this ?FollowComment

Answer: "2the recently released "AWS CloudWatch Cross-Account Observability" feature can be useful in your use case.https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-cross-account-observability/To use it, just setup one of the accounts as source account and the other as monitoring account, the logs from both accounts will show up in the monitoring account, then you can search logs using CloudWatch Insight.Certainly, other observability tools in the marketplace, like Splunk, DataDog, can also help, but it will involve some major cost.CommentShareJasencanswered 5 months ago"


Question: Amazon OpenSearch Service tenant auto-switching

Hello everyone!Faced with the problem. When I opened OpenSearch Dashboards and set auto refresh (Refresh every 5 minutes) for instance and just stay on this page (or even without auto refresh). After some time I redirected to the private tenant (from global).What’s the reason and how to fix it? Thanks in advance.FollowComment

Answer: "0Service software R20220928 includes bug fixes and performance enhancements, including improved SAML logging. The update also changes the default tenant to Global rather than Private.https://docs.aws.amazon.com/opensearch-service/latest/developerguide/release-notes.htmlCommentShareSUPPORT ENGINEERNarayan_Sanswered 8 months ago"


Question: Unable to see real time logs

"Good morning,I have a video streaming session running that is distributed using Cloudfront. I configured the real-time logs according to the docs, including the Kinesis Data Streams consumer.The video streaming session works perfectly, but I'm unable to see the Stream metrics in the Kinesis stream. It says: "No data available. Try adjusting the dashboard time range.". But the time range is not the issue.The endpoint is correctThe IAM role has all the permitsThe real-time logs are enabledThe data stream is activeWhat could be the problem? I'm out of ideas.Thank you for your kind help and have a nice day.FollowComment"


Question: Making an interior game

"I just recently downloaded this engine and started playing around with it. I'm very impressed and excited. This engine is awesome!For my first exercise with this engine I'd like to try to make a game which takes place indoors. To this end, is there a way to remove terrain, the water plane, and the sun from the world? I know the roll-up bar has options to disable rendering of ocean and terrain and the time of day editor can be used to set the sun color to 0,0,0 effectively removing sunlight from the scene, but I'd still like to know if there's a proper way to completely remove these objects from the scene. Also, I would like to turn off the distance fog as well.ThanksFollow"

Answer: "0Accepted AnswerHello @REDACTEDUSERThe totally not robots have come back with some knowledge for your brainspace! The short answer is CVARS, we 'can' accomplish all of these results through the various menus but CVARS are the quickest and most effective for your needs.To access the CVARS menu click the little {X} button to the left of the Console.REMOVEDUPLOADSearch for the Variables e_Terrain, e_WaterOcean, e_Sun, and e_Fog and set all of these to 0.REMOVEDUPLOADWith all of these set there should no longer be light in your scene, to see any meshes etc. you may have added right click on the word 'Perspective' in the Viewport Header and activate Wireframe mode.REMOVEDUPLOADAdding in Point Lights where needed to illuminate your objects will also help.REMOVEDUPLOADLet me know if you have anymore questions and I will do what I can to dispel to fog of knowledge surrounding this subject matter...or in a pinch I will answer your questions :DStay Classy People,DeranderSharerePost-User-1172586answered 6 years ago0Hello @REDACTEDUSERYour question has been received and is currently being processed by people that are not actually robots in disguise. We will be back ASAP with some answers for ya!Thanks!SharerePost-User-1172586answered 6 years ago0Awesome thanks this is exactly what i needed to know.I posted a different question here and the answer had to do with cvars as well. It seems like some really substantial engine functionality is controlled by cvars. You guys should consider standardizing a CVAR Ebus.Thanks!SharerePost-User-1217528answered 6 years ago0Feedback noted! Thanks for this perspective @REDACTEDUSERSharerePost-User-5738838answered 6 years ago0For any level you can set a config file in the leveldirectory that overloads the cvars set in any previous config file.So you don't even have to submit them programmatically yourself.For interiors you may also look at vis areas/portals as they also help to selectively exclude things that doesn't belong to your interiors(sea,sky,terran , but also other meshes, lights and enviroment probes).SharerePost-User-7621041answered 6 years ago0Please add a feature to lumberyard , for Control/Add/Inset CVAR and All nodes flowgraph to inside Script Canvas visual scripting , thanks a lotSharerePost-User-8224505answered 6 years ago"


Question: add node into node group

"Hi Folks,IHAC, They are focus on HPC and using NVME SSD for High disk performance.they need to add node into node group on EKS, and get the node specification. Then they will submit the following pods to the same node to use the same NVME SSD.Does EKS has API or other methods to add the node into node group, and get the related node info?Many thanks,FollowComment"

Answer: "0Since nodes in a node group inherit the labels and taints, it would be better to create a separate node group for the HPC/SSD requirementHPC / NVMe SSD Pods can then be scheduled on these nodes using appropriate node selector / tolerations and the regular pods could be scheduled as usual on the existing node groupRef: k8S Doc & AWS Doc--SydCommentShareSydanswered a year ago"


Question: Best path to learn Python (for AWS) - Most work will be lambda/cloudwatch work

"Morning all, as a sysadmin with some OK programming skills in PHP, Bash, I want to goto the next step as I have a lot of cloudwatch alerts in palce, and sometimes CPU > X doesn't mean restart the box vs just goto that box, kill the java server, restart, etc.So there is a lot out there, I am sure I can find someones example and update it if I look, but looking to see if there is more of an AWS type of track outside of the simple cloud9 path or will that be sufficient.It would be great to learn more python for more tools in the belt as I do most with Bash, but as I said, with cloudwatch, I can't say run this shell script, so I want to say this is happening to server x01, run this lambda, that says oh, goto x01, kill java, start, etc.   I see acloudguru and codeacadamy and someone may have input on those or others.Thanks for help/advice on this one.FollowComment"

Answer: "1Accepted AnswerIf you want to continue to use Bash, you can integrate your CloudWatch alarms with AWS Systems Manager. A CloudWatch alarm can trigger a Systems Manager OpsItem or create an incident (https://docs.aws.amazon.com/systems-manager/latest/userguide/OpsCenter-create-OpsItems-from-CloudWatch-Alarms.html). You can then automate the response to that OpsItem with System Manager automation which can run a script on the instance (https://docs.aws.amazon.com/systems-manager/latest/userguide/OpsCenter-remediating.html).If you want to go the Python route, you'll need a solid background in Developing on AWS. We offer an AWS Ramp-Up Guide for Developers here: https://d1.awsstatic.com/training-and-certification/ramp-up_guides/Ramp-Up_Guide_Developer.pdf. All of our AWS Ramp-Up Guides are here: https://aws.amazon.com/training/ramp-up-guides/CommentShareAWS-Harrisonanswered 20 days ago0In addition to above, here are a several of other Python resources.Learn Python On AWS WorkshopCode examples for SDK for Python (Boto3)Building Modern Python Applications on AWSCommentShareEXPERTkentradanswered 20 days ago"


Question: Export to CSV in NoSQL workbench exports only those that is shown in the grid

"Hi,I started using NoSQL workbench today.I am able to connect and view the Dynamo DB table in my AWS account.I set the 'Item count per page' to 100 .... but that also applies to Export to CSV.I am able to export only 100 at a time.How can I export ALL, records that satisfy the GSI being used.thanksJeevanFollowComment"

Answer: "0Hi Jeevan,There are a few ways of doing this with NoSQL Workbench. Have a read of the NoSQL Documentation for Building Complex Operations which will take you through different ways of handling data.A couple of approaches to look at:Build a Scan operation using the Operation Builder. Select the Connection, Region and Table to query, make sure "Interface-based operations" is selected, then click on "Expand Operation" near the top of the screen. Here you can choose "Scan" against your table which will retrieve all the data from your table, which you can then Export to CSV.Build a PartiQL operation using the Operation Builder. Again select the Connection, Region and Table to query but this time select "PartiQL operations" at the top of the screen. Now you can write PartiQL statements to extract data as you need from your table, and again Export to CSV is available once you've extracted the data.Please note that extracting all data from DynamoDB can carry out a Scan on your entire DynamoDB and if you're working with large amounts of data, please be aware of the DynamoDB Pricing information before you run your extracts.CommentShareAWS-User-9293507answered 2 months ago"


Question: Can I create relationship from User pool to table in DynamoDB?

"Hi everyone,I am newbie in AWS, I just wonder that can I create relationship from Cognito User pool to table in DynamoDB and what is the best practice to resolve my problem?Here is my scenario:_ I have a table named "Vehicles" and this table will only store UserID from Cognito User pool._ When I assign one Vehicle to User (from list Users of Cognito User pool), I will store UserID to this table.My concern is: How can I retrieve User's information like family_name, given_name, etc... to display to UI when I get a list of Vehicles.My current solution is: I created a table User to store all User info after user sign up or created. Then create relationship with Vehicles table and other table to handle business logic. But, when User update email, phone, or groups in User pool, I have to update to User table.Could you guys give me some instruction?Thank you!FollowComment"

Answer: "0What I do is simply fetch the data from Cognito when needed.So in your case, first fetch the Vehicle in DynamoDB, then fetch the user data, e.g. using adminGetUser (though that for some obscure reason does not support user id/sub, only username and alias, so listUsers with a filter for "sub" might be better).I was about to suggest that you could use Lambda triggers if you really needed to store the attributes in DDB, but it seems there is no Lambda trigger for attribute updates, which is quite odd, but I guess you could get around that by only letting users update attributes through your backend, which could then do the DDB update as well.CommentShareDennisDKanswered 4 years ago0Thank you very much for your reply!Yeah, I just add a Lambda trigger to save User info into DDB after they confirm signup. Then everything will be handled on backend and DDB.CommentShareThanhDanswered 4 years ago"


Question: what if i go over my limit

"I want to host simple website, but be able to manage it myself, I don't need much storage, other than to install necessary stuff.  I just want to be able to stay within same monthly limit and that not to change. (and even RAM, just to be functional, so it can handle traffic)Is there better way with other AWS services other than AWS lightsail. I checked on-demand pricing, but got confused, it says "pay what you use", but that means "how much EC2 instance is running", it will be running for 2 years continuously..And then with Reserved Instances , I checked for 3 years to pay all upfront (Linux, Shared, US East (Ohio) , and I get some offers, like ( t2.micro $115 ) . But do they depend on network traffic ? Or they just have some network bandwidth and that's it (no limit how much users I can have ? )But if you reccomend me to use AWS Lightsail, then question I also have is :If I create instance on AWS Lightsail, for 3.5 $ per month, I get 1TB of network transfer.Does that mean, if my website is 200 kB to download on client, then i can have only 5,000,000 connections ?And if it goes over that limit , how I'm gonna be priced for more network usage ?FollowComment"

Answer: "1Hi,I just want to be able to stay within same monthly limit and that not to change. (and even RAM, just to be functional, so it can handle traffic)Your instance type does not change unless you modify it. Also if you dont configure autoscaling but just 1 instance (or some static count) that's what would be running for as long as you keep it. If the load increases and the compute / RAM / storage is insufficient, then the instance would be unresponsiveI checked on-demand pricing, but got confused, it says "pay what you use", but that means "how much EC2 instance is running", it will be running for 2 years continuously..in case of on demand instance the hourly price * tenure for which you keep it running is the final price, billed monthly. If you run it for only 10 days a month, then you get billed only for 10 days.And then with Reserved Instances , I checked for 3 years to pay all upfront (Linux, Shared, US East (Ohio) , and I get some offers, like ( t2.micro $115 )You get discounts since you making an upfront commitment. If your sure to use it for 3 years, you can reserve instance. You should also compare with Savings plans. They offer some flexibility compared to Reserved instances.But do they depend on network traffic ? Or they just have some network bandwidth and that's it (no limit how much users I can have ? )the instance type you choose has a network throughput limit and you can find that information here - https://aws.amazon.com/ec2/instance-types/. The total data transfer (IN & OUT) on these instances is independent of the throughtput limits. Download / upload speeds would be capped at the throughput speedAnd if it goes over that limit , how I'm gonna be priced for more network usage ?You are charged for data transfer in excess of limits. Refer Q -What does data transfer cost? in https://aws.amazon.com/lightsail/faq/--SydCommentShareSydanswered 2 months ago0EC2 network bandwidth varies by instance type and instance family, but there is no limit based on the number of users.Also, the 1TB network transfer for AWS Lightsail that you are talking about is only charged for the amount over 1TB, not for the fact that you can only use up to 1TB.https://aws.amazon.com/lightsail/pricing/?nc1=h_lsCommentShareEXPERTRiku_Kobayashianswered 2 months ago"


Question: GuardDuty pricing investigation

"A customer is using S3 and Glue Job to combine all the files in their s3.As it stands their s3 cost is almost half of guard duty and glue job.Is there a way to find out what made the guard duty cost to go up? I mean it tells me the usage ie. this bucket but thats pretty much it, what was scan etc there is no informaiton on that, and why almost double the cost of the data that is there.I can tell the customer not to use s3 data events for guard duty by turning it off, not something I want to do, but just trying to understand if there is a way to tell or visualize why the cost almost double of the data there.FollowComment"

Answer: "0Accepted AnswerWhen enabling GuardDuty for S3, GD starts looking for S3 Data Events, e.g. GetObject, ListObjects, DeleteObject, and PutObject API operations. They are often high-volume activities, especially if used in the context of ETL processes.You can find more details by creating a Cost Usage Report (CUR) and filter by product/group = Security Services - Amazon GuardDuty Paid S3 Data Events Processed. If you are using tags, you can get a more granular view of which buckets are contributing the most (this is also available from the Events section in the GD console)CommentShareEXPERTLuca_Ianswered 2 years ago"


Question: Can't Connect to my EC2 instance with SSH

"Hey everyone,I have been suffering a hard time connecting to my EC2 instance with SSH recently. At the beginning I could connect to it but for somehow reason I lost my connection. (tried both with my VS code studio and terminal) Here are what I have been tried to troubleshoot the problem:Reboot my instanceReboot my PCDouble verify the IPv4 IP address and pair keyDouble verify my inbood rule is openDouble verify the instance is running with 2 green check passedDouble verify the ssh config set up is correctPing to google but doesn't receive anythingIt seems that the only way that I could successfully re-connect to it is stopping the current one and re-starting a new one. However, Amazon will re-allocate a new IPv4 public address to me. I have to re-set up some of my working environment. I do have bash script set up for EC2 connection and transferring file with sshfs. But it is just so annoying and time consuming that I have to do it every time. BTW, my EC2 instance doesn't have the elastic IP address because I am at the free tier and don't want to pay anything yet. Does anyone have a better idea without doing those. Appreciate your response in advance.FollowComment"

Answer: "1The fact that you can SSH to the instance's public IP when it is first started shows that your security groups and routing tables etc. are likely all correct. but then a short while later (are we talking hours or days?) you can no longer connect.Does the security group allow ingress on TCP port 22 from any address (0.0.0.0/0) or did you select "My IP" when setting up the security group for the instance? If it was limited to your IP then has that changed - depending on your ISP your Home IP address can change periodically, and obviously your IP is different at home/at work/on VPN/on cellular.The other thing I'm thinking of is that to be running within free tier you must be using a t2.micro or t3.micro instance, which are pretty puny. What else are you running on it? You may be running something that is fine at the start, but slowly consumes more and more resources until you can't login any more (it will still show up as running and 2/2 checks passed in AWS Console as it's still actually running, though unusable). Select the instance in AWS Console and look in the Monitoring tab, is there anything of concern in here?CommentShareRWCanswered a month agolj4028aim a month agoThanks for your reply. I do open port 22 for all incoming IP address. And you are 100% right. I am running a t2.micro. Not sure if this is true that I do feel it starts slowing down after few weeks since the first time I create it. When checking the monitoring tab for the instance while it still show 2/2 checks passed, I do notice that the CPU usage is up to 70%. I mean I really didn't do something seriously but just running small Docker container. I believe you are on the right track. This happened to me few time already. Every time I run the docker container, the terminal feels like freezing for few minutes and re-back up. Same thing for my visual studio code. It lost connection for few minutes and can be re-connect to it after a while. Is that just because I am using the free tier? Thanks.ShareRWC a month agoFree tier instances are, as you would expect, not very powerful and you can quickly run out of resources. 70% CPU usage isn't a problem if it's expected that your application will need this much CPU, but it can be a warning of other issues. t2.micro has only 1GB of memory, if your application is trying to use more memory than is available then the memory manager comes into play, and will try to swap processes out of main memory and onto disk (it's more complicated, but that's the general gist of it). This in itself is CPU-intensive, and as free memory gets less and less the memory manager will spend more and more of its time (and more and more CPU) trying to free up pages of main memory, leaving fewer and fewer CPU cycles for anything else (including handling SSH requests for login).It might not even be memory usage, that's just a guess. Many years ago I ran a website on a free tier EC2 and found that a memory leak in Apache was making my EC2 become unavailable after about a month. The fix was to restart Apache once a week.Unfortunately the EC2 section of the AWS Console doesn't display metrics for memory use, you'll need to setup CloudWatch agent to collect these https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.htmlSharelj4028aim a month ago@RWC Thanks for your prompt reply. After checking the monitor tab, it doesn't take 70% of CPU usage but almost close to 100% while the container is running. Wow! No idea why that happens. Now everything makes sense. Thanks for your insights.Share1Hi, what does "Ping to google but doesn't receive anything" mean?  Where did you ping from?Does EC2 Instance Connect work?I don't know what the problem is but note that to make things a little less painful you can use an Elastic IP for free so long as it remains attached; it's only the unused ones that you get charged for.CommentShareEXPERTskinsmananswered a month agolj4028aim a month agoWhat I mean by "Ping to google but doesn't receive anything" is typing command ping www.google.com, and the result is I don't receive any feedback. I have to interrupt it by using Ctrl + C. Regarding the elastic IP address, I tried it once before with attaching my running EC2 instance, and receiving an email from AWS and been notified that I am running out of 85% of my usage of the month. I looked up for amazon document, it does say it won't charge you if you attach it to your active EC2 but seeing and hearing different answer from people. Is that like totally free as long as you keep it attached to your running EC2 instance? So why am I still receiving email from AWS? Thanks.Shareskinsman EXPERTa month agoFree tier gives you a certain amount per month.  If you're running a "free tier included" instance constantly and get the 85% warning when you're 85% of the way through the month then there's nothing to worry about.Share1Another common issue with EC2 instances is that the IP address has changed.  When you assign a public IP address during the creation of the EC2 instance, it is not permanently assigned.  What happens is after a Stop and Start (not a reboot), a new IP address is assigned.  So can you verify you are trying to connect to the correct address?  And verify you didn't hard code the address in any ACLs or Security Groups.  If you want to keep your IP address permanently, you need to create the EC2 instance with no public IP and then manually assign an allocated elastic IP.  This IP address will survive reboots.CommentShareChristian Lorisanswered a month agolj4028aim a month ago@Christian Thanks for your comment. I do remind changing IP address every time I stop and re-start the instance. Never hard code the address in any ACL or security groups. After reading @RWC's comments listed above, I did couple test by running different docker container on EC2 instances and it just fine. If I didn't run anything, and the connection will stay constant perfect. So, now I believe the issue is more related to CPU or memory usage.Share"


Question: how to hook up CI/CD with sagemaker pipelines  in sagemaker sudio?

"https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-pipelines/tabular/abalone_build_train_deploy/sagemaker-pipelines-preprocess-train-evaluate-batch-transform.ipynb, I have created various pipeline step and tie those together and create a pipeline at the end. once these pipelines are created , how do i link this pipeline with steps for data preprocessing, training with other pipelines, say one for evaluation , another for deployment. is there anything inbuild in sagemaker studio to do this , or do we have to write our own code for this ? how can i link/feed output/artifact from one pipeline to another?from sagemaker.workflow.steps import TrainingStepfrom sagemaker.workflow.steps import ProcessingStepfrom sagemaker.workflow.pipeline import Pipelineprocessor_args = sklearn_processor.run(    inputs=  ...  ,    outputs=...  ,    code="code/preprocessing.py",)my_processing_step = ProcessingStep(name="my_processing_step ", step_args=processor_args).....my_training_train = TrainingStep(    name="my_training_train ",    step_args=train_args,)...pipeline_name = f"experiment_Pipeline"pipeline = Pipeline(   name=pipeline_name,    parameters= ..,     steps=[my_processing_step , my_training_train  ...],)FollowComment"

Answer: "0SageMaker PipelineAmazon SageMaker Model Building Pipelines Normally, registering a model is the end of life of a pipeline. For now,  AFAIK, there is no linking facilities in SageMaker Pipeline.However, you may consider SageMaker Project, which is another feature in SageMaker Studio. It maintain a whole CI/CD workflow, by linking sagemaker model build pipeline with codepipeline/codecommit to achieve the secondary evaluation and deployment in staging/prod. Details about SageMaker Project via https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-projects-whatis.htmlCommentShareAWS-User-5210721answered 5 months ago"


Question: The security token included in the request is invalid

"Hi,I have nested workflow for Step Functions where the outer state machine awaits for the inner state machine to return task token. Definitions:Outer State machine:{  "Comment": "A description of my state machine",  "StartAt": "ChildProcessing",  "States": {    "ChildProcessing": {      "Type": "Task",      "Resource": "arn:aws:states:::states:startExecution.waitForTaskToken",      "Parameters": {        "StateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:child",        "Input": {          "parentTaskToken.$": "$$.Task.Token",          "AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.$": "$$.Execution.Id"        }      },      "ResultPath": "$.output",      "Next": "PostJobSuccess"    },    "PostJobFailure": {      "Type": "Task",      "Resource": "arn:aws:states:::sqs:sendMessage",      "Parameters": {        "MessageBody.$": "$$.Execution.Input",        "QueueUrl": "http://queueurl"      },      "End": true    },    "PostJobSuccess": {      "Type": "Task",      "Resource": "arn:aws:states:::sqs:sendMessage",      "Parameters": {        "MessageBody.$": "$$.Execution.Input",        "QueueUrl": "http://queueurl"      },      "End": true    }  }}Inner State machine:{  "Comment": "Child processing.",  "StartAt": "GetJobStatus",  "States": {    "GetJobStatus": {      "Type": "Task",      "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",      "TimeoutSeconds": 120,      "Parameters": {        "QueueUrl": "http://childqueueurl",        "MessageAttributes": {          "jobName": {            "DataType": "String",            "StringValue": "AsyncProcessingJob"          }        },        "MessageBody": {          "taskToken.$": "$$.Task.Token",          "machineExecutionId.$": "$$.Execution.Id"        }      },      "ResultSelector": {        "input.$": "$$.Execution.Input.input",        "parentTaskToken.$": "$$.Execution.Input.parentTaskToken"      },      "Next": "CheckJobStatus"    },    "CheckJobStatus": {      "Type": "Choice",      "Choices": [        {          "Variable": "$.jobStatus",          "StringMatches": "finished",          "Next": "SendTaskSuccess"        },        {          "Variable": "$.jobStatus",          "StringMatches": "failed",          "Next": "SendTaskFailure"        }      ]    },    "SendTaskFailure": {      "Type": "Task",      "Parameters": {        "TaskToken.$": "$$.Execution.Input.parentTaskToken"      },      "Resource": "arn:aws:states:::aws-sdk:sfn:sendTaskFailure",      "End": true    },    "SendTaskSuccess": {      "Type": "Task",      "Parameters": {        "Output.$": "$.input",        "TaskToken.$": "$$.Execution.Input.parentTaskToken"      },      "Resource": "arn:aws:states:::aws-sdk:sfn:sendTaskSuccess",      "End": true    }  }}Here is my aws-stepfunctions-local-credentials.txt with fake credentials and setting STEP_FUNCTIONS_ENDPOINT to http://localhost:8083AWS_DEFAULT_REGION=us-east-1AWS_ACCESS_KEY_ID=testAWS_SECRET_ACCESS_KEY=testAWS_SESSION_TOKEN=testSTEPFUNCTIONS_PORT=8083STEP_FUNCTIONS_ENDPOINT=http://localhost:8083STEPFUNCTIONS_HOST=stepfunctions-localWhile executing this nested workflow, I am getting the error: {"Type":"TaskSubmitFailed","PreviousEventId":22,"TaskSubmitFailedEventDetails":{"ResourceType":"aws-sdk","Resource":"sfn:sendTaskSuccess","Error":"Sfn.SfnException","Cause":"software.amazon.awssdk.services.sfn.model.SfnException: The security token included in the request is invalid. (Service: Sfn, Status Code: 400, Request ID: 864f4abc-6e26-4ce7-9cfa-63ade13dd6ca)"}}Has anyone come across similar issue and found solution?FollowComment"

Answer: "0Hello,Thanks for reaching out, Tim here with Support Engineering.When you started step function local did you use the JAR method or the docker method?When you started it up, did you run into any of the following error messages?2022-07-20 16:33:28.828: Failed to load credentials from environment because Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))2022-07-20 16:33:28.865: Failed to load credentials from profile: default because profile file cannot be null2022-07-20 16:33:28.865: Failed to load credentials from system properties because Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey)2022-07-20 16:33:28.865: Failed to load credentials, default to dummy credentials, so any connection to AWS services will not work, connections to local endpoints are not affected.If you did, did you follow the documentation at these steps (one is for the JAR method to provide credentials and the other is for docker):JAR - https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local-jar.htmlDocker - https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local-config-options.html#docker-credentialsto provide credentials into the system?Could you also try to perform this test on the cloud resources and not local? That would potentially give us an idea with some execution logs as to what is occurring and isolate if it's a step function local, docker, or state machine item and helps chops up and pinpoint the issue a bit more.CommentShareSUPPORT ENGINEERTim_Panswered 10 months ago0Hi Tim, thank you for the response. Here are the answers to your queries:We use docker for setting up Step Function locallyNo I do not see any of the errors mentioned while setting up locallyYes, I have aws-stepfunctions-local-credentials.txt file which looks like this:AWS_DEFAULT_REGION=us-east-1AWS_ACCESS_KEY_ID=testAWS_SECRET_ACCESS_KEY=testSTEPFUNCTIONS_PORT=8083STEP_FUNCTIONS_ENDPOINT=http://localhost:8083/I have also tried putting AWS_SESSION_TOKEN=test to the above file, but no luck.I have tried a similar nested workflow in the testing AWS environment and the entire workflow finishes successfully.CommentSharegamathuranswered 10 months ago0Thanks for following up!Could you share the command you're using to start step function local?CommentShareSUPPORT ENGINEERTim_Panswered 10 months ago"


Question: AWS API shown "Access to requested resource is denied."

"https://developer-docs.amazon.com/sp-api-blog/docs/troubleshooting-selling-partner-api-authorization-errorsI have already follow the steps in the above link and test in postman, I am success to get a access token, but when I get the order, the response still is error with"message": "Access to requested resource is denied.","code": "Unauthorized","details": ""FollowComment"

Answer: "0Hello,As the Selling Partner API is not an AWS service, please reach out to the Amazon Selling Partner API support team through this link: https://developer.amazonservices.com/supportCommentShareEXPERTChris_Ganswered 9 months ago"


Question: [Opensearch] current ES version doesn't allow Collapse with search_after

"Hi Experts :)I wonder there is a workaround for using search_after with Collapse which isn't served on current AWS Opensearch Version 7.10https://www.elastic.co/guide/en/elasticsearch/reference/current/collapse-search-results.htmlWe tried to use an aggregation command with top hit using writer as a key, however, the presented writers bucket had low scores.https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.htmlSimply, we wanted to show the top 10 documents without duplication on the writer.How can we find a workaround for the current Opensearch Version?FollowComment"

Answer: "-1So the search_after parameter involves the point in time (PIT) API, which is an X-Pack/ElasticSearch exclusive feature due to licensing issues.I can recommend you try one or both of the following workarounds. First, you can try querying the index using the search call, without the "PIT" part of the query. For instance, like this sample query.-> GET sample/_search{"size": 2,"query": {"match_all": {}},"sort": [{"timestamp": "asc"},{"FlightNum":"asc"}]}Second, you may also find the scroll API useful. This will help you to retrieve large sets of results from a single scrolling search request.https://www.elastic.co/guide/en/elasticsearch/reference/7.10/paginate-search-results.html#scroll-search-resultshttps://www.elastic.co/guide/en/elasticsearch/reference/7.10/scroll-api.htmlCommentShareMargaret-at-AWSanswered 4 months agorePost-User-0531034 4 months agoAs I mentioned, I want to search the result only with unique writers(no duplication on the list) using Collapse and search_after.But current Opensearch version on AWS couldn't support this feature.Is there any workaround to get the search result with unique writers and paging?Share"


Question: Is there any plan for a FPS player?

"First, I would like to say I loved the new component system.So, for the cameras component I see a lot of features to build a FPS player. I believe it is going to that direction. Am I correct? Is there any plan to have a FPS player as a reusable asset?Tks!Follow"

Answer: "0Accepted AnswerWe are waiting for the release of a new starter game with a third person character.That would be a good starting point for you to make the fps player both with and without the full 3d body.SharerePost-User-7621041answered 6 years ago0do you know when we are going to have the new game available?SharerePost-User-4308601answered 6 years ago0No, I'm sorry.SharerePost-User-7621041answered 6 years ago0Yeah, we don't have a date as of right now but it should be in the next update or two.SharerePost-User-8353690answered 6 years ago"


Question: Hacker got into my AWS account. How can I nuke the account? [RESOLVED]

"Hello,Roughly 13 hours ago my AWS account got hacked into, and my email address being used for it got changed to kkezwipfqdl9046@gmail.com. I saw the email for this three hours ago, and immediately changed my password that's used elsewhere and cancelled the credit card tied to the AWS account. For peace of mind, I'd like to fully delete this account before any information held within it is used for other malicious purposes. How can I go about doing this? I cannot merely sign on and change the password as the email address for the account got changed.UPDATE I followed what the user 'Liberty Brands' said in this thread (https://repost.aws/questions/QUuB6WnPtbSHGeQ8vQjBVqNA/root-account-hacked-and-email-changed) and was able to get my account backFollowComment"

Answer: 1HelloOpen a support request using this form: https://support.aws.amazon.com/#/contacts/aws-account-supportCommentShareEXPERTMatt-Banswered 3 months ago


Question: CLI or API for refreshing SCIM Access Token?

"I can see how to generate a new SCIM access token through the web interface, but I'm not finding it in the API nor in the CLI documentation. Is there such an mechanism?I looked through these commands, but did not see anything that seemed appropriate:identitystoressosso-adminThank you!FollowComment"

Answer: "0Have you tried the following?https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.htmlCommentSharevtjeananswered 2 months agoAnthony Foiani 18 days agoThank you for your reply, but it looks like your link is talking about how individual end users can access AWS using various SSO methods.What I was trying to ask for (but probably not phrasing it very well) was how to generate a new SCIM token, used between AWS Identity Center and my company's IdP (in this case, Okta). We would like to automate rotating that value, but without an API, we have to do it manually.More specifically, I'd like to find an API / CLI equivalent to the actions described here: https://docs.aws.amazon.com/singlesignon/latest/userguide/provision-automatically.html#generate-tokenShare"


Question: Permissions issue in accessing kms-encrypted bucket using Media Convert from another account

"For accessing kms-encrypted bucket from Media Convert in local account, an in-line policy is required for the Media Convert role, not clear if applying that on the remote account would help, I thought it would not.https://docs.aws.amazon.com/mediaconvert/latest/ug/granting-permissions-for-mediaconvert-to-access-encrypted-s3-buckets.htmlHave bucket access generally (bucket + kms key permissions allow access from other roles in the remote account), but not via Media Convert.Any help/pointers appreciated.FollowComment"


Question: AWS DMS - primary key verification

"Hi there,I am planning to migrate our on-prem SQL Server to RDS SQL Server. Unfortunately, this db was managed by several people and we are not sure that all of the tables have a primary key. I wish to keep the on-prem db and RDS in sync for a couple of days, so I have to be 100% sure that all tables have a primary key. Do you know an automatic way, where I can check if all the tables (1000+) have a primary key?Cheers, MarkFollowComment"

Answer: "0This little snippet will tell you whether a table has a primary key or not - SELECT OBJECTPROPERTY(OBJECT_ID(N'TabA'),'TableHasPrimaryKey')Returns 0 if no primary key, Returns 1 if there is a primary key.Now all that's left is to get the list of all your tables from the INFORMATION_SCHEMA -SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.TABLESCommentShareAbbasanswered a year ago-1Accepted AnswerHello,You may use the SQL Server diagnostic support script, which will list down all possible issues in replication.https://docs.aws.amazon.com/dms/latest/userguide/CHAP_SupportScripts.htmlCommentShareJitenderanswered a year ago"


Question: dango  connect to `AWS QLDB` dependency issue

"Hello,I'm trying perform connect to AWS QLDB using Django projectHowever, I get an issue, once I add the following codefrom pyqldb.driver.qldb_driver import QldbDriverI get the issue ImportError: cannot import name 'Enum' from 'amazon.ion.util' (/usr/local/lib/python3.9...)Appreciate any help.FollowComment"

Answer: "0Ok, there was an issue in lates amazon-ion, so need to point directly amazon-ion = "0.9.3"CommentSharerePost-User-2019375answered 2 months ago0Hello !These python requirements worked fine for me:amazon.ion~=0.9.3ionhash~=1.2.1pyqldb~=3.2.2CommentSharerePost-User-2304584answered a month ago"


Question: aws-encryption-cli vs. aws kms encrypt/decrypt (aws-cli/2.9.5)

All - if I compare the two choices in the title the aws-encryption-cli and the aws cli within the kms group of commands they seem to overlap. In other words - strictly from a command line perspective I can encrypt and decrypt using the AWS CLI - so is there a reason to use the aws-encryption-cli which requires an additional install rather than aws kms encrypt/decrypt?FollowComment

Answer: "1The following provides a great summary on this topic:How is the AWS Encryption SDK different from the AWS SDKs?The AWS SDKs provide libraries for interacting with Amazon Web Services (AWS), including AWS Key Management Service (AWS KMS). Some of the language implementations of the AWS Encryption SDK, such as the AWS Encryption SDK for .NET, always require the AWS SDK in the same programming language. Other language implementations require the corresponding AWS SDK only when you use AWS KMS keys in your keyrings or master key providers. For details, see the topic about your programming language in AWS Encryption SDK programming languages.You can use the AWS SDKs to interact with AWS KMS, including encrypting and decrypting small amounts of data (up to 4,096 bytes with a symmetric encryption key) and generating data keys for client-side encryption. However, when you generate a data key, you must manage the entire encryption and decryption process, including encrypting your data with the data key outside of AWS KMS, safely discarding the plaintext data key, storing the encrypted data key, and then decrypting the data key and decrypting your data. The AWS Encryption SDK handles this process for you.The AWS Encryption SDK provides a library that encrypts and decrypts data using industry standards and best practices. It generates the data key, encrypts it under the wrapping keys you specify, and returns an encrypted message, a portable data object that includes the encrypted data and the encrypted data keys you need to decrypt it. When it's time to decrypt, you pass in the encrypted message and at least one of the wrapping keys (optional), and the AWS Encryption SDK returns your plaintext data.You can use AWS KMS keys as wrapping keys in the AWS Encryption SDK, but it is not required. You can use encryption keys that you generate and those from your key manager or on-premises hardware security module. You can use the AWS Encryption SDK even if you don't have an AWS account.CommentShareRajarshi_Danswered 5 months agorePost-User-5583179 5 months agoThanks - though I don't think this directly addresses the question of "why would one use the "aws-encryption-cli" CLI rather than the "aws" CLI (e.g. "aws kms encrypt" / "aws kms decrypt")Share0Accepted AnswerAWS Encryption SDK (ESDK) performs encryption at the client-side, that is, within your application and on hardware/instance of your choosing. Amount of plaintext you can encrypt depends on algorithm and data key size, and you can configure ESDK to use KMS as your key provider.Encryption using KMS API is done at the server-side, meaning you send a plaintext to KMS and in return receive ciphertext. Encryption is securely done within hardware security modules that's been validated (FIPS 140-2). There's limitation on payload size that you can encrypt/decrypt (4,096 byte).As to why you would ESDK over KMS and vice versa? It depends on your use case. If you have high performance requirement (e.g., latency, amount to encrypt, etc.), ESDK might be advantageous since encryption is done locally within your application. If you have a very sensitive data that needs to be encrypted under trusted and validated hardware, KMS might fit your use case better.CommentSharekumo-hiyorianswered 5 months ago"


Question: General latency details of Elastic search (Now Amazon Open Search)

Do we have any reference for fetching the general latency metrics of elastic search. As per the details it is related to the clients data loaded into it but wanted to know if we have any general latency metrics from AWS.FollowComment

Answer: "0You can find some latency related metrics mentioned here - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html#managedomains-cloudwatchmetrics-instance-metrics noteably Indexing Latency, Search Latency, Search rate and Indexing rate.Anything specific that you are looking for when you mention "general latency metrics of elastic search" ? The above metrics would be related to the type of data ingested and queries for and would be among the go-to metrics to understand overall latency behaviourCommentShareSydanswered 7 months agoSarath 7 months agoThis document have metric names (https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html#managedomains-cloudwatchmetrics-instance-metrics ). But, I am looking for comparison between postgresql and es search latencies (In general). Currently, I have the postgresql metrics and looking for es search latencies.In this page we have those metrics explained: https://www.2ndquadrant.com/en/blog/postgresql-latency-pipelining-batching/#:~:text=Latency%20matters&text=A%20trivial%20SELECT%20can%20take,set%2C%20query%20metadata%2C%20etc.Same details I am trying to get.Share"