Question: Lost changes in SageMaker Studio notebooks

"I have repeatedly encountered issue with lost recent changes in SageMaker Studio Collaborative space. It occurs when the session expires and requires reconnection. Even though the notebook instance is running, the checkpoint files are missing and the notebook does not display at all. When checking the file via terminal, previous (even 11h old) state is displayed. Any idea how to prevent the issue?I am using VpcOnly mode with private subnets only. There are two active users but only one is currently using the collaborative space. The notebooks often runs for couple of hours.FollowComment"

Answer: "0It sounds (from "The notebooks often runs for couple of hours") like you're running long-running - perhaps non-interactive - tasks on the notebook itself?In general (not particularly relating to data loss), I'd suggest to avoid this if you can: SageMaker's containerized (processing, training, batch transform) jobs functionality should be the preferred choice for running your serious jobs/experiments in a way that's:Scalable - Supporting multi-instance distribution as well as selecting larger or smaller instance typesAutomatically trackable - Job metadata & parameters will be stored in the SageMaker APIs, exact version of code automatically uploaded to S3 (assuming "script mode"), and logs+metrics stored via CloudWatchRepeatable - In addition to the metadata logging, each job initializes with fresh, well-defined container image without the risk that somebody installed libraries / changed something in the shared notebook space.Cost-effective - Because you can keep your notebook instance (which, being interactive, may often be idle) small and instead select the instance sizes needed for your jobs and pay only for the time the jobs run.If you must, you could run your notebook as a job... But ideally I'd recommend factoring your code into .py scripts and using actual Processing/Training jobs for richer metadata history. There's an example here comparing a "local" (all-in-notebook) notebook to a "SageMaker" (code in scripts, notebook initiates the training job) one for model training. You could even use a SageMaker Pipeline to orchestrate multiple steps like pre-processing > training > evaluation.Anyway, back to Studio data loss...In my experience, disconnecting or timing out from Studio hasn't caused any interruption of actual kernel computation or loss of data written to disk... But I have seen that cell outputs (i.e. console logs) can get cut off: For example if I start a long-running cell that trains a model and then saves it, then close the browser and come back tomorrow, I'd expect the model to be trained and saved and to be able to resume the notebook from the next cell... But some training logs might be missing from the cell output.I tentatively believe from this open Jupyter GitHub issue from 2015, that this issue isn't particular to SageMaker. It seems like Jupyter's launch of real-time collaboration was likely to improve the situation a bit, but not expected to fix it completely in the case that all users disconnect? For cell output, my suggested workaround would be to log important things to file in your long-running cells (or, better yet, use SageMaker jobs instead as mentioned above)If you're instead struggling with losing unsaved work when re-authenticating, by suggestion would be:When your session times out, leave that tab as it is. Do not exit/refresh/close/etc: Because Jupyter is a hybrid IDE and some state is stored client-side.In a new browser tab, re-launch Studio and log in. When you do, it might present you with a blank workspaceClose the new tab and go back to the old one: With your auth cookie refreshed, your previous workspace should be there exactly as you left it including any unsaved files that didn't get synced to the server.If your issue is something different than disappearing cell outputs or lost changes due to refreshing for re-authentication, then apologies I haven't faced what you saw yet! Either way, if you are doing long-lived, compute-intensive work directly in the notebook I'd recommend to look at processing/training jobs instead.CommentShareEXPERTAlex_Tanswered 3 months ago"


Question: How to connect to RDS aurora MYSQL private DB using Boto3?

"I tried searching ways to connect to RDS aurora serverless MYSQL private db found a way here, but it is asking secret arn of the DB, unable to find that one outcan anyone help me with the way to generate secret ARN or a way to connect to  RDS aurora serverless MYSQL private dbi am able to fetch all remaining details for the option mentioned in the above link,steps i am following:created RDS aurora serverless MYSQL private dbconnected a ec2 instancegenerating python script to connect to rdsif you suggest any alternative, that is also fine,TIAFollowComment"

Answer: "1If you're referring to this documentation https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-api.html#data-api.calling.python then it will only work if you have an Aurora Serverless v1 (not v2) and a Data API enabled.When enabling the Data API, you'll be creating a secret, and its ARN then you'll be putting in the secret_arn variable in that python scriptCommentShareAWS-ADolganovanswered a year agoAWS-User-9443397 a year agocan you please explain how to connect to serverless V2 using python?Share"


Question: Encoding HLS with WEBVTT or DFXP as subtitles inputs

"With mediaconvert, I ave some issue setting up a job that support an sidecar input subtitle other than SRT.When I use a subtitle TTML, I get the error :Failed to initialize pipeline []. (IS)So I tried to setup a job with DFXP (subset of TTML) the type DFXP is not available in the web console.The same goes for the WEBVTT. is not available.Meanwhile the caption filename validator does give the following error :/inputs/0/captionSelectors/Captions Selector 1/sourceSettings/fileSourceSettings/sourceFile: Should match the pattern: /^((s3://(.*?)\.(scc|SCC|ttml|TTML|dfxp|DFXP|stl|STL|srt|SRT|xml|XML|smi|SMI))|(https?://(.*?)\.(scc|SCC|ttml|TTML|dfxp|DFXP|stl|STL|srt|SRT|xml|XML|smi|SMI)(\?([^&=]+=[^&]+&)*[^&=]+=[^&]+)?))$/It seams from the regex mediaconvert should be able to support TTML, DFXP.webVTT would be a nice addtition.JoelFollowComment"

Answer: "0Hi Joel,Thank you for opening this forum post! As you mentioned, DFXP is a text based subset of TTML, and the regex does suggest that a DFXP extension is supported. Can you confirm if you have tried a TTML caption selector with a .dfxp file extension? I think this should parse correctly. If not please let me know and I can look into it further. Thank you.Regards,BenCommentShareAWS-User-0287910answered 3 years ago0Hello Ben,I tried a DFXP with the TTML subtitle profil but mediaconvert createjob validation forms reject it, It limit the extention to TTML.when I use a file with the extension TTML the job is created but I dont have a subtitle in the outputed HLS...CommentShareJoelBanswered 3 years ago0Hi jbrun, apologies for the mis-information there. I have passed your request to our engineering team to add the dxfp extension support to the service. Regarding the TTML to HLS output, can you PM me an example job ID / region / account ID for this please? I will review the logs. Thank you.Regards,BenCommentShareAWS-User-0287910answered 3 years ago0Hello Ben, I replied with a PM.Thank you.CommentShareJoelBanswered 3 years ago0Hello, I retested some mediaconvert jobs with differentr sidecar Subtitles.when the input is TTML I get the error :Failed to initialize pipeline []. (IS)Error Code : 1010When I try the subtitle input WebVTT the validator rejects it.Only the sidecar subtitle input SRT works.do you have the same issues ?CommentShareJoelBanswered 3 years ago0Hi Joel,I ran a test using the settings from the Job ID you provided via PM, and my own example MP4 / TTML files. The job completed successfully so the settings are ok. It's possible the TTML is malformed or there is a timecode mismatch between the caption file and video file. Would it be possible to send me a copy of the details you just mentioned? IE, Job ID, source MP4 and source TTML? This should allow me to reproduce the same fault and I can check in more detail what the issue is. If you can generate a presigned URL for these and send via private message that would be helpful. Thanks!Regards,BenCommentShareAWS-User-0287910answered 3 years ago0Hi Joel, do let me know if you are still having issues with this workflow and I'll be happy to review your MP4 / TTML sources if you are able to provide them. Thanks!Regards,BenCommentShareAWS-User-0287910answered 3 years ago0We finnally choose to convert our sidecar subtitles to SRT as an input.CommentShareJoelBanswered 3 years ago"


Question: Can I join master account Landing Zone to another AWS Partner Org for billing purpose?

I plan to create Landing Zone (Control Tower) for my company. It will create the organization as well. I want to know if I still can join master account Landing Zone to another AWS Partner Org for billing purpose?FollowComment

Answer: "0You will not be able to join another AWS Org per the prerequisites, it states "The account that you wish to enroll must exist in the same AWS Organizations organization as the AWS Control Tower management account".More details can be found here: https://docs.aws.amazon.com/controltower/latest/userguide/enrollment-prerequisites.htmlCommentSharenaelanswered 7 months ago"


Question: Swap space being eaten up by greengrass

I see the following process on my GreenGrass device:java -Dlog.store=FILE -Dlog.store=FILE -Droot=/greengrass/v2 -jar /greengrass/v2/alts/current/distro/lib/Greengrass.jar --setup-system-service falseis this the main greengrass service ?I'm asking because it seems to use a lot of memory and seems to be growing all the time causing the swap memory to grow.jeteye@jeteye:/data/rt$ sudo smem --sort=swap -r  PID User     Command                         Swap      USS      PSS      RSS 4525 root     java -Dlog.store=FILE -Dlog   514860   353064   353386   354676 7067 ggc_user java -jar /greengrass/v2/pa    75560     2720     2986     3392 7957 root     /usr/sbin/streamcontroller     57640    60192    61500    63172 6029 root     /usr/bin/dockerd -H fd:// -    33704    21848    21848    21856and eventually  ( over 2.5+ hours) the swap process (kswapd0) is starting to take so much processor time that other processes (that need to handle live video)  don't get enough time and restart (this is a guess..still collecting more information).Is there a way to limit the amount of memory that GreenGrass uses over time ?FollowComment

Answer: "0Accepted AnswerHoi clogwog. Yes that's the main process. Did you find this link?https://docs.aws.amazon.com/greengrass/v2/developerguide/configure-greengrass-core-v2.html#jvm-tuningAlso you can use the systemResourceLimits configuration item of the Nucleus to limit the amount of RAM (and CPU) available to component processes.https://docs.aws.amazon.com/greengrass/v2/developerguide/greengrass-nucleus-component.html#greengrass-nucleus-component-configurationThe growth over time sounds concerning though.CommentShareEXPERTGreg_Banswered a year ago0Hoi Greg,no i didn't find that link. Very useful. Will set some limits and run it overnight again. It may just be because it is completely unrestricted it will just use it.Dank U well !CommentShareclogwoganswered a year agoGreg_B EXPERTa year agoAlstublieft. (And we've just about reached the end of my Dutch.)Share"


Question: AWS OCR and Unicode

"Does AWS OCR support Unicode characters? I want to scan pages that are written partially in the Apache language, which uses Unicode fonts for accents, tone, nazalizations and at least one non-Roman character, sometimes called the silent L or slashed L.FollowComment"

Answer: "0It would appear that your use case is not covered by Amazon Textract https://aws.amazon.com/textract/faqs/Q: What type of text can Amazon Textract detect and extract?Amazon Textract can detect printed text and handwriting from the Standard English alphabet and ASCII symbols. Amazon Textract can extract printed text, forms and tables in English, German, French, Spanish, Italian and Portuguese.CommentShareRWCanswered 2 days ago"


Question: ECS task defination: missing required attributes while deploying using AWS CLI

"Hello,Background:I am trying to create new revision for ECS task. I am successful in registration. But unable to run the new definition. I am working using AWS CLI (not console)Issue:After couple of hours spent on investigation, found that issue with Require attributes which ECR related stuff is not getting added.I need some one help, why ECR related require attributes are not getting added. Due to this task is getting failed to pull data from ECRJson entries for working revision and non working revision:I have fetched Json format using aws describe command for both working revision and non functional revision definationsWorking"requiresAttributes": [            {                "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"            },            {                "name": "ecs.capability.execution-role-awslogs"            },            {                "name": "com.amazonaws.ecs.capability.ecr-auth"            },            {                "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"            },            {                "name": "ecs.capability.secrets.asm.environment-variables"            },            {                "name": "com.amazonaws.ecs.capability.task-iam-role"            },            {                "name": "ecs.capability.execution-role-ecr-pull"            },            {                "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"            },            {                "name": "ecs.capability.task-eni"            }        ],Non working"requiresAttributes": [    {      "targetId": null,      "targetType": null,      "value": null,      "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"    },    {      "targetId": null,      "targetType": null,      "value": null,      "name": "ecs.capability.execution-role-awslogs"    },    {      "targetId": null,      "targetType": null,      "value": null,      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"    },    {      "targetId": null,      "targetType": null,      "value": null,      "name": "ecs.capability.secrets.asm.environment-variables"    },    {      "targetId": null,      "targetType": null,      "value": null,      "name": "com.amazonaws.ecs.capability.task-iam-role"    },    {      "targetId": null,      "targetType": null,      "value": null,      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"    },    {      "targetId": null,      "targetType": null,      "value": null,      "name": "ecs.capability.task-eni"    }  ],Task definition:Removed entries for security purpose.{  "containerDefinitions": [    {      "name": "bbbbbtask",      "image": "aaaaaaaaaa1.dks.ecr.us-east-1.amazonaws.com/somerepo:0BVoLej",      "cpu": 0,      "portMappings": [],      "essential": true,      "entryPoint": [],      "command": [        "bundle",        "exec",        "rails",        "xxxxxxxxxuyyy"      ],      "environment": [      ...........        }      ],      "mountPoints": [],      "volumesFrom": [],      "secrets": [      .......      ],      "logConfiguration": {        "logDriver": "awslogs",        "options": { ........        }      }    }  ],  "family": "xxxxxxxxxx",  "taskRoleArn": "xxxxxx,  "executionRoleArn": "xxxxxxx",  "networkMode": "awsvpc",  "volumes": [],  "placementConstraints": [],  "runtimePlatform": {    "operatingSystemFamily": "LINUX"  },  "requiresCompatibilities": [    "FARGATE"  ],  "cpu": "1024",  "memory": "2048"}Kindly help me to added ECR pull and ECR auth required attributes while deploying using AWS CLI.FollowCommentRobert Love a month agoCan you post the CLI commands you have tried?Share"


Question: How do you add permission for a scheduled lambda function to query a DynamoDB table?

"In my amplify application I need a lambda function to execute once a day and query a Database table for records matching certain criteria and send an e-mail if they are found.I added a secondary index in my graphql schema with the fields I wanted to use. I then generated a lambda function with amplify and added this statement to the execution policy:      {            "Action": [                "dynamodb:GetItem",                "dynamodb:Query"            ],            "Resource": "arn:aws:dynamodb:us-west-1:#redacted#:table/Request-#redacted#-prod",            "Effect": "Allow"        },I can see the permissions in the created role but when I test the function I get the following error message:"message":"User: arn:aws:sts::#redacted#:assumed-role/crnLambdaRole0600f8a6-prod/sendReminders-prod is not authorized to perform: dynamodb:Query on resource: arn:aws:dynamodb:us-west-1:#redacted#:table/Request/index/requestsByDate because no identity-based policy allows the dynamodb:Query action"How do I go about granting the correct permissions to my lambda function or is there some other reason I might get this error?FollowComment"

Answer: "2Accepted AnswerHi, you need to update the policy to allow you access to the index as well as the base table. Modify the Resource part of the policy to include the index:      {            "Action": [                "dynamodb:GetItem",                "dynamodb:Query"            ],            "Resource": [               "arn:aws:dynamodb:us-west-1:#redacted#:table/Request-#redacted#-prod",               "arn:aws:dynamodb:us-west-1:#redacted#:table/Request-#redacted#-prod/index/*"             ],            "Effect": "Allow"        },See more hereCommentShareEXPERTLeeroy Hannigananswered a year ago1Hi There.I understand you are finding issues when you add permissions for a scheduled lambda function to query a DynamoDB table  and you are getting an error :"message":"User: arn:aws:sts::#redacted#:assumed-role/crnLambdaRole0600f8a6-prod/sendReminders-prod is not authorized to perform: dynamodb:Query on resource: arn:aws:dynamodb:us-west-1:#redacted#:table/Request/index/requestsByDate because no identity-based policy allows the dynamodb:Query action"Please note that for a DynamoDB index, the resource path is different. So you need to add the index path as well in iamRoleStatements.For example:{        "Action": [            "dynamodb:GetItem",            "dynamodb:Query"        ],    “Resource”: "arn:aws:dynamodb:${self:provider.region}::table/${self:provider.environment.DYNAMODB_DATA}/index/”,        "Effect": "Allow"    },For further understanding please refer to the document [1]Please also note that the following need to be confirmedYour Lambda function execution role and that you have the necessary permissions for DynamoDB table on AWS Managed policy "AmazonDynamoDBFullAccess" attached to the role.Then further check you have permissions boundary attached to the role and it only has the table ARN permissions. Because you are querying on the index, you will be adding "arn:aws:dynamodb:${self:provider.region}::table/${self:provider.environment.DYNAMODB_DATA}/index/” to the resource as mentioned in the permissions boundary policy.I hope you find this information helpful.=========References==========[1] https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/using-identity-based-policies.htmlCommentShareMfaneloanswered a year agorePost-User-2043212 a year agoThank you for your answer, both you and Leeroy had the information I needed. I'm accepting Leeroy's answer since it is more succinct.ShareMfanelo a year agoThank you, I am happy that you have resolved your issue.Share"


Question: No visualization available

"Using AWS Log insights, I simply want to plot duration (unit is nanosecond, ideally shown as seconds) as the Y axis, instead it appears to count the matches. What am I missing please?@timestamp,duration2023-03-05 00:08:24.324,54409442023-03-05 00:08:15.052,161213532023-03-05 00:07:24.128,131369352023-03-05 00:07:21.830,139717992023-03-05 00:07:20.493,195365582023-03-05 00:07:12.665,139656872023-03-05 00:07:02.095,99883652023-03-05 00:06:49.970,177175792023-03-05 00:06:32.391,6515700FollowComment"

Answer: "0Hello hendry,Time series visualizations work for queries with the following characteristics:The query contains one or more aggregation functions.The query uses the bin() function to group the data by one field.When you run a query that uses the bin() function to group the returned results by a time period, you can view the results as a line graph, stacked area graph, pie chart, or bar chart. This helps you more efficiently visualize trends in log events over time.bin(period: Period)TimestampRounds the value of @timestamp to the given time period and then truncates. For example, bin(5m) rounds the value of @timestamp to 5 minutes before it truncates.CommentSharerePost-User-6766157answered 3 months ago"


Question: IAM Policy SFTP

"Reading this doc - Link: https://docs.aws.amazon.com/transfer/latest/userguide/monitoring.htmlSo, i have created 1 policy and attached it to new role.When i want to create second role i have the error:This policy contains the following error: Has prohibited field Principal For more information about the IAM policy grammar, see AWS IAM Policies{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Principal": {"Service": "transfer.amazonaws.com"},"Action": "sts:AssumeRole","Condition": {}}]}FollowComment"

Answer: "0it's not policy, it's relationship button, fixedCommentShareCarlmaxx11answered 4 years ago"


Question: Amplify multi-env setup causing my production build with wrong env access

"Hi, I use amplify react native to build both ios and Android application. I created two environment branch on both git and amplify - dev and production. Recently I built my application and ran into a very strange problem.Login in user (cognito from production env)Query data ( api graphql from production env)Force close the app or logout and login again (this cognito username exist on both production and development env)Query data (api graphql from dev env)As you can see from above, all the sudden my app switch to development environment even though the app build is intend for production. How is it possible to have mix env co-exist in my app? The way I handle env build is like this:When working on dev environment:git commit to devgit push to devamplify init or env checkout to devamplify push (if any changes)build the dev imageWhen switch to production environment:git checkout master which is production envgit merge dev to master which is production envgit push to master which is productionamplify env checkout to production envamplify push (if any changes)build the production imageThe way I make sure my amplify environment setup switch to production is by checking inside the aws-exports.js file. And as I stated above, my first launch always points me correctly to the production server. Then how is it possible to switch from a production to a development server? Is there any place I missed causing such sudden switch?I really hope to get feedback from the aws amplify development team. This issue is preventing my from launching my app to the store. It would be a disaster if I didn't run enough test before launch.In addition, I suspect the config might be cache somewhere in the device during my development testing, but tried a brand new device and the problem persist. So that is not the case.Thanks in advance for the support.Edited by: rmwh6 on Aug 29, 2019 9:00 PMEdited by: rmwh6 on Aug 29, 2019 9:01 PMEdited by: rmwh6 on Aug 29, 2019 9:02 PMEdited by: rmwh6 on Aug 29, 2019 9:05 PMEdited by: rmwh6 on Aug 29, 2019 9:08 PMEdited by: rmwh6 on Aug 29, 2019 9:09 PMEdited by: rmwh6 on Aug 29, 2019 9:10 PMFollowComment"

Answer: 0The problem has nothing to do with Amplify env. Issue has been discussed in another thread. Closed.CommentSharermwh6answered 4 years ago


Question: Not able to connect ssh which i have created vm from the existing AMI

"I am not able to connect ssh to the newly created vm using existing AMI backup and also tried creating vm by taking AMI backup from the running vm too.Either way i am unable to connect. Network and security group is all fine.I am pretty badly needed to run another instance of running vm by referring existing AMI or creating anything new AMI.Please help me to fix this :Instance-ID:i-0dea1177b03bc9c5dAMI:ami-0e69a19245c9c3248ami-0c95fb1ecf1c439f1Thanks,FollowCommentrePost-User-0424438 6 months agofyi, i am able create and connect with new vm on the same subnet and security group without any problem. But that is not working from existing AMI backup. No idea what is wrong with the IMAGE and how to fix it. Please help me to correct if anything i need to change. Please let me know we you need console log.Share"

Answer: "1Does the EC2 instance have a public IP address? If you are trying to connect from the AWS console using EC2 instance connect and it doesn't have a public IP address you won't be able to connect to it.If there is no public IP address  you should use AWS Session Manager which works without it, and make sure the instance role has the policy  AmazonSSMManagedInstanceCore attached to it.CommentSharekdambiecanswered 6 months ago0Accepted AnswerHi hayao-k and kdambiec, Sorry for the delay in responding and Thanks for the suggestion and that will truly helps in someway for sure.Funny thing and it was legacy vm from 2012, someone has hard coded local IP in SSH Listen config and that is carry forwarded it in the image. That was the mistake and it is fixed now. Thanks for the help and support :)CommentSharerePost-User-0424438answered 6 months ago0Are you using password authentication?Instances created from the AMI will automatically disable password authentication.To work around this, set password authentication to yes in /etc/cloud/cloud.cfg.ssh_pwauth: 1CommentSharehayao-kanswered 6 months ago"


Question: Freeze/close an account

"I told my team members not to use this account, and we tried to migrate to a different cloud account since last month.But AWS sent me a big surprising bill this month, there were   many unrecognized activities in our account.   I email and talked to AWS a number of time in last two weeks.   AWS staff checked my account, seeing no usage activities    as  we have deleted everything ( except  I failed to  delete the organization and the was some error message) . Meanwhile  AWS still keeps charging me everyday.It is really frustrating/upsetting  that AWS keeps charging me everyday   while we donot use anything (they have verified also).  Appreciate any advice.  ThanksFollowComment"

Answer: "0If you want to close an account you can do it by login using root user and go to account settings page at the bottom you can find account close option..https://aws.amazon.com/premiumsupport/knowledge-center/close-aws-account/CommentShareNitin Goyalanswered 5 months ago0Thank you for your answer.There is an AWS backup plan activated in my account. When I tried to remove it , the web page says that I donot have a permission and needs to ask the administrator. I am the account owner and has the root permission.AWS customer service says that they cannot delete this backup plan, meanwhile they do not know the reason, saying they ask other teams to solve, meanwhile AWS still runs this backup and charges me everyday.Any tip will be appreciated on how to handle that.CommentSharerePost-User-8356931answered 5 months ago"


Question: How to Push Step Functions to Repo

"Hello,My team and I are working on implementing Repos for our ETL functions (Glue, Step Functions and Lambdas). I was wondering if there was a way to push changes to step Functions to a Repo? I know I can copy and paste code out the Edit window, but that feels pretty manual. I was hoping for a less manual way of having Version Control over Step Functions changes, as our step Functions are pretty extensive and would be better off editing in the console and then pushing the update  with the click of  a few buttons.Thanks!FollowComment"

Answer: "0There is no such way currently. You will need to use the visual editor to create the initial state machine, you will then need to copy and paste it into a file and add it to the repo. Going forward, you can use the AWS toolkit in your IDE to make the changes to the json file, which the SDK can visualize. If you want to use the visual editor again, you will need to copy the file over again.CommentShareEXPERTUrianswered 7 months ago0Thank you for your response.Do you know if this is on AWS' roadmap to add version control integration much like there is for AWS Glue?CommentSharerePost-User-3191225answered 7 months ago0Perhaps you can use CLI/SDK to describe the state machine and get the definition and suggest also look for other output provided to consider also versioning those other information, not only the definition.For the CLI you can look at describe-state-machine for step functions.  For Lambda look at the get-function, that gives you link to download the deployment package.  SDK for different languages would have same functionality.CommentShareromerogtanswered 7 months ago0This feature is not natively supported by Step Functions.A workaround will be to have a process which does a describe-state-machine using the SDK/CLI, then save the output as your state-machine.asl.json and track the state machine  in your version control.When you would like to update a new version of your state machine you will have to make the update-sate-machine api call with the new state machine definition file.CommentShareAWS-User-1578347answered 7 months ago"


Question: Why do I get errors when MediaLive tries to write data to MediaStore ?

"I'm getting some errors when I try to do a live stream using MediaLive. In MediaLive, I have a RTMP(push) input and MediaStore configured as destination.For MediaStore I have configured container policy like here:https://docs.aws.amazon.com/mediastore/latest/ug/policies-examples-default.htmlI enabled Cloudwatch logs for the MediaLive channel and these kind of PUT failed errors show whenever I start streaming.Setting alert [6006] [1-2] [6006-1-2] [OutputDataS3: PUT failed [mediastore://xyz.data.mediastore.us-west-2.amazonaws.com/mystreams/stream_720p30_00005.ts] size [549148]. Response=[403] Message[] Headers[x-amzn-RequestId: EHYZJ3SDA4KJBHRSPELTDV2KW5FUAWVDJYWBC2VPYR7WHWOBDNFGIH3HEAJUJ]Also in the channel's alert section, I see erros like this:Failed to Create Output File or SocketMPEGTS muxer for mediaID [1] unable to open output or stream [mediastore://xyz.data.mediastore.us-west-2.amazonaws.com/mystreams/stream_720p30.000007.m3u8].For debugging purposes I've gave full access to the IAM role that is associated with the channel.Any idea on what I'm missing ?Edited by: adrianmg on Jun 22, 2020 10:39 PMEdited by: adrianmg on Jun 22, 2020 10:42 PMFollowComment"

Answer: "0Accepted AnsweradrianmgWhen writing to AWS Elemental MediaStore you should use "mediastoressl://" instead of "mediastore://".See https://docs.aws.amazon.com/medialive/latest/ug/hls-protocol-fields.html & https://docs.aws.amazon.com/medialive/latest/ug/hls-destination-complete-fields-step.html as reference.Additionally in the HLS Output Group settings - HLS Settings - CDN Settings should be set to HLS MediaStore.CommentShareAWS-User-5437070answered 3 years ago0mediastoressl did the trick. I'll also change the hls webdav to hls media store. Live stream also works with webdav, but I need to read more about the difference between them.Thanks @fxeataws !CommentShareadrianmganswered 3 years ago"


Question: About DMS Costs

"dms costAccording to the above web page on DMS costs, the listed fees are added on an hourly basis depending on the instance type.Question.Is there a charge for just having a replication instance (even if the database migration task is not running)? Or is the fee charged only when the database migration task is running?For example, if I create a replication instance and run the database migration task for 2 hours a day with an api like boto3 below, do I have to pay for only 2 hours of usage?boto3 apiFollowComment"

Answer: "3Accepted AnswerTo save costs on the running DMS instance you could resize it to a smaller t3 or t2 instance depending on the region, when it is not performing a replication task. You cannot stop the instance in DMS as you pointed out. If you automate the creation of the instance and the replication tasks you could look at creating it each time you needed to use it. An example of this is shown in this blog https://aws.amazon.com/blogs/database/automating-database-migration-and-refreshing-activities-with-aws-dms/ using cloudformation. You should be able to convert this to gain an idea of automating the setup and tear down of DMS. It will take some time so plan on starting the DMS resource creation at least an hour before your window.CommentShareAWS-JRG-1317534answered a year ago1DMS makes use of an EC2 instance to act as a replication instance. EC2 is pay as you go, so if you stop an instance, AWS will shut it down so you will not be charged for the usage of the stopped instances or any data transfer fees. What you do get charged for is other resources that are attached to your stopped instances such as storage of any EBS volume or S3 storage. So to answer your question, if the instance is powered on, you'll be charged for the compute (EC2) being used, as well as the data transfer, storage, etc. If the instance is powered off, you'll only be charged for the storage attached to the instance.Take a look at our AWS Pricing Calculator and play around with the numbers to see exactly what you'll be charged for when using DMS.CommentShareAWSJoeanswered a year agoYasshi-Cookie a year agoDear AWSJoeThank you for your reply.If the instance is powered off,You mean that stop replication instance, right?Is it possible to stop the replication instance?Additionally, when using DMS, is it usually necessary to keep the replication instance running at all times?Best Regards,ShareAWSJoe a year ago@Yasshi-Cookie. The DMS Replication Instance is a fully managed EC2, so you don't have the ability to power down as you would a regular EC2 instance. With DMS, after your instance is running you have the ability to reboot, modify, delete. Generally, customer leave the instance running for the duration of their DB migration, then delete the instance after their data has successfully been migrated.ShareYasshi-Cookie a year ago@AWSJoeThank you very much.I see. Usually, We need to keep a replication instance of DMS running during migration. Also we can delete the replication instance but cannot power down the instance.Share"


Question: List of AWS Spice datasets and it's size

"In Redshift, we can use below query to get list of tables and it's sizeselecttable,sizefromsvv_table_infowhere "database" = 'dev' and "schema" = 'public'Do we have anything similar in quicksight?FollowComment"

Answer: "1Hi,you could use the QuickSight APIs to list the data sets in the response there is a attribute named ImportMode filtering the results where this attribute is set to SPICE you can retrieve all the DataSetId and Name for the dataset.Then looping on the list of datasets you could use the API describe_data_set to retrieve details on the dataset , one of the attribute is: ConsumedSpiceCapacityInBytesAlso, for QuickSight related questions we have launched a fully dedicated community; in case you want to check there to see if there are further answer to your questions.hope this helps,CommentShareEXPERTFabrizio@AWSanswered a year agoAWS-User-2264889 a year agoHi,Thank you for the response. We did use this but challenge here is, it's difficult to read the output and get it into a tabular formatAnd is there a way to restrict response of describe datasets and list datasets? Only thing we need in the result is Name, ConsumedSpiceCapacityInBytes and maybe couple of other thingsShareFabrizio@AWS EXPERTa year agoThank you for the feedback, currently I am not aware of other methods, as for restricting the response currently aside from scripting it there is no other option I will bring the feedback back to the team.Share"


Question: Cloud function with better GPU

I'm currently experimenting with the usage of Lambda to render videos using puppeteer. The lack of a GPU combined with the 15 minutes maximum duration limits the duration of the videos that can be rendered. Is there an infrastructure alternative where I could retain the serverless mode but have more options on the hardware?FollowComment

Answer: "1You can consider using AWS Fargate with Amazon Elastic Container Service or Amazon Elastic Kubernetes Service for a serverless compute option without the 15 minutes duration constraint. Alas, Fargate wouldn't allow you to request GPU support either.CommentShareDmitry_Ganswered a year agoNuthinking a year agoGPU presence is important.ShareNuthinking a year agoLooks like that once I start requiring GPU in AWS Jobs, then the startup time becomes ridiculous. What's the cost/complexity difference between the option you proposed and AWS Batch? Thanks!Share0You can't specify the runtime environment for AWS Lambda functions, so no, you can't require the presence of a GPU.But you could run GPU workloads on  AWS Batch, by using an AMI with GPU support.https://docs.aws.amazon.com/batch/latest/userguide/batch-gpu-ami.htmlAWS Batch automatically provisions compute resources. There's no need to install or manage batch computing software.  AWS Batch jobs are not limited by the 15 minute runtime.Hope this helpsCommentShareDennis_Aanswered a year agoNuthinking a year agoYeah, I read about AWS Batch, I'm looking for tutorials on how to use it with a node script at the moment. Any clue? Thanks!ShareNuthinking a year ago@dennis_a I'm struggling to run puppeteer on EC2 and these instructions I found don't seem to work: https://www.cloudsavvyit.com/13461/how-to-run-puppeteer-and-headless-chrome-in-a-docker-container/ShareNuthinking a year agoLooks like I could get it to work following the settings used on this repo: https://github.com/beemi/puppeteer-headfulShareNuthinking a year agoLooks like that once you are a bit more specific about the hardware requirements (i.e. GPU) it can take many minutes for a job to start. This means that I can have longer processes than 15 minutes. But that I can't really use hardware acceleration.Share"


Question: Oracle sys.utl_recomp grant permission to user

"Is it possible within AWS RDS Oracle to grant sys.utl_recomp permission to a user?exec utl_recomp.recomp_parallel(8);The command does work as master user for the database, but I need to be able to grant permission to execute this command to another user. I've not been able to find information on how to complete this task. The command tasks page for AWS shows how to assign sys objects, but it doesn't work for the package utl_Recomp.All help appreciated.FollowComment"

Answer: "0Accepted AnswerHi there!The master user in RDS Oracle should be direct-granted execute privileges on SYS.UTL_RECOMP, so it can grant the privilege to other users via the GRANT statement. For example,grant execute on sys.utl_recomp to otheruser;It can also be granted using the RDSADMIN_UTIL package as described at https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.Oracle.CommonDBATasks.System.html#Appendix.Oracle.CommonDBATasks.TransferPrivileges (note, you will need to specify the grantee username in upper case with this command).exec rdsadmin.rdsadmin_util.grant_sys_object('UTL_RECOMP', 'OTHERUSER', 'EXECUTE');Hope this helps. Thanks!MichaelCommentShareAWS-User-9938760answered 4 years ago0Thanks Michael, sometimes I get so deep in the weeds I cannot see the answer.Appreciated.CommentSharefoxxf1reanswered 4 years ago"


Question: instance status unknown - stop process hung

"We have a lightsail instance in Oregon, Zone A (us-west-2a) that has a status 'unknown' and we're unable to stop it - tried to stop it over an hour ago and it's still trying to stop, still in a 'stopping' stateAny suggestions besides waiting it out?FollowComment"

Answer: "0Hi, thanks for using Lightsail. We checked your instance in us-west-2a and it is under running state now. Did you still meet the same issue with it?ThanksCommentShareAWS-User-3138555answered 4 years ago0We waited around 3 hours and the instance eventually stopped and we were able to re-start it.CommentSharedan3zanswered 4 years ago"


Question: Binary Logging must be enabled for MySQL server

Im trying to run a CDC task using AWS DMS from our Beta DB to our Dev DB and Im getting the following error:Binary Logging must be enabled for MySQL server; Errors in MySQL server binary logging configuration. Follow all prerequisites for 'MySQL as a source in DMS' from https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MySQL.html or'MySQL as a target in DMS' from https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.MySQL.html ; Failed while preparing stream component 'st_0_core-beta'.; Cannot initialize subtask; Stream component 'st_0_core-beta' terminated [1020418]  (replicationtask.c:2888)I have enabled the binlog_format to set to ROW and Im using the writer cluster endpoints for both the beta and dev DB. Is there any other parameter I need to change/set please?FollowComment

Answer: "0Accepted AnswerGreat thanks. Also, I was using the reader instance and not the clusters writer, I think that was the problem.CommentSharemarc-epropanswered a year ago0Hi, @marc-epropYou can check other parameters from the following documents.https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MySQL.html#CHAP_Source.MySQL.AmazonManagedOther parameters such as binlog_row_image and binlog_checksum are also mentioned here.CommentShareEXPERTiwasaanswered a year ago"


Question: How do I customize/limit content for multiple customers on a static S3 site?

How do I customize/limit what a web user (customer - not administrator!) sees on my static S3 site?  I have searched high and low for which approach to take.  I've successfully limited access with CloudFront & Lambda but that approach only allows one user/password for all restricted content.  I would like my customers to log on individually (lowest level) and have content filtered for each customer.  I do not want my customers to see other customers or other customer's content.  I have tried following "Signed cookie-based authentication with Amazon CloudFront and AWS Lambda@Edge: Part 1 & 2" but boy did that leave a gaping hole between the two documents (i.e. where do you point the login form???)!I think I am looking for:Individual customer sign on / password (cognito?)Filtering/limiting content (by prefix?) determined by cookie or token or ???Please just point me in a direction?FollowComment

Answer: "0Accepted AnswerHi sschlegelmilch - Thank you for adding additional details!Here is how I would set this up.Implement Cognito on the static app. Full disclosure, I have never done this, but the goal is to get a JWT generated when a user logs in. Per the Cognito documentation that I linked above, it looks like the JWT is generated by default on a successful login so I hope that this step is not too difficult.When the user successfully logs in, set the JWT generated above as a cookie.In CloudFront, create a new cache behavior for /pricing (I am making this up, but some path that should be protected). In this behavior you will want to use this CloudFront function to handle checking the JWT and verifying if the request should be allowed/denied. This Lambda might require a few small modifications to check the JWT token in a cookie value, as I think it checks for a JWT in a query string by defaul, but it isn't too difficult. If you get stuck let me know.Change the default error behavior in the CloudFront function above to issue a redirect back to the Cognito login page.So now a user will access your site, and can browse around just fine. When they go to the Cognito login and successfully login they will get a JWT cookie. When they go to the protected path (/pricing) they will hit the Lambda that checks their JWT token. If the cookie exists and the JWT token is valid then the request is allowed through and the user can see the protected content. If the JWT token is missing, or is invalid then the user will be redirected back to the Cognito login page.Again, I haven't set this up myself personally, but I believe this should work.CommentShareJeremyAWSanswered a year ago0Hi @sschlegelmilch,Without knowing the specifics, I will make some assumptions. Please feel free to reply and add additional details if you think it will help.I am going to assume that your static site that is running on S3...Does not have user authentication built-in, and you are trying to find a way to add it.Has a specific URI that is unique to each user, and you want to make sure user A gets their unique content, user B gets their content, etc. For example, a page named /myaccountStarting with #1, you can add authentication to your site using AWS Cognito. Take a look at the integration code snippets half-way down the Cognito page found here. For a deeper dive, check out the Cognito documentation here. Once a user is logged in, you can route that user within your web application accordingly.At this point, I am assuming you have authentication setup correctly, and you are able to route individual users within your web application. However, if you add in any caching (like CloudFront for instance) then you risk serving cached content to the wrong user. For example, user A hits /myaccount and the page is cached. User B attempts to access /myaccount and they get user A's cached version of the page.  To resolve this, you need to make sure that you are creating a unique cache key for each version of the page (user A's page, user B's page, unauthenticated users, etc). To accomplish this you need to know how your application identifies authenticated users. In the case of Cognito, this is done using a JWT passed in the query string. So using that example, you would go into your CloudFront cache policy, and add the appropriate query parameter to the cache policy. This will now include the specified query parameter in the cache key, so each user will have their own unique copy in cache.Please let me know if any of my assumptions are incorrect, or if you have additional questions.CommentShareJeremyAWSanswered a year agosschlegelmilch a year agoJeremy,I also thank you for taking time to address my question.  I really didn't expect someone from AWS to answer, but that is impressive!Your assumptions...Yes, I am running a static website on S3.  I have blocked all public access to the S3 bucket and have been using CloudFront & Lambda to access both public (index, login, css, js files...) and restricted content.However,I have tried two methods to PERSONALIZE content...The first lambda I tried worked fine by requiring web users to enter a username/password to go past the login/index pages.  Again, it worked well except for the fact that once a user had authenticated they had access to ALL restricted content.  I wanted to go one level deeper...So I followed (several attempts) at using cognito to perhaps set a cookie that I could read later on a login page.  I tried to follow the AWS tutorial entitled "Signed cookie-based authentication with Amazon CloudFront and AWS Lambda@Edge: Part 1 & 2".Sidenote... Many times I have encountered several of the "how-to" documents on AWS to be outdated (i.e. referencing menu items that have changed, etc.), but that is another discussion. I got stuck between Part 1 & 2.  (to be continued...)Sharesschlegelmilch a year agoI was thinking I could use javascript to set the login button to point to the individual customer page.  I was HOPING that a cookie could be set so that individual customers could only see files with their username (or account #) in the prefix ('folder') of their customized content and would restrict them from seeing other's content (and identity/# of other customers).Your caution of CF serving up cached content is interesting and I will study that.  Also, whether to use tokens or cookies does not matter to me as long as it works.*I will take your advice and study using Cognito to set JWTs for unique identifiers.Question...I would like to verify (with you) that setting a JWT will allow me to restrict web users to certain prefixes of content?Let me give you an example...  As I said before, I am trying to give public access to 'assets' (css, js, etc.) and product info but I want to individualize pricing information.  Does that make sense?Again, I appreciate your help!  Really!!P.S. It would be great to have someone from AWS critically step through the two part document I referenced above.  That MAY have been a solution for me but I believe there is a gap between the two parts.  Part 1 illustrates using a form to authenticate but does not indicate what "get action" the form should point to.Share0Hi, @sschlegelmilchYou can authenticate by delivering a front-end website from CloudFront and accessing the back-end Lambda from JavaScript.However, in general, it is necessary to communicate from JavaScript to the backend via HTTP / HTTPS, so use Lambda Function URL / API Gateway + Lambda to enable communication from the frontend JavaScript to the server-side API server. recommend.Besides Lambd, you can also use containers and EC2. In either case, JavaScript delivered from CloudFront communicates via HTTP / HTTPS.CommentShareEXPERTiwasaanswered a year agosschlegelmilch a year agoiwasa,Thank you for taking the time to address my question.  I appreciate that.  That said, your answer to me is confusing.  Yes, I do understand and currently use CloudFront to connect to my S3 bucket.  I have also tried two different methods of using Lamda to add authentication through Cloudfront and the first method worked just fine.  What I don't follow is your reference to HTTP / HTTPS.  That has me scratching my head.  Yes, of course I am using HTTP (redirect to HTTPS) / HTTPS but I have no idea what that has to do with anything.  Sorry for MY confusion.Share"


Question: "If only one of the tunnels of site to site vpn is down, is communication possible?"

"One of the tunnels of site to site vpn is down and one is up.In this case, is it possible to communicate with the vpn network and the network on AWS?FollowComment"

Answer: "3Accepted AnswerYes, it is certainly possible, even with 1 Tunnel you could have bi-directional connectivity.A Site-to-Site VPN connection consists of two VPN tunnels between a customer gateway device and a virtual private gateway or a transit gateway. It is recommended that you configure both tunnels for redundancy. From time to time, AWS also performs routine maintenance on your VPN connection, which might briefly disable one of the two tunnels of your VPN connection. For more information, see Tunnel endpoint replacement notifications.CommentShareEXPERTTushar_Janswered 10 months agoEXPERTBrettski-AWSreviewed 10 months ago"


Question: ACM certificate validation status is still pending and CNAME record isn’t added to the correct DNS configuration

"I created a hosted zone for a valid registered domain ” skillbuilder.training.aws.dev” in Route 53. Then requested a public certificate with DNS validation for the domain name in ACM and also created the record accordingly. This added CNAME record in the hosted zone(Route 53).Now, the certificate is still under pending stage. When I tried the test record in Route 53, it returns empty response.I verified the CNAME,there is no additional characters or is missing charactersthere is no repeating bare domain to the end of its DNS recordsI also tried the dig command on generated CNAME, getting Record not found.Please provide some assistance.FollowComment"

Answer: "1Accepted AnswerHello There,ACM is unable to verify the CNAME record due to that you're seeing the state "pending".If you added the CNAME in the record correctly and the dig is not showing the response, then it might be the case where you have added to the wrong hosted zone (meaning you might have multiple hosted zones).Check your hosted zone NS server records where you have added the CNAME record are matching to the below set or not.%dig NS skillbuilder.training.aws.dev +shortns-1505.awsdns-60.org.ns-1811.awsdns-34.co.uk.ns-856.awsdns-43.net.ns-3.awsdns-00.com.if not, most likely you need to find on which account you have this Hosted zone and you need to add their.Regards,CKCommentShareChethan SEanswered 10 months ago"