Question: "why sagemaker is not giving free micro instance , when only micor instance is free for free tier? then how to use sagemaker  free as promised ?"

"why sagemaker is not giving free micro instance , when only micor instance is free for free tier?then how to use sagemaker  free as promised ?Free tierAs part of the AWS Free Tier, you can get started with Amazon EC2 for free. This includes 750 hours of Linux and Windows t2.micro instances (t3.micro for the regions in which t2.micro is unavailable), each month for one year. To stay within the Free Tier, use only EC2 Micro instanceshttps://aws.amazon.com/ec2/pricing/FollowComment"

Answer: "0Hi,Amazon SageMaker is free to try (Amazon SageMaker 2 Months). As part of the AWS Free Tier, you can get started with Amazon SageMaker for free. Your free tier starts from the first month when you create your first SageMaker resource.250 hours per month of ml.t3.medium on Studio notebooks OR 250 hours per month of ml.t2.medium or ml.t3.medium on on-demand notebook instances-25 hours per month on ml.m5.4xlarge on SageMaker Data Wrangler10M write units, 10 M, read units, 25 GB storage per month on SageMaker Feature Store50 hours per month of m4.xlarge or m5.xlarge instances on Training125 hours of m4.xlarge or m5.xlarge instance per month on InferenceSageMaker Free Tier: https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRank&all-free-tier.sort-order=asc&awsf.Free%20Tier%20Types=*all&awsf.Free%20Tier%20Categories=categories%23ai-mlThe details of the free tier and pricing for Amazon SageMaker are in the following link:  https://aws.amazon.com/sagemaker/pricing/?nc=sn&loc=3&refid=ft_cardCommentShareAWS-User-9882077answered 23 days ago"


Question: Can you use an Amazon ECR private repository with LightSail?

It says on this page that Container image sources can come from a "Public Registry" or "Images pushed from your local machine" what is not clear is if LightSail can use images stored in a private ECR repository? If not why not?Thanks.FollowComment

Answer: "1As of today, you can use ECR private repositories with Lightsail:https://aws.amazon.com/about-aws/whats-new/2022/05/amazon-lightsail-containers-deploying-images-amazon-ecr-private-repositories/CommentSharesisira_nanswered a year ago0As per my understanding your container images needs to be pushed directly to Lightsail where they will be kept private. Please refer the below link for more details.https://lightsail.aws.amazon.com/ls/docs/en_us/articles/amazon-lightsail-pushing-container-images#push-container-imagesCommentShareSUPPORT ENGINEERNirmal_Kariaanswered a year ago"


Question: AWS Greengrass - Unit test components

"Hi there,I am trying to develop unit tests in python language for simple AWS Greengrass components, using pytest framework and simulating a temperature sensor.The point here is that I have a SonarQube process running right now locally on my computer, and its main purpose is to check both the code quality and the testing coverage. The first task was easy to make it work, but for testing task, I am able to achieve it only if I ignore the AWS parts, but that is not the way I would like to follow, the whole code should be checked.Is there any guide I might follow in order to achieve it?Thanks beforehand, looking forward to hearing from you.Kind regardsDaniel.FollowComment"

Answer: "0Hi Greg_B,First of all, thank you very much for your reply, I was able to achieve one unit test publishing a message to AWS Cloudwatch metric by using the AWS Greengrass component SDK, yet I have many doubts maybe you can solve me, since I cannot see how this procedure is actually working.I am getting a passed test for the one previously indicated,  which its code is attached below:    def test_ipc_connect(self, mocker):        ipc_connect = mocker.patch('awsiot.greengrasscoreipc.connect')        mocker.patch('json.loads', return_value='1')        init_ipc_client()        ret_cpu = publish_message(ipc_connect, aws_cloudwatch_metric_payload("Greengrass-CoreDevice",                  "System.Cpu.Usage", "hostname", "co2sink-dev-CoreDevice02", get_cpu_usage()),                 "cloudwatch/metric/put")        ipc_connect.assert_called_once()        assert ret_cpu == 1But what really worries me is how it works and how the credentials are internally managed by the mock framework. As fas as I concern, the components that are running within the AWS Greengrass software work since it manages everything by using its own credentials giving at the installation process. What is it really being tested? Is it some kind of simulation?I have a 75% coverage, which is absolutely rather good, but I am quite demanding, so I would like to have a 90% at least, the problem here is I am not able to test all exceptions that the AWS Greengrasas component SDK might throw.https://imgur.com/k4k0mX0How can I test those cases? That is to say, the wrong ones (the publish message method is one from your example codes, but modified to fit in my application).I have tried something like this, but I am sure that is not ok, because in order to achieve the test properly, the exception should be thrown first within the publish_message method:    def test_get_secret_exception(self, mocker):        ipc_connect = mocker.patch('awsiot.greengrasscoreipc.connect', side_effect=Exception('mocked exception'))        with pytest.raises(Exception) as e:            init_ipc_client()            publish_message(ipc_connect, aws_cloudwatch_metric_payload("Greengrass-CoreDevice",                            "System.Cpu.Usage", "hostname", "co2sink-dev-CoreDevice02", get_cpu_usage()),                            "cloudwatch/metric/put")            assert e.type == SystemExit            assert e.code == 1        ipc_connect.assert_called_once()https://imgur.com/WectiAfThanks beforehand, looking forward to hearing from you.Kind regards.Daniel.CommentSharedpergarcanswered a year agoGreg_B EXPERTa year agoWhat is it really being tested? Is it some kind of simulation?Generally speaking, all of the GG library calls should be mocked so you only test the logic of your code. There's no real credentials being used in your test case.How can I test those cases?For each error, you have both a return value and a raise. But the raise will prevent the return value being returned.To test the raises, you would need to mock the future_response.result() call and use a side_effect to throw the right exception.ShareGreg_B EXPERTa year agoSide effect example here: https://github.com/awslabs/aws-greengrass-labs-component-for-home-assistant/blob/0056ab6f23ac9949cf6d1b561ac6635e756da663/tests/test_artifacts_secret.py#L20However, you should possibly read more widely on Pytest and mocker as this is all outside of AWS scope.Sharedpergarc a year agoHi Greg_BThank you very much for your reply, I will try the exception approach you have just mentioned before, but first I have to replace everything developed by pytest to nosetests, since it is the tool used by my work colleagues, so I will let you know how it goes.Kind regards.Daniel.Share0Hi Daniel,Not a guide, but you can find a small Python unit test example here, showing how to mock the IPC: https://github.com/awslabs/aws-greengrass-labs-component-for-home-assistant/blob/main/tests/test_artifacts_secret.pyThis is the code under test: https://github.com/awslabs/aws-greengrass-labs-component-for-home-assistant/blob/main/artifacts/secret.pyBeyond the IPC, you can find more Python unit tests in the directory: https://github.com/awslabs/aws-greengrass-labs-component-for-home-assistant/tree/main/testsAnd in this project: https://github.com/awslabs/aws-greengrass-labs-component-for-the-things-stack-lorawan/tree/main/testsHopefully that is enough to get you going.CommentShareEXPERTGreg_Banswered a year ago"


Question: S3 Intelligent - Tiering - 'Monitoring and Automation' costs

"Hi,We have a large amount of small files (< 1KB) randomly mixed with some large files (300 MB).If we would use 'S3 Intelligent - Tiering' the small files are not eligible for auto-tiering (which is quite logical, because putting them in the 'Infrequent Access Tier' would actually be more expensive).In order to determine if 'S3 Intelligent - Tiering' is cheaper than 'S3 Standard' the storage cost savings of the large files have to outweigh the 'Monitoring and Automation' costs.So the question is: Do small files (< 128KB) count to the number of object for the "Monitoring and Automation" costs?Thanks,RutgerFollowComment"

Answer: "0Hi Rutger,Thanks for posting on the forums.You are right in pointing out that small files are not eligible for auto tiering. So, for example, if you use a lifecycle policy to move your objects to INT after a specified time frame, objects that are less than 128 KB will not move to INT and you will not incur any monitoring charges for those. Objects that are bigger than 128 KB will be moved to INT and will continue to be monitored and automatically moved to infrequent access tiers based on usage.Hope this helps,SouvikCommentShareAWS-User-2349703answered 3 years ago"


Question: Storage Gateway Appliance

My initiator (Windows VM) is capable of Jumbo Frames and I was hoping the Target could be as well (Storage Gateway Appliance (VM)Is there a way to do this on the appliance? I don't see the command(s) in the appliance when I type Help inside the VM...FollowComment

Answer: "0Accepted AnswerThank you for asking, but the Storage Gateway Appliance (VM) does not support Jumbo Frames.CommentShareBill_Hanswered a year agoAWS-User-0688903 a year agoThank you for the insight. Perhaps AWS should add it in the documentation if not already done so.Share"


Question: Old EFS Access Points work but cannot mount a new one?

"I am using EFS Access Points from an EC2 instance. I had initially set the access points I needed and can mount those with no problem using the efs file type, eg:sudo mount -t efs -o tls,accesspoint=access-point-1-id efs-fs-id:/ mnt1This works and I can see and update files in the mounted file system. I recently added new access point under the same file system. However when I attempt to mount the new access point I get the following:sudo mount -t efs -o tls,accesspoint=access-point-2-id efs-fs-id:/ mnt2mount.nfs4: access denied by server while mounting 127.0.0.1:/Exception in thread Thread-1 (most likely raised during interpreter shutdown):Traceback (most recent call last):_File "/usr/lib64/python2.7/threading.py", line 804, in _bootstrap_innerFile "/usr/lib64/python2.7/threading.py", line 757, in runFile "/sbin/mount.efs", line 796, in poll_tunnel_process<type 'exceptions.TypeError'>: 'NoneType' object is not callableYet the original Access Point can be still mounted with no problem.These are both under the same Mount Target for the Subnet, which is available. The Network Interface shows up in the EC2 console as in-use and the associated Security Group has the NFS port accessible. It's certainly allowing access to the first Access Point.I have tried deleting the Access Point and the Mount Target. I then recreated the Access Point and added a Mount Target back, but I have the same result, the old Access Point mounts but I cannot mount the new one.My question is, why is access to the newly added Access Point denied? Have I forgotten to add the Access point to another security list or is there something in the system I need to restart for the new Access Point to be noticed?Edited by: JSDev on Sep 20, 2020 11:29 AMFollowComment"

Answer: "0Could you share how you are configuring your Access Point? One thing to make sure of for new EFS Access Points is that the UID/GID you are configuring for your enforced identity has at least read+execute access for its home directory. For example, if you did not specify a home directory, it will default to /, which by default is not accessible by any users other than root.CommentShareAWS-User-4040733answered 3 years ago0The access points are defined very simply, they both share the same filesystem, they both have paths defined and I used names to identify them. So they both look like:efs-file-system, access-point-1-name , /path/ap1/efs-file-system, access-point-2-name, /path/ap2/I did not add any of the optional POSIX user or creation user definitions.  For the older Access Point this is working and files can be written and updated in the directory.  I never get to attempt to write to the new Access Point as it's refusing to mount :(Edited by: JSDev on Sep 21, 2020 9:35 AMCommentShareJSDevanswered 3 years ago0Thanks for the details. Could you PM me your access point id? I can look from our side and see what's going on.CommentShareAWS-User-4040733answered 3 years ago0Thanks for the info. In your access point configuration, you left the enforced identity (UID/GID) blank, which means EFS is going to trust the identity passed from the server, which will be the UID/GID that you are logged into your server as. Also, you are specifying a directory without CreationInfo, which means the directory won't be automatically created, and must already exist and be readable+executable by your server UID/GID. Therefore, there is one of two reasons why this is not working:The directory in your second access point does not already exist in your EFS file system.The directory exists, but the UID/GID you're logged into your server as doesn't have at least read+execute access to it.The best way to make sure you don't get permission denieds with Access Points is to specify both the UID/GID of your user and use CreationInfo with your directory to make sure it gets created. For instance, you can create an access point with uid:1000, gid:1000, directory:/mydirectory, owneruid: 1000, ownergid:1000. With this, the directory will be created for you, and the owner information will match that of the access point, so you know you'll have access to it.CommentShareAWS-User-4040733answered 3 years ago0Ok Thanks, that was the missing step, creating the directories in the efs file system before or after creating the mount points.  They need to exist before attempting to mount, otherwise you are correct the creation user and POSIX user id's need to be specified.CommentShareJSDevanswered 3 years ago"


Question: amplify npm not installing all packages

"I have a weird situation with amplify console frontend not installing all of the packages and thus I am getting a module not found error.It worked initially but I had errors with environment variables that I had to fix. I then fixed the variables and nothing I can do will allow me to get past this point. I reverted everything and I still get the same error.Things I have triedRemoving the cache in the yml file and node 12 the nvm environment.Removing all node_modules and directory prior to running npm ciUsing npm install instead of npm ciUsing npm install --force (to bypass cache)updating my package-lock.json filerunning locally with npm ci which, I receive ("added 1963 packages in 33.668s") and it works.Here is my yml file:version: 1frontend:  phases:    preBuild:      commands:        - npm ci    build:      commands:        - npm run build  artifacts:    # IMPORTANT - Please verify your build output directory    baseDirectory: build    files:      - '**/*'  cache:    paths:      - node_modules/**/*Here is the error2020-07-10T15:35:02.103Z [INFO]: # Executing command: npm ci2020-07-10T15:35:05.346Z [INFO]: added 170 packages in 2.427s2020-07-10T15:35:05.352Z [INFO]: # Completed phase: preBuild                                 # Starting phase: build                                 # Executing command: npm run build2020-07-10T15:35:05.500Z [INFO]: > team-chat@0.8.0 build /codebuild/output/src889768276/src/pubnub-chat                                 > node setup/populate.js --quick-test && react-scripts build2020-07-10T15:35:05.678Z [WARNING]: internal/modules/cjs/loader.js:716                                    throw err;                                    ^                                    Error: Cannot find module 'cli-progress'                                    Require stack:                                    - /codebuild/output/src889768276/src/pubnub-chat/setup/populate.js                                    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:713:15)                                    at Function.Module._load (internal/modules/cjs/loader.js:618:27)                                    at Module.require (internal/modules/cjs/loader.js:771:19)                                    at require (internal/modules/cjs/helpers.js:68:18)                                    at Object.<anonymous> (/codebuild/output/src889768276/src/pubnub-chat/setup/populate.js:3:22)                                    at Module._compile (internal/modules/cjs/loader.js:868:30)                                    at Object.Module._extensions..js (internal/modules/cjs/loader.js:879:10)                                    at Module.load (internal/modules/cjs/loader.js:731:32)                                    at Function.Module._load (internal/modules/cjs/loader.js:644:12)                                    at Function.Module.runMain (internal/modules/cjs/loader.js:931:10) {                                    code: 'MODULE_NOT_FOUND',                                    requireStack: [                                    '/codebuild/output/src889768276/src/pubnub-chat/setup/populate.js'                                    ]                                    }2020-07-10T15:35:05.682Z [WARNING]: npm2020-07-10T15:35:05.684Z [WARNING]: ERR! code ELIFECYCLE                                    npm ERR! errno 1                                    npm ERR! team-chat@0.8.0 build: `node setup/populate.js --quick-test && react-scripts build`                                    npm ERR! Exit status 1                                    npm ERR!FollowComment"

Answer: 0I figured out the issue. It was my own issue with devDependencies not loading because env is set to prod by default.CommentShareyzfr1answered 3 years ago


Question: StreamManager error Could not find or load main class com.amazonaws.iot.gre

"Hi,I've been running Greengrass v2 succesfully on my IoT Jetson Nano device without any problems. I wanted to install greengrass on my local computer - for easier code testing/debugging.The problem I'm facing is that the StreamManager is unable to start.Here are the logs:2021-01-19T09:28:29.254Z [INFO] (pool-2-thread-20) aws.greengrass.StreamManager: shell-runner-start. {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING, command=["java  -cp /home/szymon/dev/greengrass/packages/artifacts/aws.greengrass.Stream..."]}2021-01-19T09:28:29.298Z [WARN] (Copier) aws.greengrass.StreamManager: stderr. Error: Could not find or load main class com.amazonaws.iot.greengrass.streammanager.StreamManagerService. {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING}2021-01-19T09:28:29.298Z [WARN] (Copier) aws.greengrass.StreamManager: stderr. Caused by: java.lang.ClassNotFoundException: com.amazonaws.iot.greengrass.streammanager.StreamManagerService. {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING}2021-01-19T09:28:29.302Z [INFO] (Copier) aws.greengrass.StreamManager: Startup script exited. {exitCode=1, serviceName=aws.greengrass.StreamManager, currentState=STARTING}I'm not going to paste the rest because this part occurs multiple time and nothing else.I checked the config file and I can see that:{"TS":1611048501769,"TP":["services","aws.greengrass.StreamManager","lifecycle","startup","script"],"W":"changed","V":"java  -cp /home/szymon/dev/greengrass/packages/artifacts/aws.greengrass.StreamManager/2.0.7/aws-iot-device-sdk-v2-1.0-SNAPSHOT.jar:/home/szymon/dev/greengrass/packages/artifacts/aws.greengrass.StreamManager/2.0.7/AWSGreengrassGreenlake-1.0-super.jar com.amazonaws.iot.greengrass.streammanager.StreamManagerService"}So I tried this part to run it manually:java  -cp /home/szymon/dev/greengrass/packages/artifacts/aws.greengrass.StreamManager/2.0.7/aws-iot-device-sdk-v2-1.0-SNAPSHOT.jar:/home/szymon/dev/greengrass/packages/artifacts/aws.greengrass.StreamManager/2.0.7/AWSGreengrassGreenlake-1.0-super.jarIt looks like it started but obviously with some errors but it's not important here.root@dfgdfg:/home/szymon/dev/greengrass/config# java  -cp /home/szymon/dev/greengrass/packages/artifacts/aws.greengrass.StreamManager/2.0.7/aws-iot-device-sdk-v2-1.0-SNAPSHOT.jar:/home/szymon/dev/greengrass/packages/artifacts/aws.greengrass.StreamManager/2.0.7/AWSGreengrassGreenlake-1.0-super.jar com.amazonaws.iot.greengrass.streammanager.StreamManagerServiceJan 19, 2021 11:03:40 AM com.amazonaws.util.EC2MetadataUtils getItemsWARNING: Unable to retrieve the requested metadata (/latest/dynamic/instance-identity/document). Failed to connect to service endpoint:com.amazonaws.SdkClientException: Failed to connect to service endpoint:at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:403)at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:372)at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:368)at com.amazonaws.util.EC2MetadataUtils.getEC2InstanceRegion(EC2MetadataUtils.java:283)at com.amazonaws.regions.InstanceMetadataRegionProvider.tryDetectRegion(InstanceMetadataRegionProvider.java:59)at com.amazonaws.regions.InstanceMetadataRegionProvider.getRegion(InstanceMetadataRegionProvider.java:50)at com.amazonaws.regions.AwsRegionProviderChain.getRegion(AwsRegionProviderChain.java:46)at com.amazonaws.client.builder.AwsClientBuilder.determineRegionFromRegionProvider(AwsClientBuilder.java:475)at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:458)at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)at com.amazonaws.iot.greengrass.streammanager.export.AWSClientFactory.createIotAnalyticsClient(AWSClientFactory.java:56)at com.amazonaws.iot.greengrass.streammanager.dagger.ExportModule.awsIoTAnalytics(ExportModule.java:80)at com.amazonaws.iot.greengrass.streammanager.dagger.ExportModule_AwsIoTAnalyticsFactory.get(ExportModule_AwsIoTAnalyticsFactory.java:29)at com.amazonaws.iot.greengrass.streammanager.dagger.ExportModule_AwsIoTAnalyticsFactory.get(ExportModule_AwsIoTAnalyticsFactory.java:10)at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)at com.amazonaws.iot.greengrass.streammanager.dagger.ExportModule_ExportStreamFactoryFactory.get(ExportModule_ExportStreamFactoryFactory.java:68)at com.amazonaws.iot.greengrass.streammanager.dagger.ExportModule_ExportStreamFactoryFactory.get(ExportModule_ExportStreamFactoryFactory.java:16)at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)at com.amazonaws.iot.greengrass.streammanager.dagger.ExportModule_ExportManagerFactory.get(ExportModule_ExportManagerFactory.java:90)at com.amazonaws.iot.greengrass.streammanager.dagger.ExportModule_ExportManagerFactory.get(ExportModule_ExportManagerFactory.java:20)at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)at com.amazonaws.iot.greengrass.streammanager.dagger.StreamServerModule_StreamServerFactory.get(StreamServerModule_StreamServerFactory.java:70)at com.amazonaws.iot.greengrass.streammanager.dagger.StreamServerModule_StreamServerFactory.get(StreamServerModule_StreamServerFactory.java:16)at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)at com.amazonaws.iot.greengrass.streammanager.dagger.DaggerStreamManagerServiceComponent.streamServer(DaggerStreamManagerServiceComponent.java:281)at com.amazonaws.iot.greengrass.streammanager.StreamManagerService.main(StreamManagerService.java:44)Caused by: java.net.SocketTimeoutException: connect timed outat java.base/java.net.PlainSocketImpl.socketConnect(Native Method)at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)at java.base/java.net.Socket.connect(Socket.java:609)at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253)at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1232)at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)... 30 moreThe StreamManager code started as you can see.FYI I tried on java 8 and java 11. Same problem on both versions.Looks like the greengrass is not able to start StreamManager and I'm out of ideas why...Please help.FollowComment"

Answer: "0The reason for the issue is most likely file system permissions since you have installed greengrass under your personal user's home directory. Greengrass will run the app using "ggc_user" which won't have permissions to read your own home directory. Set the permissions on "/home" "/home/szymon" "/home/szymon/dev" "/home/szymon/dev/greengrass" "/home/szymon/dev/greengrass/v2" all to be 755 so that other users have read and execute permissions on those directories. This will then allow ggc_user to read the files it needs access to.CommentShareEXPERTMichaelDombrowski-AWSanswered 2 years ago0Hi,Yes. You are right but I also found interesting behavior during installation of the greengrass itself. When you run the initial script to install the greengrass it changes the root folder to 700 which is strange because this was causing the problem so eventually it blocks itself (?!).As an example I installed greengrass:java -Dlog.store=FILE -jar ./GreengrassCore/lib/Greengrass.jar --aws-region eu-central-1 --root /opt/greengrass ....and by default /opt is 755 but after installation /opt has been changed to 700 and the problem with stream manager occurred.After changing the /opt to 755 problem was resolved. I'm guessing that there is some bug in the code which is initializing greengrass.Edited by: szymon888 on Jan 21, 2021 9:17 AMCommentShareszymon888answered 2 years ago"


Question: SES Delegate Sender and ConfigurationSetDoesNotExist

"I have a setup in which several AWS accounts need to send mails for a given domain. I configured SES on a shared account, and SES delegate sending on other accounts.Everything worked fine, until I attached a Configuration Set to the SES identity (the domain) on the shared account. From that moment on:Sending mails with credentials for the shared account keeps working as expected, including the functionalities of the Configuration SetBUT sending mails from the delegate sender account started failing with ConfigurationSetDoesNotExist errorsI guess this is related to missing permissions, but I do not find how to grant (delegate) permissions for the Configuration Set for the account that is configured as delegate.Anyone managed to get this working?Thanks.FollowComment"

Answer: "1Accepted AnswerHello, rik2803,I reviewed your post and did some research internally here at AWS, noting a case opened (and closed) the same date this post was entered.  I see from that case you believe you found the solution, but I wanted to provide a link to SES documentation on the topic of "Sending Authorization" (1) for reference.  If you do need assistance, you can reopen the case you previously closed.RESOURCESUsing sending authorization with Amazon SES - https://docs.aws.amazon.com/ses/latest/dg/sending-authorization.htmlSending emails for the identity owner for Amazon SES sending authorization  - https://docs.aws.amazon.com/ses/latest/dg/sending-authorization-delegate-sender-tasks-email.htmlCommentShareSUPPORT ENGINEERRon_Hanswered a year agorik2803 a year agoThanks for the feedback Ron, that's exactly how I solved it. I wanted the notifications to be sent to the AWS account that owns the SES identity, but was a bit misled because the SES console does not allow SNS topics in other accounts when configuring the notification destination. Turned out that this is possible using the AWS CLI. Of course, I had to update the topic access policy.Everything is working as expected.Share"


Question: How to add certificate to TrustStore for EMR Serverless application?

"How can a root CA certificate required by an EMR Serverless application be added to Java's TrustStore, or, alternatively, how can the location of a custom TrustStore be specified? In my case the TrustStore is required to contain the Amazon RDS SSL certificate in order to connect to a DocumentDB cluster via the MongoDB Spark Connector using TLS. Ordinarily the process of adding the certificate to the TrustStore on each node could be carried out by a bootstrap script, but EMR Serverless does not provide this option. I've downloaded the certificate from https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem and added it to a JKS file that can be sent to the cluster using the --files option. I have attempted to specify the location of this file by using the configuration properties spark.driver.extraJavaOptions and spark.executor.extraJavaOptions to pass the JMV option -Djavax.net.ssl.trustStore=cacerts.jks, but the file's absolute path is not the same across nodes, and cannot be known until after the job has been submitted and the SparkSession initialised. What then is the correct approach?FollowComment"

Answer: "1Hello there,Thank you for raising this question on re:Post.I would like to answer the question in two partsIf you would like to refer to a file in an Serverless EMR Spark Application, you will need to use --archive option. The --files does not help as the files will be downloaded to /tmp/spark-${UUID}/ hence not useful in this case. Please find below an example on how you can use the --archives option.a) Use --archives/spark.yarn.dist.archives option in spark to download the jks file archive under home directory of hadoop (/home/hadoop). Set the archives config like below, please note #rds marks the name of the folder the zip will be extracted to, you may name it differently as per your preference--archives 's3://YOUR-BUCKET/test-scriptsprefix/cacerts.jks.zip#rds'b) Now you can refer the file under the folder eg: /home/hadoop/rds/cacerts.jksIf you would like to use the SSL certificate bundle in the connection URL, you can follow the steps similar to belowPlease note: Steps below is an example for a connection to an external hive metastore with SSL in a EMR Serverless Spark application and are shared for reference only. You will need to change it based on how you would form the connection URL for the MongoDB Spark connector you are using.a) Prepare the certificates zip file and copy it to your S3 bucketzip -r rds-combined-ca-bundle.pem.zip rds-combined-ca-bundle.pemaws s3 cp rds-combined-ca-bundle.pem.zip s3://your-bucket/prefix/c) Similar to the previous option, we will use --archives option here as wellSample CLI:aws emr-serverless start-job-run \  --application-id "00f2hg0781i57409" \  --execution-role-arn "arn:aws:iam::111111111111:role/emrsvrlss" \  --job-driver '{        "sparkSubmit": {            "entryPoint": "s3://your-bucket/prefix/spark-jdbc.py",            "sparkSubmitParameters": "--jars s3://your-bucket/prefix/mariadb-connector-java.jar --conf spark.hadoop.javax.jdo.option.ConnectionDriverName=org.mariadb.jdbc.Driver --archives 's3://your-bucket/prefix/rds-combined-ca-bundle.pem.zip#rds' --conf spark.hadoop.javax.jdo.option.ConnectionUserName=hive --conf spark.hadoop.javax.jdo.option.ConnectionPassword=******* --conf spark.hadoop.javax.jdo.option.ConnectionURL='jdbc:mysql://database-1.************.us-east-1.rds.amazonaws.com:3306/hive\?createDatabaseIfNotExist=false\&useSSL=true\&serverSslCert=/home/hadoop/rds/rds-combined-ca-bundle.pem\&enabledSslProtocolSuites=TLSv1.2' --conf spark.driver.cores=2 --conf spark.executor.memory=10G --conf spark.driver.memory=6G --conf spark.executor.cores=4"        }    }' \    --configuration-overrides '{        "monitoringConfiguration": {        "s3MonitoringConfiguration": {            "logUri": "s3://DOC-EXAMPLE-BUCKET/emrlogs/"        }    }}'Important:Make sure you escape special characters and reserved keywords within your AWS CLI (as shown in working example).Use .zip to zip appropriate pem file and upload to your S3 bucket and make sure execution role arn has access to the S3 bucket used.Make sure you download appropriate jar file for connecting to your databaseCommentShareSUPPORT ENGINEERKrishnadas Manswered 6 months agoali-m-d 6 months agoThank you for your answer Krishnadas M. I'm still struggling with this, however, due to the problem detailed in the post below.Share0Thank you for your answer Krishnadas M. I’ve used the approach you suggested to extract the zipped JKS archive to the /home/hadoop directory. However, passing the JKS file location to the -Djavax.net.ssl.trustStore JVM option causes a com.amazonaws.SdkClientException error, presumably because the client requires access to a CA certificate that is stored in the default cacerts store, and the default store can no longer be located due to my system-wide override. The connection URI approach won’t work in the case of the MongoDB Spark Connector, as the tlsCAFile parameter is not supported (https://jira.mongodb.org/browse/DOCS-14874). Is there any way that I could add the CA certificate for my DocumentDB cluster to the default cacerts store, or obtain a copy of the default cacerts store that I could modify and submit using the —archives option? A partial trace is included below.22/12/06 13:29:29 ERROR DefaultEmrServerlessRMClient: Encountered exception when requesting SPARK_EXECUTOR containers. Future container launch rate will be slowed down until recovered. Next call will be allowed after 4000ms.com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested targetat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1216)at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1162)at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)at com.amazonaws.services.emrserverlessresourcemanager.EmrServerlessResourceManagerClient.doInvoke(EmrServerlessResourceManagerClient.java:682)at com.amazonaws.services.emrserverlessresourcemanager.EmrServerlessResourceManagerClient.invoke(EmrServerlessResourceManagerClient.java:649)at com.amazonaws.services.emrserverlessresourcemanager.EmrServerlessResourceManagerClient.invoke(EmrServerlessResourceManagerClient.java:638)at com.amazonaws.services.emrserverlessresourcemanager.EmrServerlessResourceManagerClient.executeRequestContainers(EmrServerlessResourceManagerClient.java:604)at com.amazonaws.services.emrserverlessresourcemanager.EmrServerlessResourceManagerClient.requestContainers(EmrServerlessResourceManagerClient.java:573)at org.apache.spark.deploy.emrserverless.client.DefaultEmrServerlessRMClient.createSingleBatchContainers(DefaultEmrServerlessRMClient.scala:136)at org.apache.spark.deploy.emrserverless.client.DefaultEmrServerlessRMClient.$anonfun$createContainers$2(DefaultEmrServerlessRMClient.scala:64)at scala.collection.Iterator.foreach(Iterator.scala:943)at scala.collection.Iterator.foreach$(Iterator.scala:943)at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)at org.apache.spark.deploy.emrserverless.client.DefaultEmrServerlessRMClient.createContainers(DefaultEmrServerlessRMClient.scala:63)at org.apache.spark.scheduler.cluster.emrserverless.ExecutorContainerAllocator.requestNewExecutors(ExecutorContainerAllocator.scala:282)at org.apache.spark.scheduler.cluster.emrserverless.ExecutorContainerAllocator.processSingleResourceProfile(ExecutorContainerAllocator.scala:259)at org.apache.spark.scheduler.cluster.emrserverless.ExecutorContainerAllocator.$anonfun$processExecutorsForAllResourceProfiles$5(ExecutorContainerAllocator.scala:182)at org.apache.spark.scheduler.cluster.emrserverless.ExecutorContainerAllocator.$anonfun$processExecutorsForAllResourceProfiles$5$adapted(ExecutorContainerAllocator.scala:180)at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)at scala.collection.Iterator.foreach(Iterator.scala:943)at scala.collection.Iterator.foreach$(Iterator.scala:943)at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)at scala.collection.IterableLike.foreach(IterableLike.scala:74)at scala.collection.IterableLike.foreach$(IterableLike.scala:73)at scala.collection.AbstractIterable.foreach(Iterable.scala:56)at scala.collection.TraversableLike.map(TraversableLike.scala:286)at scala.collection.TraversableLike.map$(TraversableLike.scala:279)at scala.collection.AbstractTraversable.map(Traversable.scala:108)at org.apache.spark.scheduler.cluster.emrserverless.ExecutorContainerAllocator.processExecutorsForAllResourceProfiles(ExecutorContainerAllocator.scala:180)at org.apache.spark.scheduler.cluster.emrserverless.ExecutorContainerAllocator.onNewSnapshot(ExecutorContainerAllocator.scala:130)at org.apache.spark.scheduler.cluster.emrserverless.ExecutorContainerAllocator.$anonfun$start$1(ExecutorContainerAllocator.scala:79)at org.apache.spark.scheduler.cluster.emrserverless.ExecutorContainerAllocator.$anonfun$start$1$adapted(ExecutorContainerAllocator.scala:79)at org.apache.spark.scheduler.cluster.emrserverless.store.ExecutorContainerStoreImpl$SnapshotsSubscriber.org$apache$spark$scheduler$cluster$emrserverless$store$ExecutorContainerStoreImpl$SnapshotsSubscriber$$processSnapshotsInternal(ExecutorContainerStoreImpl.scala:129)at org.apache.spark.scheduler.cluster.emrserverless.store.ExecutorContainerStoreImpl$SnapshotsSubscriber.processSnapshots(ExecutorContainerStoreImpl.scala:119)at org.apache.spark.scheduler.cluster.emrserverless.store.ExecutorContainerStoreImpl.$anonfun$addSubscriber$1(ExecutorContainerStoreImpl.scala:53)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:750)Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested targetat sun.security.ssl.Alert.createSSLException(Alert.java:131)at sun.security.ssl.TransportContext.fatal(TransportContext.java:324)at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)at sun.security.ssl.TransportContext.fatal(TransportContext.java:262)at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152)at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1397)at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1305)at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:142)at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)at com.amazonaws.http.conn.$Proxy33.connect(Unknown Source)at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1343)at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)... 46 moreCaused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested targetat sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456)at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323)at sun.security.validator.Validator.validate(Validator.java:271)at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315)at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:223)at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)... 76 moreCaused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested targetat sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451)... 82 moreCommentShareali-m-danswered 6 months ago0Hello there,Thank you for your reply.While I'm not able to test this currently, I can see that the documentation for MongoDB Connection URL does have reference for tlsCAFile for TLS connections. would you be able to quickly run a test using the same? As in use the 's3://your-bucket/prefix/rds-combined-ca-bundle.pem.zip#rds' and then refer the same in the connection URL like for eg: mongodb://db0.example.com,db1.example.com,db2.example.com/?tls=true&tlsCAFile=/home/hadoop/rds/rds-combined-ca-bundle.pemPlease note, this is untested and shared with assumption that this may help, please test and confirm. If you need further support on this, I would highly recommend reaching out to the AWS Support by raising a case with us.CommentShareSUPPORT ENGINEERKrishnadas Manswered 6 months ago0Krishnadas M the MongoDB Spark Connector uses the MongoDB Java Driver, which, unlike the mongo shell and other drivers, doesn't support tlsCAFile as one of the connection options. Accordingly, the driver reports WARN uri: Connection string contains unsupported option 'tlscafile'. when the parameter is included in the connection URI (https://jira.mongodb.org/browse/JAVA-3066).CommentShareali-m-danswered 6 months ago0Thank you for your reply.I understand the issue better now and I'll need to run tests at my end to see if there is any workaround to accomplish what you are asking here.Is it possible for you to share the steps you followed including the code (please remove any sensitive information before sharing) that I can use as a template to run my tests?I can also use the sample available in the documentation to prepare and run my tests, but unsure if that would be the same as what you are using. Please advise.CommentShareSUPPORT ENGINEERKrishnadas Manswered 6 months ago0Sorry for the delay in replying Krishnadas M. Below is the CDK code I used to create the ZIP archive containing the JKS file with the SSL certificate (cacerts.Dockerfile contains only the line FROM --platform=linux/amd64 amazoncorretto:11.0.17-alpine), along with the CLI command for starting the Spark job on EMR Serverless.Since your last message, the AWS team has announced the ability to customise the EMR Serverless base image. This allows me to use a custom Java image that has the certificate pre-installed into the TrustStore, so this approach, rather than the approach demonstrated below, is the one I am now using to solve the problem.Thank you for your help!const certificatesAsset = new assets.Asset(    this,    'BundledCertificates',    {        path: path.join(os.homedir(), '.ssh'),        bundling: {            image: DockerImage.fromBuild('../ml-pipeline', {                file: 'cacerts.Dockerfile'            }),            command: [                'sh',                '-c',                `keytool -importcert \                -keystore /asset-output/cacerts.p12 \                -storepass ${process.env.TRUSTSTORE_PASSWORD} \                -file rds-ca-2019-root.pem \                -alias RDS \                -no-prompt; \                keytool -importkeystore \                -srckeystore /asset-output/cacerts.p12 \                -srcstoretype pkcs12 \                -srcstorepass ${process.env.TRUSTSTORE_PASSWORD} \                -destkeystore /asset-output/cacerts.jks \                -deststorepass ${process.env.TRUSTSTORE_PASSWORD} \                -deststoretype jks`            ]        }    });const certificatesDeployment = new s3deploy.BucketDeployment(    this,    'CertificatesDeployment',    {        sources: [            s3deploy.Source.bucket(                certificatesAsset.bucket,                certificatesAsset.s3ObjectKey            )        ],        destinationBucket: bucket,        destinationKeyPrefix: 'cacerts',        extract: false    });new CfnOutput(this, 'CertificatesURI', {    value: certificatesDeployment.deployedBucket.s3UrlForObject(        path.join(            'cacerts',            Fn.select(0, certificatesDeployment.objectKeys)        )    ),    exportName: 'CertificatesURI'});aws emr-serverless start-job-run \    --region eu-west-1 \    --application-id <application-ID> \    --execution-role-arn <role-ARN> \    --job-driver '{        "sparkSubmit": {            "entryPoint": "s3://<bucket-name>/jobs/<filename>.py",            "entryPointArguments": [],            "sparkSubmitParameters": "--conf spark.archives=s3://<bucket-name>/cacerts/<asset-hash>.zip#cacerts,s3://<bucket-name>/artifacts/packages.tar.gz#environment --conf spark.jars=s3://<bucket-name>/artifacts/uber-JAR.jar --conf spark.emr-serverless.driverEnv.PYSPARK_DRIVER_PYTHON=./environment/bin/python --conf spark.emr-serverless.driverEnv.PYSPARK_PYTHON=./environment/bin/python --conf spark.emr-serverless.executorEnv.PYSPARK_PYTHON=./environment/bin/python --conf spark.mongodb.input.uri=mongodb://<username>:<password>@<instance-name>.••••••.eu-west-1.docdb.amazonaws.com:27017/••••••?tls=true&replicaSet=rs0&readPreference=secondaryPreferred&directConnection=true&retryWrites=false --conf spark.mongodb.output.uri=mongodb://<username>:<password>@<instance-name>.••••••.eu-west-1.docdb.amazonaws.com:27017/••••••?tls=true&replicaSet=rs0&readPreference=secondaryPreferred&directConnection=true&retryWrites=false --conf spark.driver.extraJavaOptions=-Djavax.net.ssl.trustStore=./cacerts/cacerts.jks --conf spark.executor.extraJavaOptions=-Djavax.net.ssl.trustStore=./cacerts/cacerts.jks"        }    }' \    --configuration-overrides '{        "monitoringConfiguration": {            "s3MonitoringConfiguration": {                "logUri": "s3://<bucket-name>/logs/"            }        }    }'CommentShareali-m-danswered 3 months ago"


Question: Creating metric filters encompassing multiple log groups

"I'm trying to create a metric filter that scans for ERRORS between a bunch of log groups. I see a lot of articles that show how to create a metric filter for a single log group but we have >100 log groups so it would be horrible to manually add it. I'm also open to new suggestions if I'm approaching this incorrectly, still learning how to use Cloudwatch.FollowComment"

Answer: "1Accepted AnswerCloudwatch Logs Insights will allow you to filter on up to 20 log groups at a time.  If you need to aggregate across a larger number of groups, or with more complex queries, then the suggested approach is to use subscriptions to send logs from each log group to a Kinesis stream, where you could do further processing either with Kinesis Analytics or Lambda.  Alternatively, you could stream your log groups into an OpenSearch cluster.CommentShareAWS-User-4340776answered 7 months agorePost-User-6771947 7 months agoAh got it, so what we currently do is use subscriptions on two of our log groups but I was wondering if there was a different way instead of creating a subscription for each log group manually and then further process it with Lambda.I'll check out OpenSearch cluster, to see if that is something feasible and within our budget. Thank you!ShareAWS-User-4340776 7 months agoThe best way to deal with large configurations is always automation.  Ideally you would use a complete Infrastructure as Code solution such as CloudFormation or Terraform to bring these resources under control.  If that's not possible, the next-best thing is to write a small script in the language of your choice, using either the AWS CLI or an AWS SDK, to find all of the log groups and modify them to suit your needs.  Python is a popular language used for small scripts like this.If this answer / comment was helpful, please upvote it! Thanks :)Share"


Question: Kinesis Video updateMediaStorageConfiguration AccessDeniedException (403)

"I am utilizing the aws-sdk (v2.1289.0) on a Node server to set up video related features for an application. I am using the KinesisVideo class (aws-sdk/clients/kinesisvideo). I am able to successfully create channels and streams, but when I try to run updateMediaStorageConfiguration I get an AccessDeniedException. The exact error reads: AccessDeniedException: The account id XXXXX is not allowed to access the resources.The relevant code:        kinesisVideoClient.updateMediaStorageConfiguration({            ChannelARN: channel.ChannelARN,            MediaStorageConfiguration: {                Status: 'ENABLED',                StreamARN: stream.StreamARN,            },        }, (err, data) => {            if(err) console.log('error', err)        })ChannelARN and StreamARN are both defined. It doesn't appear to be a permissions issue as even with admin access to all services, the error still occurs. I am wondering what I can do to troubleshoot. The same error also occurs when using describeMediaStorageConfiguration.FollowComment"

Answer: "1Accepted AnswerThe new preview feature for WebRTC Ingestion, which is what you're configuring via update-media-storage-configuration, is currently only available in the us-west-2 region. Please confirm that your AWS region is being set to us-west-2. See the following resource for more details on how to set the region in your code: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-region.htmlCommentShareEXPERTDavid Maloneanswered 5 months ago"


Question: Does AWS Timestream support UDFs (user defined functions)

Does AWS Timestream support UDFs (user defined functions) similar to the UDFs support in Amazon Athena?FollowComment

Answer: "0No, Timestream doesn't support user-defined functions (UDFs) at this time.If your function is simple enough, you might be able to accomplish your goal with a reduce function, which really lets you apply two inline functions (an input and output function) to transform your data. Here's a link to the docs,https://docs.aws.amazon.com/timestream/latest/developerguide/timeseries-specific-constructs.functions.filter-reduce.html#w3aac25c13c13c23b9Otherwise, you could write an AWS Lambda that consumes you table and writes back to either the same table or a different table. Timestream will support sparse columns, so you can have a lambda that is back-filling a column in a multi-measure row that initially doesn't exist for all records.  You would need to read out the row, enrich it, and then write back the same data for the other columns with a higher version number.Finally, scheduled queries are another way to automatically enrich your data. You could have a scheduled query that computes your "function" (separate query) result and pushes the record back to the same table or to a different table, in a different format. The nice thing is that the scheduled query engine will bump the version number for you. Here are the docs on scheduled queries:https://docs.aws.amazon.com/timestream/latest/developerguide/scheduledqueries.htmlCommentShareNadirSanswered a year ago"


Question: How to get a raw response like XML using s3 client from aws/sdk-javascript.

"When I execute ListBucketscommand() using the SDK's S3 client, you can get the following result.How to get a raw response like XML using s3 client from aws/sdk.// sample code & resultconst command = new ListBucketsCommand({});const response = await client.send(command);console.log(response);'$metadata': {httpStatusCode: 200,requestId: '1759FE4B14A997E0',extendedRequestId: 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',cfId: undefined,attempts: 1,totalRetryDelay: 0},Buckets: [{ Name: 'test', CreationDate: 2023-03-13T04:45:29.702Z },{ Name: 'test-0424', CreationDate: 2023-04-24T03:25:08.831Z },]// xml format<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">  <Owner>    <DisplayName>minio</DisplayName>    <ID>02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4</ID>  </Owner>  <Buckets>    <Bucket>      <Name>test</Name>      <CreationDate>2023-03-13T04: 45: 29.702Z</CreationDate>    </Bucket>    <Bucket>      <Name>test-0424</Name>      <CreationDate>2023-04-24T03: 25: 08.831Z</CreationDate>    </Bucket>  </Buckets></ListAllMyBucketsResult>FollowComment"

Answer: "1The SDK parses the raw XML responses from the S3 API and converts them into JavaScript objects before returning them to the user.https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-select.htmlBut you can directly query S3 APIhttps://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.htmlYou can use axios library to do this.CommentShareEXPERTsdtslmnanswered a month ago1For AWS SDK for JavaScript v2, you can access the raw response via the response field on the request object.  Note that response field on the request object will not be set until after the request has been completed, thus the need to await in the example below before accessing.const listBucketsRequest = S3.listBuckets({});const listBucketsResponse = await listBucketsRequest.promise()console.log(listBucketsRequest.response.httpResponse.body.toString());If AWS SDK for JavaScript v3 does not expose a similar set of objects.  Access to the raw response would be done at the middleware level:https://github.com/aws/aws-sdk-js-v3/tree/main/packages/middleware-stackhttps://aws.amazon.com/blogs/developer/middleware-stack-modular-aws-sdk-js/CommentShareAWS-User-5738042answered a month ago"


Question: RDS Specification and Burstable Performance to what level?

"Long time reader, first time poster...I'm just looking for clarification and hopefully this is a simple one that everyone (except me) knows.As regards the specification of the burstable RDS instances, it states the vCPU and RAM details. When the instance is performing below a certain baseline, it accrues credits, I understand that.My question is, to what level of performance? If I am running a t3.2xlarge instance and it has been running at a low level for a while (and therefore has a supply of credits), when there is a spike in requirement, what is the performance level? Is it 100% of the resources for the instance (so in this case 8 vCPU and 32 GB of RAM), or does it go above this - and if so, by how much?Any advice greatly appreicated.FollowComment"

Answer: "0Accepted AnswerYou always get the full amount of RAM for the instance type. As long as you have credits available you also get up to 100% of the vCPU resources for the instance type. The credits accumulate and are spent the same way they are on EC2 (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html). For RDS T2 instances you drop to baseline when credits are exhausted.Since RDS only supports the T3 family in Unlimited Mode, you always have the full performance of the instance available. But be aware that if the average usage over a 24-hour period exceeds baseline then you will be charged for the overage. Again, see the EC2 description on how these instances work. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances-unlimited-mode-concepts.html#how-burstable-performance-instances-unlimited-worksEdited by: HalTemp on Mar 19, 2020 8:29 AMEdited by: HalTemp on Mar 19, 2020 8:29 AMCommentShareHalTempanswered 3 years ago0Thanks Hal - that was exactly what I was looking for.CommentShareMatthewBlackwellanswered 3 years ago"


Question: Filtering / Paging gives strange results

"I have a dataset with seven records:I want to return all records between 14 and 20 Dec, paging them two at a time. This is my first call, for the first page of results:Dictionary<string, AttributeValue> expressionAttributeValues = new Dictionary<string, AttributeValue>();            Dictionary<string, string> expressionAttributeNames = new Dictionary<string, string>();            string filterExpression = string.Empty;            AmazonDynamoDBClient client = new Amazon.DynamoDBv2.AmazonDynamoDBClient(creds);            List<RecordingMetaData> recordings = new List<RecordingMetaData>();            expressionAttributeValues.Add(":startTime", new AttributeValue { S = $"2022-12-14" });            expressionAttributeValues.Add(":endTime", new AttributeValue { S = $"2022-12-20" });            filterExpression = $"#start > :startTime AND #start <= :endTime";            expressionAttributeNames.Add("#start", "StartTime");            ScanRequest scanRequest = new ScanRequest()            {                TableName = "SCRecordingMetaData",                ExpressionAttributeNames = expressionAttributeNames,                FilterExpression = filterExpression.Length > 0 ? filterExpression : null,                ExpressionAttributeValues = expressionAttributeValues,                Limit = 2            };            var response = new ScanResponse();            do            {                response = client.Scan(scanRequest);                response.Items.ForEach(i => recordings.Add(new RecordingMetaData()                {                    RecordingID = i["RecordingID"].S,                    Duration = Convert.ToInt32(i["Duration"].N),                    StartTime = Convert.ToDateTime(i["StartTime"].S),                    EndTime = Convert.ToDateTime(i["EndTime"].S),                    ANI = i["ANI"].S,                    DNS = i["DNS"].S,                    CreatedTime = Convert.ToDateTime(i["CreatedTime"].S),                    FilePath = i["FilePath"].S,                    AgentIDs = i.ContainsKey("AgentIDs") ? i["AgentIDs"].L : null,                    QueueIDs = i.ContainsKey("QueueIDs") ? i["QueueIDs"].L : null                }));            }            while (response.LastEvaluatedKey.Count > 1);it should, and does, return 3 records when there's no Limit set in the request. When it's set to 2, I would expect 2 results to be returned for the first page, with the LastEvaluatedKey being "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad" since this is the second key in the list that meets the filter criteria. However, only one result is returned, "2a45d769-fc43-4325-96ec-76e5cf0750fe" and this is also the LastEvaluatedKey. If I pass this value to the ExclusiveStartKey of the next call I get 0 results returned, but the LastEvaluatedKey is "8bce9df3-3921-4259-a70a-c9cf6d34f61f".So, what seems to be happening is that it only returns records that meet the filter criteria, but it is still in some way recognising that the other records that haven't been returned still exist, i.e. the first record set with no search filter should have "8c8c7afb-e529-4cb5-adb2-ab79d51f5741" and "2a45d769-fc43-4325-96ec-76e5cf0750fe", but because of the filter it only returns one record "2a45d769-fc43-4325-96ec-76e5cf0750fe". Surely it should return "2a45d769-fc43-4325-96ec-76e5cf0750fe" and "5dcb2cd5-ddcc-41d8-b147-ec2aa95883ad".How can you write any kind of paging with behaviour like this?FollowComment"

Answer: "1This is how filtering on DynamoDB works. It first reads up to the number of items defined by Limit and then applies the filter. In your case you set Limit=2 meaning you will get ~2 items, but not guaranteed to be two.A FilterExpression is applied after the items have already been read; the process of filtering does not consume any additional read capacity units.https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html#DDB-Scan-request-FilterExpressionCommentShareEXPERTLeeroy Hannigananswered 5 months agoNick 5 months agoI guess that's logical, but not great when trying to explain to non-technical users why there's a different number of results on each page. I've had to change the design now so that all results are displayed on one page, but there's a "Get More Results" button that add records to the bottom. It's an OK solution.Share"


Question: AWS Forecast accuracy

"Hello:I have time series data from farming sensors (with temperature, humidity, soil moisture, etc) and I would like to make predictions for each parameter. I 'm trying AWS Forecast, using the console, and using CUSTOM (as the other options seems more suitable for business operations, demand planning, etc) and the DEFAULT Predictor configuration. I actually split the real data (30 days) into the training data set (27 days) and reserved the last 3 days to compare with the predictions. Data frequency is 30 minutes and forecast horizons is 144 (3 days forecast) . I added geolocation, with TimeZone and weather index as well for better accurracy (I tried the same without these additions and accuracy was poor as well).The results given in P10, P50 and P90 shows a huge variation of course, but so far, I don't see a good accuracy between the predicted values and the real ones... There are many values outside the P10 ~ P90 range which I would not expect...As an example, this is  chart comparing forecasted values (p10, p50 and p90) vs the real values:As we can see, the real values (yellow) are higher than predicted values...I'm not sure if I'm doing something wrong here, but so far, these results don't seem to be very useful in a real life case... Am I missing something?UPDATE: I trained the model using much more data, and now it looks very good:Thanks in advance;SOSFollowComment"

Answer: "0Dear User,Thanks for updating you query on re:post.We are glad that your issue is resolved.Feel free to post a query again on re:post in future.Regards,NikhilCommentShareAWS-User-8643694answered 5 months ago"


"I have MySQL Aurora 5.7 cluster running. The only error logs that are being written showhandler.cc:8249Does anyone know why this is happening?FollowCommentrePost-User-5722448 2 months agoWe have encountered the same, occuring once a day, but not every day.0x2b4722a7c700 (handler.cc:8305)Share"


Question: AWS::EC2::AvailabilityZone::Name Parameter Validation...

"I have an AWS::EC2::AvailabilityZone::Name as a parameter for my CloudFormation template like this:AvailabilityZone:Type: AWS::EC2::AvailabilityZone::NameDescription: New subnets will be created for this zone.ConstraintDescription: "A valid availability zone must be selected."It populates nicely with the list of Availability Zones for the region where the CloudFormation template is coming up. However, CloudFormation does not validate that an Availability Zone was selected (i.e. it can be left blank). After the "Create Stack" button is pushed the stack creation process will fail.I would like to be able to validate the parameter and make sure that it is not empty before the stack fails. I tried specifying a "MinLength" property of 9. I uploaded the template and it would not let me enter an empty Availability Zone (i.e. it seemed to work). However, in our build process we call cfn-lint to validate the template and it gives the following error:E2001 Parameter AvailabilityZone has property MinLength which is only valid for \['String']Can anyone tell me what is the correct way to validate an AWS::EC2::AvailabilityZone::Name field to be non empty at least, and preferably an item in the list provided by the GUI?Thanks in advance,Edited by: hdsiii on May 15, 2020 11:24 AMEdited by: hdsiii on May 15, 2020 11:26 AMEdited by: hdsiii on May 18, 2020 4:24 PMFollowComment"

Answer: "0AWS tech support sent me an answer. Although MinLength is not supported on AvailabilityZone::Name, AllowedPattern is. Here is a pattern that ensures the AvailabilityZone::Name is at least 1 character long. Since this is a popup field it ensures that the popup is populated and not left blank.===================================================AvailabilityZone:Type: AWS::EC2::AvailabilityZone::NameDescription: EC2 AvailabilityZoneAllowedPattern: ".+"                # -> to have a non blank valueFYI: Using a DefaultValue is not really an option since I don't know what Region the script is being run in.CommentSharehdsiiianswered 3 years ago"


Question: php-zip missing in linux 2022

upgraded to new servers with AWS Linux 2022 (php 8.1.7).  No longer have php-zip module available.  No matching package found when trying to install manually.Just tried installing Webmin and it failed looking for "perl-Encode-Detect".  Tried installing this and it also seems to be missing from latest Linux 2022 distro.FollowCommentregattaman 4 days agoAlso missing php-imap module.  Any workarounds?Share

Answer: "0that command only returns:  unable to find match: php-zipthis is AWS Linux 2022  RC from Nov.CommentShareregattamananswered 5 months agoBrettski-AWS EXPERT5 months agoI just launched a brand-new instance in the Sydney region and that package is available. Can you provide more details? Which AMI? Which region?Shareregattaman 5 months agoAMI used was:  amazon/al2022-ami-minimal-2022.0.20221103.1-kernel-5.15-arm64Region:  US-East (N. Virginia)ShareBrettski-AWS EXPERT5 months agoThat is a preview release so it's unlikely to have support for everything yet. The 5.10 kernel release (which is the non-preview version) is what I tested earlier.Share0When you say "install manually" what commands are you running?I've just run sudo yum install php-zip on my instance and the package appears to be there.CommentShareEXPERTBrettski-AWSanswered 5 months ago0Just upgraded to RC 20230118.  still cannot install php-zipalso, when system is restarted it says that there is a newer release available but lists "20221207".CommentShareregattamananswered 2 months ago0I had this issue too but there is a workaround: https://github.com/amazonlinux/amazon-linux-2023/issues/320bwaters commented May 4, 2023:sudo yum install php8.1-devel php-pear libzip libzip-develsudo pecl install zipecho "extension=zip.so" | sudo tee /etc/php.d/20-zip.iniCommentShareGeorgeMcKanswered 5 days ago"


Question: How to create a super user on MSK

What's the easiest way to have a super user on MSK using SASL/SCRAMFollowComment

Answer: "0As super.users parameter is currently not supported in MSK, following is one way to achieve something similar -- allow.everyone.if.no.acl.found is by default set to true, so you can start by adding ACLs for the intended principal/user allowing all 'operations' on resource types like topic, group. And on resource type 'cluster' add ACLs for all operations except CLUSTER_ACTIONS(to avoid blocking brokers, don't set ACLs on CLUSTER_ACTIONS)(1).After you set the Apache Kafka ACLs for the user, update the cluster's configuration to have the property allow.everyone.if.no.acl.found to false for the cluster(2). This way above user will act similar to a super user and all other users will be denied access unless ACLs are added for other users as well.(1) https://docs.aws.amazon.com/msk/latest/developerguide/msk-acls.html(2) https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-operations.htmlCommentShareSUPPORT ENGINEERHarshith_Manswered a year agowspurgin 9 months ago"to avoid blocking brokers, don't set ACLs on CLUSTER_ACTIONS"Where is that in the documentation? The link in (1) doesn't mention CLUSTER_ACTIONS (in fact it's pretty sparse on details in general).ShareIstvan Darvas 7 months agoHi what can you do if you have a cluster with public access? - this cannot be work, because it is prohibited by the API. So my question: how can you manage the users in that scenario?Share"


Question: Error logging out from Cognito in ASP.NET Core MVC

"I'm struggling with the ASP.NET Core MVC app and the integration to Cognito. I am using the AWS SDK and the hosted UI option.I have managed to get the sign-in functionality working but am now struggling with the sign-out functionality.I am basing this on the AWS code at https://catalog.us-east-1.prod.workshops.aws/workshops/02696107-09ac-4313-a6cb-3798048b07d7/en-US/9-how-to-authenticate-users-in-my-application and the example code it supplies.This has the signout code as the following:    public IActionResult SignOut()    {        var callbackUrl = Url.Page("/", pageHandler: null, values: null, protocol: Request.Scheme);        return SignOut(            new AuthenticationProperties { RedirectUri = callbackUrl },            CookieAuthenticationDefaults.AuthenticationScheme, OpenIdConnectDefaults.AuthenticationScheme        );    }After tweaking it for my app, this ends up as:    public IActionResult SignOut()    {        var baseUri = $"{Request.Scheme}://{Request.Host}";         //var callbackUrl = Url.Page("/", pageHandler: null, values: null, protocol: Request.Scheme);        return SignOut(            new AuthenticationProperties { RedirectUri = baseUri },            CookieAuthenticationDefaults.AuthenticationScheme, OpenIdConnectDefaults.AuthenticationScheme        );    }When I run the app and attempt to sign-out, the sign-out seems to function correctly (in that the cookies are removed). However, the Cognito redirect fails with the following error:When debugging the code it can be seen that the redirect URI is set to "https://localhost:7085"Within AWS the App Client is defined as follows. I have gone to town trying various options so the URLS are currently set as the followingAny ideas about what I am doing wrong here?FollowComment"

Answer: "0Hey,From the docs you need to use the logout endpoint passing your redirect urlhttps://mydomain.auth.us-east-1.amazoncognito.com/logout?    client_id=ad398u21ijw3s9w3939&    logout_uri=https://myclient/logoutHaven't tested this but this should get you going once you have replaced the values below with your values[HttpGet]public async Task<IActionResult> LogOff(){     await HttpContext.SignOutAsync("OpenIdConnect");     await HttpContext.SignOutAsync(CookieAuthenticationDefaults.AuthenticationScheme);     return Redirect($"https://mydomain.auth.us-east-1.amazoncognito.com/logout?client_id={ClientId}&logout_uri={Request.Scheme}://{Request.Host}");}Hope that helpsCommentShareGrayNiallanswered 5 months ago"


Question: AWS CDK make Route53 records in another account

"I am working with 2 AWS accounts, one is the root account which contains the primary Route53 hosted zone domain.com. The other account is the production account, and it runs our production environment, which is deployed using CDK. It contains a delegated hosted zone with the domain prod.domain.com.  There is also an amplify application running on the production account available at app.prod.domain.com.The goal is to make the production application available via app.domain.com as it is more user-friendly. However, I have been unable to find a method that will allow me to create records in the root account hosted zone from the production account. Is there any method to do this automatically?I am aware that I can manually enter the records in the root account, but I would rather have an automated solution, as this would reduce the chance of errors and make the entire infrastructure clear to other maintainers.I have already tried to make a delegated hosted zone with the domain name domain.com in the production account, but that does not work.FollowComment"

Answer: "0What type of principal was used to create the zone(s)?  Depending on whether you used the root account credentials or an IAM user/role, the behavior/ownership is slightly different.  Have you created the appropriate roles in each account to be used for resource creation?  The documentation on resource permissions and using IAM roles is a good place to start.Route53 only supports identity-based IAM policies, so you'll need to create the appropriate permissions policy and attach it to the principal(s) you're using.  There's an example there specifically for delegating CreateHostedZone that is likely a good start.CommentSharethat_mikeanswered a year agoBob a year agoI found a solution. The delegated zone creation was already working, so I just created a dedicated hosted zone for the app.domain.com. This has the added benefit of allowing amplify to create certificates. Hosted zones aren't free, but given that this is only a single subdomain, the costs are insignificant.Share"


Question: Can we use dynamodb without nodejs ?

"Hello guys, how are you I am a beginner in AWS and I wanna know can I use plain JavaScript to store values and manipulate them using DynamoDB.const { DynamoDBClient } = require('@aws-sdk/client-dynamodb');const { DynamoDBDocument } = require('@aws-sdk/lib-dynamodb');// Create service client module using ES6 syntax.//import { DynamoDBClient } from "@aws-sdk/client-dynamodb";// Set the AWS Region.const REGION = "ap-south-1"; //e.g. "us-east-1"// Create an Amazon DynamoDB service client object.const ddbClient = new DynamoDBClient({ region: REGION });// Import required AWS SDK clients and commands for Node.jsimport { CreateTableCommand } from "@aws-sdk/client-dynamodb";// Set the parametersconst params3 = {    AttributeDefinitions: [      {        AttributeName: "Season", //ATTRIBUTE_NAME_1        AttributeType: "N", //ATTRIBUTE_TYPE      },      {        AttributeName: "Episode", //ATTRIBUTE_NAME_2        AttributeType: "N", //ATTRIBUTE_TYPE      },    ],    KeySchema: [      {        AttributeName: "Season", //ATTRIBUTE_NAME_1        KeyType: "HASH",      },      {        AttributeName: "Episode", //ATTRIBUTE_NAME_2        KeyType: "RANGE",      },    ],    ProvisionedThroughput: {      ReadCapacityUnits: 1,      WriteCapacityUnits: 1,    },    TableName: "TEST_TABLE", //TABLE_NAME    StreamSpecification: {      StreamEnabled: false,    },  };  const run = async () => {    try {      const data = await ddbClient.send(new CreateTableCommand(params3));      console.log("Table Created", data);      return data;    } catch (err) {      console.log("Error", err);    }  };  run();and I am facing this error-above is a code and output image in which I am facing credentials error. Please someone guide me where I am doing wrong. Thanks for your time and guidance.FollowComment"

Answer: "2Accepted AnswerYou seem to be missing credentials. You haven't mentioned where you are running your application from, if its from your local machine you can set up the credentials using AWS Configure. This will allow your code pick up credentials and interact with DynamoDB, so long as the credentials belong to a user who has DynamoDB access. More on Identity and Access Management here.If you're using an AWS service to host the code such as Lambda, then ensure you attach an IAM execution role to the service so it has permissions. Here is an example for Lambda but the same goes for EC2 or other services.CommentShareEXPERTLeeroy Hannigananswered 10 months agorePost-User-4667895 10 months agoLike I am using JavaScript in my local machine can you tell me which is the exactly process for passing credentials?ShareLeeroy Hannigan EXPERT10 months agoRun aws configure from the CLI. It will set up credentials locally and is the preferred method.https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.htmlOtherwise you can add the credentials to your DynamoDB Client directly, but it is strongly discouraged to do so.Share"


Question: IAM role for AWS workspace

"I have created a AWS workspaces for some users. is there any way we can add IAM role with AWS workspaces same as we do with EC2 instances, So that they do not need AWS keys and can access AWS services as per IAM role attached with workspace?Edited by: ashishnm1983 on Mar 20, 2020 1:07 PMFollowComment"

Answer: "0Looks like it is not possiblehttps://stackoverflow.com/questions/60776068/iam-role-for-aws-workspaceCommentShareashishnm1983answered 3 years ago0This is a shame - the underlying VM is an EC2 instance, and so does actually have instance metadata available, the problem being that it's running as an EC2 instance in an AWS internal account rather than managed within your own.With the current Workspaces arrangement (unlike the older TS based implementation) it seems this amounts to "can I grant IAM roles to someone else's EC2 instance", and unfortunately that doesn't seem to be possible at present. (It's possible to create a role and grant the Workspaces AWS account permission to use it, but granting access to that role to the EC2 instance itself would seem to need cooperation from the Workspaces EC2 account holder, which of course isn't available.)It might not be too hard for AWS to add a Workspaces API call to associate a role - I'll raise this with our AWS contacts next week as a request.As an interim measure I think I'll probably have to put IAM credentials in the user's AD object and retrieve those programmatically from within the instance, rather than being able to grab them straight from instance metadata, which is a bit of a shame but not the end of the world.CommentSharejamesasutherlandanswered 3 years ago0Hey James,May I ask if you were able to retrieve IAM credentials from AD object? With AWS DS SDK not exposing user-related information, it seems its not an an obvious task.Only way out looks to be: To have let workspace user configure static access keys  (with 0 access) and let him assume temporary elevated role for limited duration.Edited by: nullpointergonewild on Feb 24, 2021 10:05 AMCommentSharenullpointergonewildanswered 2 years ago"


Question: Mysql 8.0.29 issues (version disappear)

"Hello there, I have created mysql rds instance with mysql 8.0.29, but now it's says that this version is deprecated, and I can't upgrade instance, or change something via UI (only some actions over API works).Also not possible to change storage type (over UI or API):Any ideas how it can be fixed, without instance re-creation?FollowComment"

Answer: "0Dear Customer,Thanks for reaching out to us via AWS re:Post. I just saw this post and checked with my peers and found the below info.The minor version MySQL 8.0.29 release has been deprecated and that's the reason that you couldn't find the version in the list any more. At this point, recommendation is for customers that have visible symptoms of a crash we recommend they do a Point-In-Time-Restore (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIT.html  ) to recover the database. Please accept my sincere apologies on behalf of AWS for the inconvenience caused by this.When the new version is released and available on RDS we’ll inform you and the rest of our AWS community on our announcements page here: https://aws.amazon.com/new/If you would like to downgrade your instance to version 8.0.28, below options are available:Using Native MySQL import export tool:You can use tools like mysqldump/Mydumper/Myloader to import and migrate from existing RDS MySQL instance on 8.0.29 to another instance on 8.0.28 version.Please kindly review the below documentation for more details:[+] Importing data from a MySQL or MariaDB DB to a MySQL or MariaDB DB instance - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Procedural.Importing.SmallExisting.html[+] Importing Data to an Amazon RDS MySQL or MariaDB DB Instance with Reduced Downtime - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Procedural.Importing.NonRDSRepl.htmlUsing AWS Database Migration Service:Please use the below resources to get more familiar with DMS working and settings before getting stared with the service:[+] Migrate to an Amazon RDS or Aurora MySQL DB instance using AWS DMS: https://aws.amazon.com/premiumsupport/knowledge-center/migrate-mysql-rds-dms/[+] What is AWS Database Migration Service? - https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html[+] How AWS Database Migration Service works - https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.htmlHope this helps in your project.CommentShareEXPERTNIRUPAM_danswered 10 months agorePost-User-7498861 10 months agoHello, Why the minor version MySQL 8.0.29 release has been deprecated? There is no information about this on the MySQL or AWS pages.Share0Dear Customer - there was an issue found with this version, for more details pertaining to it and w.r.t. your RDS instance I would request you to kindly log a case with premium Support AWS RDS Team..CommentShareEXPERTNIRUPAM_danswered 10 months ago"


Question: Problems in Publishing Task in AMT

"Hi,  I am trying to publish a task in AMT, but I am getting the error, "you have exceeded your monthly credit limit". I have signed up for AWS billing and my credit card information is valid and up-to-date. Can anyone kindly help me with what to do in this scenario?I have submitted a request to the support team about this, but I need this resolved urgently.Any help will be greatly appreciated.FollowComment"

Answer: "0Hello,Based on the error you've received, contacting our Mechanical Turk (MTurk) support team would be the correct action to take.If this is the support team you submitted your request to, they'll get back to you soon with more info. If not, you can contact us here for assistance: https://support.aws.amazon.com/#/contacts/aws-mechanical-turk.Regards,-- Ben G.CommentShareEXPERTAWS Support - Benanswered a year agorudra0713 a year agoHi Ben, thanks for your response. I realized that I submitted the request through the link that you provided. I got a response from one of the members of Mechanical Turk Requester Support saying that a request has been sent to increase my monthly spending limit. But there was no other follow-up.As I mentioned in my original question, I am in dire need of launching the survey asap. Is there anything I can do to speedup the request approval process?SharerePost-User-9687991 4 months agoSame. I'm new and my limit says it's 2,500 already yet I couldn't publish something for 1,008. I dropped it to 2 and that exceeded the limit too. So I'm pretty sure when you're new the limit is 0. We are forced into AWS billing and can't use pre-paid HITS or pay upfront in any way and then they don't give you a workable account. I've been waiting for 5 days and this is after being told 3 times my case is being expedited. I cannot wait a month because I need this batch for doctoral research to graduate.Share"


Question: Crypto Module Error when using AWS4Axios

"I am using the aws4-axios package for authentication on a React App. I am using this package in order to authenticate an API call with AWS credentials. When I attempt to start the app, I receive an error about the crypto module. I have attempted to remediate this issue by setting crypto to false in my package.json but the error still occurs. If anyone could help, I would deeply appreciate it.I am inserting the error below.ERROR in ./node_modules/aws4/aws4.js 4:13-30Module not found: Error: Can't resolve 'crypto' in '/mnt/c/Users/dalade/code/repos/prism/prism-dashboard/node_modules/aws4'BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.This is no longer the case. Verify if you need this module and configure a polyfill for it.If you want to include a polyfill, you need to:        - add a fallback 'resolve.fallback: { "crypto": require.resolve("crypto-browserify") }'        - install 'crypto-browserify'If you don't want to include a polyfill, you can use an empty module like this:        resolve.fallback: { "crypto": false }webpack compiled with 1 errorFollowComment"


Question: Question about API Gateway VPC link and Network Load Balancer

"Hi. i associated API GW VPC link and had a latency issue. and i figured it through activate cross-zone load balancing, thanks to the link:https://repost.aws/questions/QU4zhD4Tb4Qg6z9U1YXeMAxg/api-gateway-to-nlb-via-vpc-link-latency-issuesi understand latency can be made, but i wonder why it can be solved through cross-zone load balancing.can anybody explain some details?thanks.FollowComment"


Question: On an apigateway websocket is the status returned from the lambda handler?

"When an error is returned from a lambda websocket handler I was expecting something to be seen on the client side. But nothing. I can send messages from the lambda function over the websocket so I know that path works. Is there some configuration or other magic I'm missing?exports.handler = async (event, context) => {try {return { statusCode: 200, body: "OK" }} catch (status) {return { statusCode: 500, body: "XXXX" }}}FollowComment"

Answer: "0Accepted AnswerBy default, WebSocket route is one-way, and it only sends a message to the client when API Gateway couldn't process the incoming message correctly.In your case, API Gateway successfully executed the API, but your Lambda is returning 5xx. To send a result as a message to the client, you need to configure your route as "two-way" by configuring a route response.https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api-route-response.htmlCommentShareAWS-User-3921561answered 3 years ago"


Question: ETL Connection reset error when pulling data from RDS DB

"Looking for a little insight on this:ETL job fails with the following error:  “Connection reset on https://xxxxxx.com:9200 https://xxxxxxx.com:9200"Basic information:*ETL job is pulling data from an RDS PostgresSQL database*This is a nightly job that starts at 6PM - error only happens on Thursday nights at 7pm (every thursday). Runs as expected every other night without issue*Job that has been running without issue for 5 years, and just starting 5 weeks ago.FollowCommentAWS-User-0357409 3 months agoHi Jennifer,Any chance you may have your RDS maintenance window setup during that time?  Any indication in logging or the RDS console that a failover might be happening during that (if multi-AZ)?Share"

Answer: "0That was my first thought, but no, maintenance is scheduled for saturdays.  Its not Multi-az.  The only other thing i can think of was that maybe backups  are happening but that is scheduled for 5 UTC.  I havent worked much with DB's, could it be that there is more data on thursdays to query?CommentShareJenniferanswered 3 months ago"


Question: Italy - Milan alternative SMTP Server

"Dear Community,Based on the actual documentation (https://docs.aws.amazon.com/ses/latest/dg/smtp-credentials.html) there is no possibility to create a SMTP servers in the Italy - Milan region.Do you know a solution or a workaround to use the SES service for solutions and services located in Italy?Thanks so muchFollowComment"


Question: Connect Amazon DocumentDB Cluster from Outside  Amazon VPC

"Connect Amazon DocumentDB Cluster from Outside  Amazon VPCAmazon DocumentDB (with MongoDB compatibility) clusters are deployed within anAmazon Virtual Private Cloud (Amazon VPC). They can be accessed directly by AmazonEC2 instances or other AWS services that are deployed in the same Amazon VPCI setup an EC2 (same VPC as DocumentDB) with security permissions for port 22 SSH and can do this command below successfully to the EC2 $ ssh -i "AWSshkeyForEC2.pem" ubuntu@xxxx.compute-1.amazonaws.comNo problem.I then try to setup a tunneling (for the DocumentDB at port 27017)issuing this command line$ ssh -i "AWSshkeyForEC2.pem"  -L 27017:docdbwhatever.us-east-1.docdb.amazonaws.com:27017   ubuntu@xxxx.compute-1.amazonaws.com  -N and saysWarning: Permanently added 'xxxx.compute-1.amazonaws.com,yyyyy' (ECDSA) to the list of known hosts.and just hangs there?I am on a OSX  machine.Overall I rather create the DocumentDB without the VPC for a DBaaS. There is a reasons for this the live hardware is on site at the factory for data and then the data should be stored at the Mongo Database in the cloud. Other Apps run in the cloud and can reference the DocumentDB. We are not going to have an App in the cloud to access hardware outside of the cloud due to security.So maybe the solution is to somehow create the DocumentDB without the VPC. I saw I could have deleted the default(VPC) and maybe I should have. The address is the magic 27017 of course for Mongo to the DocumentDB.FollowComment"

Answer: "0Firstly, Amazon DocumentDB is a VPC only service, hence it is not possible to create outside a VPC. If you delete the default VPC, you will need to create a new VPC, subnets, security group, DB subnet group, etc. before setting up a new DocumentDB cluster.Secondly, since the command hangs, I would recommend you to check the connectivity setup with subnets and security group. Is the Security Group being used by the DocumentDB cluster allowing access on port 271017 from the private IP of the EC2, or the CIDR range if in same VPC?To test connectivity, you can SSH to the EC2, and try a telnet to DocumentDB endpoint to confirm the connection and port are accessible.CommentShareShivam_Ganswered a year ago0Hi!  Good question.  DocumentDB in this case will require a VPC configuration.To your point, what you are doing is setting up an EC2 instance in the same VPC to forward connections to the DocumentDB as described here: https://docs.aws.amazon.com/documentdb/latest/developerguide/connect-from-outside-a-vpc.html.  Do you have TLS enabled on the DocumentDB Cluster?  If so, you will need to download the public key.  This could also be an issue with the network connectivity of the EC2 instance or the connection between the EC2 and the DocumentDB cluster (Security Groups, other network config).If you want to use a DB Service that does not require a VPC configuration, check out DynamoDB: https://aws.amazon.com/dynamodb/.  AWS calls this a "a fully managed, serverless, key-value NoSQL database."CommentSharejsoncanswered a year ago"


Question: IoT Core to Kinesis

I can define action in IoT message rule and push data to Kinesis Data Stream(single shard). Since Kinesis is going to assign sequence number to the messages as they come from Iot Core.Is there any scalability issue between iot core and Kinesis because of single shard. Example IoT core receives 1000 message and rules redirect all of them to Kinesis. Will kinesis be able to handle? what is the capacity?FollowComment

Answer: "0Hi Ozy smile. Please refer to the Kinesis Data Streams quotas and limits: https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.htmlA single shard can ingest up to 1 MB of data per second (including partition keys) or 1,000 records per second for writes. Similarly, if you scale your stream to 5,000 shards, the stream can ingest up to 5 GB per second or 5 million records per second. If you need more ingest capacity, you can easily scale up the number of shards in the stream using the AWS Management Console or the UpdateShardCount API.CommentShareEXPERTGreg_Banswered 7 months agoOzy smile 7 months agoThanks you for the answer, one more quick question, if I increase the number of the shards, is sequencing/ordering will be valid only in same shard? I mean is each shard independent from each other? ThanksShareGreg_B EXPERT7 months agoI'm quite sure the ordering is per shard, but struggling to find a documentation reference to point to. The sequence number is "unique per partition-key within its shard": https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#sequence-numberPlease be aware that for AWS IoT Core:The message broker doesn't guarantee the order in which messages and ACK are received.https://docs.aws.amazon.com/iot/latest/developerguide/mqtt.html#mqtt-differencesShare"


Question: OpenSearch Saml SP-initiated login through IAM Identity Center SSO

"Hello, i m looking for a way to activate the SP-Initiated Login to OpenSearchI have an OpenSearch that runs into an account aws A and i have another account B where OpenSearch is an application into my IAM Identity center.The login from the aws sso page works perfectly but i don't know how to enable the SP-initiated login. The documentation here https://docs.aws.amazon.com/opensearch-service/latest/developerguide/saml.html give informations about how to but i m not able to same into my IdP (IAM IdentityCenter)FollowComment"


Question: XA Global Transaction Support for Aurora Postgres

"Does AWS Aurora with postgresql engine support XA transactions meaning a transaction manager could do 2 phase commit operations on both the Aurora postgres as well as another messaging queue in a single transaction.Also if it is supported, does the XA transaction feature work with Aurora Global Database across multiple regions or does it work only within one region?Finally, does RDS postgres support the same XA transaction functionality and also if there are any constraints with that XA support between multi-region vs single region if at all it is supported on RDS postgres?FollowComment"

Answer: "1Both Aurora PostgreSQL and RDS PostgreSQL fully support the SQL capabilities of the open source community's PostgreSQL versions.  PostgreSQL supports 2 phase commit (2PC) SQL operations for PREPARE TRANSACTION, COMMIT PREPARED and ROLLBACK PREPARED operations for transactions managed by an external Transaction Manager (TM).   See the community PostgreSQL documentation at https://www.postgresql.org/docs/14/sql-prepare-transaction.html.  These SQL Operations are available on all version of Aurora PostgreSQL and RDS PostgreSQL currently supported in AWS.  Please note some of the nuances of using PREPARE TRANSACTION noted in documentation link above.Aurora Global Database currently has an architecture with a single primary region where writes occur and up to five secondary regions which are read-only versions of the primary region's database and synchronized using asynchronous replication of log records managed at the Aurora Storage layer.  Since all regions have the same data, albeit subject to very small replication lag, we do not need two-phase commit protocols.  If you plan to use 2PC coordinating different types of Resource Managers (RM) such as different databases, message queues, etc. across different regions, you will need to carefully plan how to coordinate and deal with different cross-region data consistency models and timing.  Certainly, your TM could manage transactions for independent RMs across regions but obviously there will be WAN latency considerations with this approach.Hope this helps!CommentSharePThawleyanswered a year ago"


Question: Connection Timeout Issue with DocumentDB

"I created an EC2 instance and a DocumentDB cluster, they belong to different VPC ID, from the document https://docs.aws.amazon.com/documentdb/latest/developerguide/connect-from-outside-a-vpc.html , to directly connect to DocumentDB (access port 27017), I need to use the EC2 instance running in the same VPC as DocumentDB cluster, there's no way to do it because the VPN running DocumentDB is not showing up in my VPC list, can anyone tell me how resolve this issue?ncat ( nc -zv <documentdb hostname> 27017) returned timeout errorFollowComment"

Answer: "1Accepted AnswerIf the EC2 instance is in another VPC  - VPC Peering may be a better way to connect privately without the overhead of a VPN.https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.htmlThere is a cost associated with VPC peering.  If possible you might consider snapshotting your EC2 instance into a custom AMI and launching it in the same VPC as your DB Cluster, it can be in a different subnet.  This way you would save on the VPC peering costs.CommentSharetedtrentanswered 6 months ago"