Question: [Greengrass][IDT] test component and MQTT failed with general info

"At current, my Greengrass core device (v2.5.5), is on my rasp 3b+ running normally with manual deployment from local/cloud.However, the IDT (version 4.5.3 with suite=GGV2Q_2.3.1) execution passed for 3/6 test cases as below only:========== Test Summary ==========Execution Time:         1m51sTests Completed:        6Tests Passed:           3Tests Failed:           3Tests Skipped:          0Test Groups:pretestvalidation:          PASSEDversion:                    PASSEDcoredependencies:           PASSEDmqtt:                       FAILEDcomponent:                  FAILEDFailed Tests:Group Name: mqttTest Name: mqttpubsub; Reason: Failed at 'my device is running GreengrassGroup Name: componentTest Name: cloudcomponent; Reason: Failed at 'my device is running Greengrass'Test Name: localcomponent;  Reason: Failed at 'my device is running Greengrass'Is there further detail/meaningful info from these error log/code?I have tried to look into the log folder but nothing else there.I have attached all failed logs as below:localcomponent:time="22:48:06+07:00" level=info msg=22:48:06.237 [localdeployment] [INFO] com.aws.greengrass.testing.features.LoggerSteps - Attaching thread context to scenario: 'A component is deployed locally using CLI'time="22:48:06+07:00" level=info msg=22:48:06.238 [localdeployment] [INFO] greengrass/features/localdeployment.feature - line 4: 'my device is registered as a Thing'time="22:48:08+07:00" level=info msg=22:48:08.085 [localdeployment] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Created IotPolicy in IotLifecycletime="22:48:09+07:00" level=info msg=22:48:09.615 [localdeployment] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Created IamPolicy in IamLifecycletime="22:48:10+07:00" level=info msg=22:48:10.194 [localdeployment] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Created IamRole in IamLifecycletime="22:48:10+07:00" level=info msg=22:48:10.639 [localdeployment] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Created IotRoleAlias in IotLifecycletime="22:48:10+07:00" level=info msg=22:48:10.837 [localdeployment] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Created IotPolicy in IotLifecycletime="22:48:11+07:00" level=info msg=22:48:11.040 [localdeployment] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Created IotThingGroup in IotLifecycletime="22:48:12+07:00" level=info msg=22:48:12.066 [localdeployment] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Created IotCertificate in IotLifecycletime="22:48:12+07:00" level=info msg=22:48:12.466 [localdeployment] [INFO] com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Created IotThing in IotLifecycletime="22:48:14+07:00" level=info msg=22:48:14.966 [localdeployment] [INFO] greengrass/features/localdeployment.feature - line 5: 'my device is running Greengrass'time="22:48:14+07:00" level=info msg=22:48:14.976 [localdeployment] [ERROR] greengrass/features/localdeployment.feature - Failed at step: 'my device is running Greengrass'time="22:48:14+07:00" level=info msg=com.google.inject.ConfigurationException: Guice configuration errors:time="22:48:14+07:00" level=info msg=1) [Guice/ErrorInUserCode]: Unable to method intercept: GreengrassStepstime="22:48:14+07:00" level=info msg=  while locating GreengrassStepstime="22:48:14+07:00" level=info msg=22:48:14.990 [localdeployment] [INFO] greengrass/features/localdeployment.feature - line 9: 'I create a Greengrass deployment with components'time="22:48:14+07:00" level=info msg=22:48:14.990 [localdeployment] [INFO] greengrass/features/localdeployment.feature - line 11: 'I deploy the Greengrass deployment configuration'time="22:48:14+07:00" level=info msg=22:48:14.991 [localdeployment] [INFO] greengrass/features/localdeployment.feature - line 12: 'the Greengrass deployment is COMPLETED on the device after 180 seconds'time="22:48:14+07:00" level=info msg=22:48:14.991 [localdeployment] [INFO] greengrass/features/localdeployment.feature - line 13: 'I verify greengrass-cli is available in greengrass root'time="22:48:14+07:00" level=info msg=22:48:14.991 [localdeployment] [INFO] greengrass/features/localdeployment.feature - line 14: 'I create a local deployment with components'time="22:48:14+07:00" level=info msg=22:48:14.991 [localdeployment] [INFO] greengrass/features/localdeployment.feature - line 16: 'the local Greengrass deployment is SUCCEEDED on the device after 120 seconds'time="22:48:14+07:00" level=info msg=22:48:14.991 [localdeployment] [INFO] greengrass/features/localdeployment.feature - line 17: 'the aws.greengrass.LocalHelloWorld log on the device contains the line "Hello World!!" within 20 seconds'time="22:48:16+07:00" level=info msg=22:48:16.569 [localdeployment] [INFO] com.aws.greengrass.testing.features.LoggerSteps - Clearing thread context on scenario: 'A component is deployed locally using CLI'time="22:48:16+07:00" level=info msg=22:48:16.577 [localdeployment] [ERROR] com.aws.greengrass.testing.launcher.reporting.StepTrackingReporting - Failed: 'A component is deployed locally using CLI': Failed at 'my device is running Greengrass'time="22:48:16+07:00" level=info msg=22:48:16.584 [] [] [INFO] com.aws.greengrass.testing.modules.AWSResourcesCleanupModule - Cleaned up com.aws.greengrass.testing.resources.iam.IamLifecycle$$EnhancerByGuice$$10441879@e128dc2time="22:48:23+07:00" level=error msg=Test exited unsuccessfully error=exit statuscloudcomponent:2022-May-04 15:37:28,499 [cloudComponent] [idt-b47156801aee3f29c860] [INFO] greengrass/features/cloudComponent.feature - line 9: 'I create a Greengrass deployment with components'2022-May-04 15:37:28,499 [cloudComponent] [idt-b47156801aee3f29c860] [INFO] greengrass/features/cloudComponent.feature - line 11: 'I deploy the Greengrass deployment configuration'2022-May-04 15:37:28,500 [cloudComponent] [idt-b47156801aee3f29c860] [INFO] greengrass/features/cloudComponent.feature - line 12: 'the Greengrass deployment is COMPLETED on the device after 180 seconds'2022-May-04 15:37:28,500 [cloudComponent] [idt-b47156801aee3f29c860] [INFO] greengrass/features/cloudComponent.feature - line 13: 'the com.aws.HelloWorld log on the device contains the line "Hello World!!" within 20 seconds'2022-May-04 15:37:28,500 [cloudComponent] [idt-b47156801aee3f29c860] [INFO] greengrass/features/cloudComponent.feature - line 15: 'I create a Greengrass deployment with components'2022-May-04 15:37:28,500 [cloudComponent] [idt-b47156801aee3f29c860] [INFO] greengrass/features/cloudComponent.feature - line 17: 'I deploy the Greengrass deployment configuration'2022-May-04 15:37:28,500 [cloudComponent] [idt-b47156801aee3f29c860] [INFO] greengrass/features/cloudComponent.feature - line 18: 'the Greengrass deployment is COMPLETED on the device after 180 seconds'2022-May-04 15:37:28,500 [cloudComponent] [idt-b47156801aee3f29c860] [INFO] greengrass/features/cloudComponent.feature - line 19: 'the com.aws.HelloWorld log on the device contains the line "Hello World Updated!!" within 20 seconds'2022-May-04 15:37:31,557 [cloudComponent] [idt-b47156801aee3f29c860] [INFO] com.aws.greengrass.testing.features.LoggerSteps - Clearing thread context on scenario: 'As a developer, I can create a component in Cloud and deploy it on my device'2022-May-04 15:37:31,558 [cloudComponent] [idt-b47156801aee3f29c860] [ERROR] com.aws.greengrass.testing.launcher.reporting.StepTrackingReporting - Failed: 'As a developer, I can create a component in Cloud and deploy it on my device': Failed at 'my device is running Greengrass'mqttpubsub:2022-May-04 15:36:55,929 [mqtt] [idt-614fb227a4f0913ba4be] [INFO] greengrass/features/mqtt.feature - line 9: 'I create a Greengrass deployment with components'2022-May-04 15:36:55,931 [mqtt] [idt-614fb227a4f0913ba4be] [INFO] greengrass/features/mqtt.feature - line 12: 'I deploy the Greengrass deployment configuration'2022-May-04 15:36:55,931 [mqtt] [idt-614fb227a4f0913ba4be] [INFO] greengrass/features/mqtt.feature - line 13: 'the Greengrass deployment is COMPLETED on the device after 180 seconds'2022-May-04 15:36:55,931 [mqtt] [idt-614fb227a4f0913ba4be] [INFO] greengrass/features/mqtt.feature - line 14: 'the aws.greengrass.IotMqttSubscriber log on the device contains the line "Subscribed to IoT topic idt/Mqtt/Test with QOS=AT_LEAST_ONCE" within 20 seconds'2022-May-04 15:36:55,931 [mqtt] [idt-614fb227a4f0913ba4be] [INFO] greengrass/features/mqtt.feature - line 15: 'the aws.greengrass.IotMqttPublisher log on the device contains the line "Published to IoT topic idt/Mqtt/Test with payload test message and qos AT_LEAST_ONCE" within 10 seconds'2022-May-04 15:36:58,922 [mqtt] [idt-614fb227a4f0913ba4be] [INFO] com.aws.greengrass.testing.features.LoggerSteps - Clearing thread context on scenario: 'Component publishes MQTT message to Iot core and retrieves it as well'2022-May-04 15:36:58,924 [mqtt] [idt-614fb227a4f0913ba4be] [ERROR] com.aws.greengrass.testing.launcher.reporting.StepTrackingReporting - Failed: 'Component publishes MQTT message to Iot core and retrieves it as well': Failed at 'my device is running Greengrass'2022-May-04 15:36:58,930 [] [] [INFO] com.aws.greengrass.testing.modules.AWSResourcesCleanupModule - Cleaned up com.aws.greengrass.testing.resources.s3.S3Lifecycle$$EnhancerByGuice$$11137706@7e3ca22cFollowCommentAWS-hungtn a year agoAs I checked successfully with manual creation and deployment from both of local and cloud.It seems this error during reporting step and should be revised by the test framework only.ShareAWS-hungtn a year agoAs suggestion, I have tried some workaround related to Permission/Role to fix IDT Local/Cloud Component test case, But still get same issue.The ERROR message is triggered during final logging/reporting step (After all functional verification steps were Passed as in log file).--> Failure not related to functionality of Greengrass service(Deployment/MQTT).SharesatvemulAWS 9 months agoHello AWS-hungtnCan you please test with the latest IDT-GGV2 qualification bundle (4.5.8/2.4.0) from https://docs.aws.amazon.com/greengrass/v2/developerguide/dev-test-versions.html.Also, since the errors appeared to be during invocation of Java, it could be to do with Java compatibility issues.Can you try setting up the device with java 11, and see the behavior.On our side, we are also trying to reproduce this issue.Share"

Answer: "1Hi, some things to try, can you first ensure that IDT has the correct permissions? You can follow this: https://docs.aws.amazon.com/greengrass/v2/developerguide/dev-tst-prereqs.html#config-aws-account-for-idtAlso this doesn't look to be the case here, but is a common fix which is if your tests need more time to run, you can increase the --timeout-multiplier to increase timeout.CommentShareAWS-chenthoanswered a year agoAWS-hungtn a year agoI have tried to add more time for the IDT execution with this option '--timeout-multiplier 5' but still got the same result.Here all of prerequisites have been done as you can see the local and cloud component test steps Passed at checking for the component log from local deployment and cloud deployment(upgrade).ShareAWS-hungtn a year agoAs I checked, they all failed at same step "Created IotThing in IotLifecycle" -> "'my device is running Greengrass":time="22:48:12+07:00" level=info msg=22:48:12.466 localdeployment com.aws.greengrass.testing.resources.AbstractAWSResourceLifecycle - Created IotThing in IotLifecycletime="22:48:14+07:00" level=info msg=22:48:14.966 localdeployment greengrass/features/localdeployment.feature - line 5: 'my device is running Greengrass'time="22:48:14+07:00" level=info msg=22:48:14.976 localdeployment greengrass/features/localdeployment.feature - Failed at step: 'my device is running Greengrass'Share1The issue could also be caused due to Guice error per this log -time="22:48:14+07:00" level=info msg=com.google.inject.ConfigurationException: Guice configuration errors:time="22:48:14+07:00" level=info msg=1) Guice/ErrorInUserCode: Unable to method intercept: GreengrassStepstime="22:48:14+07:00" level=info msg= while locating GreengrassStepsWe're investigating this possibility now.CommentShareAWS-User-8006754answered a year agoAWS-hungtn a year agoThank you so much, I am also thinking about it.Is there any way I can attach the configs and logs for the IDT test? I don't see any button to do it here.Share0You can paste the json files here. Please do not post sensitive info like password/keys from the json files, we won't need them for debugging anyway.Can you share the JVM version used for this setup? And also the host machine on which IDT is run - if its windows/mac/linux.CommentShareAWS-User-8006754answered a year agoAWS-hungtn a year agoThe device.json:[{"id": "<pool-id>","sku": "<sku>","features": [{"name": "arch","value": "aarch64"},{"name": "ml","value": "no"},{"name": "docker","value": "no"},{"name": "streamManagement","value": "no"},{"name": "hsi","value": "no"}],"devices": [{"id": "raspberrypi","operatingSystem": "Linux","connectivity": {"protocol": "ssh","ip": "10.3.112.32","port": 22,"auth": {"method": "password","credentials": {"user": "hungtn","password": "*********"}}}}]}]ShareAWS-hungtn a year agoThe userdata.json:{"TempResourcesDirOnDevice": "/home/hungtn/IDT/","InstallationDirRootOnDevice": "/home/hungtn/greengrass/v2","GreengrassNucleusZip": "C:\Users\hung.nguyen\greengrass-nucleus-latest.zip","GreengrassV2TokenExchangeRole": "IDTGreengrassV2TokenExchangeRole"}ShareAWS-hungtn a year agoThe host machine is Windows with JVM version:openjdk version "18" 2022-03-22OpenJDK Runtime Environment Corretto-18.0.0.37.1 (build 18+37-FR)OpenJDK 64-Bit Server VM Corretto-18.0.0.37.1 (build 18+37-FR, mixed mode, sharing)JVM version on my Rasp is:openjdk version "11.0.15" 2022-04-19OpenJDK Runtime Environment (build 11.0.15+10-post-Debian-1deb11u1)OpenJDK 64-Bit Server VM (build 11.0.15+10-post-Debian-1deb11u1, mixed mode)config.json:{"log": {"location": "../logs/"},"configFiles": {"root": "../configs","device": "../configs/device.json"},"testPath": "../tests/","reportPath": "../results/","certificatePath": "../certificates/","awsRegion": "ap-southeast-2","auth": {"method": "file","credentials": {"profile": "default"}}}ShareAWS-User-8006754 a year agoThanks for sharing, sent this info to the team to investigateShareAWS-hungtn a year agoHi @AWS-User-8006754, Is there any other point needed here?This is me(AWS-hungtn), I don't know somehow I have an another account ID now:https://repost.aws/community/users/USK6qDpMOcRVKM6K6yD6dnBQShare0Hi, can you share the json configs used for the IDT run? A zip of the logs generated by the test run would also be helpful.CommentShareAWS-User-8006754answered a year ago0Goodmorning hungtnWe are actively pursuing a resolution to this issue, and are working on an updated IDT release.We will keep you posted once the release is available.CommentSharesatvemulAWSanswered 10 months ago0Hi hungtn,We launched a new IDT v4.5.8 which should fix the Guice issues. Please download it from here. Let us know how it goes.ThanksCommentShareAWS-User-8006754answered 10 months ago"


Question: WAF rule that matches URI AND does NOT contain a string in a header

I want to create a WAFv2 rule that matches part of the URI AND does NOT contain a string in one of the headers.  I see a match type of "Contains string" but do not see how to negate this.  Is this possible?FollowComment

Answer: "0Hi Trever,This is definitely possible in WAFv2, however you will need to use the JSON editor as nesting statement is not supported through visual rule builder at the moment. Please see sample JSON code below (copy and paste to the JSON editor):{  "Name": "RuleContainingURIMatchAndNotHeader",  "Priority": 0,  "Action": {    "Block": {}  },  "VisibilityConfig": {    "SampledRequestsEnabled": true,    "CloudWatchMetricsEnabled": true,    "MetricName": "RuleContainingURIMatchAndNotHeader"  },  "Statement": {    "AndStatement": {      "Statements": [        {          "ByteMatchStatement": {            "FieldToMatch": {              "UriPath": {}            },            "PositionalConstraint": "CONTAINS",            "SearchString": "my-uri-string",            "TextTransformations": [              {                "Type": "NONE",                "Priority": 0              }            ]          }        },        {          "NotStatement": {            "Statement": {              "ByteMatchStatement": {                "FieldToMatch": {                  "SingleHeader": {                    "Name": "MyHeader"                  }                },                "PositionalConstraint": "CONTAINS",                "SearchString": "header-s",                "TextTransformations": [                  {                    "Type": "NONE",                    "Priority": 0                  }                ]              }            }          }        }      ]    }  }}Note that once you go into JSON editor and write rule, you cannot return to the visual rule builder.If you wish to learn more, I also recommend signing up for future webinar where we will do deep dive into rule writing: https://pages.awscloud.com/AWS-Online-Tech-Talks_2020_0317-SID.htmlThanks,CommentSharekumo-hiyorianswered 3 years ago"


Question: Does changing a s3 lifecycle policy change all older objects in an s3 bucket?

"I have an s3 life cycle policy using intelligent tiering where objects go to archive after 90 days and then transition to deep glacier after the next 90. And then remain in deep glacier until the retention policy hits at 305 total days.I'd like to remove the deep glacier rule and let the stuff thats currently on deep glacier age out. But if I remove the deep glacier transition, I'm concerned that all objects already on deep glacier will then hit the archive rule since technically they are over 90 days old, after which all those older objects will transition from deep glacier back to the archive storage tier.So the question is, what's the behavior: Will only objects turning 90 days old transition to the archive tier and all deep glacier objects can age out, or will ALL objects currently in deep glacier transition back to archive?FollowComment"

Answer: "1I would like to inform you that there is no option to apply lifecycle policy to a certain storage class. However if you have versioning enabled and do not have non current version of any objects (that is you have only one version of object) then you can delete the current lifecycle policy and create a new lifecycle policy to expire current version of objects after 305 days and delete the non current version of objects after 1 day. This lifecycle policy will allow you to transition all current version after 305 days to non-current version while deleting it after 1 day of being non current in all storage class. For examples of S3 Lifecycle configuration please refer to below article,https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.htmlCommentShareSUPPORT ENGINEERSathyaanswered a year ago0Thanks you for your reply. I do not have versioning enabled and I am aware that the policy doesn't apply to a storage class. I'm wondering if it applies to all objects in the bucket new and old, or does it only apply to objects turning 90 days old? For example, will objects that are already 300 days old be affected by a policy change that transitions objects at 90 days?CommentSharepdominicbanswered a year ago"


Question: Multipart Upload > 900 seconds

"What is the best practice, using JS SDK v3, to perform a multipart upload of a very large file that takes longer than 15 minutes to complete?  For example, I can imagine a scenario in which 10000 5GB parts can't be uploaded in under 900 seconds.A couple questions which may illustrate my comprehension level:Is the 900 second timeout enforced between the creation of the upload id and the UploadPart execution?  That is my interpretation of the errors I'm getting. Or, is the timeout between UploadPartCommand object creation and execution?Is it possible to use multiple UploadIds to send all the parts of a very large file?If (2) is affirmative, is it correct to assume only one upload id should be used to "complete" the multipart upload and all others should be aborted, if not, how are multiple upload id's handled in order to stitch together parts of multiple uploads into a single object?Am I even in the right ballpark here, i.e., does everything need to complete within 900 seconds? This excerpt from the userguide suggests otherwise: "Pause and resume object uploads - You can upload object parts over time. After you initiate a multipart upload, there is no expiry; you must explicitly complete or stop the multipart upload."Thx.FollowComment"

Answer: "0I've resolved the issue as I suggested previously.Platoon or stagger upload requests into arbitrary batch sizes and execute batches in sequenceWait for all current batch responses before creating the subsequent batchRepeat for all batchesI've requested a documentation modification to indicate the following:A single UploadPartCommand must be received by the server within 15 minutes (900 seconds) of its creationA single UploadPartCommand that is not received by the server within 15 minutes of its creation will fail with a misleading "RequestTimeTooSkewed" error."RequestTimeTooSkewed" error content, and documentation, and customer support all imply or explicitly state that the error is caused by incorrect client system time. In fact, the problem in this case is that the UploadPartCommand was merely created 15 minutes or more before it was received by the server.This (#3) can happen as a matter of course when processing very large files.  It requires only seconds to generate thousands of UploadPartCommand objects, each of will will have effectively the same creation time.  Depending on bandwidth, part size, and part count, it can easily take more than 15 minutes to process these requests.  Simultaneous UploadPartCommand requests must be platooned into manageable batches that can be entirely resolved within 15 minutes, lest subsequent, simultaneously created and queued requests fail.CommentSharevarontronanswered 2 years ago"


Question: Athena queries against mongodb

"Hi,I believe one can run Athena queries against a MongoDB instance using the DocumentDb connector.  I've setup a connection and Athena able to read the collections within my database.When trying to run any query, I get an error like the one below.  This happens even when I use the "Preview Table" query.  One thing I noticed straight away in the error message is that, even though I type my collection name with a starting capital letter (e.g. User) in my query, it's getting reported back in the error in all lowercase (e.g. "user").My example table nameUserHere is a sample query (fake db and collection names used):SELECT * FROM "mydb"."User" limit 10;Here is an example error message:NOT_SUPPORTED: Table has no columns: mongo_db:LambdaTableHandle{tableName=TableName{schemaName=mydb, tableName=user}}This query ran against the "xxx" database, unless qualified by the query.My first guess, based on the above error, is that Athena is lowercasing the collection name, and in so doing, not finding the correct collection.Does anyone know:If this is the case - i.e. that table names are lowercased?If there are settings to change this - i.e. not to lowercase the table name?If there is anything I can tryI'd appreciate any help or guidance!Thanks!FollowCommentwoutercouvaras 10 months agoFor anyone else stumbling upon this, I've found this:  https://github.com/awslabs/aws-athena-query-federation/issues/463#issuecomment-897628798I'll try option 1 as option 2 is not a viable option in my situation.  I'll post the results here (if I can figure it out) :)Share"


Question: Amazon Pinpoint Originating identities for tests in sandbox

"Hi! We're developing international service and trying to use Pinpoint API to send transactional SMS messages for users' phone numbers verification. I'm quite confused with choosing one of Originating identities. First of all, we need to test API integration, using Russian phone numbers, as I see from docs, to verify destinations numbers for tests, we need to have at least one originating number:Amazon Pinpoint sends the verification code using one of the phone numbers in your account. If you are attempting to verify a US-based phone number, note that you must have a dedicated phone number in your account. For more information about purchasing phone numbers, see Requesting a number.And there are short codes, long codes and sender ID. Long codes are not supported for Russia, as this table says: https://docs.aws.amazon.com/pinpoint/latest/userguide/channels-sms-countries.html#sms-support-note-1So the question is, what is the simplest and the cheapest option for originating identity to use in sandbox stage with Russian destination phone numbers to test Amazon Pinpoint SMS API?P.S. When we will add some countries support in our service, do we have to use a new number for each country or one number will be alright for most of countries (except the one which have additional restrictions such as India, for example)?FollowComment"


Question: Redshift Data Transfer Incapability

"We are trying to copy a dataset from EMR to Redshift which consist of around 13 billion records and 20-25 columns. I tried copying the dataset with the traditional method suing the COPY command through S3 Bucket pointing to EMR but it is taking more than 24 hours to copy the dataset.Current EMR configuration is 1 main Node of r5.4xlarge and 2 core nodes of r5.4xlarge and Redshift configuration is 2 ra.3xplus nodes.We have also enabled sortkey, distkey and compression. The dataset is stored in parquet format in S3 bucket.Please suggest a way to copy the dataset in less time as much as possible.FollowComment"

Answer: "0Hello,Thank you for writing on re:PostI understand that you would like to increase the speed at which your Spark application writes to S3.I would recommend experimenting with increasing the size of the executors and/or the number of executors, since it looks like the executors are finishing quite quickly.Other thing we may want to consider include:How data is being partitioned, partitioning the data can often improve write efficiency. This can be done by using a partitionBy with the write.The data format being written, this can have a significant performance impact.We can also consider using Redshift's UNLOAD function if the data isn't being heavily modified in Spark. Redshift's UNLOAD performance is typically quite good. For more information on UNLOAD: https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html#unload-parametersPlease try the above suggestions. Hope they improve your experience.Thanks and have a great day!CommentShareSUPPORT ENGINEERMudasser_Aanswered 8 months agoFabrizio@AWS EXPERT8 months agoHi , while the answer is correct and thorough, the User is asking about COPY command performance.Share0Without analyzing in detail, I know that using Redshift Spectrum to read the files may, in certain cases, be faster than COPY.  E.g. it has the ability to automatically split the file.  If you can catalog this data set as an external table, then do an insert/select, you may get better performance.CommentShareEXPERTRajiv_Ganswered 8 months agoEXPERTFabrizio@AWSreviewed 8 months ago0Hi,I understand you have an application in EMR writing to S3 in Parquet Format and you are trying to load the generated data in Redshift uding the Copy Command.You have built your table using best practices, at the same time the size of your cluster is quite small 2 ra3.xplus nodes (8 cores and 64 GB of memory in total).As mentioned in the documentation you should have multiple files, which is for sure your situation, which are multiple of the number of slices in your cluster.The nodes you are utilizing have 2 slices each, these means you will load 4 files at the time.Load time will depend on 3 factors: total size of the dataset to load ( how many GB in size is the Parquet table ) , the number and size of each file, the parallelism at which you can load.With 13 Billions rows I would expect the table to be large and have thousands of files.If your files are already quite large (closer to the upper range -125 MB at least - mentioned in the documentation I mentioned above; for ease of  read I add an excerpt here:   the file should be: "about equal size, from 1 MB to 1 GB after compression. For optimum parallelism, the ideal file size is 1–125 MB after compression"), you could speed up the copy command would be to scale out the cluster using the Fast Classic Resizing so that you can increase the number of parallel loads.If your files are small ( less than 100MB or even worst less then 10 or 1 MB) then I would first recommend to compact the files in larger ones and retry, if still too slow you have to consider resizing the cluster anyway to increase the load parallelism.Hope this helps.CommentShareEXPERTFabrizio@AWSanswered 8 months ago"


Question: Lambda function cannot run ECS task: unable to retrieve ecr registry auth

"I have run up and torn down a few ECS clusters whilst figuring out how to use them. My latest cluster has a task which runs fine when called from the ECS console "run task" option; however the task will not run if I try to run it using a python lambda function.My lambda code:import boto3def lambda_handler(event, context):    ecs = boto3.client('ecs')    response = ecs.run_task(        cluster='lighthouse-run-cluster',        taskDefinition='lighthouse-run-task-definition:5',        launchType='FARGATE',        networkConfiguration={            'awsvpcConfiguration': {                'subnets': [...],                'securityGroups': [...]            }        }    )The function completes successfully, however the ECS task does not. The task sits in PENDING for some time then stops, with the stopped reason:ResourceInitializationError: unable to pull secrets or registry auth:execution resource retrieval failed: unable to retrieve ecr registry auth:service call has been retried 3 time(s):RequestError: send request failed caused by:Post https://api.ecr.eu-west-2.amazonaws.com/: dial tcp 52.94.49.106:443: i/o timeoutI'm guessing this is some cruft left over from a previous cluster, but I'm not sure how to find the problem. Can anyone help?FollowComment"

Answer: "1Hi Simon,It looks like this is either an IAM permission or a security group permission. I'd recommend making sure that your ecsTaskExecutionRole has the following permissions:{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "ecr:GetAuthorizationToken",                "ecr:BatchCheckLayerAvailability",                "ecr:GetDownloadUrlForLayer",                "ecr:BatchGetImage",                "logs:CreateLogStream",                "logs:PutLogEvents"            ],            "Resource": "*"        }    ]}If that doesn't work, try adding those same permissions to your Lambda function's role as well.You may also need to add this permission to your ecsTaskRole:{            "Action": "ecr:GetAuthorizationToken",            "Effect": "Allow",            "Resource": "*"}If those fail, try checking the security group associated with your Lambda function to make sure it can talk over 443, and if that fails, then check the permissions associated with your ECR repository:{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "AllowCrossAccountPullTest",      "Effect": "Allow",      "Principal": {        "AWS": [          "arn:aws:iam::ACCOUNT_ID:role/ecsTaskExecutionRoleNAME",        ]      },      "Action": [        "ecr:BatchCheckLayerAvailability",        "ecr:BatchGetImage",        "ecr:GetDownloadUrlForLayer"      ]    }  ]}If all else fails, check out this article here that appears to be similar to your issue: https://stackoverflow.com/questions/61265108/aws-ecs-fargate-resourceinitializationerror-unable-to-pull-secrets-or-registryLooks like it could be a networking issue with how you're launching your tasks if they don't have internet accessibility to ECR. Hope this helps!CommentSharegrahamschuckmananswered a year agoSimon Brahan a year agoNone of this made any difference. Thanks though.Share1I fixed this by including 'assignPublicIp': 'ENABLED' to the networkConfiguration:    response = ecs.run_task(        cluster='lighthouse-run-cluster',        taskDefinition='lighthouse-run-task-definition:5',        launchType='FARGATE',        networkConfiguration={            'awsvpcConfiguration': {                'subnets': [...],                'securityGroups': [...],                #                # Added the flag below                #                'assignPublicIp': 'ENABLED'            }        }    )I think this is down to a config problem in our sandbox and it shouldn't be needed in production, but it got me moving.CommentShareSimon Brahananswered a year ago"


Question: "Please I confused, I don't know why I got this error in S3 - Codecommit and Cloudformation"

"I am currently working with this PoC = Automatically build CI/CD pipelines and Amazon ECS clusters for microservices using AWS CDK [https://docs.aws.amazon.com/es_es/prescriptive-guidance/latest/patterns/automatically-build-ci-cd-pipelines-and-amazon-ecs-clusters-for-microservices-using-aws-cdk.html]And I have this error.This is the command I use to launch the deploymentcdk deploy --context aws_account=xxxxxx --context aws_region=eu-west-2 --context vpc_nonprod_id=vpc-0b1ad2009aa6d2106 --context vpc_prod_id=vpc-00b4a914c2628a1b2 --context ecssg_nonprod_id=vpc-0b1ad2009aa6d2106 --context ecssg_prod_id=vpc-00b4a914c2628a1b2 --context code_commit_s3_bucket_for_code=cdk-hnb659fds-assets-xxxxxxxxxx-eu-west-2 --context code_commit_s3_object_key_for_code=arn:aws:kms:eu-west-2:xxxxxxxx:key/d92b32f6-d8cb-4429-adcc-d88d2d3f0ec3 --context microservice_name=devops-iris| CREATE_FAILED        | AWS::CodeCommit::Repository               | xxxxxx-codecommit-idNot Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: GRZCJ67AMTTS1318; S3 Extended Request ID: 3514cWqx2HUsD3cSLgj5zYCEw1IcAp6id5AVbBSDChEnKoFAolNI0iP1jhMgR8v1Iifktm+dYEU=; Proxy: null)"code_commit_s3_object_key_for_code", I don't have idea, How can I know this value?CheersAndyFollowComment"

Answer: "0code_commit_s3_object_key_for_code is obviously not the KMS key you have provided.Assuming you have uploaded the starter-code.zip to a S3 bucket as mentioned in the prerequisites, its the path in the S3 bucket where the code was uploaded to. If root of the bucket, then it would be starter-code.zip else the <prefix>/starter-code.zip--SydCommentShareSydanswered a year ago0First, thank you for the support.It's true, I made a rookie mistake.CommentShareAWS-User-5650834answered a year ago"


Question: Cloudfront cache mutation:

"Can anyone tell me the best practice of caching in cloudfront? Why mutation in the dynamodb table isn't reflecting? I have set the cache behaviour to max TTL:3sec, min TTL:0sec. When I'm refreshing the cloudfront url after 3 sec it is showing the mutation. But in the second refresh after 3 sec it is showing the older result.FollowComment"

Answer: "0Accepted AnswerAre you doing strongly-consistent reads rather than eventually-consistent reads in DynamoDB?  If not, then after a mutation a sequence of reads may return a mix of old and new values until full consistency is achieved.CommentShareEXPERTskinsmananswered 9 months agoAmplify_lover 9 months agoI want something like when any mutation occurs it should pass through cloudfront to dynamoDB. When a mutation will occur cloudfront will match the ID of objects in cache, if id matches it will be updated in CloudFront and then in DynamoDB.Share"


Question: My instance stop working and it show this error on the systemlogs

start request repeated too quickly for systemd-tmpfiles-clean.serviceFollowComment

Answer: "0Hello,I understand that you got the below error on the systemlogs from the instance that stopped working:"start request repeated too quickly for systemd-tmpfiles-clean.service”If the service restart exceeds the value of StartLimitBurst within the time specified by the value of StartLimitInterval, the service startup will fail and lead to that error .Ensure that frequent restarts of your services/applications within a short period of time are the expected behaviour for hat service.When not, it is important to troubleshoot the reason for the frequent restarts of the service.  This existing behaviour exists as a measure to prevent many repetitions of service restarts in a system.  Once the service restart exceeds the value of StartLimitBurst within the time specified by the value of StartLimitInterval, the service startup will fail. StartLimitInterval is set to 10 seconds by default and StartLimitBurst is set to 5 by default.The problem can be avoided by disabling the restrictions, please be mindful of implications before disabling any system setting  and please take note of any changes so they can be undone. To disable checking of service starts, please set StartLimitBurst to a value of 0.Please see the commands below to troubleshoot the problem.Check systemd's file with the following commands:[1] systemctl show -p FragmentPath systemd-tmpfiles-clean.service[2] cat /usr/lib/systemd/system/systemd-tmpfiles-clean.serviceWrite StartLimitBurst=0 in [Service] section:[3] vim /usr/lib/systemd/system/systemd-tmpfiles-clean.service...StartLimitBurst=0Re-load the modified systemd file:[4] systemctl daemon-reloadI hope the above information is helpful. Kindly let me know if you have additional concerns.CommentSharePhindianswered a year ago"


Question: RDP Cannot Connect

"Hello,We are getting issue in connecting RDP of my EC2 Instance.when we try to connect RDP than Showing a Error, Error Details and Code is below:-An internal error has occurred.Error code: 0x4Extended error code: 0x0FollowCommentCuong-Huynh-SP a year agoCan you submit more details log or try to telnet or ping to RDP endpoint, also check that security group of RDP was allow incoming traffic from your instance too,Share"

Answer: "0Hello,Few things to go over here:As commented previously, if you can confirm both the Security Group(s) allow inbound traffic from your location to port 3389? You can further test using PowerShell:PS C:\ > tnc <ip_address> -port 3389If you can reach it, let us know if the Instance is passing both System Checks in the Console, System for AWS underlying host and Instance for OS. If it's failing the first, stop and start the Instance so it lands in a new host, if it is the second, then this is related to the OS.Keep in mind to get a backup of the Instance in the form of AMI or Snapshot before continuing.Check the Screenshot of the Instance to validate there is no issue in the network card (we want to avoid yellow or red icons). If the Instance is managed by SSM, try running SSM Automation and discard these culprits to help troubleshooting.Lastly you can also use Ec2Rescue to see if something was modified in terms of Firewall or settings for Remote Desktop.Reach out to your Sys Admin to also validate no GPOs are cascading to the Instance which may be meddling with connectivity, also sometimes 3rd party software like Antivirus or Firewall could be in the way of connecting.I hope the information above is helpful.CommentShareSUPPORT ENGINEERJosueAWSanswered a year ago"


Question: "Looking for Flexmatch process flow, How to and/or samples (Unity client)"

"based from herehttps://docs.aws.amazon.com/gamelift/latest/flexmatchguide/match-server.htmlGameLift sends an onStartGameSession() for new game session, in the matched game, is this automatically calledor should the client application still have to issue create Game session command?Ive been following the code samples here: https://github.com/BatteryAcid/unity-custom-gamelift-clientthough it doesnt use flexmatch, it calls CreateGameSessionAsync() and in turn calls onStartGameSession in the custom gamelift server.I cant seem to find example codes in unity that uses flexmatch in client side that has a complete process from, starting/looking for match,accepting match and starting a gamesession from that match.I dont get the process and sequence in the client side and process.Anyone can recommend a good resource?FollowComment"

Answer: "0The callback function, onStartGameSession, that is defined by you and passed to ProcessReady gets called whenever a game session is created by GameLift on whatever server process it chose to host that game session. There are many ways to create a game session. Flexmatch is one of them, or you can create a game session through one of the various methods, for e.g. CreateGameSessionAsync, in the GameLift class in the AWS SDK.CommentSharerePost-User-5100160answered a year ago0Hey @REDACTEDUSERJust make terminology clear in this flow, what you've been referencing to as a client is a GameLift client that calls GameLift AWS Service APIs. Usually this is its own service and not a game client that players will directly interact with.This section of the docs has a pretty good flow of how FlexMatch works https://docs.aws.amazon.com/gamelift/latest/flexmatchguide/gamelift-match.html#gamelift-match-howitworks Specifically, during this step is when onStartGameSession will be called (from the GameLift remote server to your game)FlexMatch with a GameLift hosting solution: The game session queue locates the best available game server for the match. Depending on how the queue is configured, it tries to place the game session with the lowest-cost resources and where players will experience low latency (if player latency data is provided). Once the game session is successfully placed, the GameLift service prompts the game server to start a new game session, passing on the matchmaking results and other optional game data.CommentSharerePost-User-4754183answered a year ago0Hi, did you manage to figure this out?CommentShareND Bhavanswered 7 months ago"


Question: Lightsail Django demo app 404 Bad request after setting static IP

"Hi, I’m hosting a demo Django application on an AWS Lightsail instance. It was working correctly at the original assigned IP address, but the documentation recommended adding a static IP address, which I did (it’s 50.112.67.77). I can SSH into the instance using that IP, and the ipv4 firewall rules are set to allow HTTP on port 80 and HTTPS on 443.But since making that change, I keep getting 400 Bad Request when I try to access the Djang site from a browser. I first tried setting ALLOWED_HOSTS=[‘50.112.67.77’]. When that didn’t work, I tried temporarily setting ALLOWED_HOSTS=[‘*’] and DEBUG=True, but still am getting 400 Bad Request.I can still use manage.py runserver to run the app while in an SSH session and access the app in a second SSH session using lynx http://127.0.0.1:8000. When I do that, I see the expected ‘hello world’ http response.I’d appreciate any help in how to troubleshoot this.Thanks!FollowComment"

Answer: "0Accepted AnswerIt started working after I added a letsencrypt cert and restarted services sudo /opt/bitnami/ctlscript.sh restart.  I think the issue with 400 Bad request would have been solved by just restarting services.CommentSharerePost-User-0570699answered 18 days agoAWS-SUM 18 days agoThat is correct. After attaching a Static IP, rebooting the instance or restarting the bitnami services helps update the application config that uses/needs the instance's public IP.Share"


Question: Best way to lock bucket item behind authentication?

"I have an application that sends audio files to a bucket. I then want users to be able to access these audio files attached in a ticket. But of course I want the users to authenticate before accessing the file.The upload now returns me:  https://bucket.s3.eu-central-1.amazonaws.com/RECORDING-478468.mp3 , I was hoping to embed this url to the ticket so users can use this to access the file. Is there a way to restrict what users can use the URL and have them authenticate? As far as I can see it is either public, locked, or programmatic access via CDK's or CLI. Not after clicking the url.I also thought about using pre signed urls, but I want the file to be available for longer and also thought it was unsafe to store voice recordings there for a longer time.Any advice?FollowComment"

Answer: "0You can add authentication to S3 via the AWS Cognito service.Here is the link to the documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_cognito-bucket.htmlCommentSharevtjeananswered 14 days ago0You can use Amazon CloudFront signed URLs.You can distribute private content using a signed URL that is valid for a short time—possibly for as little as a few minutes or for a longer time, possibly for years. Check out below link for more info.https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-urls.htmlCommentShareRavikant Sharmaanswered 14 days ago0There are two things necessary to control access, authentication and then authorization.  Authentication is how you "know" your users.  You haven't mentioned how this works for you today, but without a way to convey that trust to S3 you won't be able to start the authorization process.It's not possible to answer how to authenticate your users in the scope of this answer because that is dependent on your relationship with your users.  If you already have a way of authenticating them, then you can start there.S3 only supports one type of authentication natively, and that is AWS access credentials.  You need to bridge your authentication to the S3 recognized form of authentication.Depending on your authentication method, you may have other tools to help bridge authentication to AWS access credentials.  AWS Cognito is one example which would allow direct communication to S3 without signing or relays.But let's say that's not practical for your scenario. I can outline the general solution space that remains.You have a two major sets of options on how to bridge authentication methods.  There are request signing mechanisms, and then there are message relay mechanisms.  Signing mechanisms have the advantage of not requiring a service you design to relay the bytes of the objects retrieved. Relaying the bytes is practical for smaller volumes,  but at larger volumes may decrease efficiency and introduce scalability challenges that you would rather be handled by a service that was already designed to handle that scale (S3 or CloudFront being the two I'll discuss).If you choose the message relay method, you host a service, your users make requests to it, and your service has access to AWS credentials (temporary credentials recommended).  You use those credentials to make requests on behalf of your user.  You can perform authorization via custom methods and have S3 trust your service.  S3 would in this case allow your service to retrieve any data, but your service wouldn't send any requests that weren't authorized.  It is possible to have authorization still done by S3, but that would be a more complex topic, so I'll avoid discussing it here.If you choose the signing method, you host a service, your users make requests to it, and that service returns signed requests that your users then forward on to another service.  Your service has to perform authorization for the users requests before returning signed requests using the authentication method you have available.  If your service is serving HTTP requests, you can also use redirects.  You'll need to handle Cross-Origin Resource Sharing (CORS) for a browser to accept this type of redirect.For S3 presigned URLs these are signed with AWS access credentials.  User who receive these aren't able to change any of the headers, and there is an expiration time.  Generally the model here is that the presigned URLs should be used quickly, as it's a stand-in for direct access.  If these users need to access the URL again, they can requested again.For CloudFront, Signed URLs, you control a set of public/private key pairs and use these to sign request URLs.  You have more flexibility here as to how long a signed URL would be valid.Now, all this said, the situation you describe, where your trying to embed a link in a ticket, makes me uncertain that you have access to the authentication information of your users at all, and that the only authentication you have is that they managed to land on the page where you want to embed the link.While it's best to know your users for the entire flow, if you are in this situation in which you only have that information, you can create a CloudFront Signed URL and embed it directly, rather than a link to a service that validates authentication and authorization.  But keep in mind that the URL there wouldn't retain any authentication or authorization controls.. if users forward the content that contains the link, it would be like them forwarding the content of the link too.You'll have to evaluate if that fits the use case you have in mind.  If you trust the users who have access to the ticketing system highly, and the content from the link is not too sensitive, it might be acceptable, but be sure to evaluate the sensitivity and your expectation of user behavior.  If you're uncertain, I'd suggest a solution with lower expirations, and more live authentication processes.CommentShareAWS-User-5738042answered 10 days ago"


Question: SSIS Best Practices with Amazon Redshift

A Customer uses SSIS in their SQL Server environment to perform ETL.  They would like to move to Amazon Redshift for all the value Redshift brings to data warehousing.What are the best practices for integrating Redshift into a SSIS workflow?FollowComment

Answer: "0Accepted AnswerCustomers coming from SQL Server might first start with an Amazon Redshift pilot:Create their first Redshift cluster (and secondary AWS things like AWS VPC subnet groups, Amazon S3 buckets, etc.).Load some sample data and get familiar with Redshift fundamentals (COPY command, zone mapping, etc.)Hook up their favorite BI tool to check for compatibility.As they progress, they can load a bulk copy of data from their existing MSSQL database using AWS DMS so they can experiment with their existing data model, see how it performs in Redshift, find out changes in data types, changes to their queries (especially things like PIVOTs), etc.Finally, when they are ready to start integrating their data sources directly into Redshift, they might find they don't need SSIS and they can explore cloud-native services such as Glue, or Marketplace offerings like Matillion. If there is a hard requirement for SSIS, I would recommend a 3rd party plugin called ZappySys. It handles the Redshift loading best practices (i.e. staging data in S3, using COPY commands on multiple files in S3, etc.).Hope this helps!CommentShareLaith-at-AWSanswered 4 years ago"


Question: SecretsManagerRotation Lambda getting created even with false condition

"Hi,I am trying to use Secret Rotation functionality however want to keep it optional based on user input.When I run the template its ignoring all the resources where the condition is false however still goes ahead and creates the SecretsManagerRotation Lambda function.To elaborate further based on below shortened template , even when condition PasswordRotationEnabled is false (parameter PasswordRotation = -1) , CF attempts to create the Lambda function. It however does not create the SecretsManagerVPCEndpoint which means that condition is working as expected.I also noticed that if I remove/comment out "Transform: AWS::SecretsManager-2020-07-23" from the template then CF does NOT create the lambda function.To my mind, the existence of  "Transform: AWS::SecretsManager-2020-07-23" itself triggers the creation of Lamdba.Unless I am doing something wrong, this seems to be bug. Kindly help.AWSTemplateFormatVersion: 2010-09-09Transform: AWS::SecretsManager-2020-07-23Parameters:PasswordRotation:Description: Specify the password rotation cycle in days (1-1000). Use -1 to disable.Type: NumberDefault: -1Conditions:PasswordRotationEnabled: !Not [!Equals [!Ref PasswordRotation, -1]]Resources:SecretsManagerVPCEndpoint:Type: AWS::EC2::VPCEndpointCondition: PasswordRotationEnabledProperties:SubnetIds:- !Ref PrimarySubnetId- !Ref SecondarySubnetIdSecurityGroupIds:- !Ref DatabaseSecurityGroupVpcEndpointType: 'Interface'ServiceName: !Sub "com.amazonaws.${AWS::Region}.secretsmanager"PrivateDnsEnabled: trueVpcId: !Ref DatabaseVpcIdSecretRDSInstanceAttachment:Type: AWS::SecretsManager::SecretTargetAttachmentCondition: PasswordRotationEnabledProperties:SecretId: !Ref MasterUserPasswordTargetId: !Ref DBInstanceTargetType: AWS::RDS::DBInstanceMySecretRotationSchedule:Type: AWS::SecretsManager::RotationScheduleCondition: PasswordRotationEnabledDependsOn: SecretRDSInstanceAttachmentProperties:SecretId: !Ref MasterUserPasswordHostedRotationLambda:RotationType: MySQLSingleUserRotationLambdaName: SecretsManagerRotationVpcSecurityGroupIds: !Ref DatabaseSecurityGroupVpcSubnetIds:Fn::Join:- ","- - !Ref PrimarySubnetId- !Ref SecondarySubnetIdRotationRules:AutomaticallyAfterDays: 1Edited by: DivAWS on Apr 15, 2021 7:42 AMFollowComment"

Answer: "0Answering my own question -If a CF template references AWS::SecretsManager, CF generates a Lambda function to perform secrets rotation. We cannot make it conditional if Transform is present.We have two options to create the AWS pre-defined Lamdba secret-rotation functions for databases.:Refer to below link which provides the source code of Lambda functions: https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_available-rotation-templates.htmlYou can find the template in AWS Serverless Application Repository. You can then either create the function using AWS console or Create it as a SAM resource via CF template.CommentShareDivAWSanswered 2 years ago"


Question: Add "insecure" to export options on NFS server.

"Hello,I am running into an issue mounting our NFS server on some osx devices, with the errorcan't mount /<name> from <ip> onto /Users/mac: Operation not permittedDid some googling and one suggested solution was to add “insecure” to the export options on the NFS server side, located in the /etc/exports file on the server.However, I seem to be unable to get into the files on the EC2 server we're hosting it on. It won't allow me to log in as root and admin just takes me to a management console for the storage gateway which doesn't have any of the normal commands. cd, sudo, ls, etc are all unavailable.How can I get into the server to change this setting? Or if there's a better way to fix this issue, i'm all ears.Thanks!FollowComment"

Answer: "0Accepted AnswerHi,This looks like a permissions issue. Are you trying to mount the NFS share to a home directory? Try changing the mount point to an empty directory where you have write permissions. You may need to use "sudo".Cheers,JKCommentShareJohn Kennedy - AWSanswered 4 years ago0Thank you, it was indeed a permissions issue. We were trying to mount the fileshare to ~CommentSharerileyhempanswered 4 years ago"


Question: AWS should not take capital letters in DNS URL

"Why is AWS ELB field designed to allow capital letters ?The global truth is that DNS in browser always is parsed as lowercase letters . When using them as PUBLIC IP in applications, we must parse it as lowercase as well although AWS Load Balancer field allows capital letters. Then why allow capital letters at all ? Should not this be an improvement on AWS side ?FollowComment"


Question: iam role trust policy behavior

"Hiaws docs mentioned that iam role trust policy  should be treated as a resource based policy but in fact it doesn't .regularly iam user could get a permission from his identity policy (resource level permission) lets say s3:getobject then he will be allowed to do that action unless an explicit deny exist  regardless of the default implicit deny on bucket policy .so in case of iam role trust policy  , lets say :role "A" trust user "B" in the same account if another user "C"  in the same account had "sts:assumerole" permission in his identity based policy as a resource level permission then he should be able to assume the role even if user "c" is not in the trust policy which does not happen .the current behavior is more like an explicit deny for any principal not specified in the trust policy .it is not the default/documented behavior of the resource based policy which should be an implicit deny .any thoughts ?thanksFollowComment"

Answer: "1Accepted AnswerThe documentation has been update to account for this exception.Role trust policies and KMS key policies are exceptions to this logic, because they must explicitly allow access for principals.https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html#policy-eval-denyallowCommentShareEXPERTkentradanswered 10 months agoYasserAlhawary 10 months agoActually I read that before but didn't notice, Thanks AlotShare1A user or resource can only assume an identity given the user/resource has "sts:assumerole" permissions for Role A, and role A trusts the user or the entire account that includes users B and C. However, User C would not be able to assume role A unless trusted by Role A even with "sts:assumerole" as a result of least privilege. Principles are not allowed to assume a role unless they are explicitly allowed to in the role’s trust policy. This is because there is an implicit deny by default. An explicit deny would require a Deny statement which would override any allow. This is done to prevent user C from assuming a role with more permissions than they should be allowed.Attaching the following documentation regarding role trust policies here.https://aws.amazon.com/blogs/security/how-to-use-trust-policies-with-iam-roles/CommentShareDavid_Tanswered 10 months ago"


Question: Marketplace Product ID Generation

"Hello,I want to publish a product through AWS Marketplace, and we need to rely on the Product ID to perform some housekeeping tasks within the product, I have some questions,What is the recommended approach for experimenting?Are product IDs dynamically generated every time you publish the product?Can we add any other metadata to the product identity document?Thanks in advance!FollowComment"

Answer: "1The publishing process for AWS Marketplace includes a step after building a product listing where the product is available to you and allowlisted test accounts but not to the public, allowing for testing.  The Product ID is per product listing, each publishing event creates a new version of the product but the Product ID is the same.Information on Metadata is available in the AWS Marketplace Seller Guide: https://docs.aws.amazon.com/marketplace/latest/userguide/categories-and-metadata.htmlCommentSharethat_mikeanswered 10 months ago"


Question: API Error N. Virginia ?

"I cannot access my services via Dashboard, the Ressources section on Dashboard indicates API Error.See screenshot : https://drive.google.com/file/d/1Spo99NrydNETG4unh9UI3odrFpUAIWa9/view?usp=sharingOn Health Dashboard, all services are marked functional.Anyone can help me please ?FollowComment"

Answer: "3Hi,There is an Operational Event affecting the US-East-1 region: https://status.aws.amazon.com/RegardsCommentShareVazmigueanswered a year agoEXPERTHernitoreviewed a year agoMODERATORAdeebreviewed a year agoEXPERTScottreviewed a year ago"


Question: How to get semantic segmentation output using Batchtransform

"Hello,I've been trying to use batch transform to perform inference on a semantic segmentation problem but I've been struggling to get the correct output.The code of my estimator is:ss_estimator.set_hyperparameters(    backbone="resnet-101",  # This is the encoder. Other option is resnet-101    algorithm="fcn",  # This is the decoder. Other options are 'psp' and 'deeplab'    use_pretrained_model="True",  # Use the pre-trained model.    crop_size=320,  # Size of image random crop.    num_classes=3,  # Pascal has 21 classes. This is a mandatory parameter.    epochs=30, # Number of epochs to run.    learning_rate=0.0001,                                 optimizer='adam', # Other options include 'adam', 'rmsprop', 'nag', 'adagrad'.    lr_scheduler='poly', # Other options include 'cosine' and 'step'.       mini_batch_size=4,  # Setup some mini batch size.    validation_mini_batch_size=4,    early_stopping=True,  # Turn on early stopping. If OFF, other early stopping parameters are ignored.    early_stopping_patience=5,  # Tolerate these many epochs if the mIoU doens't increase.    early_stopping_min_epochs=2,  # No matter what, run these many number of epochs.    num_training_samples=num_training_samples,  # This is a mandatory parameter, 1464 in this case.)After training the model, I am configuring the batch transform as below:timestamp = time.strftime("-%Y-%m-%d-%H-%M-%S", time.gmtime())batch_job_name = "image-segmentation-model-batch" + timestampmodel_name = "DEMO-full-image-segmentation-model" + time.strftime(    "-%Y-%m-%d-%H-%M-%S", time.gmtime())output_folder="s3://client/private_datasets/experiments/batch_transform_output/"batch_input = "s3://client/private_datasets/experiments/test-images-small/"request = {    "TransformJobName": batch_job_name,    "ModelName": model_name,    "MaxConcurrentTransforms": 16,    "MaxPayloadInMB": 6,    "BatchStrategy": "SingleRecord",    "TransformOutput": {"S3OutputPath": "{}/output".format(output_folder)},    "TransformInput": {        "DataSource": {"S3DataSource": {"S3DataType": "S3Prefix", "S3Uri": batch_input}},        "ContentType": 'image/jpeg',        "SplitType": "None",        "CompressionType": "None",    },    "TransformResources": {"InstanceType": "ml.c5.xlarge", "InstanceCount": 1},}print("Transform job name: {}".format(batch_job_name))print("\nInput Data Location: {}".format(batch_input))And I'm starting the batch transform job as:sagemaker = boto3.client("sagemaker")sagemaker.create_transform_job(**request)print("Created Transform job with name: ", batch_job_name)while True:    response = sagemaker.describe_transform_job(TransformJobName=batch_job_name)    status = response["TransformJobStatus"]    if status == "Completed":        print("Transform job ended with status: " + status)        break    if status == "Failed":        message = response["FailureReason"]        print("Transform failed with the following error: {}".format(message))        raise Exception("Transform job failed")    time.sleep(30)My problem is that the batch transform outputs a "*.out" file, which I have no idea how to open.Can you help me out? Or is there some tutorial to use batch transform for mask prediction/image segmentation?FollowComment"


Question: ACM Notifications for expiring certificates

"Customer wants an internal team to be notified when a certificate is approaching expiration so that they can respond and investigate the issue.The plan is to issue public certs with auto-renewal configured so the primary goal is to get notified if that auto-renewal process is failing but a customer can accomplish that by simply getting a notification if the cert is less than 60 days from expiring, since AWS will auto-renew before the 60 day expiration timer.  What are the available options to do this in AWS ?FollowComment"

Answer: 0Accepted AnswerCustomers can make use of the acm-certificate-expiration-check AWS Config managed rule and get notified using the AWS-PublishSNSNotification SSM automation document to publish a notification to Amazon SNS.CommentShareAWS-User-2530299answered 3 years ago


Question: Does changing the protocol of the aws alb taraget group to http2 improve the speed? And how to set it up?

"I know that changing from http1 to http2 improves the speed.So I allowed alb and cloudfront to be http2.In the developer tools of the Chrome browser, I confirmed that the protocol was h2 and http2 was applied.However, after searching for stalled, there was an opinion that changing the protocol of the aws elb target group to http2 would improve stalled and improve speed.So I changed the protocol to http2 with the same settings as http1, but a 502 error occurred. (When connecting to a random port (eg: 8080 port) with the load balancer dns address) There was no error in the http1.1 setting.The question is, if h2 is displayed when checking the protocol in the browser, is it applied to http2 or should the target group protocol be set to http2 internally?If I need to set it up, how can I avoid getting a 502 error?FollowComment"

Answer:
"0Accepted AnswerWhen you enable ALB target group protocol as HTTP/2 you need to make sure that target support protocol version HTTP/2 as well. For example if you are using nginx you can update server section as :server {        listen       80 http2 ;        listen       [::]:80 ;If server is not configured to support protocol version, Target will send 400 bad request to ALB as ALB was making HTTP/2 request. Tested this on my env and packet capture on target show below response towards ALB.PRI * HTTP/2.0HTTP/1.1 400 Bad RequestServer: nginx/1.20.0Date: Thu, 23 Feb 2023 05:06:42 GMTContent-Type: text/htmlContent-Length: 157Connection: close<title>400 Bad Request</title>
400 Bad Request

nginx/1.20.0Request packet from ALB to Target:As the ALB get this 400 Bad request from target it can generate 502 back to the client.If you have already configured your Target for HTTP2, then you can check other possible reasons for 502 here.As far as performance , you can get the benefit of compressed header when you have end to end HTTP2, which can save some data transfer cost.CommentSharenavazanswered 3 months ago"


Question: Error in Athena: database does not exist

"Hi !I have been a long time trying to solve this error without success. I have already verified the existence of the database, the region, etc. Any help would be great. Thanks!FollowComment"

Answer: "1I have faced this issue earlier and I feel the error message here is misleading. I was able to resolve this issue by providing access to the database and tables through Lakeformation. Let me explain what I had to do:Do you have Lake formation turned on? If so, you would have to provide Database level and table level access from there. The fact that you are seeing your databases and tables implies that you have you have describe access on the database/table but not select access.You don't have lakeformation turned on? Check if your user/role has these permissions: "glue:GetDatabases", "glue:GetDatabase","glue:GetTables", "glue:GetPartitions","glue:GetPartition","glue:GetTable"In some cases, this could be due to compatibility of the table such as XML table - https://aws.amazon.com/premiumsupport/knowledge-center/athena-glue-tables-not-visible/If you still have the issue, verify that your tables, columns, and queries comply with Athena requirements. For more information, see Names for Tables, Databases, and Columns and Troubleshooting in the Athena User Guide.CommentShareAnanth Tirumanuranswered 5 months agorePost-User-4649751 5 months agoUnfortunately this didn't work. I verified that the Lake formation is off, I have all the permissions needed in the user/role, my tables meet the formatting requirements and my tables and components comply with Athena requirements. If you discover another reason for the error would be great.Share"


Question: Unable to connect ECS Service to ALB Target Group in separate account

"I am using the Compliant Framework for Federal and DoD Workloads in AWS GovCloud. I have 3 Mission Application Accounts corresponding to each of my environments (dev, qa, prod). Within each Mission Application Account I have an ECS Service that creates one ECS Fargate Task. I am attempting to use Terraform to configure the ECS Service to connect the Fargate Task to an IP style Target Group attached to an ALB I have in a separate (Transit) account (I have been able to do this manually already without the ECS Service by just spinning up the Fargate Task on its own and passing the private IP to the Target Group), however, I receive an error saying:InvalidParameterException: Unable to assume role and validate the specified targetGroupArn. Please verify that the ECS service role being passed has the proper permissionsI attempted pass in a custom IAM Role that had said permissions but was given this error:InvalidParameterException: You cannot specify an IAM role for services that require a service linked role.As Service Linked Roles do not seem to be editable, I am unsure on how to provide the proper permissions to my ECS service in my Mission Application Account to the Target Group in my Transit Account.FollowComment"

Answer: "0Actually in case of ECS, the load balancer/ target group should be in same account in which ECS cluster exist. It is not possible to share target group from other account.CommentShareSUPPORT ENGINEERAshish_Tanswered 4 months ago"


Question: Lambda Memory Quota Inconsistent in São Paulo region

"Hi,I had deployed a Lambda function in the São Paulo region with 2500 MB of memory. I wanted to increase the allocated memory to 3500 MB through the Console, but received a quota error: 'MemorySize' value failed to satisfy constraint: Member must have value less than or equal to 3008.I understand that 3008 MB used to be the memory quota, but the documentation lists the memory quota as "128 MB to 10,240 MB, in 1-MB increments", independently of region as far as I can tell, and the Service Quota Dashboard in the São Paulo region lists "Function memory maximum" as "10,240 megabytes" as well under "AWS default quota value".Is there a way to enable the higher memory quota of 10,240 MB?Thank you.FollowComment"

Answer: "0Please raise a ticket with the support team if you require higher memory support in Lambda.CommentShareEXPERTBrettski-AWSanswered a year agoshai a year agoI tried to do sit through the Support Center, but the option to request memory support increases does not seem to be there. The options I get for the "Lambda" type of request are "Concurrent Requests", "Elastic network interfaces per VPC" and "Function and layer storage". The documentation also says that the memory limit can't be changed.ShareBrettski-AWS EXPERTa year agoRaise the ticket against "Function and layer storage". It'll get routed appropriately.Share"


Question: Using Cognito and Cloudfront to control access to user files on S3

"Hi, I'm putting together a media viewer website for myself to learn how AWS works. My first step was to host a webpage (index.html) on S3, and have this webpage allow for image/video uploads to a folder in my bucket using the AWS Javascript SDK (v2), and having the mediaviewer on the web page access these files directly through http. I have lambda functions that convert media formats appropriately, and hold metadata in DynamoDB that can be queried by the website using the javascript SDK. This all works fine.Now, I'd like to make it a bit more secure, and support users who login, individual user directories within the buckets, and control access to the media files so users can only view their own files. So the steps I used to do this were the following:Create a user pool and identity pool in Cognito.Add a google sign in button, and enable user pool sign in with the google button... To do this, Google requires the webpage to be served via https (not http).Since S3 can't serve files via https, I put the S3 bucket behind cloudfront.Modify my bucket to have a user directory, and subdirectories for each cognito identityid. Modify the access policies so that users can only read/write to their individual subdirectory, and can only read/write to a subset of DynamoDB based on their identity ID. The webpage uses AWS Javascript SDK calls to login with cognito, upload to S3, access dynamodb. It all appears to work well, and seems to give me secure user access control.Now, the hole... I want the media viewer portion of my app to access the images/media via https:// links, and not via the javascript sdk. The way its currently configured, https access goes through cloudfront, and cloudfront has access to all the files in the S3 bucket. I'm trying to figure out how to make an https request via cloudfront (along with a cognito token), and then have cloudfront inspect the token, determine the identity ID of the user, and only serve contents for that user if he is logged in. Does this require lambda@edge or is there an easier way? I don't want to use signed urls, because I anticipate having a single user view hundreds of urls at a single time (in a gallery view), and figure generations signed urls will slow things down too much.In the future, I may want to enable sharing of files... Could I enable that by having an entry in DynamoDB for every file, and have cloudfront check if the user is allowed to view the file, before serving it? Would this be part of the lambd@edge function?ThanksFollowComment"

Answer: "0This sample solution comes somewhat close to what you want, and could be tweaked to fit your needs: https://github.com/aws-samples/cloudfront-authorization-at-edgeCurrently it misses an upload feature you need, and also it misses the feature of having users only see/edit their own files.It would not require signed URLs because it relies on good old cookies for authentication.However I do like to point out that generating signed URLs is in fact very quick, and usable for your purpose too. I doubt it would be too slow for hundreds of URLs even. Signing an (https) S3 URL using AWS credentials is quick--this does not need to invoke any AWS service over the wire, it's a cryptographic operation that runs locally on the client (in your browser's JavaScript in this case).I hope that helps somewhat, I realize it's not the whole story. There's a lot to your question, it's quite a full blown application you're trying to build :)CommentShareOtto Kruseanswered 5 months agorrrpdx 5 months agoThanks. I did end up putting together a simple lambda@edge function that is similar to what you pointed to. I am uploading content by using presigned urls, but assume that using cookies with a lambda@edge function will be more performant for downloading content.Share0One consideration might be to look at Origin Access Control (OAC) and custom bucket polices: OACCommentShareAWSdaveanswered 5 months ago"


Question: Password issue (facing issue in login iam account)

"i have forgotten my iam user account password, and i don't know how to change it . Just needed this information about passwordFollowComment"

Answer: "0This might be helpfulhttps://repost.aws/knowledge-center/recover-aws-passwordhttps://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys_retrieve.htmlCommentSharerePost-User-5208427answered 2 months ago0Take a look at this page: Resetting lost or forgotten passwords or access keys for AWS.The easiest method is to login as the root user and reset the IAM user's password. If this is not an option, contact support.CommentShareEXPERTkentradanswered 2 months ago"


Question: CloudFormation deleting nlb did not release public IP

"I deleted my C/f stack which included an NLB but that hasn't released the public IPs associated with each subnet. To be clear, the NLB instances have been deleted but their public IPs are still marked as in use. I'm now unable to detach , or delete them - as they are marked as being in use by another service - the NLB that has successfully been deleted.None of the troubleshooting articles (which are primarily related to EIPs) offer any material resolution. Even logging into the console as root, I get the error that I do not have permissions.Any thoughts on what I'm missing?FollowComment"

Answer: "0Hi ThereThis is happening because the EIP is associated with the managed network interface of the load balancer.  It takes some time for the deleted NLB to release the network interface.  You cant delete the EIP until the network interface is gone, and unfortunately CloudFormation doesnt have any knowledge of this.You can do any of the following:Dont use an Elastic IP and just let AWS assign a public IP to the NLB.  Example:  loadBalancer:    Type: 'AWS::ElasticLoadBalancingV2::LoadBalancer'    Properties:      Type: network      IpAddressType: ipv4      Scheme: internet-facing      Subnets: !Ref SubnetsPerform a delete-stack operation again, after verifying that any network interfaces(belonging to network load balancer) are deleted by elbv2 service itself and association id is removed from EIP.Use DeletionPolicy/UpdateReplacePolicy attributes with AWS::EC2::EIP resource and set value to Retain to achieve single step stack deletion. Note that you will still need to delete the EIP manually.CommentShareEXPERTMatt-Banswered 9 months agoEXPERTToni_Sreviewed 8 months ago"


Question: How to debug cloudfront behaviours not sending requests to correct origin?

"I have two custom origins set up pointing to two different apps on Heroku, A and B.I have three cache behaviours set up, all of which have caching disabled, and pass through all headers. In priority order:Path pattern /v3 points to app A.Path pattern /v3/* points to app A.The default behaviour points to app B.However, no matter what request I make (I've tried different variations on /v3, /v3/ and /v3/some-page, they are all routed to app B. App A is never reached.I haven't been able to find any debugging info to help me figure out what's going on. Is there any? I turned on logs for the distribution, but these appear to be mainly access logs, and don't give any indication as to why a particular origin was chosen.FollowComment"

Answer: "0Figured it out:I had the behaviours configured to pass through all headers (including the Host header). As Heroku uses global routing, app-a.herokuapp.com and app-b.herokuapp.com both resolve to the same thing, and Heroku uses the host header to route to the correct heroku app. So for e.g. curl app-a.herokuapp.com -H "Host: app-b.herokuapp.com" will make a request to app b, not app a.For historical reasons, the overall domain name (let's call it example.com) had been "added" to heroku app b, so curl app-a.herokuapp.com -H "Host: example.com" would resolve to app b. Very confusing!To fix this I created a custom cache policy that sent through all headers except the Host header, and changed the origins to add a hardcoded X-Forwarded-Host header set to the domain name, in this case example.com. Now the routing works, and the X-Forwarded-Host can be used by the two apps to know what host they've been accessed from. (One of them is Rails, which picks up the X-Forwarded-Host header automatically).CommentSharerosollanswered 2 years ago"


Question: How to view email headers in WorkMail

I am trying to resolve email delivery issue and wanted to look at email headers. According to AWS documentation it can be viewed from mail client by clicking on gear mailbox icon on the top of message but I can't find that icon anywhere. I tried using AWS email app using Chrome and Firefox.Link to documentation:https://docs.aws.amazon.com/workmail/latest/adminguide/email-headers.htmlFollowComment

Answer: "1Accepted AnswerHi,The documentation might not be 100% clear but you need to open the email and not view it in the preview window. Once opened the gear icon will appear in the top right corner of the email.Kind regards,RobinCommentShareMODERATORrobinkawsanswered a year agoAWS-User-6626260 a year agoHi,I just tried to open email in no preview mode but I still can't see the icon with sent time. I see only two gear icon one on the top right corner is for global email settings and another one is next to search box for search settings. Documentation says there should be icon next to sent time on email. I still can't see it anywhere.Are you able to view email headers from the gear icon ? I am still confused why it is not showing for me.Thanks,SimarpreetEdit: It just worked. I had to double click on the email after opening in no preview mode. Now I can see gear and mailbox icon on the right top.Thanks very much!!Share"


Question: Memory leaks when upgrading from Aurora MySQL 2.10.0 to 3.01.0

"We recently upgraded a cluster from MySQL 5.7 (Aurora 2.10.0) to MySQL 8.0.23 (3.01.0) using a snapshot restore, as we could really benefit from the new features of MySQL 8. The moment the new cluster became available, memory usage on our writer instance started decreasing until it started consuming swap, and then once it ran out of swap, AWS initiated an automatic failover. This cycle has been repeating since we've performed the upgrade.We have a separate test cluster we upgraded (using the same snapshot), and it does not appear to be experiencing the same issue. That cluster does not have much load on it, and also wouldn't be used for replication to other databases.We tried increasing our instance sizes (db.t3.large to db.r5.large), turning off our cross region read replicas, turning off our DMS tasks, tweaking some memory settings, and opened a support ticket, but so far none of these avenues have helped us out at all.Has anyone else run into this issue? Are there other troubleshooting steps we could try?FollowComment"

Answer: "1Accepted AnswerThe new version is now GA:https://docs.aws.amazon.com/AmazonRDS/latest/AuroraMySQLReleaseNotes/AuroraMySQL.Updates.3011.htmlCommentShareRoni Vered - AWSanswered a year agoholdcontrast a year agoWe just upgraded to Aurora 3.02.0 and turned our binlogs back on, and it appears that the memory usage is remaining stable. Thanks!Share1Hi,I understand troubleshooting an issue of this nature can be frustrating, but we're here to help. Considering the issue is specific to an AWS Account/Aurora Cluster, we will need to review the cluster metrics and logs to understand where memory is being utilized. I see an investigation is on-going through the support case created on this topic. I have triggered an internal engagement with the Aurora development team to aid with investigating the behavior you're seeing with memory consumption.Please refer to the case in the Support Center for the latest updates: https://console.aws.amazon.com/support/home#/If you have any other questions or concerns, please engage us via your case as this is being actively monitored for the quickest turn around time. Thank you for your understanding and patience in working with us concerning this.CommentShareShivanswered a year ago1Same issues here. We raised a ticket with AWS who (eventually) responded confirming it's a bug with Aurora 3.0.1.0. A potential workaround is to disable binlog replication which isn't an option for us. A fix is pending "some time in Q1 2022", which is incredibly helpful when you're relying on this for production use.CommentShareAWS-User-0681491answered a year ago"


Question: How to delete a service-linked role

"I am trying to delete Roles by AWS instructions.It says, "If the role is being used, then you must wait for the session to end before you can delete the role". Unfortunately, I have no idea about ending sessions. Here I attach the profile of the Role, where you can see many services. What should I do to delete this Role? Any suggestions and help will be appreciated! My account was hacked, I have to delete the Roles that the hacker created.FollowComment"

Answer: "0Hello,Delete a service-linked role for Amazon ConnectYou don't need to manually delete the AWSServiceRoleForAmazonConnect prefixed role. When you delete your Amazon Connect instance in the AWS Management Console, Amazon Connect cleans up the resources and deletes the service-linked role for you.What do I do if I notice unauthorized activity in my AWS account?CommentShareAWS-User-7653869answered 7 months agoxgxofdream 7 months agoThanks, but it is not a prefixed role.Share0Hello,Please see the below blog post if it helps:https://aws.amazon.com/blogs/security/now-use-aws-iam-to-delete-a-service-linked-role-when-you-no-longer-require-an-aws-service-to-perform-actions-on-your-behalf/ThanksCommentShareGautam Kumaranswered 7 months ago"


Question: How to load annotation from local computer in Amazon rekognition custom label service

"Hello Everyone,I am creating object detection model using Amazon rekognition custom label service. I have already uploaded the image data in s3 and attached it to rekognition project. In Label part, I can select only 9 Images at a time After saving the annotation it takes me to 1st page from there I have to go to 30 or 40 page wherever I last saved the annotation. Doing annotation on that tools is very time consuming.So can we use the locally annotated images in amazon rekognition service ? Is there any way to upload the json file of the annotation and we can directly train the model.Any help will be appreciated.Thank youViraj HapaliyaFollowComment"

Answer: "1You can upload the label as a SageMaker Ground Truth format manifest file. It's a JSON lines format and you can find some example here: https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/md-create-manifest-file.html . You might have to transform the format if you local annotation tool generates a different format.Once you have the manifest file you can import it using the Console or SDK, see https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/understanding-custom-labels.html#tm-create-modelCommentShareS Lyuanswered a year agovirajhapaliya a year agoI was able to load my local annotation into sagemaker by making changes in the json. But How do I download the manifest file of the annotation which I have done in amazon rekognition custom labels.I have checked my S3 bucket but I am not to find output.menifest in that.And One more question I have already added the sagemaker image files in amazon rekognition service. Now I want to update manifest file in rekognition service.ShareS Lyu a year agoHi Viraj,You could export the JSON Lines information of your dataset using this API: https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/md-listing-dataset-entries-sdk.htmlI don't see a easy way to update the manifest file in-place. But I believe you can upload a new manifest file to a S3 bucket, keeping all the images untouched, and then delete and recreate the dataset in Rekognition using the new manifest file.Share"


Question: RDS Enhanced Monitoring

"Greetings! I want to enable Enhanced Monitoring for my RDS Instance but don't how much it costs. Can you please provide a link or any info considering this topic?Edited by: JoeyBlack on Feb 6, 2019 6:03 PMFollowComment"

Answer: "0Enhanced Monitoring has no cost itself, but it does write to Cloudwatch. So when/if you exceed the free tier for Cloudwatch Logs (5GB) you'll start to pay for the extra usage. You can limit the amount of data being sent to Cloudwatch by using coarser granularity monitoring.  Cloudwatch pricing is at https://aws.amazon.com/cloudwatch/pricing/CommentShareHalTempanswered 4 years ago0see bottom of this page https://aws.amazon.com/rds/faqs/Copied and pasted here for convenience :Q: What impact does Enhanced Monitoring have on my monthly bills?Since the metrics are ingested into CloudWatch Logs, your charges will be based on CloudWatch Logs data transfer and storage rates once you exceed CloudWatch Logs free tier. Pricing details can be found here. The amount of information transferred for an RDS instance is directly proportional to the defined granularity for the Enhanced Monitoring feature. Administrators can set different granularities for different instances in their accounts to manage costs.The approximate volume of data ingested into CloudWatch Logs by Enhanced Monitoring for an instance is as shown below:Granularity 60 seconds 30 seconds 15 seconds 10 seconds 5 seconds 1 secondData ingested in CloudWatch Logs* (GB per month)                     0.27       0.53       1.07      1.61      3.21      16.07Edited by: awskyleh on Feb 6, 2019 4:27 PMCommentShareAWS-User-1731424answered 4 years ago"


Question: Domain registration

"Hi,I was trying to register a domain name for my association  but unfortunately I have received an email says "We weren't able to register the domain name." we can't finish registering your domain. Contact AWS Support at this linkDid anybody have the same problem? Kindly help to solve this issue!.FollowComment"

Answer: "1Availability: The domain name you're trying to register might already be taken by someone else. You can check its availability using a domain registrar like AWS Route 53 or other services like Namecheap or GoDaddy.Invalid characters: Ensure that the domain name you're trying to register doesn't contain any invalid characters or spaces. Domain names can include letters, numbers, and hyphens, but cannot start or end with a hyphen.TLD restrictions: Some top-level domains (TLDs) have specific requirements or restrictions. For example, certain TLDs may require the registrant to be a resident of a specific country. Make sure you're aware of any requirements for the TLD you're trying to register.Payment issues: Make sure your payment method is valid and has sufficient funds to cover the domain registration cost.Account limitations: If you're using AWS, check if there are any limitations or restrictions on your account that could prevent you from registering a domain.CommentShareEXPERTsedat_salmananswered 2 months ago"


Question: No EC2 instance showing to run but the OPENVPN is working fine.

"Hi,I followed a youtube video and setup a OPENVPN EC2 and tunneled my home network through that and it was working fine.Now after a month later, the VPN server is still running fine and I am seeing payment amount increasing and forcasted for next month but when I log into my aws account and go to EC2 I don't see any instances running. 0 instance. but the VPN is working fine.So, I wonder how to be sure that it is my VPN is what I am using and not a hacker's VPN now. And why the bill is adding up?Any help for this novice will be appreciated.Thanks,RepostFollowComment"

Answer: 0Are you looking in the correct Region?CommentShareEXPERTskinsmananswered 2 months ago


Question: No module error while trying "Connect to AWS IoT"

"I have followed the "Connect to AWS IoT" steps provided in AWS IOT (1.Register a device 2.Download a connection kit 3.Configure and test your device).When I tried to run the script, I am getting No module named AWSIoTPythonSDK.MQTTLibEnvironment:Platform: LinuxChoose a AWS IoT Device SDK: PythonMore info:user::~/Downloads/connect_device_package$ ./start.sh Downloading AWS IoT Root CA certificate from AWS...  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100  1188  100  1188    0     0   7378      0 --:--:-- --:--:-- --:--:--  7378Cloning the AWS SDK...Cloning into 'aws-iot-device-sdk-python'...remote: Enumerating objects: 870, done.remote: Counting objects: 100% (135/135), done.remote: Compressing objects: 100% (77/77), done.remote: Total 870 (delta 71), reused 58 (delta 58), pack-reused 735Receiving objects: 100% (870/870), 334.31 KiB | 1.92 MiB/s, done.Resolving deltas: 100% (423/423), done.Installing AWS SDK...~/Downloads/connect_device_package/aws-iot-device-sdk-python ~/Downloads/connect_device_packageRequirement already satisfied: AWSIoTPythonSDK in /home/om/.local/lib/python3.8/site-packages (1.4.9)~/Downloads/connect_device_packageRunning pub/sub sample application...Traceback (most recent call last):  File "aws-iot-device-sdk-python/samples/basicPubSub/basicPubSub.py", line 18, in <module>    from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClientImportError: No module named AWSIoTPythonSDK.MQTTLibFollowComment"

Answer: "1Accepted AnswerHi Om. I'm guessing you have both python2 and python3 installed. Can you run the following command to check the version?python --versionIf it's 2.x, that would explain what you're seeing. If so, easiest way out is probably to edit line 32 of start.sh to call python3 instead of python.CommentShareEXPERTGreg_Banswered a year agoOm a year agoThanks Greg. It is working now :). Yes, I have both 2.x and 3 versions.Share"


Question: Accessing S3 Gateway VPC Endpoint from another VPC (VPC Peering established between both source and destination VPCs)

"Hi,IHAC who is looking for a solution to access their S3 VPC Endpoints in one VPC from another VPC in another region/account.They have set VPC Peering in between.Source VPC:region: us-east-1resources in private subnetvpc peering established with Destination VPCDestination VPC:region: ca-central-1resources in private subnetvpc peering established with Source VPCS3 VPC endpoint createdNow, from Destination VPC, the S3 VPC endpoint is accessible on regional S3 url: s3.ca-central-1.amazonaws.comHowever, the S3 VPC endpoint is not accessible from the Source VPC.Is there something different that we need to do to achieve this?FollowComment"

Answer: "0Accepted AnswerS3 Gateway VPC Endpoint is a Gateway endpoint type. You can't access from another VPC with VPC peering or TGW without placing Proxy in Destination VPC.S3 Interface endpoint is in private preview. With S3 Interface Proxy will not be required.Also, endpoints currently do not support cross-Region requests—ensure that you create your endpoint in the same Region as your bucket.CommentShareMODERATORAWS-User-6274870answered 3 years ago"


Question: Generate OIDC token from EC2 assigned IAM Role temporary credentials

"Hi,I wish to use "Workload Identity Federation" to use an AWS generated OIDC token with Azure AD. The use-case for this is so that workloads running in AWS can use an assigned IAM Role to access services that are using Azure AD as the Idp. This would mean that the AWS workload would not have to store any AAD credentials.(https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation)I had assumed that there would be a way to use an IAM role Assigned to my EC2 instance to get temporary credentials (this bit is ok), and then use those to get an Open ID Connect (OIDC) token which could then be sent to AAD to be exchanged for an AAD issued token). But I haven't been able to find a way to get an OIDC token from AWS.Would this require AWS SSO?I can only find a way to get a token from AWS SSO by first registering a client using this API:https://docs.aws.amazon.com/singlesignon/latest/OIDCAPIReference/API_RegisterClient.html... but registering a client responds with the credentials to use for getting a token, and having to store these credentials defeats the object of using the assigned IAM Role.Does anyone know how I can get a JWT ODIC token from AWS that represents my IAM Role as the principal?ThanksFollowComment"

Answer: "2See my blog post: https://blog.identitydigest.com/azuread-federate-aws/ which provides a walkthrough for how to achieve this scenarioCommentShareuday hegdeanswered 6 months ago0Hi,To get an OpenID token for your EC2 instance you need an OpenID provider to issue these tokens. AWS itself is not an OpenID provider, the only service that can issue tokens of this type is Cognito (for example using client-credentials flow with a client-id and secret) and this will require your EC2 instance to use credentials other than the IAM role. So it will be similar to keeping AAD credentials.You should be able to store AAD credentials in secrets manager and use the IAM permissions associated with your machine to access these secrets as needed at runtime to authenticate with workload identity federation.CommentShareEXPERTMahmoud Matoukanswered a year agopcresswell a year agoThanks for the response Mahmoud. I did look at Cognito too, but as you said there wasn't a way to get a token using IAM Role temporary credentials.Your workaround is our current state (using Secrets manager), I was hoping I would be able to avoid key refreshing and distribution of the AAD credentials by using IAM role.It seems like you can do this in reverse ok (i.e. use an OIDC token from AAD Managed Identity or GCP to access AWS resources), just not when the 'workload' is in AWS.Thanks again for the reply.Share0@pcresswell were you ever able to find a solution to this? I'm looking to do the same thing.CommentShareSamanswered a month ago0@Sam At the time I was looking at this it wasn't possible (so I was told by MS). But I've not tried Uday Hegde's method described in his excellent blog post above - which looks like it achieves what I was wanting to do. So looks like it now is possible using Managed Identities.The solution I used back then was to use the Azure Arc agent on the EC2 workload, which is a very simple setup and results in an Azure Managed Identity on the EC2 workload. (The Arc Agent includes a local service that is queried to get Azure AD tokens).https://learn.microsoft.com/en-us/azure/azure-arc/servers/managed-identity-authenticationCommentSharepcresswellanswered a month ago0Thanks @pcresswell. I actually implemented Uday Hegde's blog post pattern with a lambda... works great to get an Azure AD token. Just wish that I was able to use IAM instead of cognito to get that OIDC token directly. Hoping that's a feature that AWS will incorporate into IAM/STS.CommentShareSamanswered 18 days ago"


Question: Asked Pinpoint voice production access from AWS but still in sandbox mode

"I asked Pinpoint voice production access from support centre, but my account tier of Pinpoint "SMS and voice" overview. still show "Sandbox".Unfortunately, AWS support centre can't provide technical support.So, is there any step I missed?Please help!!!FollowComment"

Answer: "0Hi,I'm sorry to hear that you are still seeing your account in Sandbox mode. I think the initial support case may have been created under "Account and billing".https://docs.aws.amazon.com/awssupport/latest/user/case-management.htmlCan you kindly create a new support case under "Technical" with "Service" as Pinpoint. You can choose Category as General Guidance/Limit Increase and a Premium Support engineer will be able to help you. We would need to review your account in order to determine what is happening here. Unfortunately, this is not possible to review via re:post.Please do let me know if you are still not able to get help even by following these steps. Please let me know if you have any other questions as wellCommentShareSUPPORT ENGINEERRyan_Aanswered 5 months ago"


Question: [XRT] ERROR: xclRegRW: can't map CU: -  when trying to access to read/write register inside the fpga

"Hi,Using xrt application and trying to access to read/write register and got this error:[XRT] ERROR: xclRegRW: can't map CU:My xrt.ini contains this content:[Debug]profile=true[Runtime]rw_shared=trueruntime_log=consoleand inside the my code I used those commands:/ Read clock count from clk_cnt* registers and unite into 64-bit value// There is issue with [XRT] error : xclRegRW: can't map CU: clock count is 0xclOpenContext(handle, xclbinId, cuidx, false);xclRegRead(handle, cuidx, clk_cnt_lsb_offset, &clk_cnt_lsb);xclCloseContext(handle, xclbinId, cuidx);xclOpenContext(handle, xclbinId, cuidx, false);xclRegRead(handle, cuidx, clk_cnt_msb_offset, &clk_cnt_msb);xclCloseContext(handle, xclbinId, cuidx);long int clock_count = ((long int)clk_cnt_msb << 32) | (long int)clk_cnt_lsb;std::cout << std::endl << "Clock count is: " << clock_count << std::endl;FollowCommentTaylorAWS a year agoHi, you might want to add the FPGA Development tag to this question.Share"