Question: Hive Sync fails: AWSGlueDataCatalogHiveClientFactory not found

"I'm syncing data written to S3 using Apache Hudi with Hive & Glue.Hudi options:hudi_options:    'hoodie.table.name': mytable    'hoodie.datasource.write.recordkey.field': Id    'hoodie.datasource.write.partitionpath.field': date    'hoodie.datasource.write.table.name': mytable    'hoodie.datasource.write.operation': upsert    'hoodie.datasource.write.precombine.field': LastModifiedDate    'hoodie.datasource.hive_sync.enable': true    'hoodie.datasource.hive_sync.partition_fields': date    'hoodie.datasource.hive_sync.database': hudi_lake_dev    'hoodie.datasource.hive_sync.table': mytableEMR Configurations:...{                "Classification": "yarn-site",                "Properties": {                    "yarn.nodemanager.vmem-check-enabled": "false",                    "yarn.log-aggregation-enable": "true",                    "yarn.log-aggregation.retain-seconds": "-1",                    "yarn.nodemanager.remote-app-log-dir": config[                        "yarn_agg_log_uri_s3_path"                    ].format(current_date),                },                "Configurations": [],            },            {                "Classification": "spark-hive-site",                "Properties": {                    "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"  # noqa                },            },            {                "Classification": "presto-connector-hive",                "Properties": {"hive.metastore.glue.datacatalog.enabled": "true"},            },            {                "Classification": "hive-site",                "Properties": {                    "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",  # noqa                    "hive.metastore.schema.verification": "false",                },            },...Getting the following error:Traceback (most recent call last):  File "/home/hadoop/my_pipeline.py", line 31, in <module>    ...  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 1109, in save  File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco  File "/usr/lib/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_valuepy4j.protocol.Py4JJavaError: An error occurred while calling o84.save.: org.apache.hudi.hive.HoodieHiveSyncException: Got runtime exception when hive syncingat org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:74)at org.apache.hudi.HoodieSparkSqlWriter$.syncHive(HoodieSparkSqlWriter.scala:391)at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$4(HoodieSparkSqlWriter.scala:440)at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$4$adapted(HoodieSparkSqlWriter.scala:436)at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:436)at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:497)at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:223)at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145)at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:194)at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:232)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:229)at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:190)at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:134)at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:133)at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)at py4j.Gateway.invoke(Gateway.java:282)at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)at py4j.commands.CallCommand.execute(CallCommand.java:79)at py4j.GatewayConnection.run(GatewayConnection.java:238)at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to create HiveMetaStoreClientat org.apache.hudi.hive.HoodieHiveClient.<init>(HoodieHiveClient.java:93)at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:69)... 46 moreCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to instantiate a metastore client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory due to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory not found)at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:239)at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:402)at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:335)at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:315)at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:291)at org.apache.hudi.hive.HoodieHiveClient.<init>(HoodieHiveClient.java:91)... 47 moreCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to instantiate a metastore client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory due to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory not found)at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3991)at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:251)at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:234)... 52 moreCaused by: MetaException(message:Unable to instantiate a metastore client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory due to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory not found)at org.apache.hadoop.hive.ql.metadata.HiveUtils.createMetaStoreClientFactory(HiveUtils.java:525)at org.apache.hadoop.hive.ql.metadata.HiveUtils.createMetaStoreClient(HiveUtils.java:506)at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3746)at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3726)at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3988)... 54 more22/01/12 16:37:53 INFO SparkContext: Invoking stop() from shutdown hookFollowComment"

Answer: 0This is due to Glue catalog libraries missing in the Hudi class path.Kindly consider to modify the script (run_sync_tool.sh) as below to make it work.       # added for AWS Glue Catalog hive metastore libraries.(For EMR cluster with version 6.X)       HIVE_METASTORE=/usr/share/aws/hmclient/lib/aws-glue-datacatalog-hive3-client.jar:$HIVE_METASTORE(or)# added for AWS Glue Catalog hive metastore libraries.(For EMR clusters with version 5.X)HIVE_METASTORE=/usr/share/aws/aws-java-sdk/aws-java-sdk-bundle-1.11.970.jar:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-hive2-client-1.15.0.jar:$HIVE_METASTORESteps:SSH Login to the master node of EMRcd /usr/lib/hudi/binsudo cp run_sync_tool.sh run_sync_tool.sh_backupsudo vi run_sync_tool.sh(Add below lines in this section : ## Include only specific packages from HIVE_HOME/lib to avoid version mismatches : add it after the line GLUE_JARS  or line 56)       # added for AWS Glue Catalog hive metastore libraries.       HIVE_METASTORE=/usr/share/aws/hmclient/lib/aws-glue-datacatalog-hive3-client.jar:$HIVE_METASTORECommentShareSUPPORT ENGINEERAWS-User-0909871answered a year agospree a year agoIs there an option to apply these changes? I've added the following after GLUE_JARS line:HIVE_METASTORE=/usr/share/aws/aws-java-sdk/aws-java-sdk-bundle-1.12.31.jar:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-hive3-client-3.4.0.jar:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-hive3-client.jar:$HIVE_METASTOREBut I'm running into the same error.Share


Question: Device Farm - Code Pipeline Integration fails

"Hi everyone,I'm actually unable to integrate a Device Farm test phase in my pipeline.I try to use Appium for NodeJS to test a Web Application using a Test Stage in my Code Pipeline but I've got the same error every time and I don't figure out how to correct it.The error is not very explicit so it's difficult to detect the issue :  "Invalid option for TestType configured, should be a valid TestType."Here is the part of my Cloudformation stack describing the Test Stage :- Name: Tests          Actions:            - Name: TestOnDeviceFarm              ActionTypeId:                Category: Test                Owner: AWS                Version: 1                Provider: DeviceFarm              Configuration:                Test: package_device_farm.zip                RecordAppPerformanceData: "true"                RecordVideo: "true"                AppType: Web                TestType: APPIUM_WEB_NODE                ProjectId: arn:aws:devicefarm:us-west-2:xxxxxx                DevicePoolArn: arn:aws:devicefarm:us-west-2:yyyy                RadioBluetoothEnabled: false                RadioWifiEnabled: true                RadioNfcEnabled: false                RadioGpsEnabled: true                AppiumVersion: "1.9.1"              RunOrder: 1              InputArtifacts:                - Name: BuildArtifactThe Zip file comes from the Input Artifacts of the stage.The test type "APPIUM_WEB_NODE" seems to be correct, so I don't understand the error message.Can someone help me ?Edited by: wmegel on Feb 11, 2020 12:24 AMFollowComment"

Answer: "0Hello,The reason you are getting this failure is because APPIUM_WEB_NODE is not supported on code pipeline.Currently we do not have any plans of extending support on code pipeline. However if you are looking for ways to pipeline, you can use AWS CLI or DeviceFarm Jenkins plugin.RegardsCommentShareAWS-User-9707822answered 3 years ago0I've found this answer. The only TestType supported with a Code Pipeline integration are : CALABASH INSTRUMENTATION UIAUTOMATION UIAUTOMATOR XCTEST XCTEST_UI"APPIUM_WEB_NODE" type is not yet supported with code pipeline.I've got this information from the AWS Support Team.Unfortunately the documentation was not very clear about that !CommentSharewmegelanswered 3 years ago"


Question: CPU usage over 100% on amazon-ssm-agent

"Hi,Hi, we have an ec2 instance with amazon-ssm-agent installed on it.. it's topping out with over 100% CPU just a few minutes after restart, causing all other ECS services on the machine to get killed due to lack of CPU.We've tried two versions - latest = 2.3.701.1and a downgraded version = 2.3.662.0Both versions exhibit this behavior...FollowComment"

Answer: "0Hi JonasL,Thank you for posting here. Could you please provide the following information to help us investigate this issue?Instance typeAMI idRegionLogs about amazon-ssm-agentCommentShareAWS-User-0703893answered 4 years ago0Sure:AMI id - amzn-ami-2017.09.f-amazon-ecs-optimized (ami-acb020d5)Instance type - m5.xlargeRegion - eu-west-1Logs attachedCommentShareJonasLanswered 4 years ago0Turns out there was a bug in the executable that the ssm-agent tried to restart. This caused the ssm-agent to re-try to execute the script even though it was failing - causing the 100% cpu.CommentShareJonasLanswered 4 years ago0Hi JonasL,Base on the log, there are over 180 command executions within 2 mintues. All the commands have the same content, all excutions completed with failed status (no retry).The high CPU usage was related to the nubmer of the commands have been executed.Do you have configuration (StateManager, CloudWatch) that sends commands to the instance after restart?Regards,ShihuaCommentShareAWS-User-7548355answered 4 years ago"


Question: "Object lambda access point policy, policy document malformed"

"Hi,Despite olap is declared as "optional", when saving it I've got an error message "invalid policy". Then I edit it and from there I'm struggling to define the following:{"Version": "2012-10-17","Statement": [{"Sid": "Statement1","Principal": {"AWS": "*account-ARN*"},"Effect": "Allow","Action": ["s3-object-lambda:*"],"Resource": ["*bucketAccesspoint-ARN*"]}]}In the underneath status bar: Security = 0, errors = 0, warnings = 0, suggestions = 0 - So should be fine.However, when saving the policy I have an error "Invalid Object Lambda Access Point policy -> API response -> Policy document is malformed"What am I missing?Thx in advance for any suggestionFollowComment"

Answer: "0Hi,I understand that you are encountering a MalformedPolicyDocument error.  You get the "MalformedPolicyDocument" error when the policy document isn't syntactically or semantically correct, as per the grammar of the policy language[1]. Refer to the resources posted below.To solve this error you can modify your policies to the below format:{"Version": "2012-10-17","Statement": [{"Sid": "Statement1","Principal": {"AWS": "*account-ARN*"},"Effect": "Allow","Action": ["s3:*"],"Resource": [                             arn:aws:s3:::xxxxxxxxxxxxxxx/*]}]}Resources:-[1]https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-policies.htmlCommentShareAsisiphoanswered a year ago0In my understanding, olap policy must point, resource-wise, to an access-point, action-wise to object-lambda action - not a directly to a bucket nor s3 actions.Any, attempting to change action to what you suggest raises the error "Unsupported Action In Policy: The action s3: is not supported for the resource-based policy attached to resource type S3 Bucket.*" while the resource raises "Unsupported Resource ARN In Policy: The resource ARN is not supported for the resource-based policy attached to resource type S3 Object Lambda Access Point.".So, it doesn't to be solving the issue...CommentShareAWS-User-3194526answered a year ago"


Question: Unable to access ec2 instance | SSH Connection Timed Out

Every time I try to ssh into my virtual machine -- it saysDW-MBP:Desktop dWw ssh -i "rf_secure_key.pem" ubuntu@ec2-accesspoint.comssh: connect to host ec2-access_point.com port 22: Operation timed outHow do I fix this issue? I believe the firewall was mistakenly turned on during a routine update of nginx configs. I was just on the brink of completing deployment.FollowComment

Answer: 0changed naclsCommentSharedwxcod3answered 2 years ago


Question: Unable to publish CloudWatch latency and result metrics with timestamp

"reproductionCreated Canaryspecific free api (https://weather.tsukumijima.net/api/forecast/city/400040)I didn't edit lambda script not at all(default).Not use VPCwhat happenedSummary status is failedbut step is successI found logs published lambda is belowERRORExecution error: Unable to publish CloudWatch latency and result metrics with timestamp: Tue Oct 04 2022 03:52:10 GMT+0000 (Coordinated Universal Time) for canary name: test result: PASSED with start time: Tue Oct 04 2022 03:52:17 GMT+0000 (Coordinated Universal Time) and end time: Tue Oct 04 2022 03:52:17 GMT+0000 (Coordinated Universal Time) and stepName: 検証 weather.tsukumijima.net, Exception:  InvalidParameterValue: The value for parameter MetricData.member.1.Dimensions.member.2.Value contains non-ASCII characters.What is any hint or solution?thank uFollowComment"

Answer: "0I ran into the same problem.It seems that if the string "検証" is included, the error occurs due to a character code problem. Why don't you try using only English instead of Japanese?CommentSharetakagianswered 5 months ago"


Question: "The security token included in the request is invalid" for cdk deploy on cloud9

"Hi, I just did as I always used to do:Create a Cloud9 using Amazon Linux 2upgrade AWS CLI to v2initialize a CDK app and add some sample codeI am logged on using a role that has 'AdministratorAccess' policy attached and it is assumed by Cloud9.CDK is bootstrapped.Running 'cdk deploy' now gives the following error:current credentials could not be used to assume 'arn:aws:iam::ACCOUNTNUMBER:role/cdk-hnb659fds-deploy-role-ACCOUNTNUMBER-eu-central-1', but are for the right account. Proceeding anyway. ❌ Building assets failed: Error: Building Assets Failed: Error: TestStack: The security token included in the request is invalid    at buildAllStackAssets (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/cdk/node_modules/aws-cdk/lib/index.js:362:115279)    at processTicksAndRejections (node:internal/process/task_queues:96:5)    at async CdkToolkit.deploy (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/cdk/node_modules/aws-cdk/lib/index.js:362:142800)    at async exec4 (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/cdk/node_modules/aws-cdk/lib/index.js:417:51795)Building Assets Failed: Error: TestStack: The security token included in the request is invalidI deleted the CDKToolkit stack and did run 'cdk bootstrap' again, it created the stack again. So apparently this works.Running ' aws sts get-caller-identity' also shows the right role.I also tried on an old instance that was stopped, getting same result.Ideas?FollowComment"

Answer: 0I also encounter same error here in idt not sure why does this happenedCommentShareloordbanswered 2 months ago


Question: Cloudwatch Connector V2 Upgrade Deployment Fails

"I just tried to upgrade the GG Cloudwatch connector to the new version 2 and can no longer get that connector to deploy.  Here's the message:Failed: Deployment ---abcdefg--- of type NewDeployment for group --foobar--- failed error: Greengrass deployment error: unable to execute download step in deployment. error while processing: unable to load the group file downloaded: lambda definition is invalid: [arn::lambda:us-west-2:aws:function:CloudWatchMetricConnector:2] is invalid [FunctionArn]Deployment works just fine with the connector removed.  Is there a way to fix this, or a procedure for re-installing version 1 of the connector?Thanks.FollowComment"

Answer: "0Accepted AnswerHi JennaSys,Thank you for your patience, this issue has now been resolved, and the connector V2 is working.Thank youCommentShareredab_awsanswered 4 years ago0Hi JennaSys,We identified a problem with that Connector version 2 that we are fixing right now. In the meantime, you can still use the version 1 of the connector using the AWS CLI [1], or redeploy your older version of the Greengrass group[1] https://github.com/awsdocs/aws-greengrass-developer-guide/blob/master/doc_source/connectors-cli.md#connectors-cli-create-connector-definitionCommentShareredab_awsanswered 4 years ago0Thank you for letting me know about the situation.  I look forward to the fix.I didn't know redeploying an older version was possible.  I will look into doing that in the mean time.CommentShareJennaSysanswered 4 years ago0Thank you, I was able to deploy V2 now without any issues.Is there a changelog available for the differences between V1 and V2?Thanks.CommentShareJennaSysanswered 4 years ago0Hi JennaSys,Changelog can be found here:https://docs.aws.amazon.com/greengrass/latest/developerguide/cloudwatch-metrics-connector.html#cloudwatch-metrics-connector-changelogThanks,KR-AWSCommentShareKR-AWSanswered 4 years ago"


Question: Amazon Timestream: Is there a cost associated with empty measure values?

"If I have multi-measure records where most of the measures are empty, does that still affect my storage cost as if they were populated?Example: Will a multi-measure record of 250 empty values and 2 populated values take up storage cost for 252 values or 2 values?Also, if I have single-measure records with different data types, does the unused measure value have a cost to it?Example: "measure_name='my_bigint', measure_value::double=NULL, measure_value::bigint=5000", does the NULL take up 8 bytes of storage?FollowComment"

Answer: "2Accepted AnswerSince we do not support NULL values on ingest, if an ingested record does not include a measure either in a single measure or multi measure record, then it will not be charged for storage. The storage cost is associated with the measures, dimensions and values in the ingested record.CommentShareRajesh Iyer - AWSanswered a year ago1This page (https://aws.amazon.com/timestream/pricing/) explains how pricing works for Timestream and provides some examples of billing. Based on that doc, billing is based in part on amount stored in GB. I don't know for sure whether or not NULL takes up 8 bytes of storage. This page (https://docs.aws.amazon.com/timestream/latest/developerguide/metering-and-pricing.storage.html) shows how to calculate the size of each record. The doc says 8 bytes for values. It also shows that each dimension consumes a number of bytes to hold the dimension name. According to the Supported data types documentation, null data is contained in the 'unknown' data type. So NULL is a valid datatype. Based on all that, I going to assume storing 252 values, regardless of values, is going to require more memory in storage than 2 values due to the increased size of every record containing 252 dimensions.CommentShareChris_Panswered a year ago"


Question: [Announcement] Amazon Kendra launches support for query language

"Amazon Kendra is an intelligent search service powered by machine learning. Today, we are excited to announce Kendra’s support for query language, allowing customers to use more advanced search terminology and techniques when needed, to retrieve more precise answers.With Amazon Kendra’s query language, customers can now query Amazon Kendra using advanced query syntax and operators. For example, you can use familiar operators such as OR, AND, NOT, (e.g. “history” AND “movies” NOT “fiction” to find documents about historical movies, but exclude documents about historical fiction movies), specify field-level matches (e.g. “author: John Doe”), and perform range searches (e.g. “pub_date: < 2021-01-02”). Amazon Kendra’s query language expands Kendra’s search capabilities to power a broader range of search scenarios and applications.Amazon Kendra’s query language is available in the following AWS regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), Asia Pacific (Sydney), Asia Pacific (Singapore), Canada (Central) and AWS GovCloud (US-West).For more information, please visit the query language documentation (https://docs.aws.amazon.com/kendra/latest/dg/searching-example.html#advanced-query-syntax) for a full list of supported query syntax and operators.FollowCommentNguyen_M SUPPORT ENGINEERa year agoThis is an announcement migrated from AWS Forums that does not require an answer.Share"

Answer: 0[Announcement] Does not require an answer.CommentShareEXPERTIsraa-Nanswered 20 days ago


Question: Credits are not being used

"I can see Credits worth $10000 in my AWS account but the amount used is 0$ and my account was suspended due to outstanding invoices.I can see that the services I am using are included in the list of applicable products under the credit, yet the bill is generated without credit usage.Can someone tell me how to use the credits?FollowComment"

Answer: "0When you add credits to your account, they can only apply to usage starting from current month, and going forward into future months. If you already have outstanding bills and charges, billing periods for those charges are already over (and the invoices are already created), then credits cannot apply to those past charges.You'd need to pay for your outstanding charges separately, and then you would be able to use credits for any future usage.UPD - if you have issues or difficulties paying for your past charges, I suggest opening a support case from your account (even if it's suspended, you can still open a support case) with the billing category. Ask support team which options you might have, maybe to pay those bills in parts (sort of a "payment plan"), or maybe to get a payment term extention.CommentShareEXPERTNataliya_Ganswered 2 months ago"


Question: glue job not running ??

''failed to execute with exception Task allocated capacity exceeded the limit.  (Service: AWSGlueJobExecutor; Status Code: 400; Error Code: InvalidInputException; Request ID: 8d31d112-91e4-4892-900f-048d993e0da1; Proxy: null) this error is coming when ever I run a job.FollowComment

Answer: "0It would be helpful to understand more details to understand why you are facing this issue.In my experience, I have faced this exact same issue when I use more than 300 under "Requested number of workers". The maximum number of workers you can define is299 for G.1X (For AWS Glue version 2.0 or later jobs)149 for G.2X (For AWS Glue version 2.0 or later jobs)For older job types that use "Standard", choose an integer from 2 to 100. The default is 10.Another reason could be the service quotas and you could check your quotas. You can use Service Quotas console to view and request increases for most AWS quotas.CommentShareAnanth Tirumanuranswered 9 months ago"


Question: time out cloud front

I am keep getting timout from cloud 504x-cache: Error from cloudfrontyou can check this https://www.ddexplorer.com/Do I know why it is happening ? I have deployed this using awsamplifyFollowComment

Answer: "0Pls checkhttps://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-504-gateway-timeout.htmlResolution504: "Gateway Time-out" errorVerify that the correct ports are open on your security group.Make sure that the origin server allows inbound traffic from CloudFront, typically on port 443 or 80.If your origin uses Elastic Load Balancing, then review the ELB security groups. Make sure that the security groups allow inbound traffic from CloudFront.Verify that the origin server firewall allows connections from CloudFrontDepending on your OS, confirm that the firewall allows traffic for port 443 and 80.If you're using Redhat Linux View, verify that your firewall rules match the following settings.Firewall Rules:$ sudo firewall-cmd --permanent --zone=public --list-portsPermanently Add Rules:$ sudo firewall-cmd --permanent --zone=public --add-port=80/tcp$ sudo firewall-cmd --permanent --zone=public --add-port=443/tcpIf you're using Ubuntu Linux, verify that your firewall rules match the following settings.Ubuntu Linux View Firewall Rules:$ sudo ufw status verbosePermanently Add Rules:$ sudo ufw allow 80$ sudo ufw allow 443If you use Windows Firewall on a Windows server, then see Add or Edit Firewall Rule in the Microsoft documentation.Make sure that your custom server is accessible over the internetIf CloudFront is unable to access your origin over the internet, then CloudFront returns a 504 error. To check that internet traffic can connect to your origin, confirm that your HTTP and HTTPS rules match the following settings.For HTTPS Traffic:nc -zv OriginDomainName/IP_Address 443telnet OriginDomainName/IP_Address 443For HTTP Traffic:nc -zv OriginDomainName 80telnet OriginDomainName 80504: "The request could not be satisfied" errorMeasure the typical and high-load latency of your web applicationUse the following command to measure the responsiveness of your web application:curl -w "DNS Lookup Time: %{time_namelookup} \nConnect time: %{time_connect} \nTLS Setup: %{time_appconnect} \nRedirect Time: %{time_redirect} \nTime to first byte: %{time_starttransfer} \nTotal time: %{time_total} \n" -o /dev/null https://www.example.com/yourobjectNote: For https://www.example.com/yourobject, enter the URL of the web application that you're testing.The output looks similar to the following:DNS Lookup Time: 0.212319Connect time: 0.371254TLS Setup: 0.544175Redirect Time: 0.000000Time to first byte: 0.703863Total time: 0.703994Depending on the location of the request, troubleshoot the step that shows high latency.Add resources or tune servers and databasesMake sure that your server has enough CPU, memory, and disk space to handle viewer requests.Set up persistent connections on your backend server. These connections help latency when connections must be re-established for subsequent requests.Adjust the CloudFront timeout valueIf the previous troubleshooting steps didn't resolve the HTTP 504 errors, then update the time that is specified in your distribution for origin response timeout.CommentShareAWS-User-3429271answered 24 days agoEXPERTkentradreviewed 23 days ago"


Question: Cognito PinPoint User data

"If I decided to share the user data from the Cognito user pool with pinpoint (email), will I have to gain GDPR consent from the user to do so?FollowComment"


Question: Cloudfront is forwarding cookies when it was suppose not to

"I have set a Cloudfront origin request policy with no cookie.But Cloudfront is sending the cookies to the origin anyways.Here is the data obtained on Webpagetest for request (https://www.webpagetest.org/result/220816_BiDcWR_ACG/1/details/#waterfall_view_step1)::authority: fisiculturismo.com.br:method: GET:path: /applications/core/interface/font/fontawesome-webfont.woff2?v=4.7.0:scheme: httpsaccept: */*accept-encoding: gzip, deflate, braccept-language: en-US,en;q=0.9cookie: AWSALB=V+qLZCNbjEfbsPzZCvXjy8lR1d7lJw+6Qz1bNnwYg3ri9BdDQEtMndfBsf/Hz6jHSj9ffTMEA4MsyUU2es6+KXvX4j590g0Rnn2XevQuROzwR/vyxmaPt32qn142; AWSALBCORS=V+qLZCNbjEfbsPzZCvXjy8lR1d7lJw+6Qz1bNnwYg3ri9BdDQEtMndfBsf/Hz6jHSj9ffTMEA4MsyUU2es6+KXvX4j590g0Rnn2XevQuROzwR/vyxmaPt32qn142; ips4_IPSSessionFront=fi6hu5jv1pl00tp6jshi3uf2kaorigin: https://fisiculturismo.com.brreferer: https://fisiculturismo.com.br/sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="104", "Google Chrome";v="104"sec-ch-ua-mobile: ?1sec-ch-ua-platform: "Android"sec-fetch-dest: fontsec-fetch-mode: corssec-fetch-site: same-originuser-agent: Mozilla/5.0 (Linux; Android 8.1.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Mobile Safari/537.36 PTST/220727.131331customize waterfall • Vie===As long as there is a session cookie, the response miss the Cloudfront cache:accept-ranges: bytescache-control: max-age=2592000, publiccontent-length: 77160content-type: application/font-woff2date: Tue, 16 Aug 2022 14:33:40 GMTetag: "12d68-5e3c8209e1ce0"expires: Thu, 15 Sep 2022 14:33:40 GMTlast-modified: Thu, 14 Jul 2022 18:32:43 GMTserver: Apache/2.4.54 (Ubuntu)set-cookie: AWSALB=ldcCFgF+iJ0E/9dkC7wI4cjnuEVQbpZIdhNTudEvrd2RNGyXq1KOUVxtocvI6fgV6ZgUUbC34vikqmDhDGNxJuswDudtAo0P8RpZDyi/k2/Njzu5uQUSS0REf8QM; Expires=Tue, 23 Aug 2022 14:33:40 GMT; Path=/set-cookie: AWSALBCORS=ldcCFgF+iJ0E/9dkC7wI4cjnuEVQbpZIdhNTudEvrd2RNGyXq1KOUVxtocvI6fgV6ZgUUbC34vikqmDhDGNxJuswDudtAo0P8RpZDyi/k2/Njzu5uQUSS0REf8QM; Expires=Tue, 23 Aug 2022 14:33:40 GMT; Path=/; SameSite=None; Secure:status: 200===Why Cloudfront is forwarding the cookies to the origin (ELB cookies and APP cookie) if it was set not to?FollowComment"

Answer: "0You mention that your Origin Request policy is not configured to forward cookies, but what about your Cache policy? If cookies are included in the cache policy, they will automatically be forwarded to the origin. Please refer to https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/controlling-the-cache-key.html for details on Cache policy.CommentSharePaul_Lanswered 10 months agoIgor 10 months agoCache policy was also no cookie. To leave no doubt, I have tested again with Managed-CachingOptimized for caching and no policy for origin request and response headers. Same issue with cookies being fowarded.Share0I think the problem here was related to another question that you asked - the DNS records for your domain did not point to CloudFront, so requests were being made directly to the ALB. This is evident in the response headers above - if the request had been handled by CloudFront then the Server response header would have a value of 'CloudFront' and you'd also have CloudFront specific headers like x-amz-cf-id, x-amz-cf-pop and x-cache. It looks like your DNS is now correctly configured so I expect you are no longer experiencing this issue.CommentSharePaul_Lanswered 9 months ago"


Question: Streaming data to an HTTP endpoint with Amazon Kinesis Data Firehose

"Hi All, Trying to implement "Streaming data to an HTTP endpoint with Amazon Kinesis Data Firehose" using AWS Big Data blog https://aws.amazon.com/blogs/big-data/stream-data-to-an-http-endpoint-with-amazon-kinesis-data-firehose/.  Blog does not tell about "REST API in Amazon API Gateway" Mapping template - for Integration Request and Integration Response is         configured. I am getting error HTTPEndpoint "missing requestId". How to set Integration Response & Method Response in API Gateway to work. Any help is much appreciated.FollowComment"

Answer: "1There are some good tutorials that can help you understand how to configure API Gateway - https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-rest-tutorials.htmlCommentShareEXPERTIndranil Banerjee AWSanswered 10 months agoIndranil Banerjee AWS EXPERT10 months ago@UdAWS - If my answer has helped you understand API Gateway better, can I request you to please accept my answer. ThanksShare"


Question: S3 bucket permissions to run CloudFormation from different accounts and create Lambda Funtions.

"Not sure what I am missing but I keep getting permission denied errors when I launch CloudFormation using https URLHere are the details.I have a S3 bucket "mys3bucket" in ACCOUNT A. In this bucket, I have a CloudFormation template stored at s3://mys3bucket/project1/mycft.yml . The bucket us in us-east-1. It uses S3 Serverside Encryption using S3 key [not KMS]For this bucket, I have disabled ACLs , bucket and all objects are private but I have added a bucket policy which is as below:{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Principal": {"AWS": "arn:aws:iam::ACCOUNT_B_NUMBER:root"},"Action": ["s3:GetBucketLocation","s3:GetObject","s3:GetObjectTagging","s3:ListBucket"],"Resource": ["arn:aws:s3:::mys3bucket","arn:aws:s3:::mys3bucket/project1/*"]}]}Now, I login to Account B --> CloudFormation --> Create new stack --> Template is Ready --> Amazon S3 URL and the I enter the object path to my template in this formathttps://mys3bucket.s3.amazonaws.com/project1/mycft.ymlWhen I click next, I get the following message on the same page as a banner in redS3 error: Access Denied For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.htmlAlso, just for your information, I am able to list the bucket and objects from Account B if I use Cloud9 and runaws s3 ls s3://mys3bucket/project1/mycft.ymlaws s3 cp s3://mys3bucket/project1/mycft.yml .What am I missing?  [I think this should work even when bucket is set a private but bucket policy allows cross-account access].Does this use case require my bucket to be hosted as static website?FollowComment"

Answer: "0I found the problem. I had SSE encryption at bucket level but all objects had default S3 KMS key which doesn't allow objects to be shared outside that account.CommentShareAlexaanswered a year ago0Hi Alexa,Glad you found your problem.  One useful tip for setting up cross-account access via a resource policy (such as the bucket policy you've used):Given Bucket/Resource in Account R and IAM Entity in Account A.Check the Resource Policy in Account R to ensure  it allows access to the IAM Entity.If the Resource is encrypted, check the KMS Key as well.  KMS Keys have Resource Policies and Grants that can be used to give cross-account access.Check the IAM Entity for the right permissions to access the Resource in Account R.  I like to add the resource explicitly in the resource block here.Note: Not all resources support resource policies for cross-account access and some resources have more complex access mechanisms (such as S3 ACLs).KMS Cross-Account Access: https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying-external-accounts.htmlCommentSharejsoncanswered a year ago"


Question: EC2 instance doesn't get a new public IP after Elastic IP is disassociated

"I recently had a need to attach an Elastic IP to an instance that previously had a Public IP - got tired of changing SSH keys whenever I rebooted as I was assigning the same private IP to the instance.  However, the need for a static IP across reboots has now waned and I wanted to revert to the Public IP setting.  However, after detaching the Elastic IP, the instance did not get another Public IP as stated in this document:https://aws.amazon.com/premiumsupport/knowledge-center/ec2-recover-ip-address/Instead, I had to resort to creating an AMI and launching a new instance with a Public IP.  Is this now the only way to get a Public IP after an Elastic IP is detached?FollowComment"

Answer: "0The reason why you can't get the public IP address when you detach the EIP you asked for the first time is also because the secondary ENI is attached.And we will advise you on your additional questions.I understand that you are attaching and detaching Elastic IP between nodes to achieve floating IP.You can build a Network load Balancer for inbound and outbound to communicate by installing a NAT gateway in each AZ from the private subnet.Or, if you just want to operate the AWS CLI while keeping the current configuration, how about using the VPC endpoint to operate the Elastic IP with private communication?CommentShareEXPERTiwasaanswered a year agoak2766 a year agoThanks @iwasa.  I'm new to load balancers on AWS so I'm going to have to go read about them before I proceed.  Need to get a full understanding before I start migration of production workloads.I'll post back once my understanding is clear; will most likely accept this as the answer.Share0Hi, @ak2766Detaching an Elastic IP address usually assigns it a public IP address.Isn't it assigned even if stop → start?In that case, you may have multiple ENIs attached to your EC2. In that case, the public IP address will not be automatically assigned when the Elastic IP address is detached.https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html#concepts-public-addressesIf the public IP address of your instance in a VPC has been released, it will not receive a new one if there is more than one network interface attached to your instance.You should detach the non-primary ENI once, stop EC2 and then start. You should be assigned a public IP address.Then reattach the detached secondary ENI.CommentShareEXPERTiwasaanswered a year ago0Thanks for all your answers and I see I'm in a bit of a quandary.I'm in the process of setting up a highly available NFS cluster using BRDB across AZ's and hence need secondary interfaces as well as Elastic IP's.  I was intending on making use of the OCF Heartbeat RA's - namely awseip and awsvip.  The problem occurs when the floating IP is moved to another node.  When that happens, the node can no longer reach the AWS API via AWSCLI since there's no longer an IP address that can reach the Internet.  Currently, I'm forced to do it all in the same node AZ but that means if the AZ is down, then so too is that "highly (un)available" NFS cluster.Out of the box thinking will be required here.CommentShareak2766answered a year ago0You can setup instance with 2 Network Interfaces. The primary network interface (eth0) would have a Public IP which would remain the same for the life of the instance. The secondary interface would have the Elastic IP which could be the floating IP you need. When the Elastic IP get reassigned to the other node due to the failover or any other underlying reason as configured, the instance wont lose access to the internet since the Public IP stays the same.Why not try Amazon EFS? It would save you the overhead of managing NFS cluster. (Unless you have already done so and have overwhelming reason not to use EFS )--SydCommentShareSydanswered a year ago"


Question: Support for RDS Oracle as source while using DMS SCN for CDC

"I have a scenario where the customer would like to have a ongoing replica for an RDS Oracle (SE2) database as source to another RDS Oracle (SE2) database as target in a different region. Given the new support for SCN in DMS, can we do an offline backup and export of the source databse using Oracle native tools, and use DMS with SCN for CDC? I know this was not recommended before, but going through the documentation at https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html it looks possible. Any comments? The database has LOBs and is large at over 5-6 TB. ThanksFollowComment"

Answer: "0Accepted AnswerYes this was possible before the native SCN support also, without the SCN support you had the option to start the CDC task using the timestamp only and the issue was that the specific timestamp there could be several SCN's associated with it . Now with Native SCN support you can specify the SCN to the CDC task.CommentShareEXPERTAWS-User-7019129answered 5 years ago"


Question: AWS Security Hub: Sending Findings to S3 for Athena

"I am trying to find a way to create Athena queries that handle information from AWS Security Hub, such as the 'Findings' displayed within it. Athena's input data comes from S3. Is there a way to specify a location in S3 that will receive the findings from AWS Security Hub, or is there already a location I should try looking into? Is there any other way to feed Security Hub information into Athena?FollowComment"

Answer: "0Hi,In principle, you should move security hub logs to an s3 bucket of choice, and then use Athena to query from that bucket.You can check these out:https://github.com/aws-samples/aws-security-hub-findings-exporthttps://aws.amazon.com/blogs/security/export-historical-security-hub-findings-to-an-s3-bucket-to-enable-complex-analytics/Hope it helps ;)CommentShareEXPERTalatechanswered 2 months ago0Hey there!You can use the new service, Amazon Security Lake, which automatically sends security hub findings to an S3 bucket and sets up Athena for you.For more details, see here: https://docs.aws.amazon.com/security-lake/latest/userguide/internal-sources.htmlCommentShareAWS-User-3160679answered 3 days ago"


Question: How to Mount an EFS in WSL2

"I´m trying the same using WSL2 to mount an EFS (I don´t have access to any Linux Distro because I rely on a corporate VPN). I must move some data from a Docker container to EFS on the first run. I read that I could do that with an IAM user but I can't create one for corporate reasons.From my side, I already configured the efs-utils in my Ubuntu running under WSL2 but every time, I´m trying to mount the EFS, I´m getting this error:Error retrieving region. Please set the "region" parameter in the efs-utils configuration file.I already set my region in both files that I found in my PC (I couldn´t find more):•./efs-utils/build/debbuild/etc/amazon/efs/efs-utils.conf•./efs-utils/dist/efs-utils.confDo you have any idea where the efs-utils.conf file you require is located? I cannot find more files beyond those two locations on my PC.Thanks for your time.Edited by: fanmixco on May 14, 2021 4:38 AMFollowComment"

Answer: 1You need to go manually to the root (/) and there you will find /etc/amazon and there is the file.CommentSharefanmixcoanswered 2 years ago0I provided the answer.CommentSharefanmixcoanswered 2 years ago0The efs-utils config file normally located at /etc/amazon/efs/efs-utils.conf. Did you check there?CommentShareCappuccinuoanswered a year ago


Question: AWS Workspace Multiple User access

"Hi,I am currently working on a college project for a company along with my batchmates. Our team has been provided client access to AWS workspace but has been given only one user id. How can we ensure that each member of the team can work on the code simultaneously while ensuring that this doesn't affect the work of the other individual? Is there any way to duplicate the contents of this workspace and add users? Or can I add different users to the same workspace? Right now the issue is that since there is only 1 login credential, only 1 person can access the workspace at a time. Please suggest alternatives. Can we do something about this ourselves or do we need to talk to the company which has admin credentials?FollowComment"

Answer: "1Workspaces is a single-user service so it probably isn't suitable for your project. Have you considered Cloud9 - it allows for multiple users and is designed specifically as a developer environment.CommentShareEXPERTBrettski-AWSanswered 5 months agoAk 5 months agoCan't the client company which gave me the login credentials duplicate the contents and create another workspace? Is this expensive? They seem to prefer the workspace as some of their internal team members also use the same. So any solution involving a workspace would be easier.ShareBrettski-AWS EXPERT5 months agoYou can have multiple Workspaces; each has a different user (and therefore login) assigned to it. But then you will need some sort of tool to allow code sharing and collaboration.Share0Each WorkSpace is assigned to a single user and cannot be shared by multiple users. By default, only one WorkSpace per user per directory is allowed.[+] https://docs.aws.amazon.com/workspaces/latest/adminguide/administer-workspace-users.htmlFor your use-case, I would suggest like creating an EC2 and RDP access with RDP license you have 2 license free with AWS. the license for the Windows Server operating system allows two simultaneous remote connections for administrative purposeshttps://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/connecting_to_windows_instance.htmlCommentShareaymann_Banswered 5 months ago-1You can use the collaboration feature provided by the AWS Workspace. AWS Workspaces allows multiple users to access the same environment with their individual credentials. You will have to check with your company if they provided the client access to AWS workspace if they have enabled this feature or not.You will need to talk your company which has admin credentials to add new users or make any changes to the AWS Workspace.CommentShareFrank Idugboeanswered 5 months ago"


Question: Move instance from one subnet to another

"A customer wants know the process to move an instance from one subnet to another. I referred them to creating a snapshot, then image and creating a new instance. They now have the following question.Does mac address change when you launch a new machine from snapshot?Does Window SSID change when you launch a new machine from snapshot?Based on my testing mac does change but SID does not. But I haven't tested this on a domain joined machine. Any insight would be appreciated on how to get this done.FollowComment"

Answer: "0Accepted AnswerSince this is a new instance on a new host the MAC address will indeed change as you noted as there is new underlying hardware. If you are restoring from a snapshot the SID will not change as the operating system remains the same (and still domain joined) as this is an exact block for block copy (just make sure the machine is shutdown prior to snapshot). For the most part SIDs are not exposed outside of the machine in any meaningful way so even having duplicate SIDs would not be a concern. Just make sure the source machine is shut down as two duplicate machines referencing the same machine object on AD will cause problems (possibly locking the machine account).This is of course if you are moving the instance and not cloning the instance. If you are cloning the instances you should be using Sysprep before you create the AMI as there are a lot of duplicate settings that could be problematic for other applications like SCCM, WSUS, etc. that use other agents and identifies to reference that machine. The SID here is rarely the culprit of these issues. This is the same process you would use to create any base AMI for Windows.This is much better explained by the following article from 2009 which is still very relevant today.The Machine SID Duplication Myth (and Why Sysprep Matters)https://blogs.technet.microsoft.com/markrussinovich/2009/11/03/the-machine-sid-duplication-myth-and-why-sysprep-matters/CommentShareAWS-User-2522362answered 7 years ago"


Question: How to test Cloudfront continuous deployment?

"I have created a Cloudfront distribution for an S3 static site with continuous deployment enabled. I have version 1 code in the S3 bucket which is the origin for the Cloudfront distribution. I have configured staging distribution to receive 10% of the traffic. I deployed version 2 code to the S3 bucket. I expect 10% of the requests to get version 2 and 90% to get version 1, but I get version 2 for all 100% of requests. Is my understanding correct? How can I test the traffic split?FollowComment"

Answer: "0I always try to learn new things, I have liked your courage, so I was crusing the web and I have found a blog of AWS which may assist you.https://aws.amazon.com/blogs/networking-and-content-delivery/use-cloudfront-continuous-deployment-to-safely-validate-cdn-changes/Please let me know the update, so I will also get courage to assist others and encourage me by voting my effort.CommentSharerePost-User-7136343answered 3 months agorePost-User-2220416 3 months agoI have the continuous deployment set up but I am unable to test the expected behavior in traffic split.Share"


Question: Lake Formation Governed tables: truncate and reload?

Doing a POC with LF Governed table for a small dimension. I'd like to purge and reload the table each time it runs. I have tried to include a purge_table command within a governed table transaction but I receive the below error. I've tried adding a transactionId param to the purge_table command directly or within the options dict and neither has worked. Any ideas? Or anyone found a good way to truncate and reload a governed table?"An error occurred while calling o119.purgeTable. Specify Either Transaction Id or Query AsOf Time"FollowComment

Answer: "0Hello,I tried to purge the table and reload but unable to reload the table. Could you please share the purge_table command you are using for us to test ? Also, we recommend you to please open a support ticket with us so that we can get on a screen-share and assist you by checking your Lake Formation configuration and parameters used.CommentShareSUPPORT ENGINEERVishali_Nanswered a year agotjtoll a year agoThanks for your reply. See below. I'll also open a support ticket. Thank youdest_path = <s3 path>db = <database name>tbl = <table name>tx_id = glue_context.start_transaction(False)sink = glue_context.getSink(connection_type="s3", path=dest_path,enableUpdateCatalog=True,transactionId=tx_id)sink.setFormat("glueparquet")sink.setCatalogInfo(catalogDatabase=db, catalogTableName=tbl)try:glue_context.purge_table(db, tbl,options={'transactionId':tx_id})sink.writeFrame(glue_data_frame)glue_context.commit_transaction(tx_id)except Exception:glue_context.cancel_transaction(tx_id)raisejob.commit()Share0In my POC to work with a clean LF table, I do the following:import awswrangler as wr# delete table if existswr.catalog.delete_table_if_exists(database=DATABASE, table=TABLE)# clean up S3 location (delete_table do not remove data itself, just a metadata)wr.s3.delete_objects(TABLE_LOCATION)#Create a test tablewr.catalog.create_parquet_table(        database=DATABASE,        path=TABLE_LOCATION,        table=TABLE,        partitions_types = partition_types,        columns_types=columns_types,        description='Table for testing lakeformation',        table_type = "GOVERNED",        compression= 'snappy'    )CommentShareStas Khirmananswered 13 days ago"


Question: Support for Radius/Accuracy values

My asset tracking project requires including an accuracy or radius (in meters) for each of the location entries on a tracker. Does ALS support the inclusion of a radius and drawing it on the map with the asset?FollowComment

Answer: "0Hi AaronBenz,Thank you for your interest in Amazon Location. Unfortunately, Amazon Location Tracker resources don't currently support metadata with location updates so you can't include an accuracy or radius with your updates. We will keep your use case in mind as we continue improving our service.CommentShareAWS-User-7896559answered 2 years ago"


Question: ElasticSearch version change gets stuck in Amazon OpenSearch Service

"I wanted to change the ElasticSearch version from 6.2 to 6.8 and it has been at the same percentage for many hours. It seems that it got stuck and I don't know how to unlock it.Upgrade from 6.2 to 6.8In progress (42.86%)Tue, 07 Jun 2022 17:33:15 GMTFollowCommentrePost-User-whunter a year agoHaving the EXACT same issue, except ours only made it to 33.33% 🥲Share"

Answer: "0Hi , hope this situation has been resolved by now.For the next time , or otherwise if you still have the issue. This is a case in which AWS Support should be able to help you, because no general answer may apply and something contextual to your account might be causing the issue.Your best next steps would be to open a Support case, in case you have not yet a Support subscription, consider to upgrade to Developer Support or Business support depending on the urgency of your case.Without looking at the internal logs or having more detailed information on the situation it is not possible to provide guidance, sorry.Alternatively, Please make sure you follow the best practices recommended by the Amazon open search to protect your cluster from getting in stuck state in future.Best Practices: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.htmlHope this helpsCommentShareAnkush_Aanswered a year ago"


Question: Getting a '403 Forbidden' error on deploying a Laravel application to AWS Elastic Beanstalk

"I have deployed a Laravel application to AWS Elastic Beanstalk. However, on navigating to the URL of the application I see a 403 Forbidden error.The last two lines of my /var/log/nginx/error.log are as follows.2022/11/21 08:23:20 [error] 3614#3614: *31864 directory index of "/var/www/html/" is forbidden, client: 172.31.39.157, server: , request: "GET / HTTP/1.1", host: "172.31.47.15"2022/11/21 08:23:31 [error] 3614#3614: *31865 directory index of "/var/www/html/" is forbidden, client: 172.31.11.188, server: , request: "GET / HTTP/1.1", host: "172.31.47.15"How do I resolve this issue?FollowComment"


Question: Help needed for a data collection project (mTurk)

"First time mTurker.  I’m having difficulty setting up a project.  I have looked at the forums and tutorials and found some relevant help but still having difficulty figuring this out.  I would be grateful for any help!I have a data collection project where I want to setup thousands of HITs to perform the same task.  Each HIT would visit a website, enter a unique addressline1, and then collect 3 values from the page if the address is found.Since there will be a high volume of these HITs I’d like to use a spreadsheet to set it up.  So I would envision the columns: website, addressline1, value1, value2, value3, addressfoundYN.Here are my questions:I’m not sure how to present instructions to the Worker to visit the website for that row + addressline1 for that row.  I see how to present them with the website based on the “data collection” basic template.  Do I just use html & then add “${addressline1}” to present the address on the HIT?If I want multiple Workers accessing this project (not redundant workers), how can I setup the job? It appears changing “number of assignments per task” would just have more Workers doing the same HIT rather than working on unique HITs.In the event that I want 2 redundant workers on each HIT (as a data verification mechanism) do I simply change the “number of assignments per task” to 2?Is there a way to embed a .jpg in the Worker instructions to show which fields from the website they’ll be collecting data from (a small snip with the values circled as an example)?Thanks for any help!FollowComment"

Answer: "0You can delete most of the HTML in the requester interface for the data collection and write your own.You set the value to 1 and upload your csv, each line in the csv will have one worker doing it. They will be able to do multiples.yesI write instructions in a google doc. and link to itCommentSharejimmyz22answered 2 years ago0For #4, note that you can also include instructional images and examples in your task using plain HTML<img src="https://yourwebsite.com/example.jpg" width="500" height="600"/>  https://www.w3schools.com/tags/tag_img.aspCommentShareAWS-User-6870314answered 2 years ago"


Question: NICE DCV fps settings

"I currently have a setup where I run NICE DCV in a docker container and stream an application to a webpage using the webclient SDK.Unfortunately, the fps on the stream seems to somehow be limited to 30 fps, however, despite setting the targetFps in the dcv.conf file to 60.The grabber-target-fps settings is set to 0.We use a g4dn.xlarge EC2 instance with Nvidia grid 14.3 drivers.The docker container is an ubuntu container.How can I increase the fps?FollowComment"

Answer: "0The target FPS can be set on Linux in the display section of dcv.conf:[display]target-fps=60Maybe you can show the relevant lines of your dcv.conf to verify?CommentShareKarstenNISPanswered 3 months agorePost-User-4598580 3 months agoHere's my display section of the dcv.confenable-client-resize=trueenable-qu=truegrabber-target-fps=0max-compressor-threads=4max-head-resolution=(4096, 2160)min-head-resolution=(426, 240)target-fps=60web-client-max-head-resolution=(0, 0)Share0Looking good. Related to the DCV Web SDK - if the user is on a recent Edge or Chrome version using the latest web client SDK, they will be able to have 60 FPS (no setting needed). Which browser are you using?CommentShareKarstenNISPanswered 3 months ago"


Question: Timestream Query running from Lambda

"Hello:I want to retrieve my Timestream data from API Gateway, so I need to write query code in lambda. I tried the Timestream SDK in Nodejs, and it worked fine running from the CLI. However, trying to replicate same example in AWS Lambda, it runs without errors, but so far, I can't get any data... I was trying to get Query examples to run from Lambda but I couldn't find...Any help would be appreciated.Regards;FollowComment"

Answer: 0It's tough to tell without knowing the details of your implementation. We do have examples in the Timestream documentation that might help.CommentShareEXPERTjohn_lanswered a year ago0If you are using NodeJS in lambda make sure that your handler is defined as an async function and that the code await on the Timestream API call. (see a generic example here https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html#nodejs-handler-async)Alternatively use Python with is synchronous.https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/timestream-query.htmlCommentShareEXPERTMassimilianoAWSanswered a year ago


Question: AccessDenied when create ebs-csi-driver

"HiI'm having a problem when I install ebs-csi-driver for AWS EKSI used the account with administratoraccess to install via AWS console but had an error "namespaces "kube-system" is forbidden: User "eks:addon-manager" cannot patch resource "namespaces" in API group "" in the namespace "kube-system""I don't know why the administrator permission can not install add-on.I also tried to create IAM role following Linkhttps://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html but received error : Error from server (NotFound): serviceaccounts "ebs-csi-controller-sa" not foundI just started with EKS so don't know how to resolve,Anyone can help?Thank youFollowComment"

Answer: "0How you create EKS cluster? Via Console?CommentSharemyronix88answered 2 months agoCuong 2 months agono, from CLI, is it different?Share0Hello,Kindly note that the error below can happen because an important ClusterRoleBinding does not exist:"eks:addon-manager" cannot patch resource "namespaces" in API group "" in the namespace "kube-system""Execute the following command and check if you are able to see the 2 ClusterRoleBinding in your EKS Cluster :kubectl get clusterrolebinding -o wide | grep addoneks:addon-cluster-admin                                ClusterRole/cluster-admin                                          3d21h   eks:addon-manager                                                                  eks:addon-manager                                      ClusterRole/eks:addon-manager                                      3d21h   eks:addon-manager If the eks:addon-cluster-admin is missing, you can use the following yaml to create it:---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata:  name: eks:addon-cluster-adminsubjects:- kind: User  name: eks:addon-manager  apiGroup: rbac.authorization.k8s.ioroleRef:  kind: ClusterRole  name: cluster-admin  apiGroup: rbac.authorization.k8s.io---Therefore, you can apply this yaml file to your EKS Cluster:kubectl apply -f eks-addon-cluster-admin.yamlThe eks:addon-cluster-admin ClusterRoleBinding binds the cluster-admin ClusterRole to the eks:addon-manager Kubernetes identity. The role has the necessary permissions for the eks:addon-manager identity to create Kubernetes namespaces and install add-ons into namespaces. If the eks:addon-cluster-admin ClusterRoleBinding is removed, the Amazon EKS cluster continues to function, however Amazon EKS is no longer able to manage any add-ons.[1][1] - https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.htmlCommentSharerePost-User-8679597answered 21 days ago"


Question: Organizing Greengrass Cores and Lambdas

"Successfully deployed my first GG group, with one GG core a lambda, a local resource (USB0) and rules to handle the message queue. Quite pleased. It is measuring local seismic activity.This was just 1 site out of 10 sites. Now starting to add my second core that will perform the exact same functions and many questions arise on how to organize things:I have only one core per site. I guessed I needed to use groups to enable lambda's. As such, one site has one group has one core. Is that correct?These 10 form a functional group of measurements. How to organize those? I cannot find hierarchy of some kind. And groups do not seem to allow that hierarchy.The lambda for each location has the same code, but some parameters differ. E.g. the MQTT topic has the region and site name in it. Also the hardware id of the seismic measuring device is needed by the lambda. My question: If I would call region, site, hwId parameters, how should I make those availalble to the lambda? How should I maintain these?Thank you, excited to get this to work as intended and scale up to 100s of higher value measurements.FollowComment"

Answer: "0Hi jerome68,Regarding hierarchy, there are cores (exactly 1 core per group), and there are Greengrass aware devices (GGADs). GGADs connect to the cores. There's currently no way to make a group of groups or something like that. Instead you can organize your devices by the MQTT topics, something like topic/region/device . Then you can use the wildcards when setting up subscriptions.If the Lambda code is the same but you need to pass in some variables you can add Environment variables to the Lambda inside each Greengrass group.Please let me know if you have any other questions.Thanks,KeagenCommentShareKR-AWSanswered 4 years ago0That helps, thank youCommentSharejerome68answered 4 years ago"


Question: Please support .build domains in Route53

I would like to transfer my .build domain to Route53 but that is not in the supported list.FollowComment

Answer: "0Hi there,I understand that you want to transfer .build domain to Route 53 and it is currently not in the supported generic top-level domains of Amazon. Please correct me if I misunderstood.You can submit a feature request to AWS Support for the .build domain to be supported.To submit a feature request, open a case with AWS Premium Support and choose technical. You may refer to the following link:https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-caseI would also advise you to keep an eye on what’s new for future releases on this link [1].I hope this was helpful!Reference:[1] https://aws.amazon.com/new/?whats-new-content-all.sort-by=item.additionalFields.postDateTime&awsf.whats-new-serverless=*all&awsf.whats-new-storage=*all&whats-new-content-all.sort-order=desc&awsf.whats-new-analytics=*all&awsf.whats-new-app-integration=*all&awsf.whats-new-arvr=*all&awsf.whats-new-blockchain=*all&awsf.whats-new-business-applications=*all&awsf.whats-new-cloud-financial-management=*all&awsf.whats-new-compute=*all&awsf.whats-new-containers=*all&awsf.whats-new-customer-enablement=*all&awsf.whats-new-customer%20engagement=*all&awsf.whats-new-database=*all&awsf.whats-new-developer-tools=*all&awsf.whats-new-end-user-computing=*all&awsf.whats-new-mobile=*all&awsf.whats-new-gametech=*all&awsf.whats-new-iot=*all&awsf.whats-new-machine-learning=*all&awsf.whats-new-management-governance=*all&awsf.whats-new-media-services=*all&awsf.whats-new-migration-transfer=*all&awsf.whats-new-networking-content-delivery=*all&awsf.whats-new-quantum-tech=*all&awsf.whats-new-robotics=*all&awsf.whats-new-satellite=*all&awsf.whats-new-security-id-compliance=*allCommentShareKaraboanswered a year ago"


Question: MultiDataModel with different inference scripts

"I found a similar posting here but I'm hoping my situation is a little simpler.Q: Is there a way to provide two separate inference scripts for each model in the multi-endpoint or does some dynamic/custom inference script need to be made to handle both?I have two model pipeline built using SageMaker Python SDK Scikit-learn processing/models:One is a clustering model to return cluster prediction and centroid distances when requesting inferencingOther is simply PCA, returning 3-components when requesting inferencingBecause of the odd formating of the data and the way that the output needs to be provided, both models are using custom inference scripts (e.g. predicting vs. transformation).From what I can see in the examples for MultiDataModel, it only accepts a single entry_point for inference.py when passing the model information and just the model artifacts are "added" later:cluster_model = SKLearnModel(    model_data=cluster_artifact,    role=role,    entry_point="scripts/cluster_inference.py",    sagemaker_session=sagemaker_session)pca_model = SKLearnModel(    model_data=pca_artifact,    role=role,    entry_point="scripts/pca_inference.py",    sagemaker_session=sagemaker_session)mme = MultiDataModel(    name='model',    model_data_prefix=model_data_prefix,    model= cluster_model,    sagemaker_session=sagemaker_session,)Deployed as separate endpoints, both perform inference as expected, but I cannot get to function as one endpoint.Below is the most recent error I receive, but it is hard to understand where is failing, seeing as the serialization should be handled properly in my inference script and when invoking the endpoint:ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (503) from primary with message "{  "code": 503,  "type": "InternalServerException",  "message": "Unsupported model output data type."}".Any pointers or alternatives are appreciated!ThanksFollowComment"

Answer: "0Accepted AnswerHi there!In short, our SageMaker scikit-learn container do not currently support model specific inference script.The entry_point script that you referenced in your MultiDataModel object will be the inference script used for all the models. If you've added logging in your script, you will be able to see them in CloudWatch logs.If you have some pre/post-processing script that needs to be performed on a specific model, you need to write them all in one universal inference.py . You can then add some extra attributes in your data when invoking the endpoint and have the same script read the extra attribute so it knows which pre/post-processing script to perform.One important thing to note is that although you reference a model object in MultiDataModel, i.e.mme = MultiDataModel(    name='model',    model_data_prefix=model_data_prefix,    model= cluster_model,    sagemaker_session=sagemaker_session,)The only information fetched from the model object is the image_uri and entry_point, these information is needed for during endpoint deployment.All the model.tar.gz sitting in 'model_data_prefix' should not have a inference.py as it confuses the container and forces it to go back to default handler, hence why you're probably receiving ModelError.Try something like below:cluster_model = SKLearnModel(    model_data=cluster_artifact,    role=role,    entry_point="scripts/cluster_inference.py",    sagemaker_session=sagemaker_session)pca_model = SKLearnModel(    model_data=pca_artifact,    role=role,    entry_point="scripts/pca_inference.py",    sagemaker_session=sagemaker_session)mme = MultiDataModel(    name='model',    model_data_prefix=model_data_prefix,#make sure the directory of this prefix is empty, i.e. no models in this location    model= cluster_model,    sagemaker_session=sagemaker_session,)list(mme.list_models()) # this should be emptymme.add_model(model_data_source=cluster_artifact, model_data_path='cluster_artifact.tar.gz')#make sure model artifact doesn't contain inference.pymme.add_model(model_data_source=pca_artifact, model_data_path='pca_artifact.tar.gz') #make sure model artifact doesn't contain inference.pylist(mme.list_models()) # there should be two models listed now, if you look at the location of model_data_prefix, there should also be two model artifactoutput_cluster = predictor.predict(data='<your-data>', target_model='cluster_artifact.tar.gz')print(output_cluster) #this should work since it's using the inference.py from cluster_inference.pyoutput_pca = predictor.predict(data='<your-data>', target_model='pca_artifact.tar.gz')print(output_pca) #this might fail since it's using cluster_inference.py, add this model's inference script into cluster_inference.py to make it work I know this approach is not ideal since if you have a new model with new pre/post-processing script, you'd have to redeploy your endpoint for the new script to come into effect.We actually just added support in our tensorflow container to allow model specific inference script: https://github.com/aws/deep-learning-containers/pull/2680You can request for the same feature for our scikit container here: https://github.com/aws/sagemaker-scikit-learn-container/issuesCommentShareBenHanswered 2 months ago"


Question: AWS Glue ETL from SQL Server DB on premises to Snowflake on AWS cloud

Need to know if it is possible and feasible to do ETL with help of AWS Glue from SQL Server DB on premises(JDBC Connector used ) to Snowflake(JDBC Connector)?FollowComment

Answer: "0yes, it is.AWS Glue offers a native connector for SQL server and has a connector available for in the Glue connector Marketplace for SnowFlake.you can refer to this post to set up the connectivity to on-premises  and to this other post to view how to use the Snowflake connector.hope this helpsCommentShareEXPERTFabrizio@AWSanswered a year agoKhushi19995 a year agoThanks Fabrizio@AWS,Do we need to store data in S3; or we can directly fetch the data from SQL Server and do the transformation and pushes directly into Snowflake? What do you suggest ?Share"


Question: Lambda function for S3 tagging with multiple tags

"Hi Guys,I am trying put multiple tags on objects when they are copied on a particular folder, it works perfectly fine with single tags or multiple tags if I specify them explicitly, how to do it using a loop.Below works perfectly fine.response = s3.put_object_tagging(Bucket='mybucket',Key='Test/Test_test_xyz.csv',Tagging={'TagSet': [{'Key': 'key1','Value': 'value1'},{'Key': 'key2','Value': 'value2'},]})However I want to achieve it using a loop, Below is what I am trying, any help would be greatly appreciated.import boto3s3 = boto3.client("s3")all_objects = s3.list_objects(Bucket = 'Test')tag={'TEST':{'test': {'key1': 'value1' , 'key2': 'value2'}}}j=1for i in all_objects['Contents']:x=i['Key']try:y=(x.split('.')[0])if y:z=(y.split('/')[-1])if z:p=(z.split('')[0])q=(z.split('')[1])if p == 'Test':for s, t in tag[p][q].items():# Put S3 object taggingresponse = s3.put_object_tagging(Bucket='Test',Key=x,Tagging={'TagSet': [{'Key': s,'Value': t},]})except IndexError:passThanksSidFollowComment"

Answer: "0Got it, you just need to create a list of dictionaries and assign it to tag_setCommentSharesid1987answered 4 years ago"


Question: How to increase Docker pull timeout in Elastic Beanstalk?

"Hello,I am working on deploying an application that is packaged using Docker onto Elastic Beanstalk with a single EC2 instance currently.I have a multi-stage Dockerfile that is as small as I could possibly make it. Initially, I tried to deploy it to Elastic Beanstalk by deploying my Dockerfile, but the builds took too long so it would fail.So currently, I am building my image locally, pushing it to an AWS ECR repository, then deploying to elastic beanstalk using a Dockerrun.aws.json file. This, however, still gets timeout errors on deployment! When looking at the logs, it appears the build gets stopped because the command used to pull my pre-built image takes too long to download for some reason. So is there any way to increase this timeout?I have already tried running eb deploy with the --timeout flag, but it doesn't seem to change anything. I have also tried making a config file to increase the timeout:.ebextensions/increase-timeout.configoption_settings:    - namespace: aws:elasticbeanstalk:command      option_name: Timeout      value: 1800But that also fails to change the 300 second timeout.Does anyone have any idea of how I could fix this? Thanks!FollowComment"

Answer: "0A Docker engine timeout settings in backend code as “const (defaultCommandTimeout = 300)”. The aws:elasticbeanstalk:command timeout option (Timeout config in Console, default is 600s) is in fact applied. Nevertheless, this applies for the whole Beanstalk deployment. The commands during building Docker app are timing out before and failing. The 300 seconds timeout is fixed and cannot be modified at the moment.Please create a support case to AWS with below information that helps AWS support team to provide work around if available.A time frame when the issue appears.Upload failed log files.Dockerfile / docker-compose.yaml / Dockerrun.aws .json filesThe size of the image you would like to pull from ECR, also cloud you please kindly provide some context of the image if it’s with a large size.CommentShareRam_AWSanswered 2 months ago"


Question: S3 Bucket Security

"HiTotal noob when it comes to AWS.I am looking to shift my veeam backups to capacity tier which will be an S3 bucket.The S3 bucket will have all public access blocked, use versioning and object lock.My question is really around how best to secure the bucket in terms of IAM/Bucket Policy/ACLWe don't have a VPC setup at present as we use Azure for cloud workloads and email.Basically what I want to do is to restrict access to this S3 bucket to a particular user and lock it down to a single public IP address.Both conditions must be met to get access to the bucket.For a noob, the number of options when creating the policy is a bit confusing.I have used the policy generator but it throws an error when I try add the principal (IAM user)Any help greatly appreciated.ThanksPFollowComment"

Answer: "1Good day.To limit access to specific IAM principals, here is our public guide: https://aws.amazon.com/blogs/security/how-to-restrict-amazon-s3-bucket-access-to-a-specific-iam-role/To limit access to specific IP addresses, here is out public guide: https://aws.amazon.com/premiumsupport/knowledge-center/block-s3-traffic-vpc-ip/I'm hoping this is able to help you.Truly,Jason H.CommentShareJason_Hanswered a year ago0See this page for recommended access controls: https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-best-practices.htmlIn short, disable ACLs on the bucket and govern access via IAM. More details on how to do this here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/about-object-ownership.htmlCommentShareRodney Lesteranswered a year ago0Hi Guys,Sorry for the slow response and thanks for the responses I did receive.This is kind of what i want to do but the policy gives out about the principal.I want access to be allowed for a particular group AND only from a particular IP4 block.{"Id": "Policy1639567223578","Version": "2012-10-17","Statement": [{"Sid": "Stmt16395456545451","Action": "s3:","Effect": "Allow","Resource": ["arn:aws:s3:::bucket1","arn:aws:s3:::bucket1/"],"Principal": {"AWS": ["arn:aws:iam::awsaccount:group/backup_admins"]}},{"Sid": "Stmt1631234560507","Action": "s3:","Effect": "Deny","Resource": ["arn:aws:s3:::bucket1","arn:aws:s3:::bucket1/"],"Condition": {"NotIpAddress": {"aws:SourceIp": "ip4subnet"}},"Principal": "*"}]}This policy was built using the policy generator, when i add it to the bucket, I get this error:Unsupported Principal: The policy type IDENTITY_POLICY does not support the Principal element. Remove the Principal element.CheersCommentShareAWS-User-7490208answered a year ago0You cannot specify an AWS User Group as a principal in an IAM policy:https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.htmlhttps://docs.aws.amazon.com/IAM/latest/UserGuide/id_groups.htmlYou could specify one or more User ARNs. Or, if you want to keep using a Group you can grant access to the bucket from a policy associated with the Group itself:IAM Group Policy (this can be an inline policy or a managed policy attached to the Group):{  "Version": "2012-10-17",  "Statement": [{    "Action": [      "s3:ListBucket",      "s3:GetObject*",      "s3:GetBucket*",      "s3:List*",      "s3:Abort*",    ],    "Effect": "Allow",    "Resource": [      "arn:aws:s3:::bucket1",       "arn:aws:s3:::bucket1/*"    ],  }]}Note: the above is not tested and be sure to limit the actions to least privileged.And keep the Bucket Policy to restrict access from a specific IP (note: this will break access to the bucket from the AWS Console):  "Version": "2012-10-17",  "Statement": [{    "Action": "s3:*",    "Effect": "Deny",    "Resource": [      "arn:aws:s3:::bucket1",       "arn:aws:s3:::bucket1/*"    ],    "Condition": {      "NotIpAddress": {        "aws:SourceIp": "x.x.x.x"      }    },    "Principal": "*"  }]}CommentSharewdavjonesanswered a year ago"


Question: How can I achieve multiple-object detection with Amazon Rekognition Custom Labels?

Amazon Rekognition Custom Labels allows selecting several instances of the same type of objects when I select the bounding box labeling method. Can Rekognition Custom Labels detect several instances of the same objects on provided images?FollowComment

Answer: "0Accepted AnswerFor object detection, Amazon Rekognition Custom Labels can train a multi-class/multi-label object detection model with up to 250 labels per dataset. You can label more than one object with the same class in a single image during data labeling using the object drawing bbox.Amazon Custom Labels can return up to 50 labels per image.For information on quotas in Amazon Rekognition Custom Labels, see [Guidelines and quotas in Amazon Rekognition Custom Labels][1].[1]: https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/limits.htmlCommentShareWoody Borraccinoanswered 3 years ago"


Question: Web hosting for Raffle Saas Platform

"HelloI am developing a Saas platform, precisely a raffle business where raffle will be run LIVE and I would like to know what web hosting would be best for me and what would be the cost?Amazon Hosting S3, plus route 53 and cloud frontAmazon WordPress LightsailElastic Load BalancingAmazon EC2FollowComment"

Answer: 0Each of the services you mentioned has different use cases and cost. I have listed down their perspective website for your reference.Amazon Hosting S3: https://aws.amazon.com/s3Amazon WordPress Lightsail: https://aws.amazon.com/lightsailElastic Load Balancing: https://aws.amazon.com/elasticloadbalancing/Amazon EC2: https://aws.amazon.com/ec2CommentShareAWS-User-2776000answered 3 months ago


Question: ESP32 AWS tutorial Lamba question

"Hey guys, this might be a newb question but im trying to follow this guide- https://aws.amazon.com/blogs/compute/building-an-aws-iot-core-device-using-aws-serverless-and-an-esp32/And at step "Deploying the lambda-iot-rule AWS SAR application": the "lambda-iot-rule" page is no longer available at AWS Serverless Application Repository. How should I proceed?FollowComment"


Question: site can't be reached

"In all the exercises I did for AWS Cloud Technical Essentials, none of the pages I tried to view using the public IP address worked. I did every solution I saw on YouTube and here on this website, but they didn't work.I did the following:Checked the sg to see if I set the http, ssh, https accessible anywhere.I also connect to the shell, and installed https, checked the status and restarted the server. This one I was able to display something. It displayed "It works!". Every time I do these steps, the page always displays the sameI also tried to add ICMP in the security group. I was able to ping the IP address, but then, the page still displays "site can't be reached"I also disabled my antivirusI also checked my proxy settings, and follow the steps I read to clear themI also added the IP address in the list of sites that are safe to browse in my chrome settingsI DID ALL THESE in both my Mac and Windows computers.If I made a mistake or I missed something in the steps provided for the exercises, I don't believe that I did in all the exercises, especially the first few where the steps were still simple and easyAny help will be greatly appreciated. Thanks.FollowCommentkentrad EXPERT2 months agoCan you post the User Data script used in this lab?Share"

Answer: "1Accepted AnswerMake sure to choose Amazon Linux 2 instead of Amazon Linux 2023 when you launch the instance. In the console, Amazon Linux 2023 is now the default. You will need to terminate the existing instance and create a new one.CommentShareEXPERTkentradanswered 2 months agorePost-User-3867614 2 months agoThank you so much for your help. This solved my problem.Share0What is the HTTP status code returned when connecting to EC2 and executing the following command?curl localhost -o /dev/null -w '%{http_code}\n' -sIn case of 5XX error, there is most likely a problem with the web server configuration.CommentShareEXPERTRiku_Kobayashianswered 2 months agoRiku_Kobayashi EXPERT2 months agoIt is possible that the web server is not started.Please execute the following command to start it.sudo systemctl start httpdorsudo systemctl start apache orsudo systemctl start nginxShareRiku_Kobayashi EXPERT2 months agoDid you solve the problem here?SharerePost-User-3867614 2 months agoThanks for your answer. This is what I got.[ec2-user@ip- ~]$ curl localhost -o /dev/null -w '%{http_code}\n' -s000Share0[ec2-user@ip- ~]$ sudo systemctl start httpdFailed to start httpd.service: Unit httpd.service not found.[ec2-user@ip- ~]$ sudo yum install httpdLast metadata expiration check: 1 day, 3:24:20 ago on Thu Apr  6 16:11:44 2023.ec2-user@ip- ~]$ sudo systemctl start httpd[ec2-user@ip- ~]$ service httpd statusRedirecting to /bin/systemctl status httpd.service● httpd.service - The Apache HTTP ServerLoaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; preset: disabled)Active: active (running) since Fri 2023-04-07 19:37:10 UTC; 23s agoDocs: man:httpd.service(8)Main PID: 25460 (httpd)Status: "Total requests: 0; Idle/Busy workers 100/0;Requests/sec: 0; Bytes served/sec:   0 B/sec"Tasks: 177 (limit: 1112)Memory: 12.8MCPU: 77msCGroup: /system.slice/httpd.service├─25460 /usr/sbin/httpd -DFOREGROUND├─25472 /usr/sbin/httpd -DFOREGROUND├─25473 /usr/sbin/httpd -DFOREGROUND├─25474 /usr/sbin/httpd -DFOREGROUND└─25475 /usr/sbin/httpd -DFOREGROUNDApr 07 19:37:10 ip-.us-east-2.compute.internal systemd[1]: Starting httpd.service - The Apache HTTP Server...Apr 07 19:37:10 ip-.us-east-2.compute.internal systemd[1]: Started httpd.service - The Apache HTTP Server.Apr 07 19:37:10 ip-.us-east-2.compute.internal httpd[25460]: Server configured, listening on: port 80###############################When I entered my IP address in my browser, it displayed this:It works!BUT, that's not the output I was expecting. I think that "It works!" is a default message.CommentSharerePost-User-3867614answered 2 months agokentrad EXPERT2 months agoReally need to see the User Data script from the lab to determine what application is running. We don't know if Apache is even the web server.Share0To: kentradI pasted below my  user data. Thank you.#!/bin/bash -exwget https://aws-tc-largeobjects.s3-us-west-2.amazonaws.com/DEV-AWS-MO-GCNv2/FlaskApp.zipunzip FlaskApp.zipcd FlaskApp/yum -y install python3 mysqlpip3 install -r requirements.txtamazon-linux-extras install epelyum -y install stressexport PHOTOS_BUCKET=${SUB_PHOTOS_BUCKET}export AWS_DEFAULT_REGION=<INSERT REGION HERE>export DYNAMO_MODE=onFLASK_APP=application.py /usr/local/bin/flask run --host=0.0.0.0 --port=80CommentSharerePost-User-3867614answered 2 months ago"


Question: AWS IDT problem [Error: 106] problem

"HiI just got some problem when I using device testerThe following are my error message and my config fileI don't know what I missing casue that problem?I already use IAM create account and export in terminal.yichen@yichen-desktop:~/devicetester_greengrass_linux/bin$ sudo ./devicetester_linux_x86-64 run-suite[sudo] password for yichen:time="2019-10-02T15:23:48+08:00" level=info msg=Using suite: GGQ_1time="2019-10-02T15:23:48+08:00" level=info msg=Using pool: 1time="2019-10-02T15:23:48+08:00" level=info msg=Running test case... executionId=9326d229-e4e5-11e9-a446-080027639d9 suiteId=GGQ groupId=version testCaseId=ggc_version_check_test deviceId=123time="2019-10-02T15:23:48+08:00" level=error msg=AWS credentials not found: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment suiteId=GGQ groupId=version testCaseId=ggc_version_check_test deviceId=123 executionId=9326d229-e4e5-11e9-a446-0800276f39d9time="2019-10-02T15:23:48+08:00" level=info msg=Finished running test case... testCaseId=ggc_version_check_test                                 deviceId=123 executionId=9326d229-e4e5-11e9-a446-0800276f39d9 suiteId=GGQ groupId=versiontime="2019-10-02T15:23:48+08:00" level=info msg=--- FAIL: TestGGCVersion (0.00s) suiteId=GGQ groupId=versiontestCaseId=ggc_version_check_test deviceId=123 executionId=9326d229-e4e5-11e9-a446-0800276f39d9time="2019-10-02T15:23:48+08:00" level=info msg=FAIL executionId=9326d229-e4e5-11e9-a446-0800276f39d9 suiteId=GG                                              Q groupId=version testCaseId=ggc_version_check_test deviceId=123time="2019-10-02T15:23:48+08:00" level=error msg=Test exited unsuccessfully executionId=9326d229-e4e5-11e9-a446-800276f39d9 testCaseId=ggc_version_check_test error=exit status 1time="2019-10-02T15:23:48+08:00" level=info msg=All tests finished. executionId=9326d229-e4e5-11e9-a446-0800276f9d9time="2019-10-02T15:23:49+08:00" level=info msg=========== Test Summary ==========Execution Time:         1sTests Completed:        1Tests Passed:           0Tests Failed:           1Tests Skipped:          0Test Groups:version:            FAILEDFailed Tests:Group Name: versionTest Name: Test GGC Version versionReason: [Error: 106] ValidationError: AWS credentials not found: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment. Refer to the logs and troubleshooting section of IDT User Guide                                               https://docs.aws.amazon.com/greengrass/latest/developerguide/device-tester-for-greengrass-ug.html for more information.Path to AWS IoT Device Tester Report: /home/yichen/devicetester_greengrass_linux/results/9326d229-e4e5-11e9-a446                                              -0800276f39d9/awsiotdevicetester_report.xmlPath to Test Execution Logs: /home/yichen/devicetester_greengrass_linux/results/9326d229-e4e5-11e9-a446-0800276f9d9/logsPath to Aggregated JUnit Report: /home/yichen/devicetester_greengrass_linux/results/9326d229-e4e5-11e9-a446-0800276f39d9/GGQ_Report.xml{"log": {"location": "../logs/"},"configFiles": {"root": "../configs","device": "../configs/device.json"},"testPath": "../tests/","reportPath": "../results/","certificatePath": "../certificates/","awsRegion": "us-west-2","auth": {"method": "environment"}[{"id": "1","sku": "8783d","features": [{"name": "os","value": "ubuntu"},{"name": "arch","value": "x86_64"}],"kernelConfigLocation": "","greengrassLocation": "/greengrass","devices": [{"id": "123","connectivity": {"protocol": "ssh","ip": "10.1.70.127","auth": {"method": "pki","credentials": {"user": "yichen","privKeyPath": "/home/yichen/.ssh/id_rsa"}}}}]}]FollowComment"

Answer: "0Hi yichen,IDT needs AWS credentials to run, and allows two way to provision them using credentials file and environment variables(https://docs.aws.amazon.com/greengrass/latest/developerguide/set-config.html).Based on your config file given below, you seem to be using environment variables( see "auth" set in config.json). The console output shows "AWS credentials not found: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment".IDT is expecting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set as environment variables to run. Please set these variables and export them as environment variables. Refer https://docs.aws.amazon.com/greengrass/latest/developerguide/set-config.html for more details.Thanks,CommentShareAWS-User-4357211answered 4 years ago0HiI already tried but not working.yichen@yichen-desktop:/devicetester_greengrass_linux/configs$ cat /.aws/credentials[default]aws_access_key_id = *********************aws_secret_access_key = *********************yichen@yichen-desktop:/devicetester_greengrass_linux/configs$ cat config.json{"log": {"location": "../logs/"},"configFiles": {"root": "../configs","device": "../configs/device.json"},"testPath": "../tests/","reportPath": "../results/","certificatePath": "../certificates/","awsRegion": "us-west-2","auth": {"method": "environment"}}yichen@yichen-desktop:/devicetester_greengrass_linux/configs$ export AWS_ACCESS_KEY_ID=*********************yichen@yichen-desktop:/devicetester_greengrass_linux/configs$ export AWS_SECRET_ACCESS_KEY=*********************yichen@yichen-desktop:/devicetester_greengrass_linux/configs$ export -p | grep AWSdeclare -x AWS_ACCESS_KEY_ID=""declare -x AWS_SECRET_ACCESS_KEY=""yichen@yichen-desktop:~/devicetester_greengrass_linux/configs$ sudo ./../bin/devicetester_linux_x86-64 run-suite[sudo] password for yichen:time="2019-10-03T08:31:38+08:00" level=info msg=Using suite: GGQ_1time="2019-10-03T08:31:38+08:00" level=info msg=Using pool: 1time="2019-10-03T08:31:38+08:00" level=info msg=Running test case... suiteId=GGQ groupId=version testCaseId=ggc_version_check_test deviceId=123 executionId=28f97589-e575-11e9-9731-0800276f39d9time="2019-10-03T08:31:38+08:00" level=error msg=AWS credentials not found: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment suiteId=GGQ groupId=version testCaseId=ggc_version_check_test deviceId=123 executionId=28f97589-e575-11e9-9731-0800276f39d9time="2019-10-03T08:31:38+08:00" level=info msg=Finished running test case... executionId=28f97589-e575-11e9-9731-0800276f39d9 suiteId=GGQ groupId=version testCaseId=ggc_version_check_test deviceId=123time="2019-10-03T08:31:38+08:00" level=info msg=--- FAIL: TestGGCVersion (0.00s) testCaseId=ggc_version_check_test deviceId=123 executionId=28f97589-e575-11e9-9731-0800276f39d9 suiteId=GGQ groupId=versiontime="2019-10-03T08:31:38+08:00" level=info msg=FAIL deviceId=123 executionId=28f97589-e575-11e9-9731-0800276f39d9 suiteId=GGQ groupId=version testCaseId=ggc_version_check_testtime="2019-10-03T08:31:38+08:00" level=error msg=Test exited unsuccessfully testCaseId=ggc_version_check_test error=exit status 1 executionId=28f97589-e575-11e9-9731-0800276f39d9time="2019-10-03T08:31:38+08:00" level=info msg=All tests finished. executionId=28f97589-e575-11e9-9731-0800276f39d9time="2019-10-03T08:31:38+08:00" level=info msg=========== Test Summary ==========Execution Time:         1sTests Completed:        1Tests Passed:           0Tests Failed:           1Tests Skipped:          0Test Groups:version:            FAILEDFailed Tests:Group Name: versionTest Name: Test GGC Version versionReason: [Error: 106] ValidationError: AWS credentials not found: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment. Refer to the logs and troubleshooting section of IDT User Guide https://docs.aws.amazon.com/greengrass/latest/developerguide/device-tester-for-greengrass-ug.html for more information.Path to AWS IoT Device Tester Report: /home/yichen/devicetester_greengrass_linux/results/28f97589-e575-11e9-9731-0800276f39d9/awsiotdevicetester_report.xmlPath to Test Execution Logs: /home/yichen/devicetester_greengrass_linux/results/28f97589-e575-11e9-9731-0800276f39d9/logsPath to Aggregated JUnit Report: /home/yichen/devicetester_greengrass_linux/results/28f97589-e575-11e9-9731-0800276f39d9/GGQ_Report.xmlCommentShareyichenanswered 4 years ago0Hi yichen,You have two ways to fix this:1).  You can just run IDT without using sudo command: ./devicetester_linux_x86-64 run-suite.Because when you use sudo, it will open a new shell which will not have the environment variables you configured before.2). I noticed that you have configured ~/.aws/credentials, the other way to fix this is to change your config.json to use file instead of environment.i.e."auth": {"method": "file","credentials": {"profile": "default"}}Please go to https://docs.aws.amazon.com/greengrass/latest/developerguide/set-config.html for more details.ThanksEdited by: aws-jessica on Oct 3, 2019 4:18 PMCommentShareAWS-User-2027723answered 4 years ago0Run device test without sudo solve error106 problemThanks for your helpCommentShareyichenanswered 4 years ago"


Question: How to you manage application state when using KCL

"how to you mange application state when using KCL.I create a simple application which will save my application state on efs when creating checkpoint, but the state is per shard. it willnot work when resharding. This is should be a common issue for KCL application. how do you solve it?FollowComment"


Question: HIVE_BAD_DATA: Error parsing field value '2021-10-31 22:00:00.0' for field 3: For input string: "2021-10-31 22:00:00.0"

"I am reading files from S3 and using a Glue ETL job to populate a Data Catalog Table.My S3 files look like this:VID,ALTID,VTYPE,TIME,TIMEGMT,VALUEABC, ABC, DATAFEED,31-10-2021 22:00:00,01-11-2021 02:00:00,   11775.685The scheme read in looks like thisroot|-- VID: string|-- ALTID: string|-- VTYPE: string|-- TIME: string|-- TIMEGMT: string|-- VALUE: stringI am changing fields 3 and 4 from "strings" to timestamps, which matches the schema of my Data Catalog Table.  I am also doing a few other transformations.I am transforming them like so:df = df.withColumn("time", to_timestamp("time", 'dd-MM-yyyy HH:mm:ss'))df = df.withColumn("timegmt", to_timestamp("timegmt", 'dd-MM-yyyy HH:mm:ss'))When I try to read the data with Athena, I get an error:HIVE_BAD_DATA: Error parsing field value '2021-10-31 22:00:00.0' for field 3: For input string: "2021-10-31 22:00:00.0"The Data Catalog Table Schema looks like this:Column NameData TypevidstringaltidstringvtypestringtimetimestamptimegmttimestampvalueintfilenamestringAnd the line in the run-1647806179090-part-r-00000 file it is choking on looks like this:vid,altid,vtype,time,timegmt,value,filenameABC,ABC, DATAFEED,"2021-10-31 22:00:00.0","2021-11-01 02:00:00.0",11775,"lf_buf_20211101_005904.csv"Does anyone know why it would throw this error? I believe according to the documentation, this is the correct timestamp format.FollowComment"

Answer: "0Accepted AnswerIt turns out, the documentation for Athena is either incorrect or at best misleading.  The excellent answer by Alexandre says it best here https://stackoverflow.com/questions/52564194/athena-unable-to-parse-date-using-opencsvserdeBasically you need to store the date or the timestamp in UNIX Epoch time.  You wouldn't know that, because of all the emphasis on the format of the time.  I tried as a timestamp, and that is why I got this error.  As soon as I stored it as UNIX time I got somewhere.  However, the unix_timestamp() function only returns time in seconds (long) and timestamp wants time in milliseconds (double).  So I simply multiplied by 1000:df = df.withColumn("time", f.unix_timestamp("time", 'dd-MM-yyyy HH:mm:ss') * 1000) After doing this, you will have a 13 digit double, and Athena will properly produce a timestamp from it if you have selected Timestamp as the Data Type.CommentSharebfeenyanswered a year ago"