Question: Issue with accessing Glue Data Catalog with Spark

"Hello!I'm forwarding my issue from StackOverflow where I was unable to find correct answer.I'm using Spark 2.4.0 on EMR from spark-shell (executed by user hadoop on master node) and trying to store simple Dataframe in S3 using AWS Glue Data Catalog. EMR has automatically generated default IAM roles. The code is below:val peopleTable = spark.sql("select * from emrdb.testtableemr")val filtered = peopleTable.filter("name = 'Andrzej'")filtered.repartition(1).write.format("hive").mode("append").saveAsTable("emrdb.destDir")table emrdb.testtableemr exists in my Glue Data Catalog and was created by Glue Crawler on S3 directory where only one json file exists:{"Name": "Andrzej", "Surname": "WenWen", "age": "32"}{"Name": "Tomasz", "Surname": "Tomtom", "age": "42"}{"Name": "Andrzej", "Surname": "Golota", "age": "52"}Above code works as expected- data is filtered and stored in s3 directory that is linked with AWS Glue table emrdb.destDir. (emrdb.destDir table was also created by crowler- in table's directory I put same file for crowler to create same structure) The issue I got is: although it works correctly it still throws below exception:scala> filtered.repartition(1).write.format("hive").mode("append").saveAsTable("emrdb.destDir")org.apache.spark.sql.AnalysisException: java.lang.IllegalArgumentException: Can not create a Path from an empty string;  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)  at org.apache.spark.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.loadTable(ExternalCatalogWithListener.scala:159)  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:259)  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)  at org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand.run(CreateHiveTableAsSelectCommand.scala:66)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)  at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)  at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)  at org.apache.spark.sql.DataFrameWriter.createTable(DataFrameWriter.scala:465)  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:444)  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:400)  ... 49 elidedCaused by: java.lang.IllegalArgumentException: Can not create a Path from an empty string  at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163)  at org.apache.hadoop.fs.Path.<init>(Path.java:175)  at org.apache.hadoop.hive.metastore.Warehouse.getDatabasePath(Warehouse.java:172)  at org.apache.hadoop.hive.metastore.Warehouse.getTablePath(Warehouse.java:184)  at org.apache.hadoop.hive.metastore.Warehouse.getFileStatusesForUnpartitionedTable(Warehouse.java:520)  at org.apache.hadoop.hive.metastore.MetaStoreUtils.updateUnpartitionedTableStatsFast(MetaStoreUtils.java:180)  at com.amazonaws.glue.shims.AwsGlueSparkHiveShims.updateTableStatsFast(AwsGlueSparkHiveShims.java:62)  at com.amazonaws.glue.catalog.metastore.GlueMetastoreClientDelegate.alterTable(GlueMetastoreClientDelegate.java:534)  at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.alter_table(AWSCatalogMetastoreClient.java:400)  at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:497)  at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:485)  at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1669)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  at java.lang.reflect.Method.invoke(Method.java:498)  at org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:878)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply$mcV$sp(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275)  at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213)  at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212)  at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258)  at org.apache.spark.sql.hive.client.HiveClientImpl.loadTable(HiveClientImpl.scala:779)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply$mcV$sp(HiveExternalCatalog.scala:845)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)  ... 74 moreI got same error trying to execute below:filtered.repartition(1).write.mode("append").insertInto("emrdb.destDir")but I'm able to store data bypassing Glue Data Catalog:filtered.repartition(1).write.format("json").mode("append").save("s3://awenclaw-emr-test/destDir/")This make me thinking the issue is on Data Catalog site.The solution suggested on StackOverflow throws same error:filtered.repartition(1).write.option("path", "s3://awenclaw-emr-test/destDir/").format("hive").mode("append").saveAsTable("emrdb.destDir")So my question is how to correctly store Spark DataFrame into Glue Data Catalog table without all error messages mentioned above?And link to StackOverflow if you need more details on answer I received.https://stackoverflow.com/questions/54441163/writing-spark-dataframe-to-hive-table-through-aws-glue-data-catalougThanks in advance.AndrzejEdited by: awenclaw on Feb 6, 2019 1:25 AMFollowComment"

Answer: "0awenclaw wrote:Hello!I'm forwarding my issue from StackOverflow where I was unable to find correct answer.I'm using Spark 2.4.0 on EMR from spark-shell (executed by user hadoop on master node) and trying to store simple Dataframe in S3 using AWS Glue Data Catalog. EMR has automatically generated default IAM roles. The code is below:val peopleTable = spark.sql("select * from emrdb.testtableemr")val filtered = peopleTable.filter("name = 'Andrzej'")filtered.repartition(1).write.format("hive").mode("append").saveAsTable("emrdb.destDir")table emrdb.testtableemr exists in my Glue Data Catalog and was created by Glue Crawler on S3 directory where only one json file exists:{"Name": "Andrzej", "Surname": "WenWen", "age": "32"}{"Name": "Tomasz", "Surname": "Tomtom", "age": "42"}{"Name": "Andrzej", "Surname": "Golota", "age": "52"}Above code works as expected- data is filtered and stored in s3 directory that is linked with AWS Glue table emrdb.destDir. (emrdb.destDir table was also created by crowler- in table's directory I put same file for crowler to create same structure) The issue I got is: although it works correctly it still throws below exception:scala> filtered.repartition(1).write.format("hive").mode("append").saveAsTable("emrdb.destDir")org.apache.spark.sql.AnalysisException: java.lang.IllegalArgumentException: Can not create a Path from an empty string;  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)  at org.apache.spark.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.loadTable(ExternalCatalogWithListener.scala:159)  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:259)  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)  at org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand.run(CreateHiveTableAsSelectCommand.scala:66)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)  at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)  at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)  at org.apache.spark.sql.DataFrameWriter.createTable(DataFrameWriter.scala:465)  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:444)  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:400)  ... 49 elidedCaused by: java.lang.IllegalArgumentException: Can not create a Path from an empty string  at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163)  at org.apache.hadoop.fs.Path.<init>(Path.java:175)  at org.apache.hadoop.hive.metastore.Warehouse.getDatabasePath(Warehouse.java:172)  at org.apache.hadoop.hive.metastore.Warehouse.getTablePath(Warehouse.java:184)  at org.apache.hadoop.hive.metastore.Warehouse.getFileStatusesForUnpartitionedTable(Warehouse.java:520)  at org.apache.hadoop.hive.metastore.MetaStoreUtils.updateUnpartitionedTableStatsFast(MetaStoreUtils.java:180)  at com.amazonaws.glue.shims.AwsGlueSparkHiveShims.updateTableStatsFast(AwsGlueSparkHiveShims.java:62)  at com.amazonaws.glue.catalog.metastore.GlueMetastoreClientDelegate.alterTable(GlueMetastoreClientDelegate.java:534)  at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.alter_table(AWSCatalogMetastoreClient.java:400)  at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:497)  at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:485)  at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1669)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  at java.lang.reflect.Method.invoke(Method.java:498)  at org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:878)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply$mcV$sp(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275)  at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213)  at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212)  at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258)  at org.apache.spark.sql.hive.client.HiveClientImpl.loadTable(HiveClientImpl.scala:779)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply$mcV$sp(HiveExternalCatalog.scala:845)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)  ... 74 moreI got same error trying to execute below:filtered.repartition(1).write.mode("append").insertInto("emrdb.destDir")but I'm able to store data bypassing Glue Data Catalog:filtered.repartition(1).write.format("json").mode("append").save("s3://awenclaw-emr-test/destDir/")This make me thinking the issue is on Data Catalog site.The solution suggested on StackOverflow throws same error:filtered.repartition(1).write.option("path", "s3://awenclaw-emr-test/destDir/").format("hive").mode("append").saveAsTable("emrdb.destDir")So my question is how to correctly store Spark DataFrame into Glue Data Catalog table without all error messages mentioned above?And link to StackOverflow if you need more details on answer I received.https://stackoverflow.com/questions/54441163/writing-spark-dataframe-to-hive-table-through-aws-glue-data-catalougThanks in advance.AndrzejEdited by: awenclaw on Feb 6, 2019 1:25 AMawenclaw wrote:Hello!I'm forwarding my issue from StackOverflow where I was unable to find correct answer.I'm using Spark 2.4.0 on EMR from spark-shell (executed by user hadoop on master node) and trying to store simple Dataframe in S3 using AWS Glue Data Catalog. EMR has automatically generated default IAM roles. The code is below:val peopleTable = spark.sql("select * from emrdb.testtableemr")val filtered = peopleTable.filter("name = 'Andrzej'")filtered.repartition(1).write.format("hive").mode("append").saveAsTable("emrdb.destDir")table emrdb.testtableemr exists in my Glue Data Catalog and was created by Glue Crawler on S3 directory where only one json file exists:{"Name": "Andrzej", "Surname": "WenWen", "age": "32"}{"Name": "Tomasz", "Surname": "Tomtom", "age": "42"}{"Name": "Andrzej", "Surname": "Golota", "age": "52"}Above code works as expected- data is filtered and stored in s3 directory that is linked with AWS Glue table emrdb.destDir. (emrdb.destDir table was also created by crowler- in table's directory I put same file for crowler to create same structure) The issue I got is: although it works correctly it still throws below exception:scala> filtered.repartition(1).write.format("hive").mode("append").saveAsTable("emrdb.destDir")org.apache.spark.sql.AnalysisException: java.lang.IllegalArgumentException: Can not create a Path from an empty string;  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)  at org.apache.spark.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.loadTable(ExternalCatalogWithListener.scala:159)  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:259)  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)  at org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand.run(CreateHiveTableAsSelectCommand.scala:66)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)  at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)  at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)  at org.apache.spark.sql.DataFrameWriter.createTable(DataFrameWriter.scala:465)  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:444)  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:400)  ... 49 elidedCaused by: java.lang.IllegalArgumentException: Can not create a Path from an empty string  at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163)  at org.apache.hadoop.fs.Path.<init>(Path.java:175)  at org.apache.hadoop.hive.metastore.Warehouse.getDatabasePath(Warehouse.java:172)  at org.apache.hadoop.hive.metastore.Warehouse.getTablePath(Warehouse.java:184)  at org.apache.hadoop.hive.metastore.Warehouse.getFileStatusesForUnpartitionedTable(Warehouse.java:520)  at org.apache.hadoop.hive.metastore.MetaStoreUtils.updateUnpartitionedTableStatsFast(MetaStoreUtils.java:180)  at com.amazonaws.glue.shims.AwsGlueSparkHiveShims.updateTableStatsFast(AwsGlueSparkHiveShims.java:62)  at com.amazonaws.glue.catalog.metastore.GlueMetastoreClientDelegate.alterTable(GlueMetastoreClientDelegate.java:534)  at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.alter_table(AWSCatalogMetastoreClient.java:400)  at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:497)  at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:485)  at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1669)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  at java.lang.reflect.Method.invoke(Method.java:498)  at org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:878)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply$mcV$sp(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275)  at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213)  at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212)  at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258)  at org.apache.spark.sql.hive.client.HiveClientImpl.loadTable(HiveClientImpl.scala:779)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply$mcV$sp(HiveExternalCatalog.scala:845)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)  ... 74 moreI got same error trying to execute below:filtered.repartition(1).write.mode("append").insertInto("emrdb.destDir")but I'm able to store data bypassing Glue Data Catalog:filtered.repartition(1).write.format("json").mode("append").save("s3://awenclaw-emr-test/destDir/")This make me thinking the issue is on Data Catalog site.The solution suggested on StackOverflow throws same error:filtered.repartition(1).write.option("path", "s3://awenclaw-emr-test/destDir/").format("hive").mode("append").saveAsTable("emrdb.destDir")So my question is how to correctly store Spark DataFrame into Glue Data Catalog table without all error messages mentioned above?And link to StackOverflow if you need more details on answer I received.https://stackoverflow.com/questions/54441163/writing-spark-dataframe-to-hive-table-through-aws-glue-data-catalougThanks in advance.AndrzejEdited by: awenclaw on Feb 6, 2019 1:25 AMawenclaw wrote:Hello!I'm forwarding my issue from StackOverflow where I was unable to find correct answer.I'm using Spark 2.4.0 on EMR from spark-shell (executed by user hadoop on master node) and trying to store simple Dataframe in S3 using AWS Glue Data Catalog. EMR has automatically generated default IAM roles. The code is below:val peopleTable = spark.sql("select * from emrdb.testtableemr")val filtered = peopleTable.filter("name = 'Andrzej'")filtered.repartition(1).write.format("hive").mode("append").saveAsTable("emrdb.destDir")table emrdb.testtableemr exists in my Glue Data Catalog and was created by Glue Crawler on S3 directory where only one json file exists:{"Name": "Andrzej", "Surname": "WenWen", "age": "32"}{"Name": "Tomasz", "Surname": "Tomtom", "age": "42"}{"Name": "Andrzej", "Surname": "Golota", "age": "52"}Above code works as expected- data is filtered and stored in s3 directory that is linked with AWS Glue table emrdb.destDir. (emrdb.destDir table was also created by crowler- in table's directory I put same file for crowler to create same structure) The issue I got is: although it works correctly it still throws below exception:scala> filtered.repartition(1).write.format("hive").mode("append").saveAsTable("emrdb.destDir")org.apache.spark.sql.AnalysisException: java.lang.IllegalArgumentException: Can not create a Path from an empty string;  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)  at org.apache.spark.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.loadTable(ExternalCatalogWithListener.scala:159)  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:259)  at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)  at org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand.run(CreateHiveTableAsSelectCommand.scala:66)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)  at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)  at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)  at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)  at org.apache.spark.sql.DataFrameWriter.createTable(DataFrameWriter.scala:465)  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:444)  at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:400)  ... 49 elidedCaused by: java.lang.IllegalArgumentException: Can not create a Path from an empty string  at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163)  at org.apache.hadoop.fs.Path.<init>(Path.java:175)  at org.apache.hadoop.hive.metastore.Warehouse.getDatabasePath(Warehouse.java:172)  at org.apache.hadoop.hive.metastore.Warehouse.getTablePath(Warehouse.java:184)  at org.apache.hadoop.hive.metastore.Warehouse.getFileStatusesForUnpartitionedTable(Warehouse.java:520)  at org.apache.hadoop.hive.metastore.MetaStoreUtils.updateUnpartitionedTableStatsFast(MetaStoreUtils.java:180)  at com.amazonaws.glue.shims.AwsGlueSparkHiveShims.updateTableStatsFast(AwsGlueSparkHiveShims.java:62)  at com.amazonaws.glue.catalog.metastore.GlueMetastoreClientDelegate.alterTable(GlueMetastoreClientDelegate.java:534)  at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.alter_table(AWSCatalogMetastoreClient.java:400)  at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:497)  at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:485)  at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1669)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  at java.lang.reflect.Method.invoke(Method.java:498)  at org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:878)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply$mcV$sp(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:780)  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275)  at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213)  at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212)  at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258)  at org.apache.spark.sql.hive.client.HiveClientImpl.loadTable(HiveClientImpl.scala:779)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply$mcV$sp(HiveExternalCatalog.scala:845)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:843)  at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)  ... 74 moreI got same error trying to execute below:filtered.repartition(1).write.mode("append").insertInto("emrdb.destDir")but I'm able to store data bypassing Glue Data Catalog:filtered.repartition(1).write.format("json").mode("append").save("s3://awenclaw-emr-test/destDir/")This make me thinking the issue is on Data Catalog site.The solution suggested on StackOverflow throws same error:filtered.repartition(1).write.option("path", "s3://awenclaw-emr-test/destDir/").format("hive").mode("append").saveAsTable("emrdb.destDir")So my question is how to correctly store Spark DataFrame into Glue Data Catalog table without all error messages mentioned above?And link to StackOverflow if you need more details on answer I received.https://stackoverflow.com/questions/54441163/writing-spark-dataframe-to-hive-table-through-aws-glue-data-catalougThanks in advance.AndrzejEdited by: awenclaw on Feb 6, 2019 1:25 AMI had a lot of issues like that these days and after dozens of hours investigating, I found the solution:Set up a location URI for the database you are using in Glue Data Catalog UI (in your case for "emrdb").https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-glue.htmlThe documentation talks about this problem but it was not so clear to me."Having a default database without a location URI causes failures when you create a table. As a workaround, use the LOCATION clause to specify a bucket location, such as s3://mybucket, when you use CREATE TABLE. Alternatively create tables within a database other than the default database."CommentSharempierreanswered 3 years ago"


Question: sagemaker custom ml model in container

"2022/11/09 05:21:09 [error] 11#11: *213 upstream prematurely closed connection while reading response header from upstream, client: 169.254.178.2, server: , request: "POST /invocations HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/invocations", host: "169.254.180.2:8080"In this model we provide S3 audio object URL to given output audio S3 URL after processing.nginx.conf is as follows:daemon off; # Prevent forkingpid /tmp/nginx.pid;error_log /var/log/nginx/error.log;events {  # defaults}http {  include /etc/nginx/mime.types;  default_type application/octet-stream;  access_log /var/log/nginx/access.log combined;  client_max_body_size 0;  client_body_timeout 999;  client_header_timeout 999;  keepalive_timeout 999;  fastcgi_buffers 8 16k;  fastcgi_buffer_size 32k;  fastcgi_connect_timeout 999;  fastcgi_send_timeout 999;  fastcgi_read_timeout 999;  upstream gunicorn {    #server 0.0.0.0:4602;    server unix:/tmp/gunicorn.sock fail_timeout=0;  }  server {    listen 8080 deferred;    client_max_body_size 0;    keepalive_timeout 1200s;    proxy_read_timeout 3600s;    location ~ ^/(ping|async-invocations|invocations) {      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;      proxy_set_header Host $http_host;      proxy_redirect off;      proxy_pass http://gunicorn;    }    location / {      return 404 "{}";    }  }}```and serve.py as follows:```pythonimport multiprocessingimport osimport signalimport subprocessimport syscpu_count = multiprocessing.cpu_count()model_server_timeout = os.environ.get('MODEL_SERVER_TIMEOUT', 999)model_server_workers = int(os.environ.get('MODEL_SERVER_WORKERS', cpu_count))def sigterm_handler(nginx_pid, gunicorn_pid):    try:        os.kill(nginx_pid, signal.SIGQUIT)    except OSError:        pass    try:        os.kill(gunicorn_pid, signal.SIGTERM)    except OSError:        pass    sys.exit(0)def start_server():    print('Starting the inference server with {} workers.'.format(model_server_workers))    print(model_server_timeout)    # link the log streams to stdout/err so they will be logged to the container logs    subprocess.check_call(['ln', '-sf', '/dev/stdout', '/var/log/nginx/access.log'])    subprocess.check_call(['ln', '-sf', '/dev/stderr', '/var/log/nginx/error.log'])    nginx = subprocess.Popen(['nginx', '-c', '/opt/program/nginx.conf'])    gunicorn = subprocess.Popen(['gunicorn',                                 '--timeout', str(model_server_timeout),                                 '-k', 'sync',                                 '-b', 'unix:/tmp/gunicorn.sock',                                 '-w', str(model_server_workers),                                 'wsgi:app'])    signal.signal(signal.SIGTERM, lambda a, b: sigterm_handler(nginx.pid, gunicorn.pid))    # If either subprocess exits, so do we.    pids = set([nginx.pid, gunicorn.pid])    while True:        pid, _ = os.wait()        if pid in pids:            break    sigterm_handler(nginx.pid, gunicorn.pid)    print('Inference server exiting')# The main routine just invokes the start function.if __name__ == '__main__':    start_server()FollowComment"

Answer: "0The error means that it's upstream prematurely closed connection, here, Gunicorn. It's not quite related to NGINX.What's the timeout parameter you have configured when start gunicorn , i.e. --timeout option ? By default it's only 30s as per gunicorn doc https://docs.gunicorn.org/en/stable/settings.html#timeout it may change with versions but definitely won't be 3600s.Also, if you are using SageMaker Endpoint, the maximum processing time for each invocation is only 60s. If your algorithm can't answer within 60s, SageMaker, as a client of your NGINX server in your container, will fail this request. There is not something we could configure this hard time out.CommentShareAWS-User-5210721answered 7 months agorePost-User-1658948 7 months agotimeout parameter is 999 and Processing time for model can't be more than 60s because it give this error before 60s. I tried still that errorShare"


Question: NICE DCV Client 2022.1 (r4241) for MacOS has distorted audio

"We are using DCV to connect to g4dn instances running Windows Server 2019. Audio coming from the remote workstations have heavily distorted audio when viewed by MacOS clients with DCV Viewer 2022.1.4241. Downgrading the MacOS DCV Viewer to 2022.0.3973 fixes the issue. We are fine to use 2022.0, but just a heads up for anyone else experiencing this. Hopefully this can be submitted as a bug report.FollowComment"


Question: cloudfront access log to s3  | how to pass pass the user defined headers

"Hi All,I have set up a CloudFront access log to s3, but I am trying to log user-defined headers.  is this possible to log the user-defined custom headers as a CloudFront access log into the s3?for example, if I pass user-defined headers like transaction id and device id, then it be visible to the CloudFront enables s3 log file.kindly suggest. As I tried hards. the below but no successallow the custom header from the behaviorallow the other predefined headers just to see is these can be seen in s3 or not.and many more. but still no success.kindly suggest  will appreciate your cooperationFollowComment"


Question: ECS Anywhere -- Register External Instances link dead

"We've started experimenting with ECS Anywhere and just in the past few days, the register external instances link is dead/inaccessible:curl --proto "https" -o "/tmp/ecs-anywhere-install.sh" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh" returns:This XML file does not appear to have any style information associated with it. The document tree is shown below.<Error>AccessDenied<Message>Access Denied</Message><RequestId>Q8QPSN1GP90X3ZX2</RequestId><HostId>DREyWsYwxRMDC1It6G/ThV7gYa6AAzN5U4lO3kfVAzrf2T5A6XPq1ALRVeE6bDSGhmLk/gI/Jug=</HostId></Error>FollowComment"

Answer: "1Accepted AnswerI have made the team aware.CommentShareEXPERTmreferreanswered a year ago0Same here. The file for windows downloads correctly, but (un)fortunately, it's of no use to me.CommentSharerik2803answered a year ago0It's back!!curl --proto "https" -o "/tmp/ecs-anywhere-install.sh" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh"                                                       <assumerole:ixor.tooling>  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                 Dload  Upload   Total   Spent    Left  Speed100 22282  100 22282    0     0  28370      0 --:--:-- --:--:-- --:--:-- 28530CommentSharerik2803answered a year ago"


Question: Synthetics: Get S3 bucket/key location at runtime

"Is there a way in a CloudWatch Synthetics script (nodejs) to get the S3 bucket and key location for artifacts during runtime? I see log messages like:S3 destination for uploading artifacts determined: {    "s3Bucket": "cw-syn-results-accountid-region",    "s3Key": "canary/region/name-uuid/2021/12/28/time"}Can I get that bucket and key for use in my script, even if it's unofficial and may break in future canary versions?FollowComment"

Answer: "0There is no currently supported way to do that.If you need that information, this currently works (testing using syn-nodejs-puppeteer-3.3):const uploader = require('SyntheticsUploader');await uploader.setUpUploader();const s3Path = 's3://' + uploader.getS3Path();It is completely unsupported, though.CommentShareAWS-User-9784718answered a year ago"


Question: Amazon Polly's voice license

"Hi, I work as a copywriter and I'd like to create some voice posts with TTS. I'd like to know if Polly's synthetic voices have a license that can be extended to the employer (a web site owner in this case), and if Polly's voices can be used to create audio files for third parties. Thank you so much.FollowComment"

Answer: "1Hi,Yes you can use Polly generated file in 3rd party application with no additional costs.You can cache and replay Amazon Polly’s generated speech at no additional cost.Source https://aws.amazon.com/polly/faqs/?nc=sn&loc=8Store & redistribute speechAmazon Polly allows for unlimited replays of generated speech without any additional fees. You can create speech files in standard formats like MP3 and OGG, and serve them from the cloud or locally with apps or devices for offline playback.Source https://aws.amazon.com/polly/?nc=sn&loc=1Therefore you are good.ThxDanCommentShareDanBloyanswered a year agoLBajard a year agoWonderful, thank you very much Dan!Share"


Question: Can we use a conda environment on AWS Glue-ETL?

"I have a python file that runs on a conda environment (i.e, qiime2 which is a conda env installed from "https://data.qiime2.org/distro/core/qiime2-2021.4-py38-linux-conda.yml")Is there a way to setup a conda environment on AWS Glue-ETL before running a job using my python file?Is seems it is possible to provide pip-installable python modules or whl files in the "Python library path" field, but I cannot find how to set up the running-environment using yml file.FollowComment"

Answer: "0Hi,AWS Glue lets you install additional Python modules and libraries for use with AWS Glue ETL. If you want to install Python modules that aren’t included out of the box, please follow the steps lined out in the documentation here:https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.htmlThanks.CommentShareNick Furranswered 10 months agorePost-User-9256543 10 months agoHi, thanks for your answer.I am aware of that python modules can be installed by assigning "Python library path" as described in the document you mentioned. However, conda environment also includes "non-python" library dependencies.I wonder if a conda environment including "non-python" library (e.g., https://data.qiime2.org/distro/core/qiime2-2021.4-py38-linux-conda.yml) can be set up in AWS Glue.Share"


Question: do not connect to domain  lncomputers.online with   44.198.26.218:8080

"i am using  tomcat 10 as server .In browser  44.198.26.218:8080 is working properly. opens my index.jsp file. already configured security group  as    {  ssh  22  anywhere   } {  custom tcp   8080  anywhere } .  configured route 53 with lncomputers.online .   with  type   {  A   lncomputers    44.198.26.218 }    {  CNAME   www.lncomputers.online      lncomputers.online  }  ]  pasted all  four NS to my   hostinger  domain as custom domain.    1 day ago  .  but when i type lncomputers.online in browser still shows error as  this site cant be reached........................plzzzzz help me . i am very disappoint from this.  *****@gmail.com  is my mail id , you can  connect to me  through this mail alsoFollowCommentBrettski-AWS EXPERT8 months agoThis is a public forum; I would not recommend putting your personal contact details on it.ShareChris_G EXPERT8 months agoUpdated your question and removed your emailShare"

Answer: 1Accepted AnswerElaborating on @Brettski-AWS comment:You have configured your server to listen on port 8080. When accessing via the domain name lncomputers.online & www.lncomputers.online you are sending traffic to port 80 and not 8080. You need to access it as lncomputers.online:8080 or configure some redirection from 80 to 8080 on the server or configure tomcat to listen on port 80. And when you configure it for port 80 you'll also need to allow inbound traffic for port 80 in the Security groupCommentShareSydanswered 8 months ago1I've just tested that domain name and the CNAME (www) on port 8080 using curl and I get a response back. DNS looks ok to me. Server looks to be running. I think it's operating as planned.CommentShareEXPERTBrettski-AWSanswered 8 months ago


Question: Amazon Connect "Get Customer Input" block exits with "Error" branch for last Lex slot question (custom data type) contains unknown slot value (Fallback transcription) handled via Lambda

"All, this one has really stumped me all week... I will do my best to try and describe the behavior as clearly as possibly can...We use Amazon Connect Contact Flows with the widget "Get Customer Input" to route a caller through an automated Lex V2 bot experience. We are gathering information for our firm about a potential client's needs. In Contact Flow "Get Customer Input" we exit only on two paths: Default and Error (then from Default, we check session attributes and determine next steps in the Flow). All this is working as desired.The part that bombs out on us, is the situation outlined below:At the end of every different matter intent a caller could choose from and trigger, we have a custom data type that captures a lengthy user issue description (for a caller describing their particular legal case like: tell me about your injuries, or what's the main issue you are looking to address, etc. etc.)The point is, this custom data type is a generic "catch up" Speech-to-Text ONLY use case. We aren't actually trying to divulge meaningful attributes from within for this slot. It's essentially a bit like a voice mail recording message. So the answers are very open-ended by the users and extremely varied. We totally realize and appreciate this is not how a slot is usually built...So, it's very hard to train data for this custom slot type, because it's a bit like a mini-story the user tells, and it could be anything. We've tried to train with literally hundreds of slot utterances, and still we experience lots of Fallback (which we expect and handle). We've spoken with AWS Lex SAs before and they thought if we trained on 200-300 phrases, we should be able to catch everything. But we still get a high rate of answers that fall through the cracks...So we DON'T want to validate the answer against any trained slot type utterance data--we just want to save the inputTranscript to the slot, and move on.We use Lambda codehook on every conversation turn, and check for fallback, and if fallback occurs, we literally take the inputTranscript Speech-to-Text and manually save it to the slot.This works great when we have custom data type questions that are not the last question in the slots list. It totally works as expected: senses the Fallback, takes the inputTranscript, saves it, and moves on by explicitly calling DialogAction type = ElicitSlot for the next question.HOWEVER, on the last question which is custom data type, when Invocation changes from DialogCodeHook to FulfillmentCodeHook, we again save the last value, and we call DialogAction type = Close.** If a fallback value occurs here, neither Lambda, nor Lex Conversations logs record any error. BUT the Amazon Connect "Get Customer Input" goes down the "Error" branch instead of the "Default" branch.** NOTE: If we give a "clean" answer (one of the test trained answers) to this last question, the FulfillmentCodeHook logic for DialogAction type = Close works just great, no problems...The payload of intent that I pass to DialogAction Close for either the working or breaking case is exactly the same -- with only the last question slot value being different, of course.No error is showing up in my Lambda. No error in the Lex conversation log. But the Amazon Connect fails hard on "Error" path coming out of the Get Customer Input widget, and there's literally no more info in CloudWatch from the Connect side that shows "why" it hit the error branch.I've tried everything I can think of; does anyone else have any ideas or seen similar behavior on last question custom data type fallback issues?One last note: I get this behavior in either the pre-August 17 and post-August 17 release of Lex V2 bots.Thanks all,JeremyFollowComment"

Answer: "1Accepted AnswerAmazon Lex now has a FreeFormInput Slot Type available that might be a good thing to try. The built-in can recognize strings made up of words or characters. The resolved value is the complete free form entry provided by the user.https://docs.aws.amazon.com/lexv2/latest/dg/built-in-slot-free-form.htmlCommentShareThomas_Ranswered 7 months agoEXPERTAWS-samfredreviewed 2 months agoJeremy 7 months agoThank you Thomas!! We just heard this release yesterday and we are VERY excited and hopeful to test this out now. Will update for everyone later.Share0Y’ALL…. AMAZON.FreeFormInput is my new JAM!!This is the data type I’ve been waiting for since the beginning of our bot building project. When you just need only full “any string” capture (and you don’t care or need or want a check against pre-trained utterances), this is your new best friend in Lex. Thank you Lex team, thanks so much Thomas_R for the post. I’m stoked!CommentShareJeremyanswered 7 months agoJeremy 7 months agoAnd forgot to mention—because this data type accepts ANYTHING, we don’t fail  and hit the strange error case on the ending slot any longer that I outlined above. Awesome!Share0I just don't see how you're going to be able to overcome this. Ultimately seems like you're using the wrong tool for the job. Can you change your IVR to use the media streaming for the last question?davidCommentSharedmaciasanswered 8 months ago"


Question: RDS Proxy for PostgreSQL major version 14

"Hi AWS TeamWhen do you guys think we can get to use RDS Proxy for PostgeSQL major version 14. I know support was added for major version 13 early thia year, so any launch plans in the upcoming weeks or months for 14?FollowComment"

Answer: 1Hello! Amazon RDS Proxy now supports PostgreSQL major version 14 - https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-rds-proxy-supports-postgresql-major-version-14/CommentSharePriyanka_Sanghvianswered 4 months ago


Question: Can one use the SDK call to receive events when Kinesis stream changes number of shards?

"Hello;Using the SubscribeToShard SDK call, one can receive events via HTTP when records are added to the shard.  And other SDK calls allow one to determine what shards are in a Kinesis stream.Is there a way to receive events via HTTP when a shard is added to or removed from a stream, so that a program can know to add or remove a consumer for that shard?  Or is it necessary to regularly call the ListShards API to determine what shards are in the stream?Thank you.FollowCommentNico_S 2 months agoHello,Using ListShards is one way of subscribing to new shards after resharding operations take place and you could regularly check with the API if new shards are available to do so.One thing to note is that the SubscribeToShard API also returns the list of the child shards of the current shard when the end of the current shard is reached, which means that you are able to see the newly split or merged shard ID's which you can then make use of to Subscribe to them.If event.ContinuationSequenceNumber returns null, it indicates that a shard split or merge has occurred that involved this shard. This shard is now in a CLOSED state, and you have read all available data records from this shard. In this scenario, per example above, you can use event.childShards in your code logic to subscribe to the new shards.For more information you are welcome to have a look at the complete SDK examples that can be found here: https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/kinesis/src/main/java/com/example/kinesis/KinesisStreamEx.javaYou can also reference the following documentation: https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.htmlShare"


Question: New Gem doesn't have LmbrCentral Option in #include?

"Hi,I've made a new gem and am moving my c++ files to it so that I can reuse it in multiple projects. I changed the namespaces etc. to fit, added it to the waf files etc. but when building get this error:        Build failed<++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>Compilation failed - File: 'RigidBodyOscillatorComponent.cpp.144.obj', Module: 'Playground', Configuration: 'win_x64_vs2015|profile', error code 2Input Files: 'd:\Repos\UES-lumberyard\dev\Gems\Playground\Code\Source\Components\RigidBodyOscillatorComponent.cpp'Output Files: 'd:\Repos\UES-lumberyard\dev\BinTemp\win_x64_vs2015_profile\Gems\Playground\Code\Source\Components\RigidBodyOscillatorComponent.cpp.144.obj'Command: 'D:\VS2015\VC\BIN\amd64\CL.exe /nologo /W3 /WX /MP /Gy /GF /Gm- /fp:fast /Zc:wchar_t /Zc:forScope /Gd /wd4530 /wd4351 /Ox /Ob2 /Ot /Oi /Oy- /FS /Wv:18 /bigobj /showIncludes /MD /Z7 /YuPlayground_precompiled.h /Fpd:\Repos\UES-lumberyard\dev\BinTemp\win_x64_vs2015_profile\Gems\Playground\Code\Source\Playground_precompiled.144.pch /GR- /Id:\Repos\UES-lumberyard\dev\BinTemp\win_x64_vs2015_profile\Gems\Playground\Code /Id:\Repos\UES-lumberyard\dev\Gems\Playground\Code /Id:\Repos\UES-lumberyard\dev\Code\SDKs\boost /Id:\Repos\UES-lumberyard\dev\Code\CryEngine\CryCommon /Id:\Repos\UES-lumberyard\dev\Gems\Playground\Code\Include /Id:\Repos\UES-lumberyard\dev\Gems\Playground\Code\Source /Id:\Repos\UES-lumberyard\dev\Code\CryEngine\CryAction /Id:\Repos\UES-lumberyard\dev\BinTemp\win_x64_vs2015_profile\Code\Framework\AzFramework /Id:\Repos\UES-lumberyard\dev\Code\Framework\AzFramework /Id:\Repos\UES-lumberyard\dev\BinTemp\win_x64_vs2015_profile\Code\Framework\GridMate /Id:\Repos\UES-lumberyard\dev\Code\Framework\GridMate /Id:\Repos\UES-lumberyard\dev\BinTemp\win_x64_vs2015_profile\Code\Framework\AzCore /Id:\Repos\UES-lumberyard\dev\Code\Framework\AzCore /ID:\VS2015\VC\INCLUDE /ID:\VS2015\VC\ATLMFC\INCLUDE /IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt /IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um /IC:\Program Files (x86)\Windows Kits\8.1\include\shared /IC:\Program Files (x86)\Windows Kits\8.1\include\um /IC:\Program Files (x86)\Windows Kits\8.1\include\winrt /Id:\Repos\UES-lumberyard\dev\Code\SDKs\rapidjson\include /Id:\Repos\UES-lumberyard\dev\Code\SDKs\rapidxml /Id:\Repos\UES-lumberyard\dev\Code\SDKs\Lua /Id:\Repos\UES-lumberyard\dev\Code\SDKs\zlib-common\include /D_WIN32 /D_WIN64 /DCODE_BASE_FOLDER="d:/Repos/UES-lumberyard/dev/Code/" /D_PROFILE /DPROFILE /DNDEBUG /DSCRIPTCANVAS_ERRORS_ENABLED /DBINFOLDER_NAME="Bin64vc140" /DPLATFORM_SUPPORTS_AWS_NATIVE_SDK /DUSE_WINDOWS_DLL_SEMANTICS /DLY_BUILD=526767 /D_MT /D_DLL /DAZ_ENABLE_TRACING /DAZ_ENABLE_DEBUG_TOOLS /DEXE_VERSION_INFO_0=1 /DEXE_VERSION_INFO_1=12 /DEXE_VERSION_INFO_2=0 /DEXE_VERSION_INFO_3=0 d:\Repos\UES-lumberyard\dev\Gems\Playground\Code\Source\Components\RigidBodyOscillatorComponent.cpp /c /Fo d:\Repos\UES-lumberyard\dev\BinTemp\win_x64_vs2015_profile\Gems\Playground\Code\Source\Components\RigidBodyOscillatorComponent.cpp.144.obj'Output:RigidBodyOscillatorComponent.cppd:\Repos\UES-lumberyard\dev\Gems\Playground\Code\Source\Components\RigidBodyOscillatorComponent.cpp(9): fatal error C1083: Cannot open include file: 'LmbrCentral\Physics\PhysicsComponentBus.h': No such file or directory<++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>It appears that the lmbrCentral thing is not available as a #include, and when I try to add it manually by manually typing out the #include <Lmbr....> it comes up with LmbrAWS, but not LmbrCentral. It works perfectly fine in the original c++ files in the project itself.Thanks in advance for any helpFollow"

Answer: "0Accepted AnswerDoing:#include <..\..\LmbrCentral\Code\include\LmbrCentral\Physics\PhysicsComponentBus.h>works, but is a messy solution. Any better fixes?SharerePost-User-8843326answered 5 years ago0I tried to add the test1 project as a dependency to my playground gem like this:    {"GemFormatVersion": 4,"Uuid": "9e1bf6a6cbb3432cb108c9e812a3e55a","Name": "Playground","DisplayName": "Playground","Version": "0.1.0","Summary": "All the code and assets we share between projects","Tags": ["Untagged"],"IconPath": "preview.png","Modules": [{"Type": "GameModule"}],"Dependencies": [{"Uuid": "4f2993c739444e33a766cc3d64efb8a1", //the uuid of the test1 project gem"VersionConstraints": [">=0.1"],"_comment": "Test1"}]}But it doesn't work :(SharerePost-User-8843326answered 5 years ago0My gem worked fine when I added LmbrCentral dependencies to the gem.JSON file{"Dependencies": [    {"Uuid": "ff06785f7145416b9d46fde39098cb0c",      "VersionConstraints": [">=0.1"],  "_comment": "LumbrCentral"}],"GemFormatVersion": 4,"Uuid": "7cf95b92295a4076b0093befaa36e7fd","Name": "Hands","DisplayName": "Hands","Version": "0.1.0","Summary": "Hands Gem.","Tags": ["Untagged"],"IconPath": "preview.png",    "Modules": [{"Type": "GameModule"} ]}SharerePost-User-4778853answered 5 years ago"


Question: Unable to run GlueStudio Jobs & getting aws sdk java.lang.NoSuchMethodError

"2023-03-10 03:52:55,572 ERROR [stream execution thread for [id = bea7911a-d787-44e1-836b-8488a43e16ea, runId = ba5fb563-91a6-4d39-befb-f740a9be9a08]] streaming.MicroBatchExecution (Logging.scala:logError(94)): Query [id = bea7911a-d787-44e1-836b-8488a43e16ea, runId = ba5fb563-91a6-4d39-befb-f740a9be9a08] terminated with errorjava.lang.NoSuchMethodError: software.amazon.awssdk.utils.SystemSetting.getStringValueFromEnvironmentVariable(Ljava/lang/String;)Ljava/util/Optional;at software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor.lambdaFunctionNameEnvironmentVariable(TraceIdExecutionInterceptor.java:63)at software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor.modifyHttpRequest(TraceIdExecutionInterceptor.java:40)at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyHttpRequestAndHttpContent(ExecutionInterceptorChain.java:90)at software.amazon.awssdk.core.internal.handler.BaseClientHandler.runModifyHttpRequestAndHttpContentInterceptors(BaseClientHandler.java:151)at software.amazon.awssdk.core.internal.handler.BaseClientHandler.finalizeSdkHttpFullRequest(BaseClientHandler.java:78)at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:151)at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:175)at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)at software.amazon.awssdk.services.glue.DefaultGlueClient.getTable(DefaultGlueClient.java:7227)at org.apache.iceberg.aws.glue.GlueTableOperations.getGlueTable(GlueTableOperations.java:185)at org.apache.iceberg.aws.glue.GlueTableOperations.doRefresh(GlueTableOperations.java:103)at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:96)at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:79)at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:44)at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406)at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404)at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387)at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)at org.apache.iceberg.CachingCatalog.loadTable(CachingCatalog.java:161)at org.apache.iceberg.spark.SparkCatalog.load(SparkCatalog.java:526)at org.apache.iceberg.spark.SparkCatalog.loadTable(SparkCatalog.java:141)at org.apache.iceberg.spark.SparkCatalog.loadTable(SparkCatalog.java:98)at org.apache.spark.sql.connector.catalog.TableCatalog.tableExists(TableCatalog.java:119)at org.apache.spark.sql.execution.datasources.v2.AtomicCreateTableAsSelectExec.run(WriteToDataSourceV2Exec.scala:108)at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:40)at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:40)at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:46)at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:229)at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3724)at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3722)at org.apache.spark.sql.Dataset.<init>(Dataset.scala:229)at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:615)at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650)at SparkSqlQuery$.execute(sync_cj_data_shopify_product_updates_stream.py:24)at GlueApp$.$anonfun$main$1(sync_cj_data_shopify_product_updates_stream.py:48)at GlueApp$.$anonfun$main$1$adapted(sync_cj_data_shopify_product_updates_stream.py:40)at com.amazonaws.services.glue.GlueContext.$anonfun$forEachBatch$1(GlueContext.scala:1366)at com.amazonaws.services.glue.GlueContext.$anonfun$forEachBatch$1$adapted(GlueContext.scala:1347)at org.apache.spark.sql.execution.streaming.sources.ForeachBatchSink.addBatch(ForeachBatchSink.scala:35)at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:586)at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$15(MicroBatchExecution.scala:584)at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:584)at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:226)at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:333)at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)Glue version3.0We use the Apache Iceberg Connector for Glue 3.0 to read data from Kinesis, and write data to Glue Catalog.Job Generated Script :import com.amazonaws.services.glue.GlueContextimport com.amazonaws.services.glue.MappingSpecimport com.amazonaws.services.glue.errors.CallSiteimport com.amazonaws.services.glue.util.GlueArgParserimport com.amazonaws.services.glue.util.Jobimport com.amazonaws.services.glue.util.JsonOptionsimport org.apache.spark.SparkContextimport scala.collection.JavaConverters._import com.amazonaws.services.glue.DynamicFrameimport org.apache.spark.sql.Datasetimport org.apache.spark.sql.Rowimport org.apache.spark.sql.SaveModeimport org.apache.spark.sql.streaming.Triggerimport java.util.Calendarimport com.amazonaws.services.glue.DynamicFrameimport org.apache.spark.sql.SQLContextimport com.amazonaws.services.glue.errors.CallSiteobject SparkSqlQuery {  def execute(glueContext: GlueContext, sqlContext: SQLContext, query: String, mapping: Map[String, DynamicFrame]) : DynamicFrame = {    for ((alias, frame) <- mapping) {      frame.toDF().createOrReplaceTempView(alias)    }    val resultDataFrame = sqlContext.sql(query)    return DynamicFrame(resultDataFrame, glueContext)  }}object GlueApp {  def main(sysArgs: Array[String]) {    val spark: SparkContext = new SparkContext()    val glueContext: GlueContext = new GlueContext(spark)    val sql: SQLContext = new SQLContext(spark)    // @params: [JOB_NAME]    val args = GlueArgParser.getResolvedOptions(sysArgs, Seq("JOB_NAME").toArray)    Job.init(args("JOB_NAME"), glueContext, args.asJava)    // Script generated for node Kinesis Stream    val dataframe_KinesisStream_node1 = glueContext.getSource(connectionType="kinesis",connectionOptions={JsonOptions("""{"typeOfData": "kinesis", "streamARN": "arn:aws:kinesis:us-west-2:133981716884:stream/shopify_product_updates_stream", "classification": "json", "startingPosition": "earliest", "inferSchema": "true"}""")}, transformationContext="dataframe_KinesisStream_node1").getDataFrame()    glueContext.forEachBatch(dataframe_KinesisStream_node1, (dataFrame: Dataset[Row], batchId: Long) => {      if (dataFrame.count() > 0) {        val KinesisStream_node1 = DynamicFrame(dataFrame, glueContext)        // Script generated for node INSERT INTO        val SqlQuery0: String = """INSERT INTO my_catalog.cj_data.shopify_product_updates_stream SELECT * FROM myDataSource;        ||""".stripMargin        val INSERTINTO_node1678417695303 = SparkSqlQuery.execute(glueContext = glueContext, sqlContext = sql, query = SqlQuery0, mapping = Map("myDataSource" -> KinesisStream_node1)        )        // Script generated for node CREATE TABLE        val SqlQuery1: String = """CREATE TABLE IF NOT EXISTS my_catalog.cj_data.shopify_product_updates_stream AS         ||(SELECT * FROM myDataSource LIMIT 0);        ||""".stripMargin        val CREATETABLE_node1678417075933 = SparkSqlQuery.execute(glueContext = glueContext, sqlContext = sql, query = SqlQuery1, mapping = Map("myDataSource" -> KinesisStream_node1)        )        // Script generated for node NO ROW SQL        val SqlQuery2: String = """select * from my_catalog.cj_data.shopify_product_updates_stream         ||limit 0;        ||""".stripMargin        val NOROWSQL_node1678417543220 = SparkSqlQuery.execute(glueContext = glueContext, sqlContext = sql, query = SqlQuery2, mapping = Map("myDataSource" -> CREATETABLE_node1678417075933, "myDataSource2" -> INSERTINTO_node1678417695303)        )        // Script generated for node Apache Iceberg Connector for Glue 3.0        val ApacheIcebergConnectorforGlue30_node3 = glueContext.getSink(connectionType="marketplace.spark", connectionOptions=JsonOptions("""{"path": "my_catalog.cj_data.shopify_product_updates_stream", "connectionName": "my-iceberg-conn"}"""), transformationContext="ApacheIcebergConnectorforGlue30_node3").writeDynamicFrame(NOROWSQL_node1678417543220)      }    }, JsonOptions(s"""{"windowSize" : "100 seconds", "checkpointLocation" : "${args("TempDir")}/${args("JOB_NAME")}/checkpoint/"}"""))    Job.commit()  }}FollowCommentGonzalo Herreros 3 months agoAre you running a type "Streaming" job? Have you tried using the Iceberg support provided out of the box instead of the connector? https://docs.aws.amazon.com/glue/latest/ug/gs-data-lake-formats-delta.htmlShare"


Question: IoT Tunneling LocalProxy Raspberry support

"Hi,I am looking to try out IoT Secure Tunneling to see if it fits my needs.I am trying to build the files for the localproxy on a raspberry device (raspbian stretch / buster) but have a lot of issues with the packages (mostly with boost).I have 2 questions -Is there a pre-built version for raspbian, specifically for stretch?If not, how can I skip building the tests (as it seems most issues Im having are there) ?And last question - is it possible to use Secure tunneling with any other proxy client ?Thank youEdited by: EcoMan on Oct 13, 2020 8:43 AMFollowComment"

Answer: "0Hi,We don't have a pre-built version for raspbian. We only offer the source code of the local proxy and customer can build in the platform of their choice. I encourage you to try the docker image https://github.com/aws-samples/aws-iot-securetunneling-localproxy#building-the-local-proxy-via-docker , which eliminates the trouble for building for specific platform and provide a general application isolation. Thanks for the idea and the feedback though, our team will look into making it available natively on Raspbian.You can modify the cmake file to not build the binary of localproxytest https://github.com/aws-samples/aws-iot-securetunneling-localproxy/blob/master/CMakeLists.txt#L5. localproxytest binary is suppose to run before localproxy and helps to detect if your platform is building and running successfully. If you have problem with localproxytest building, it might also indicate localproxy may not build/run successfully.We're not exactly sure what alternative proxy client you are referring to. But HTTP proxy is currently not supported. You can refer here https://github.com/aws-samples/aws-iot-securetunneling-localproxy/issues/4 for more details. There isn't a different local proxy implementation currently available to try at this time.ShantanuSAWSCommentShareShantanuSAWSanswered 3 years ago"


Question: Origin Group getting 403 on 404 failover

"Hi,I have set up two different distributions as so:Distribution A: s3 originDistribution B: custom originWhen I attempt to GET a known resource from distro A, everything works and a 200 OK is returned. When I attempt to GET a known resource from distro B, everything works and a 200 OK is returned. So, everything is fine with an s3 origin and a custom origin.Now, the resource on each does not exist on the other distros. In other words, the resource on distro A is not in distro B and the resource in distro B is not in distro A.I have then set up a third distribution that has both of these origins configured and an origin group is set up with each origin and the failover criteria of 404 are set. The s3 origin is priority 1 and the custom origin is priority 2. So, looks like this:Distribution C: origin grouppriority 1: s3 originpriority 2: custom originfailover criteria: 404 Not FoundWhen I perform a GET request for a known resource on distro C the priority 1 origin (s3 origin), a 200 OK is returned. When I perform a GET request that I know will 404 (because it is on the custom origin), I get a 403 Forbidden response.Is anyone able to explain to me what is happening here, am I missing a configuration setting?FollowComment"

Answer: "0I just want to give an update. I was able to figure out the problem. It seems that the 404 failover was not taking place. In the response from origin group it had a 403 forbidden reply from "AmasonS3". Digging into that I found this nugget of information:"If a user doesn’t have s3:ListBucket permissions, then the user gets Access Denied errors for missing objects instead of 404 Not Found errors. Run the head-object AWS CLI command to check if an object exists in the bucket."I was able to test that the 403 was from the S3 origin and not failing over by setting an additional origin group failover criteria to include a 403 response. This initially routed to my custom origin.However, to ensure the correct reason for failing over was due to 404 and not a misconfigured permission, I added the s3:ListBucket permission , removed the 403 criteria, and the failover from s3 origin to custom origin on a 404 works correclty.TL;DR -- Make sure CF has not only s3:GetObject permissions on the S3 origin bucket, but also s3:ListBucket permissions.CommentSharefortycreekanswered 3 years ago"


Question: Is there a way to can create an AWS Quicksights dashboard with more than 20 sheets/reports?

I have a dashboard published(with 20 sheets currently) that is used by more than 250 users every day. I wanted to add some more analysis reports/sheets to this existing dashboard.FollowComment

Answer: "0Hello,Each analysis can contain 30 (default) to 50 sheets. Refer [1]You can share analyses and publish dashboards with multiple sheets. However, scheduled email reports only include the top sheet in an analysis.Use the following list of actions to work with sheets:To add a new sheet, choose the plus-sign (+) to the right of the sheet tabs.To rename a sheet, choose the name of the sheet and start typing. Rename is also available from the sheet menu (  ).To duplicate a sheet, choose the name of the sheet, then choose Duplicate from the sheet menu (  ). You can only duplicate a sheet if Autosave is turned on.To delete a sheet, choose the name of the sheet, then choose Delete from the sheet menu (  ). You can't delete the sheet if it's the only sheet in the analysis.To change the order of the sheets, choose the name of the sheet and drag it to a new position.To copy a visual to a new sheet, choose Duplicate visual to from the on-visual menu. Then choose the target sheet. Filters exist only on the sheet that you create them on. To duplicate filters, recreate them on the target sheet.You can use the parameter controls on the top sheet to control multiple sheets. To do this, open each sheet that you want to work with the parameter. Then add a filter that uses the same parameter used in the control on the top sheet. Or, if you want a new sheet to operate independently, you can add parameters and parameter controls to it that are separate from those on the top sheet.Link- https://docs.aws.amazon.com/quicksight/latest/user/working-with-multiple-sheets.html -- [1]CommentShareSUPPORT ENGINEERAWS-User-Chiraganswered 10 months agoEXPERTFabrizio@AWSreviewed 10 months ago"


Question: Internal Server Error from OpenSearch Dashboard after integrating with SAML (KeyCloak)

"After integrating with KeyCloak, SAML post is giving 500 response.  Nothing in logs. What is the best way to debug/log the error?FollowComment"

Answer: "0Hello,I have exactly the same problem, do you have find a way to get this works ?Currently I'm trying to map my user against the group "all_access" but I have no other error than this.An example of my assertion:<samlp:Response xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol"                xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion"                Destination="https://xxxxxx/_dashboards/_opendistro/_security/saml/acs"                ID="ID_f2e7e757-2c9c-4904-8336-ddd809a6d4b0"                InResponseTo="ONELOGIN_c48844cf-c1b3-47e1-9242-ffe508c269b8"                IssueInstant="2022-07-15T07:08:21.365Z"                Version="2.0"                >    <saml:Issuer>https://xxxxx/auth/realms/xxxxx</saml:Issuer>    <dsig:Signature xmlns:dsig="http://www.w3.org/2000/09/xmldsig#">        <dsig:SignedInfo>            <dsig:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />            <dsig:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256" />            <dsig:Reference URI="#ID_f2e7e757-2c9c-4904-8336-ddd809a6d4b0">                <dsig:Transforms>                    <dsig:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />                    <dsig:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />                </dsig:Transforms>                <dsig:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256" />                <dsig:DigestValue>xxxxx</dsig:DigestValue>            </dsig:Reference>        </dsig:SignedInfo>        <dsig:SignatureValue>xxxxx</dsig:SignatureValue>        <dsig:KeyInfo>            <dsig:KeyName>xxxx</dsig:KeyName>            <dsig:X509Data>                <dsig:X509Certificate>xxxxxxxx</dsig:X509Certificate>            </dsig:X509Data>            <dsig:KeyValue>                <dsig:RSAKeyValue>                    <dsig:Modulus>xxxxxxx</dsig:Modulus>                    <dsig:Exponent>AQAB</dsig:Exponent>                </dsig:RSAKeyValue>            </dsig:KeyValue>        </dsig:KeyInfo>    </dsig:Signature>    <samlp:Status>        <samlp:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success" />    </samlp:Status>    <saml:Assertion xmlns="urn:oasis:names:tc:SAML:2.0:assertion"                    ID="ID_5d953166-ef46-48c5-bd54-a9adfce95fe0"                    IssueInstant="2022-07-15T07:08:21.365Z"                    Version="2.0"                    >        <saml:Issuer>https://xxxx/auth/realms/xxxx</saml:Issuer>        <saml:Subject>            <saml:NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified">xxxx@xxxx.com</saml:NameID>            <saml:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">                <saml:SubjectConfirmationData InResponseTo="ONELOGIN_c48844cf-c1b3-47e1-9242-ffe508c269b8"                                              NotOnOrAfter="2022-07-15T07:13:19.365Z"                                              Recipient="https://xxxxx/_dashboards/_opendistro/_security/saml/acs"                                              />            </saml:SubjectConfirmation>        </saml:Subject>        <saml:Conditions NotBefore="2022-07-15T07:08:19.365Z"                         NotOnOrAfter="2022-07-15T07:09:19.365Z"                         >            <saml:AudienceRestriction>                <saml:Audience>https://xxxx</saml:Audience>            </saml:AudienceRestriction>        </saml:Conditions>        <saml:AuthnStatement AuthnInstant="2022-07-15T07:08:21.365Z"                             SessionIndex="3256b6e5-fdf3-492d-8089-038c06b7fa3c::43aacbea-219e-4210-a258-ccca424f990f"                             SessionNotOnOrAfter="2022-07-15T17:08:21.365Z"                             >            <saml:AuthnContext>                <saml:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:unspecified</saml:AuthnContextClassRef>            </saml:AuthnContext>        </saml:AuthnStatement>        <saml:AttributeStatement>            <saml:Attribute Name="groups"                            NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"                            >                <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"                                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"                                     xsi:type="xs:string"                                     >all_access</saml:AttributeValue>            </saml:Attribute>            <saml:Attribute Name="email"                            NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"                            >                <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"                                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"                                     xsi:type="xs:string"                                     >xxxx@xxx.xxx</saml:AttributeValue>            </saml:Attribute>            <saml:Attribute FriendlyName="audience"                            NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"                            >                <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"                                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"                                     xsi:type="xs:string"                                     >https://xxxxx</saml:AttributeValue>            </saml:Attribute>        </saml:AttributeStatement>    </saml:Assertion></samlp:Response>CommentSharerePost-User-3668023answered a year ago"


Question: SSM Network firewall audit

"Hi all, I'm using SSM on some hybrid linux nodes. I was going through the documentation and there is a mention of being able to use SSM to check open network ports on the nodes but there isn't any example of how to do it. I'm trying to setup a proof of concept right now and if there is a tutorial on how to do that that would be awesome, and would help my case. Are there any available resources?FollowComment"

Answer: "0Hello!Usually we would suggest using AWS Firewall Manager as this is the best way to have a single view for your Security Groups and enforce a baseline policy across applications/ many instances.This is described in detail under this documentation: https://aws.amazon.com/blogs/security/how-to-continuously-audit-and-limit-security-groups-with-aws-firewall-manager/https://aws.amazon.com/firewall-manager/AWS Firewall Manager may be more suitable for what you are trying to accomplish.For your questions regarding this under SSM, I was not able to find the documentation which you are referring to. Could I please ask that you include a link here so that I can check on that for you?CommentShareSUPPORT ENGINEERJasper_Ranswered a year ago0Hi!If you're looking to see if the security groups on the instances allow for overly permissive access (such as wide open 0/0), you could use Trusted Advisor: https://docs.aws.amazon.com/awssupport/latest/user/trusted-advisor-check-reference.html#security-groups-specific-ports-unrestricted.  Trusted Advisor can be displayed in Systems Manager: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-trusted-advisor-and-phd.html.If you're looking for something else, you could also use Systems Manager OpsCenter to configure alerts based off AWS Security Hub events: https://docs.aws.amazon.com/systems-manager/latest/userguide/OpsCenter.html.AWS Firewall Manager can offer more network security checks, but requires integration with more services (Organizations, possibly Network Firewall) and can be pricier as well.And lastly, AWS Config offers the ability to check resources and their configuration as well: https://docs.aws.amazon.com/config/latest/developerguide/managed-rules-by-aws-config.html.  It's possible to create your own custom Config rules too.CommentSharejsoncanswered a year ago"


Question: AWSAmplify DataStore iOS: Populating relationships

"Hi,I am trying to implement offline app using Datastore in iOS. I was going through this guide saving-relationsIn the example if we want to create a Comment and associate to a Post we have to supply the entire Post object. Is there a possibility to pass in just the postID and the comment then gets associated with that Post?  I tried this by adding (fields: ["postID"]) but the comments never gets populated in the Post object. Whats the correct way to model this?Objective is to just pass in the postId for creating and associating a comment to the post and make sure when querying the post, I get the comments field populated.FollowCommentanoop4real a year agoAny inputs?Share"


Question: AWS VPN Client Federated users

"Hi all,i have an issue with my client vpn and federated users.When i open the connection using the aws vpn client i get forwarded correctly to my company login page, but then after login i get a 403 error.The user has admin priveleges and can acces all vpc/vpn settingsin the vpn i allow traffic from all usersso i dont realy find whats wrong.FollowComment"


Question: "Amazon Connect - "You've reached the limit of Phone Numbers. To increase limit, contact support""

"Hi AWS Support, I got locked out of my Amazon Connect instance so I deleted it. I created a new instance, but now I CANNOT Claim a phone number. I get a message "You've reached the limit of Phone Numbers. To increase limit, contact support" - see error attached as screenshot.NOTE: I deleted the previous Amazon Connect instance so I do NOT have access to the original Phone Number DID that I claimed.Please let me know how I can claim phone number and continue using Amazon Connect?Thank you.ArtFollowComment"

Answer: "0Hi,as the error suggests, contact Support so they can solve the issue. This is also mentioned in this guide (check Phone numbers per instance section)https://docs.aws.amazon.com/connect/latest/adminguide/amazon-connect-service-limits.htmlAlso there are certain implications when deleting a Connect instance and its phone number, as pointed out in this post: https://repost.aws/questions/QU75xdZbLARqiEPvAYL0Nkow/amazon-connect-restoring-phone-numberHope it clarifies.CommentShareEXPERTalatechanswered 2 months agoEXPERTAWS-samfredreviewed 2 months ago"


Question: Upgrade LZ to v2.4.4 - Java 1.8.0 error

"Trying to upgrade LZ to V2.4.4 but getting stuck in the first step of the codebuild (container configuration).It seems that is using an OS image that is crashing due packages during the process. At the moment apt-get upgrade is trying to install a newer version of java JDK, which's requiring a manual interaction not mapped (potentially causing the error bellow):Setting up java-1.8.0-amazon-corretto-jdk:amd64 (1:8.332.08-1) ...Configuration file '/usr/lib/jvm/java-1.8.0-amazon-corretto/jre/lib/security/policy/limited/local_policy.jar' ==> File on system created by you or by a script. ==> File also in package provided by package maintainer.   What would you like to do about it ?  Your options are:    Y or I  : install the package maintainer's version    N or O  : keep your currently-installed version      D     : show the differences between the versions      Z     : start a shell to examine the situation The default action is to keep your current version.*** local_policy.jar (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing package java-1.8.0-amazon-corretto-jdk:amd64 (--configure): end of file on stdin at conffile promptThat causes Google Chrome packages to fail, further down the line.Errors were encountered while processing: java-1.8.0-amazon-corretto-jdk:amd64W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/google-chrome.list:3 and /etc/apt/sources.list.d/google.list:1E: Sub-process /usr/bin/dpkg returned an error code (1)First of all, not sure why AWS still using apt-get instead of APT to manage packages and further more using Chrome packages to containers executions.Anyway, have anyone faced and fixed this?Thanks.FollowComment"


Question: AWS IoT core MQTT publish message to a topic

"Hello all I'm very new to using AWS IoT core. I am using a raspberry pi to send and receive messages with sensor data to and from the console. I can successfully send messages, but I am unable to receive messages I publish to the same topic (I don't see them on the raspberry pi when I post to the topic) I am using a template I found for room occupancy here: https://aws.amazon.com/blogs/iot/monitor-and-visualise-building-occupancy-with-aws-iot-core-amazon-quicksight-and-raspberry-pi/I also would like to know how to save the message I send down into a variable to use in my program.from datetime import datetimeglobal messageimport requestsimport timefrom AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClientmyMQTTClient = AWSIoTMQTTClient("outsidecam")myMQTTClient.configureEndpoint("myendpoint", 8883)myMQTTClient.configureCredentials("/home/pi/certs/Amazon-root-CA-1.pem", "/home/pi/certs/private.pem.key", "/home/pi/certs//home/pi/certs/certificate.pem.crt")print ('Initiating Realtime Data Transfer From Raspberry Pi...')Myvar= myMQTTClient.connect()date = datetime.now().strftime("%Y_%m_%d-%I:%M:%S_%p")print (f"Timestamp:{date}")while True:    print("Sending message")    message= "This is a Test"    myMQTTClient.publish("topic/pi", "{\"MotionMessage\":\""+ message + "\", \"Timestamp\" :\""+ str(date)+ "\"}", 0)    time.sleep(60)FollowComment"

Answer: "1Accepted AnswerHi AWS-currydem,from what I can see your code only publishes messages. To receive messages you need to subscribe to a topic. You are also using the old version of the AWS IoT Device SDK.You can try the current AWS IoT Device SDK v2 for Python. It includes also a pubsub example.In the sample mentioned above the message is stored in a variable before it is published. This would address your use-case to store the message in a variable.Cheers,PhilippCommentSharePhilipp Sanswered 3 months agoAWS-currydem 3 months agoI took your advice and it worked like a charm. for the second part of the question I'm still confused.When I publish a message I can see it in my python program, but I still am not sure on how to save my specific message into a string variablefor example If I publish{"message": "Blue"}I receive in my programb'{\n  "message": "Blue"\n}'How do I save the "Blue" to a variable like "color"color = Str(payload)color = "Blue"SharePhilipp S 3 months agoAssuming the message is stored in the variable message:import json  color = json.loads(message.decode())['message']ShareAWS-currydem 3 months agoI tried using what you suggested and got this error message: "AttributeError: 'str' object has no attribute 'decode'"Upon further investigation of the pubsub.py file I theorized that the message is stored in the function "def on_message_received" and the message is stored in the "payload" variableCallback when the subscribed topic receives a messagedef on_message_received(topic, payload, dup, qos, retain, **kwargs):print("Received message from topic '{}': {}".format(topic, payload))global received_countreceived_count += 1if received_count == cmdUtils.get_command("count"):received_all_event.set()I then tried to change the line to "color = json.loads(payload.decode())['message']" and got this error message at first: NameError: name 'payload' is not definedI assumed this was caused because "payload" is a parameter of the function "def on_message_received" instead of a declared variable, so I then tried these ways of declaring "payload";payload = ""  #I then got this error message:"AttributeError: 'str' object has no attribute 'decode'"payload = 0  #I then got this error message:"AttributeError: 'int' object has no attribute 'decode'"I'm pretty sure that "payload" is where the messages received are stored but I'm still not sure on how to save it to a variableI appreciate your help and I apologize if I'm not catching on as fast, I did try to find tutorials on how to do what I need.SharePhilipp S 3 months agoYou are using a variable (payload) that is not defined in your code. Replace payload with the variable name that contains your message in bytes sequence format.ShareAWS-currydem 3 months agoThank you so much for your help, it's almost working perfectly besides one error.I changed the "def on_message_received" functionCallback when the subscribed topic receives a messagedef on_message_received(topic, payload, dup, qos, retain, **kwargs):print("Received message from topic '{}': {}".format(topic, payload))global received_countreceived_count += 1if received_count == cmdUtils.get_command("count"):received_all_event.set()color = json.loads(payload.decode())['message']print(color)This is the format of the message I am sending from the topic{"message": "blue"}This allows me to save the received message to color and just prints blue. However, I also get this error message TypeError: string indices must be integers. And I know this is because there is supposed to be an integer in the brackets instead of the word 'message' but when I place either a 0 or a 1 the output, I get is either: A or 1I also tried changing the function and just having color = json.loads(payload) But when I do this the output is {'message': 'blue'}Is there a way to still get the output of print(color) as blue without getting the error message: TypeError: string indices must be integers?ShareShow 1 more comment"


Question: Veeam backup to S3 and deletion protection

A customer wants to ensure that data that is locked in Veeam can also not be deleted from S3.What's the best way to achieve that?FollowComment

Answer: "1Accepted AnswerYou can achieve the deletion protection by using the Object Immunity feature in Veeam Backup and Replication. Please note that this functionality only exists in Version 10 of the product, so if they are on a lower version - they would need to upgrade.CommentShareAnthony_Fanswered 3 years ago"


Question: mxnet-lambda function creation fails

"Hi, I'm trying to run this from an EC2 instance (sagemaker), but it fails with:An error occurred (RequestEntityTooLargeException) when calling the CreateFunction operation: Request must be smaller than 69905067 bytes for the CreateFunction operationWhat is wrong?FollowComment"

Answer: 0Accepted AnswerYou zipped function code is too large. See the limits here:https://docs.aws.amazon.com/lambda/latest/dg/limits.htmlIt has to be smaller than 50 MBTry to upload your zip to s3 first. Then reference the archive in S3 in the CreateFunction request. This could work when your unzipped code is smaller than 250 MB.CommentSharechristiananswered 4 years ago


Question: Timestream Eventual Consistency - What is the Consistent Read Latency on Magnetic Writes?

"I have a Timestream table used for testing in a development environment. The table is set to 1 hour memory store retention-- the minimum-- and 30 days of magnetic store retention. I also have magnetic writes enabled.We have custom code to restore data from a production table archive, saved as hive-partitioned, parquet files in S3, to the development table. When I process 10 days of records, about 1.2M records, and write them to the development table it seems the eventual consistency is really, very "eventual". The writes occur without error over the course of a minute or two, but we don't see the data until several hours later or sometimes even the next day.What is the expected, consistent read latency for magnetic writes in Timestream?FollowComment"

Answer: "0Hi NadirSLooks like you have a late data arrival issue, when query finished data does not occur after several hours?"The write occur without error" --> Do you mean query finished fast in a minute or two?This does feel need more investigation, I would suggest open a support case with us to investigate further.CommentShareSUPPORT ENGINEERKevin_Zanswered a year agoNadirS a year agoThanks for your response, Kevin. I mean that the query finishes fast (sub 10 seconds), but the data isn't there. If I re-run the same query hours or a day after I do the batch, magnetic write, the data is there.I have opened a support case, but I haven't heard back. The documentation for Timestream does say it is eventually consistent, so I don't expect the data to return immediately post-write, but the documentation says a "short time". I would like further clarification on the definition of "short time". I also expect that the magnetic write consistency latency will be higher than the memory write consistency latency, but the amount of latency I'm currently seeing seems high.https://docs.aws.amazon.com/timestream/latest/developerguide/writes.html#writes.eventual-consistencyShare0We opened a case with AWS Support and they pointed us to the relevant documentation.The maximum latency for reads after magnetic writes is 6 hours. You can find that number in the documentation here, https://docs.aws.amazon.com/timestream/latest/developerguide/data-ingest.html, emphasis mine:If you expect late arriving data, turn on magnetic store writes for your table. This will allow ingestion for data with timestamps that fall outside your memory store retention period but still within your magnetic store retention period. You can set this by updating the EnableMagneticStoreWrites flag in the MagneticStoreWritesProperties for your table. This property is false by default. Please note that writes to the magnetic store will not be immediately available to query. They will be available within 6 hours.There is also a useful metric in CloudWatch for tracking this latency. The MagneticStorePendingRecordsLatency provides the oldest magnetic write not available for querying.  https://docs.aws.amazon.com/timestream/latest/developerguide/metrics-dimensions.html#mcs-metricsIn my case, the MagneticStorePendingRecordsLatency was about 4.5 hours for a recent batch write, which is within the documented 6 hour limit.CommentShareNadirSanswered a year ago"