Question: "java.lang.IllegalArgumentException: path must exist: /tmp/scratchyaEyhw.scratch/test-package04aCEg/src/test/resources/MobileFeatures getting this error  unable to run cucumber framework,please help?"

"[DEVICEFARM] Setting up your device. This usually takes 2-3 minutes.[DEVICEFARM] ########### Start executing testspec ###########[DEVICEFARM] ########### Entering phase install ###########[DeviceFarm] export NVM_DIR=$HOME/.nvm[DeviceFarm] . $NVM_DIR/nvm.sh[DeviceFarm] nvm install 14.19.1v14.19.1 is already installed.Now using node v14.19.1 (npm v6.14.16)[DeviceFarm] export APPIUM_VERSION=1.22.2[DeviceFarm] avm $APPIUM_VERSION/usr/bin/avm: line 261: appium: command not found    exists : [DeviceFarm] ln -s /usr/local/avm/versions/$APPIUM_VERSION/node_modules/.bin/appium  /usr/local/avm/versions/$APPIUM_VERSION/node_modules/appium/bin/appium.js[DEVICEFARM] ########### Entering phase pre_test ###########[DeviceFarm] export CLASSPATH=$CLASSPATH:$DEVICEFARM_TESTNG_JAR[DeviceFarm] export CLASSPATH=$CLASSPATH:$DEVICEFARM_TEST_PACKAGE_PATH/*[DeviceFarm] export CLASSPATH=$CLASSPATH:$DEVICEFARM_TEST_PACKAGE_PATH/dependency-jars/*[DeviceFarm] echo "Start appium server"Start appium server[DeviceFarm] appium --log-timestamp --default-capabilities "{"deviceName": "$DEVICEFARM_DEVICE_NAME", "platformName":"$DEVICEFARM_DEVICE_PLATFORM_NAME", "app":"$DEVICEFARM_APP_PATH", "udid":"$DEVICEFARM_DEVICE_UDID", "platformVersion":"$DEVICEFARM_DEVICE_OS_VERSION", "chromedriverExecutable":"$DEVICEFARM_CHROMEDRIVER_EXECUTABLE"}" >> $DEVICEFARM_LOG_DIR/appiumlog.txt 2>&1 &[DeviceFarm] start_appium_timeout=0; while [ true ]; doif [ $start_appium_timeout -gt 60 ];thenecho "appium server never started in 60 seconds. Exiting";exit 1;fi;grep -i "Appium REST http interface listener started on 0.0.0.0:4723" $DEVICEFARM_LOG_DIR/appiumlog.txt >> /dev/null 2>&1;if [ $? -eq 0 ];thenecho "Appium REST http interface listener started on 0.0.0.0:4723";break;elseecho "Waiting for appium server to start. Sleeping for 1 second";sleep 1;start_appium_timeout=$((start_appium_timeout+1));fi;done;Waiting for appium server to start. Sleeping for 1 secondWaiting for appium server to start. Sleeping for 1 secondWaiting for appium server to start. Sleeping for 1 secondWaiting for appium server to start. Sleeping for 1 secondWaiting for appium server to start. Sleeping for 1 secondWaiting for appium server to start. Sleeping for 1 secondAppium REST http interface listener started on 0.0.0.0:4723[DEVICEFARM] ########### Entering phase test ###########[DeviceFarm] echo "Navigate to test package directory"Navigate to test package directory[DeviceFarm] cd $DEVICEFARM_TEST_PACKAGE_PATH[DeviceFarm] echo "Start Appium TestNG test"Start Appium TestNG test[DeviceFarm] java -Dappium.screenshots.dir=$DEVICEFARM_SCREENSHOT_PATH org.testng.TestNG -testjar *-tests.jar -d $DEVICEFARM_LOG_DIR/test-output -verbose 10[TestRunner] Running the tests in 'Test' with parallel mode:none[RunInfo] Adding method selector: org.testng.internal.XmlMethodSelector@dd3b207 priority: 10[TestClass] Creating TestClass for [ClassImpl class=cucumberOptions.TestNGTest][TestClass] Adding method AbstractTestNGCucumberTests.runScenario(io.cucumber.testng.PickleWrapper, io.cucumber.testng.FeatureWrapper)[pri:0, instance:null] on TestClass class cucumberOptions.TestNGTest[XmlMethodSelector] Including method io.cucumber.testng.runScenario()[TestNG] Running:/tmp/scratchyaEyhw.scratch/test-package04aCEg/testng.xml[SuiteRunner] Created 1 TestRunners[TestRunner] Running test Test on 1  classes,  included groups:[] excluded groups:[]===== Test classcucumberOptions.TestNGTest@BeforeClass AbstractTestNGCucumberTests.setUpClass(org.testng.ITestContext)[pri:0, instance:cucumberOptions.TestNGTest@20d3d15a]@Test AbstractTestNGCucumberTests.runScenario(io.cucumber.testng.PickleWrapper, io.cucumber.testng.FeatureWrapper)[pri:0, instance:cucumberOptions.TestNGTest@20d3d15a]@AfterClass AbstractTestNGCucumberTests.tearDownClass()[pri:0, instance:cucumberOptions.TestNGTest@20d3d15a][TestNG] RUNNING: Suite: "Test" containing "1" Tests (config: /tmp/scratchyaEyhw.scratch/test-package04aCEg/testng.xml)[Invoker 102065302] Keeping method AbstractTestNGCucumberTests.setUpClass(org.testng.ITestContext)[pri:0, instance:cucumberOptions.TestNGTest@20d3d15a] for class [TestClass name=class cucumberOptions.TestNGTest][Invoker 102065302] Invoking @BeforeClass AbstractTestNGCucumberTests.setUpClass(org.testng.ITestContext)[pri:0, instance:cucumberOptions.TestNGTest@20d3d15a]===============================================TestTests run: 0, Failures: 0, Skips: 0Configuration Failures: 1, Skips: 0[TestNG][TestNG] ===============================================[TestNG]     Test[TestNG] INVOKING CONFIGURATION: "Test" - @BeforeClass io.cucumber.testng.AbstractTestNGCucumberTests.setUpClass(org.testng.ITestContext)(value(s): org.testng.TestRunner@19d37183)Failed to invoke configuration method io.cucumber.testng.AbstractTestNGCucumberTests.setUpClass:path must exist: /tmp/scratchyaEyhw.scratch/test-package04aCEg/src/test/resources/MobileFeatures/Features/EarlysalaryQECjourney.feature[TestNG] FAILED CONFIGURATION: "Test" - @BeforeClass io.cucumber.testng.AbstractTestNGCucumberTests.setUpClass(org.testng.ITestContext)(value(s): org.testng.TestRunner@19d37183) finished in 0 ms[TestNG] java.lang.IllegalArgumentException: path must exist: /tmp/scratchyaEyhw.scratch/test-package04aCEg/src/test/resources/MobileFeatures/Features/EarlysalaryQECjourney.feature[TestNG] at io.cucumber.core.resource.PathScanner.findResourcesForPath(PathScanner.java:49)[TestNG] at io.cucumber.core.resource.PathScanner.findResourcesForUri(PathScanner.java:31)[TestNG] at io.cucumber.core.resource.ResourceScanner.findResourcesForUri(ResourceScanner.java:61)[TestNG] at io.cucumber.core.resource.ResourceScanner.scanForResourcesUri(ResourceScanner.java:134)[TestNG] at io.cucumber.core.runtime.FeaturePathFeatureSupplier.loadFeatures(FeaturePathFeatureSupplier.java:62)[TestNG] at io.cucumber.core.runtime.FeaturePathFeatureSupplier.get(FeaturePathFeatureSupplier.java:45)[TestNG] at io.cucumber.testng.TestNGCucumberRunner.<init>(TestNGCucumberRunner.java:122)[TestNG] at io.cucumber.testng.AbstractTestNGCucumberTests.setUpClass(AbstractTestNGCucumberTests.java:27)[TestNG] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[TestNG] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[TestNG] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[TestNG] at java.lang.reflect.Method.invoke(Method.java:498)[TestNG] at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)[TestNG] at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564)[TestNG] at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)[TestNG] at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138)[TestNG] at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:175)[TestNG] at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:107)[TestNG] at org.testng.TestRunner.privateRun(TestRunner.java:767)[TestNG] at org.testng.TestRunner.run(TestRunner.java:617)[TestNG] at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)[TestNG] at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)[TestNG] at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)[TestNG] at org.testng.SuiteRunner.run(SuiteRunner.java:254)[TestNG] at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)[TestNG] at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)[TestNG] at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)[TestNG] at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)[TestNG] at org.testng.TestNG.run(TestNG.java:1057)[TestNG] at org.testng.TestNG.privateMain(TestNG.java:1364)[TestNG] at org.testng.TestNG.main(TestNG.java:1333)[Invoker 102065302] Keeping method AbstractTestNGCucumberTests.tearDownClass()[pri:0, instance:cucumberOptions.TestNGTest@20d3d15a] for class [TestClass name=class cucumberOptions.TestNGTest][Invoker 102065302] Invoking @AfterClass AbstractTestNGCucumberTests.tearDownClass()[pri:0, instance:cucumberOptions.TestNGTest@20d3d15a][TestNG] INVOKING CONFIGURATION: "Test" - @AfterClass io.cucumber.testng.AbstractTestNGCucumberTests.tearDownClass()[TestNG] PASSED CONFIGURATION: "Test" - @AfterClass io.cucumber.testng.AbstractTestNGCucumberTests.tearDownClass() finished in 0 ms===== Invoked methodsAbstractTestNGCucumberTests.setUpClass(org.testng.ITestContext)[pri:0, instance:cucumberOptions.TestNGTest@20d3d15a]org.testng.TestRunner@19d37183  550752602AbstractTestNGCucumberTests.tearDownClass()[pri:0, instance:cucumberOptions.TestNGTest@20d3d15a] 550752602Creating /tmp/customer_log_directoryZc6WXr/test-output/Suite/Test.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/Suite/Test.xmlFAILED CONFIGURATION: @BeforeClass setUpClass(org.testng.TestRunner@19d37183)java.lang.IllegalArgumentException: path must exist: /tmp/scratchyaEyhw.scratch/test-package04aCEg/src/test/resources/MobileFeatures/Features/EarlysalaryQECjourney.featureat io.cucumber.core.resource.PathScanner.findResourcesForPath(PathScanner.java:49)at io.cucumber.core.resource.PathScanner.findResourcesForUri(PathScanner.java:31)at io.cucumber.core.resource.ResourceScanner.findResourcesForUri(ResourceScanner.java:61)at io.cucumber.core.resource.ResourceScanner.scanForResourcesUri(ResourceScanner.java:134)at io.cucumber.core.runtime.FeaturePathFeatureSupplier.loadFeatures(FeaturePathFeatureSupplier.java:62)at io.cucumber.core.runtime.FeaturePathFeatureSupplier.get(FeaturePathFeatureSupplier.java:45)at io.cucumber.testng.TestNGCucumberRunner.<init>(TestNGCucumberRunner.java:122)at io.cucumber.testng.AbstractTestNGCucumberTests.setUpClass(AbstractTestNGCucumberTests.java:27)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564)at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138)at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:175)at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:107)at org.testng.TestRunner.privateRun(TestRunner.java:767)at org.testng.TestRunner.run(TestRunner.java:617)at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)at org.testng.SuiteRunner.run(SuiteRunner.java:254)at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)at org.testng.TestNG.run(TestNG.java:1057)at org.testng.TestNG.privateMain(TestNG.java:1364)at org.testng.TestNG.main(TestNG.java:1333)[TestNG]     Tests run: 0, Failures: 0, Skips: 0[TestNG]     Configuration Failures: 1, Skips: 0[TestNG] ==============================================================================================SuiteTotal tests run: 0, Failures: 0, Skips: 0Configuration Failures: 1, Skips: 0[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 1 msCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/toc.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/Test.propertiesCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/index.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/main.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/groups.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/methods.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/methods.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/methods.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/methods.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/methods-alphabetical.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/methods-alphabetical.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/methods-alphabetical.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/methods-alphabetical.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/classes.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/reporter-output.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/methods-not-run.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/Suite/testng.xml.htmlCreating /tmp/customer_log_directoryZc6WXr/test-output/old/index.html[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@2df32bf7: 14 ms[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@32cf48b7: 1 ms[TestNG] Time taken by org.testng.reporters.jq.Main@3bd94634: 22 ms[TestNG] Time taken by org.testng.reporters.XMLReporter@7181ae3f: 3 ms[TestNG] Time taken by org.testng.reporters.EmailableReporter2@1ee807c6: 5 ms[DEVICEFARM] ########### Entering phase post_test ###########[DEVICEFARM] ########### Finish executing testspec ###########[DEVICEFARM] ########### Setting upload permissions ###########[DEVICEFARM] Tearing down your device. Your tests report will come shortly.FollowComment"

Answer: "0Hello,Thanks for reaching out!It could be one of several items that are occurring from what I’ve been able to determine.To get a deeper understanding into the issue could you please open a support case with us: https://console.aws.amazon.com/support/home#/case/create?issueType=technicalAfter you open the case with us under the Device Farm service it may have a few additional questions it asks to fill in order for us to get a deeper idea into the issue, but please do feel free to include the error you’ve shared here as well.CommentShareSUPPORT ENGINEERTim_Panswered a year ago"


Question: How to check for InvalidKeyPair.Duplicate in AWS SDK Go v2

"Hello,I see an error when a pair is already present, I would like to check for error type correctly on Go with errors.As/Is but I am unable to find the proper type for this particular error in the SDK. It is so huge, apparently it's generated somehow. Any hint what is the correct type to check for?Thanks!FollowComment"

Answer: "0Hi LzapI understand you want to check if a key pair is invalid. Please correct if my understanding is wrong.Please ensure that the key pair name is not existing on your AWS region, If it does, create or import a key that is unique from the existing key pair names. To check if a key pair exist  within your region run the following on AWS CLI  "aws ec2 wait key-pair-exists --key-names my-key-pair" [1],You can also check by displaying the key pair using fingerprint for your key pair, because it is generated from it and can be used to verify that the private key on your local machine matches the public key stored in AWS[2].I hope the information above is helpful to resolve the issue in question.References:1.https://docs.aws.amazon.com/cli/latest/reference/ec2/wait/key-pair-exists.html2.https://docs.aws.amazon.com/cli/latest/userguide/cli-services-ec2-keypairs.htmlCommentShareNonkululekoanswered a year ago"


Question: Micrsoft SQL Server tempdb to instance/ephemeral disks on Amazon EC2

The customer having the requirement to store the Tempdb files in  instance/ephemeral  storage. Is there any script available to restore the tempdb drive after shutdown if we are configuring  the MS SQL Server Tempdb in  instance/ephemeral  storage?FollowComment

Answer: "0Accepted AnswerBy placing tempdb on instance store the gain is twofold, not only the instance store is fast because its local to the server but also the IO against the instance store is not charged separately. Plan carefully, as you may be aware ephemeral\instance store volume is lost when the virtual machine is started on another host because ephemeral disk is locally attached to the machine-The volume should be initialized  before SQL Server service starts else the SQL Server startup procedure will fail-The permission on the instance store volumeYou may use solution from one of the below links to accomplish the same:EBS Volumes Don't Initialize on Windows Server 2016 and Later AMIshttps://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/common-issues.html#init-disks-win2k16How to move Micrsoft SQL Server tempdb to instance/ephemeral disks on Amazon EC2https://www.ifm.net.nz/cookbooks/amazon-sql-tempdb/index.htmlCommentShareEXPERTAbhishek Sonianswered 3 years ago"


Question: Need AWS Glue to store bad records/ records with error when reading Mongo db data to a S3 path and process the rest of the data.

"Hi,Working on a solution to read Mongo db data and storing in S3. But due to conflicting schema the job does not run the second time, it always runs in the first time. I need Glue to handle the records with such conflicting schemas, tried providing badRecordsPath and mode = PERMISSIVE but it does not consider the options. Tried reading data using both -> with catalog and with from_options. Below is the snippet of the same.dynamic_frame = glueContext.create_dynamic_frame_from_catalog(database = "mongo_app_config_db",table_name = "s3app_config_intf_logs",additional_options = {"database":"app_config", "collection":"intf_logs","badRecordsPath": s3_path,"mode": "PERMISSIVE" },transformation_ctx = "dynamic_frame" )dynamic_frame = glueContext.create_dynamic_frame.from_options(connection_type="mongodb",connection_options=read_mongo_options, transformation_ctx = "dynamic_frame", additional_options = {"badRecordsPath": s3_path , "mode": "PERMISSIVE"})Please help me to handle such records and continue the job run for rest of the records.FollowComment"

Answer: "0Hello,Glue or Spark uses a connector called "MongoDB Spark connector" when reading from Mongodb and as per the documentation here https://www.mongodb.com/docs/spark-connector/current/configuration/read/ I do not see any options available for handling corrupted/bad records. I could also see a feature request was placed recently here https://jira.mongodb.org/browse/SPARK-327 to the community.You can try exporting mongo collection as JSON or CSV files to an s3 location using something like https://www.mongodb.com/docs/database-tools/mongoexport/  and consume them using Spark data frames. Spark should support handling bad records for these file formats.You can also convert between Spark data frame and Glue dynamic frame easily as shown in the below linkshttps://stackoverflow.com/a/59668469https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame.htmlCommentShareSUPPORT ENGINEERChiranjeevi_Nanswered a year ago"


Question: Server-side lambda: providerName doesn't match issuer

"I'm attempting to set AWS.config.credentials in a server-less lambda (running in a REST endpoint).  The login is from a Cognito User Pool which uses ADFS as a Federated Identity.  I also have a Cognito Identlty Pool in the mix.My lambda handler looks like this:exports.handler = (event, context, callback) {   ...}The event.headers.Authorization has a valid JWT (from the id_token) which was placed in the Authorization header when the POST call on the HTML client to the REST endpoint was executed.  I validated the JWT myself here: https://jwt.io.  In addition, the context.authorizer object also looks to be valid indicating that I am logged in, it has my email and username information and valid issuance and expiry dates.  In short, I'm pretty sure I've got a valid login.But when I try to execute the following commands I get an error that says "NotAuthorizedException: Invalid login token. Issuer doesn't match providerName"AWS.config.credentials = new AWS.CognitoIdentityCredentials({    IdentityPoolId: 'us-east-1:my-identity-pool-id',    Logins: {      "cognito-idp:us-east-1.amazonaws.com/us-east-1_myUserPoolId": event.headers.Authorization    } }); AWS.config.credentials.get(err => {    if (!err) {      var id = AWS.config.credentials.identityId;      console.log('Cognito Identity ID '+ id);    } else {      console.log('cred error: ', err);    } });of course, my-identity-pool-id and myUserPoolId are replaced with their actual values.  I've seen many other posts on this topic indicating incorrectly formulated Logins objects produce this error, but I've been over that several times in this code - and had others look it over for me as well.  I don't think I got that part wrong.  In so far as the error text is concerned, well it is actually correct.  If I look at my decoded JWT, the issuer and the providerName are different:{  "at_hash": "pciSj0Hcjk1Pp9noJIj4GQ",  "sub": "e41500b1-d987-49be-81c0-8fbc36a59ce1",  "email_verified": false,  "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_xxxxxxx",  "cognito:username": "ge2cadmin2-saml-idp_ravi",  "cognito:roles": [    "MyFancyRole"  ],  "aud": "7r7v0ren5fm3m6vmjgj6nasr79",  "identities": [    {      "userId": "ravi",      "providerName": "ge2cadmin2-saml-idp",      "providerType": "SAML",      "issuer": "http://adfs.careevolution.com/adfs/services/trust",      "primary": "true",      "dateCreated": "1590770196859"    }  ],  "token_use": "id",  "auth_time": 1591118267,  "exp": 1591126227,  "iat": 1591122627,  "email": "ravi@careevolution.com"}As you can see, in the identities[] section, the issuer is http://adfs.careevolution.com/adfs/services/trust and the providerName is ge2cadmin2-saml-idp.  But I think in this scenario, that is to be expected.  However, if the problem is indeed this difference (and not some misconfiguration elsewhere) how can I setup things so AWS.CognitoIdentityCredentials is OK with the issuer being different from the providerName?Edited by: RaviSDesai on Jun 2, 2020 12:50 PMFollowComment"

Answer: "0The answer turns out to be simple.  The magic string:  "cognito-idp:us-east-1.amazonaws.com/us-east-1_myUserPoolId" was incorrect.  Instead I should have used: "cognito-idp.us-east-1.amazonaws.com/us-east-1_myUserPoolId" (with a dot after cognito-idp instead of a colon).  While the error message is terrible and misleading, the error turned out to be simple user error.I posted this here in order to help anyone who comes by with a search for that same misleading error message.CommentShareRaviSDesaianswered 3 years ago"


Question: Subtracting two dates and comparing if result >365 days

"I have this sqlSELECT instrument_id, maturity_dt, issue_dt, (maturity_dt - issue_dt) as daysFROM instrument_desc_currentwhere maturity_dt is not nulland issue_dt is not nulland (maturity_dt - issue_dt) > 365limit 100but get this error...  SYNTAX_ERROR: line 5:30: '>' cannot be applied to interval day to second, integercommenting out "and (maturity_dt - issue_dt) > 365 my result set looks like...instrument_idmaturity_dtissue_dt        days1000244483      3/22/20113/21/20111 00:00:00.0001000244596      7/1/2020        4/4/20007393 00:00:00.0001000244665      4/1/2019      12/8/20112671 00:00:00.0001000245009      6/1/2021        6/1/20113653 00:00:00.0001000245547      5/23/20114/6/201147 00:00:00.0001000245766    11/8/2012      11/7/20121 00:00:00.000How do I select only those records which are greater than 365 days?FollowComment"

Answer: "0I think based on the tags you're using Athena, so date_diff may be the answer. The official AWS documentation links to Presto.Other database systems also support DATEDIFF: MySQL, MS SQL Server and Oracle. With POSTGRES you could use DATE_PART, see http://www.sqlines.com/postgresql/how-to/datediff.CommentShareedeastwoodanswered a year ago"


Question: Use of ACTIVE/STANDBY ActiveMQ results in "supressing duplicate delivery on connection" errors when used with the redelivery plugin

"When connecting to the active broker ActiveMQ via JMS in java without a failure url (e.g. ssl://broker_address), the activeMQ redelivery plugin works as expected, i.e retries X times before moving to the DLQ.When copying the Java failover string from the AWS MQ console  ( failover:(ssl://broker1:61617,ssl://broker2:61617), and using that to connect, the re-delivery plugin  no longer functions as expected. The scheduler runs, and messages are infact re-dispatched, but the the activeMQ java consumer throws an exception, stating "suppressing duplicate delivery on connection".Note everything else remained constant between testing with a failure url and testing without the failure url.Version of ActiveMQ being used is 5.16.3.The first thing I want to ask, is anyone using redelivery plugin with failover strings successfully?Secondly, anyone have the specific issue I've faced and can suggest any solutions?=====SOLUTION=======This ended up being a problem with the default client library for activemq with springboot, upgrading the activemq library to 5.16.3 fixed this bug.FollowComment"

Answer: "0Hi,Apologies for the delay. I was just about to reach out to our MQ team but it seems you have found a solution. Please note that you can also answer your own questions if you have found a solution.I just want to confirm if upgrading the library fixed all your issues or if there's anything else which needs to be answered?Hope you have a great day/evening ahead!CommentShareSUPPORT ENGINEERRyan_Aanswered a year ago"


Question: Inspector Lambda Scanning – CWE-798 - Hardcoded credentials in package-lock.json

"Hi,We've been testing out Inspector on our Lambda code (NodeJS) and one of the vulnerabilities it highlights is hardcoded credentials in some of our package-lock.json files. I've reviewed the files in question and can only identify one that has a http username encoded in a url (but no password) and the other I can't even find a username in any of the urls. The only thing I can think that is causing this is that we are using some dependencies from a private repository although I can't see any credentials in the file.Has anyone else observed this issue or can suggest what else might be triggering the detector?FollowComment"

Answer: "0Hello,Generally, CWE-798: Use of Hard-coded credentials checks if product contains any hard-coded credential such as passwords, cryptographic key, Username and password combination, which it uses for its own inbound authentication, outbound communication to external components, or encryption of internal data. [1]From the query posted, I understand that though your files doesn’t contain passwords, inspector is still detecting the vulnerability CWE-798. To further debug into this behaviour, I would suggest you to create a support case with us, so that we have visibility to the inspector findings and could fetch more details from internal team regarding this.Reference[1] https://cwe.mitre.org/data/definitions/798.htmlCommentShareDivya_Aanswered 9 days ago"


Question: Cognito external provider user email cannot be automatically verified

"According to this documentation, my pre-sign up lambda function handler written in Python is already setting the autoVerifyEmail in the response attribute to True, yet after a user logs in with a external provider of a Active Directory Federation Service using SAML, the user is created in the user pool, but yet the email remains unverified.Here is the event object from cloudwatch with sensitive data being replaced with '--' for the purpose of this post:{    "version": "1",    "region": "ap-southeast-1",    "userPoolId": "--",    "userName": "--",    "callerContext": {        "awsSdkVersion": "aws-sdk-unknown-unknown",        "clientId": "--"    },    "triggerSource": "PreSignUp_ExternalProvider",    "request": {        "userAttributes": {            "email_verified": "false",            "cognito:email_alias": "",            "cognito:phone_number_alias": "",            "given_name": "--",            "family_name": "--",            "email": "--"        },        "validationData": {}    },    "response": {        "autoConfirmUser": true,        "autoVerifyEmail": true,        "autoVerifyPhone": false    }}Thank you in advance for any solution to this problem!FollowComment"

Answer: "0Prior to discussion about Pre sign-up Lambda trigger, I would first like to mention some details around the integration of Cognito with External identity providers (such as Facebook/Google/Apple/Azure):→ When a user is created in Cognito after Sign-up using these social providers, all the attributes of the user depend upon the attribute mapping which is configured in Cognito (under the “Attribute Mapping” tab in Cognito User pool console).→ External identity users on SignIn are automatically confirmed by the Cognito service & the account status is set to "EXTERNAL_PROVIDER" (there is no need of using any Lambda trigger).→ On every federation i.e. each time the user sign-in, all the attributes of this user are again updated in the user pool based on all the attributes which are sent from the social provider side & also the attribute mapping configured in Cognito.→ If there is no mapping configuration for email_verified in Cognito for that provider, then upon the federation of your user, this attribute is set to a default value of false.For more details around mapping in user pool, please refer this documentation linkNow coming to the Pre sign-up Lambda trigger, please note that the trigger is indeed called each time a user sign-up using any of the external providers. However, the values returned by the trigger to verify phone & email are not considered for the federated identities. Hence, this is the reason you are observing this behaviour, where even after returning “autoVerifyEmail = true”, the email_verified is set to false. Though it would work perfectly fine for the native Cognito user which signed-up using username & password.That been shared, Pre sign-up Lambda trigger cannot be used for your use-case.In regard to your use-case, there can be two workarounds:Please refer to “Option 1“ in case if you just want to map the actual value of email_verified attribute for the user, which can be true or false depending upon what is configured for that user at the social provider end. However, in case if you want to make the value for the email_verified attribute to be always true, then please refer to the ”Option 2“.OPTION 1In this case, we may use the Attribute Mapping section (for example,  "Federation" →  "Attribute Mapping" → "SAML") for the user pool to ensure that all the required attribute which needs to present on the user are configured. Hence, in this scenario, we will create a mapping between "email_verified" attribute which is coming from the federated identity provider side & the one present in Cognito. Once that is done for each identity provider, the user created in the user pool will have a value for "email_verified" attribute to be true or false, depending upon what is sent by the provider.For more details around configuring the attribute mappings for your user pool, please refer the documentation linkPlease note, we may use the "Discovery endpoint" of the respective social provider to confirm if it will return a value of “email_verified” or notOPTION 2As mentioned earlier in the response that each time the user sign-in with any external provider, all the attributes of this user are again updated in the user pool based on all the attributes values which are sent from the identity provider side. Hence, setting the “email_verified” attribute before authentication using any trigger would lead to no effect, as immediately after signing-in the attribute will be updated to either its default value or to a value coming from the provider side.Hence, for this solution, we may use the following two Lambda triggers along with the “AdminUpdateUserAttributes” API call which will be used to update the “email_verified” to true during the respective SignUp or SignIn flow:a) Post confirmation Lambda Trigger: This trigger is the one which is invoked only during the SignUp flow when the user is confirmed, however before the tokens are being issued for the first time. Hence, this trigger will allow us to set the “email_verified” attribute to true before the tokens are issued during the SignUp flow. So, once the token is issued, you will be able to view the same in the Id token as well as in the user details page in the user pool console. For more details around this trigger, please refer to the documentation linkb) Post authentication Lambda Trigger: This trigger is invoked after the user is authenticated during the SignIn flow, however before tokens are issued for the user. Hence, each time when the user sign-in, this trigger will allow us to set the “email_verified” attribute to true before the tokens are issued. Again, to confirm the same, you can either decode the Id token or refer to the user details page in the console. For more details around this trigger, please refer to the documentation linkI hope the above information resolves your query.CommentShareSUPPORT ENGINEERVarunanswered 8 months agorePost-User-0582860 7 months agoHi Varun,Thank you very much for your answer. It has given me the explanation as to why the email was not automatically verified.As for the solution, I am currently attempting Option 2 while calling the AdminUpdateUserAttributes API within the Post Confirmation lambda trigger function. The issue I've been facing with this option however, is that although I'm following this documentation and calling with the right arguments, I'm not able to get a response from the cognito-idp client, as well as no errors being thrown when calling this API.Please advise if possible. Thank you again!ShareVarun SUPPORT ENGINEER7 months agoI would advise you to log the events in the lambda code and check Cloudwatch logs to investigate what is happening in the background.Share"


Question: Unable to refresh Quicksight dataset. ERROR: Query (39995467) cancelled on user's request

"Hello,I am not able to refresh the dataset anymore receiving the following error message:This is a general SQL error. This can be caused by query timeouts, resource constraints, unexpected DDL alterations before or during a query, and other database errors. Check your database settings and your query, and try again.Source error message:ERROR: Query (39995467) cancelled on user's requestSQL queries behind are working correctly, how can I dive deep and fix it?Thanks a lotNicolòFollowComment"

Answer: "0If you open up the Athena console and check the query history, you should be able to locate the queries that were executed at the time of the Quicksight dashboard refresh. Try refreshing the dashboard once while you have the Athena query history open. Here each query that executes in Athena would be listed and you should be able to see detailed query errors as well that would help identify which queries are timing out or erroring out.CommentShareAnanth Tirumanuranswered 10 months ago"


Question: Cloudfront can not serve s3 website via custom origin deployed via Serverless Framework

"Hi, I am setting up CI/CD pipeline for the frontend of an application developed in REACT, everything is working as expected.Build is builtbuild is being pushed to s3site can be visited via s3 static site urlWhat's not working is when the s3 site is set up via CloudFront as the custom origin. I'm attaching my code for creating a bucket and cloudfront distribution.Note: the custom domain works fine as well (it to redirects to the error that cloudfront distribution domain returns, i.e. 504 ERROR)Error message is in the end of the questionType: AWS::CloudFront::DistributionProperties:  DistributionConfig:    Aliases:      - ${param:deploymentUrl}    CustomErrorResponses:      - ErrorCode: 404        ResponseCode: 200        ResponsePagePath: /index.html        ErrorCachingMinTTL: 10    DefaultCacheBehavior:      AllowedMethods:        - GET        - HEAD      TargetOriginId: ${self:custom.cloudfrontOriginId}      ForwardedValues:        QueryString: 'false'        Cookies:          Forward: none      ViewerProtocolPolicy: redirect-to-https      CachePolicyId: 658327ea-f89d-4fab-a63d-7e88639e58f6 # CachingOptimized    DefaultRootObject: index.html    Enabled: 'true'    HttpVersion: 'http2'    Origins:      - DomainName: ${param:s3BucketName}.s3-website-${self:provider.region}.amazonaws.com        CustomOriginConfig:          HTTPPort: 80          HTTPSPort: 443          OriginProtocolPolicy: https-only        Id: ${self:custom.cloudfrontOriginId}    ViewerCertificate:      AcmCertificateArn: ${param:certificateArn}      MinimumProtocolVersion: 'TLSv1.2_2021'      SslSupportMethod: 'sni-only'Type: AWS::S3::BucketProperties:  BucketName: ${param:s3BucketName}  AccessControl: PublicRead  WebsiteConfiguration:    IndexDocument: index.html    ErrorDocument: index.htmlType: AWS::S3::BucketPolicyProperties:  Bucket:    Ref: Bucket  PolicyDocument:    Statement:      - Sid: PublicReadGetObject        Action:          - s3:GetObject        Effect: Allow        Principal: '*'        Resource: arn:aws:s3:::${param:s3BucketName}/*The request could not be satisfied.CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.Generated by cloudfront (CloudFront)Request ID: ...FollowComment"

Answer: "1The static website server provided by the S3 service for a bucket is http only, no support for https.  Your custom origin config says https-only, so it won't be able to contact the origin.The best way to do this is disable the bucket's website and public access and instead of using a custom origin, use an S3 origin secured by an Origin Access Identity.  That's a special CloudFront principal that you grant permissions to in your bucket policy.Then for best practice enable all the Block Public Access settings on the bucket.  Bonus points for doing that at the account level too and never making any of your buckets public!CommentShareEXPERTskinsmananswered 10 months ago"


Question: lustre client does not update properly on ubuntu 22

"Typically when I update my ubuntu servers I have to reinstall the lustre client per the documentation or the FSx lustre volume no longer mounts.I am using this documentation:https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.htmlThis time, as expected, the mount failed after an OS update and I followed the doc above for ubuntu 22. When I try to mount the fsx volume, I get this error:mount.lustre: mount fs-xxxxxxxxxxx.fsx.us-east-2.amazonaws.com@tcp:/xxxxx at /mcb failed: No such deviceAre the lustre modules loaded?Check /etc/modprobe.conf and /proc/filesystemsThe filesystem is available and mounts to amazon linux servers. The fragility of the ubuntu lustre client is an ongoing issue for me.FollowComment"

Answer: "0Accepted AnswerHi, are you on Ubuntu 22 kernel version 5.19? FSx Lustre has not released support for version 5.19 yet.CommentSharerePost-User-9323546answered 24 days agoLinda 24 days agoThanks, is there a way to know when a new kernel is or is not supported? The AWS documentation: https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.htmlgives no indication that 5.19 is not supported. It says: "The Lustre client on Ubuntu 22.04 requires kernel 5.15.0-1015-aws or later..."ShareAaron_B 23 days agoFor client kernel compatibility, you can reference the compatibility matrix at the end of the installation page:https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html#lustre-client-matrixIf your compute instance isn't running the Linux kernel specified in the installation instructions, and you can't change the kernel, you can build your own Lustre client. Additional details:https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.htmlhttps://wiki.lustre.org/Compiling_LustreShare"


Question: Criteria for choosing which regions to use on a multi-region setup for improved reliability

"Based on the recent outage at us-east-1, where we have most of our AWS resources, the company has taken the path to start transitioning into a multi-region architecture, but picking which extra region(s) to use is not a matter of chance and the decision must be thoughtfully taken.Right now the first criteria to consider is cost as most US-based regions are cheaper for example than the one in Sao Paulo.Latency for our customers (distributed across LATAM and US) might be worthwhile considering, but right now everything is working from Virginia. For major disasters maybe replicating to a different country, or perhaps even continent could be the best for geographic isolation, but at a cost.How do you actually decide which regions to use? How to weigh different factors? Is there an actual guide for such?FollowComment"

Answer: "4There was a recent blog post on this:"What to Consider when Selecting a Region for your Workloads"https://aws.amazon.com/blogs/architecture/what-to-consider-when-selecting-a-region-for-your-workloads/Hope this will help you identify the criteria that you could use.CommentShareDennis_Aanswered a year ago1From the blog post above, you will likely mainly consider latency (some regions such as Stockholm are cheap to use but would significantly impact your US/SA userbase's experience) and cost as you mentioned.Cost though is something you can tune and optimise by carefully consider what you need to replicate (this will be based on your risk modelling). For example, if you go for active/active and plan to do old-fashioned backups, you can keep them in the cheapest region. Traffic flows can be optimised to avoid going too much "cross border" (ie make sure once an user's request enters into a region, it stays there).I know you haven't directly requested it, but if you're planning things now, I suggest to do it with this in mind: https://aws.amazon.com/builders-library/static-stability-using-availability-zones/ - most of Amazon/AWS are designed based on this principle and generally limiting your dependency on control planes or changes in general during a failure will make everything better.CommentShareGiorgio@AWSanswered a year ago"


Question: from the web application to how can I run the Python script in the Ubuntu terminal?

"Is there a way to run the python script terminal and get the output EC2 instance?As an example, I get two values from the web app as input. With these two values as arguments, a Python script runs on the terminal.Python3 add.py value1 value2What AWS services are required?Is there a way to get these values to be sent to a web application?What is the best way to run or trigger the Python script on the EC2 instance from the web app?It's challenging for me to figure this out.FollowComment"

Answer: "0If I understand your question correctly: You have a web application and you want to send two values to a Python script; have it perform some operation and return a result.While you could do this with EC2 I wouldn't recommend it - you would have a whole instance running all the time even when your web application isn't requesting any results.Instead, consider using Lambda. Create a Lambda function using Python then publish that function using Lambda function URLs - you pass the two values to the URL and it executes the Lambda function, returning the appropriate value.CommentShareEXPERTBrettski-AWSanswered a year agosakthivel a year agoBrettski, I know it's not recommended for simple tasks, but the actual application is image processing.The S3 bucket folder name is the argument value. It takes a minimum of two to three hours to complete.I have tested the script manually and it works well. I want to automate it on the EC2 instanceThanksShareBrettski-AWS EXPERTa year agoThanks for expanding on the question. In that case, I would use SQS: Send the job details to a SQS queue; have EC2 instances scale based on the length of the queue (that way you can run zero instances when there are no jobs; and you can scale to more instances if necessary); then your script on the EC2 instances reads the job details from the SQS queue. Don't forget to set the message visibility timeout to several hours. Note that you can create the SQS message as a trigger from S3 when the object is uploaded.Share"


Question: AppSync did't return Highlight response with opensearch resolver

"Hi have the following schema,example:type Demo { id: ID! code: String name: String         title: String highlight: String }and my query resolver template looks like this,{     "version":"2017-02-28",     "operation":"GET",     "path":"/index/_search",     "params":{         "headers":{},         "queryString":{},         "body":{         "query": {           "bool": {             "filter": {               "term": {                 "code": $util.toJson($context.arguments.code)               }             },             "must": [               {                 "bool": {                   "should": [                     {                       "match_phrase_prefix": {                         "title": {                           "query": $util.toJson($context.arguments.title)                         }                       }                     },                     {                       "match_phrase_prefix": {                         "name": {                           "query": $util.toJson($context.arguments.name)                         }                       }                     }                   ]                 }               }             ]           }         },         "highlight": {           "fields": {             "name": {}           }         }         }     } }all results are getting fine except highlightDoes anyone have any idea about this? I want to get highlighting responses in the same query.FollowComment"


Question: AWS CloudWatch Log Insights Generate URL

"Is it possible to deterministically generate a URL for CloudWatch Log Insights given the log group, start time, and query (text area that allows fields, sorts, filters, etc...)? The URL is extremely long and does not have standard URL encoding from what I can tell.There are plenty of suggestions in the below Stack Overflow post about the topic but I was curious if there was a more official guide to doing this?https://stackoverflow.com/questions/60796991/is-there-a-way-to-generate-the-aws-console-urls-for-cloudwatch-log-group-filtersFollowComment"

Answer: "2Accepted AnswerIn addition to the suggestions in the Stack Overflow post, here is some further info on how the Logs Insights URLs are generated that may help you.The deep link of Logs Insights query is generated in three steps:Step 1. Define the queryDetails in a JSON structure {        editorString: '',        end: '',        queryId: '',        source: '',        start: '',        timeType: '',        tz: '',        unit: ''    };Step 2. It uses jsurl (https://www.npmjs.com/package/jsurl) to encode the JSON object logs-insights?queryDetail=<encoded js url>Step 3. It uses following map to replace the special characters inside logs-insights?queryDetail=<encoded js url> {  '!' = '$21',  '#' = '$23',  '%' = '$25',  '&' = '$26',  // Spaces are encoded as +. That's what GWT uses when we do not manually  //   encode our routes, so it is backwards compatible with customer links  //   generated prior to this manual encoding process.  ' ' = '$2B',  ':' = '$3A',  ';' = '$3B',  '=' = '$3D',  '?' = '$3F',}Step 4. add prefix https://console.aws.amazon.com/cloudwatch/home?region=<region>#logsV2:To get more details, you can use some example URL and reverse the steps and see what the original JSON structure looks like.CommentShareSUPPORT ENGINEERJustin_Banswered a year ago"


Question: AWS WorkSpaces Windows Bundle choice

"Hello,  I'm looking to deploy WorkSpaces for my company.  We primarily have two types of users, average office type and developers.   The developers use Visual Studio, eclipse and or IntelliJ applications.What would be the lowest recommended bundle type for these two user types?TIA---KennyFollowComment"

Answer: "0Hello Kenny,I have had a look at the Visual Studio 2019 System Requirements [1] and I believe that the best option for your developers is the "Performance" type while for the other users the "Value" or "Standard" type [2].Having said that, if you are going to use the Windows bundle my suggestion is to start with testing the "Standard" type and increase or decrease the Compute Type later as per your needs [3].Thank you,VasosAWSReferences:[1] https://docs.microsoft.com/en-us/visualstudio/releases/2019/system-requirements[2] https://aws.amazon.com/workspaces/pricing/[3] https://docs.aws.amazon.com/workspaces/latest/adminguide/modify-workspaces.htmlCommentShareAWS-User-5546426answered 3 years ago0VasosAWS,  Thanks for the reply.   Will start testing.--KennyCommentShareBESCanswered 3 years ago"


Question: Deploying ECS cluster + task with Javascript SDK

"I'm trying to deploy an ECS Cluster with a task to run a docker container with a web application. I found the methods to create the cluster, registering the task definition and running it, but I can't find how to choose the instance type that I want to be used and also the disk size.In this post ( https://dev.to/raphaelmansuy/deploy-a-docker-app-to-aws-using-ecs-3i1g ) the person is making a deploy from the AWS Console and he is able to choose the instance type and disk type and size.Is it possible to do the same using the Javascript SDK ( https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/ECS.html ). I can't find where.If it's not possible, do you suggest me some other strategy to make this work? Basically what I need is to create the whole stack triggered from an admin page. May be I trigger the process from a lambda function or directly from the back-end of my application.Thanks.FollowCommentcbevilaqua a year agoAs I can understand now, this option does really does not exist and we must change the memory and cpu requirements for our application, doesn't matter the EC2 instance type, please correct me if I'm wrong. Thanks!Share"

Answer: "0Accepted AnswerHi,About the post screenshots - If you look at the main ECS cluster creation screen, when you choose EC2, it says it will create for your an Autoscaling Group of EC2 instances, then, at the Task creation step, you'll be choosing the EC2 provider to deploy the tasks on the EC2 AutoScaling Group.If you have to reproduce the same setup from the JS SDK, first you need to create an AutoScaling Group from the EC2 APIThen you create a Capacity Provider from the ECS API and reference the Autoscaling Group ARN previously created.CommentShareJulienanswered a year agocbevilaqua a year agothank you so muchShare"


Question: AWS GLUE - java.sql.SQLIntegrityConstraintViolationException: Duplicate entry Error

"I was doing a POC using Glue to Migrate data from RDS MySql to RDS Postgres. I have created Connectors to both source and target, and a crawler which connected to source. Then  created a job and tried to migrate data with out any transformation and started getting java.sql.SQLIntegrityConstraintViolationException: Duplicate entry errorjava.lang.reflect.Constructor.newInstance(Constructor.java:423)\n\tat com.mysql.cj.util.Util.handleNewInstance(Util.java:192)\n\tat com.mysql.cj.util.Util.getInstance(Util.java:167)\n\tat com.mysql.cj.util.Util.getInstance(Util.java:174)\n\tat com.mysql.cj.jdbc.exceptions.SQLError.createBatchUpdateException(SQLError.java:224)\n\tat com.mysql.cj.jdbc.ServerPreparedStatement.executeBatchSerially(ServerPreparedStatement.java:385)\n\tat com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchInternal(ClientPreparedStatement.java:435)\n\tat com.mysql.cj.jdbc.StatementImpl.executeBatch(StatementImpl.java:794)\n\tat org.apache.spark.sql.jdbc.glue.GlueJDBCSink$.saveBatch(GlueJDBCSink.scala:400)\n\tat org.apache.spark.sql.jdbc.glue.GlueJDBCSink$.$anonfun$saveTable$4(GlueJDBCSink.scala:77)\n\tat org.apache.spark.sql.jdbc.glue.GlueJDBCSink$.$anonfun$saveTable$4$adapted(GlueJDBCSink.scala:77)\n\tat org.apache.spark.sql.jdbc.glue.GlueJDBCSink$.savePartition(GlueJDBCSink.scala:261)\n\tat org.apache.spark.sql.jdbc.glue.GlueJDBCSink$.$anonfun$saveTable$3(GlueJDBCSink.scala:77)\n\tat org.apache.spark.sql.jdbc.glue.GlueJDBCSink$.$anonfun$saveTable$3$adapted(GlueJDBCSink.scala:76)\n\tat org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1020)\n\tat org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1020)\n\tat org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2278)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:131)\n\tat org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat ****java.lang.Thread.run(Thread.java:750)\nCaused by: java.**sql.SQLIntegrityConstraintViolationException: Duplicate entry '00002b0b-1a34-3319-b003-fb073fb8248d' for key 'transformer_fragment.PRIMARY'\n\tat ******com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:117)\n\tat com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)\n\tat com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:637)\n\tat com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:418)\n\tat com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1092)\n\tat com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1040)\n\tat com.mysql.cj.jdbc.ServerPreparedStatement.executeBatchSerially(ServerPreparedStatement.java:357)\n\t... 19 more\n","Accumulator Updates":[{"ID":130,"Update":"105","Internal":false,"Count Failed Values":true},{"ID":132,"Update":"0","Internal":false,"Count Failed Values":true},{"ID":139,"Update":"0","Internal":false,"Count Failed Values":true},{"ID":140,"Update":"1","Internal":false,"Count Failed Values":true},{"ID":141,"Update":"0","Internal":false,"Count Failed Values":true},{"ID":142,"Update":"0","Internal":false,"Count Failed Values":true},{"ID":143,"Update":"1135","Internal":false,"Count Failed Values":true},{"ID":144,"Update":"0","Internal":false,"Count Failed Values":true},{"ID":145,"Update":"1","Internal":false,"Count Failed Values":true}]},"Task Info":{"Task ID":13,"Index":5,"Attempt":1,"Launch Time":1655403050036,"Executor ID":"1","Host":"172.31.22.88","Locality":"NODE_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":1655403050162,"Failed":true,"Killed":false,"Accumulables":The following are the values I have used while creating the job,Type : SparkGlue Version -  Glue 3.0 Supports spark3.1, Scala2, Python3Language - Python3Worker Type - G.1XAutomatically Scale the number of workers - FalseRequested number of workers - 2Generate Job Insights - TrueJob Bookmark - EnableNumber of retries - 1job timeout - 2880Really appreciate any help !FollowComment"


Question: Sent Notification when EBS volume snapshot permission changed to public

"Hi Team,Could you please help me to send an alert when EBS volume snapshot permission changed to public.We don't want others to create volume from the snapshot.FollowComment"

Answer: 3You can use AWS Config to detect non-compliant and notify via Amazon SNS. Using "ebs-snapshot-public-restorable-check" managed rules seems to fit your purpose.ebs-snapshot-public-restorable-check - AWS ConfigHere is an example of a security group.Receive notifications for a non-compliant AWS resource using AWS ConfigCommentSharejhashimotoanswered 5 months agoEXPERTkentradreviewed 5 months agorePost-User-7169480 4 months agoThanks @jhashimoto and kentrad for quick answerSharerePost-User-7169480 3 months ago@All with below configuration for existing non-complaint resources not getting alert. But if the new resource is created and it's compliance changed to not-compliance then only receiving the  alert. Is there a way to get alert for exiting non-complaint resources under config ruleReceive notifications for a non-compliant AWS resource using AWS ConfigShare


Question: Can Data Pipelines be used for running Spark jobs on EMR 6.5.0?

"Hi,I have a problem in that I make heavy use of EMRs, and I orchestrate their use with Data Pipeline - multiple daily runs are automated and EMRs are launched and terminated on conclusion.However, I'd now like to make use of EMR 6.X.X releases via Data Pipelines, rather than EMR 5.X.X releases I'm currently using.  This is for two main reasons:Security compliance: the latest EMR 6.X.X release have less vulnerabilities than the latest EMR 5.X.X releasesPerformance/functionality: EMR 6.X.X releases perform much better than EMR 5.X.X releases for what I'm doing, and have functionality I prefer to useHowever...the current documentation for Data Pipeline says the following regarding EMR versions:AWS Data Pipeline only supports release version 6.1.0 (emr-6.1.0).Version 6.1.0 of EMR was last updated on Oct 15, 2020...it's pretty old.Now, if I try and use an EMR version > 6.1.0 with Data Pipeline, I get the issue that has already been raised here i.e. during initial EMR cluster bring up via the Data Pipeline, there's a failure that renders the cluster unusable.  It looks like a malformed attempt to create a symbolic link to a jar by one of the AWS scripts:++ find /usr/lib/hive/lib/ -name 'opencsv*jar'+ open_csv_jar='/usr/lib/hive/lib/opencsv-2.3.jar/usr/lib/hive/lib/opencsv-3.9.jar'+ sudo ln -s /usr/share/aws/emr/goodies/lib/emr-hadoop-goodies-3.2.0.jar /mnt/taskRunner/oncluster-emr-hadoop- goodies.jar+ sudo ln -s /usr/share/aws/emr/goodies/lib/emr-hive-goodies-3.2.0.jar /mnt/taskRunner/oncluster-emr-hive-goodies.jar+ sudo ln -s /usr/lib/hive/lib/opencsv-2.3.jar /usr/lib/hive/lib/opencsv-3.9.jar /mnt/taskRunner/open-csv.jarln: target â€˜/mnt/taskRunner/open-csv.jarâ€™ is not a directoryCommand exiting with ret '1'So - I guess my questions are:Is there a way to work around the above so that Data Pipelines can be used to launch EMR 6.5.0 clusters for Spark jobs?If there isn't, is there a different way of automating runs of EMR 6.5.0 clusters, other than writing my own script and scheduling that to bring up the EMR cluster and add the required jobs/steps?Thanks.FollowComment"

Answer: "0With this banner in the console, "Please note that Data Pipeline service is in maintenance mode and we are not planning to expand the service to new regions. We plan to remove console access by 05/12/2023", I would focus development of new workloads in Managed Apache Airflow. Using this operator, you can kick off a dag that launches an EMR cluster and terminates it one job is complete, similar to what we used to do in data pipeline.CommentShareEmananswered a month ago"


Question: VPN tunnel switching setting

"Local firewall and VPC set up VPN,the ip of tunnel 1 is first prority and ip of tunnel 2 is second prority in the Local firewallWhen there is a problem with the ip of tunnel 1,AWS will failover to the tunnel 2 to keep communicate with local firewall.But when tunnel 1 is restored, AWS will not switch back to tunnel 1 to communicate with the firewall,but will continue to remain in tunnel 2.How to set up when tunnel 1 is restored, aws can also switch back to tunnel 1 tocommunicate with the local firewallFollowCommentTushar_J EXPERT8 months agoAre you using Static route based VPN or BGP based Dynamic VPN?Share"

Answer: "0Accepted AnswerHi,Simply put, the VPN tunnel is randomly chosen by AWS and is called the preferred tunnel. If your AWS VPN connection (static route type) has an active/active configuration (both tunnels are up), you cannot configure your preferred specific tunnel in AWS to send traffic.But, for dynamic AWS VPN connections (Active/Active), you can set the customer gateway device to prefer one VPN tunnel over the other by leveraging the order of preference criteria:Advertising more specific prefix over preferred tunnelAdvertising shorter ASPATH over the preferred tunnelSetting lower MED values over the preferred tunnelHope this helpsCommentSharejcvipanswered 8 months ago0Hello,See this Knowledge center article:https://aws.amazon.com/premiumsupport/knowledge-center/vpn-configure-tunnel-preference/Note also below from our documentationWe recommend that you use BGP-capable devices, when available, because the BGP protocol offers robust liveness detection checks that can assist failover to the second VPN tunnel if the first tunnel goes down. Devices that don't support BGP may also perform health checks to assist failover to the second tunnel when needed.CommentShareEXPERTTushar_Janswered 8 months ago"


Question: S3 lifecycle rule / ExpiredObjectDeleteMarker': True failed on chalice application

"HiInside a chalice application, I have created a class for handling S3 bucket. I add a method to add a default lifecycle rule which will delete all objects after 1 day. (quite useful for a dev environment)def put_default_lifecycle_rule(self, id_name='default_lifecycle_rule'):   try:        s3 = boto3.resource('s3')        bucket_lifecycle_configuration = s3.BucketLifecycleConfiguration(self._bucketname)        response = bucket_lifecycle_configuration.put(            LifecycleConfiguration={                'Rules': [                    {                        'Expiration': {                            'Days': 1                        },                        'ID': id_name,                        'Status': 'Enabled',                        'Filter': {'Prefix': ''},                        'NoncurrentVersionExpiration': {                            'NoncurrentDays': 1                        },                        'AbortIncompleteMultipartUpload': {                            'DaysAfterInitiation': 1                        }                    }                ]            }        )    except Exception as e:        logging.error(e)        raise e    return responseWhen I add "ExpiredObjectDeleteMarker': True" in expiration block, I got an XML error (malformed).Official documentation says to put this in this block.Any idea ?Thanks guysFollowComment"

Answer: "1Accepted AnswerHi there!You cannot add the ExpiredObjectDeleteMarker configuration within the same expiration rule where you specify a Days or Date value.Try having two separate rules for the LifeCycleConfiguration. Something like this:response = bucket_lifecycle_configuration.put(            LifecycleConfiguration={                'Rules': [                    {                        'Expiration': {                            'Days': 1                        },                        'ID': 'Rule01',                        'Status': 'Enabled',                        'Filter': {'Prefix': ''},                        'NoncurrentVersionExpiration': {                            'NoncurrentDays': 1                        },                        'AbortIncompleteMultipartUpload': {                            'DaysAfterInitiation': 1                        }                    },                    {                        'Expiration': {                            'ExpiredObjectDeleteMarker': True                        },                        'ID': 'Rule02',                        'Status': 'Enabled',                        'Filter': {'Prefix': ''}                    }                ]            }        )CommentShareFabio_Lanswered a year agoJack-Angel-369 a year agoHi )))Yes I should have thought about that.This is the solutionThanks FabioShare0The documentation for S3.BucketLifecycleConfiguration.put() states:ExpiredObjectDeleteMarker (boolean) --Indicates whether Amazon S3 will remove a delete marker with no noncurrent versions. If set to true, the delete marker will be expired; if set to false the policy takes no action. This cannot be specified with Days or Date in a Lifecycle Expiration Policy.-- https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.BucketLifecycleConfiguration.put(emphasis mine)Specifying  'Expiration': { 'Days': 1  } causes current versions to expire after one day.The 'NoncurrentVersionExpiration': {                             'NoncurrentDays': 1                         }block specifies that non-current versions are deleted one day after Expiration.CommentShareChandler Howellanswered a year agoJack-Angel-369 a year agoHi thanks for the replyYes sure this is the code I wrote. The question is about the expired object delete marker)))I have read the doc before writing the code.With the AWS console I can set up the lifecycle rule for expired object delete markers but not on code...When some buckets can't be delete with the console, the only solution is to put a lifecycle rule for deletingcurrent versionsnon current versionsexpired object delete markersaccording to AWS documentation...So the issue remainsThanks for the helpShare"


Question: TLS certificate still pending even though domain and nameservers correct in route 53

"Hello,I am trying to obtain a valid tls certificate through ACM, however, the certificate is still pending. I registered my domain (let's use example.com as the example) in Route 53, and the 4 name servers under route 53 > registered domains > example.com > hosted zone details matches the 4 name servers in my hosted zone under route 53 > hosted zone > example.com - under the NS record. Also, I have 3 other records under r 53 > hosted zone > example.com, them being an A record which routes to my s3 bucket, an SOA record which routes to the first name server of the NS record, and a CNAME record with 'record name' of 'www.example.com' and routing to 'example.com'.I transferred this domain from another host a month ago, and aws support told me yesterday that the name servers hadn't transferred over correctly, so yesterday I fixed the name servers in route 53 > registered domains > example.com > hosted zone details to the name servers that were within r 53 > hosted zone > example.com - under the NS record. Also, I don't think it took 48 hours for the dns settings to update because they are already updated under route 53 > registered domains > example.com > hosted zone details.So domain and name servers are all correct, and the aws documentation here says that if all checks out it should take 30 min max to issue a valid certificate. However, it has been almost 24 hours and the certificate for example.com is still pendingFollowComment"

Answer: "1Accepted AnswerYou need to make sure the CNAME records have been created in order for ACM to validate the Domain. Within the ACM certificate, it will define the CNAME record thta needs creating.You have not mentioned you have created this CNAME record!Also make sure the domain registrar records point to the name servers for this domain.CommentShareGary Mcleananswered a month agoJeff F a month agoOh ok I see. I manually added the CNAME records to r 53 > hosted zones > example.com yesterday and now it works!Question though - Do I need only one hosted zone for example.com, or do I need a hosted zone both for example.com and www.example.com ? I'm asking because within the hosted zone r 53 > hosted zones > example.com I now have 3 CNAME records :'record name' is www.example.com with 'value/route to' being example.comrandomcharacters1.example.com routing to randomcharacters2.tftwdmzmwn.acm-validation.aws.randomcharacters3.www.example.com routing to randomcharacters4.tftwdmzmwn.acm-validation.aws.In the hosted zone for r 53 > hosted zones > www.example.com I only have one CNAME record, and I think this CNAME record auto-populated somehow. It just happens to be one of the CNAME records from example.com:randomcharacters3.www.example.com routing to randomcharacters4.tftwdmzmwn.acm-validation.aws.Share"


Question: Failure in Cloudformation template [ CommandRunenr] while running CLI command for Cloudtrail

"Hi Guys,I am trying to run CLI command to update a CloudTrail  but stack is getting failed. Requirement is to apply advanced data events to existing CloudTrail.Please find below details of CF template:CF templateAWSTemplateFormatVersion: 2010-09-09Resources:UpdateTrail:Type: AWSUtility::CloudFormation::CommandRunnerProperties:Role: ec2-role-nameSubnetId: subnet-XXXXXXXXXLogGroup: log-group-nameCommand: aws cloudtrail put-event-selectors --trail-name XXXX --region XXXX --advanced-event-selectors....ErrorResource handler returned message: "Either the command failed to execute, the value written to /command-output.txt was invalid or the Subnet specified did not have internet access. The value written to /command-output.txt must be a non-empty single word value without quotation marks. Check cloud-init.log in the LogGroup specified for more information."CLI commandaws cloudtrail put-event-selectors --trail-name XXXX --region XXXX--advanced-event-selectors'[{"Name": "S3EventSelector","FieldSelectors": [{ "Field": "eventCategory", "Equals": ["Data"] },{ "Field": "resources.type", "Equals": ["AWS::S3::Object"] },{ "Field": "eventName", "Equals": ["PutObject","DeleteObject"] },{ "Field": "resources.ARN", "StartsWith": ["arn:aws:s3:::XX","arn:aws:s3:::XX"] }]}]'Note : Command runs successfully in CLI.pre-requisites for commandRunner is installed.Also,  Subnet specified does have internet access.I sense, it might be the issue with command format or may be something else.Any assistance would be appreciated.ThanksFollowComment"

Answer: "0Hello, as indicated here, you must write the command's output to a reserved file called /command-output.txt. Now, per the code, the above error can occur if SSM PutParameter failed due to invalid value in /command-output.txt.I tested the template in my lab and noticed PutParameter event in CloudTrail failed with the following error:"1 validation error detected: Value at 'value' failed to satisfy constraint: Member must have length greater than or equal to 1.Then, observed the following error in a log named i-****/cloud-init-output.log in the CloudWatch log group that I configured using the LogGroup property in AWSUtility::CloudFormation::CommandRunner resource type.aws: error: argument --event-selectors is requiredContents of /command-output.txt =As seen above, contents of the /command-output.txt is empty.This --event-selectors argument error occurs if the session uses AWS CLI v1 and is not seen in AWS CLI v2. Observed the Amazon Linux AMI used by CommandRunner does not have the latest version of the AWS CLI installed at present, therefore we will need to update the AWS CLI on the instance to successfully execute the above put-event-selectors command. I used the following snippet to fix the issue:....Resources:  CommandRunner:    Type: 'AWSUtility::CloudFormation::CommandRunner'    Properties:      Command: >-        curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip";        unzip awscliv2.zip;        sudo ./aws/install;        aws --version;        aws cloudtrail put-event-selectors --trail-name XXX --region XXXX --advanced-event-selectors '[ { "Name": "S3EventSelector", "FieldSelectors": [ { "Field": "eventCategory", "Equals": ["Data"] }, { "Field": "resources.type", "Equals": ["AWS::S3::Object"] }, { "Field": "eventName", "Equals": ["PutObject","DeleteObject"] }, { "Field": "resources.ARN", "StartsWith": ["arn:aws:s3:::XX","arn:aws:s3:::XXX"] } ] } ]' --output text > /command-output.txt      Role: yourrole      LogGroup: log-group-nameHope it helps!CommentShareSUPPORT ENGINEERSreshtha_Kanswered 6 months ago"


Question: A certificate for this thing could not download

"Hi, I am a new user for IOT.When I try to built my first thing, I can create the certificate. However, I could not download the "A certificate for this thing", because it was blocked by Windows.message: xxxxxx-certificate.pem.crt was blocked because this type of file can harm your device.I try to disable the Windows Security but without improvement.Please help to teach me.Thank you.GordonFollowComment"

Answer: "0Hi,It could be operating system specific error. You may be able to choose either Save anyway or Discard.  If that is not the case, use aws CLI to generate certificate and attach to thing.Hope this helps,CommentShareAWS-User-3247576answered 2 years ago0Hi,The other method could not work. I will try to use aws CLI to generate the cert.Thank you.GordonCommentSharehorseHKanswered 2 years ago"


Question: Custom Entity Recognition Job Not Working with .txt files

"I am trying to run a custom entity recognition job on a couple text files. I trained the recognizer using PDF annotation, but I am sending .txt files into the job.So, I am getting this error: DOCUMENT_CORPUS_SIZE_LESS_THAN_MINIMUM: Document corpus size is less than the minimum requirement: 5120 bytes.It is true, the text files in my input folder are only 1.2 KB, but I am not sure how to proceed from here. I tried changing the options to "ONE_DOC_PER_LINE", but that gave another error saying that is unsupported for semi-structured data.UPDATE: I added extra text to the file, and it went through, but the entity recognition is very poor. Before this, I was submitting PDFs to the analysis job, and it was working well. However, I need to submit text documents because I want to eventually create an endpoint to use for this job. What do I do?FollowComment"

Answer: 0This was a bug in our validation logic which did not allow for a total input corpus size of less than 5kb for a semi-structured trained model. This issue is being addressed and should be deployed worldwide by 07/29.CommentSharednlenAWSanswered 10 months ago


Question: Is this the right way to backup my ec2 instance? (+some questions)

"Hi, I am an AWS EC2 free-tier user and I would like to know if the way I backup is a correct or not.So, the way I currently backup isMake a snapshot from an ebs volumeMake an AMI from the snapshotMake a new instance from that AMITerminate the pervious instance and detach the volume from the instance as wellAttach the new instance with a new volume to my domain(www.blahblah)Change my domain public ip to the new instance public ip.I have done my backup this way but I am not sure if the way I do it is correct or not. Please let me know if there is anything that I am missing.+I have some few more questions other than the procedure of the ebs snapshot backup. My questions areHow to check how much volume is left in the current ebs storage? (I want to check for the free space in my ebs. I have heard that we need to size up the ebs volume when there are more files and more snapshots in the volume. 'df -hT /dev/xvda1' command didn't show the ebs volume that are free.)Do I get a notification when my ebs storage is almost full?Do I need to stop my instance when I am making a snapshot? (For the current backup, I didn't stop my instance when I took a snapshot and It was fine. but on the internet, people seem to say it is better to stop the instance before taking a snapshot. so I'm confused.)Is using S3 mandatory to store ebs snapshots? (I'm currently not using S3 but I have seen people storing ebs snapshots in their S3 storage. so I'm not sure if If I need to store them in S3.)Thanks!FollowComment"

Answer: "0Take a look at this blog for how to automate backing up your EBS volumes: Automating Amazon EBS snapshot management using Data Lifecycle Manager.You don't need to create the AMI and boot up a new instance unless your are trying to restore the volume (steps 2-6).You have to use the OS/Filesystem commands to determine usage, like 'df'.No, not by default. You can install the CloudWatch agent to monitor OS level resources and get alerts, see: Collect metrics and logs from Amazon EC2 instances and on-premises servers with the CloudWatch agentNo, no need to stopS3 is used by the EBS service, you can't control this or see the bucket where your snapshots are stored.CommentShareEXPERTkentradanswered 5 months agorePost-User-9594951 5 months agoHi, Thank you for the quick response.I am a bit confused by "You don't need to create the AMI and boot up a new instance unless your are trying to restore the volume (steps 2-6).".If I don't create an AMI from the snapshot, how do I backup my current instance?In snapshot action, we havecreate volumecreate imagemanage fast snapshot restore etcIf I don't create an image(AMI) to backup, should I create a volume from that snapshot and attach it to the current instance? Or should I do the fast snapshot restore? (but I have heard that fast snapshot restore is expensive though...)Thanks!SharerePost-User-9594951 5 months agoI got it, I created a volume from the snapshot and it worked out fine.Thanks!!Share0Adding some information to question (3):Do you need to stop your instance before doing a snapshot? No, you don't. However the snapshot process is not aware of the state that the filesystem on the disks is in. Nor is it aware of the state that any application data might be in.For example, if the operating system is in the middle of writing a file but has not yet updated the file allocation table (or whatever the equivalent is for that particular operating system) and you take a snapshot right then the filesystem in the snapshot is in a state which isn't (for want of a better word) "stable". That means when you restore from snapshot the new volume will be in the same state. Most operating systems will recover from this gracefully but you might lose the contents of the file. Maybe. It depends.If you're running a database on the instance and the database is in the middle of committing a transaction and you take a snapshot at that exact moment then the database will be in an unknown state in the snapshot. Same thing applies - the database engine might automatically fix that problem on the next startup (on the new volume); or it might not. And the data might be lost. Or not. Again, it depends.So it is safer to stop your instance to do a snapshot? Yes. If the filesystem is relatively stable but you're running a database then you might want to stop write operations to the database while you're doing the snapshot - that could be done at an application level or within the database. That way you know that the database is fully written to disk before the snapshot is taken.CommentShareEXPERTBrettski-AWSanswered 5 months ago"


Question: Lumberyard and Steam

"Hey,How does it work for if i make a game and decide to host the game on steam? When i host, it on there, they host the Multiplayer servers. Would that be allowed? How would it work with how Lumberyard is free, but you use aws to host the servers.Follow"

Answer: "0According to Lumberyard's FAQ:---Quote----Q. Can my Lumberyard game connect to services like Steamworks, Xbox Live, PSN, Apple Game Center, Google Play Games, or console social services?A. Yes. Your game may read and write data to platform services and public third-party game services for player save state, identity, social graph, matchmaking, chat, notifications, achievements, leaderboards, advertising, player acquisition, in-game purchasing, analytics, and crash reporting.Q. Can my game use an alternate web service instead of AWS?A. No. By “alternate web service” we mean any non-AWS web service that is similar to or can act as a replacement for Amazon EC2, Amazon Lambda, Amazon DynamoDB, Amazon RDS, Amazon S3, Amazon EBS, Amazon EC2 Container Service, or Amazon GameLift. You can use hardware you own and operate for your game servers.Q. Is it okay for me to use my own servers?A. Yes. You can use hardware you own and operate for your game. It’s also ok for your users to connect peer-to-peer, and for users to host personal servers (as long as they’re not operating as a service provider business).---End Quote----So you can connect to Steamworks, but you can't use Steam as host service for your multiplayer game, it is not allow because it's operating as a service provider business.Hope I understand your question correctly.SharerePost-User-6460635answered 5 years ago0The steam part is an issue, as your forced to pay for the servers while hosing your game on there, so if you are putting your game on there, then you would end up paying for aws and steam. Steam its not like its optional as its part of the take they take and its already 30-40% they take. Steam can not be defined by an alternaitive web service as its just the multi player servers and you cant use them unless you have a game sold on there. I would like to get input from a LY Staff on this rather than a quote....SharerePost-User-1354821answered 5 years ago0Hi @REDACTEDUSERSharerePost-User-0479726answered 5 years ago"


Question: unable to start cloudshell

"Unable to start the environment. To retry, refresh the browser or restart by selecting Actions, Restart AWS CloudShell. I am getting that error even after following the instruction and I am using ohio region. I alread tried using other regions with cloudshell like tokyo but still facing this issue. I am also using my root account and still the same issue.FollowCommentAWS-User-2148182 a year agoI have the same problem.ShareMaulik_Makadiya a year agoAnother thread already running with the same problemhttps://repost.aws/questions/QUufc-elFvSvqYlbJw95RI6Q/unable-to-start-the-environment-to-retry-refresh-the-browser-or-restart-by-selecting-actions-restart-aws-cloud-shellSharerePost-User-5406173 3 months agoI can't use cloudshell, same problem.Share"

Answer: "0Make sure there are no Service Control Policies blocking the account from using CloudShell. The policy simulator may help narrow down what is blocking if it is a permissions issue. https://policysim.aws.amazon.com/CommentShareEXPERTkentradanswered a year agoAWS-User-2148182 a year agoPolicy simulator shows I should have access to CloudShell but I don't have.Share0The two regions you mentioned are supported refer to AWS CloudShell endpoints and quotas. If this was a permission error I would expect to see a different error. To be sure check that your current permissions includes necessary permissions, for example, AWSCloudShellFullAccess.This could be browser related. CloudShell supports below so check that your browser meets these requirementsGoogle ChromeLatest three major versionsMozilla FirefoxLatest three major versionsMicrosoft EdgeLatest three major versionsApple Safari for macOSLatest two major versionsCommentShareRoBanswered a year ago0I'm having the same issue even though I'm using Root User and I don't think it's a browser problem because another root user can be used normally. I tried to simulate and the permission column said allowed both CloudShell Create & StartEnvironment. How should I do?Thank you very much.CommentSharePlAwAnSaIanswered a year ago-1So i was able to resolve this issue with the help of AWS repost user @Maulik_Makadiya.Few things to try to create CloudShell environment:Time Synchronization: Make sure Your machine time is accurate. It means its correct based on world time. did you try from another machine to see if its working there? may be any time sync related issue?Check in different regions..check AWSCloudShellFullAccess policy to ensure it has below JSON data.{    "Version": "2012-10-17",    "Statement": [        {            "Action": [                "cloudshell:*"            ],            "Effect": "Allow",            "Resource": "*"        }    ]}Check in different browser to see if it works in different browser or not. https://docs.aws.amazon.com/cloudshell/latest/userguide/troubleshooting.htmlDid you deleted cloudshell home directory or something? Try after resetting home directory. But it can DELETE all your data exists in home directory. https://docs.aws.amazon.com/cloudshell/latest/userguide/vm-specs.html#deleting-home-directorycheck if any DENY policy created for it. remove that.It is possible your account is not 100% verified. Try this: Create a CloudFront distribution if you get the below error it confirms your account is unverified or if you can create 2 distributions and can't create the 3rd one.Your account must be verified before you can add new CloudFrontresources. To verify your account, please contact AWS Support(https://console.aws.amazon.com/support/home#/ ) and include thiserror message.Click the support linkNavigate to:Support / New case / Service limit increaseLimit type:CloudFront DistributionsIn Requests select:Limit: Web Distributions per AccountNew limit value: <TYPE_YOUR_NEW_VALUE_HERE>MY CASE: In my case, I had 2 distributions, wanted to create 3rd, but couldn't. So I have put as <TYPE_YOUR_NEW_VALUE_HERE> a number 10.Note: If nothing works choose last option as your last resort to confirm your account is verified.hope it helps.CommentShareAWS-User-3586868answered a year ago"


Question: How to : Multiple core devices and subscriptions across all core devices

"Hi,What I am trying to achieve is having a group of multiple core devices, where the lambda functions gets executed, and these devices will have to send and receive messages across these devices.these are my questions regarding the same,Can I have a group with only core devices?In the above mentioned case, the same lambda functions will be running on all the cores, so in this case, how to create the subscriptions as the target and source will be the lambda functions?Please advise on the same.Thanks.Edited by: mockingBird on Jan 11, 2019 12:22 AMFollowComment"

Answer: "0Hi MockingBird,No. You can have a group with only a core. However, you can only have one core per group.If you launch multiple groups you would setup a subscription from Lambda -> IoT Cloud and from IoT Cloud -> Lambda for each group.Can you tell me a bit more about your use case and what you're trying to do?Thanks,KeagenCommentShareKR-AWSanswered 4 years ago0Hi Keagen,Thanks for your reply.What we are trying to achieve is, we have a set of sensors, and we want to crunch the data there itself,  that means we prefer to execute the lambda in each of these device, and all these devices should communicate each other to share some data for their processing, that means all these devices need to be in a group.So could you please suggest a proper way to do the sameThanks,CommentSharemockingBirdanswered 4 years ago0Some more clarification neededCommentSharemockingBirdanswered 4 years ago0Hi mockingBird,Currently there isn't an easy way to communicate from core A to core B without going through the IoT Cloud. You can make core A a device for core B and vice versa. However, MQTT from device to core is only QoS 0. You'll need to handle missing/lost messages in your application.Please let me know if you have any other questions.Thanks,KeagenCommentShareKR-AWSanswered 4 years ago0Thanks a lot KaeganCommentSharemockingBirdanswered 4 years ago"


Question: Index in Lightsail Object Storage

Is there a way to enable static hosting with --index-document in Lightsail object storage?FollowComment

Answer: "0Hi, thanks for using Lightsail. I'm sorry to say that, currently, Lightsail object storage does not support this feature.LeslieCommentShareAWS-Leslieanswered 2 years ago"


Question: Custom DNS for MSK

"Hi, how can I "mask" an MSK endpoint with a custom domain? I'm trying to find a solution that won't cause SSL issues.Thanks!FollowComment"

Answer: "1[Updated as this answer was incorrect]CommentShareAnanth Tirumanuranswered 4 months agoEdbE 4 months agoCan you elaborate, at which point this provides connectivity with MSK/Kafka?Share0Accepted AnswerCustom domain names are not supported at this point.The only work around possible would involves non-data PLAINTEXT access for bootstrapping (fetch metadata request) and regular SSL for data-in-transfer encryption.Update advertised listeners to have port 9094 for PLAINTEXT protocolDefine R53 A-record with custom domain name resolved to an NLB with targets to all brokers to port 9092 (PLAINTEXT)a. create certificates in ACM and deploy on NLB listenersb. define NLB listeners as TLS to terminate TLS connection at NLBThis will let you use custom domain name with TLS traffic.Limitation of this approach is that it won't work with SASL (SCRAM, IAM), because listeners don't have SASL implementation mechanisms.CommentShareEdbEanswered 4 months ago"


Question: g5.48xlarge instance launch failed

"Hi, I am trying to launch a g5.48xlarge instance. But it kept saying my configuration was not supported.My configuration is listed below:Region: US-WEST-2 (Oregon)AMI: I choose one Ubuntu AMI from the "Community AMI" from verified provider and the AMI explicitly says it support G5 instance.Networking: I just choose the default existing groupStorage: I tried GP3, GP2, IO1, IO2 (requiring 2200 GB)Can anyone help me here? Also, I have encountered issues in launching P4 instance and cannot find any tutorial documentation. It is good to provide templates for each instance, so people know what should choose and what should not.Thanks.FollowComment"


Question: I can not sign into my AWS console

"I can not log into my AWS console AWS keeps sending a verification code to an old number I no longer have access to. What can I do?FollowCommentD G 9 days agoJust checking base with you about the answer provided.Let me know if you have any issues with this, or if it helps you then please accept my answer after you've tried it out - it would be much appreciated! Good luck :)Share"

Answer: "0Open up a case here: https://support.aws.amazon.com/#/contacts/aws-account-support/CommentShareEXPERTkentradanswered 12 days agoEXPERTBrettski-AWSreviewed 11 days ago0One is this a personal account? If so you could work to attain root privileges.https://docs.aws.amazon.com/signin/latest/userguide/troubleshoot-forgot-root-password.htmlOnce you have that you have "root domain" on the account in question.However just like those that messaged above. It is FAR better to sign in to submit.It's much faster to get support via the console, please click the button below to sign in and submit your requestSign in to submit your billing and account support requestLet me know if you have any issues with this, or if it helps you then please accept my answer after you've tried it out - it would be much appreciated! Good luck :)CommentShareD Ganswered 10 days ago"


Question: S3 sync command fails with "Connection broken" messages during reliable connection

"When attempting to download from S3 to a Windows 11 system using the AWS CLI s3 sync command, some files fail with the following error message:download failed: s3://bucketname/filename to D:\bucketname\filename ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))There were no connectivity issues during the transfer and other files from the same bucket during the same sync session are written as expected.aws-cli/2.7.13 Python/3.9.11 Windows/10 exe/AMD64 prompt/offWindows Terminal, Version: 1.13.11432.0Microsoft Windows 11 Enterprise, 10.0.22000 N/A Build 22000FollowCommentrePost-User-8683112 a year agoFor reference, similar behavior was reported by other users in Stack Overflow four years ago: https://stackoverflow.com/questions/51310901/aws-cli-s3-sync-error-an-existing-connection-was-forcibly-closed-by-the-remoteSharerePost-User-8683112 a year agoUsing the same sync command syntax, hardware, bucket, files, destination, and host OS, I was able to work-around the defect by running the AWS CLI from Ubuntu on WSL. The AWS client on Ubuntu/WSL was:aws-cli/2.7.12 Python/3.9.11 Linux/5.10.16.3-microsoft-standard-WSL2 exe/x86_64.ubuntu.20 prompt/offConsidering anti-virus as a possible cause per the Stack Overflow in the previous comment, the same anit-virus was running at the time both the Windows and WSL AWS CLIs were executing. The host OS was running the default Microsoft Defender Antivirus distributed with Microsoft Windows.Share"


Question: Transfer files (1GB to 2G) from web url to S3 on a schedule

My customer has a use case to access file from a web url periodically and upload to S3. This is currently implemented using lambda with schedule trigger and is working well for smaller files. But larger files need higher memory settings in lambda. Is there a way to directly upload to S3 without storing in memory or are there better serverless implementation options for this use case?FollowComment

Answer: "0Accepted AnswerI've previously used a very small container running in ECS for this. ECS has scheduling in-built, so you simply set it the schedule appropriately, and it runs regularly thereafter.The container I used was built on Alpine, and used wget to pull the files I needed.CommentShareEXPERTAndrew L.answered 3 years ago"


Question: DNS Zone TXT Records limited to 255 characters - need more

"Good day,New to Amazon Lightsail here. Migrating my instance off another vendor. Instance, firewall, static ip - all went ok. Now setting up my DNS Zone. One of my DNS TXT records is about 400 characters long. It is a google-generated DKIM record which contains key-value pairs including a public key.When I try to add the record in the DNS Zone Editor, an error pops up: "Each line must be between 0 and 255 characters and contain only printable ASCII characters". But since the 400 characters contain a public key, I don't think I should break it up. Is there any solution to this?Thanks -RicardoEdited by: RigaDorje on Jun 17, 2019 7:42 PMEdited by: RigaDorje on Jun 17, 2019 7:43 PMFollowComment"

Answer: "0Accepted AnswerStrings in SPF and TXT records should be no longer than 255 characters. However to get around this limitation, per RFC 4408 a TXT or SPF record is allowed to contain multiple strings, which should be concatenated together by the reading application.-- https://kb.isc.org/docs/aa-00356CommentShareDavid Ganswered 4 years ago0Hi David G,Thanks for the information and the reference link. I broke up the long string by making several lines each less than 255 chars (separated using the Enter key) -- trusting that the reading application will reassemble it back correctly. The editor saved my record.I then used Mxtoolbox's DKIM Lookup tool https://mxtoolbox.com/dkim.aspx to query the DNS record. It interpreted the DKIM record correctly.Thank you! -RicardoCommentShareRigaDorjeanswered 4 years ago"


Question: AWS charges for non-existent service

"Hi,on my bill I see high costs for a service called "Elastic Cloud (Elasticsearch Service) (sold by Elastisearch Inc)".I searched for a service with that name, but it seems, there is no such thing. I searched extensively. There are similar things like "CloudSearch" or "Elastic * Cloud", but I don't have anything there. If I open CloudSearch in any region it just displays "You have no search domains in *".Is there a hidden menu somewhere?Thanks,STefanFollowComment"

Answer: "1The "sold by ElasticsearchInc" makes me think this is from the AWS Marketplace. If you to the Marketplace (you can search for it in the AWS Console search bar) it should list your subscriptions.CommentShareEXPERTShlomo Dubrowinanswered 3 months ago0Thanks a lot. I just cancelled 8 active subscriptions that I didn't know of. :)CommentSharerePost-User-8067226answered 3 months agoShlomo Dubrowin EXPERT3 months agoAwesome, glad I could help.Share"


Question: "Does the Lambda , AWS Active MQ support Message Groups ?"

I want to use a single Active MQ  Queue and have it invoke lambda functions.  I want to enforce order but also have parallelism.   Active MQ has a concept of "Message Groups" which sounds perfect.    Does the AWS lambda support Message Groups and will invoke different instance of lambda synchronously with different message groups as required  based on load ?FollowComment

Answer: "0To answer my own question,  the AcitveMQ  Message Group is irrelevant as the documentation says"The Lambda service limits the concurrency to one execution environment per queue." seehttps://aws.amazon.com/blogs/compute/using-amazon-mq-as-an-event-source-for-aws-lambda/As such there will never be more than one lambda function being executed at one time as part of a synchronous call, as no no matter whether the Message Group is specified there will not be any opportunity to increase the amount of concurrent processing.CommentShareAWS-User-1467335answered a year ago"