Question: "Not able to compile to NEFF, the BERT model from neuron tutorial"

"Hi Team,I wanted to compile a BERT model and run it on inferentia. I trained my model using pytorch and tried to convert it by following the same steps in this tutorial on my amazon linux Machine. But I keep getting failure with this error:09/22/2022 06:13:56 PM ERROR 23737 [neuron-cc]: Failed to parse model /tmp/tmp64l9ygmj/graph_def.pb: The following operators are not implemented: {'SelectV2'} (NotImplementedError)I followed the installation steps here for pytorch-1.11.0 and tried to execute the code in tutorial but got the same error.We wanted to explore using Inferentia for our large BERT model but are blocked on doing so due to failure in conversion to NEFF format. I also tried following steps using TF and ran into some other ops unsupported issue. Could you please help!Below are the setup commands i ran on my Amazon Linux Desktopsudo yum install -y python3.7-venv gcc-c++python3.7 -m venv pytorch_venvsource pytorch_venv/bin/activatepip install -U pip# Set Pip repository  to point to the Neuron repositorypip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com#Install Neuron PyTorchpip install torch-neuron neuron-cc[tensorflow] "protobuf<4" torchvision!pip install --upgrade "transformers==4.6.0"pip install tensorflow==2.8.1and then executed the below script(copied from tutorial) on my amazon linux host:import tensorflow  # to workaround a protobuf version conflict issueimport torchimport torch.neuronfrom transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoConfigimport transformersimport osimport warnings# Setting up NeuronCore groups for inf1.6xlarge with 16 coresnum_cores = 16 # This value should be 4 on inf1.xlarge and inf1.2xlargenc_env = ','.join(['1'] * num_cores)warnings.warn("NEURONCORE_GROUP_SIZES is being deprecated, if your application is using NEURONCORE_GROUP_SIZES please \see https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/deprecation.html#announcing-end-of-support-for-neuroncore-group-sizes \for more details.", DeprecationWarning)os.environ['NEURONCORE_GROUP_SIZES'] = nc_env# Build tokenizer and modeltokenizer = AutoTokenizer.from_pretrained("bert-base-cased-finetuned-mrpc")model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased-finetuned-mrpc", return_dict=False)# Setup some example inputssequence_0 = "The company HuggingFace is based in New York City"sequence_1 = "Apples are especially bad for your health"sequence_2 = "HuggingFace's headquarters are situated in Manhattan"max_length=128paraphrase = tokenizer.encode_plus(sequence_0, sequence_2, max_length=max_length, padding='max_length', truncation=True, return_tensors="pt")not_paraphrase = tokenizer.encode_plus(sequence_0, sequence_1, max_length=max_length, padding='max_length', truncation=True, return_tensors="pt")# Run the original PyTorch model on compilation exapleparaphrase_classification_logits = model(**paraphrase)[0]# Convert example inputs to a format that is compatible with TorchScript tracingexample_inputs_paraphrase = paraphrase['input_ids'], paraphrase['attention_mask'], paraphrase['token_type_ids']example_inputs_not_paraphrase = not_paraphrase['input_ids'], not_paraphrase['attention_mask'], not_paraphrase['token_type_ids']# Run torch.neuron.trace to generate a TorchScript that is optimized by AWS Neuronmodel_neuron = torch.neuron.trace(model, example_inputs_paraphrase)This gave me the following error:2022-09-22 18:13:12.145617: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory2022-09-22 18:13:12.145649: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.sample_pytorch_model.py:14: DeprecationWarning: NEURONCORE_GROUP_SIZES is being deprecated, if your application is using NEURONCORE_GROUP_SIZES please see https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/deprecation.html#announcing-end-of-support-for-neuroncore-group-sizes for more details.  for more details.", DeprecationWarning)Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 433/433 [00:00<00:00, 641kB/s]Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████| 213k/213k [00:00<00:00, 636kB/s]Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████| 436k/436k [00:00<00:00, 731kB/s]Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████| 29.0/29.0 [00:00<00:00, 35.2kB/s]Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████| 433M/433M [00:09<00:00, 45.7MB/s]/local/home/spareek/pytorch_venv/lib64/python3.7/site-packages/transformers/modeling_utils.py:1968: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!  input_tensor.shape[chunk_dim] == tensor_shape for input_tensor in input_tensorsINFO:Neuron:There are 3 ops of 1 different types in the TorchScript that are not compiled by neuron-cc: aten::embedding, (For more information see https://github.com/aws/aws-neuron-sdk/blob/master/release-notes/neuron-cc-ops/neuron-cc-ops-pytorch.md)INFO:Neuron:Number of arithmetic operators (pre-compilation) before = 565, fused = 548, percent fused = 96.99%INFO:Neuron:Number of neuron graph operations 1601 did not match traced graph 1323 - using heuristic matching of hierarchical informationINFO:Neuron:Compiling function _NeuronGraph$662 with neuron-ccINFO:Neuron:Compiling with command line: '/local/home/spareek/pytorch_venv/bin/neuron-cc compile /tmp/tmp64l9ygmj/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp64l9ygmj/graph_def.neff --io-config {"inputs": {"0:0": [[1, 128, 768], "float32"], "1:0": [[1, 1, 1, 128], "float32"]}, "outputs": ["Linear_5/aten_linear/Add:0"]} --verbose 35'huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...To disable this warning, you can either:- Avoid using `tokenizers` before the fork if possible- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false).2022-09-22 18:13:52.697717: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory2022-09-22 18:13:52.697749: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.09/22/2022 06:13:56 PM ERROR 23737 [neuron-cc]: Failed to parse model /tmp/tmp64l9ygmj/graph_def.pb: The following operators are not implemented: {'SelectV2'} (NotImplementedError)Compiler status ERRORINFO:Neuron:Compile command returned: 1WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$662; falling back to native python function callERROR:Neuron:neuron-cc failed with the following command line call:/local/home/spareek/pytorch_venv/bin/neuron-cc compile /tmp/tmp64l9ygmj/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp64l9ygmj/graph_def.neff --io-config '{"inputs": {"0:0": [[1, 128, 768], "float32"], "1:0": [[1, 1, 1, 128], "float32"]}, "outputs": ["Linear_5/aten_linear/Add:0"]}' --verbose 35Traceback (most recent call last):  File "/local/home/spareek/pytorch_venv/lib64/python3.7/site-packages/torch_neuron/convert.py", line 382, in op_converter    item, inputs, compiler_workdir=sg_workdir, **kwargs)  File "/local/home/spareek/pytorch_venv/lib64/python3.7/site-packages/torch_neuron/decorators.py", line 220, in trace    'neuron-cc failed with the following command line call:\n{}'.format(command))subprocess.SubprocessError: neuron-cc failed with the following command line call:/local/home/spareek/pytorch_venv/bin/neuron-cc compile /tmp/tmp64l9ygmj/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp64l9ygmj/graph_def.neff --io-config '{"inputs": {"0:0": [[1, 128, 768], "float32"], "1:0": [[1, 1, 1, 128], "float32"]}, "outputs": ["Linear_5/aten_linear/Add:0"]}' --verbose 35INFO:Neuron:Number of arithmetic operators (post-compilation) before = 565, compiled = 0, percent compiled = 0.0%INFO:Neuron:The neuron partitioner created 1 sub-graphsINFO:Neuron:Neuron successfully compiled 0 sub-graphs, Total fused subgraphs = 1, Percent of model sub-graphs successfully compiled = 0.0%INFO:Neuron:Compiled these operators (and operator counts) to Neuron:INFO:Neuron:Not compiled operators (and operator counts) to Neuron:INFO:Neuron: => aten::Int: 97 [supported]INFO:Neuron: => aten::add: 39 [supported]INFO:Neuron: => aten::contiguous: 12 [supported]INFO:Neuron: => aten::div: 12 [supported]INFO:Neuron: => aten::dropout: 38 [supported]INFO:Neuron: => aten::embedding: 3 [not supported]INFO:Neuron: => aten::gelu: 12 [supported]INFO:Neuron: => aten::layer_norm: 25 [supported]INFO:Neuron: => aten::linear: 74 [supported]INFO:Neuron: => aten::matmul: 24 [supported]INFO:Neuron: => aten::mul: 1 [supported]INFO:Neuron: => aten::permute: 48 [supported]INFO:Neuron: => aten::rsub: 1 [supported]INFO:Neuron: => aten::select: 1 [supported]INFO:Neuron: => aten::size: 97 [supported]INFO:Neuron: => aten::slice: 5 [supported]INFO:Neuron: => aten::softmax: 12 [supported]INFO:Neuron: => aten::tanh: 1 [supported]INFO:Neuron: => aten::to: 1 [supported]INFO:Neuron: => aten::transpose: 12 [supported]INFO:Neuron: => aten::unsqueeze: 2 [supported]INFO:Neuron: => aten::view: 48 [supported]Traceback (most recent call last):  File "sample_pytorch_model.py", line 38, in <module>    model_neuron = torch.neuron.trace(model, example_inputs_paraphrase)  File "/local/home/spareek/pytorch_venv/lib64/python3.7/site-packages/torch_neuron/convert.py", line 184, in trace    cu.stats_post_compiler(neuron_graph)  File "/local/home/spareek/pytorch_venv/lib64/python3.7/site-packages/torch_neuron/convert.py", line 493, in stats_post_compiler    "No operations were successfully partitioned and compiled to neuron for this model - aborting trace!")RuntimeError: No operations were successfully partitioned and compiled to neuron for this model - aborting trace!FollowComment"

Answer: "0Hi, my suspicion is in the following line of your env setup:pip install tensorflow==2.8.1I faced similar issue in the past while trying to compile a different model. I solved it by downgrading tensorflow to 1.15.0This is the configuration of packages I used with Pytorch 1.11:pip install -U --force-reinstall torch-neuron==1.11.0.2.3.0.0 \     neuron-cc[tensorflow]==1.11.7.0+aec18907e \     "protobuf<4" tensorflow==1.15.0 \     numpy==1.19.5 torchvision==0.12.0Please, try that and let me know.CommentShareSamir Araujoanswered 8 months ago"


Question: How does AWS GameKit integrate with Facebook login?

"How is AWS GameKit integrating Facebook login, does it use inbuilt browser for opening an Amazon Cognito link or using Facebook's C++ SDK? How does this work on Android?FollowComment"

Answer: "0Accepted AnswerAWS GameKit provides an API to get a Facebook login URL which comes from Amazon Cognito. Then the game developer can open this URL in a web browser, and call an AWS GameKit API to poll for login completion. In the Unreal examples, we've demonstrated how to open the Facebook login URL in a web browser. We are working on adding mobile support to AWS GameKit (Android & iOS), and the same Facebook login example works on both Android and iOS. AWS GameKit does not use the Facebook C++ SDK.CommentShareAmzn_Adriananswered a year ago"


Question: migrate data to new Aurora cluster

"Hi team,I modified my CDK code to add the encryption option in my Existing Aurora DB cluster =>  which created a brand new cluster.now I have 2 clusters :the old cluster (non-encrypted)the new cluster created after CDK changes (the encrypted one)how can I take existing data from my old cluster (the non-encrypted one) and inject them into the newly created cluster( the encrypted one created following CDK changes)?Thank you :)FollowComment"

Answer: "0Restore a snapshot from the unencrypted database into the encrypted database by specifying the KMS key used for encryption: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Overview.Encryption.htmlCommentShareSeanManswered a year agoJess a year agoThank you for your answer :) but this doesn't answer my questionI have a new Aurora cluster created because I changed the encryption option (un-encrypted => to encrypted) via CDK codenow I would like to copy my data from the original cluster (the non-encrypted one) ==> to the newly created one (the encrypted one)Thank youShareAWS-User-8174525-subhash a year agoif you are not using snapshot to from unencrypted cluster to create an encrypted cluster.you can use dms to migrate the data.Share0When you restore a snapshot, it creates a new DB Cluster, see:https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Managing.Backups.html#Aurora.Managing.Backups.RestoreYou specify the KMS key as a parameter on your restore, see:https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/restore-db-cluster-from-snapshot.htmlLook for --kms-key-idSince the CDK creates CloudFormation templates, you'll likely have to import the new cluster into your stack. See:https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.htmlCommentShareRodney Lesteranswered a year ago"


Question: Unsupported or invalid SSH public key format

"I have deployed a Transfer Family sftp server (using an Amazon EFS ). I am having trouble configuring the user. I keep getting the error: Failed to create user (Unsupported or invalid SSH public key format)I have tried using the format according to AWS format but still, get the error. Has anyone had this issue and how did you solve it?FollowComment"

Answer: "0Hello, have you tried to use an RSA, ED25519, or ECDSA key type as the user's public key? These are the current key types supported for AWS Transfer users. There are detailed instructions on generating SSH key pairs in the AWS Transfer Family User Guide found here. If you continue to experience issues, please create a support case.CommentShareEXPERTAWSTransferFamilyanswered 9 months agoDreski 9 months agoI actually read that document and used puTTY generator initially. However, after some additional research, I discovered that the output format was incorrect. I should have used the format starting with ssh- rsa. So I modified the file in that format and now it works!  Thx for your response!Share"


Question: Trying to port a number to connect

I followed the steps to do so. When clicking on the submit button to create the case. I get a 400 bad request.FollowComment

Answer: 0Support got it fixed.CommentSharetechprospaceanswered 3 years ago


Question: Issue while using local IPC for messaging between two local component

"publish("test/test",message={"a":1})def publish(self, topic, message, message_format="json"):"""Publishes message to a topic of AWS IOT    Args:          topic (string): AWS IoT Core Topic          message (dict / string): Message to be published          message_format      """      request = PublishToTopicRequest()      request.topic_name = topic      publish_message = PublishMessage()      if message_format != "json":          publish_message.binary_message = BinaryMessage(message=bytes(message, "utf-8"))          # publish_message.binary_message.message = bytes(message, "utf-8")      else:          publish_message.json_message = JsonMessage(message=message)          # publish_message.json_message.message = message      request.publish_message = publish_message      operation = self.ipc_client.new_publish_to_topic()      operation.activate(request)      future = operation.get_response()      future.result(TIMEOUT)  accessControl permissions are:com.test.test:pubsub:1:policyDescription: "Allows communication with other components"operations:- "aws.greengrass#PublishToTopic"resources:- "test/test"and Greengrass.log are:2021-01-06T10:30:13.515Z [INFO] (Thread-6) software.amazon.awssdk.eventstreamrpc.RpcServer: New connection code [AWS_ERROR_SUCCESS] for [Id 816, Class ServerConnection, Refs 1](2021-01-06T10:30:13.515Z) - <null>. {}2021-01-06T10:30:13.515Z [INFO] (Thread-6) software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler: aws.greengrass#GreengrassCoreIPC authenticated identity: com.test.test. {}2021-01-06T10:30:13.515Z [INFO] (Thread-6) software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler: Connection accepted for com.test.test. {}2021-01-06T10:30:13.515Z [INFO] (Thread-6) software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler: Sending connect response for com.test.test. {}2021-01-06T10:30:13.517Z [INFO] (Thread-6) software.amazon.awssdk.eventstreamrpc.RpcServer: New connection code [AWS_ERROR_SUCCESS] for [Id 818, Class ServerConnection, Refs 1](2021-01-06T10:30:13.517Z) - <null>. {}2021-01-06T10:30:13.517Z [INFO] (Thread-6) software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler: aws.greengrass#GreengrassCoreIPC authenticated identity: com.test.test {}2021-01-06T10:30:13.517Z [INFO] (Thread-6) software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler: Connection accepted for com.test.test. {}2021-01-06T10:30:13.517Z [INFO] (Thread-6) software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler: Sending connect response for com.test.test. {}2021-01-06T10:30:13.518Z [ERROR] (Thread-6) com.aws.greengrass.ipc.common.ExceptionUtil: Unhandled exception in IPC. {}java.lang.NullPointerExceptionEdited by: allenkallz on Jan 6, 2021 3:21 AMFollowComment"

Answer: "0Hi and thanks for using Greengrass v2.Is there any additional logging after the NullPointerException? There ought to be a stacktrace which would tells us exactly what the problem is.Thanks!Michael Dombrowski.CommentShareEXPERTMichaelDombrowski-AWSanswered 2 years ago0I think I am also getting same error. Here is the complete stack trace:2021-01-07T14:31:08.759Z \[INFO] (Thread-17) software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler: Connection accepted for com.example.IPCExample. {}  2021-01-07T14:31:08.760Z \[INFO] (Thread-17) software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler: Sending connect response for com.example.IPCExample. {}  2021-01-07T14:31:08.760Z \[ERROR] (Thread-17) com.aws.greengrass.ipc.common.ExceptionUtil: Unhandled exception in IPC. {}  java.lang.NullPointerException          at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)          at com.aws.greengrass.builtin.services.pubsub.PubSubIPCEventStreamAgent.handlePublishToTopicRequest(PubSubIPCEventStreamAgent.java:113)          at com.aws.greengrass.builtin.services.pubsub.PubSubIPCEventStreamAgent.access$100(PubSubIPCEventStreamAgent.java:44)          at com.aws.greengrass.builtin.services.pubsub.PubSubIPCEventStreamAgent$PublishToTopicOperationHandler.lambda$handleRequest$0(PubSubIPCEventStreamAgent.java:192)          at com.aws.greengrass.ipc.common.ExceptionUtil.translateExceptions(ExceptionUtil.java:33)          at com.aws.greengrass.builtin.services.pubsub.PubSubIPCEventStreamAgent$PublishToTopicOperationHandler.handleRequest(PubSubIPCEventStreamAgent.java:176)          at com.aws.greengrass.builtin.services.pubsub.PubSubIPCEventStreamAgent$PublishToTopicOperationHandler.handleRequest(PubSubIPCEventStreamAgent.java:160)          at software.amazon.awssdk.eventstreamrpc.OperationContinuationHandler.onContinuationMessage(OperationContinuationHandler.java:271)          at software.amazon.awssdk.crt.eventstream.ServerConnectionContinuationHandler.onContinuationMessageShim(ServerConnectionContinuationHandler.java:53)    Edited by: kunupat on Jan 7, 2021 6:46 AMCommentSharekunupatanswered 2 years ago0Based on that stacktrace, that must mean that the topic is null. Please ensure that you have set a non-null topic name in the publish request.request.topic = "my/topic/name"Thanks.Michael DombrowskiEdited by: MichaelDombrowski-AWS on Jan 7, 2021 9:08 AMCommentShareEXPERTMichaelDombrowski-AWSanswered 2 years ago0HiThanks for your reply.The issue is in AWS IoT Python SDK v2 that for communication with the local component we have to create request instance PublishToTopicRequest  that accept topic with-param name "topic"but when we have to communicate with Cloud then we have to create request instance PublishToIoTTopicRequest and this class accepts topic with-param name   topic_name.    I have one question why you dont have the same param name at both places. Its confusingSee the param names of both classes in the attachment.And previously it`s not mentioned in Greengrass v2 documentation.Edited by: allenkallz on Jan 8, 2021 8:37 AMCommentShareallenkallzanswered 2 years ago"


Question: Building Project Failed . Error Executing WAF

"Here is my log :"C:\Amazon\Lumberyard\1.12.0.1\dev\lmbr_waf.bat" build_win_x64_vs2015_profile --enabled-game-projects=s -p s[WAF] Engine Root: C:\Amazon\Lumberyard\1.12.0.1\dev[WAF] Executing 'build_win_x64_vs2015_profile' in 'c:\Amazon\Lumberyard\1.12.0.1\dev\BinTemp\win_x64_vs2015_profile'[WARN] Removing target platform win_x64_vs2013 due to "Visual Studio 2013" not checked in Setup Assistant.[WARN] Removing target platform android_armv7_clang due to "Compile For Android" not checked in Setup Assistant.[WARN] Removing target platform android_armv8_clang due to "Compile For Android" not checked in Setup Assistant.[WARN] Game project 'StarterGame' configured in bootstrap.cfg is the enabled game for this build. In order to run or debug for the game 's', they need to be set in bootstrap.cfg under the 'sys_game_folder' entry accordingly[WARNING] Incredibuild disabled by build option[WARNING] 'libpath' value 'c:\Amazon\Lumberyard\1.12.0.1\dev\Code\SDKs\LibTomCrypt\lib\ios' defined in TaskGenerator "sIOSLauncher" does not exist[WARNING] 'libpath' value 'c:\Amazon\Lumberyard\1.12.0.1\dev\Code\SDKs\LibTomMath\lib\ios' defined in TaskGenerator "sIOSLauncher" does not exist[WARNING] 'libpath' value 'c:\Amazon\Lumberyard\1.12.0.1\dev\Code\SDKs\FreeType2\ios\lib' defined in TaskGenerator "sIOSLauncher" does not existunable to find QTunable to find QTTraceback (most recent call last):File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\Scripting.py", line 226, in waf_entry_pointrun_commands()File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\Scripting.py", line 342, in run_commandsctx = run_command(cmd_name)File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\Scripting.py", line 319, in run_commandctx.execute()File ".\Tools\build\waf-1.7.13\lmbrwaflib\msvs.py", line 1926, in override_build_stateold(ctx)File "c:\Amazon\Lumberyard\1.12.0.1\dev\wscript", line 317, in executereturn execute_method(self)File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\Scripting.py", line 696, in executereturn execute_method(self)File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\Build.py", line 254, in executeself.execute_build()File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\Build.py", line 277, in execute_buildself.compile()File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\Build.py", line 370, in compileself.producer.start()File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\Runner.py", line 303, in startself.refill_task_list()File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\Runner.py", line 209, in refill_task_listself.add_outstanding_tasks(next(self.biter))File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\Build.py", line 818, in get_build_iteratorself.post_group()File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\Build.py", line 789, in post_groupf()File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\build\waf-1.7.13\waflib\TaskGen.py", line 218, in postv()File ".\Tools\build\waf-1.7.13\lmbrwaflib\cry_utils.py", line 671, in add_compiler_dependencyif os.path.isabs( self.env['CC'] ):File "C:\Amazon\Lumberyard\1.12.0.1\dev\Tools\Python\2.7.12\windows\lib\ntpath.py", line 59, in isabsreturn s != '' and s[:1] in '/\'TypeError: 'in <string>' requires string as left operand, not listProcess failed with return code: 1Follow"

Answer: "0Hey @REDACTEDUSERSharerePost-User-5738838answered 5 years ago0Sorry for late post, but:I have the same problem here in 2020, using VS 2019, so you're saying MFC may have been left out of VS install ? Is that still valid in latest VS ?Using LY 1.24.1ThxSharerePost-User-2755327answered 3 years ago0It's weird, why don't you guys fill sorry for this post ?https://forums.awsgametech.com/t/lumberyardsdk-vs2017-sln-generated-using-lmbr-waf-doesnt-work/8974SharerePost-User-2441773answered 3 years ago0Hi, we have been investigating this issue and we are working on a fix for this in the next update.SharerePost-User-8164888answered 3 years ago0You'r fixing it, that's great, and I think you should fix this too.https://forums.awsgametech.com/t/cant-open-animation-editor-setup-from-branch-1-24-or-1-23/8986SharerePost-User-2441773answered 3 years ago0Hi, thanks for the great feedback.Improving the animation editor is on our to do list as well. In our next update we will be introducing a lot of bug fixes, new features and quality of life improvements, so keep an eye out for LY 1.25 announcements on our forums and blog pages.SharerePost-User-8164888answered 3 years ago"


Question: How to delete an active instance in the detector model?

"Hello:How I can delete an instance of the detector model? I mean, for each input with different key, a new instance is created from the detector model. If at some point I want to delete that particular instance, how can I do that? I couldn't get any information on that.ThanksFollowComment"

Answer: "1Accepted AnswerAWS IoT Events has released a new API for deleting the detectors in batches.API Guide for reference: https://docs.aws.amazon.com/cli/latest/reference/iotevents-data/batch-delete-detector.htmlGuide for deleting the detectors on console: https://docs.aws.amazon.com/iotevents/latest/developerguide/iotevents-delete-batch-detector.htmlCommentShareHarshil_Sanswered a year ago0The AWS IoT Events service does not define an explicit API for removing a detector instance. This is because there is no ongoing cost for the instance when it is not used. What is the use case for deleting an instance?edit: please see Harshil's more recent answerCommentShareRyan_Banswered a year agoAWS-User-SOS a year agoNo specific use case...It's just getting too messy to manage having a lot of instances, where many are no longer being used...ShareAWS-User-SOS a year agoThe use case is to be able to clean up as it gets very messy to manage it during developmentShare"


Question: AWS IoT maximum reported attributes and how I ca increase this number

"Hello,I am trying to build an IoT thing to publish message on shadow. I am able successfully published 4 parameters:{"desired": {"welcome": "aws-iot","vin": 4840,"temperature": 7,"wifi_strength": -67,"switch": true},"reported": {"welcome": "aws-iot"}}if I want to add more parameters such as: pressure or humidity it looks like "Shadow" doesnt accept the update.Can you please kindly advise what should I do to be able to publish more parameters in to the repoFollowComment"

Answer: "0This 26-second videohttps://www.youtube.com/watch?v=NnZDpnp6pvQshows hundreds of attributes in the device shadow.CommentShareuwegambitcommanswered 3 years ago0Hi uwegambitcomm,Thank you for your replay, that is very interesting video, but it doesnt show how to that,I also checked links followed to the video and it doesnt really show...Hope AWS guys can "light some sun in dark"Regards,StalkerCommentSharestalkeranswered 3 years ago0After further investigation, I find issue was NOT related to AWS IoT but to internal embedded side problem. Issue is fixed now, and close.CommentSharestalkeranswered 3 years ago"


Question: Direct Connect switching from 1G to 10G

What is the lag time to move from a 1G to 10G DX connection? Is there a downtime?FollowComment

Answer: "0Accepted AnswerUpgrading an existing connection from 1G to 10G is not possible. User will have to order/setup a new 10G connection and then migrate their VifsYou cannot change the port speed after you create the connection request. To change the port speed, you must create and configure a new connection.CommentShareNikiforos_Banswered 4 years ago"


Question: EC2 instance kept randomly crashing

"Hello,My client's EC2 instance has been kept crashing randomly for a while, instance ID:  i-0f4d7b0f00f614d62When it went down, I couldn't ssh into it. I posted a question on serverfault: https://serverfault.com/questions/1127277/ec2-instance-randomly-crashBut some times it doesn't seem to be a memory issue, while I was monitoring memory usage, the system could suddenly go down even when it had sufficient memory at the last minute.Could you investigate please?FollowComment"

Answer: 0Try a VPC peering connection between the VPCs of the two accounts.CommentSharerePost-User-2633511answered 2 months agorePost-User-4338311 2 months agoI am not suing any VPC. Just a standalone EC2.Share


Question: API Gateway claims it's invoking a custom authorizer but rejects before calling

"I set up a custom authorizer in API Gateway to validate a JWT on the Authorization header. The authorizer is working fine for some GET resources, but for a specific POST call it rejects with a 403 Forbidden (AccessDeniedException):{    "message": "User is not authorized to access this resource"}I turned on API Gateway logging and for both resources it claims it's calling the authorizer with ID vf0zk6. This is the API gateway log:Extended Request Id: d0GmcE2HCYcFrVQ=Starting authorizer: vf0zk6 for request: f23990ba-c1a4-4c71-9427-d28697735735Incoming identity: ***************voGFIgExtended Request Id: d0GmcE2HCYcFrVQ=Using valid authorizer policy for principal: ************************f8069dSuccessfully completed authorizer executionThe client is not authorized to perform this operation.Gateway response type: ACCESS_DENIED with status code: 403Gateway response body: {"message":"User is not authorized to access this resource"}Gateway response headers: {Access-Control-Allow-Headers=Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token, Access-Control-Allow-Origin=*, Access-Control-Allow-Methods=OPTIONS,POST}Gateway response type: ACCESS_DENIED with status code: 403I also enabled CloudWatch logging for the custom authorizer lambda itself. I can see logs for the GET resource it succeeds at, but no logs show up for the failing POST resources. So it looks like it's failing it without ever calling the custom authorizer lambda. Both resources are called with the exact same Authorization header.I was able to temporarily get it working by creating a new authorizer and pointing it to the same lambda function (with caching disabled). But the next day after I re-added caching on the new authorizer it was failing again without any call to the lambda function.(EDIT) It doesn't have to do with GET vs POST; some POST resources are succeeding and others failing. The requests are identical, pointing to the same authorizer but some just get rejected before the custom authorizer lambda is invoked and others invoke the lambda and work fine.FollowComment"

Answer: "1Accepted AnswerI was able to temporarily get it working by creating a new authorizer and pointing it to the same lambda function (with caching disabled). But the next day after I re-added caching on the new authorizer it was failing again without any call to the lambda function.The above suggests that authorizer caching is causing the issue. If you are able to make a GET request and subsequent GET requests succeed but subsequent POST requests fails then the policy returned by the authorizer is limiting the scope to GET method only.Use API Gateway Lambda authorizersOutput from an Amazon API Gateway Lambda authorizerI recommend following the below article to troubleshoot the issue. See the section called Resolve "not authorized to access this resource" errors from the Lambda authorizer.How do I troubleshoot HTTP 403 Forbidden errors when using a Lambda authorizer with an API Gateway REST API?CommentShareTaka_Manswered 5 months agoRandomEngy 5 months agoThanks, this pointed me in the right direction. This was a really helpful article: https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-lambda-authorization-errors/My authorizer was just passing the methodArn straight through when I needed to be returning a wildcard method ARN. It had been caching the result for all method calls when I assumed it had cache keys for each one.Share"


Question: RDS Query Editor Catch 22 - prevents me from doing anything at all

"I can't use the query editor anymore. It worked around 18 months ago when I created a PostgreSQL DB.When I try to use it now, I see this: "Currently, query editor only supports Aurora Serverless databases."And this: "To use the query editor for a database, the database must have the Data API enabled."When I try to enable the data API I see this: "Currently, you can't use the Data API with Aurora Serverless v2 DB instances."When I try to create a database the only option I have is for Serverless v2.How can I do anything with an AWS database?Are there external tools that can be used to run SQL against the DB? Of course these will need to work without me configuring any users on the database because that would require running SQL in a query tool.FollowComment"

Answer: "1Accepted AnswerHello,To provision the Aurora Serverless v1 you need to first select a compatible DB engine version: 2.07.1, 5.6.10a for MySQL or 10.12, 10.14, 10.18 for Postgres.If you select one of these, the Serverless v1 will appear as possible instance class.To answer your other question: you can connect to any of the RDS databases using external tools like for example DBeaver https://dbeaver.io/CommentShareAWS-ADolganovanswered 10 months agoneohed 10 months agoAha, ok, thanks. I only started AWS a few days ago and there's a lot to take in...  Thank you for both answers. "dbeaver.io" looks perfect for my needs.Share"


Question: API Gateway Cache Invalidation not working

"I'm trying to invalidate API Gateway cache as described in the documentation: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-caching.htmlI've tried to invalidate the cache by making a http request using Lambda and through Postman but neither worked, the result is always the same: I get a warning in the response headers saying "199 Cache-control headers were ignored because the caller was unauthorized."My API Gateway policy currently looks like this:{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Principal": "*",            "Action": [                "execute-api:Invoke",                "execute-api:InvalidateCache"            ],            "Resource": "arn:aws:execute-api:{region}:{accountId}:{apiId}/*"        }    ]}I've also applied the permission execute-api:InvalidateCache to the lambda who calls the API Gateway.FollowComment"

Answer: "0Good question.Is this for a GET method?    And are you authorized properly - does the IAM role you are using or the lambda role have the proper permissions to Invalidate Cache?If not - I would check the settings for if API Cache is enabled and the setting for per-key cache invalidation here: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-caching.html#override-api-gateway-stage-cache-for-method-cache.The error you are getting tells me that the setting for handle unauthorized requests is set to "Ignore cache control header.  Add a warning in response header."    And that it may still think you are unauthorized.CommentSharejsoncanswered a year agoAWSUser22001 a year agoYes, it is a GET method and the Lambda has the policy to execute-api:InvalidateCacheShare0I'm having the same problem here.Cache invalidation works properly only if I do not require authorization.Also, I can't understand from the documentation if authorization also works with a custom authorizer on API gateway.My authorization mecanism on API gateway is based on a bearer jwt token with a lambda authorizer that returns a policy.I tried to return the proper policy execute-api:InvalidateCache from the custom authorizer but it didn't work as expected.I also tried with an IAM based authorization and a signed http request but it did not work neither. I had a user with the proper policy (I did not try with a role).Also I did not try with a lambda calling the API gateway.I suspect there is something around roles vs policies that I don't do properly...Also, can you confirm if it can work with a custom authorizer or not ?CommentSharebrunoanswered a year ago"


Question: Can anyone explain the RDS "EBS IO Balance (Percent)" metric?

"Is anyone able to explain the EBS IO Balance (Percent) metric in RDS please?We have a MySQL t3.small instance with a 400GB GP3 disk. We're running a process from an EC2 instance to pull data from an external API and store it in our DB.We're hitting 0 on the EBS IO Balance (Percent) metric and seeing slow downs - which I pursue is a result of the read & write latency metrics increasing when we hit 0. We're struggling to understand what this metric is - and therefore how we can improve our process.As for other metrcs:We peak at around 5,000 IOPS (combined read & write) - well under the 12k we get with our GP3 disk"EBS Byte Balance (Percent)" has fallen to 25% by the time our import process endsWrite throughout peaks at around 160 mb / secondRead throughput peaks at around 30 mb / secondCPU utilisation peaks at around 50% but our credit balance is very healthyDisk capacity is certainly not an issueScreenshotted metrics are here (import process started ~16:15:00)Any help / pointers would be much appreciated.I've asked elsewhere, and someone has suggested the issue might be related to volume striping?FollowComment"

Answer: "0EBS IO Balance is a property of the t3.small instance not the EBS volumes. That instance type has burst IOPS of 11800, and baseline IOPS of 1000.aws ec2 describe-instance-types --instance-types t3.small --query "InstanceTypes[*].[InstanceType, EbsInfo.EbsOptimizedSupport, EbsInfo.EbsOptimizedInfo]" --output json[    [        "t3.small",        {            "BaselineBandwidthInMbps": 174,            "BaselineThroughputInMBps": 21.75,            "BaselineIops": 1000,            "MaximumBandwidthInMbps": 2085,            "MaximumThroughputInMBps": 260.625,            "MaximumIops": 11800        }    ]]CommentShareMODERATORphilawsanswered 4 months ago"


Question: Cannot upgrade SLES 12 SP2 to SP3 or higher

"We have 6 on-demand EC2 instances on a public VPC, with SLES 12 SP2 running a SAP workload that we need to upgrade to SP3, then SP4.This is the procedure we are following:1.) zypper up2.) cd /etc/zypp/repos.d3.) for i in SP2; do newname=${i/SP2/SP3}; cp $i $newname; done4.) sed -i s/SP2/SP3/ SP3.repo5.) sed -i s/enabled=0/enabled=1/ SP3.repo6.) zypper refresh7.) zypper in zypper8.) zypper dupThe issue we encounter at step 6 is this:Permission to access 'https://smt-ec2.susecloud.net/repo/SUSE/Products/SLE-SERVER/12-SP3/x86_64/product_debug/repodata/repomd.xml?credentials=SMT-http_smt-ec2_susecloud_net' denied.Does any one know the proper credentials to access smt-ec2.susecloud.net? Or any solution to the issue?Does any one has a proved, documented step by step procedure to share?Thanks!FollowComment"

Answer: "0I found 2 entries "Added by SMT registration" on /etc/hosts. I commented out one and now I can connect to smt-ec2.susecloud.net.However, when prompted for credentials, I provide the user and password found in /etc/zypp/credentials.d but those don't authenticate.I tried running registercloudguest --force new, with the following result:/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:264: SubjectAltNameWarning: Certificate for 54.244.244.107 has no subjectAltName, falling back to check for a commonName for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)SubjectAltNameWarningI found an old post from February 13, 2017 in which rjschwei explains that the issue was because of memory shortage on the registration servers.Would someone at AWS take a look into this? Apparently the same issue is happening now.ThanksCommentSharealuminicasteanswered 4 years ago0I believe this issue is resolved. Will you please try running the registration again?CommentShareJoelSUSEanswered 4 years ago0I think the registercloudguest issue is solved.I still see a message:nwdev:/home/admin # registercloudguest --force-new/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:264: SubjectAltNameWarning: Certificate for 54.253.118.149 has no subjectAltName, falling back to check for a commonName for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)SubjectAltNameWarningBut new credentials are available:nwdev:/home/admin # ls -l /etc/zypp/credentials.dtotal 36-rw------- 1 root root 72 Sep  3 22:58 Advanced_Systems_Management_Module_x86_64-rw------- 1 root root 72 Sep  3 22:58 Containers_Module_x86_64-rw------- 1 root root 72 Sep  3 22:58 Legacy_Module_x86_64-rw------- 1 root root 72 Sep  3 22:58 Public_Cloud_Module_x86_64-rw------- 1 root root 72 Sep  3 22:58 SCCcredentials-rw------- 1 root root 72 Sep  3 22:58 SUSE_Linux_Enterprise_Server_x86_64-rw------- 1 root root 72 Sep  3 22:58 SUSE_Linux_Enterprise_Software_Development_Kit_x86_64-rw------- 1 root root 72 Sep  3 22:58 Toolchain_Module_x86_64-rw------- 1 root root 72 Sep  3 22:58 Web_and_Scripting_Module_x86_64I will run the Service Packs update a bit later and if successful, document my findings and tag this thread as Answered.Thank you very much for the registercloudguest fix!!Edited by: aluminicaste on Sep 3, 2019 9:09 PMCommentSharealuminicasteanswered 4 years ago0With the proper SMT registration entry in /etc/hosts and with the command registercloudguest fixed, the upgrade worked perfectly. Also, I found a simpler procedure for SUSE 12, very well documented here:https://kerneltalks.com/linux/how-to-upgrade-suse-12-sp1-to-sp3-or-sp4/Briefly:0.) registercloudguest --force-new1.) zypper in zypper-migration-plugin2.) zypper patch3.) zypper migration4.) reboot5.) cat /etc/os-releaseThanks!!!CommentSharealuminicasteanswered 4 years ago"


Question: IOTCore TLS connection overhead too large. Is it possible to resume a session?

"Hi, I have remote battery powered cellular devices uploading their measurements to IOT Core. They disconnected their modem each packet upload, which could be 1/min. However the data overhead required to connect and handshake, exchange certs etc is too large. At the moment 4k-6k is required on each connect, even if just sending 100bytes of data. The handshake is killing my data usage.See this blog I found for info on the overhead required by TLS 1.2 http://netsekure.org/2010/03/tls-overhead/For devices that need to be as efficient as possible with data and power, is it possible to a) resume TLS connections? or b) what is the recommended way for these devices to connect to the MQTT IOT Core service?This is a low power sensor device that is not able to run the SDK. So we need to implement out own connection to the MQTT ports on AWS (8883). AWS also does not allow unencrypted connections...A very similar and related question that went unanswered (https://forums.aws.amazon.com/thread.jspa?messageID=891100&#891100), however considering MQTT is desinged for IOT devices and is supposed to minimise load and be efficient it is hard to accept a 4k-6k overhead on each connection.FollowCommentBora a year agoI had a similar issue, and I ended up staying connected instead of connecting again every time. You probably could adjust the cellular network parameters (NB-IoT?) to stay in low-power mode while connected. This also has the advantage of having an always-on downlink in case you need it.Share"

Answer: "0Thanks for the reply. Re the payload size. The significant cost is in cellular data. So the 4k-6k overhead is increasing my data cost by a factor of at least 6. AWS core messaging costs are certainly a consideration but at the moment far outweighed by the data costs. To the point that using AWS IOTCore as my endpoint for my fleet of IOT devices is not really viable.I'd be interested if there are alternative thought regarding this.I don't currently have an account manager I don't think. Is there another way to get this escalated?Thanks,KarlCommentShareAWS-User-6286872answered a year agorePost-User-0665975 6 days agoHello, did you manage to find a work-around for this? We are also facing similar issues with data usage...Share0Hi. TLS session resumption is not presently supported by AWS IoT Core. The benefit of the feature is clear. If you have an Account Manager, please request this feature with them.You mention that a typical message is only 100 bytes. Please be aware that MQTT messages are metered in 5KB increments. I realize this may not be practical for your application, but if you could batch your messages (and hence connect less frequently), you would reduce both your cellular network data usage and your AWS IoT costs.CommentShareEXPERTGreg_Banswered a year ago0doesn't the MQTT Server Keep Alive option help with this?CommentShareesiranswered 4 months ago"


Question: AWS DMS Table Mapping Mongo Embedded document to Dynamo Table

"how we can write a table mapping rule for this kind of JSON? any suggestion{"_id" : ObjectId("6274105feb404050c7300005"),"ADDITIONALINFO" : {"PASSPORTISSUEDATE" : "01/04/2018 12:00:00 AM","PASSPORTEXPIRYDATE" : "12/18/2024 12:00:00 AM","IACOUNTRY" : "puk","IACOUNTRYID" : "5"},"CDETAILS" : [{"CHECKID" : "1","CHECKSTATUS" : "WIP","REVERIFICATION" : {"REVERIFICATIONCOUNT" : null,"OLDCHECKID" : null,"OLDDFREFNUMBER" : null},"CHECKSTATUSID" : "1","ISSUINGAUTHORITYCOUNTRY" : "","ISSUEDATE" : "02/16/2022 12:00:00 AM"}],"CSTAGE" : {"STAGEID" : "2","STAGE" : "New"},"CSTATUS" : {"STATUSID" : "1","STATUS" : "WIP",}}FollowComment"


Question: Not receiving application code in Url

"I am having trouble with the oauth2/idpresponse request to the IDP. I am failing to get the the code and state; after the waterfall of request. I see the oauth2/idpresponse being preformed in the network and resulting in a 302 while using devtools, but I think the state parameter erases what the oauth2/idpresponse request creates in the URI. So in result I only receive my DNS in the URI after login not Dns?code=code.login?client_id=5f2u0pc7f&redirec…>idpresponse?code=36a460d6-0e28-4dad-9b0f-b0b990755…>DNSFollowComment"


Question: EC2 BYOIP: signature couldn't be verified

"The goalI am trying to bring my /46 IPv6 prefix to EC2. It is part of a /44 IPv6 assigned to my ASN with the status "ASSIGNED" within the RIPE database. The ROA records have been set which I could also verify under https://rpki.cloudflare.com/.What I did so farI have basically followed this doc, yet provisioning fails: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-byoip.html#prepare-for-byoipWhen I send the aws ec2 provision-byoip-cidr request, the status is failed-provision with the message "The CidrAuthorizationContext signature could not be verified with the X509 certificates in the RIR records". The command whois -r -h whois.ripe.net abcd:efab:cde::/46 | grep descr | grep BEGIN delivers my certificate succesfully.My request looks like this:# ! bin/shtext_message="1|aws|123456789012|abcd:efab:cde::/46|20230101|SHA256|RSAPSS"signed_message=$(echo $text_message | tr -d "\n" | openssl dgst -sha256 -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-1 -sign private-key.pem -keyform PEM | openssl base64 | tr -- '+=/' '-_~' | tr -d "\n")aws ec2 provision-byoip-cidr --cidr abcd:efab:cde::/46 --cidr-authorization-context Message="$text_message",Signature="$signed_message" --region eu-central-1So, I checked the signature:$ echo "1|aws|123456789012|abcd:efab:cde::/46|20230101|SHA256|RSAPSS" > file.txt$ cat file.txt | openssl dgst -sha256 -sign private-key.pem -keyform PEM > rsasign.txt$ openssl sha256 -verify certificate.pem -signature rsasign.txt file.txtunable to load key fileIt only works when I use the public key instead of the certificate:$ openssl sha256 -verify public-key.pem -signature rsasign.txt file.txtVerified OKI also tried adding just the public key to the inet6num object's descr in the RIPE database, but that results in "No X509 certificate could be found in the Whois remarks", so that won't do it.Question: Any ideas on how to bring my IPv6 prefix to AWS? The linked documentation alone is of no help at this moment..FollowComment"


Question: Successful WAF CAPTCHA challenge is not updating aws_waf_token cookie

"My application is rendering the CAPTCHA challenge from a WAF intercepted 405 response in an iframe. While successful completion of the puzzle renders the "That is correct, Success! You will be redirected shortly" text, the aws_waf_token cookie does not get updated in the chrome/firefox/safari/edge browser.Looking more closely at the network traffic, when user submits the puzzle answer a successful POST call from the challenge.js to the "verify" endpoint completes but the subsequent POST request to the "voucher" endpoint fails with an 'InvalidRequest' 400 error. The request payload for the failed voucher call has two properties:a 'captcha_voucher' with the value taken from the verify responsea 'existing_token' property with a value of null.Given that the CAPTCHA challenge is essentially a black box, I'm at a loss on how to address this issue.Has anyone else run into this?FollowComment"


Question: "I need to read S3 data, transform and put into Data Catalog.  Should I be using a Crawler?"

"Files are uploaded every hour to an S3 bucket.  I currently have a Glue ETL job reading the S3 bucket, transforming data and inserting into a Glue Data Catalog.  I have seen examples where people have a Glue Crawler which reads the S3, writes data to Data Catalog table, and then an ETL job reads from a table transforms and then writes back to another table (or wherever it needs to go).  Should I be using a Crawler? I don't see the need for it if I can just use the ETL job to go S3->Transform->Data Catalog.  It would seem the ETL job supports bookmarking (init/commit) just like Crawlers do.FollowComment"

Answer: "1Accepted AnswerHi,AWS Glue Crawlers are used to automatically discover the schema of the data in Amazon S3 or other data sources. They also help in capturing schema evolution.If your schema is fixed (do not change often), already known and you do not have issues creating your tables manually via the console or your code using the APIs, then you do not need to use them.Consider also that the Crawler do have a cost, so cost optimization might be another reason to not use them if you are fine with self managing the schemas of your datasets.for additional information on Crawlers, you can refer to this section of the AWS Glue Documentation.hope this helpsCommentShareEXPERTFabrizio@AWSanswered a year agoSwwapnil a year agoAs Fabrizio said correctly, You only need to run the AWS Glue Crawler again if your schema changes. Also, ETL job support bookmarking and its recommended to use when your data grows day by day and when your job runs, you don't want to perform ETL job operations all over your data again, with bookmarking it will start processing operation on new data (wont perform operation on a processed data).Read more at:https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.htmlShare"


Question: Which fonts are actually available for burning in subtitles

"I am seeing an issue when trying to burn in Subtitles using Elemental media encoder. I have tried to choose a Proportional sans serif font as fallback, but I can on get a monospace to appear in the the out no matter if the font in my caption file is Arial, Verdana or literally proportionalsansserif. This seems like a bug. Likewise for traditional Chinese, i can only get one single font to appear in the output. Something that looks like SimSum when I looked for something like SimHei. Is it possible to get a list of the fonts that are actually supported? The current selection of fonts seems very limited. And since i get the same font no matter what I try, it seems like a bug.FollowComment"

Answer: "0Hello,The MediaConvert supports the following generic font families listed in the TTML2 W3C recommendation:default, monospace, sansSerif, serif, monospaceSansSerif, monospaceSerif, proportionalSansSerif, proportionalSerif.  Please follow the below documents for more details:https://docs.aws.amazon.com/mediaconvert/latest/ug/ttml-style-formatting.htmlhttps://docs.aws.amazon.com/mediaconvert/latest/ug/burn-in-output-captions.htmlPlease note: You can burn in only one track of captions in each output.Currently there are no plans to add the more font options for burn-in output captions. If this is a requirement for your workflow, please let me know and we can forward your request as a feature request with our product team for prioritisation.Regarding the bug, we require further details that are non-public information . Please open a support case with AWS using the following link https://console.aws.amazon.com/support/home#/case/createCommentShareSUPPORT ENGINEERrePost-User-6579332answered 7 months ago"


Question: Can consumer routers be used for AWS site-to-site VPN?

"I am looking at the Customer Gateway Device Requirements hereInternet key exchange (IKE) security association. This is required to exchange keys used to establish the IPsec security association.IPsec security association. This handles the tunnel's encryption, authentication, and so on.Tunnel interface. This receives traffic going to and from the tunnel.(Optional) Border Gateway Protocol (BGP) peering. For devices that use BGP, this exchanges routes between the customer gateway device and the virtual private gateway.How can I verify if a home router (for example ASUS RT-82U or ASUS TUF-AX5400) would meet the above specifications? The closest related thing I can find in the official specifications is VPN server IPSec.FollowComment"

Answer: 1Accepted AnswerI doubt above routers support it.You are essentially looking for this. Apparently Asus call it Net-To-Net Tunnel.https://www.asus.com/support/FAQ/1033578/I would suggest to open a chat/email case to Asus through their website and ask for IKEV1 or IKEV2 Site to Site IPSEC VPN capable device model.BRT-AC828 Model apparently support it as per above article.CommentShareMuhammadanswered 4 months ago


Question: Default display resolution (Windows ec2 server)

"Hi,I've been experimenting with Nice DCV installs on my g4dn ec2 instances with great results but I'm struggling to work out how to force the servers to default to 1920x1080 resolution rather than 1280x1024. At the moment I have to open a NICE DCV connection to each instance and manually set it to 1080p before disconnecting which is a pain - is there any way to force it to boot up at 1920x1080? I tried to set it via the registry display parameters but nothing seems to stick!Many thanks,Joe MurphyFollowComment"

Answer: "0Hi Joe,did you try the following parameter:console-session-default-layoutStringsession[]Default screen resolution and position for console sessions — Specifies the default screen resolution and position for console sessions. If this is set, DCV sets the requested layout at startup. Each monitor can be configured with resolution (w,h) and position (x,y). All specified monitors are enabled. Default layout example value: [{'w':<800>, 'h':<600>, 'x':<0>, 'y': <0>}, {'w':<1024>, 'h':<768>, 'x':<800>,'y':<0>}]from https://docs.aws.amazon.com/dcv/latest/adminguide/config-param-ref.htmlBest regards,KarstenCommentShareKarstenNISPanswered 3 years ago0Hi Karsten,Yes, I did try that console-session-default-layout value but I still seemed to get 1024x768 after a reboot. I might have gotten the syntax of the string wrong though - on the docs it mentions:Default layout example value: [{'w':<800>, 'h':<600>, 'x':<0>, 'y': <0>}, {'w':<1024>, 'h':<768>, 'x':<800>,'y':<0>}]On the Windows registry would that mean including either the square or curly brackets? And are both screens needed there or could I specify just a single screen? (which presumably would be: 'w':<1920>, 'h':<1080>, 'x':<0>, 'y': <0>Or: {'w':<1920>, 'h':<1080>, 'x':<0>, 'y': <0>}Many thanks!JoeCommentSharejoemurphyanswered 3 years ago0Hi Joe,should be with square and curly brackets : [{'w':<1920>, 'h':<1080>, 'x':<0>, 'y': <0>}].Seems it is eating the config due to the square brackets ;))  SB for opening and closing square brackets  SB{'w':<1920>, 'h':<1080>, 'x':<0>, 'y': <0>}SBWhich reg key are you setting in case this does not work?Best regards,KarstenEdited by: KarstenNISP on Oct 16, 2020 6:07 AMCommentShareKarstenNISPanswered 3 years ago0Hi Karsten,I was sure I'd tried every variation of brackets but obviously not as that's definitely done the trick! Fantastic - many thanks.Oh, one other quick question - I noticed I had quite a bit of lag on the audio stream coming from my ec2 instance today but having delved into the docs there doesn't seem to be a way to tweak the audio latency. Is it tied to the frame rate or is there much of a difference in audio latency between Best Quality vs Best Responsiveness? (I was on Best Responsiveness today btw).Thanks again,JoeCommentSharejoemurphyanswered 3 years ago0Hi Joe,hey great to hear it worked! With best quality you will consume bandwidth but assuming you have enough it is the latency of your connection causing the lag. You can check the latency in the information about your connection at Settings (wheel upper left) -> Streaming Mode -> Display Streaming Metrics (Keep Best responsiveness)   Best regards,KarstenCommentShareKarstenNISPanswered 3 years ago0Thanks Karsten,My stats showed my latency at a pretty constant 25ms all day and the audio delay seemed to be a constant 1500ms so I'm not sure it was my connection causing it. That said, I'll do some more testing and see if I can narrow down whether it's worse with specific applications running on the instance, etc.One final question (I promise!) - is it possible to get the instances to log into Windows when they're started without needing me to connect a client? Again, I'm fairly sure with the right registry setting it is possible but I've not managed to achieve it!Thanks,JoeCommentSharejoemurphyanswered 3 years ago0Hi Joe,1.5 sec delay sounds quite high - maybe you can check with the AWS support if they can help further here as we dont have the details of your environment there?Related to the Windows startup - would you like to turn off the DCV authentication maybe? Or the Windows session should initiate itself immediately without connecting with the DCV client?Regards,KarstenCommentShareKarstenNISPanswered 3 years ago0Hi Karsten,Yes, the 1.5 second delay is a really weird and irritating one - I've tested spinning up a few ec2 g4dn instances from the NICE DCV AMI and nothing else running but even playing a web video or simple mp4 from the desktop has the same issue. I can only assume it might be something to do with my client - I know the audio is in sync on the server itself as my main production server is able to stream out RTMP or SRT streams and they're fine, I just can't monitor them at the moment!With regards to the auto logon stuff, I think I've worked out what I need it to do using Microsoft's AutoLogon application. Essentially I need to be able to spin up multiple lightweight capture servers and stream their video output to my main production instance using NDI. I think I've now got it worked out the way I want so that the instances automatically log onto windows at 1920x1080 resolution and they open up the NDI Tools Scan Converter app which allows me to capture and control via KVM without needing to manually log into each instance.If I can only crack that audio lag I'll be happy! Is the HTML5 client still available anywhere by the way? I see it referenced in the docs but can't find a link.Thanks,JoeCommentSharejoemurphyanswered 3 years ago0Hi Joe,thanks for the interesting update! Autologon looking good! Audio lag less - maybe indeed the client as we usually do not see these issues.The HTML5 client is essentially your browser opening https://IP_ADDR_OF_YOUR_DCV_RUNNING_INSTANCE:8443 - or your DNS name of course where you then can apply certificates as well.Best regards,KarstenEdited by: KarstenNISP on Oct 21, 2020 7:11 AMCommentShareKarstenNISPanswered 3 years ago0Hi, Joe!Did you ever figure out the audio sync issue?I am struggling with the same problem..RemiCommentSharereminettanswered 3 years ago0Hi Remi,No, I've not yet managed to crack the audio sync issue here - I've been using a separate SRT stream to verify my output is in sync and behaving itself but it's not ideal. I wonder if the new QUIC UDP protocol in 2020.2 might help but if not upgraded my main production server yet to test it as we've been live all the last couple of weeks.Karsten - thanks for your reply and a bit of a d'oh moment about the html5 client from my side!! I'm going to give that a try today to see if I get anything different in terms of the audio lag vs the standard Windows client.UPDATE -So I just did that quick test with the HTML5 client (which is great by the way, and useful to know that's available in a pinch if I need to use a device where I don't have access to a main client) but no change to the audio sync issues.Remi - did you deploy your instance from the AMI or did you install and configure on an existing instance? My machine was deployed from the NICE DCV AMI so wanted to see if we can narrow down potential sources for the issue. I'm seeing the same audio lag on any application the system runs - my typical use is vMix but tests with VLC, online videos and other applications seem the same so it must be something to do with either how the system is sending the video and audio data or the way it's being processed and played out on the client side.Karsten - any ideas on potential avenues I could explore to help narrow down a cause?All the best,JoeEdited by: joemurphy on Nov 19, 2020 1:21 PMAdded notes on HTML5 test.CommentSharejoemurphyanswered 3 years ago"


Question: Give cluster admin access to EKS worker nodes.

We have a EKS cluster running 1.21 version. We want to give admin access to worker nodes. We modified the aws-auth config map and added "system:masters" for eks worker nodes role. Below is the code snipped for the modified configmap.data:  mapRoles: |    - groups:      - system:nodes      - system:bootstrappers      - system:masters      rolearn: arn:aws:iam::686143527223:role/terraform-eks-worker-node-role      username: system:node:{{EC2PrivateDNSName}}After adding this section the EKS worker nodes got admin access to cluster successfully. But in EKS dashboard the nodegroups are in degraded state. It shows below error in Health issues section.Your worker nodes do not have access to the cluster. Verify if the node instance role is present and correctly configured in the aws-auth ConfigMap.FollowComment

Answer: "0Hi @abhinav,Confirm that your control plane's security group and worker node security group are configured with the recommended settings for inbound and outbound traffic. Also, confirm that your custom network ACL rules are configured to allow traffic to and from "0.0.0.0/0" for ports 80,443 and 1025-65535.Refer-https://aws.amazon.com/premiumsupport/knowledge-center/eks-worker-nodes-cluster/If the Answer is helpful, please click  Accept Answer & UPVOTE, this can be beneficial to other community members.CommentSharelearn2skillsanswered 8 months agoabhinav 8 months agoIt's all fine. Whenever I remove the line system:masters everything works fine. Something wrong with this line.Shareabhinav 8 months agoI removed the system:masters from EKS worker node role and gave access to jenkins agent pods via service account bound to cluster-admin role via ClusterRoleBinding and the error was gone.Share0I've tested this with EKS 1.22 and 1.23, and was able to reproduce. Besides the node group health issues surfaced in the console and via the below CLI command, I saw no actual degradation of the nodes.aws eks describe-nodegroup --cluster-name <CLUSTER_NAME> --nodegroup-name <NG_NAME>...        "health": {            "issues": [                {                    "code": "AccessDenied",                    "message": "Your worker nodes do not have access to the cluster. Verify if the node instance role is present and correctly configured in the aws-auth ConfigMap.",                    "resourceIds": [                        "eksctl-<CLUSTER_NAME>-nodegroup-<NG_NAME>-<NODE_INSTANCE_ROLE>"                    ]                }            ]        },I think this error message is benign. While the console reports an unhealthy node group, the individual nodes show as healthy.kubectl get nodes -oyaml | grep conditions -A 30CommentSharejimmyrayanswered 8 months agoabhinav 8 months agoyeah all the working is good. Just the EKS dashboard shows unhealthy. Due to that we were not able to update the cluster and also was not able to run Terraform code because it gave error that cluster is unhealthy.I removed the system:masters from EKS worker node role and gave access to jenkins agent pods via service account bound to cluster-admin role via ClusterRoleBindinga and things worked fine and these errors were gone.Shareabhinav 8 months agoHow were you able to reproduce this error?Share0I reproduced with your settings. I am looking deeper into this.CommentSharejimmyrayanswered 8 months ago0HI!This could be perhaps a simple answer, but I think the problem could be the missing dash in the pipe, -Correct:data:      mapRoles: |-Incorrect:data:      mapRoles: |See? missing dash, this basically is explained by "Block Chomping Indicator"Block Chomping Indicator: The chomping indicator controls what should happen with newlines at the end of the string. The default, clip, puts a single newline at the end of the string. To remove all newlines, strip them by putting a minus sign (-) after the style indicator. Both clip and strip ignore how many newlines are actually at the end of the block; to keep them all put a plus sign (+) after the style indicator. https://yaml-multiline.info/When this is applied to Kubernetes: [~]$ k -n kube-system get cm aws-auth-test  -oyaml    apiVersion: v1data:  mapRoles: |-    - groups:      - system:bootstrappers      - system:nodes      - system:masters      rolearn: myrole      username: system:node:valuekind: ConfigMapmetadata:  annotations:    kubectl.kubernetes.io/last-applied-configuration: |      {"apiVersion":"v1","data":{"mapRoles":"- groups:\n  - system:bootstrappers\n  - system:nodes\n  - system:masters\n  rolearn: myrole\n  username: system:node:value"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"aws-auth-test","namespace":"kube-system"}}  creationTimestamp: "2023-02-18T05:16:35Z"  name: aws-auth-test  namespace: kube-system  resourceVersion: "1947"  uid: eb241efd-a31a-45e8-b0fc-9f486ef5fff1 [~]$ k -n kube-system get cm aws-auth-test-bad -oyaml apiVersion: v1data:  mapRoles: |    - groups:      - system:bootstrappers      - system:nodes      - system:masters      rolearn: myrole      username: system:node:valuekind: ConfigMapmetadata:  annotations:    kubectl.kubernetes.io/last-applied-configuration: |      {"apiVersion":"v1","data":{"mapRoles":"- groups:\n  - system:bootstrappers\n  - system:nodes\n  - system:masters\n  rolearn: myrole\n  username: system:node:value\n"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"aws-auth-test-bad","namespace":"kube-system"}}  creationTimestamp: "2023-02-18T05:19:12Z"  name: aws-auth-test-bad  namespace: kube-system  resourceVersion: "1990"  uid: 1beebe54-58d5-4515-808c-b8333b1aec0bIt seems valid, but now, there is a line break at the end of the role: system:node:value\n under the annotation kubectl.kubernetes.io/last-applied-configuration, and maybe that's what Console is retrieving. Its an educated guess, I haven't tested under any  AWS Cluster, but you might want to give it a try.If you run the same test you will notice a blank space on the yaml if retrieved:# bad one with break at the end [~]$ k -n kube-system get cm aws-auth-test-bad -oyaml | yq e '.data.mapRoles'- groups:  - system:bootstrappers  - system:nodes  - system:masters  rolearn: myrole  username: system:node:value                                                       #<----- this is line break from the bad configmap# good one, no break at the end. [~]$ k -n kube-system get cm aws-auth-test -oyaml | yq e '.data.mapRoles' - groups:  - system:bootstrappers  - system:nodes  - system:masters  rolearn: myrole  username: system:node:valueHope this helps!CommentShareDavidCaballeroanswered 3 months ago"


Question: Purchasing savings plans

"Hello, if I have ec2 instances that belong to an EKS group, and I buy savings plans of the ec2 type, is the discount applied or does it only apply to instances that are not under a managed service?FollowComment"

Answer: "1Accepted AnswerHi,According to https://docs.aws.amazon.com/savingsplans/latest/userguide/what-is-savings-plans.html, EC2 Instance saving plans do NOT automatically apply to Amazon ECS and Amazon EKS using Fargate. I have updated above answer to reflect also this.On the other hand, Compute saving plans do apply also for ec2 under managed services. https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-fargate-aws-eks-included-compute-savings-plan/Hope it helpsCommentShareEXPERTalatechanswered 4 months agohmunoz 4 months agoHello,I understand that with compute savings plans I would include the EC2 instances under EKS, but the same thing also happens for the EC2 savings plans type?Sharealatech EXPERT4 months agoI see what you mean. According to the table here https://docs.aws.amazon.com/savingsplans/latest/userguide/what-is-savings-plans.html EC2 Instance saving plans do NOT automatically applies to Amazon ECS and Amazon EKS using Fargate. I have updated above answer to reflect also this.Share"


Question: Permissions needed to get shadow from IOT Core from a greengrass code device

"I am trying to get a shadow named cycle_topic from AWS IOT thing definition. The permissions/configurations are what are tripping me up. It seems to me that I have a correct permission in ShadowManager for SliceOfPi thing to get count_topic. I am believing that this is a permission issue. What is wrong with my config?I have the following configs for the following components:Configure aws.greengrass.clientdevices.mqtt.Bridge:{  "reset": [],  "merge": {    "mqttTopicMapping": {      "HelloWorldIotCoreMapping": {        "topic": "#",        "source": "Pubsub",        "target": "IotCore"      }    },    "ShadowsPubsubToLocalMqtt": {      "topic": "$aws/things/+/shadow/#",      "source": "Pubsub",      "target": "LocalMqtt"    }  }}Configure aws.greengrass.clientdevices.Auth:{  "reset": [],  "merge": {    "deviceGroups": {      "formatVersion": "2021-03-05",      "definitions": {        "MyPermissiveDeviceGroup": {          "selectionRule": "thingName: *",          "policyName": "MyPermissivePolicy"        }      },      "policies": {        "MyPermissivePolicy": {          "AllowAll": {            "statementDescription": "Allow client devices to perform all actions.",            "operations": [              "*"            ],            "resources": [              "*"            ]          }        }      }    }  }}aws.greengrass.ShadowManager:{  "reset": [],  "merge": {    "strategy": {      "type": "realTime"    },    "synchronize": {      "coreThing": {        "classic": true,        "namedShadows": [          "count_topic"        ]      },      "shadowDocuments": [        {          "thingName": "SliceOfPi",          "classic": true,          "namedShadows": [            "count_topic"          ]        }      ],      "direction": "cloudToDevice"    }  }}The error I am getting in greengrass.log is:2022-09-20T21:51:46.237Z [WARN] (Thread-4) com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler: handle-get-thing-shadow. Not authorized to get shadow. {thing name=SliceOfPi, shadow name=count_topic}com.aws.greengrass.authorization.exceptions.AuthorizationException: Principal com.xxxxxxxxxx.productivity.cycle_count is not authorized to perform aws.greengrass.ShadowManager:aws.greengrass#GetThingShadow on resource $aws/things/SliceOfPi/shadow/name/count_topicat com.aws.greengrass.authorization.AuthorizationHandler.isAuthorized(AuthorizationHandler.java:247)at com.aws.greengrass.authorization.AuthorizationHandler.isAuthorized(AuthorizationHandler.java:255)at com.aws.greengrass.shadowmanager.AuthorizationHandlerWrapper.doAuthorization(AuthorizationHandlerWrapper.java:73)at com.aws.greengrass.shadowmanager.AuthorizationHandlerWrapper.doAuthorization(AuthorizationHandlerWrapper.java:56)at com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler.lambda$handleRequest$0(GetThingShadowRequestHandler.java:87)at com.aws.greengrass.ipc.common.ExceptionUtil.translateExceptions(ExceptionUtil.java:33)at com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler.handleRequest(GetThingShadowRequestHandler.java:75)at com.aws.greengrass.shadowmanager.ipc.GetThingShadowIPCHandler.handleRequest(GetThingShadowIPCHandler.java:82)at com.aws.greengrass.shadowmanager.ipc.GetThingShadowIPCHandler.handleRequest(GetThingShadowIPCHandler.java:28)at software.amazon.awssdk.eventstreamrpc.OperationContinuationHandler.onContinuationMessage(OperationContinuationHandler.java:291)at software.amazon.awssdk.crt.eventstream.ServerConnectionContinuationHandler.onContinuationMessageShim(ServerConnectionContinuationHandler.java:53)FollowComment"

Answer: "2Accepted AnswerHow are you trying to access the shadow? I'm noticing that you are using the MQTT bridge component to bridge from PubSub to IoT Core. Generally you shouldn't need to do that, as components are able to publish directly to IoT Core. The Client Device Auth and MQTT Bridge components are only necessary if you are trying to connect external client devices to Greengrass.Based on that stack trace, I suspect you are actually trying to access shadow from a component, using the GetThingShadow IPC API. You can see the authorization failure log below:com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler: handle-get-thing-shadow. Not authorized to get shadow. {thing name=SliceOfPi, shadow name=count_topic} com.aws.greengrass.authorization.exceptions.AuthorizationException: Principal com.xxxxxxxxxx.productivity.cycle_count is not authorized to perform aws.greengrass.ShadowManager:aws.greengrass#GetThingShadow on resource $aws/things/SliceOfPi/shadow/name/count_topic...You can fix this by updating the accessControl section for the cycle_count component.https://docs.aws.amazon.com/greengrass/v2/developerguide/ipc-local-shadows.html#ipc-local-shadow-authorizationCommentShareJoeAtAWSanswered 8 months agoflycast 8 months agoI'm noticing that you are using the MQTT bridge component to bridge from PubSub to IoT Core. Generally you shouldn't need to do that, as components are able to publish directly to IoT Core. The Client Device Auth and MQTT Bridge components are only necessary if you are trying to connect external client devices to Greengrass."OH MY GOSH!!! So helpful!!! WOW! I thought the bridge and auth was necessary for that based on some AWS tutorials. Thanks a BUNCH for clearing that up.Shareflycast 8 months agoThis was super helpful. Part of the key was that I don't need auth, bridge. That simplified things quite a bit and cleared my confusion up. The link referenced was also pointed to by @Joseph Cosentino. The gamechanger was simplifying.Share1There are two sets of permissions needed. They ones you noted (thanks for the detail!) are from a component perspective, but not for how the Nucleus and ShadowManager component interact with AWS IoT.Most likely the permissions are missing from the Role Alias, or the IoT policy associated with the core device. Please check and see for the Role Alias the permissions called out here, specifically for AllowGreengrassAccessToShadows are included. If they are, check the IoT policy.Please let me know if this addresses your problem of if you have other details.CommentShareGavin_Aanswered 8 months agoflycast 8 months agoThanks @Gavin_A. I posted an "Answer" because the text was too long for a comment.Share0Too long to make this a comment.My service role currently looks like this:{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "AllowGreengrassAccessToShadows",            "Action": [                "iot:DeleteThingShadow",                "iot:GetThingShadow",                "iot:UpdateThingShadow"            ],            "Effect": "Allow",            "Resource": [                "arn:aws:iot:*:*:thing/GG_*",                "arn:aws:iot:*:*:thing/*-gcm",                "arn:aws:iot:*:*:thing/*-gda",                "arn:aws:iot:*:*:thing/*-gci"            ]        },        {            "Sid": "AllowGreengrassToDescribeThings",            "Action": [                "iot:DescribeThing"            ],            "Effect": "Allow",            "Resource": "arn:aws:iot:*:*:thing/*"        },        {            "Sid": "AllowGreengrassToDescribeCertificates",            "Action": [                "iot:DescribeCertificate"            ],            "Effect": "Allow",            "Resource": "arn:aws:iot:*:*:cert/*"        },        {            "Sid": "AllowGreengrassToCallGreengrassServices",            "Action": [                "greengrass:*"            ],            "Effect": "Allow",            "Resource": "*"        },        {            "Sid": "AllowGreengrassToGetLambdaFunctions",            "Action": [                "lambda:GetFunction",                "lambda:GetFunctionConfiguration"            ],            "Effect": "Allow",            "Resource": "*"        },        {            "Sid": "AllowGreengrassToGetGreengrassSecrets",            "Action": [                "secretsmanager:GetSecretValue"            ],            "Effect": "Allow",            "Resource": "arn:aws:secretsmanager:*:*:secret:greengrass-*"        },        {            "Sid": "AllowGreengrassAccessToS3Objects",            "Action": [                "s3:GetObject"            ],            "Effect": "Allow",            "Resource": [                "arn:aws:s3:::*Greengrass*",                "arn:aws:s3:::*GreenGrass*",                "arn:aws:s3:::*greengrass*",                "arn:aws:s3:::*Sagemaker*",                "arn:aws:s3:::*SageMaker*",                "arn:aws:s3:::*sagemaker*"            ]        },        {            "Sid": "AllowGreengrassAccessToS3BucketLocation",            "Action": [                "s3:GetBucketLocation"            ],            "Effect": "Allow",            "Resource": "*"        },        {            "Sid": "AllowGreengrassAccessToSageMakerTrainingJobs",            "Action": [                "sagemaker:DescribeTrainingJob"            ],            "Effect": "Allow",            "Resource": [                "arn:aws:sagemaker:*:*:training-job/*"            ]        }    ]}Right at the top it looks like I am allowed shadow access.CommentShareflycastanswered 8 months agoGavin_A 8 months agoThe resources for the shadows actions looks to be different than one of the shadows you're trying to get. This is from your logs: $aws/things/SliceOfPi/shadow/name/count_topic,but the resource filters are:                "arn:aws:iot:*:*:thing/GG_*",                "arn:aws:iot:*:*:thing/*-gcm",                "arn:aws:iot:*:*:thing/*-gda",                "arn:aws:iot:*:*:thing/*-gci"which don't match. To test, try adding the resource arn:aws:iot:*:*:thing/SliceOfPi and see what the log file shows.Share0It appears that IPC permissions are missing for aws.greengrass#GetThingShadow.Please take a look at the following guide for allowing access: https://docs.aws.amazon.com/greengrass/v2/developerguide/ipc-local-shadows.htmlCommentShareJoseph Cosentinoanswered 8 months agoflycast 8 months agoThank you for that. Where do these permissions go? ShadowManager? Auth? Bridge?Share"


Question: SSH- Server Refused our Key

I am getting server refused our key when connecting to my EC2 instance from MobaXterm. Security group and key pair name is correct. I’m using a puttgen .ppk version of the Private key. It worked the first and only other time I connected to this instance. Could something have happened to the key file since then or am I missing something else?FollowComment

Answer: "1Sometimes this is caused by not sending the correct login id (ec2-user, ubuntu, etc) when connecting.CommentShareEXPERTkentradanswered a year ago0Another possible reason is incorrect permissions for user's (ec2-user, ubuntu,centos etc) home directory and SSH files (~/.ssh/).  You can use the AWSSupport-TroubleshootSSH Systems Manager Automation runbook to troubleshoot common SSH issues in EC2 Linux instances.  Refer to following AWS Knowledge Centre article for further details.https://aws.amazon.com/premiumsupport/knowledge-center/ec2-ssh-errors-automation-workflow/CommentShareRenjith_Ranswered a year ago"


Question: pass custom event to a cognito lambda trigger : Post authentication Lambda trigger

"Hi team,I have a lambda function#1 that creates a user inside a Cognito user pool using adminCreateUser API call,after that, I call (in the same lambda function #1) adminInitiateAuth to have the 'NEW_PASSWORD_REQUIRED' challenge.then Cognito will call my second lambda function#2: Post authentication Lambda trigger.I would like to pass to the Post authentication Lambda trigger (lambda function#2) the initAuthResponse result from the adminInitiateAuth call (made in my first lambda function#1).so I can have the Cognito session (initAuthResponse.session) in lambda function#2 and store it in Redis cacheI'm wondering if that's feasible to pass an event to a Post authentication Lambda trigger or only Cognito can do that (pass a custom event to a Post authentication Lambda trigger in Cognito)?does the session (initAuthResponse.session) result from the adminInitiateAuth have an expiry time? or never expire, don't find documentation about the expiration date of the session (adminInitiateAuthResult.session)FollowCommentOwen 8 months agoUnfortunately it isn't possible to pass your own information into the post authentication trigger.  Cognito issues the event as described here - https://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-post-authentication.html.  What are you trying to accomplish with the triggers?Share"

Answer: "0Hello,Greetings from AWS !I understand that you would like to pass to the Post authentication Lambda trigger (lambda function#2) the admininitAuthResponse result from the adminInitiateAuth call (made in my first lambda function#1).Please feel free to correct me in case I have misunderstood your concern.Please find the answers below :1)I'm wondering if that's feasible to pass an event to a Post authentication Lambda trigger or only Cognito can do that (pass a custom event to a Post authentication Lambda trigger in Cognito)?Answer: As per the link[1], Please be informed that The ClientMetadata value is passed as input to the functions for only the following triggers:a)Pre signupb)Pre authenticationc)User migrationSo, I would like to confirm that it is not feasible to pass an event to a Post authentication Lambda trigger.2)Does the session (initAuthResponse.session) result from the adminInitiateAuth have an expiry time? or never expire, don't find documentation about the expiration date of the session (adminInitiateAuthResult.session)Answer: Yes, It has expiry time. By default, your users must complete each challenge within 3 minutes. If you want to provide your users with more time, up to 15 minutes per API request, you can adjust this value. please refer to the link[2] for configuring Authentication flow session duration.I believe the information is helpful to you. In case you have any further queries/concerns then please let us know. We will be more than happy to assist you further.Wish you an Awesome day ahead and stay safe !--References--[1] https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_AdminInitiateAuth.html#API_AdminInitiateAuth_RequestParameters[2] https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-authentication-flow.htmlCommentShareSanjay_Sanswered 8 months ago"


Question: NICE DCV with external authentication -- can I avoid supplying a password to unlock the screen? (Web client with Linux)

To unlock the screen (gnome lock screen) the user needs a password.  We're trying to avoid giving passwords; we use external authentication so users log in via our login system.  Any workarounds?FollowComment

Answer: 0I logged this is a bug (the said feature request). See my original question here: https://repost.aws/questions/QUNQuF_O0BQjuPNDg1eEhz9Q/use-dcv-gateway-but-it-is-not-logging-into-instance-brings-up-ctrl-alt-delete-screenBefore External Authentication we were passing username and passwords as query params when connecting via Browser with a really long bogus/unused param before it to push the username and password to the end.CommentShareLtnSnoopyanswered 10 months ago


Question: what permision on lambda Role are needed for "compute optimizer" to work?

basically as in the tilte.  What permissions are needed on a lambda Role for "Compute optimizer" to work with it?FollowComment

Answer: "0The following policy statement grants access to opt in to Compute Optimizer. It grants access to create a service-linked role for Compute Optimizer. This role is required to opt in. For more information, see Using Service-Linked Roles for AWS Compute Optimizer. It also grants access to update the enrollment status to the Compute Optimizer service.{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": "iam:CreateServiceLinkedRole","Resource": "arn:aws:iam:::role/aws-service-role/compute-optimizer.amazonaws.com/AWSServiceRoleForComputeOptimizer","Condition": {"StringLike": {"iam:AWSServiceName": "compute-optimizer.amazonaws.com"}}},{"Effect": "Allow","Action": "iam:PutRolePolicy","Resource": "arn:aws:iam:::role/aws-service-role/compute-optimizer.amazonaws.com/AWSServiceRoleForComputeOptimizer"},{"Effect": "Allow","Action": "compute-optimizer:UpdateEnrollmentStatus","Resource": ""}]}For other details & policies, refer to the link below:https://docs.aws.amazon.com/compute-optimizer/latest/ug/security-iam.html -- [1]CommentShareSUPPORT ENGINEERAWS-User-Chiraganswered 9 months ago"


Question: Athena scans all data when querying for struct field

"Hi,Let's say I have an AWS Glue table with the following schema:CREATE EXTERNAL TABLE `simulated_data`(  `stream_src` string,  `raw_data` string,   `my_struct` struct<date:string,time:string,value:float>)ROW FORMAT SERDE   'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT   'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT   'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'LOCATION  's3://my-bucket/'TBLPROPERTIES (  'classification'='parquet',   'parquet.compress'='SNAPPY',   'transient_lastDdlTime'='1565760085')The data is stored in S3 in multiple Apache Parquet files. The raw_data field contains JSON string with the same data (and structure) as my_struct struct. Example of data row is below:{    "stream_src": "src1",    "raw_data": "{\"date\":\"\",\"value\":18.7,\"time\":\"5:30\"}",    "my_struct": {        "date": "",        "time": "5:30",        "value": 18.7    }}I have a problem with querying the structure - Athena scans all data stored in the S3 bucket. The query that I use is:Run time: 2.36 seconds, Data scanned: 1.88 MB (same size as select * from simulated_data )SELECT my_struct.timeFROM simulated_dataWHERE stream_src='src1'Surprisingly, the problem does not exist with JSON_extractRun time: 2.02 seconds, Data scanned: 1.12 MBSELECT json_extract(raw_data, '$.time')FROM simulated_dataWHERE stream_src='src1'What is the reason for that? Why reading field of structure leads to scanning the whole data set? Answering those questions probably will give me a light on why is JSON_extract faster than selecting data from the struct.FollowComment"

Answer: "0Accepted AnswerThere's some quirks with Presto (underlying engine) and nested types (array/map/struct). This is mentioned briefly here - https://docs.aws.amazon.com/athena/latest/ug/other-notable-limitations.html"When you query columns with complex data types (array, map, struct), and are using Parquet for storing data, Athena currently reads an entire row of data, instead of selectively reading only the specified columns as expected. This is a known issue."If you dig on the PrestoDB git repo (Athena is based on an older version of this afaik e.g. 0.176 or something like that) there's various issues about predicate pushdown/optimizations of reading nested fields e.g. this issue and the linked issues - https://github.com/prestodb/presto/issues/11326 - although I'm not sure on the current state of this.I'm not sure if this is better for ORC possibly (maybe worth testing if you're willing to change file formats). Otherwise it's basically:Optimization makes its way into PrestoAthena uses the version of Presto that has that optimization (or just backports it/implements it themselves)CommentSharerruppmgpanswered 4 years ago0Thank you for your answer. I will try with ORC.Edit:It seems to be working better with ORC. However, it is worth to notice, that ORC has other limitation - as long as stripe size is below 8MB (default for Presto), it will read the whole file anyway - doesn't matter whether query regarded struct or simple field.Edited by: offroader on Aug 20, 2019 8:37 PMCommentShareoffroaderanswered 4 years ago0Yeah, I ran into this as well, even with the example data sets that AWS uses, mentioned here - https://forums.aws.amazon.com/message.jspa?messageID=841338#841338 and here https://forums.aws.amazon.com/thread.jspa?messageID=846250I think maybe there's some tool or version of tool (old version of Hive/Spark or something?) that must set 8MB as the default or something because I've seen this happen multiple times from other people even when the actual files they're generating are > 8MB e.g. the AWS example files I think were 200MB+ * ~28 files (file per year).If I look at the ORC Java library for instance the default stripe size is 64MB - https://orc.apache.org/docs/hive-config.htmlEdited by: rruppmgp on Aug 21, 2019 6:52 AMCommentSharerruppmgpanswered 4 years ago"


Question: Resolve "protocol version mismatch" error when launching new terminal on SSH based environment

"I have created a new SSH based Environment with a inf2.xlarge based instance and am having problems when creating a new terminal window as it is giving me the following repeated error message:protocol version mismatch (client 8, server 7)Do I need to update my SSH server on the instance ? The current version of SSH on the EC2 instance is the following:OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017FollowComment"

Answer: 0Accepted AnswerIt seems to be a tmux issue. I resolved it by killing all the tmux sessions with the following command: sudo killall -9 tmuxCommentShareEXPERTAWS-MattManswered a month agoEXPERTAWS-User-Nitinreviewed a month ago


Question: RemoteCertificateNameMismatch error when using Gamelift generated certificat

When trying to securely connect a client through an Gamelift SSL generated certificate to a Game session we got an RemoteCertificateNameMismatch error. We are using a Websocket communication between game client based on Unity and the game session.FollowComment

Answer: "0Accepted AnswerWhen connecting to a game session that is running on a TLS-enabled fleet, you must use the DNS name, not the IP address.DNS name can be retrieved when getting GameSession object and is format following this pattern:<unique identifier>.<region identifier>.amazongamelift.com.https://docs.aws.amazon.com/gamelift/latest/apireference/API_GameSession.html#gamelift-Type-GameSession-DnsNameCommentShareSteve_Hanswered 8 months ago"


Question: Data Migration Service from RDS to Redshift Failed

"Last Error Error 1236 reading binary log. Task error notification received from subtask 0, thread 0 [reptask/replicationtask.c:2883] [1020493] Error executing source loop; Stream component failed at subtask 0, component st_0_GI7QJS2UGYJBQ2O7OEJDBDVH3ZAU2QC53GUFBTQ; Stream component 'st_0_GI7QJS2UGYJBQ2O7OEJDBDVH3ZAU2QC53GUFBTQ' terminated [reptask/replicationtask.c:2891] [1020493] Stop Reason FATAL_ERROR Error Level FATALHere is the error message I received? Can someone help explaining why this happens and how can I resolve this?FollowComment"

Answer: "0I'm not familiar enough with DMS to explain the reasoning behind this error, however I did see that someone had a similar problem with the same error which is discussed here: https://stackoverflow.com/questions/69579734/aws-dms-task-failed-with-error-error-executing-source-loop-stream-component-fa - It may lead you to resolving your problemCommentShareStuart Scottanswered 5 months agorePost-User-8868240 4 months agoThank you so much! This makes senseShare"


Question: Additional information on GuardDuty DNS Findings

I am receiving DNS related GuardDuty findings for "querying algorithmically generated domains" that we suspect are not algorithmically generated.An example URL is from the following Facebook page: https://www.facebook.com/SOMD-167556163301693/The URL in question is http://stuckonmsdawn.com/So my questions:Why is GuardDuty flagging this as algorithmically generated?Is there a way of checking with AWS whether they think a domain is algorithmically generated (or as part of a "Command & Control server" also) before actually making the DNS query itself?ThanksFollowComment

Answer: "1There are 2 types of DGA related GuardDuty findings i.e. DGADomainRequest.B and DGADomainRequest.C!DNS.For this finding EC2/DGADomainRequest.B: it is based on analysis of domain names using advanced heuristics and may identify new DGA domains that are not present in threat intelligence feeds. If you believe the domain has been incorrectly identified, please raise a technical support ticket with AWS support.For this finding DGADomainRequest.C!DNS: it is based on known DGA domains from GuardDuty's threat intelligence feeds.Please refer to this link for additional details of the DGA related findings.CommentShareKuok_Kanswered a year agoAWS-User-2792197 a year agoFinding a lot of this lately, does not seem very intelligent. Lots of false positives.Share"


Question: EKS - Get availability zone of pod deployment from within pod

"In my EKS cluster, I have a few node groups that cover availability zone A and B. The pods running on the nodes are consumers/producers on various Kafka topics (MSK). There isn't any internal communication between the pods.I have two brokers for the MSK cluster, one in each availability zone that the pods use.I am concerned that since the pods can be deployed in either availability zone, I might be unknowingly charged for data that is being transferred between the availability zones as the pod doesn't have a preference to which broker it talks toTo avoid this, I am thinking of having the pod select the broker server to talk to once it has started, and to do this it needs to know what availability zone the underlying node its been deployed on.I was imagining that it would be possible to have the deployment pull the zone information like any other spec field e.g. https://docs.microsoft.com/en-us/answers/questions/330925/how-can-a-pod-know-which-availability-zone-it-loca.html but that isn't a valid field.I have tried hitting the ec2 metadata ip address like the link below but since its a pod with its own IP on the node, I assume this mechanism doesn't work. https://www.quora.com/How-do-I-find-out-in-which-availability-zone-a-AWS-EC2-server-isNone of the field listed https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podspec-v1-core seem to have any field that could identify this.Is there a way of doing this? Or is there a better way of networking the pods/node groups?FollowComment"

Answer: 1Accepted AnswerEC2 Instance Metadata Service v2 can be leveraged to get the AZ details. Amazon EKS supports containers to access instance metadata using the IMDSv2 format.https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-eks-supports-ec2-instance-metadata-service-v2/CommentShareRama Krishna Sanjeevaanswered a year agoKieran Bacon a year agoGreat thanks.Share


Question: How to use Reserved Instances for Gamelift fleets

"When reserved instances are purchased from EC2 > Reserved Instances, then I would guess that gamelift fleets can not use these instances? Is there a way to reserve instances to be used for gamelift fleets? My gamelift fleet has predictable usage and I would like to reserve instances for next 12 months and save costs.Thanks!FollowComment"

Answer: "0You have a few options in Gamelift including Gamelift Spot which, if suitable for you, is likely to yield the best savings.  Or Gamelift Anywhere allows you to use self-managed EC2 instances which you can cover with Reservations or Savings Plans.  Savings Plans are definitely a better option than Reservations for compute resources - see https://www.linkedin.com/pulse/aws-savings-plans-reservations-perils-over-optimising-steve-kinsman/ for a discussion of trade-offs.CommentShareEXPERTskinsmananswered 4 months ago"