Question: How to run the tensorflow neuron in SageMaker endpoint for production

"We have a huggingfacemodel with zero-shot-classification with neuron infernetia. It's based on the pretrained huggingface pipelines distilBert with TensorFlow2 neuron with zero-shot-classification model.We planned to use it in production environment since it reduced the latency from 1s to 100ms.However, the Sagemaker Python SDK HuggingFaceModel seems not support tensorflow 2 neuron. It gave error like bellow.My question is that how to run this tensorflow2 neuron under sageamaker?If huggingfacemodel doesn't support tensorflow2, can you provide a pytorch version for hugginface pipeline. There isn't any example of implement neuron for huggingface pipeline.Is there any other way like create dockerfile?Thanks a lot# create Hugging Face Model Classhuggingface_model = HuggingFaceModel(   model_data="s3://sagemaker-us-west-2-**********/inf1/model.tar.gz",      # path to your model and script   role=role,                    # iam role with permissions to create an Endpoint   transformers_version="4.6.1",  # transformers version used   tensorflow_version="2.4.1",        # pytorch version used   py_version='py37',            # python version used)huggingface_model._is_compiled_model = True# deploy the endpoint endpointpredictor = huggingface_model.deploy(    initial_instance_count=1,      # number of instances    instance_type="ml.inf6.xlarge" # AWS Inferentia Instance)We got responseDefaulting to the only supported framework/algorithm version: 4.12.3. Ignoring framework/algorithm version: 4.6.1.---------------------------------------------------------------------------ValueError                                Traceback (most recent call last)/tmp/ipykernel_18249/2024686093.py in <module>      2 predictor = huggingface_model.deploy(      3     initial_instance_count=1,      # number of instances----> 4     instance_type="ml.inf1.xlarge" # AWS Inferentia Instance      5 )~/anaconda3/envs/amazonei_pytorch_latest_p37/lib/python3.7/site-packages/sagemaker/huggingface/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, async_inference_config, serverless_inference_config, volume_size, model_data_download_timeout, container_startup_health_check_timeout, inference_recommendation_id, **kwargs)    303     304         return super(HuggingFaceModel, self).deploy(--> 305             initial_instance_count,    306             instance_type,    307             serializer,~/anaconda3/envs/amazonei_pytorch_latest_p37/lib/python3.7/site-packages/sagemaker/huggingface/model.py in serving_image_uri(self, region_name, instance_type, accelerator_type, serverless_inference_config)~/anaconda3/envs/amazonei_pytorch_latest_p37/lib/python3.7/site-packages/sagemaker/workflow/utilities.py in wrapper(*args, **kwargs)    386     387 --> 388 def execute_job_functions(step_args: _StepArguments):    389     """Execute the job class functions during pipeline definition construction    390 ~/anaconda3/envs/amazonei_pytorch_latest_p37/lib/python3.7/site-packages/sagemaker/image_uris.py in retrieve(framework, region, version, py_version, instance_type, accelerator_type, image_scope, container_version, distribution, base_framework_version, training_compiler_config, model_id, model_version, tolerate_vulnerable_model, tolerate_deprecated_model, sdk_version, inference_tool, serverless_inference_config)    172             )    173         _validate_arg(full_base_framework_version, list(version_config.keys()), "base framework")--> 174         version_config = version_config.get(full_base_framework_version)    175     176     py_version = _validate_py_version_and_set_if_needed(py_version, version_config, framework)~/anaconda3/envs/amazonei_pytorch_latest_p37/lib/python3.7/site-packages/sagemaker/image_uris.py in _validate_arg(arg, available_options, arg_name)    569     """Creates a tag for the image URI."""    570     if inference_tool:--> 571         return "-".join(x for x in (tag_prefix, inference_tool, py_version, container_version) if x)    572     return "-".join(x for x in (tag_prefix, processor, py_version, container_version) if x)    573 ValueError: Unsupported base framework: tensorflow2.4.1. You may need to upgrade your SDK version (pip install -U sagemaker) for newer base frameworks. Supported base framework(s): version_aliases, pytorch1.9.1.FollowComment"

Answer: "0Hi Xin Tong,Thanks for posting the question. HuggingFace Neuron Inference Containers are currently only available for PyTorch.  Please file a feature request on https://github.com/aws/deep-learning-containers for TensorFlow 2.x HuggingFace Neuron Inference Container support.CommentShareJeffH-AWSanswered 4 months ago"


Question: "How to setup Lambda Authorizer for a REST API added in Amplify,"

"I cannot find a way to set up Lambda Authorizer for the Rest API I added using AWS amplify in my amplify app. The only options it shows are for authenticated and unauthenticated users and does not let me configure anything else.Maybe it can be done with overrides.ts but I could not find an example of how to do it, any guidance will be appreciated.FollowComment"

Answer: "0This is how I solved it.amplify override API to create an overrides.ts file and add the following. // Create a lambda authorizer  resources.restApi.addPropertyOverride("Body.securityDefinitions", {    Custom: {      type: "apiKey",      name: "Authorization",      in: "header",      "x-amazon-apigateway-authtype": "custom",      "x-amazon-apigateway-authorizer": {        type: "request",        authorizerUri: {          "Fn::Sub":            "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AuthorizerFunctionName}/invocations",        },        authorizerResultTtlInSeconds: 300,        identitySource: "method.request.header.Authorization",      },    },  });  // For every path in your REST API, add the authorizer for all methods  for (const path in resources.restApi.body.paths) {    resources.restApi.addPropertyOverride(      `Body.paths.${path}.x-amazon-apigateway-any-method.parameters`,      [        ...resources.restApi.body.paths[path]["x-amazon-apigateway-any-method"]          .parameters,        {          name: "Authorization",          in: "header",          required: false,          type: "string",        },      ]    );    // Use your new Cognito User Pool authorizer for security    resources.restApi.addPropertyOverride(      `Body.paths.${path}.x-amazon-apigateway-any-method.security`,      [ { Custom: [], }, ]    );  }CommentSharesarinanswered 21 days ago"


Question: Publish and Subscribe Time out in my code while sample project is working

"Hi,Im created my own AWS client class that i am using in one of my python project.The sample project connect_device_package is working perfectly and i can see the message coming on the aws iot console but when i'm trying to subscribe and publish in my code i have time out exceptionAny help would be greatly appreciated!def __init__(self, host, rootCA, cert,key):        #for websocket use 443        port = 8883          clientId = "basicPubSub"        self.topic = "Telemetry"        mode = "both"        useWebsocket = False        # Init AWSIoTMQTTClient        self.myAWSIoTMQTTClient = None        if useWebsocket:            self.myAWSIoTMQTTClient = AWSIoTMQTTClient(clientId, useWebsocket=True)            self.myAWSIoTMQTTClient.configureEndpoint(host, port)            self.myAWSIoTMQTTClient.configureCredentials(rootCA)        else:            self.myAWSIoTMQTTClient = AWSIoTMQTTClient(clientId)            self.myAWSIoTMQTTClient.configureEndpoint(host, port)            self.myAWSIoTMQTTClient.configureCredentials(rootCA, key, cert)        # AWSIoTMQTTClient connection configuration        self.myAWSIoTMQTTClient.configureAutoReconnectBackoffTime(1, 32, 20)        self.myAWSIoTMQTTClient.configureOfflinePublishQueueing(-1)         self.myAWSIoTMQTTClient.configureDrainingFrequency(2)  # Draining: 2 Hz        self.myAWSIoTMQTTClient.configureConnectDisconnectTimeout(15)  # 10 sec        self.myAWSIoTMQTTClient.configureMQTTOperationTimeout(15)  # 5 sec        self.myAWSIoTMQTTClient.connect()        if mode == 'both' or mode == 'subscribe':           self.myAWSIoTMQTTClient.subscribe(self.topic, 1, self.customCallback)        time.sleep(2)           self.myAWSIoTMQTTClient.publish(self.topic, 'This is a test Publish', 1)Exceptions:INFO:AWSIoTPythonSDK.core.protocol.mqtt_core:MqttCore initializedINFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Client id: basicPubSubINFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Protocol version: MQTTv3.1.1INFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Authentication type: TLSv1.2 certificate based Mutual Auth.INFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Configuring endpoint...INFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Configuring certificates...INFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Configuring reconnect back off timing...INFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Base quiet time: 1.000000 secINFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Max quiet time: 32.000000 secINFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Stable connection time: 20.000000 secINFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Configuring offline requests queueing: max queue size: -1INFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Configuring offline requests queue draining interval: 0.500000 secINFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Configuring connect/disconnect time out: 15.000000 secINFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Configuring MQTT operation time out: 15.000000 secINFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Performing sync connect...INFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Performing async connect...INFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Keep-alive: 600.000000 secINFO:AWSIoTPythonSDK.core.protocol.mqtt_core:Performing sync subscribe...ERROR:AWSIoTPythonSDK.core.protocol.mqtt_core:Subscribe timed outTraceback (most recent call last):  File "main.py", line 75, in <module>    main(args)  File "main.py", line 44, in main    client = NetworkQueue(local_dump, AWSNetworkClient(mqttHost,rootCA,cert,key))  File "AWSNetworkClient.py", line 45, in __init__    self.myAWSIoTMQTTClient.subscribe(self.topic, 0, self.customCallback)  File "AWSIoTPythonSDK/MQTTLib.py", line 668, in subscribe    return self._mqtt_core.subscribe(topic, QoS, callback)  File "AWSIoTPythonSDK/core/protocol/mqtt_core.py", line 302, in subscribe    raise subscribeTimeoutException()AWSIoTPythonSDK.exception.AWSIoTExceptions.subscribeTimeoutExceptionSame thing for the Publish:File "AWSIoTPythonSDK/MQTTLib.py", line 600, in publish    return self._mqtt_core.publish(topic, payload, QoS, False)  # Disable retain for publish by now  File "AWSIoTPythonSDK/core/protocol/mqtt_core.py", line 269, in publish    raise publishTimeoutException()AWSIoTPythonSDK.exception.AWSIoTExceptions.publishTimeoutExceptionDid someone had the same issue ?If yes, How did you solve that ?ThanksEdited by: Doombqr on Feb 17, 2019 12:27 PMFollowComment"

Answer: "0It was just a policy issue.I followed this article and it fixes it.http://www.fhilitski.com/2016/09/connecting-mqtt-fx-to-aws-iot/Thanks anywayEdited by: Doombqr on Feb 17, 2019 2:56 PMCommentShareDoombqranswered 4 years ago"


Question: cloudwatch alarm for redshift queryduration

"I have below cloudwatch alarm defined in CF template for altering me on queries running for 30 mins or more.    Type: AWS::CloudWatch::Alarm      Properties:         AlarmName: !Sub "awsredshift-${RSClusterName}-QueryDuration"        AlarmDescription: Redshift QueryDuration Alarm        Namespace: AWS/Redshift        MetricName: QueryDuration        Dimensions:           - Name: ClusterIdentifier            Value: !Ref RSClusterName          - Name: latency            Value: long        ActionsEnabled: true        AlarmActions:           - !Ref TopicARN        OKActions:           - !Ref TopicARN        ComparisonOperator: GreaterThanOrEqualToThreshold        DatapointsToAlarm: 1        EvaluationPeriods: 1        Period: 300        Statistic: Average        Threshold: 1800000000        TreatMissingData: missing  But its activating the alarms when there are no queries running that long, am I missing something?Also is there any way to customize the alarms to put logic in them, I would like to get the SQL text of the query which is running longer. Is there any way to do this via cloudwatch alarms? If not whats the best way to do it - probably lambda?FollowComment"

Answer: "0Accepted AnswerHi nmakb,I think you're assuming the Redshift CloudWatch QueryDuration metric is something it's not.  It's not a query level metric.  It's a binned a QueryDuration dimensioned by ClusterIdentifier, latency (short, medium, long bins).  Then in your CloudWatch Alarm you're aggregating that in an average over a 300 period.Furthermore, I don't see a query id or anything related to join the individual metric data points data back to the source query.I think the best you could do with the current QueryDuration metric would be to create a CloudWatch Alarm on the metric.  trigger the Alarm on a theshold, set the action to a AWS Lambda to find queries in the alarm interval that were above the threshold.  Then lastly do something with the result in the Lamdba, like trigger another SNS topic or send an email.I hope this helps clarify what the CloudWatch QueryDuration metric is and isn't and provides you at least an idea of how you could build toward what you're looking to do.Regards,-KurtCommentShareklarsonanswered 4 years ago"


Question: Preview request responded with incorrect contents via email

"We have requested for the preview a few days back and received an email subjected "Waitlisted: Amazon Textract Preview".But in the content, it says, the account ID is invalid. We have doubled checked the ID (I don't know that I can share the account ID here in public) and it is valid.Further, the content of email says, "Our next approval run is scheduled for March 18. If your new application containing the 12 digits is received by March 17, 12 PM Pacific Time you will have early access by March 20. If you are not able to re-apply by then with your freshly located 12 digit AWS ID, new approval runs are scheduled every Monday for the next few weeks."It says March 18th??Could you please assist on this?FollowComment"

Answer: 0Resolved by contacting AWS support.CommentShareSureshkumaranswered 4 years ago


Question: Unable to create and save Jupyter Notebook in AWS Glue

"I'm attempting to write some ETL jobs in a jupyter notebook, however whenever I leave the page the notebook isn't listed on my written jobs. When I attempt to save the notebook manually I get this error"Failed to update job[s3.eu-west-2.amazonaws.com] putObject: NetworkingError: Network Failure"This happens for the spark python shell as well, all other job types seem to save fine.FollowComment"

Answer: 0Do you refresh your browser when you go back to the Glue jobs list?CommentSharevtjeananswered 2 months ago


Question: "Can an AWS EventBridge Rule target a Kinesis Data Stream (with a custom event, not AWS event)?"

"In EventBridge I have an Event Bus with 2 Rules. In Kinesis I have a Data Stream.One of the Rules has as a Target a Log Group in CloudWatch. The second Rule has as a Targe my Data Stream in Kinesis. This Rule has a Role with a Policy to execute all needed actions on my Data Stream.When I go to my Event Bus and test it using "Send events" button, I see the event in the Log Group but nothing in my Data Stream.In "Event source" I enter: com.mycompany.poc.eventbusEventBridge -> Rule -> Kinesis StreamsI don't find any other activity in Cloud Trail nor Cloud Watch. My Event Pattern is like this:{  "source": ["com.mycompany.poc.eventbus"]}Does someone have any clue?FollowComment"

Answer: "1Accepted AnswerEvents in EventBridge can only accept the following fields: Time, Source, Resources, DetailType, Detail, EventBusName, TraceHeader. If you need to pass additional information, such as a partition key, you need to include it inside the Detail field.If you need to include a Partition key when sending to Kinesis, you will need to use an Input Transformer to create the appropriate structure, and use the key from the Detail fiel.CommentShareEXPERTUrianswered 2 months ago0Thanks Uri,another option that I found is to include PartitionKey inside Detail,then in Target Rule add:\"InputPath\":\"$.detail\" , \"KinesisParameters\":{\"PartitionKeyPath\":\"$.detail.PartitionKey\"}CommentSharePablo DCanswered 2 months agoUri EXPERTa month agoThis is exactly what I means by using the Input Transformer.Share0It looks like the Kinesis Data Stream needs to receive PartitionKeybut I can't include PartitionKey in the EventBridge Event.I'm trying this:aws events put-events --entries '[{"Source": "my.source", "DetailType": "my.detailtype", "Detail": "{\"key\": \"value\"}", "PartitionKey": "my_partition_key", "EventBusName": "myEventBus"}]'But I get this error:Unknown parameter in Entries[0]: "PartitionKey", must be one of: Time, Source, Resources, DetailType, Detail, EventBusName, TraceHeaderAny clue?CommentSharePablo DCanswered 2 months ago"


Question: I want to send a notification to discord when amplify's app is deployed.

"It seems that only email notifications are possible.In the case of slack, I understand that you can receive notifications via your email address.But discord has no way.I want to be notified via discord.FollowComment"

Answer: "0Hi, Good questionI think it is possible to use Event Bridge/CloudWatch events with Lambda and WebhooksHere is an examplehttps://towardsdatascience.com/discord-notification-using-cloudwatch-alarms-sns-and-aws-lambda-71393861699fHere is another example to slack using Webhook (please note it's not same as the one you asked in the question, however it gives you an idea)https://security-hub-workshop.awssecworkshops.com/04-enrichment-and-integration/hope this helps!CommentShareSrianswered a year ago"


Question: Logs only updated on redeploy

"Hi, I have a single container on elastic beanstalk.It's a .net core 2.2 application deployed with a docker container running microsoft/dotnet:2.2-aspnetcore-runtime.I've been trying to setup logging to cloudwatch, and I went intoElastic Beanstalk -> Environments -> my-environment -> Configurationand enabled "log streaming" (rotate logs is disabled)I could now see my log's in cloudwatch, and I can see the log statements from my application in:/aws/elasticbeanstalk/my-environment/var/log/eb-docker/containers/eb-current-app/stdouterr.logThe problem is, that they are never updated. The only time more data is put into the logs, is if I redeploy the application (even after 12H, there's still nothing new in the logs).I've tried adding a file:.ebextensions/logs.config with the content:option_settings:namespace: aws:elasticbeanstalk:cloudwatch:logsoption_name: StreamLogsvalue: truenamespace: aws:elasticbeanstalk:cloudwatch:logsoption_name: DeleteOnTerminatevalue: falsenamespace: aws:elasticbeanstalk:cloudwatch:logsoption_name: RetentionInDaysvalue: 7But it's the same. The content in the logs is only updated when I redeploy the application.What am I missing? Shouldn't the logs be updated every 5 minutes or so? There isn't a lot of traffic on the site. I've been reading about file_fingerprint_lines and setting awslogs as driver on docker, but can't seem to get any of that to work.I don't have much experience with EBS or Cloudwatch, so any help is highly appreciated.ThanksFollowComment"

Answer: "1I got this solved, by having this in my .config file:option_settings:    - namespace: aws:elasticbeanstalk:cloudwatch:logs      option_name: StreamLogs      value: true    - namespace: aws:elasticbeanstalk:cloudwatch:logs      option_name: DeleteOnTerminate      value: false    - namespace: aws:elasticbeanstalk:cloudwatch:logs      option_name: RetentionInDays      value: 14    files:    "/etc/awslogs/config/stdout.conf":      mode: "000755"      owner: root      group: root      content: |        \[/var/log/containers/docker-stdout]        log_group_name=/aws/elasticbeanstalk/`{ "Ref" : "AWSEBEnvironmentName" }`/docker-stdout.log        log_stream_name={instance_id}        file=/var/log/eb-docker/containers/eb-current-app/*-stdouterr.log  The answer itself is actually not as interesting, as how I got it debugged (posted here in case it will help others):SSH into the instance by using eb sshFound the log: /var/log/awslogs.log and saw the error: cwlogs.push.stream - WARNING - 6563 - Thread-1 - No file is found with given path '/var/log/containers/*-stdouterr.log'I followed the path (/var/log/containers) only to find that the folder structure did not exist. From the logs that was send to cloudwatch, I could see the path was: /var/log/eb-docker/containers/eb-current-app/ which I checked, and found the log files.From some of the other questions that I've been reading, I could see that the configuration for awslogs (which is the agent sending output to aws) was located in: /etc/awslogs/config/stdout.conf. Looking in this file, I saw the path /var/log/containers/*-stdouterr.logI changed the path to /var/log/eb-docker/containers/eb-current-app/*-stdouterr.log and restarted the awslogs service sudo service awslogs restartNow I could see the log being updated.Why it did send logs to cloudwatch on deploy, I don't know, but the above fixed the issueCommentShareMyMooseHenryanswered 3 years ago"


Question: Cognito can not create Lambda trigger

"I would like to add Pre-SignUp Lambda trigger to Cognito. However, if you create a lambda function and try to add it as a trigger in Cognito, you will get an error in either category.The contents of the error are as follows.code: InvalidSmsRoleTrustRelationshipExceptionmessage: Role does not have a trust relationship allowing Cognito to assume the roleIt's well explained, but I'm not familiar with AWS, so it's hard to solve it. Where do we need to modify and edit?FollowComment"

Answer: "0I don't know why, but accurate.Normally after completing the trigger settings in the messaging in the console cognito sms saengseongdoe.CommentShareSangHyunanswered 7 months ago0yes, you can add pre-Signup Lambda trigger in cognito.When you create a Lambda trigger outside of the Amazon Cognito console, you must add permissions to the Lambda function. When you add permissions, Amazon Cognito can invoke the function on behalf of your user pool. You can add permissions from the Lambda Console.some useful links :-https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools-working-with-aws-lambda-triggers.htmlhttps://docs.aws.amazon.com/cognito/latest/developerguide/user-pool-lambda-pre-sign-up.htmlCommentShareutkarsh-awsanswered 7 months ago"


Question: How can we accurately define "better" for recommender metrics with CW Evidently?

"I'm exploring using the new CloudWatch Evidently feature for measuring success of recommendation model deployments with Amazon Personalize.In this context, assigning a user or session to a particular feature variation (baseline recommendations list vs Personalize campaign 1 vs Personalize campaign 2)  might trigger one or more valuable "events":Maybe a click/view for an individual item (What's this item's price? How far down the recommendation list was it?)Maybe a checkout for a basket of products (with an overall total price or total margin)If I understand right (?), Evidently experiment dashboards for "statistical significance" and "improvement" today look just at the distribution of values of recorded events in terms of averages and distribution, right? The number of data points is used for assessing "how significant" but not "what's better"?If so, this seems like a challenge for "optional" events: For example what if one treatment gives me really high basket value on average (only recommending expensive products), but very few users convert? I could see really high metrics for the new treatment, even though its overall value was very poor.Do I understand correctly here? And if so, how might you recommend defining Evidently metrics for these kinds of use cases?For example maybe we'd need to find a way of generating zero-value metric events when a session is abandoned?FollowComment"

Answer: "0Hi, the question of what is "best" will depend on your KPIs. Amazon CloudWatch Evidently, can record the events and assign them to the testing groups, but it is for your business to decide what metric works best.You can sent the metrics to Amazon CloudWatch Evidently at any point in the user journey that matches your use case. For instance, if you don't want to record metrics for abandoned sessions, only send the metrics to Amazon CloudWatch Evidently when the customer "checks out"/"purchases". These can be the metrics for all previous events in the session.With Amazon CloudWatch Evidently you can decide what metric to track, because you can decide what to send to the service. You can control what you send to match your use-case.Documentation: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Evidently.htmlCommentShareAnna_Ganswered 7 months ago"


Question: AWS Glue:  Crawler worked with connection but using it in Glue Jobs result to "while calling o145.pyWriteDynamicFrame. The connection attempt failed."

"When I created a crawler to crawl an RDS (Postgres), it was able to connect and crawl one table I specified.  When I created a job, using the node type "AWS Glue Data Catalog table with PostgreSQL as the data target" and pointing to the database and table, it won't connect to the target.  It is giving me the "An error occurred while calling o145.pyWriteDynamicFrame. The connection attempt failed."  I've checked the security group and subnet of the RDS and the connection in Glue.  What else should I be checking?FollowComment"

Answer: "0Hi,I see that you are receiving the following error while trying to connect to RDS as target using glue job."An error occurred while calling o145.pyWriteDynamicFrame. The connection attempt failed."Common causes for this error can be caused by the subnet the Job is running not being able to connect to the RDS instance or the Security Group of the RDS instance not allowing access to the Security Group being used by the Glue Job. As a first reference, we have a step-by-step guide on how to set up the environment for access the RDS data stores, which covers the configurations that are needed here - https://docs.aws.amazon.com/glue/latest/dg/setup-vpc-for-glue-access.htmlAs you mentioned your glue crawler was working with RDS using same glue connection then I would also request you to check your data source with the glue job if you have used another connection for reading the data catalog table. Then kindly note that, AWS Glue supports one connection per job. If you specify more than one connection in a job, AWS Glue uses the first connection only. If your job requires access to more than one virtual private cloud (VPC) you have to create a dedicated VPC for Glue connection and then configure a peering connection with the VPCs where you have your data source. Please check below link for more details on same.https://aws.amazon.com/premiumsupport/knowledge-center/connection-timeout-glue-redshift-rds/https://aws.amazon.com/blogs/big-data/connecting-to-and-running-etl-jobs-across-multiple-vpcs-using-a-dedicated-aws-glue-vpc/If the issue still persist, then please open a support case with AWS providing the connection details and code snippet used - https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-caseThank you.CommentShareSUPPORT ENGINEERNirmal_Canswered 4 months ago0Following up just in case people are wondering about this:  What I actually did was started with a new canvas, filled in the target first (and went backwards) and my source last.  This fixed the issue and the run was successful.  Very quirky.  Don't know if AWS knows about this issue.CommentSharerePost-User-1795545answered 4 months ago"


Question: An error occurred while setting up your landing zone

"Hello!We are evaluating AWS Control Tower and trying to set up a landing zone. It looks like it has set up the log archive and audit accounts when I look under AWS Organizations, and also set up AWS SSO, but errored out with "AWS Control Tower failed to set up your landing zone completely: An error occurred while setting up your landing zone. Try again later. If this persists, contact AWS Support." When I click "Retry", I see the following error appear: "AWS Control Tower was unable to find your previous shared accounts, so we will need to create new shared accounts. You will need to enter new email addresses for those accounts." Filling in the emails and clicking "Set up landing zone" results in the first error again, after about 2 minutes.I also looked in the browser console when I hit "Retry" and this 400 error shows up on the endpoint "https://console.aws.amazon.com/controltower/api/controltower":{"__type":"ResourceNotFoundException","Message":"No core service resource of type LOGGING found."}This is a brand new AWS account just created a few hours ago, with no resources in it, and no organization associated at first. It seems now that I have attempted to provision a Landing Zone, my account is a member of an organization, and Control Tower is stuck in this retry loop. Any thoughts? Thank you!FollowComment"

Answer: "0I have very similar error. A brand new account and I tired to use ControlTower after few hours of account creation. I can see it created the two OUs and new AWS accounts for Log and Audits as well as SSO but now ControlTower screen is complaining that it cannot use an account which is already member of AWS Org or have SSO Setup which is pretty annoying.CommentSharenzuser105answered 4 years ago0Few things to check:Check https://console.aws.amazon.com/servicequotas/ to ensure you have not exceeded any quotasCheck CloudWatch for errorsCheck CloudFormation StackSets to see errors if it got that farEnsure the email addresses you are using for the Logging & Audit accounts use the same domain as your master accoun, are less than 65 characters long, and are not associated with any other AWS accountsEnsure your payment method is valid and you do not have a past due invoiceFollow this guide to make sure you have your account cleaned up and retry the creation: https://docs.aws.amazon.com/controltower/latest/userguide/walkthrough-delete.htmlCommentSharejstarcheranswered 4 years ago0Looks like the error persisted, as the message warned it might. Did you contact AWS Support? I'd recommend doing so if you continue to encounter this issue.CommentShareDan - AWSDocsanswered 4 years ago0Hi @Dan@AWS, I did not contact AWS support. We do not have a support subscription on this account, since it is for evaluation purposes. Our primary account, which is separate, has Business support. I think I'm going to end up paying for Developer support on this account to open the ticket.It has persisted even after cleaning everything it created up and deleting the sub-accounts.CommentShareSightSoundanswered 4 years ago0After deleting all resources, waiting a day, and trying again, it seems to be working now. There must've been some kind of delay in resource/account/organization deletion that required the wait. Thanks!CommentShareSightSoundanswered 4 years ago0Glad to hear your issue cleared up!CommentShareDan - AWSDocsanswered 4 years ago"


Question: How to solve the ECS Error: You've reached the limit on the number of tasks you can run concurrently.

"When I try to place the 3rd service (I already have 2 services in running) into my ECS service of a cluster, I got this error event:(service xxx) was unable to place a task. Reason: You've reached the limit on the number of tasks you can run concurrently. For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.I have tried to use a bigger "Max running tasks"(600%) when deploying the service, but it didn't work.I have googled, but still didn't figure out how to solve it.What should I do? Thank you!FollowComment"

Answer: "2While this is not documented clearly, check out Service Quotas for both Fargate and ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-quotas.html.Another solution that others have said worked (to activate a new region) is to launch an EC2 instance in the region to "activate" the region.From previous questions, those 2 have typically fixed the issue since it seems that unactivated/new accounts may come with a limit of 2 tasks that can be run concurrently.CommentSharejsoncanswered a year agoEXPERTmreferrereviewed a year agoBorislav Kirilov a year agoI tried to start EC2 instance in the region, chose medium size instance, left it running for almost one hour, but no change - I still get the limit of maximum 2 running tasks. While in the service quotas my Fargate On-Demand resource count is set to 100, which is far more than 2 :( Any other ideas?Share"


Question: Copy AWS Deepracer model to s3

I am trying to copy Deepracer model to s3. It fails with error message "A DeepRacer access role could not be created. Make sure you have the appropriate permissions and try again" I am executing as root user. Also tried with another user with Administrator and other Deepracer access policies granted. No success. Please help.FollowCommentdccn-aws 7 months agoIs your account a member of an AWS organization? It's possible that there may be a Service Control Policy (SCP) blocking your account's access to DeepRacer or to create an IAM role.Share

Answer: 0There has been a bug related to S3 exports around 3 months ago. Is this still occurring?CommentShareTomasz Ptakanswered 4 months ago


Question: API Gateway to Oracle database in private cloud

Can API in AWS cloud interact with On-premises Oracle database?FollowComment

Answer: "1Do you have a DirectConnect or IPSec VPN connectivity from your on-prem data center to an AWS VPC?Also is there an application that is running in your on-prem that interacts with the Oracle database in your on-prem data center and exposes APIs.If the answer to both is yes, then you can expose the APIs over AWS API Gateway.You will need to use API Gateway Private Integration, for which you will need to create a Network Load Balancer and a VPCLink in your AWS VPC. There is a good tutorial that demonstrates how private integration can be configured, although the example does not specifically talk about connecting to on-prem resources - https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-with-private-integration.htmlYou can have your on-prem resources as targets for an AWS Network Load Balancer. There are some restrictions in terms of permitted IP ranges as target for the AWS Network Load Balancer and the details are available here - https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html. Read up the Target Type section.The reason I asked the second question about an application is because AWS API Gateway cannot directly connect to an Oracle database, without an application in between. If you don't have an application, you can use AWS Lambda functions or create your own microservices in AWS ECS or EKS or even on EC2, that can connect to your on-prem database, although you have to run tests to make sure the latency is acceptable to your API consumers.CommentShareEXPERTIndranil Banerjee AWSanswered 8 months ago"


Question: Batch and Spot Interruptions

"Customer is asking if their Batch job that execute on spot is interrupted due to spot price increasing beyond their threshold, will Batch resubmit the job when the spot price falls again, or will the customer have to manually resubmit?FollowComment"

Answer: "0Accepted AnswerAWS Batch has a mechanism for job retry. Documentations here: https://docs.aws.amazon.com/batch/latest/userguide/job_retries.htmlSome quote from the doc:You can apply a retry strategy to your jobs and job definitions that allows your jobs to be automatically retried if they fail. Possible failure scenarios include:Any non-zero exit code from a container jobAmazon EC2 instance failure or terminationInternal AWS service error or outageWhen a job is submitted to a job queue and placed into the RUNNING state, that is considered an attempt. By default, each job is given one attempt to move to either the SUCCEEDED or FAILED job state. However, both the job definition and the job submission workflows allow you to specify a retry strategy with between 1 and 10 attempts. For more information, see Retry Strategy.CommentShareAWS-User-4709969answered 5 years ago"


Question: Reserved instance - Payment pending

"Hi there, I just bought a reserved instance, and the payment deducted 1 hour ago. I can see the receipt in the billing dashboard but the state of the reserved instance is still "payment pending".Could anyone help me to deal with that?My account number is 980414319295 and Reserved Instance Id: a7ba0ad8-609b-4745-b6a8-dc9455df8abeThanks!FollowComment"

Answer: 0it's now activeCommentShareXinchenanswered 2 years ago


Question: Lambda container and oci image manifest type

"Hello,I am trying to use podman to build lambda container.When manifest type is application/vnd.oci.image.manifest.v1+json I get the following error:"The image manifest or layer media type for the source image <snip> is not supported."When I set manifest type to application/vnd.docker.distribution.manifest.v2+json it works.In lambda faq (https://aws.amazon.com/lambda/faqs/) I read:"""You can use any container tooling as long as it supports one of the following container image manifest formats: Docker Image Manifest V2 Schema 2 (used with Docker version 1.10 and newer) or Open Container Initiative (OCI) Specifications (v1.0 and up). For example, you can use native container tools (i.e. docker run, docker compose, Buildah and Packer) to define your functions as a container image and deploy to Lambda."""So it should be supported.I do not see what is wrong as all seems right.Diff of inspect if I remove GraphDriver and expected timestamp and hash modifications... ends up with:-        "ManifestType": "application/vnd.docker.distribution.manifest.v2+json",+        "ManifestType": "application/vnd.oci.image.manifest.v1+json",Does anyone have a clue?Reproduce with this simple Containerfile:from public.ecr.aws/lambda/python:3.9run touch a# works:podman --format docker build .# Does not work:podman --format oci build .FollowComment"

Answer: 0I've tested with your Containerfile and reproduced the issue. The problems come with the multi arch manifest it automatically generates which apparently is not supportedSpecify a single arch to fix the issue:podman build . --arch=amd64 --os=linuxCommentShareabanswered a month ago


Question: HOW TO SECURE(SSL/TLS) RDP ACCESS with AWS ACM Private certificate (CA)

"Hi Team,Good Day!!I have some queries about ACM private certificate.Here's  my queries:1.How to deploy a AWS (ACM) private certificate to RDP for Windows server OS and Windows machine OS?2.While taking RDP "The identity of the remote computer cannot be verified" it shows, So how to install AWS  ACM private certificate for it? 3.How do I install an SSL/TLS certificate for RDP connection using AWS ACM with private certificate?Regards,DivyaFollowComment"

Answer: "0Accepted AnswerCertificates created within ACM cannot be saved or exported for the purpose of installing within the Windows OS (or any other OS). They can be used with Elastic Load Balancers, CloudFront, and other services listed here https://docs.aws.amazon.com/acm/latest/userguide/acm-services.htmlCommentShareRWCanswered a month agorePost-User-9323488 a month agoWe have created Root CA and Subordinate CA in ACM AWS Private Certificate Authority After the status is Active , We can able to export the certificate body and chain from Root and Subordinate CA. Now, I want to install the certificate for RDP connection which is running on EC2 windows OS While taking RDP "The identity of the remote computer cannot be verified" it shows, So how to install AWS ACM private certificate for it? What can i do further to proceed?Share0We have created Root CA and Subordinate CA in ACM AWS Private Certificate AuthorityAfter the status is Active , We can able to export the certificate body and chain from Root and Subordinate CA.Now, I want to install the certificate for RDP connection which is running on EC2 windows OSWhile taking RDP "The identity of the remote computer cannot be verified" it shows, So how to install AWS  ACM private certificate for it?What can i do further to proceed?CommentSharerePost-User-9323488answered a month ago0https://docs.aws.amazon.com/privateca/latest/userguide/Create-CA-console.htmlhttps://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-private.html#request-private-consoleThis step for Machine A where RDP service is running:https://serverfault.com/questions/1044614/windows-10-pro-as-rdp-host-with-ssl-certificate-howThis step for Machine B where you use RDP client to access Machine A:Certificates signed by a private CA are not trusted by default, and ACM does not support any form of validation for them. Consequently, an administrator must take action to install them in your organizations's client trust stores.CommentSharerePost-User-6621528answered a month ago"


Question: AWS IOT v2 SDK migration from v1 sdk

"We need to know why we are migrating from v1 sdk to v2.?What are the major issues in v1 sdk and how is it impacting our clients.?What are major advantages of IOT v2 sdk. and how many clients it will benefit.?what should be the rollout plan for v2 sdk changes. Can we cross v2 API with some v1 API and then test it.FollowCommentDavid Malone EXPERT10 months agoWhich SDK are you using? Regarding your questions for rollouts or cross testing with v2 and v1, both v2 and v1 continue to use the standard AWS IoT MQTT, Websocket, and HTTP endpoints and commands; i.e. pub/sub to MQTT topics, or Shadow endpoints remain the same regardless of which version you are using.As stated in the README for the Javascript SDK (https://github.com/aws/aws-iot-device-sdk-js-v2):This SDK is built on the AWS Common Runtime, a collection of libraries (aws-c-common, aws-c-io, aws-c-mqtt, aws-c-http, aws-c-cal ...) written in C to be cross-platform, high-performance, secure, and reliable. The libraries are bound to JS by the awscrt package.Each of the V2 IoT SDKs have been developed to use the AWS Common Runtime with the same goals in mind.SharerePost-User-8590118 10 months agoThanks DavidWe are already using using IOT java v1 SDK. while doing load testing we encountered AWSIotException: too many publishes. One possible reason could be maxInflightMessage count which is 10 by default and can't be changed in v1 SDK. So can it be resolved in v2 sdk.Also we get disconnect after every 24hrs from Mqtt Client and we need to refresh credentials for that. Is the new version using same methods?Share"

Answer: "0Hi. Per the READMEs (and as David just mentioned in his comment), the V2 SDKs are:built on the AWS Common Runtime, a collection of libraries written in C to be cross-platform, high-performance, secure, and reliable.The CPP, Python, Java and JS V2 SDKs all use the awscrt and all benefit from the reuse of common code. Since the V2 SDKs have been around for a long time already, and the V1 SDKs only get security updates, the V1 SDKs lack many of the more recent AWS IoT features compared to V2. For example (depending on the language), named shadows, Greengrass IPC, PCKS11, fleet provisioning, device defender and more.Additionally, all of the V2 SDKs are validated against AWS IoT Core Device Advisor.CommentShareEXPERTGreg_Banswered 10 months ago"


Question: Synthesis & CSim hangs on Vitis HLS GUI

"Hi. This is a follow-up question to: https://forums.aws.amazon.com/message.jspa?messageID=990108#990108I am using Vitis HLS GUI on AWS M5.xlarge instance with FPGA Developer AMI 1.10.0I am currently in Los Angeles and logging into N. California AWS server, so latency shouldn't be too bad.I create a simple project with source files which I know synthesized correctly in my own servers with Vitis HLS 2020.2.But when I pressed "Synthesis" (the green arrow button) or when I pressed CSim (in the pull-down menu),Vitis HLS simply hangs and does not respond.No error message - simply hangs. The program itself is not dead - it does accept other commands, such as switching to Debug tab.Could anyone help me solve this problem?FollowComment"

Answer: "0Hi Young,I'll check with Xilinx on this. Are there any logs that you can share?-DeepCommentShareDeep_Panswered 2 years ago0Thanks. Actually, I decided to use NICE DCV as you suggested, and that flow does not have this problem. Thanks for clarifying that it is free of use :)CommentShareYoungChoianswered 2 years ago"


Question: .es registration semi-broken

"There seems to be a bug in your validation code for .es domain registration.The domain contact's ES_LEGAL_FORM is set to PUBLIC_LIMITED_COMPANY, and the console forces me to specify a citizen identification number in ES_IDENTIFICATION. This is incorrect; ES_IDENTIFICATION must be a company number according to the documentation[1].Entering a citizen identification number causes Gandi to reject the registration, as expected:2400 : Command failed (541 Invalid attribute value; INVALID CONTACT \[OWNERCONTACT (Invalid attribute value syntax; The identification is not valid)])The console should hide the ES_IDENTIFICATION_TYPE when ES_LEGAL_FORM is not INDIVIDUAL, and should apply CIF validation to the ES_IDENTIFICATION field.[1] https://docs.aws.amazon.com/Route53/latest/APIReference/API_domains_ExtraParam.htmlEdited by: mildsunrise on Apr 9, 2021 10:28 AMEdited by: mildsunrise on Apr 9, 2021 10:28 AMFollowComment"

Answer: "0Okay, after poking a bit more I've figured it out. The API uses "Contact type" instead of "ES Legal Type" to decide if it has to validate a CIF or a citizen number, and I was incorrectly setting Person as the Contact type. It must be Company for the Registrant, but Person for the other two.So, in summary. You must:Uncheck "My Registrant, Administrative and Technical Contacts are all the same"Set registrant to "Company" and the others to "Person"In "ES ID Number", enter (1) the CIF for registrant and (2) citizen ID for the others.Disable privacy guard on all contacts.CommentSharemildsunriseanswered 2 years ago"


Question: How is processor type determined for T3 instances?

Hi.  We're running four T3 instances in two Availability Zones in the Asia Pacific (Sydney) region.The AWS docmentation on instance types (https://aws.amazon.com/ec2/instance-types/) states that the CPUs for T3 instances will be "Up to 3.1 GHz Intel Xeon Scalable processor (Skylake 8175M or Cascade Lake 8259CL)".Our t3.small and two t3.micro instances have the Cascade Lake 8259CL processor.  Our t3.2xlarge instance - which does most of our work - has a Skylake 8175M processor.From my simple benchmark tests it looks like the Cascade Lake 8259CL processor is something like 2-3 times faster than the Skylake 8175M processor.  I'd dearly love the T3.2xlarge instance to acquire the Cascade Lake 8259CL processor; the performance benefit for our application would be terrific!What determines the selection of processor type for T3 instances?  How can I endeavour to have our t3.2xlarge instance to procure the faster processor?  Thanks for any help!FollowComment

Answer: "0Accepted AnswerYou can't control the selection of CPU type - as the documentation you've linked to states; there are multiple CPU types in use for the instance type. You could launch an instance; check the CPU type; and keep doing that until you get one that you prefer but it's not a great solution.Otherwise, you would need to choose an instance type where a single CPU type is used for that type.CommentShareEXPERTBrettski-AWSanswered 3 months ago"


Question: RDS connections limit

"Hello,I have a RDS Postgres used by a lambda. This lambda is used on a front end experience.Recenlty I reached the limit of connections on the rds.My concern is that the logs where showing this messages like a 100 times:2023-02-27 11:00:22 UTC:3.0.139.252(47570):@:[23428]:FATAL: password authentication failed for user ""2023-02-27 11:00:22 UTC:3.0.139.252(47570):@*******:[23428]:DETAIL: Role "" does not exist.Connection matched pg_hba.conf line 13: "host all all all md5"2023-02-27 11:00:45 UTC:3.0.139.252(52788):@:[23453]:FATAL: password authentication failed for user ""2023-02-27 11:00:45 UTC:3.0.139.252(52788):*******@:[23453]:DETAIL: Role "" does not exist.Connection matched pg_hba.conf line 13: "host all all all md5"2023-02-27 11:01:26 UTC:3.0.139.252(58080):@:[23498]:FATAL: password authentication failed for user ""2023-02-27 11:01:26 UTC:3.0.139.252(58080):@:[23498]:DETAIL: Role "" does not exist.Connection matched pg_hba.conf line 13: "host all all all md5"2023-02-27 11:01:33 UTC:3.0.139.252(39560):@**:[23539]:FATAL: password authentication failed for user ""2023-02-27 11:01:33 UTC:3.0.139.252(39560):@*****:[23539]:DETAIL: Role "******" does not exist.Connection matched pg_hba.conf line 13: "host all all all md5"This is easier a log issue because the connection limit was reached or either someone trying to bruteforce the database.Can you confirm that this is link to the connection limit being reached ?Also now when I look at my connections limit I see pikes like bellow:But when I look my trafic analytics I don't have huge pikes of views at the same time.What can this be related to ? What exatcly count as a connection ?Thank you in advance,FollowComment"

Answer: "1I can't speak to what your exact issue is.  Every time your Lambda function runs, you may be establishing a new database connection.   Since Lambda reuses execution environments, it is recommended that the DB connection be established outside of the init code and left open.  Additionally It could be a security issue, depending on your configuration. Is your DB Instance located in a private subnet?  Do you have Security Group rules that only allow DB traffic from the Lambda functions?Database connection management in a serverless environment is a common challenge, there is a tool to help you manage it.RDS Proxy for PostgreSQL.This can be setup for most apps without code changes. The idea is to fix connection management by connection pooling and multiplexing. You can also use RDS Proxy can also help add  another layer of  security and enforce IAM authentication for databases as well as securely store credentials in AWS Secrets Manager rather than in code.RDS proxy info https://aws.amazon.com/rds/proxy/A Lab using RDS Proxy for PostgreSQL https://catalog.us-east-1.prod.workshops.aws/workshops/2a5fc82d-2b5f-4105-83c2-91a1b4d7abfe/en-US/3-intermediate/rds-proxySecurity Group settings for Lambda connections: https://aws.amazon.com/premiumsupport/knowledge-center/connect-lambda-to-an-rds-instance/Also see this post on RDS and Lambda -https://repost.aws/questions/QUdJCGX5f1SD-OESCqqOCT_w/rds-connections-in-lambdaCommentSharetedtrentanswered 3 months ago0Thank you for all the details we have definlty not done the right things the right way we are going to change that.By the way we find our bug we did not added a "client.end()" at the end of our lambda that's why db connection keep added up.In the mean time the main issue is fixed. But we are going to change our architecture to be more secure and efficient thanks to your help.Best,CommentSharerePost-User-1044214answered 3 months ago"


Question: Facing server error in Lambda(Node.js) after MSSQL package update

"On updating the MSSQL package from Version6.2.0 to Version9.1.0 in package.json file , we are facing the below error .Failed to connect to Server1_name  - Hostname/IP does not match certificate's altnames: Host: Server1_name is not in the cert's altnames: DNS:Server2_nameThe lambda is written in Node.js, lambda connects to mssql server in npm mssql version 6.2.0 but throws the above error in npm mssql version 9.1.1We upgraded the version for resolving the security vulnerability reported for dependency package (xmldom).Please guide what can be the ways to resolve the error and the root cause.FollowCommentKirk_D 3 months agoIt would be helpful if you gave the exact library that you're referring to.  For .NET code in Lambda, typically people are using either the older System.Data.SqlClient (max version 4.8.5) or the newer and recommended package Microsoft.Data.SqlClient (max version 5.1.0).  There's an entity framework core package Microsoft.EntityFrameworkCore.SqlServer (max version 7.0.3) and the older Entity Framework 6 package EntityFramework (max version 6.4.4).Since none of these are available in version 9.1.0, it's not clear exactly what library you're using. Are you possibly using the magic.lambda.mssql library? Or something else?ShareChaitra Anjanappa 3 months agoHi , The lambda is written in Node.js and we have package.json file were the packages required and its dependencies mentioned . Based on a security vulnerability reported for dependency package(xmldom) we had to upgrade the MSSQL package in Visual Studio Code  from Version 6.2.0 to Version 9.1.1 which is leading to the above error . We dont have the sql server connection error in lambda when the MSSQL version is 6.2.0. The mentioned versions are Npm MSSQL versionsShare"

Answer: "2According to error, there is a certificate mismatch between the server name and the alternative names listed in the certificate.To resolve this issue, you can try the following steps:Check the certificate to make sure it is valid and has the correct alternative names listed.Ensure that the server name is correct and matches the name listed in the certificate.If the certificate is not valid or does not have the correct alternative names, update or replace the certificate.If the server name has changed, update any references to the old name with the new name.Check the network configuration to ensure that there are no issues with DNS resolution or routing that could be causing the error.CommentShareTasadduq Burneyanswered 3 months ago"


Question: AWS Data Migration

"Hello,geting error while migrating data from aws open search to elastic search.Error:body={"error":{"root_cause":[{"type":"illegal_argument_exception","reason": "No search type for status":400}]Can any one help in solving the above issue.FollowComment"

Answer: "0Are you following the steps provided by Elastic to do this?https://www.elastic.co/guide/en/cloud/current/ec-migrate-from-aws.htmlLooks like you don't need to use AWS DMS (if you are trying to use that), but the steps walk you through exporting your data to S3 then having Elastic ingest the data from there.Hope this helps!CommentShareCarlos del Castilloanswered a month ago"


Question: AWS Network Firewall to protect against DDoS layer 7 attacks

"Hello,We have a public API (many of them) and I am looking into how we can protect against large scale layer 7 DDoS type attacks, for example a HTTP GET. The proposed architecture that we have come up with involves asynchronously detecting bad IP addresses based on signals (web server logs), and blocking by IP at network level (osi 3). We want to block at network level as it’s likely to be much more performant and cost effective than attempting to do it at application/compute level.We already have AWS Shield Advanced.We have identified the AWS Network Firewall as a possible place to enforce the blocking of these bad IPs. However, I have a question about the scale in which the AWS Network Firewall can block bad IP addresses. In the FAQ (https://aws.amazon.com/network-firewall/faqs/) it says:Q: Can I use AWS Network Firewall for protection against DDoS attacks?AWS Network Firewall is designed to protect and control access to and from your VPC, but not to mitigate volumetric attacks, like distributed denial of service (DDoS), that can impact the availability of your application. To protect against DDoS attacks and ensure application availability, we recommend customers review and adhere to our AWS Best Practices for DDoS Resiliency, and also explore AWS Shield Advanced, which offers managed DDoS protection customized to your specific application traffic.My question is this: is this recommendation based on the AWS Network Firewalls inability to apply rules (I.e. block by IP address’) at a large scale, or is this specifically the volumetric attacks.To clear some things up:We have AWS shield advanced - so I expect that we have most volumetric type attacks coveredThe DDoS attacks we are trying to protect against are mostly layer 7 targeted (e.g. HTTP GET index.html)Ability to detect bad IP addresses based on signals is out of scope, assume we have a 3rd party OTT product that can detect bad IP’s from our web server logs and apply custom rules managed by usWe cannot use cloudflare/fastly etcScale of layer 7 DDoS that we are likely to want to protect against: ~10k source ip addresses each doing around 1k HTTP requests per second, (total  ~10M http requests p/s) - typically any ip doing over 1k requests per second would be a strong signal and we would likely want to block that IP for some time. Sorry, I don’t know how to translate the volume of http layer 7 requests into volume of layer 3 packets I can only comprehend it in http requests per second.FollowComment"

Answer: "4its primary purpose is not to mitigate large-scale DDoS attacks, especially volumetric attacks that can impact the availability of your application. AWS Network Firewall is designed to provide fine-grained control over network traffic, and it can be used to block specific IP addresses.For Layer 7 DDoS protection, you can consider a combination of AWS Shield Advanced and AWS WAF (Web Application Firewall). AWS Shield Advanced provides DDoS protection at the network and transport layers (Layers 3 and 4). AWS WAF, on the other hand, offers Layer 7 protection by allowing you to create custom web access control rules based on various parameters such as IP addresses, HTTP headers, query strings, etc.CommentShareEXPERTsdtslmnanswered a month agorePost-User-2820918 a month agoHello, I appreciate your answer, however I don’t think it answers the question being asked. My question is specifically around the recommendation from the FAQ.Share4Hello.For layer-7 protection for an API the recomended service is Web Application Firewall, WAF protection is improved when integrated with Shield Advanced.For using WAF (L7) with Shield Standard (L3-4) I would suggest you check the WAF Security Automation solution that using WAF and integrating other serverless technologies has bot protection, https-flood prevention and ease use of block/allow ip-lists.  Deployment is straight forward following the deployment guided and using CloudFormationTo read about Shield Advanced (L7 plus other features) protections with WAF you can look at this page.Addingt to my response:There are several layers of protection you can use, Network Firewall is not designed for large DDoS as AWS Shield Advanced is pre-existing service for this purpose and has other important benefits as incident response team.  Customers can take advantage of Shield Advanced even when choosing other firewall or perimeter protection solutions from recognized security partners.CommentShareromerogtanswered a month ago"


Question: Multiple questions about Route 53

I have a customer asking the following questions about Route 53:Does Amazon route 53 offer the following:Full registrar services?UK specific DNS suffixes?Unlimited DNSqueries?2 factor-authentication?Sub-domain delegation?Single signon?Sell ssl certificates?Thanks!FollowComment

Answer: "0Accepted AnswerSome of the things of your list are not DNS related, so Route 53 won't help but you might find something in other AWS services, below my quick comments:Full registrar services?Here is the list of Domains you can use Route 53 as a registrar: http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/registrar-tld-list.htmlUK specific DNS suffixes?Search on the page above.Unlimited DNS queries?There isn't a limit, however you pay per query.  (customer may be asking for an unlimited plan)2 factor-authentication?The access to  the Route 53 console (for DNS administration) is controlled by IAM and therefore you can enable 2 factor authentication. http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa.htmlSub-domain delegation?Yes.Single sign on?Yes, again the access to  the Route 53 console (for DNS administration) is controlled by IAM and thus you can implement SSO: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_saml_3rd-party.htmlSell ssl certificates?Route 53 don't, we have another service for that , it's called AWS Certificate Manager, however we don't sell certificates, we give them for FREE :)  https://aws.amazon.com/certificate-manager/CommentShareLucas_Ranswered 6 years ago"


Question: class CfnFramework how to add scopeTags to controlScope

"Hi all, exactly what the title says.https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_backup.CfnFramework.html#class-cfnframework-constructconst cfnFramework = new backup.CfnFramework(this, 'MyCfnFramework', {  frameworkControls: [{    controlName: 'controlName',    // the properties below are optional    controlInputParameters: [{      parameterName: 'parameterName',      parameterValue: 'parameterValue',    }],    controlScope: controlScope, # how do we add scope tags  }],});I was trying to do something like this but it didn't work because I have got the wrong key there but I have also tried tags instead of scopeTags and that didn't work either.const cfnFramework = new backup.CfnFramework(this, 'MyCfnFramework', {  frameworkControls: [{    controlName: 'controlName',    // the properties below are optional    controlInputParameters: [{      parameterName: 'parameterName',      parameterValue: 'parameterValue',    }],    controlScope: {      scopeTags: # extraneous key [scopeTags] is not permitted    }  }],});Adding errors that I am seeing on screen -`2:41:05 AM | CREATE_FAILED | AWS::Backup::Framework | BackupFrameworkProperties validation failed for resource BackupFramework with message:#/FrameworkControls/0/ControlScope: extraneous key [complianceResourceTypes] is not permitted#/FrameworkControls/0/ControlScope:extraneous key [tags] is not permitted#/FrameworkControls/1/ControlScope: extraneous key [complianceResourceTypes] is not permitted#/FrameworkControls/1/ControlScope: extraneous key [tags] is not permitted0 comments`FollowComment"

Answer: "0You receive this error "extraneous key [complianceResourceTypes] is not permitted" when the properties are not correctly specified or you haven't defined one or more properties for controlScope.The control scope can include one or more resource types, a combination of a tag key and value, or a combination of one resource type and one resource ID. If no scope is specified, evaluations for the rule are triggered when any resource in your recording group changes in configuration.NOTE : To set a control scope that includes all of a particular resource, leave the ControlScope empty or do not pass it when calling CreateFramework .You can try adding scope tags to controlScope by defining it using ControlScopeProperty as shown below :================// The code below shows an example of how to instantiate this type.// The values are placeholders you should change.import { aws_backup as backup } from 'aws-cdk-lib';const controlScopeProperty: backup.CfnFramework.ControlScopeProperty = {complianceResourceIds: ['complianceResourceIds'],complianceResourceTypes: ['complianceResourceTypes'],tags: [{key: 'key',value: 'value',}],};I suggest you test it on a demo environment before using in a production setup.Refer to this documentation for more information :==> https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_backup.CfnFramework.ControlScopeProperty.html==> https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-model-validation-failure/CommentShareSUPPORT ENGINEERSatyam_Danswered 4 months ago"


Question: How can I export and import EC2 Launch Templates?

"How can I export and import EC2 Launch Templates, including version history and launch scripts (advanced)? Exporting to another EC2 account/region is not sufficient, I need a way to get them off AWS and stored locally.Thanks!FollowComment"

Answer: "1Accepted AnswerHi Sensible Saas,There is no out of the box way to export a launch template.What you could do is to exploit the describe template (https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-launch-templates.html) and describe versions (https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-launch-template-versions.html) api/sdk and use a lambda function to extract to a specific JSON structure that fits your use case.Similarly you can then refer engineer your logic and use the create launch template and create launch template versions to recreate the resources.Hope it inspires youCommentShareEXPERTalatechanswered 3 months agoSensible SaaS 3 months agoThanks, was hoping for a trivial way but sounds like it would be easier to just rebuild with Terraform/AnsibleShare1A better way than exporting ones made in the console is leveraging CloudFormation to create the LaunchTemplates, this way you can always have a JSON/YAML format of the Launch Template that you can go back and look at.https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-launchtemplate.htmlCommentShareEXPERTRob_Hanswered 3 months agoSensible SaaS 3 months agoWhy would anyone use CloudFormation over Terraform?Share"


Question: Lightsail instance randomly shut down at exactly 11pm

"Hello!I have a Lightsail instance that I've been running for about a year now without any issues. Last night at exactly 11pm (the timestamp is 4:00:11 UTC) the server randomly shut down. I checked the syslog and it appears to have been shutdown rather than crashed. I checked the system metrics and everything seems normal there (CPU was hovering around 3% usage, no status check failures). I don't have any cronjobs or backups that run at that time, so I'm sort of at a loss currently. Is it possible that AWS ran some sort of scheduled task that accidentally shut the server down?FollowComment"

Answer: "1HelloWarm greetings from AWS!To Investigate the issue, we require details that are non-public information i.e details that are specific to your AWS account. First of all, I would suggest you to check cloudtrail for any stop/reboot events performed on the instance[1]. Please follow the below steps:Open Cloudtrail console from AWS Management ConsoleSelect Event history from the left panel.In the Lookup attributes select Event Source from dropdown and select “lightsail.amazonaws.com” as event source.Set the time frame around the time when you observed the issue.Search for “StopInstance” or “RebootInstance” callIf the shutdown was performed explicitly, you will be able to find the username who stopped/rebooted the instance. Any information regarding the shutdown would be mentioned in the logs.In case the shutdown occurred due to hardware failure, it will not be recorded in the cloudtrail. In such case, the best approach here would be to open a case with AWS Support[2] along with the resource details in order to investigate the issue better.[1] https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events-console.html#viewing-details-for-an-event[2] https://console.aws.amazon.com/support/home#/case/createCommentSharerePost-User-1052802answered 4 months ago"


Question: "Passing Default Arguments in GreenGrass recipe, fails to do so on IoT Edge device"

"Problem passing arguments with non alpha chars, being 'ignored'When I configure a component with an argument, as below"ComponentConfiguration": {"DefaultConfiguration": {"Slot": "1-2-3-4"}},At the end, when the pythonscript is being launched using that argument, say:"Run": "python {artifacts:decompressedPath}/program/myscript.py "{configuration:/Slot}""At the Edge device it is being passed as myscript 1 <------------missing -2-3-4It seems documentation is missing, I need more arguments being passed into 'Slot' but not filtered by some unknown mechanism.How to solve it? Spaces and comma's have been tried already.THANKSFollowComment"

Answer: "1Hi enierop,First, can you check your <greengrass-install-dir>/config/config.tlog? This is a transaction log that contains all the configuration that has been applied.By checking the config.tlog for Slot and your component lifecycle, you can see what Greengrass thinks is the value and what it will run:e.g.#  grep Slot /greengrass/v2/config/config.tlog{"TS":1647492231773,"TP":["services","com.example.HelloWorld","configuration","Slot"],"W":"changed","V":"1-2-3-4"}# grep Run /greengrass/v2/config/config.tlog{"TS":1647492236124,"TP":["services","com.example.HelloWorld","lifecycle","Run"],"W":"changed","V":"python3 /greengrass/v2/packages/artifacts/com.example.HelloWorld/1.1.2/hello_world.py \"1-2-3-4\""}Is it possible that you deployed an earlier version of your component that contained the "DefaultConfiguration" value of "1" for the Slot parameter?Greengrass does not do any interpretation of the data when it replaces the values in the recipe. However, if you had previously set the default configuration to a value, like 1, then that value would be used unless the configuration is reset via a deployment.If you change the DefaultConfiguration value from one version of a recipe to another, and you want that new default to be applied, in the deployment you must also add configuration to reset the config. For instance, in your deployment, if you configure the component with:  "reset": [    ""  ]it will reset all the applied configuration back to the defaults.The following is my attempt to replicate your issue. I have installed Greengrass v2.5.3 and python3.8.10. I created a python3 script hello_world.py which contains the following:import sysmessage = "Hello, %s!" % sys.argv[1]print(message)I created a recipe that looks like:{  "RecipeFormatVersion": "2020-01-25",  "ComponentName": "com.example.HelloWorld",  "ComponentVersion": "1.1.2",  "ComponentType": "aws.greengrass.generic",  "ComponentDescription": "Hello world",  "ComponentPublisher": "Me",  "ComponentConfiguration": {    "DefaultConfiguration": {      "Slot": "1-2-3-4"    }  },  "Manifests": [    {      "Platform": {        "os": "linux"      },      "Name": "Linux",      "Lifecycle": {        "Run": "python3 {artifacts:path}/hello_world.py \"{configuration:/Slot}\""      },      "Artifacts": [        {          "Uri": "s3://<my_bucket_name>/hello_world.py",          "Unarchive": "NONE",          "Permission": {            "Read": "OWNER",            "Execute": "NONE"          }        }      ]    }  ]}It prints out "Hello, 1-2-3-4!" when it runs:2022-03-17T04:43:56.347Z [INFO] (Copier) com.example.HelloWorld: stdout. Hello, 1-2-3-4!. {scriptName=services.com.example.HelloWorld.lifecycle.Run, serviceName=com.example.HelloWorld, currentState=RUNNING}When I change the value of "Slot" in the DefaultConfiguration and update the recipe and component version, using reset: [""]  always makes it print the expected response.CommentShareRobanswered a year ago0Thanks, It once was "1" indeed.In the tlog it shows first as one{"TS":1646767501143,"TP":["services","com.mystuff","configuration","Slot"],"W":"changed","V":"1"}Later it becomes{"TS":1646927815345,"TP":["services","com.mystuff","configuration","Slot"],"W":"timestampUpdated","V":null}Unfortunately, the argument in Python, I used it the same as you, but it stays "1".I will try your suggestions, though I must say, it does not make any sense to me that it does not detect argument changes. I will (need to) be doing this more often.CommentShareenieropanswered a year agoRob a year agoThe configuration interpolation happens when the device receives the deployment. The "default" value is only applied to a component running on a device if that device does not have a configuration value applied to it already.Once a "default" value has been applied - it is stored in the configuration as the value for that configuration key - just as if you set that configuration value in a deployment. When you change the default value in the recipe, that won't change the value on your device as it still has the old value applied and no longer needs to lookup the value.You would need to deploy the new version along with the reset configuration.If the argument needs to change frequently without changing the code artifact - you may be better just deploying the change as a configuration via:merge: { "Slot": "1-2-3-4'}Here's the documentation for merge and reset for more info: https://docs.aws.amazon.com/greengrass/v2/developerguide/update-component-configurations.html#merge-configuration-updateShare"


Question: Unable to Connect to EC2 Instance thru SSH

"The instance was working perfectly fine months ago. I kept it running and did not check on it for a couple of months. Since then, I can have access to it with PuTTy, not even the session manager of AWS.As a matter of fact, I can't even ping the instance IP in cmd.Instance ID: i-0f75db51a7825b76aThanks,AlvinFollowComment"

Answer: "0Hello AlvinCheung,I have checked your instance and I can see that it is healthy with 2/2 successful health checks. Also, the access list and route table are correctly configured.Having said that, I was able to reach the server on port 80 from my end as it is open to the world but not via the port 22 because traffic is allowed only from 1 IP.To make sure that the allowed IP belongs to you please update the security group rule for port 22 and choose as "source", "My IP". Then please try to SSH to the instance again. Additionally, for your reference I would like to share this useful link too. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesConnecting.htmlIn regards the ping failure, please note that there is no ICMP rule configured at the security group to allow ICMP messages. Therefore, if you want to ping the server you will have to create a new rule as per your requirements.Lastly, in order to be able to use the Session manage please make sure to meet the prerequisites as per the document below. By looking at the instance I can see that you did not attach an instance profile.https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started.htmlIf the issue persists please let me know what error message PuTTY returns and remember to hide any public IP addresses.Thank you,VasosAWSCommentShareAWS-User-5546426answered 3 years ago0Thanks! The problem is solved.CommentShareAlvinCheunganswered 3 years ago"


Question: Install virtualization software on EC2

"A customer would like to have a virtual server and then install a virtualization software in it (they are currently working with Proxmox VE), to virtualize Linux, FreeBSD, and Windows machines. Is that possible?FollowComment"

Answer: "0Accepted AnswerNested virtualisation is a very tricky problem, which can bring further issues to the surface; for example, Xen-on-Xen virtualisation works up to a point but opens a security hole, which is why we don't have our base Xen configured to permit it. For the new Nitro system, KVM has a nesting capability, but I don't know (yet) whether our specific KVM implementation allows it (I'll edit this piece when I've found out more).If a customer wants to have an environment where they have control over a virtualisation layer that sits on top of their own environment (for example, so they can instrument the hypervisor in ways that CloudWatch doesn't), then their best option is to get one of the new bare-metal instances, and install an OS and layered hypervisor of their choice on that.CommentShareEXPERTDave_Wanswered 5 years ago"


Question: BYOIP and Direct Connect

"Are "Bring your own IP" addresses advertised over a DX public VIF?  I'm assuming they would be, as the process authorizes Amazon to advertise it outward, but my customer is looking for confirmation.FollowComment"

Answer: "0Accepted AnswerYes, BYOIP ranges bought to be used with EC2 are included in the DX Public VIF advertisements. They are not however listed in the ip-ranges.json file.CommentShareEXPERTSteveS-AWSanswered 3 years agoEXPERTBrettski-AWSreviewed a year ago"


Question: Amplify vs S3: Why is this behaviour different?

"HI All, I'm still very much a novice with AWS and I'm following this tutorial to learn more Build a Serverless Web Application. In my own test account I've followed this tutorial without issue. I'm currently trying to apply the same tutorial in a restricted environment where I don't have access to Amplify. I figured I could get away with swapping Amplify in this tutorial S3, however I'm getting some odd behaviour.Amplify hosted siteS3 hosted siteI'm using this github repo website for both. I'm creating the bucket and uploading the files via TF.resource "aws_s3_bucket" "mrw-example-bucket" {  bucket = "mrw-example"  tags = {    Name = "mrw-example"  }}resource "aws_s3_bucket_acl" "mrw-example-bucket-acl" {  bucket = aws_s3_bucket.mrw-example-bucket.id  acl    = "public-read"}resource "aws_s3_bucket_website_configuration" "mrw-example-bucket-website-config" {  bucket = aws_s3_bucket.mrw-example-bucket.id  index_document {    suffix = "index.html"  }  error_document {    key = "error.html"  }}resource "aws_s3_bucket_public_access_block" "mrw-example-bucket-pab" {  bucket = aws_s3_bucket.mrw-example-bucket.id  block_public_acls   = false  block_public_policy = false}resource "aws_s3_bucket_policy" "mrw-example-bucket-policy" {  bucket = aws_s3_bucket.mrw-example-bucket.id  policy = file("policy.json")}resource "aws_s3_bucket_cors_configuration" "mrw-example-cors-config" {  bucket = aws_s3_bucket.mrw-example-bucket.id  cors_rule {    allowed_headers = ["*"]    allowed_methods = ["PUT", "POST", "GET"]    allowed_origins = ["*"]    max_age_seconds = 3000  }}###WEBSITE FILES###locals {  content_type_map = {    html        = "text/html",    js          = "application/javascript",    css         = "text/css",    svg         = "image/svg+xml",    jpg         = "image/jpeg",    ico         = "image/x-icon",    png         = "image/png",    gif         = "image/gif",    pdf         = "application/pdf"  }}resource "aws_s3_object" "mrw-example-bucket-objects" {  for_each = fileset("website/", "**/*.*")  bucket   = aws_s3_bucket.mrw-example-bucket.id  key      = each.value  source   = "website/${each.value}"  etag     = filemd5("website/${each.value}")  content_type  = lookup(local.content_type_map, regex("\\.(?P<extension>[A-Za-z0-9]+)$", each.value).extension, "application/octet-stream")}My question to you all, is why is the behaviour different? It feels like I've missed something obvious somewhere.FollowComment"

Answer: 0Accepted AnswerSeems like caching issue. Cleared and resolved.CommentShareAWS-User-5216987answered a year ago


Question: Unable to create Workspace

"After creating several workspaces, today I am getting the error message "An error has occurred.  If you continue to experience an issue contact the AWS Support Team on the community forums and via AWS Premium Support at: AWS Support Center."Not much info here.  Does anyone have any tips on where to look for the error?  Or what it might be?FollowComment"

Answer: "1If you have AWS support, reach out to them to look at the logs around the deployment.Sometimes running the deployment again after 15-20min will make it successful.Be sure to look at the Directory Service accounts and make sure that the accounts are not being used anywhere else in another WorkSpaces desktop.Be sure you have enough resources available in your VPC like IP addressesLink to Troubleshooting Amazon WorkSpacesCommentShareAWS-User-EPanswered 6 months ago"


Question: SIN SERVICIO EN ECUADOR

"Good afternoon,Today, May 10, the COMPLETE aws service was not available for more than 3 hours, that is, the browser showed; "This website cannot be accessed" when trying to enter our web system (created in aws), not even the aws administration console.Any reason for this ???FollowComment"


Question: "Websoket with API Gateway, ECS and VPC link"

"Hi, I have a WebSocket deployed in ECS with load balancers on top of that and use the VPC link to link the API gateway with the ECS. For some unknown reason, I was able to connect to the API Gateway WebSocket successfully but can't send the data to the ECS. I send the message with the action "sub" but nothing hit the Docker container in the ECS. Is that because it is not possible to have the API Gateway WebSocket with load balancer and ECS connected by the VPC link? Can that be achievable? Or is the API Gateway WebSocket can only talk with Lambda?FollowComment"


Question: EventBridge target System Manager Automation pass in Execution Role

"I defined a EventBridge Rule which targets a System Manager Automation I wrote. The Automation runs a Document that starts like:schemaVersion: '0.3'assumeRole: "{{ AutomationAssumeRole }}"parameters:  AutomationAssumeRole:    type: String    description: "(Required)"    default: ""For the EventBridge Rule, I define the Execution Role to be an IAM Role I created. I also configure an Input Transformer with some data about the event that triggered the Rule.I want to pass in the Execution Role into the Automation's "AutomationAssumeRole" parameter.If I manually trigger the Automation in the UI and select the Role from the dropdown, the Automation execution works as expected. I have not been able to figure out the right way to define the Input Transformer to have the EventBridge Rule trigger the Automation and populate the parameter "AutomationAssumeRole".FollowComment"

Answer: "0To answer your question, we require details that are non-public information. Please open a support case with AWS using the following link: https://console.aws.amazon.com/support/home#/case/createCommentSharerePost-User-3152870answered 4 months ago"


Question: RDS Serveless V1 Timeouts

"Since a few hours ago, we have been getting seemingly random timeouts from our RDS Serverless 1 cluster. They are very simple single table queries, eg looking up a user. A query can fail, then succeed as normal when running immediately again. Successful queries don't appear to be any slower than usual.We have been using serverless since its launch, and have not experienced this before. Queries are called from different EC2 instances, they all exhibit the same problem, so I don't think the problem is there. The servers have been running for months, no recent changes.The timeouts are similar to what happens when I try to access the database from a different VPC without granting access. Maybe the connections are being routed through a different something??GaryFollowCommentrePost-User-2057524 9 months agoSetting up some tests and I am totally lost. Simple php page on a new server setup:$pdo = new PDO("mysql:host=$servername", $username, $password);Keep reloading page, connects OK about 4/5 times.Share"

Answer: "0Any chance your usage of the database is less than it used to be, and you're scaling down to 0?  Cold starts are pretty long - usually over 30 seconds so do cause timeouts.CommentShareEXPERTskinsmananswered 9 months agorePost-User-2057524 9 months agoNo, it is set to a minimum level 2 capacity units.SharerePost-User-2057524 9 months agoFor example, we have a button on our dashboard to reload a list of the user documents. Click that 4 times in a row and it will work, then the 5th one will fail. And may then work on the next 2, then timeout again. All running the same code on the same server.Share0And as quickly as the timeouts started, they stopped about 10 hours after they started. I did start a new database and the servers had no trouble connecting to it, so I was about to do a snapshot / restore to a new serverless database, when the old database started working again. No idea why.CommentSharerePost-User-2057524answered 9 months ago"


Question: Using RDS Proxy doubles or triples average response times

"When we swap over to routing our READ traffic to go through RDS Proxy our average response time (web app) goes from ~200ms to ~600ms. As soon as we swap back to using the Aurora cluster endpoint the average response time significantly drops. No other changes, the only change we make is the hostname for the DB.Is this normal? Is this a sacrifice we have to make for connection management and faster failover?See here https://i.imgur.com/Vi1zPoP.png it's painfully obvious when we swap over to route through the RDS Proxy.RDS Proxy and Aurora MySQL clusterSame region, same subnets, same VPC.Aurora MySQL cluster 1x Write 1x Read, db.r5.large instancesap-southeast-2 regionRunning Aurora v2.10The web application we're running is a standard PHP/Laravel web app. (php8, laravel 8)UpdateCloudwatch Metrics of the RDS Proxy (when in use)https://i.imgur.com/WXv1qzw.pnghttps://i.imgur.com/5U7zi9t.pnghttps://i.imgur.com/imq1e3N.pngFollowCommentAcceptMyAnswer a year agoWhat language & db client in your web app are used to make these db queries?SharePataTepartie a year agoJasonF, did you ever manage to fix your issue? We have a similar behavior where DB queries take roughly twice as long since we started talking to our aurora MySQL (1.x for us) through an RDS proxy.Share"

Answer: "0This is indeed strange behavior, Are you using IAM for authentication ? I am really curious if your connections are getting pinned and every call to the database has to get a new connection.  There are couple of metrics that you can monitor that might give you some idea where the delay is coming from, DatabaseConnections  and DatabaseConnectionsCurrentlyBorrowed to get the number of connections and connections in use, and DatabaseConnectionsBorrowLatency, DatabaseConnectionsCurrentlySessionPinned.  These are available in cloudwatch https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.monitoring.htmlCommentShareDevinder-theDBGuyanswered a year agoJasonF a year agoHi Devinder,We don't use IAM for authentication. It's username/password. I believe RDS Proxy stores this in AWS Secrets?Thanks, I'll check out those metrics.ShareJasonF a year agoI've added some metric screenshots if it helps.https://i.imgur.com/WXv1qzw.pnghttps://i.imgur.com/5U7zi9t.pnghttps://i.imgur.com/imq1e3N.pngShareGleepwurp 2 months agoHi!  Did you get a resolution for your issue?Share0These all look good. I would suggest please open a ticket with support as they will have access to logs and all and can help troubleshoot the issue.CommentShareDevinder-theDBGuyanswered a year ago"


Question: Creation/Modified Time of Spectrum tables (do not appear in pg_class_info)

"Hi I was wondering what was the best way to get the creation/last modified time for Redshift Spectrum Tables.For non Spectrum tables, I am planning to query PG_CLASS_INFO for relcreationtime but I don't see my external tables there.My current idea is to use the transient_lastDdlTime from parameters in svv_external_tables. It is the last modified time (not creation date) but will suffice in my use case.Wondering if anyone else has better suggestions.FollowComment"

Answer: "0Hi Camilla11,I too found the Redshift catalog aspects of external tables confusing.  In part IMO this is due Redshift skipping a catalog declaration step when Spectrum was first created.  In hindsight I think this may haunt them.  However, in your case we can get past it.The key concept to know is that Redshift Spectrum is really federating an external catalog, typically that catalog is the Glue Data Catalog, but it can also be a Hive Metastore Service running on an EC2 instance, an on-prem host you can reach over the network, or on the master node of an EMR cluster.  Because it's a federated catalog most of the external catalog metadata comes from the external catalog.  All Redshift is really putting into it's catalog is a pointer to where the federated catalog is via the external schema declaration.  This is where the skipping that catalog declaration step coming into play making it confusing.  Look carefully you external schema declaration and you see that the one of the key clauses in the statement points to where on the network to find the external catalog.  This is somewhat opaque for the Glue Data Catalog because Redshift already knows where to look, but for a Hive Metastore Service it's connection information.  You can also look directly at PG_CATALOG.SVV_EXTERNAL_SCHEMAS in the ESOPTIONS column for this same information.  I think it's a little clearer now that that view has been extended to cover more than the Spectrum use case and now includes federating to RDS PostgreSQL, Aurora, and other Redshift databases.So, to solve your original question.  There are several ways you can get to the external table metadata from the external data catalog.  One way is go to the data catalog itself directly.  Chances are you're using the Glue Data Catalog for Redshift external tables and you go to the Glue console or CLI/API to get this table metadata.  Another option is you can use the Athena console app to browse the table metadata in the Glue Data Catalog or Hive Metastore.  Lastly you should be able to see the external table metadata directly in Redshift by querying PG_CATALOG.SVV_EXTERNAL_TABLES.   Look at the PARAMETERS column of that view to see the table properties.  The external catalog contains a "Create Time" table property that has the value you're looking for.  AFAIK, there is not concept of a last modified time for an external table in the external catalog because by definition the data "in", or more accurately that the external table currently maps to, is managed outside of any database transaction control and thus the catalog does not have an opportunity to know when the data changes because there is no transnational control over the data.I hope this helps you get to the metadata you need.Regards,-KurtCommentShareklarsonanswered 2 years ago0Thank you!That is what I thought. When I look at the parameters I see only 1 field,  "transient_lastDdlTime" which I plan to use. Thanks!CommentShareCamilla11answered 2 years ago0Hi Camilla11,The external table properties you get are indicative of what created the table.  Different SQL engines or the service APIs will set different table properties.   You should be able to pattern match the table properties to a set of likely external table creation engines.Regards,-KurtCommentShareklarsonanswered 2 years ago0Hi Camilla11,The external table properties you get are indicative of what created the table.  Different SQL engines or the service APIs will set different table properties.   You should be able to pattern match the table properties to a set of likely external table creation engines.Regards,-KurtCommentShareklarsonanswered 2 years ago"


Question: Is there a way to create a Redshift Table from a Glue table's schema?

"Athena tables can be created from Glue tables which can have schemas based on crawlers.**Is it also possible to use the schema of a Glue table to generate a Redshift-compatible CREATE TABLE statement? **I tried SHOW CREATE TABLE encounter; in Athena. And then I tried plugging in the resulting CREATE TABLE statement in Redshift, but got an error:ERROR: syntax error at or near "`" Position: 23.I can go through the statement Athena generated and clean it up to fit Redshift requirements, like taking out the back-ticks, but I'm wondering if there's any more direct way to generate a table based on a Glue table?This is that CREATE TABLE statement that Athena generated:CREATE EXTERNAL TABLE `encounter`(  `resourcetype` string COMMENT 'from deserializer',   `id` string COMMENT 'from deserializer',   `meta` struct<lastupdated:string,profile:array<string>> COMMENT 'from deserializer',   `identifier` array<struct<use:string,system:string,value:string>> COMMENT 'from deserializer',   `status` string COMMENT 'from deserializer',   `class` struct<system:string,code:string> COMMENT 'from deserializer',   `type` array<struct<coding:array<struct<system:string,code:string,display:string>>,text:string>> COMMENT 'from deserializer',   `subject` struct<reference:string,display:string> COMMENT 'from deserializer',   `participant` array<struct<type:array<struct<coding:array<struct<system:string,code:string,display:string>>,text:string>>,period:struct<start:string,end:string>,individual:struct<reference:string,display:string>>> COMMENT 'from deserializer',   `period` struct<start:string,end:string> COMMENT 'from deserializer',   `location` array<struct<location:struct<reference:string,display:string>>> COMMENT 'from deserializer',   `serviceprovider` struct<reference:string,display:string> COMMENT 'from deserializer',   `reasoncode` array<struct<coding:array<struct<system:string,code:string,display:string>>>> COMMENT 'from deserializer')ROW FORMAT SERDE   'org.openx.data.jsonserde.JsonSerDe' WITH SERDEPROPERTIES (   'paths'='class,id,identifier,location,meta,participant,period,reasonCode,resourceType,serviceProvider,status,subject,type') STORED AS INPUTFORMAT   'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'LOCATION  's3://bucket/Encounter/'TBLPROPERTIES (  'CrawlerSchemaDeserializerVersion'='1.0',   'CrawlerSchemaSerializerVersion'='1.0',   'UPDATED_BY_CRAWLER'='healthlake-export-crawler',   'averageRecordSize'='1561',   'classification'='json',   'compressionType'='none',   'objectCount'='14',   'recordCount'='53116',   'sizeKey'='83059320',   'typeOfData'='file')Here's an example of what the original data looks like (it's synthetic data, so not PHI):{"period": {"start": "2019-11-18T13:53:49-08:00","end": "2019-11-18T14:23:49-08:00"},"subject": {"reference": "Patient/92e36d1e-66a2-4e77-9f50-155f7edf819c","display": "Cyndi533 Bogan287"},"serviceProvider": {"reference": "Organization/3ecb1bdd-03d7-3fd2-b52d-8df2a04f5b0a","display": "SOUTH SHORE SKIN CENTER, LLC"},"id": "b39745ae-14dd-46b3-9345-2916efa759ad","type": [{"coding": [{"system": "http://snomed.info/sct","code": "410620009","display": "Well child visit (procedure)"}],"text": "Well child visit (procedure)"}],"class": {"system": "http://terminology.hl7.org/CodeSystem/v3-ActCode","code": "AMB"},"participant": [{"period": {"start": "2019-11-18T13:53:49-08:00","end": "2019-11-18T14:23:49-08:00"},"individual": {"reference": "Practitioner/c51e847b-fcd0-3f98-98a7-7e4274a2e6f3","display": "Dr. Jacquelyne425 O'Reilly797"},"type": [{"coding": [{"system": "http://terminology.hl7.org/CodeSystem/v3-ParticipationType","code": "PPRF","display": "primary performer"}],"text": "primary performer"}]}],"resourceType": "Encounter","status": "finished","meta": {"lastUpdated": "2022-04-08T15:40:39.926Z"}}FollowComment"

Answer: "0Accepted AnswerHi,A Glue ETL job can be configured to create tables in the data target. If your goal is to create a table in Redshift and write data to it, consider looking into Glue ETL referenced below. There is an option to have Glue create tables in your data target, so you wouldn't have to write the schema yourself. You could then see the schema by generating the table DDL afterwards in Redshift.https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/load-data-from-amazon-s3-to-amazon-redshift-using-aws-glue.htmlCommentSharehamltmanswered a year agoFabrizio@AWS EXPERTa year agothe link is broken it actually open the same repost page.Sharehamltm a year agoThank you Fabrizio@AWS, link is fixed.ShareData_dozer a year agoThat link gives nothing more than a high-level overview that isn't useful. It says for the target, choose "create tables in your data target" but doesn't say which transform might allow that option because I don't see it in the Apply Mapping or Select Fields transforms. Then for related resources it sends you to the generic AWS Glue documentation where there might possibly be something helpful in the hundreds of pages of documentation.Shareyann_stoneman a year agoThanks! So it sounds like the answer is "no, you can't use an existing glue data catalog table, but the end goal can be achieved via a Glue ETL job."Share0Have you considered creating an external schema in Redshift pointing to a database in Glue catalog, so the data can be accessed from Redshift via Redshift Spectrum?https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_EXTERNAL_SCHEMA.htmlCommentShareDenis_Aanswered a year agoyann_stoneman a year agoThe name of the Glue db is healthlakedb. The schema is different for each table in the Glue db. However, I don't see a field for glue table, so this was my command: CREATE EXTERNAL SCHEMA healthlake FROM DATA CATALOG DATABASE 'healthlakedb' REGION 'us-east-1' IAM_ROLE 'default' CREATE EXTERNAL DATABASE IF NOT EXISTSGot this error: "Unknown std exception when calling external catalog API ... error: Assert code: 1000 context: arn.valid() || g_xencppunit - query: 0 location: datacatalogclient.cpp:228 process: padbmaster"Any tips on how this works with Glue tables? Using Redshift ServerlessShare"


Question: Method to output individual records to S3 objects with Glue ?

"Hi,I have a list of json records that's been ingested (daily process), data is crawled, I'd like to output the records individually to S3 objects. i.e. for each unique record, create an S3 object. I'd then take that object and use its creation event as a trigger for onward processing. How do I get Glue to output the object for the individual record  ?I guess I need to iterate in PySpark and output each DynamicRecord (?) from the DynamicFrame or is there another option / idea ? any examples on doing that ? Many thanks for any advice.FollowCommentAnanth Tirumanur 9 months agoIs there a reason you would like to create an unique S3 object for each unique record? Too many files would not be ideal for storage (S3) or for processing (Glue). The size of each S3 file or number of records depends on the number of executors used by Glue. Glue distributes your computing into multiple executors and each of those would process and write those records. Writing out one record per file is not an ideal usage of Glue functionalityShare"