Question: Cannot get Websocket (Python) working with Elastic Beanstalk

"Hi,I am new to coding and also to AWS, so please do not be to hard on me if this is a dumb question. For training purposes I am trying to create an app (Kotlin) with the backend (Python) running on Elastic Beanstalk. Everything is working fine except the websocket data transfer.Here is the shortened Backend-Code:Import...app = application = Flask(__name__)ping_timeout = 120ping_interval = 60socketio = SocketIO(application, cors_allowed_origins="*", engineio_logger=True, allowEIO3=True, ping_timeout=ping_timeout, ping_interval=ping_interval)logging.basicConfig(level=logging.DEBUG)@application.route('/send_test', methods=['POST'])def send_test():    socketio.emit('test', {'response': 'TEST RECEIVED!!!!!'})    return "Success", 200And this is the shortened Frontend-Code:class MainActivity : AppCompatActivity(), TextToSpeech.OnInitListener, NavigationView.OnNavigationItemSelectedListener {    private lateinit var socket: Socket    override fun onCreate(savedInstanceState: Bundle?) {        connectWebSocket()    private fun connectWebSocket() {        Log.d("MainActivity", "connectwebsocket called")        val socketUrl = "ws://travel32.us-east-1.elasticbeanstalk.com"        val job = Job()        val coroutineScope = CoroutineScope(Dispatchers.IO + job)        IO.setDefaultOkHttpWebSocketFactory(OkHttpClient.Builder().addInterceptor(HttpLoggingInterceptor().setLevel(HttpLoggingInterceptor.Level.BODY)).build())        IO.setDefaultOkHttpCallFactory(OkHttpClient.Builder().addInterceptor(HttpLoggingInterceptor().setLevel(HttpLoggingInterceptor.Level.BODY)).build())        coroutineScope.launch {            socket = IO.socket(socketUrl, IO.Options().apply {                reconnection = true                reconnectionAttempts = Integer.MAX_VALUE                reconnectionDelay = 1000 // 1 second                reconnectionDelayMax = 5000 // 5 seconds            })            socket.connect()            Log.d("MainActivity", "WebSocket connected")            socket.on("test", testsocket)            socket.on(Socket.EVENT_CONNECT, onConnect)            socket.on(Socket.EVENT_DISCONNECT, onDisconnect)            socket.on(Socket.EVENT_CONNECT_ERROR, onError)        }    }    private val testsocket = Emitter.Listener { args ->        runOnUiThread {            val testresponse = args[0] as JSONObject            Log.d("MainActivity", "Received TEST response: $testresponse")        }    }In short, when pressing a button in the app, it sends a post to "domain/send_test" and just for test purposes it should send via websocket "Test Received!!!!" to the frontend. When I push the button, I can see that the send_test function in the backend gets called, but "Received TEST response: $testresponse" in MainActivity never gets logged. The web.stdout.log of my environment outputs this when send_test is called:May 14 14:19:27 ip-172-31-13-189 web[2211]: jxIFnNaeB7crO36TAAAC: Sending packet MESSAGE data 2["test",{"response":"TEST RECEIVED!!!!!"}]May 14 14:19:27 ip-172-31-13-189 web[2211]: INFO:engineio.server:jxIFnNaeB7crO36TAAAC: Sending packet MESSAGE data 2["test",{"response":"TEST RECEIVED!!!!!"}]May 14 14:19:39 ip-172-31-13-189 web[2211]: jxIFnNaeB7crO36TAAAC: Sending packet MESSAGE data 2["test",{"response":"TEST RECEIVED!!!!!"}]May 14 14:19:39 ip-172-31-13-189 web[2211]: INFO:engineio.server:jxIFnNaeB7crO36TAAAC: Sending packet MESSAGE data 2["test",{"response":"TEST RECEIVED!!!!!"}]May 14 14:20:01 ip-172-31-13-189 web[2211]: jxIFnNaeB7crO36TAAAC: Sending packet PING data NoneMay 14 14:20:01 ip-172-31-13-189 web[2211]: INFO:engineio.server:jxIFnNaeB7crO36TAAAC: Sending packet PING data NoneMay 14 14:20:02 ip-172-31-13-189 web[2211]: luoag_u_sK4lq15kAAAE: Sending packet OPEN data {'sid': 'luoag_u_sK4lq15kAAAE', 'upgrades': ['websocket'], 'pingTimeout': 120000, 'pingInterval': 60000}May 14 14:20:02 ip-172-31-13-189 web[2211]: INFO:engineio.server:luoag_u_sK4lq15kAAAE: Sending packet OPEN data {'sid': 'luoag_u_sK4lq15kAAAE', 'upgrades': ['websocket'], 'pingTimeout': 120000, 'pingInterval': 60000}May 14 14:20:02 ip-172-31-13-189 web[2211]: luoag_u_sK4lq15kAAAE: Received packet MESSAGE data 0May 14 14:20:02 ip-172-31-13-189 web[2211]: INFO:engineio.server:luoag_u_sK4lq15kAAAE: Received packet MESSAGE data 0May 14 14:20:02 ip-172-31-13-189 web[2211]: luoag_u_sK4lq15kAAAE: Sending packet MESSAGE data 0{"sid":"1FGFeiKmah61v5UEAAAF"}May 14 14:20:02 ip-172-31-13-189 web[2211]: INFO:engineio.server:luoag_u_sK4lq15kAAAE: Sending packet MESSAGE data 0{"sid":"1FGFeiKmah61v5UEAAAF"}May 14 14:20:02 ip-172-31-13-189 web[2211]: luoag_u_sK4lq15kAAAE: Received request to upgrade to websocketMay 14 14:20:02 ip-172-31-13-189 web[2211]: INFO:engineio.server:luoag_u_sK4lq15kAAAE: Received request to upgrade to websocketMay 14 14:20:03 ip-172-31-13-189 web[2211]: luoag_u_sK4lq15kAAAE: Upgrade to websocket successfulMay 14 14:20:03 ip-172-31-13-189 web[2211]: INFO:engineio.server:luoag_u_sK4lq15kAAAE: Upgrade to websocket successfulAs you can see it tries to send the response but nothing is being received on the frontend. The original purpose is, that I have a function on the Backend which should return a value right away, but in another thread also calls an external API. When the answer of the external API is given, the function should additionally to the returned value also transfer the API response to the frontend.I read a lot that the load balancer must be configured somehow to make it work with websockets, but all tutorials and forum posts are from long ago when the Elastic Beanstalk UI looked way different and I could not manage to get it working.My app is running on "Single Instance (Free Tier)" Configuration and I did not configure a load balancer or anything else.I am now trying to get it work for over a week and hope that anyone can help me out :)Thank you!FollowCommentlarryjkl 9 days agoI'm not sure this helps; but with Beanstalk it configures a proxy automatically I think; either Apache or NGINX? I use Apache and I had to add some configuration to the apache config in order for it to support web sockets. If this seems helpful, I can post what I had to do to get it work, let me know.Share"


Question: IoT Core: KafkaAction failed to send a message to the specified bootstrap servers. SSL handshake failed

"I'm trying to create an IoT Core Rule that triggers a Kafka action to send our device data directly to our Kafka cluster, using Basic Ingest. I have successfully setup the VPC Destination that this action needs, specifying our subnets that contain the Kafka brokers; however, I am getting the following log in CloudWatch when the KafkaAction fails:{    "ruleName": "testKafkaAction",    "topic": "",    "cloudwatchTraceId": "<id>",    "clientId": "test",    "base64OriginalPayload": "<payload>",    "failures": [        {            "failedAction": "KafkaAction",            "failedResource": "iot-core-sensordata-stream",            "errorMessage": "KafkaAction failed to send a message to the specified bootstrap servers. SSL handshake failed. Message arrived on: , Action: kafka, topic: test-kafka-action, bootstrap.servers: <bootstrap_servers>"        }    ]}Our Kafka cluster is self hosted, and uses a self-signed CA. I have created a binary secret in Secrets Manager containing the truststore in pkcs12 format. The truststore contains our self-signed CA. I am using SASL_SSL as the security protocol for the rule, and SCRAM-SHA-512 as the mechanism. The username and password are also stored as a separate secret in Secrets Manager. My rule's IAM policy is setup correctly to access these secrets. One other thing to note: My bootstrap servers list for the rule are the private ip:port of the nodes running the Kafka brokers in our VPC, and I have ensured these IPs are on the SAN list for the cert.Here is the json template for my rule as a reference:{    "sql": "SELECT *",    "ruleDisabled": false,    "awsIotSqlVersion": "2016-03-23",    "actions": [        {            "kafka": {                "destinationArn": "<VPC_DESTINATION_ARN>",                "topic": "test-kafka-action",                "clientProperties": {                    "bootstrap.servers": "<KAFKA_BOOTSTRAP_SERVERS>",                    "key.serializer": "org.apache.kafka.common.serialization.StringSerializer",                    "value.serializer": "org.apache.kafka.common.serialization.ByteBufferSerializer",                    "security.protocol": "SASL_SSL",                    "ssl.truststore": "${get_secret('<SECRET_NAME>', 'SecretBinary', '<KAFKA_RULE_ROLE_ARN>')}",                    "ssl.truststore.password": "{{SSL_TRUSTSTORE_PASSWORD}}",                    "sasl.mechanism": "SCRAM-SHA-512",                    "sasl.scram.username": "${get_secret('<SECRET_NAME>', 'SecretString', 'kafkaUser', '<KAFKA_RULE_ROLE_ARN>')}",                    "sasl.scram.password": "${get_secret('<SECRET_NAME>', 'SecretString', 'kafkaPassword', '<KAFKA_RULE_ROLE_ARN>')}"                }            }        }    ],    "errorAction": {        "cloudwatchLogs": {            "logGroupName": "AWSIotLogsV2",            "<KAFKA_RULE_LOGS_ROLE_ARN>"        }    }}This error message does not provide any reason as to why the handshake failed, so my only guess is that IoT Core does not allow self-signed CAs for the KafkaAction in a rule. Is this true, or am I missing something elsewhere potentially? I am able to connect and publish a message to our Kafka cluster with no issues using a python client, giving it the same credentials and self-signed CA, after having to add our servers' IP addresses to the SAN list on the cert.FollowComment"

Answer: "1Accepted AnswerI ended up getting things working with a slightly different configuration, so answering my own question to help out anyone. I am still using a self signed CA for our Kafka cluster, and am providing this to the KafkaAction in my IoT Core Rule via a PKCS12 formatted truststore in Secrets Manager, stored there as a binary secret (I have confirmed that using the old JKS format for the truststore works as well FYI). I was able to get things working by generating a X.509 cert and private key for the KafkaAction to use from the self signed CA, and switching the protocol to SSL. I created a keystore, also in PKCS12 format, containing this cert and private key, and likewise stored it in Secrets Manager as a binary secret. For those unfamiliar, you can only create binary secrets via the CLI or SDKs, it is impossible through the console. In my rule's configuration, I used the get_secret method as shown in the docs to instruct IoT Core to get the binaries from Secrets Manager. My IAM role for this rule also has a policy allowing access to these secrets. One last thing to note: My truststore and keystore both are password protected, so I also had to specify this password (the same for both in my case) in the configuration for the rule. I was never able to get things working with SASL and SCRAM-SHA-512, so my only guess is that this is unsupported with self signed CAs, or the documentation is confusing regarding what regions this is actually supported in.CommentShareOtto45answered a year agoJyoti 13 days ago@Otto45 How did u automate the creation of secrets for cert ? I also got it working the same manner you have .. thanks to you but I am facing difficulties while automating it so that IoT rule always get the right cert. For reference entire details are on my post https://stackoverflow.com/questions/76280908/aws-iot-rule-to-self-managed-ssl-enabled-kafka-clusterShare0Hi, I am facing the same problem. In addition to what is shared here, I also tried using SSL method and generated a key pair signed by the CA in my cluster. I also created a CA myself and created key pairs for brokers and IoT Core. But all of them ended up to the same error.  @Otto45, did you folks solve the issue? If so, can you please share it here. Thanks!CommentShareacsgnanswered a year agoOtto45 a year agoI got things working, and I had to switch to SSL from SASL_SSL. I posted a detailed answer, hope it helps!Share-1Hello,Thank you for reaching out to us. I understand that you were getting SSL handshake failed error while trying to create an IoT Core Rule that triggers a Kafka action to send our device data directly to our Kafka cluster, using Basic Ingest.Could you please confirm the region which you are using. I am asking the region because you have mentioned that you are using SASL_SSL as the security protocol for the rule, and SCRAM-SHA-512 as the mechanism, SCRAM-SHA-512 is the only supported security mechanism in the cn-north-1, cn-northwest-1, us-gov-east-1, and us-gov-west-1 Regions. Please refer to this link [1] for more information on this security mechanism supported regions.References:https://docs.aws.amazon.com/iot/latest/developerguide/apache-kafka-rule-action.html#apache-kafka-rule-action-parametersCommentShareSUPPORT ENGINEERNamratha_Kanswered a year agoOtto45 a year agoI am using us-east-2, and that link says you have to use SCRAM-SHA-512 for SASL_SSL in those specific regions. The way I understand it I should be fine using SCRAM-SHA-512 in any region.Share"


Question: What's an AWS Practice exams used for?

"In this URL https://aws.psiexams.com/#/dashboard/compact-dashboard, what's a Practice exams used for? Just look at the scores that we pass, should we really go to the exam or not? Because there is no solution.Thank you very much.FollowComment"

Answer: "0PlAwAnSaI,Practice exams are a great way to mimic the exam experience. You will be able to use the look and feel of the exam as well as answer questions. You will receive a score report for your result although you will not receive feedback on each individual question.KimAWS CertificationCommentShareKimWD-AWSanswered 2 years ago"


Question: Help! How do you turn off AWS config!

"Hi,I wanted to learn about AWS config so I started it with some rules.  Then I got the bill so I decided to stop the service but I can't seem to.  I stopped recording and I have no rules running but I still get billed for $4.00 per month.It seems like config is easy to turn on but not so easy to off.FollowComment"

Answer: "2Turn off Recording for that region using the consoleDelete the Rule by going to actions, delete ruleUse the AWS CLI and delete the default recording byaws configservice delete-configuration-recorder --configuration-recorder-name default --region <region-name>Delete the service linked role created for AWS ConfigRefresh the Config home page to make it appear fresh.If necessary delete the config bucket and its objects.Edited by: ecooper on Sep 18, 2021 10:28 AMCommentShareecooperanswered 2 years ago1AWS CLI:aws configservice describe-config-rules | grep ConfigRuleName | cut -d":" -f2 | cut -d"," -f1 | xargs -L1 aws configservice delete-config-rule --config-rule-nameCommentShareiothreat1answered 2 years ago0I had to use CLI to remove all the rules. Just executed - will see how it affects billing tonight.P.S. instead of grep/awk feel free to use --query parameter of aws commandhttps://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.htmlaws configservice describe-config-rules | grep ConfigRuleName | gawk 'match($0, /:.+"(.+)"/, a) {print a[1]}' | while read rule_name; do echo $rule_name; aws configservice delete-config-rule --config-rule-name $rule_namedoneCommentShareeugenjanswered 3 years ago0In your billing console, you can view what charges are still being applied. It is possible that the Config recording is still turned on. This is the capability that records the resource configuration changes and provides the trigger for change triggerd/ hybrid rules to evaluate when there are resource specific changes.You can find steps to manage the Config recorder  https://docs.aws.amazon.com/config/latest/developerguide/stop-start-recorder.html .CommentShareAWS-User-4960325answered 3 years ago-1In Console,Go to the setting page of config and disable recorder.Delete all the config rulesyou will be fine nowCommentSharerePost-User-9257957answered 9 months ago"


Question: I am trying to write an ETL job to the Data Catalog but its writing the Headers as Data

"I am trying to read data from S3, transform it and insert it into a table I have created in the Data Catalog.My ETL job reads data with create_dynamic_frame, converts to a Spark DataFrame, does its transforms, converts back to a Dynamic DataFrame and then writes that frame to the Data Catalog.I do the write to the Data Catalog table like so:DataCatalogtable_node2 = glueContext.write_dynamic_frame.from_catalog(    frame = dynamic_df,    database = db_name,    table_name = tbl_name,    transformation_ctx = "DataCatalogtable_node2",)I manually defined my Data Catalog tableColumn NameData TypevidstringaltidstringvtypestringtimetimestamptimegmttimestampvalueintfilenamestringThis matches the schema of my Dynamic DataFrame write before I do the write operationroot|-- vid: string|-- altid: string|-- vtype: string|-- time: timestamp|-- timegmt: timestamp|-- value: int|-- filename: stringWhen I attempt to look at the Data in Athena after the ETL job runs, I get errors, because stored in "time" the first value is called "time".  My timestamps look like and I think they are in the correct format.vid,altid,vtype,time,timegmt,value,filenameABC,ABC,DATAFEED,"2021-10-31 22:00:00.0","2021-11-01 02:00:00.0",11775,"s3://bpf-load-data/abc_data/lf_buf_20211101_005904.csv"Originally (in s3) time is in the format 31-10-2021 22:00:00 and so in order to get it into the correct format that the Data Catalog/Athena use for timestamp I am doing:df = df.withColumn("time", to_timestamp("time", 'dd-MM-yyyy HH:mm:ss'))I am not sure if this is correct.  I get 2021-10-31 22:00:00.0 instead of what I would suspect which is 2021-10-31 22:00:00In any case, in Athena when I attempt to read the table, the actually first Data in time is the word time and therefore it does not match timestamp and it all errors out.HIVE_BAD_DATA: Error parsing field value 'TIME' for field 3: For input string: "TIME"It is almost as if it is trying to insert the "header" of the dataframe as actual data.Does anyone know what could be going on?FollowComment"

Answer: "1Accepted AnswerI was able to figure this out.  I was creating my Data Catalog table manually, and I needed to add the table property "skip.header.line.count = 1".CommentSharebfeenyanswered a year agoEXPERTFabrizio@AWSreviewed a year ago"


Question: GUI for Linux2 on-premises Hyper-V instance

"Hi all,I've just started working with AmazonLinux2 and want to use my normal developer tools like kdevelop and other GUI tools. I have been able to create a Hyper-V VM with AmazonLinux2 on my local machine but it seems to be command-line only. Can I run a GUI such as the one used in Ubuntu? If so, how can I set this up?Thanks.FollowComment"

Answer: "0Hi @technovelist By default Amazon Linux 2 is just a CLI version since we don't have GUIs on Linux EC2 instance, We don't have repos setup for X display on Amazon Linux 2, but you may be able to leverage this article to help https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-2-install-gui/  You may need to add custom repos if you really need a GUI.CommentShareEXPERTRob_Hanswered a year ago"


Question: Upgradation and Downgradation of any AWS Service

"Is there any proper person of AWS to whom we can contact to know whenever AWS upgrades or rollback any service, then where it maintain details of older version of that service ??FollowComment"

Answer: "0Hello,Firstly, I would like to mention that Security and Compliance is a shared responsibility between AWS and the customer.Going deeper into this, a customer responsibility will be determined by the AWS Cloud services that a customer selects.For example -Customers that deploy an Amazon EC2 instance are responsible for management of the guest operating system (including updates and security patches), any application software or utilities installed by the customer on the instances, and the configuration of the AWS-provided firewall (called a security group) on each instance.For services such as Amazon S3, AWS operates the infrastructure layer, the operating system, and platforms, and customers access the endpoints to store and retrieve data. Customers are responsible for managing their data (including encryption options), classifying their assets, and using IAM tools to apply the appropriate permissions.Now, in regards to updates related to a service, say in terms of changing behavior of existing features, or introduction of new feature etc; there are several methods of communication channel lines -When your current resources are affected by any upcoming changes, you will be notified by the service team for the corresponding service using support cases and health dashboard notifications.For any new features, you can see the announcement on our What's New with AWS? page here or AWS blog here.For our enterprise support customers, you can also get in touch with your Technical Account managers, who can assist with answering such queries. Please note such queries maybe subjected to NDAs, limited discretionary disclosures etc which you can confirm with your account team. Furthermore, you can also work with your Technical account manager to submit feature requests as well for future updates for the service.Finally, you can also proactively check with our Support team by creating a support case if you have any questions regarding any upcoming changes, or new features etc.Additionally, you can also find information on post event summaries here following the closure of the issue.CommentShareSUPPORT ENGINEERYash_Canswered 4 months ago"


Question: "64-bit Positioning & World Sizes,  Large World Coordinates, and 64-bit Physics Refactoring"

"With StarCitizen migrating to lumberyard, and recent tech of Procedural Planets (v2), Pre-built Biomes (Mountain, Desert, Woodland, etc.), Advanced Character Tech, Planet Editor, Item 2.0, and advanced Weather System, will that or a similar tech (& tools) also be available for lumberyard? Is it possible for Amazon to add each of these new tech/features/tools to Lumberyard as Amazon Gems? (that can be selected and included through the Amazon Project Configurator and from the command line)Follow"

Answer: "0@REDACTEDUSERCIG/RSI/Chris Roberts has already created all the code/tools for Star Citizen. It would only be a matter of Amazon Lumberyard Developers working together with CIG/Star Citizen/Foundry 42 developers to "re-factor" that code (and development tools/assets) into Amazon Lumberyard "Gems" (that can easily be added to any Lumberyard build). As well as creating a sample asset pack/game/tutorial for Lumberyard (slimmed down version of Star Citizen).Users have already donated/given over $140 Million to Star Citizen's Kickstarter project/stretch goals. It would be nice if some of that development code/work was given back to the Lumberyard Community (as a "StarEngine Gem" module and assets). So that others in the Lumberyard Community could use the same tech, and tools that CIG/Foundry 42/Star Citizen are using with Lumberyard.Refactor and convert Star Citizen/CIG's Lumberyard source code modifications to a new optional Lumberyard "Gem" that can be installed. Then bundle it with the Tools/Editors and create a sample asset pack (using some Star Citizen sample assets) and bundle it with the new Lumberyard "StarEngine Gem" to create assets/tools/demo game that Lumberyard developers and community can learn from, share, and use to develop AAA-quality games using the same content generation tools that CIG/Star Citizen are using.Using powerful procedural content-generation tools, a very small team of developers can customize and build a "AAA-quality" game using procedural content generation. Small developers and indie studios need Advanced Procedural Tools for Procedural Planets, Pre-built Biomes (Mountain, Desert, Woodland, etc.), Advanced Character Tech, Planet Editor, Advanced Weather System, Item/Inventory 2.0, etc.Extremely powerful and advanced content and games can be created by very small teams of Indie developers (3 to 13 developers). We just need the tools (and tutorials).These content-generation tools (and sample assets) need to be "built once" but can be used by many. If these advanced features/tools were added to the Lumberyard engine (like CIG/Star Citizen has done), if these tools/features were added to the engine, users can easily build large AAA-quality universes (using advanced procedural generation tools), and look at included sample game assets and simply add/create additional vehicles, Flora, Fauna, etc.New Fauna/Flora/Vehicle Assets could easily be modified or built by artists and shared/sold on Amazon Lumberyard Storefront for Star Citizen (or any other developer) to use in their game.With the correct tools, a small team of 3 to 13 developers can easily create a procedural game with an "infinitely" large Universe (like No Man's Sky or something more advanced like Star Citizen).SharerePost-User-9149768answered 6 years ago0If you have the budget and the stuff to make a game so big that cannot be done without that technology you may surely do few other great games working with Lumberyard as it is right now.I suggest to start with them while you wait...who knows...it could be a 64bit long wait...better to get busy :)SharerePost-User-7621041answered 6 years ago0Ya ,wishing....I was drooling in a way, watching video wishing I had those tools,,wow what I could do with them, but unless they already partnered with LY, hmm ...Unless they are holding back on announcing it ;)If not, I can't imagine how much money would exchange hands for that, but I can't assume LY etal couldn't manage something.If not so be it, I'm not that worried as I'm enough of a artists and have enough ideas that I can pull it off in time, much longer, but..manually with some clever thought and my worlds won't be 'procedural', but from my heart with ideas from nature, et cetera.I love the idea of 'help' , but I can do without if needed, and in my case I prefer it. I could stand to use a much larger team and in time I will acquire one, but manual words vs procedural has pros and cons as we all know.I tend to prefer a 'mix',or at very least the former not latre.But yes as OP noted, having ability to 'edit' worlds after creation is a bit of a unique merging of those thoughts.Trouble is, for the pleasure, would LY have to pass along those costs,if so where and how much :?ONe note,,,watching the video, which is amazing,it was easy to spot the procedural nature of it,which is a turn off to me; just saying...tho awesome it is.I did not watch entire thing no time, but on char. editing, seems to me LY is already incredible unique there,and what LY doesn't have, I'm sure maya, max or blender could provide via other mechanisms ?SharerePost-User-2755327answered 6 years ago0@REDACTEDUSERDevelopers can already easily make "other great games" in Unity or Unreal Engine, but what I am asking for is advanced tools for large (unlimited-size) maps, planets, and procedural content generation specific to CryEngine/Lumberyard development (which would allow/make it EASY to build extremely large universes and large maps using procedural content generation) which is why Chris Roberts (and others) developed such tools.These tools already exist for Lumberyard (look at CIG/Star Citizen), which is using a heavily modified version of Lumberyard.I am just asking whether a similar tech (and tools) will also be available for Lumberyard? Could these tools be "converted" from CIG-only tools and be "converted" into Amazon Gems that others in the Lumberyard Community can learn from and use to develop their own AAA-quality games?These tools have already been created for Lumberyard (by CIG for Star Citizen). Recent tech of Procedural Planets (v2), Pre-built Biomes (Mountain, Desert, Woodland, etc.), Advanced Character Tech, Planet Editor, Item 2.0, and advanced Weather System, will that or a similar tech (& tools) also be available for lumberyard?Is it possible for Amazon to work with CIG and add each of these new tech/features/tools to Lumberyard as Amazon Gems? (that can be selected and included through the Amazon Project Configurator and from the command line)It's nearly impossible to use Lumberyard (or CryEngine) to create a large universe without these powerful development tools.Artists, Content Creators and Game Developers need advanced procedural tools, and advanced content-generation tools (which currently don't exist in the current version of Lumberyard or CryEngine). This is why I am asking.I'm not asking for someone to just tell me to use Lumberyard (or CryEngine) to build something simple like "Tetris" instead (or go back to the "old way" of development, which does NOT work when trying to build something as large-scale as a planet, or universe).As developers, we need powerful development and content-generation tools, and sample assets/games/tutorials. I am just asking if Amazon can "partner" with CIG/Chris Roberts, and get the development tools and source code (from CIG) converted into Amazon Gems, that everyone in the community can use (and benefit from) and added to Lumberyard as Amazon Gems.SharerePost-User-9149768answered 6 years ago0Yes it would be nice, point is, star citizen devs are under no obligation , to do so, thus I won't get my hopes up.As noted, and while the video is very cool,I'm going in a more manual direction.I can see you are excited about it, and I hope you get your wish, and if so, I'll certainly 'take a look at it' ,,who wouldn't ;)GLSharerePost-User-2755327answered 6 years ago0Great conversation going here! Large maps and worlds is something that we hear a lot of requests for. We don't really have any details or timelines to share at this time, but we will as soon as we can.SharerePost-User-8353690answered 6 years ago0A bit too much to hope to have all the custom made toolsets that Star Citizen have made for themselves, but 64-bit precision/the ability for very large worlds would definitely be a great addition that I am sure will be implemented along the line... (Crytek were working on it (segmented worlds) for years then changed their approach).SharerePost-User-9959924answered 6 years ago0On the other hand Star Citizen is essentialy made by public money (crowd sourcing). You can make case, that by doing so, they should made all their changes available to everyone.SharerePost-User-9496452answered 6 years ago0It would be extremely helpful, if the Lumberyard Devs could work together with the Star Citizen Devs, and help migrate/integrate/refactor some of that code directly into Lumberyard, and make the tools available to developers. (Universe Editor, Planet Editor, procedural generation tools, etc.) Using powerful procedural content-generation tools, a very small team of developers can customize and build a "AAA-quality" game using procedural content generation. Small developers and indie studios, as well as for teaching students... I hope that Lumberyard devs can collaborate and work closely with the Star Citizen dev team. I hope many of these tools (and code) can be re-factored (as gems) and integrated into Lumberyard (as "Gems") that we can add to Lumberyard, and we definitely need Advanced Procedural Tools for Procedural Planets, Pre-built Biomes (Mountain, Desert, Woodland, etc.), Advanced Character Tech, Planet Editor, Advanced Weather System, Item/Inventory 2.0, etc. Has any progress on this been made?SharerePost-User-9149768answered 6 years ago0If the Lumberyard Devs agreed to work with (and help) Chris Roberts (and Star Citizen devs), hopefully the code developed could be shared equally between Star Citizen (and Lumberyard), and hopefully the code could be "re-factored" by Lumberyard Devs and included as modules (Gems) that Lumberyard users/developers could add/install as part of their Lumberyard install. It would definitely be great to have 64-bit positioning & Large World Sizes and Large World Coordinates, and 64-bit physics refactoring in Lumberyard. With StarCitizen migrating to lumberyard, and recent tech of Procedural Planets (v2), Pre-built Biomes (Mountain, Desert, Woodland, etc.), Advanced Character Tech, Planet Editor, Item 2.0, and advanced Weather System, will that or a similar tech (& tools) also be available for lumberyard? Is this something that the Lumberyard Devs can bring to the Lumberyard Community? It would be great to have the "Item 2.0", and "Advanced Character Tech", Advanced Weather, and Procedural Planets (as well as the Planet and System Editors) made available to Lumberyard users/developers. Has any work or progress been made on this? I believe it would benefit both sides (both Chris Roberts, and Lumberyard), as well as help grow the Lumberyard Community by having such tools. I would be willing to help with development, as well as video tutorials, etc. Is this something that we can get added to Lumberyard? Possibly work on "merging" the code changes that Star Citizen has made, into "gems" (by re-factoring the code), and turning those changes/modifications into "gems" that can easily be added to a basic Lumberyard install. It would simply the code base (for Star Citizen), and also be great for the Lumberyard Community. Has any progress on this been made?SharerePost-User-9149768answered 6 years ago0Hi Binky. I just wanted to tell you that I would gladly port my tech from UE4 to Lumberyard if it used double precision coordinates. This is the #1 hindrance to next-gen gaming -- and it's relatively easy to get around the GPU float limitation: you just need to convert all locations to a position relative to the camera and cast down to float. Then you can hand the location to the GPU and the precision loss is pushed back to the objects that are far away, which gets obscured with distance -- so it works out perfectly (I did it in XNA way back).The only reason(s) to use floats is network bandwidth and/or RAM, but both of those resources have grown way beyond the limitations that existed when floats were chosen for GPU architectures, so now is the time to make double precision coordinates a reality.If you can get the code from CIS then holy cow, what are you waiting for?SharerePost-User-0696492answered 6 years ago0Amazon seems to have this on low priority. Maybe this might be realized from the community side at some point. I had a look at the Cryengine engine code myself to identify what classes would need to be modified to realize this. Not all that easy, especially the physics system is complex. All the Vec3 would have to be replaced with Vec3d all over the place. All the geometry-related code like intersection math would have to be switched to f64 as well. I don't think this could be turned into gems, it goes too deep into the systems. If you wanna work with planets then you would also have to do the same thing like CIG and rewrite the terrain-system. Certainly not a two weeks work, but not impossible to do by a couple of folks with enough patience and dedication.Also to find the appropriate spots for all passes to convert the 64-bit coordinates back to camera-relative 32-bit values prior to CPU to GPU shader transfer / rendering is difficult. I simply don't know the engine well enough. What i tried so far didn't work.Would be interesting to start a research group of like-minded engine nuts. Not sure though if there would be actually some folks with enough engine knowledge out there, that would be up for it.SharerePost-User-9439563answered 5 years ago0Agreed. It only took Sean Tracy (CIG / CryTek) two weeks to write the original code/modifications to the original CryEngine for 64-bit world sizes (64-bit double precision), and it took less than 8 hours for Sean Tracy to "refactor" and "port" that old code over to Lumberyard (for the 64-bit double precision implementation) in Star Citizen / CIG on Lumberyard. It would seem that it really shouldn't take too long (or be too difficult to do) with the resources that Amazon has (as long as they can dedicate a few developers to this, and have a developer like Sean Tracy, that knows what they are doing). Any thoughts on hiring Sean Tracy to implement these features into Lumberyard?SharerePost-User-9149768answered 5 years ago0Any news on this? Is this something that has been added to the Lumberyard Roadmap, and if so, is there a public timeline (and expected release date) for these features? Any news from the developers? Thank-you!SharerePost-User-9149768answered 5 years ago0I agree that something has to be done about the terrain system as is. The biggest problem I had with Unreal Engine was implementing larger terrains. Unreal never addressed it. Sure they made the Kite Demo, but that thing wouldn't even run on lesser machines and barely run on mid level graphics cards, not sure how they figured it was a showcase for large open worlds when it was useless for real time gaming. Even though Unreal had those problems, it is still better than what Lumberyard has. I can't even use an 8k heightmap. Sure I can make an 8k terrain, but you have to use a 4k heightmap that is just stretched over 8k. The terrain quality really suffers from it. World Machine and other such programs are heavily used by devs and artists, there really should be better and easier integration from such programs. Just my 2 cents.SharerePost-User-6267791answered 5 years ago0Maybe Crytek will sue Amazon then saying 64bit precision and all tech developed by Cloud Imperium belong to Crytek. Everyone should abandon Lumberyard and anything associated to Crytek in case their projects do take off and you actually make money. Will you feel safe from Crytek deciding they want some of it?SharerePost-User-2650819answered 5 years ago0@REDACTEDUSERTo answer your question: Amazon owns a completely different/separate license, and Amazon has the ability to distribute "Lumberyard" (source code, tools, etc.). Amazon can make all of the changes they want, and even turn this "Lumberyard" into a very completely different engine (different/heavily modified source code, Tools, etc.) from what CryEngine 3.8.x used to be. As you can see probably about 50% to 60% of the old Crytek code (used in early Lumberyard alpha/beta's) has already been removed/changed, and replaced by completely new Amazon Lumberyard code. Amazon Lumberyard developers are working hard to create a "completely new" engine (based formerly on the old CryTek code, but it's being completely changed and re-written/re-factored/improved and Lumberyard is basically a "new engine" completely different than the old CryTek 3.x engine).As for the silly CryTek lawsuits against CIG, that is a completely different topic, and has nothing to do with 64-bit positioning or infinitely large world sizes. (So please stay "on topic").As for the silly Crytek lawsuit against CIG, that is pertaining to a separate contract/license agreement that CIG signed with CryTek. It has nothing to do with Lumberyard, and more than likely the courts will probably "throw out" that frivolous nonsense lawsuit that CryTek filed against CIG (for "not using Crytek" and switching to Lumberyard). There doesn't even seem to be a contract, and only a license agreement and Crytek can't FORCE CIG to use CryEngine (as part of a license agreement). Just because CIG signed a license agreement, doesn't mean that CIG can't switch engines to Lumberyard. It's absurd thinking, and more than like the judge will just throw that silly frivolous nonsense case/lawsuit out.@REDACTEDUSERSharerePost-User-9149768answered 5 years ago0@REDACTEDUSERBefore diving into research forks, proper analysis and discussion is a good first step.It needs to be determined what subsystems besides Physics would be affected by introducing 64-bit entity positions.What would happen to animations, lighting?Anyone has some thoughts on this?The goal would be, having to touch as few as possible.Anywhere in the engine code where entity world matrices are read/written to,you would have to handle the higher precision floats and their relationship to other entities/positions.352 results for search term "GetWorldTM" in the engine github-repo.Hypothetically, if all entities would "know" about the camera that is currently active,a camera-relative 32-bit matrix could be returned from the entity instance directly and in one spot, which might let you get away with not having to change Physics, 'cause everything is still 32-bit , but not sure.This however violates OOP/code-architecture/SOC principles.Also all bounding box-related code needs to be adapted.A first version could maybe exclude any physics.Amazon would probably only ever consider merging or working on something like that if it didn't touchthe existing 32 bit processing. That would mean developing a parallel system for 64-bit.Sean Tracy or Marco Corbetta would for sure be the guys to talk to, i see a very low chance thoughthat they would want/are allowed to share too much details.Certainly a bit more concrete info on what systems exactly CIG had to touch to get this to work would help.The info publicly available from interviews is unfortunately too general.Chris Bolte gave a good overview about how they avoided passing 64-bit data to the GPU bytransforming to camera-relative spaces.A dedicated thread or some forum about such research project would be a good start.Maybe this can then reach some momentum...SharerePost-User-9439563answered 5 years ago0@Phaeton  - Yes, I completely agree. We don't have a detailed "Public Roadmap" from Lumberyard Dev Team, so we have no clue what they are currently doing (or where this "Lumberyard" project is even headed).I supposed we could create a "fork" (Lumberyard Research Fork) and begin implementing some type of 64-bit precision (similar to what CIG/RSI has done for Star Citizen), and possibly even reach out to Sean Tracy (CIG/Crytek) and a few members of the CIG/RSI team, and possibly see if he/they'd be willing to help contribute some of their existing Lumberyard modifications/code changes back to this new "Research Project" (Lumberyard fork).As part of the Research Project, we can also begin working on Lumberyard Tools/Editor (Planet Editor, Solar System Editor, Procedural Generation Tools, etc.) and also BIOMES, BIOME Editor, etc.It might help us get started, and Sean Tracy seems very familiar with the "inner workings" of CryEngine/Lumberyard. There are a few former Crytek engine devs working over at CIG, maybe they will join us in forming a "Lumberyard Research Group" (fork) that we can work together on, and eventually merge back into the main Lumberyard branch (after some refactoring/modifications, optimizations, etc.).Once we have something working, then maybe we can work together with the Lumberyard devs and try to "refactor" the code changes/fork back into a mainline Lumberyard branch (especially if you feel that there would be no way to create "gems" or "modules" that we could just add/remove to support these 64-bit precision).Would be interesting to start a research group of like-minded engine nuts. Not sure though if there would be actually some folks with enough engine knowledge out there, that would be up for it.I completely agree. It doesn't seem to be a "priority" to the current Lumberyard Devs. I supposed we can create a separate "Research Group" and begin working on this Lumberyard project. Maybe we can get a few former Crytek engine devs, and a few CIG Devs to help contribute to the project as well (especially Sean Tracy from Crytek/CIG), and hopefully (since CIG uses Lumberyard anyways) they'd be willing to help contribute some of their existing Lumberyard code changes/modifications to our research project "Lumberyard Fork", and then once we can get the code modifications and Tools done, then maybe it can be refactored/integrated back into Lumberyard main branch.I've been pushing for this for quite some time now. I'm definitely willing to help, and if we start the project (as a Lumberyard fork), maybe I can try to get a few CIG devs, a few former Crytek devs (that are familiar with CryEngine), and a few Lumberyard Devs to "help us out" and help out with the modifications/refactoring/implementation and eventually get this completely refactored, optimized and integrated back into the main Lumberyard Dev branch.It might take some time, but I think this is a good thing to eventually add back into the main Lumberyard branch. I think this is something a LOT of game developers need (not just CIG). Unfortunately we can't seem to get the Lumberyard Devs to do anything about it, or do it. It seems to be "very low priority" (to the Amazon Devs) but hopefully if we start the project, it can "pick up steam" and we can get a few more devs to help us out with it and make it a reality.SharerePost-User-9149768answered 5 years ago"


Question: Enable detailed monitoring for EC2 instance

Does enabling detailed monitoring for an EC2 instance provide memory usage metric?FollowComment

Answer: "1No, enabling detailed monitoring will just provide data in 1-minute periods. It does not include metrics on memory usage. The available CloudWatch metrics for the basic monitoring (5-minute period) and detailed monitoring (1-minute period) can be found here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html.If you need to view memory usage metrics, you need to use a CloudWatch Agent. More details here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html and https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html.CommentSharejoahnaanswered 9 months ago0By default, Amazon EC2 metrics related to CPUUtilization, a set of disk Read and Write metrics, and a set of NetworkIn and NetworkOut can be fetched.But, EC2 doesn't provide metrics related to OS-level memory usage or disk usage metrics.You will need to configure custom metrics for memory. Please follow steps in this article to configure memory metrics.https://aws.amazon.com/premiumsupport/knowledge-center/cloudwatch-memory-metrics-ec2/#:~:text=By%20default%2C%20Amazon%20EC2%20delivers,usage%20or%20disk%20usage%20metrics.CommentSharerePost-VivekPophaleanswered 9 months ago"


Question: Build error on『MLOps and integrations』

"Hello, I have a question about one of the Workshops published in the AWS Workshop studio.I am going through the "MLOps and integrations" hands-on, but it is not working well during the process.Here is the URL for the hands-on.Specifically, the build process performed when deploying the provided CloudFormation template will generate an error occurs.I think that the URL for S3 listed in the distributed source(source/test.py) in the workshop is probably not valid.Do any of you know how to solve this problem?----Excerpts from the CodeBuild error log-----Traceback (most recent call last):File "source/test.py", line 20, in <module>test_file,File "/usr/local/lib/python3.6/site-packages/wget.py", line 526, in download(tmpfile, headers) = ulib.urlretrieve(binurl, tmpfile, callback)File "/usr/local/lib/python3.6/urllib/request.py", line 248, in urlretrievewith contextlib.closing(urlopen(url, data)) as fp:File "/usr/local/lib/python3.6/urllib/request.py", line 223, in urlopenreturn opener.open(url, data, timeout)File "/usr/local/lib/python3.6/urllib/request.py", line 532, in openresponse = meth(req, response)File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response'http', request, response, code, msg, hdrs)File "/usr/local/lib/python3.6/urllib/request.py", line 570, in errorreturn self._call_chain(*args)File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chainresult = func(*args)File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_defaultraise HTTPError(req.full_url, code, msg, hdrs, fp)urllib.error.HTTPError: HTTP Error 403: ForbiddenFollowComment"

Answer: "0Hi , Have you created your user token from github, as indicated in the 'Set up your environment' part of the workshop and then input the token on the "Create the Cloud Formation stack" config?  - (Create Github token - https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)CommentShareEXPERTLorea-AWSanswered 10 months agorePost-User-9533151 10 months agoThank you for your comment!  I had issued and set up the Github token as per the instructions. However, I would like to check the 'Set up your environment' section again.SharerePost-User-9533151 10 months ago'Set up your environment' was checked again and implemented, but without success. However, when I rewrote the S3 URL in "source/test.py" to my own URL where I uploaded the MNIST test image, it worked and the build completed. For some reason, the distributed resource did not seem to be valid.Share"


Question: EC2 Autoscaling

"I have a question. I'm running my entire infrastructure in EC2. If I autoscale it 1st step is to create an AMI. But after taking creating an image. How should I capture and save my live data from user. Autoscaling is working fine but how should I save the data. I'm new to it. Please, If anyone have an idea please help.FollowComment"

Answer: "0Hi Devashish RokshitMy understanding is you want to capture and save the live data.What you want to know is how to save the data.I am quite not sure about the quesiton. But let me assume there are two scenario;Scale-out event, new instance will have empty data. You need to fill the instance with the data.Scale-in event, existing instance will be removed. You need to save the data before the instance terminated.Either scenario, the best approach is using EFS.EFS is storage living on network (VPC). you can mount EFS drive on multiple instance under Autoscaling group. This way, you don't have to worry about scaling event.Check this page to learn morehttps://docs.aws.amazon.com/efs/latest/ug/how-it-works.htmlCommentShareEXPERTResaanswered 3 months ago0EFS as described above would definitely work but it may be costly compared to EBS volumes. You can also take advantage of lifecycle hooks. You can define a specific procedure before scaling-out ( copy data to another storage solution or S3 bucket ). You can have this lifecycle hooks for scale - in or out meaning that you can even copy data or run a script before scale-in and before your instance comes in-service.Summary of how it works:An Amazon EC2 instance transitions through different states from the time it launches until it is terminated. You can create lifecycle hooks to act when an instance transitions into a wait state.The lifecycle hook puts the instance into a wait state (Pending:Wait) and then performs a custom action.https://docs.aws.amazon.com/autoscaling/ec2/userguide/lifecycle-hooks-overview.htmlCommentSharemojtothanswered 3 months ago"


Question: Unable to delete user with special character

"Hi there,I have some users in my database with special characters which can't be deleted, any idea how could I do it?dwh_sch=# select * from pg_user where usename like 'jenn%' ;usename     | usesysid | usecreatedb | usesuper | usecatupd |  passwd  | valuntil | useconfig--------------------------------------------------------------------------------+-----------jenni.tulamito |      125 | f           | f        | f         | ******** |          |(1 row)dwh_sch=# drop user jenni.tulamito ;ERROR:  syntax error at or near "."LINE 1: drop user jenni.tulamito ;ThanksFollowComment"

Answer: 0Accepted AnswerWrap the username in double quotes.CREATE USER "test.user" PASSWORD 'Testing123';-- CREATE USERselect * from pg_user where usename like 'test%';--   usename  | usesysid | usecreatedb | usesuper | usecatupd |  passwd  | valuntil | useconfig-- -----------+----------+-------------+----------+-----------+----------+----------+-------------  test.user |      121 | f           | f        | f         | ******** |          |DROP USER "test.user";-- DROP USER select * from pg_user where usename like 'test%';--  usename | usesysid | usecreatedb | usesuper | usecatupd | passwd | valuntil | useconfig-- ---------+----------+-------------+----------+-----------+--------+----------+----------- CommentShareAWS-User-2965899answered 3 years ago


Question: Migrating from Cisco UCCX to AWS

"Hi TeamCustomer would like to migrate from Cisco UCCX to AWS Connect. I need some clarifications and understanding on below topicsCurrently they are having DID numbers dispatched on the Cisco Gateway through the service Providers ISDN/SIP, is it possible to use the same DID numbers on the AWS connect? If yes how we can achive that? I would appreciate if anyone ould share some documents related to the process and configurations.Eg - Customer current did 1234 100 - 1234 199 , these numbers are provided by local service providers how to get these calls from Cisco gateway to AWS connect.How many calls can be in the queue? How to set this?In Cisco UCCX we have IVR ports which meanseg - If the customer has 50 IVR ports 50 customers can call the hotline simultaneosly and can be in the queue (before answered by any agents) ?In Amazon connect how many customers can call simultaneosly? lets assume if 100 customers can call simultaneosly is it chargeble or free?I do not see any these charges information on https://aws.amazon.com/connect/pricing/?p=ft&c=rs&z=2FollowComment"

Answer: "1Porting Numbers:Primarily you need to raise a support ticket with AWS with the LOA to port the numbers to Amazon Connect.Please refer the section "Porting numbers purchased from other contact center providers" in the document.https://docs.aws.amazon.com/connect/latest/adminguide/what-numbers-can-be-ported.htmlThere is a soft limit of 10 Concurrent calls per instance and can be increased by raising a service quota request form.Please refer the document for opening the support request:https://docs.aws.amazon.com/connect/latest/adminguide/amazon-connect-service-limits.htmlCalls will be charged as per the usage /pricing  (minutes), but not by concurrency.CommentSharePhani_Lanswered a year agoPragash Venugopal a year agoThank you Phani_L for your time and answeres.With related to the answeres on the 2nd question, If that raised by service quota request form does that chargable?Share0I've seen customers do different things. First, is to port the numbers to AWS like it was mentioned. The other is to have the telco provider forward calls to a new dummy number which AWS owns. This way you can control your main DIDs if you ever need to do any sort of DR or alternative routing.There's no charge for creating a service request to increase your limit.I have a lot of UCCX experience, make sure you do a deep dive into Connect works to avoid any surprises. Because UCCX is so tightly coupled with UCM you can do some cool telephony things when it comes to transfers/outbound calls, but this is not the case in Connect.CommentSharedmaciasanswered a year ago"


Question: Unexpected DataTransfer-Regional-Bytes traffic

"Hello,I have a few EC2 instances in East-1 and East-2 regions.East-1 instances are all in the same availability zone (us-east-1a). Most Nano, one Small.While trying to better understand usage costs, I noticed a significant part of it is 'DataTransfer-Regional-Bytes' in my Cost Explorer when filtering by East-1 Region. As far as I know, this shouldn't be possible as all traffic within the same availability-zone should be free.Any advice on how I could find out what could be causing these unexpected data transfer charges?Thanks in advance.FollowComment"

Answer: "3Accepted AnswerDo you use any other services? Like S3? And your EC2 instances are communicating with S3 buckets in us-east-1 region? If so, make sure that you're using Gateway VPC endpoints for S3.Otherwise, when you go to Cost Explorer, and apply filter "Usage Type" - and search for %Regional-Bytes%, and then Group your result by Service. Are these charges associated only/mostly with EC2?Billing tools (even most detailed AWS Cost and Usage Reports with resource IDs) only can show information about resources that generate the charges, however they won't be able to show exact information about source or destination of the traffic depending on the traffic type. For example, for data transfer OUT you can see the resource IDs (in CUR) that traffic is coming from, but you can't see where exactly that traffic goes; and for data transfer IN you can see the resource that's accepting ingress traffic, but you can't see which exact resource it's coming from.[ADDITION] - if all instances are in the same AZ, but they are using Public IP (like Elastic IP) to communicate, this will be the most likely cause of generated DataTransfer-Regional-Bytes charges. Instances in the same AZ that communicate within private network (using private IPs) within AWS, have no data trasnfer charges. However, if with the same setup you use public IP, you will see data transfer charges associated with those resources. Check in Cost Explorer - filter DataTransfer-Regional-Bytes usage type, and group by API Operation. The result might show operations like "public IP" or something similar in this case.If this is mostly for EC2, your best bet could be using VPC Flow Logs - they may actually show the source/destination of traffic, including ports, IPs, etc. Note though, that flow logs can get quite expensive if you leave them turned on for long period or time and you are tranferring large volumes of data, and they also won't show what happened in the past before you turned them on. If you don't use them regulrly, you can turn them on for a period of time for troubleshooting, and turn them off later.CommentShareEXPERTNataliya_Ganswered 4 months agoEXPERTTushar_Jreviewed 4 months agoDave 4 months agoThanks for the reply. The only services am using are EC2, no S3. Cost explorer grouped by Service + Regional-bytes filter shows 100% of the data/charges as EC2-other. I'll give VPC Flow Logs a shot.ShareNataliya_G EXPERT4 months agoAh, one more thing - if all instances are in the same AZ, are you using Public IP on them to communicate with each other? Like... Elastic IP for example? If so - this will be the cause of generated DataTransfer-Regional-Bytes charges. If instances are in the same AZ, and they communicate within private network (using private IPs) within AWS, then that traffic is free. However, if with the same setup you use public IP, you will likely see data transfer charges associated with those resources. Check one more thing in Cost Explorer - when you filter DataTransfer-Regional-Bytes usage type, can you group by API Operation? If the results will show the operations like "public IP" or something similar, then my last theory is right :)ShareDave 4 months agoThat's it, it's all PublicIP-In / PublicIP-Out usage. It wasn't obvious to me from the documentation so thanks for pointing that out.Share"


Question: AWS SNS FCM/GCM "direct_boot_ok" property support for push notifications

"I utilize AWS SNS for sending push notifications to an Android application. I want to be able to send push notifications to the android app immediately after device boot and before the security pin has been entered. The capability to do this is fully supported by FCM/GCM by adding an additional property to the push notification request:{"direct_boot_ok": true}See: Firebase documentation for more details on this property.I am able to successfully trigger this capability when using a REST client to send the push notification, but not when sending the same message payload directly though AWS SNS. Is it possible to have this functionality supported with AWS SNS?FollowComment"

Answer: "0Dear Customer, thank you for reaching out us regarding the above query.Kindly note that unfortunately this attribute is not supported with AWS SNS. However, in order to send push notifications to the android app immediately after device boot and before the security pin has been entered, try the following payload on the Pinpoint consoleGo to Pinpoint > Project > Test messagingSelect Push notifications, and enter the Device tokenSelect Push notifications service as FCMUnder 'Message', select 'Raw message' and use the following payload:{ "GCMMessage": {     "notification": {         "body": "Test PinPoint",         "title": "Test PinPoint",         "direct_boot_ok": true     } }}In case you are facing any challenges or any error messages, I would request you to kindly provide a test payload or the SDK/CLI call (with the complete payload) that you are trying, along with any error message or the behaviour observed at your end. Please feel free to add any follow-up query you may have.CommentSharerePost-User-7792115answered 9 months ago"


Question: "EC2 unresponsive ,status check failed , not reaching via ping/ssh"

"Ec2 is not unresponsive after the force shut down and the normal start of ec2 instances.When starting, the Status check failing and the below logs are seen via Get system log option on aws console:Now as there is no way to access to ec2, how can I resolve it?----[    3.766101] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11[    4.210518] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0[    4.862401]  nvme0n1: p1 p2[    4.876284] random: nonblocking pool is initializedBegin: Loading essential drivers ... done.Begin: Running /scripts/init-premount ... done.Begin: Mounting root file system ... Begin: Running /scripts/local-top ... [    4.929843] device-mapper: uevent: version 1.0.3[    4.931134] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised: dm-devel@redhat.comdone.Begin: Running /scripts/local-premount ... done.Begin: Will now check root file system ... fsck from util-linux 2.25.2[/sbin/fsck.ext4 (1) -- /dev/nvme0n1p2] fsck.ext4 -a -C0 /dev/nvme0n1p2 /dev/nvme0n1p2 contains a file system with errors, check forced./dev/nvme0n1p2: Inodes that were part of a corrupted orphan linked list found.  /dev/nvme0n1p2: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.(i.e., without -a or -p options)fsck exited with status code 4done.Failure: File system check of the root filesystem failedThe root filesystem on /dev/nvme0n1p2 requires a manual fsck[    5.052672] ACPI: bus type USB registered[    5.053800] usbcore: registered new interface driver usbfs[    5.055203] usbcore: registered new interface driver hub[    5.056611] usbcore: registered new device driver usb[    5.058820] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver[    5.060950] ehci-pci: EHCI PCI platform driver[    5.065352] uhci_hcd: USB Universal Host Controller Interface driver[    5.068489] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver[    5.071678] hidraw: raw HID events driver (C) Jiri Kosina[    5.074091] usbcore: registered new interface driver usbhid[    5.075511] usbhid: USB HID core driverFollowComment"

Answer: "0Are you able to use the EC2 Serial Console. This provides a way to troubleshoot boot and network connectivity issues by establishing a connection to the serial port of an instance. This access can be used for interactive troubleshooting.The AWS article Why is my EC2 Linux instance unreachable and failing one or both of its status checks? may be of useCommentShareRoBanswered a year ago0Yes, I am able to login using serial port , but no idea how to fix this thing. If I run the command manually on serial console then i get the same error.Begin: Running /scripts/local-premount ... done.Begin: Will now check root file system ... fsck from util-linux 2.25.2[/sbin/fsck.ext4 (1) -- /dev/nvme0n1p2] fsck.ext4 -a -C0 /dev/nvme0n1p2 /dev/nvme0n1p2 contains a file system with errors, check forced./dev/nvme0n1p2: Inodes that were part of a corrupted orphan linked list found.  /dev/nvme0n1p2: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.(i.e., withou[   13.089321] ACPI: bus type USB registeredt -a or -p optio[   13.158239] usbcore: registered new interface driver usbfsns)fsck exited[   13.160407] usbcore: registered new interface driver hub with status cod[   13.163360] usbcore: registered new device driver usbe 4done.Failure: File system check of the root filesystem failedThe root filesystem on /dev/nvme0n1p2 requires a manual fsck[   13.265290] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver[   13.357764] ehci-pci: EHCI PCI platform driver[   13.364183] uhci_hcd: USB Universal Host Controller Interface driver[   13.368666] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver[   13.373432] hidraw: raw HID events driver (C) Jiri Kosina[   13.458440] usbcore: registered new interface driver usbhid[   13.460332] usbhid: USB HID core driver(initramfs) Failure: File system check of the root filesystem failed/bin/sh: 1: Failure:: not found(initramfs) fsck -r -Vfsck from util-linux 2.25.2```CommentShareAWS-User-1149198answered a year ago"


Question: Amazon Textract server side encryption

"When we are talking about encryption in Amazon Textract, there are following questions:Who chooses which encryption method is selected?If no method is selected, is there any encryption in the Amazon S3 storage?Source: https://docs.aws.amazon.com/textract/latest/dg/encryption.htmlThanksFollowComment"

Answer: "0Hi,In general, customer (security team, engineer team) should choose which encryption method to use according to their requests. AWS provides multiple exclusive options, and recommend to ensure your service encrypt at rest, encrypt at transit.For S3, by default, the encryption is disabled. You can enable it by choosing either Amazon S3-Managed Keys (SSE-S3) or AWS Key Management Service (SSE-KMS). In summary, the former option use the key managed by AWS, which reduce the overhead to configure KMS key. The latter you have the flexibility to create and manage your KMS key.More details:https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.htmlhttps://docs.aws.amazon.com/kms/latest/developerguide/services-s3.htmlThanks,CommentShareJadyanswered 5 months ago"


Question: Why is Auto Scaling Group not taking Security Group from Launch Template?

"First created a Launch Template with Auto Scaling Guidance (no Networking info except providing Security Group)Next created Auto Scaling Group in the console. As the launch template is selected in the first step, it can be seen that the security group IDs is just a dash (-), which indicates that the Auto Scaling group is not taking the Security Group from the Launch Template.Moreover, there is no way in this screen to override/correct the security group.Confirmed this when the Auto Scaling Group creates the instances - those instances have the default security group instead of the one specified in the Launch Template.Could this be a bug or a pebcak problem?FollowComment"

Answer: 0Accepted AnswerEdit: Looks like it took some time for the security groups to get reflected in the instances.CommentShareAWS-User-5154567answered a year ago


Question: Using SCIM with AWS IAM as IdP

"Hi everyone,during my research, I found a lot of documentation that describes how you can use another IdP, such as Azure Active Directory, to synchronize identities from that IdP to AWS IAM. But what about the other way around? Is it possible to use AWS IAM as my central IdP, and then synchronize users stored in AWS to another service provider? Azure for example provides that functionality, I can create an Enterprise Application for user provisioning and then select any arbitrary target endpoint that receives the SCIM requests to synchronize users between AAD and another application. Does AWS provide something comparable?Thanks in advance!FollowComment"

Answer: "0Have you looked into IAM Identity Center (successor to AWS Single Sign-On)? This service is has more centralized capabilities vs AWS IAM.Check out this list of applications that already has built integrations with IAM Identity Center. These integration I know does support Federation between AWS and those service providers, but as far as SCIM integration, you would likely need to implement a custom solutions. For example :You can use AWS Lambda or another compute service to create a script that periodically exports IAM user data and transforms it into the appropriate format, such as SCIM, for the target service provider. Then, you can set up an API to receive the transformed data in the target service provider.Keep in mind that this custom solution will require ongoing maintenance and updates to ensure compatibility and security.See the official documentation for AWS IAM Identity Center for more detailsCommentSharesamwale-awsanswered 2 months ago"


Question: User: anonymous is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:us-east-1:**xxxx

"I have a lamdba function at account A that requests a private API at account B. There is a VPC Peering between VPC account A and VPC account B.At account A I created a VPC endpoint (com.amazonaws.us-east-1.execute-api).The API Gateway at account B, was created as Private, bonded to VPC at account B, created resource and method without any type of authentication and the method points to a Lambda function (account B) that does an insert on QLDB table.The lambda is configured as proxy.When I execute the test of API Gateway (account B), it executes with sucess the lambda function and inserts a document at QLDB table.When I execute the lambda (at account A) requesting the API Gateway, I get this error message:User: anonymous is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:us-east-1:**xxxx I've been trying to overcome this issue without success.Thanks in advance,Fernando PossebonFollowComment"

Answer: "0Make sure that the resource policy on the API lists both VPC Endpoints. For an example see: Use the AWS CLI to associate VPC endpoint with a private REST APICommentShareEXPERTkentradanswered 6 months agopossebon 6 months agoHi kentrad, thanks for your post.I changed the resource policy to this, deploy the API and I'm still getting the same error message.{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Principal": "","Action": "execute-api: Invoke","Resource": "execute-api:vpce-accountb.execute-api.us-east-1.vpce.amazonaws.com///"},{"Effect": "Allow","Principal": "","Action": "execute-api:Invoke","Resource": "execute-api:vpce-accounta.execute-api.us-east-1.vpce.amazonaws.com///"},{"Effect": "Allow","Principal": "","Action": ["logs:CreateLogGroup","logs:CreateLogStream","logs:DescribeLogGroups","logs:DescribeLogStreams","logs:PutLogEvents","logs:GetLogEvents","logs:FilterLogEvents"],"Resource": ""}]}Sharekentrad EXPERT6 months agoThese vpc endpoints should be listed in the condition section of the resource policy attached to the API. See: https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-resource-policies-examples.html#apigateway-resource-policies-source-vpc-exampleShare"


Question: Transfer to queue question

"We have only one agent andWhen the call get transferred to queue and the agent doesn't answer the call and the agent state goes to "Missed" and the call stays there for ever. How can we get the call back and send it to the voicemail.We are using desk phones, one option is reduce the ring time no answer to less than 20. But I want to make sure if there is any other option?FollowComment"

Answer: "0For the described use case, you can implement the voicemail solution that Amazon Connect offer. This voicemail solution is made up of many different AWS services which includes Lambda, Amazon Kinesis Data Streams, Amazon DynamoDB, Amazon S3 and Event streams to process, transcribe and deliver these transcripts and audio recordings based on the agent’s delivery preference. It will require some additional setup that includes deploying the necessary Cloudformation templates that can be found in the voicemail for amazon connect repository [1].The configured voicemail solution would allow your customers to call into your call center and leave a voicemail for an agent by choosing the agent’s extension and then these voicemails recordings and transcripts are delivered to the agent through SMS and email [2].Please check the referenced documents below for more information and steps on how to implement the voicemail solution.Document Reference:[1] https://s3.amazonaws.com/solutions-reference/voicemail-for-amazon-connect/latest/voicemail-for-amazon-connect.pdf[2] https://aws.amazon.com/solutions/implementations/voicemail-for-amazon-connect/[3] https://github.com/amazon-connect/voicemail-for-amazon-connectCommentSharerePost-User-9848202answered 14 days ago"


Question: "Glue supports the iceberg framework and uses the MERGE INTO syntax, but it can't succeed."

"At present, Glue supports the iceberg framework, but the MERGE INTO syntax is needed here. I set it according to the official website's article, but I can't succeed. There will always be an error of "unable to get the table".How can I solve it?FollowCommentGonzalo Herreros 21 days agowhat's the stacktrace?Share"


Question: Trouble with Websockets on elastic beanstalk node js envirom

"Hello everybody,I am currently trying to integrate Websocket into my server running on an AWS Beanstalk NodeJS environment. Unfortunately I get the error message in the browser:WebSocket connection to 'wss: // ***' failed: Error during WebSocket handshake: Unexpected response code: 404Local (localhost) everything is running fine.On the server side, I log the calls and see that the value "upgrade" has not been transferred in the header.In the Beanstalk configuration, I have a Classic Load Balancer deposited with the following configuration:Port 443Protocol SSLInstance port 80Instance protocol TCPSSL Certificate: ...In my deployment package is next to the server also the following file: .ebextensions / WebSocketsUpgrade.confwith the following content:container_commands:enable_websockets:command: |sed -i '/\sproxy_set_header\sConnection/c \proxy_set_header Upgrade $http_upgrade;\proxy_set_header Connection "upgrade";\' /tmp/deployment/config/#etc#nginx#conf.d#00_elastic_beanstalk_proxy.confHave I missed something that needs to be configured?Thanks a lot!FollowComment"

Answer: "0Hello, the problem is solved. It is an incorrect name of the config file, which has been adopted in many forums. The file name must be: WebSocketsUpgrade.config.CommentShareakoranswered 4 years ago"


Question: Migrate SQL Server DB (outside of AWS) to Aurora DB in AWS

"I've created an empty Aurora DB inside AWS.I backed up my SQL Server DB and upload that backup to an S3 bucket.I thought I might be able to import that SQL Server backup either into my Aurora DB or into a SQL Server instance in AWS then migrate that to Aurora...  I can't see how to do this. I've tried creating an endpoint, thinking I could use that to link to the SQL Server backup file but can't see a way.Is this possible or should I use a different approach?FollowComment"

Answer: "0Accepted AnswerYou can not just migrate a backup from Sql Server to Aurora. There are two optins you can consider1/ Use SCT and DMS to migrate your Sql Server database to Aurora.There are two parts for heterogeneous migration, Schema conversion and data migration.SCT ( Schema Conversion Tool)  as the name implies is a tool to convert your database objects from  one db engine to another and then you can use DMS to migrate your data from Sql Server to Aurora.https://aws.amazon.com/dms/schema-conversion-tool/https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html2/ Use babelfish endpoint and then use dms to migrate your data into Aurora.https://aws.amazon.com/blogs/database/migrate-from-sql-server-to-amazon-aurora-using-babelfish/I hope it helps.** Please make sure to accept the answer **CommentShareDevinder-theDBGuyanswered 10 months agoneohed 10 months agoThanks for the help. I'm trying the first approach. The SCT has connected to my SQL Server DB. I can't connect to my Aurora DB because it asks for an Amazon Aurora (PostgreSQL compatible) driver. I've looked around but cannot find this. Do you know how to download this?Thanks,ShareDevinder-theDBGuy 10 months agoPl take a lookhttps://docs.aws.amazon.com/dms/latest/oracle-to-aurora-postgresql-migration-playbook/chap-oracle-aurora-pg.tools.awssct.htmlor download the driver from herehttps://jdbc.postgresql.org/download.htmlShareneohed 10 months agoGreat, thanks. I'm making progress, very very slowly - been at this all day. In SCT I'm on the final tab "Choose a target" I don't know what username and password I'm supposed to use. The only login credentials I have are the ones I login to AWS portal with. RDS Management Console does not let me create any users for my Aurora DB. I did associate it with IAM credentials and created an IAM user, but that has no password that I can see, only an access key that SCT doesn't seem to care about...ShareDevinder-theDBGuy 10 months agoUse the credentials for the admin user that was created when you provisioned the aurora cluster.Share"


Question: Tracking record-level failure from Databrew validation

"Hi,This is question with Databrew data profile capability. I am looking for record-level tracking per rule validation status. This will provide while rules has failed on a record from the dataset.At least, I am looking to separate records after data profiling, valid vs invalid records.Thanks,FollowComment"

Answer: "0Hello,this functionality is currently not supported, I will provide feedback to the team owning the service.Today, to solve your issue you could use a implement in a project recipe the same logic in the quality check you are interested in and using a conditional transform associate a value (true/False or 1/0) to a new column isCol1Valid .when you run the processing job and write the output to your target you could easily find all rows that are not satisfying the specific check.I am aware this is currently a workaround.hope it helpsCommentShareEXPERTFabrizio@AWSanswered a year ago"


Question: [AWS EMR] HTTP 403: Forbidden (Workspace is not attached to cluster]

"Hi everyone,I am a bit new to AWS so pardon me on this. I am trying to run Spark on a notebook in EMR. I have created a cluster to achieve this effect. However, when I tried to select Pyspark or Spark as the kernel, the aforementioned error pops out.I have tried creating workspaces in EMR Studio but it doesn't seem to solve the problem.The next thing I have tried was to try other versions of EMR Release versions. The ones that have succeeded areemr-6.0.0 (JupyterHub 1.0.0, Hive 3.1.2, Hue 4.4.0, Livy 0.6.0, Spark 2.4.4)emr-5.18.0 (Hive 2.3.3, Pig 0.17.0, Hue 4.2.0, JupyterHub 0.8.1, Spark 2.3.2)emr-5.2.3 (Hive 2.1.0, Pig 0.16.0, Hue 3.10.0, Spark 2.0.2)However, the following failed and gave the error as shown in the picture.emr-6.3.1 (JupyterHub 1.2.0, Hive 3.1.2, Pig 0.17.0, Hue 4.9.0, Spark 3.1.1, JupyterEnterpriseGateway 2.1.0)emr-5.36.0 (Hive 2.3.9, Pig 0.17.0, Hue 4.10.0, JupyterEnterpriseGateway 2.1.0, JupyterHub 1.4.1, Spark 2.4.8)What could be the issue behind this?Thanks so much everyone!FollowComment"

Answer: "0If the notebook is attached to cluster and if you are still encountering this error, it probably means notebook is unable to communicate to EMR Cluster's JupyterEnterpriseGateway(JEG) endpoint.Please make sure your EMR cluster / notebook security groups are not restrictive.https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-security-groups.htmlyou may check logs on your EMR cluster on /var/log/applications/Jupyter*Unfortunately, you will not able to see logs on EMR notebooks as its managed by EMR service.if this is working differently between versions, Please make sure you review theseEMR Studio Considerations :https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-considerations.htmlEMR Notebook Considerations :https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-considerations.htmlCommentShareMannem Ranswered 5 months ago"


Question: How to dynamically update the policy of user(Cognito identity) from backend/lambda?

"I am building an IoT solution using the IoT Core.The end-user will be using Mobile App and will be authenticated and authorized using Cognito.I want to authorize users to allow iot:Publish and iot:Subscribe action only on the devices that the user owns.The IAM Role attached to the Cognito Identity pool has only iot:Connect permission when the user is created. The User won't have any additional permission at this point.{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "iot:Connect"            ],            "Resource": "arn:aws:iot:us-east-1:1234567890:client/${cognito-identity.amazonaws.com:sub}"        }    ]}Now, when the user finishes the device provisioning, I want to attach the inline Policy to Cognito identity of that user to authorize him to publish and subscribe to the shadow of that device. Let's assume the ThingName is Thing1 so the policy should be as below:{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "iot:Connect"            ],            "Resource": "arn:aws:iot:us-east-1:1234567890:client/${cognito-identity.amazonaws.com:sub}"        },        {            "Effect": "Allow",            "Action": [                "iot:Publish",                "iot:Subscribe"            ],            "Resource": "arn:aws:iot:region:account-id:topic/$aws/things/Thing1/shadow/*"        }    ]}The user may keep adding new devices and I want to scale this policy to include resource ARNs of those devices.This is an example of IoT Core, but my question is very generic to IAM policies. (e.g. the same can be applied to dynamically allow access  to the S3 bucket folders)So, here is my question:What is the best approach for dynamically adding or removing the inline policy granted to the Cognito identity?Can I use the STS service for updating/attaching the policy on my backend/Lambda when new Things are added or removed?Note:I can use the Customer Managed Policy, but it is not the right approach for granting policies to federated users as per my knowledge.I know I can use the intelligent naming of the device as mentioned in this approach. But, I have a very basic requirement.https://aws.amazon.com/blogs/iot/scaling-authorization-policies-with-aws-iot-core/FollowComment"


Question: "Grouping Behavior for S3 Data" on Crawler does not work.

"I have a s3 bucket with this following path:s3://a/b/c/productsand inside the products folder I have 2 subfolders called version_0 and version_1. Each version is a "database snapshot" for the products table.What we needed is to create a crawler that merges the two versions together and adds it to the glue gatalog. So in the crawler properties we have set the s3 bucket path to be the one mentioned above, and then the Output options we have this:I already tried 3 different configurations:Box ticked "Create a single schema for each S3 path" and "Table Level" emptyBox ticked "Create a single schema for each S3 path" and "Table Level" set to 4.Box ticked "Create a single schema for each S3 path" and "Table Level" set to 5.Nothing worked and the crawler always created 2 tables (one for each version).I followed this documentation: https://docs.aws.amazon.com/glue/latest/dg/crawler-configuration.html#crawler-table-levelFollowCommentFabrizio@AWS EXPERT10 months agoHi, do the 2 versions of the table have a similar schema or are there many differences?Share"

Answer: "0There are four reasons why a crawler would create separate tables:The source files might not be having the same type of file (CSV, parquet or JSON)Please check whether the source files in your given folder are of same type or not.The source files might be of different compression types (snappy, gzip,  bzip2)Make sure whether the compression types are of the same type for your source files.The source files might not be having the same schema.As said in the earlier answer, for the crawler to detect a single schema for both the folders, the threshold of 70% must be met. That means, the similarity of schema between both the sources must be atleast 70%.The structure of Amazon S3 storage partitions are different for both of the files.*I do not think this might be an issue because the structure of S3 partitions is almost the same. But please do check this.You can know the exact reason behind your crawler creating multiple tables by checking through your crawler logs. You can login to your console -> select your crawler -> choose logs to view the logs of your crawler.For more details please follow this article.CommentShareSUPPORT ENGINEERChaituanswered 9 months agorePost-User-4108613 9 months agoHi Chaitu, Thank you for your answer.I have checked everything and all seems good...My S3 structure is as follows:s3://a/b/c/products/version_0_temporary0_$folder$part-00000-c5... ...c000.snappy.parquet/version_1_temporary0_$folder$part-00000-29... ...c000.snappy.parquetWhat is making the crawler create multiple tables in this case?The schemas are exactly the same. Tried with Table Levels: 4 5 and 6.Nothing worked...Share0For the schemas to be defined as one schema, there needs to be similarity and a threshold limit.See the examples in the below reference link that talk about partition threshold higher than 70%. In your case, I am assuming there are only 2 schemas for each version that you have.The crawler infers the schema at folder level and compares the schemas across all folders. If the schemas that are compared match, that is, if the partition threshold is higher than 70%, then the schemas are denoted as partitions of a table. If they don’t match, then the crawler creates a table for each folder, resulting in a higher number of tables.Reference : https://aws.amazon.com/premiumsupport/knowledge-center/glue-crawler-detect-schema/CommentShareAnanth Tirumanuranswered 10 months ago"


Question: Kajal is not added as a valid VoiceId in Amazon Polly API Action SynthesizeSpeech

"https://docs.aws.amazon.com/polly/latest/dg/API_SynthesizeSpeech.html#API_SynthesizeSpeech_RequestSyntaxHi,If you will see the valid voices, it does not contain Kajal as one of the valid VoiceId. This is causing our API to fail when trying to synthesize speech for the neural voice Kajal (Indian English, Female). Please find below list of valid VoiceId currently:Aditi | Amy | Astrid | Bianca | Brian | Camila | Carla | Carmen | Celine | Chantal | Conchita | Cristiano | Dora | Emma | Enrique | Ewa | Filiz | Gabrielle | Geraint | Giorgio | Gwyneth | Hans | Ines | Ivy | Jacek | Jan | Joanna | Joey | Justin | Karl | Kendra | Kevin | Kimberly | Lea | Liv | Lotte | Lucia | Lupe | Mads | Maja | Marlene | Mathieu | Matthew | Maxim | Mia | Miguel | Mizuki | Naja | Nicole | Olivia | Penelope | Raveena | Ricardo | Ruben | Russell | Salli | Seoyeon | Takumi | Tatyana | Vicki | Vitoria | Zeina | Zhiyu | Aria | AyandaFollowComment"

Answer: "0Accepted AnswerLooks like we are not passing neural as an engine for Kajal, which is why it is failing. Still, you guys can fix the voiceId supported docs page.CommentShareprakharjain17answered 6 months ago"


Question: DOWNLOAD_SOURCE => Failed => YAML_FILE_ERROR: YAML file does not exist

"Hi,I have set up the codepipeline. I'm experiencing build failure details are 'DOWNLOAD_SOURCE => Failed => YAML_FILE_ERROR: YAML file does not exist' .Can you clarify the need of explicit yaml file in the code level as I already defined my settings at the time of creating 'CodePipeine' please ?Thanks,Krish.FollowComment"

Answer: "0Hi Krishnan,CodePipeline is a continuous delivery orchestration service, it does not run builds itself but instead delegates that work to CodeBuild, which looks for the buildspec.yml to determine how to build your code.You have the choice of either creating a buildspec.yml in your source repository, or editing the CodeBuild Project attached to your pipeline and specifying the buildspec.yml inline.CommentShareAWS-User-6139167answered 3 years ago"