>

I agree that AI is not quite at the point where it can be creative in the same way that humans are creative, and in its current state, it is really only useful when the human prompts it in the correct way.
Still, my view is that if AI gets to the point where it can formulate hypotheses, develop experimental methods to test the hypotheses, and then carry out the experiments in a simulated environment (which I don't think we're too far off of), then that could be detrimental for humanity from an economic standpoint and from a psychological standpoint. Again, in its current state, I don't think we will be seeing something like this within the next year or so, but in 10-20 years, I could absolutely see this being a possibility.
And if it does become possible, it would mean that AI has the ability to make technological and scientific advancements limited only by computational speed, which would lead to a sort of acceleration in human knowledge that would require our brains to adapt at a rate we are not capable of keeping up with, and who knows where that path would lead. 
Also, I don't think any of us have any good enough understanding of ChatGPT to label it as simply a fancy language model, it is clearly much more than that. It is a versatile system capable of generating genuinely impressive solutions across a wide range of fields and disciplines. It is undoubtedly smarter and more knowledgeable than I will ever be (even considering some of the times when it is blatantly incorrect), so if you want to also call me as a human being just a fancy language model, I guess I'll take that.
You reference machines being creative. However if you look at the chatGPT subreddits, the language model is notoriously terrible at logic and reasoning games. It can’t formulate its own original arguments in order to solve a puzzle. Seems like it isn’t that creative to me. >


I used to think this kind of stuff was BS since AI is no more than matrix multiplication at the end of the day, but as I've seen the power of AI art tools and ChatGPT

AI art tools are still just matrix multiplication and regression (or some similar technique).
And ChatGPT is just a fancy language model, not a compendium of all human knowledge, which is again just matrix multiplication and regression when training.

What I thought was an inherently human characteristic (creativity) has been shown to be no problem for AI.

You've yet to give a solid example of an AI being "creative". Sure, we can train AI to create things we didn't directly instruct them to make by seeding them with starting points and references, but we've yet to make an AI that can come up with its own ideas or apply known ideas to solve problems.
Even for those of us that use AI or machine learning in research, we have to find a specific program that fits the exact requirements for each one of our experiments then train it ourselves and only then do we even stand a chance at getting any useful information out of it. Even with an output that appears significant, it's often practically useless.
So no, we aren't anywhere close to AI being creative enough to come up with, execute, and extrapolate from scientific experiments. We can't even get an AI robot to wander around a room and explore without programming in an explicit "explore" function, and you think they're a few small steps away from evolving past humans and killing humanity? Nah.