I'm building my first ever web-app in python, haven't even decided on a framework yet,
does it make sense to start out with a MySQL database to manage users and credentials?
or is it a completely ridiculous way to approach it?
what are some existing solutions and best practices for managing user credentials?
The answer is, as always, it depends. There are many ways to build a web app in Python so you'll first need to decide on what you want to build or what technology you want to learn. 
If you just want to focus on a Python backend as a learning exercise then you could use Flask which can run as a server and includes many modules to help you get started including managing users. If you plan to expose your app publicly though it is generally recommended to use Apache or some other battle tested server which can route the requests to Flask.
There are other Python frameworks like bottle which I believe is meant to be even simpler than Flask and Django which is more complicated but has more features. It all depends on what you want to do. You can also look at things like dash if you're end goal has a data analysis flavor.
One thing to note though is that managing user credentials is not trivial. It can be a useful exercise if you like to learn and tinker, but to do it correctly youll need to learn about salting passwords, cryptographically secure hashing, session management, https (and ideally which ciphers should be deprecated), how to protect against sql injection (good to know how to do this anyway if you don't already), cross site scripting, CORS, etc. The list goes on. None of these things are exclusively just for managing user credentials but you should understand all the ways things can go south on you.
Managing User Authentication for a web-app with MySQL