I have a data science project in Python and I wonder how to manage my data. Some details about my situation:

My data consists of a somewhat larger number of football matches, currently around 300000, and it is supposed to grow further as time goes on. Attached to each match are a few tables with different numbers of rows/columns (but similar column formats across different matches).
Now obviously I need to iterate through this set of matches to do some computations. So while I don’t think that I can hold the whole database in memory, I guess it would make sense to have at least chunks in memory, do computations on that chunk, and release it.
At the moment I have split everything up into single matches, gave each match an ID and created a folder for each match with the ID as folder name. Then I put the corresponding tables as small individual csv files into the folder that belongs to a given match. Additionally, I have an „overview“ DataFrame with some „metadata“ columns, one row per match. I put this row as a small json into each match folder for convenience as well.
I guess there would also be other ways to split the whole data set into chunks than match-wise, but for prototyping/testing my code with a small number of matches, it actually turned out to be quite handy to be able to go to a specific match folder in a file manager and look at one of these tables in a spreadsheet program (although similar inspections could obviously also be made from code in appropriate settings). But now I am at the point where this huge number of quite small files/folders slows down the OS so much that I need to do something else.
Just to be able to deal with the data at all right now, I simply created an additional layer of folder hierarchy like „range-0“ contains folders 0-9999, „range-1“ contains 10000-19999 etc. But I‘m not sure if this is the way to go forward.
Maybe I could simply save one chunk - whatever one chunk is - as a json file, but would lose some of the ease of the manual inspection.
At least everything is small enough, so that I can do my statistical analyses on a single machine, such that I think I can avoid map/reduce-type algorithms.
On another note, I have close to zero knowledge about database frameworks (I have written a few lines of SQL in my life), and I guess I would be the only person making requests to my database, so I am in doubt that this makes sense. But in case it does, what are the advantages of such an approach?

So, to the people out there having some experience with handling data in such projects - what kind of way to manage my data, on a conceptual level, would you suggest or recommend to use in such a setting (independent of specific tools/libraries etc.)?
Your arrangement is not bad at all. We are not used to think of it this way, but modern filesystems are themselves very efficient (noSQL) databases.
All you have to do is having auxiliary files to work as indexes and metadata so your application can find its way. From your post, it looks like you already have that done to some degree.
Since you don't give more specific details of the specific files and data you are dealing with, we can't suggest specific arrangements. If the data is proper to be arranged in an SQL tabular representation, you could get benefits from putting all of it in a database and use some ORM - you'd also have to write adapters to get the Python object data into Pandas for your numeric analysis if you that, and it might end up being a superfluous layer if you are already getting it to work.
So, just be sure that whatever adaptations you do to get the files easier to deal with by hand (like the extra layer of folders you mention), don't get in the way of your code - i.e., make your code so that it automatically find its way across this, or any extra layers you happen to create (this can be as simple as having the final game match folder full path as a column in your "overview" dataframe)
How should I handle a data set with around 300000 small groups of data tables?