I'm assuming that the more commits I make to my database, the more put requests I make. Would it be less expensive to commit less frequently (but commit larger queries at a time)?
I am assuming you're either using RDS for MySQL or MySQL-Compatible Aurora; in either case, you're charged based on the number of running hours, storage and I/O rate, and data transferred OUT of the service (Aurora Serverless pricing is a different story).
In RDS, you're not charged by PUT requests, and there is not such a concept with pymysql.
The frequency of commits should be primarily driven by your application functional requirements, not cost. Let's break it down to give you a better idea of how each cost variable would relate to each approach (commit big batches less frequently vs. commit small batches more frequently).

Running hours: Irrelevant, same for both approaches.
Storage: Irrelevant, you'll probably consume the same amount of storage. The amount of data is constant.
I/O rate: There are many factors involved in how the DB engine consumes/optimizes I/O. I wouldn't get to this level of granularity.
Data transferred IN: Irrelevant, free for both cases.
Is using connection.commit() from pymysql more frequently to AWS RDS more expensive?