Which works, but I feel like there should be a better way to do it.

“Better way” in terms of the API design? If yes, do you have an API in mind? Or better in terms of speed?
Removing rows is not easy to implement (efficiently) because PyArorw tables, which datasets use to store data, are immutable. You could use pandas for that (ds.to_pandas()) if your dataset is not too big and can fit in memory.