This seems to be the behavior that I am facing. I don’t know how pyarrow works, but for a single CSV file with 250 Mb I got out of disk quickly.
I was able to make this work by using my own Dataset implementation, in this way pyarrow isn’t used and all works fine.
I could not find a mention of this limitation on datasets documentation. I think this is worth adding since it requires some knowledge of pyarrow internals.