The solution seems to be to first doing the partitioning and then doing the map and set_format on the DatasetDict. That being said, I still feel like the behaviour that I mentioned before is a bug - but I am not sure.