Hi,
we are currently working on the Image feature, which will make this whole process faster.
In the meantime, you can use Dataset.with_transform and load the images on the fly when you need them. This way, only the current batch will occupy some memory. The only downside of this approach is that you’ll have to implement any sort of additional formatting by yourself in with_transform (you can also use DataLoader’s collate_fn for this) because with_format and with_transform currently cannot be chained.