Ha, nothing like a best practice question for some bikeshedding.

I think serializing the dataset object is not a good idea.

If applicable, I tend to favor some native format for the given datatype. It also comes down to performance considerations (I/O is typically slower than some processing, so e.g. saving a tensor representation instead of an JPEG for image data is not a good idea). I’ve also had similar things with medical imaging.
I would avoid pickling. Typical options are using (dicts/lists of) tensors (using torch.save/load), numpy’s format, or, if you feel fancy, something like hdf5.


I think I’m much too grumpy for that.  IMO whether or not yaml is a good format (as opposed to command line parameters, a .py or whatnot) depends on how large the number of parameters are and how generalizable you need it to be. For example, I would not encode model architecture in anything but code unless you really need to vary it a lot across your experiments. But then I might not be the best source for advice on this part.

Best regards
Thomas