Hello everyone!
The tittle is pretty much self explanatory, but I would nonetheless like to elaborate a bit. Namely, I have two questions:


Very often we run multiple experiments on a single dataset and very often we also have a bunch of preprocessing steps that we want to perform on the data we have. I am wondering what is the best practice for serializing data after preprocessing? Is it considered best to serialize the entire torch.utils.data.Dataset object? Is it perhaps better to simply generate a preprocessed text file and read the input from there? Is there any preference when it comes to using pickle/dill? Feel free to elaborate and answer any other connected question that I have not asked, but you believe it’s important.


A common practice in Machine Learning is to use YAML config files to specify all sorts of things for your data preprocessing pipeline, your model, your layers, hyperparams etc. Are there any best practices on using YAML config files?


Looking forward to your answers