I am trying to use openNMT on the WMT german-english dataset. This dataset is about 600 Mb.
After I preprocess the data, I try to save them with the syntax
torch.save(train, open('trainFile', 'wb'), pickle_module=dill)¨¨
This results in a MemoryError that I am not sure how to fix. I am aware that there are some issues with torch.save when the models are store on GPU (at least according to some of the questions on this forum) but I am still in the pre-processing phase, so everything is still on CPU.
How can I solve the problem?