Suppose I’m using the Dataset and Dataloader classes to load vectors (e.g. sentence embeddings trained from another model) on-the-fly (meaning I only load e.g. batchsize=16 embeddings at a time for the batch) for training my model. I might want to do this if the number of embeddings I have is large and I can’t fit it all into memory.
What is the best format to save the sentence embeddings in order to speed up the on-the-fly loading and hence training? What is the best way to load them?
The embeddings have size of 1024 dimensions. They can be thought of as being convertible to numpy arrays or torch tensors. I think the values of the embedding are quite random since they’re the latent vector of another model.