Hello,
I plan to use pre-trained CNN for image classification. Also, I plan to train only the FC layers, hence the features shall stay fixed. With this in mind, Its beneficial to generate the features only once and use them over all epochs. My question is how can I save features into disk efficiently? Assume that each image has an image ID so during the forward pass, I can save features based on the image ID. But with high number of images, the size of my dictionary (where I will save features) will keep on expanding and eventually RAM shall not be able to handle it.  So how can I (efficiently) append features pertaining to the current batch in a dictionary (Saved on disk) without loading the dictionary into memory?
Or any other suggestions?
Thanks