I have 20 .csv files which have data about some organs. They are way too big to combine since I run out of RAM even trying to merge them. I managed combine 10 of them but my RAM runs out when I load it so they cannot be merged.
I would like to train my CNN on 2-20 and then test it with 1. However my approach seems to reset the net each time I use a new set.
I pasted the output here: https://justpaste.it/28tk0 3
But to summarize, even after 9 hours of training, the test set is still at between 10% and 20% accuracy (btw there are 23 classes in total). The problem is it doesn’t improve at all. However the most important part is, for some reason, after file 14, the test accuracy is always around 40%. For file 3, it’s literally always as 13%.
From these results I conclude that I have failed to train the CNN on multiple sets and instead it always resets when I use a new set.
This is what I tried:
n_epochs = 10
for epoch in range(n_epochs):
    for i in range(2,21):
        train_dataset = mydata('./data/{}.csv'.format(i), transform= transforms.Compose(
                            [transforms.ToPILImage(), 
                             transforms.ToTensor(), 
                             transforms.Normalize(mean=(0.5,), std=(0.5,))]))
        train_loader = torch.utils.data.DataLoader(dataset=train_dataset,batch_size=batch_size, shuffle=True)
        train(epoch, i)