hhaoao,
thanks for the reply, i think you may have missunderstood the problem, due to my bad explaination.
The problem I have is that when i take 2 datasets and combine them in the doubledataset class (which works fine) I cannot normalise the data across the whole dataset due to the concatination.
To get over this I then take the concatacted datasets and use the normalised function to normalise the whole dataset. This then gets recombined, however at this stage instead of being a dataset with entries [[a,b,c,d],[a’,b’,c’,d’]], i end up with [a,b], where the dataloader has length 1 instead of the size of the dataset and a and be have size of the length of the dataset.
eg:
length of dataset 1000, batchsize=1 .
what i want is dataloader length 1000 and dataset[0] is [[a,b,c,d],[a’,b’,c’,d’]] where each entry is a numerical 2D arrray and some other metadata.
what i am ending up with is:
length of dataset 1000, batchsize=1 .
what i want is dataloader length 1and dataset[0] is [[a,],[a’]] where each entry just the a numerical 2D arrray.
to compond matters, datasets a and a’ before combining are of size [a,b,c,d] and [a’,b’,c’,d’] but they are only combining a & a’.
I aplogise if this is confusing, I am confused. the problem is something to with how i am recreating the datasets, but with torch.utils.data.ConcatDataset the options are severly restricting…
thanks for your patience.
chaslie