How to create patches / windows of image dataset?
I would now like to know where would be the best place to implement this type of preprocessing?
bodokaiser
Yeah that’s a tricky one. There’s no way to tell the DataLoader to dynamically skip some inputs and replace them with next ones, because it would violate the guarantee that it will always return the data in the same order as the given by the sampler.
apaszke
As you said I think I will need to add another stage which converts the data from HDF5 to patches and then feed that data into pytorch. Thanks in advance!
bodokaiser
trypag
You don’t need to save the patches ahead of time, it should be enough to store the file names and locations to retrieve them later. But if you don’t have a lot of data it might be simpler to preprocess the dataset.
apaszke
bodokaiser
bodokaiser
sure, I emailed you!!
trypag
apaszke
bodokaiser
Tepp
See you
trypag
Anyways, a very simple approach to sampling 2D patches from 3D MRI is to extract a random slice (see my package torchsample for that transform) then just take a random crop. You can use the “MultiSampler” sampler in torchsample to go through each image more than once in a single epoch. No need to extract EVERY possible patch haha just do it randomly at train time. It’s still a fun exercise to reimplement them tho… which paper specifically are you interested in? I think I have implementations of most of them and can share or help you develop the right sampler.
ncullen93
trypag
Yeah you kinda have more data but it’s misleading bc you don’t have more region-specific subject variability. Data is definitely an issue… I train on a lot of subjects (a few thousand) from multiple large datasets… Also, the KEY to MRI segmentation is data augmentation and applying random affine transforms specifically since images differ that way across scanners - hence why I developed torchsample. With good augmentation we have done quite well with ~200 subjects.
ncullen93
rogetrullo
Meghal_Dani
akbarnejad
Najeh_Nafti
fepegar