Hello,
I’m a fairly new Pytorch user and wondering if anyone could help me with this problem associated with Dataloader.
Here’s a screenshot of my dataframe, inputs are values from ‘y+, index, Re_tau, DU_DY, Y’ column.

dataframe1807×823 38.5 KB

Every point in this dataframe, DU_DY & Y always have the same size. However, for different Re_tau values, the size for DU_DY are different (hence, so is the size for Y). And this is where the problem starts in network training. I get a feeling the training data for DU_DY & Y wasn’t split correctly, then checked a few threads and see many CNNs have encountered the same problem and they were solved by calling a custom collate_fn in Dataloader. I attempted to implement my own collate_fn but failed after many tried…
This dataframe ‘df_train’ is first converted to datasets ‘ds_train’, then convert to dataloader.
trainloader = DataLoader(ds_train, batch_size=5,  shuffle=True)
Could anyone point me into the right direction on how to write a collate_fn for this different input size problem? I heard there was a Pytorch preview version solve this problem automatically, sadly not the latest stable version.