Hello,
Here’s my scenario. I have abundance of data, tens of millions of diverse samples, proper dataset hygiene maintained, train/test/val partitioning and as good statistical distrubution as data allows, custom datagenerator maintains that distrubution throughout the process.
Network itself is rather, i presume, small, couple million parameters and performs convolutional multiclass classification. Batch size 32 training occupies 97% of 16G of GPUs memory and takes 10 hours per 10% of dataset.
The problem i encounter happens early in training. At around 5-8% of dataset the network begins overfitting, i diagnose that with regular snapshots. Typical stalling of test accuracy while test loss ramps up. The problem resists typical solution, batch normalizations, various dropout strategies, hyper parameter exploration fails as well.
From starting point to now i managed with architecture changes to get the network to solve test data from top-1 15% → 30% before overfitting occurs, with overfitting occuring at 5% → 8% of dataset.
Since training is rather time costly and already matches my cards capabilities, the approach i will decide to take will take around two weeks to fully explore, so i’m looking for guidance which route might be better to take first from people with experience with non trivial projects.
Here’s the two divergent approaches i consider

Decrese the batch size to 24/16 and try to explore wider architectures that 16G GPUs allows.
Increse computational complexity/depth of network.
Any insight/suggestion will be welcome.

Regards,
A L