I want a CNN designed by NVIDIA to work on a regression task of “Driving image to steering angle prediction”. Although the model (image 1) was designed for a slightly different task (with driving images and other signals), I was planning to try this first and modify along for my purposes.

Screen Shot 2021-08-04 at 8.09.43 PM870×1228 61.5 KB

Hyperparamters I have been using are:

Learning rate: 0.0001
Train and Val batch sizes: 64, 64
Optimizer: Adam
Activations: ELU

I have studied code from various people who implemented this model to try a bunch of different strategies:

Normalize pixels in range (0,1), in the range (-1,1) and z-score normalization (separately)
One Dropout 0.2 between CONV 2 and 3 layers (also tried 0.4)
One Dropout of 0.2 between CONV and Linear layers (also tried 0.4, 0.5)
Batch normalization after each convolutional layer
Reduction of model complexity (removal of CONV and Linear layers)
Different values of LR, Batch
Different variations of filter sizes, stride and linear layers

However, despite trying about many many variations of the model for the entire day today, I am still not able to get validation loss to decrease (see figure 2)

qi3DCDZP1600×500 34 KB

Some more strategies that I have in mind but am not sure about their effectiveness are:

Implementing a decaying learning rate scheme

Any help would be greatly appreciated.
Thanks