Thank you for your reply. Sorry, I forgot to mention about the dataset.
Its from Honda and Waymo (on real road images not in simulation), each image has size (220, 66, 3)
There are about 2,50,000 images in training set and about 43,000 in validation.
And yes the data distribution is comparable. Similar angles and road conditions. Although inherently driving data is a little bit non identically distributed.
Yes, Data Augmentation with:

Random selection of various lighting, brightness, contrast, saturation and hue conditions
Spatial shift, rotation, flip and perspective

are in my considerations for the future. However, I was under the impression that I should be able to get at least some generalization with the current version of the model and pipeline as well. And that Augmentation would help improve that performance.
Please feel free to add your thoughts and comments on this.
I appreciate you taking the time to answer.