@ptrblck, Thanks for the reply. I did try that - but no luck.
The Loss still remains high. Other things I have tried - changing the architecture, changing the learning rate, and changing the batchsize/number of epochs but still the same result.
I am absolutely dumbstruck as what to try next. To be clear though, there is something wrong in the architecture right?