Solved by nihalnayak in post #22 
              
              
                Update: The main error was due to the difference between floating point precision in cpu and gpu for float32 type. 
I converted all my operations to float64 type including the neural layers. The loss is reducing on cpu and gpu at almost the same rate. The reason I say “almost” is because I am using n…