The loss looks reasonable in FP32 , however its fluctuating. And the loss scale decrease after 4 epochs.

Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Epoch 1 loss is 14325.70703125 and accuracy is 0.7753031716417911
Epoch 2 loss is 10062.3505859375 and accuracy is 0.8111007462686567
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 128.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 64.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16.0
Epoch 3 loss is 114.29520416259766 and accuracy is 0.8091962064676618
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.5
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.25
Epoch 4 loss is 4.909278869628906 and accuracy is 0.7796175373134329
Epoch 5 loss is 0.1522625833749771 and accuracy is 0.8130830223880597
Epoch 6 loss is 0.15188875794410706 and accuracy is 0.8128498134328358
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.125
Epoch 7 loss is 0.1318310648202896 and accuracy is 0.812616604477612

I am training as we speak. However i think i should change the architecture cause its kind of slow training. I am sure i am using everything multiple of 8