The difference appears during training. Here’s a look at the total_loss for each batch while training.
For gpu -
batch no 1 loss =  1.48519766330719
batch no 2 loss =  1.349387288093567
batch no 3 loss =  1.2793351411819458
batch no 4 loss =  1.1422659158706665
batch no 5 loss =  1.065197229385376
batch no 6 loss =  0.8988766074180603
batch no 7 loss =  0.795573353767395
batch no 8 loss =  0.7166022062301636
batch no 9 loss =  0.5900906920433044
batch no 10 loss =  0.5402706265449524
batch no 11 loss =  0.5099382400512695
batch no 12 loss =  0.5270498991012573
batch no 13 loss =  0.4964383840560913
batch no 14 loss =  0.4695017337799072
batch no 15 loss =  0.4864051938056946
batch no 16 loss =  0.5001940131187439
batch no 17 loss =  0.5504444241523743
batch no 18 loss =  0.4995945990085602
batch no 19 loss =  0.6236541867256165
batch no 20 loss =  0.5803393125534058
batch no 21 loss =  0.6943713426589966
batch no 22 loss =  0.6918058395385742
batch no 23 loss =  0.7231294512748718
batch no 24 loss =  0.8104425072669983
batch no 25 loss =  0.8485711216926575
batch no 26 loss =  0.9337277412414551
batch no 27 loss =  1.040302038192749
batch no 28 loss =  1.0833561420440674
batch no 29 loss =  1.2168757915496826
batch no 30 loss =  1.3513448238372803
batch no 31 loss =  1.3978110551834106
batch no 32 loss =  1.5755630731582642
batch no 33 loss =  1.643385887145996
batch no 34 loss =  1.8431206941604614
batch no 35 loss =  2.0576276779174805
batch no 36 loss =  2.138352155685425
batch no 37 loss =  2.420424222946167
batch no 38 loss =  2.6673426628112793
batch no 39 loss =  2.9037210941314697
batch no 40 loss =  3.0415878295898438
batch no 41 loss =  3.3367340564727783
batch no 42 loss =  3.61075758934021
batch no 43 loss =  3.875593900680542
batch no 44 loss =  4.022735595703125
batch no 45 loss =  4.232641220092773
batch no 46 loss =  4.469114303588867
batch no 47 loss =  4.6978302001953125
batch no 48 loss =  4.908476829528809
batch no 49 loss =  5.408933639526367
batch no 50 loss =  5.91743278503418
batch no 51 loss =  6.056540489196777
batch no 52 loss =  6.3848652839660645
batch no 53 loss =  6.919095993041992
batch no 54 loss =  7.30460262298584
batch no 55 loss =  7.652694225311279
batch no 56 loss =  8.113967895507812
batch no 57 loss =  8.591318130493164
batch no 58 loss =  8.899323463439941
batch no 59 loss =  9.512104034423828
batch no 60 loss =  9.753629684448242
batch no 61 loss =  10.346807479858398
batch no 62 loss =  11.033885955810547
batch no 63 loss =  11.304972648620605
batch no 64 loss =  11.373547554016113
batch no 65 loss =  12.382369041442871
batch no 66 loss =  12.86313533782959
batch no 67 loss =  13.56275749206543
batch no 68 loss =  13.181768417358398
batch no 69 loss =  14.387351989746094
batch no 70 loss =  14.72762680053711
batch no 71 loss =  14.963506698608398
batch no 72 loss =  15.535669326782227
batch no 73 loss =  17.41385841369629
batch no 74 loss =  17.897266387939453
batch no 75 loss =  18.659147262573242
batch no 76 loss =  18.77376365661621
batch no 77 loss =  19.839771270751953
batch no 78 loss =  20.197492599487305
batch no 79 loss =  20.273174285888672

and this keeps increasing for gpu.
For cpu:
batch no 1 loss =  1.5018784999847412
batch no 2 loss =  1.236070990562439
batch no 3 loss =  1.094645380973816
batch no 4 loss =  0.9613571166992188
batch no 5 loss =  0.8980026841163635
batch no 6 loss =  0.7537975311279297
batch no 7 loss =  0.6927058696746826
batch no 8 loss =  0.6422103047370911
batch no 9 loss =  0.5273403525352478
batch no 10 loss =  0.4687236547470093
batch no 11 loss =  0.4736068546772003
batch no 12 loss =  0.4473775029182434
batch no 13 loss =  0.44782334566116333
batch no 14 loss =  0.3897721469402313
batch no 15 loss =  0.37255093455314636
batch no 16 loss =  0.40343356132507324
batch no 17 loss =  0.41588154435157776
batch no 18 loss =  0.3447286784648895
batch no 19 loss =  0.3894757628440857
batch no 20 loss =  0.3367255628108978
batch no 21 loss =  0.36912691593170166
batch no 22 loss =  0.3307099938392639
batch no 23 loss =  0.31340059638023376
batch no 24 loss =  0.3092980682849884
batch no 25 loss =  0.2926962673664093
batch no 26 loss =  0.30344510078430176
batch no 27 loss =  0.29070281982421875
batch no 28 loss =  0.2682488262653351
batch no 29 loss =  0.28866472840309143
batch no 30 loss =  0.2943607568740845
batch no 31 loss =  0.2776503562927246
batch no 32 loss =  0.2795364260673523
batch no 33 loss =  0.2474450021982193
batch no 34 loss =  0.24592795968055725
batch no 35 loss =  0.25801122188568115
batch no 36 loss =  0.2531129717826843
batch no 37 loss =  0.22873735427856445
batch no 38 loss =  0.2336498498916626
batch no 39 loss =  0.24088449776172638
batch no 40 loss =  0.21383072435855865
batch no 41 loss =  0.2321128398180008
batch no 42 loss =  0.21124516427516937
batch no 43 loss =  0.2158881574869156
batch no 44 loss =  0.22324740886688232
batch no 45 loss =  0.20954640209674835
batch no 46 loss =  0.18683622777462006
batch no 47 loss =  0.20678168535232544
batch no 48 loss =  0.17524667084217072
batch no 49 loss =  0.18902267515659332
batch no 50 loss =  0.19290755689144135
batch no 51 loss =  0.17181488871574402
batch no 52 loss =  0.1689278483390808
batch no 53 loss =  0.17976917326450348
batch no 54 loss =  0.1698329597711563
batch no 55 loss =  0.16441339254379272
batch no 56 loss =  0.16918373107910156
batch no 57 loss =  0.16696254909038544
batch no 58 loss =  0.1432090401649475
batch no 59 loss =  0.16007395088672638
batch no 60 loss =  0.14895042777061462
batch no 61 loss =  0.157599538564682
batch no 62 loss =  0.17643079161643982
batch no 63 loss =  0.14895623922348022
batch no 64 loss =  0.136954203248024
batch no 65 loss =  0.15611445903778076
batch no 66 loss =  0.15810908377170563
batch no 67 loss =  0.14348700642585754
batch no 68 loss =  0.12974749505519867
batch no 69 loss =  0.13568048179149628
batch no 70 loss =  0.14065878093242645
batch no 71 loss =  0.12928995490074158
batch no 72 loss =  0.14016589522361755
batch no 73 loss =  0.15469464659690857
batch no 74 loss =  0.14998523890972137
batch no 75 loss =  0.13565079867839813
batch no 76 loss =  0.1321781426668167
batch no 77 loss =  0.1252971589565277
batch no 78 loss =  0.1260882318019867
batch no 79 loss =  0.12096744030714035