Yes I just tried reducing the learning rate.
It takes a few more batches to reach the lowest batch loss and starts increasing again.
But the rate of increase of total loss is lesser.
Here’s how the output looks -
batch no 1 loss =  1.48519766330719
batch no 2 loss =  1.4672067165374756
batch no 3 loss =  1.4816129207611084
batch no 4 loss =  1.434296727180481
batch no 5 loss =  1.440385341644287
batch no 6 loss =  1.4176286458969116
batch no 7 loss =  1.40739107131958
batch no 8 loss =  1.3973239660263062
batch no 9 loss =  1.3658576011657715
batch no 10 loss =  1.3650410175323486
batch no 11 loss =  1.337887167930603
batch no 12 loss =  1.3581486940383911
batch no 13 loss =  1.3573602437973022
batch no 14 loss =  1.31741201877594
batch no 15 loss =  1.2534146308898926
batch no 16 loss =  1.3125391006469727
batch no 17 loss =  1.294553279876709
batch no 18 loss =  1.1889533996582031
batch no 19 loss =  1.2660199403762817
batch no 20 loss =  1.2492356300354004
batch no 21 loss =  1.2197800874710083
batch no 22 loss =  1.1822484731674194
batch no 23 loss =  1.1688812971115112
batch no 24 loss =  1.2103825807571411
batch no 25 loss =  1.1710189580917358
batch no 26 loss =  1.215803861618042
batch no 27 loss =  1.157057285308838
batch no 28 loss =  1.1300220489501953
batch no 29 loss =  1.0991148948669434
batch no 30 loss =  1.0601304769515991
batch no 31 loss =  1.0228008031845093
batch no 32 loss =  1.0912868976593018
batch no 33 loss =  0.9856505990028381
batch no 34 loss =  0.9819539189338684
batch no 35 loss =  1.0306510925292969
batch no 36 loss =  0.9848380088806152
batch no 37 loss =  0.989624559879303
batch no 38 loss =  0.9476594924926758
batch no 39 loss =  0.9612619876861572
batch no 40 loss =  0.8986958861351013
batch no 41 loss =  0.9100659489631653
batch no 42 loss =  0.9121372699737549
batch no 43 loss =  0.9362645745277405
batch no 44 loss =  0.8442355990409851
batch no 45 loss =  0.8276706337928772
batch no 46 loss =  0.803051233291626
batch no 47 loss =  0.8012357354164124
batch no 48 loss =  0.7831454873085022
batch no 49 loss =  0.7508223056793213
batch no 50 loss =  0.8096636533737183
batch no 51 loss =  0.7562276721000671
batch no 52 loss =  0.764530599117279
batch no 53 loss =  0.7601000666618347
batch no 54 loss =  0.710577130317688
batch no 55 loss =  0.6691561341285706
batch no 56 loss =  0.6948921084403992
batch no 57 loss =  0.6729640364646912
batch no 58 loss =  0.6475369930267334
batch no 59 loss =  0.7118080854415894
batch no 60 loss =  0.6325834393501282
batch no 61 loss =  0.6638092398643494
batch no 62 loss =  0.6681099534034729
batch no 63 loss =  0.6030600070953369
batch no 64 loss =  0.6064748167991638
batch no 65 loss =  0.5821509957313538
batch no 66 loss =  0.5825245976448059
batch no 67 loss =  0.5944026708602905
batch no 68 loss =  0.5315052270889282
batch no 69 loss =  0.5525216460227966
batch no 70 loss =  0.5441896915435791
batch no 71 loss =  0.5337881445884705
batch no 72 loss =  0.5284881591796875
batch no 73 loss =  0.5651391744613647
batch no 74 loss =  0.5703287124633789
batch no 75 loss =  0.5127482414245605
batch no 76 loss =  0.5207087397575378
batch no 77 loss =  0.5141681432723999
batch no 78 loss =  0.492698073387146
batch no 79 loss =  0.46884605288505554
batch no 80 loss =  0.511565089225769
batch no 81 loss =  0.5005688071250916
batch no 82 loss =  0.5198782682418823
batch no 83 loss =  0.47924041748046875
batch no 84 loss =  0.4710450768470764
batch no 85 loss =  0.46020805835723877
batch no 86 loss =  0.4849032163619995
batch no 87 loss =  0.4721143841743469
batch no 88 loss =  0.4703673720359802
batch no 89 loss =  0.4440164566040039
batch no 90 loss =  0.4755094647407532
batch no 91 loss =  0.4569631814956665
batch no 92 loss =  0.47981956601142883
batch no 93 loss =  0.4545298218727112
batch no 94 loss =  0.44351571798324585
batch no 95 loss =  0.46746325492858887
batch no 96 loss =  0.47358807921409607
batch no 97 loss =  0.4544294774532318
batch no 98 loss =  0.4962926506996155
batch no 99 loss =  0.48395419120788574
batch no 100 loss =  0.44108179211616516
batch no 101 loss =  0.4728832542896271
batch no 102 loss =  0.4890323281288147
batch no 103 loss =  0.4716647267341614
batch no 104 loss =  0.48249995708465576
batch no 105 loss =  0.47763603925704956
batch no 106 loss =  0.46557939052581787
batch no 107 loss =  0.4828740060329437
batch no 108 loss =  0.4845884144306183
batch no 109 loss =  0.4839400351047516
batch no 110 loss =  0.533776044845581
batch no 111 loss =  0.5043434500694275
batch no 112 loss =  0.4820551872253418
batch no 113 loss =  0.4765048027038574
batch no 114 loss =  0.5152161121368408
batch no 115 loss =  0.5205423831939697
batch no 116 loss =  0.45017296075820923
batch no 117 loss =  0.5060521364212036
batch no 118 loss =  0.5015444755554199
batch no 119 loss =  0.5357353687286377
batch no 120 loss =  0.5455232262611389
batch no 121 loss =  0.5464028716087341
batch no 122 loss =  0.5404470562934875
batch no 123 loss =  0.5515747666358948
batch no 124 loss =  0.48644503951072693
batch no 125 loss =  0.5170029401779175
batch no 126 loss =  0.4973011016845703
batch no 127 loss =  0.5444160103797913
batch no 128 loss =  0.5032432079315186
batch no 129 loss =  0.5079444646835327
batch no 130 loss =  0.5730089545249939
batch no 131 loss =  0.5962690114974976
batch no 132 loss =  0.5836448073387146
batch no 133 loss =  0.5500009655952454
batch no 134 loss =  0.5935165286064148
batch no 135 loss =  0.5874279737472534
batch no 136 loss =  0.5787685513496399
batch no 137 loss =  0.6307970285415649
batch no 138 loss =  0.6106826663017273
batch no 139 loss =  0.6551612019538879
batch no 140 loss =  0.5904539227485657
batch no 141 loss =  0.6462585926055908
batch no 142 loss =  0.6867714524269104
batch no 143 loss =  0.6145952939987183
batch no 144 loss =  0.6478555202484131
batch no 145 loss =  0.6670128107070923
batch no 146 loss =  0.6696873903274536
batch no 147 loss =  0.6415550708770752
batch no 148 loss =  0.7150164842605591
batch no 149 loss =  0.6920061707496643
batch no 150 loss =  0.7582559585571289
batch no 151 loss =  0.7134975790977478
batch no 152 loss =  0.7674281001091003
batch no 153 loss =  0.757271409034729
batch no 154 loss =  0.7957542538642883
batch no 155 loss =  0.7181404232978821
batch no 156 loss =  0.7426053881645203
batch no 157 loss =  0.7954193353652954
batch no 158 loss =  0.822559118270874
batch no 159 loss =  0.8057528734207153
batch no 160 loss =  0.8383210897445679
batch no 161 loss =  0.8529059290885925
batch no 162 loss =  0.9087692499160767
batch no 163 loss =  0.8549301624298096
batch no 164 loss =  0.8934707641601562
batch no 165 loss =  0.8587768077850342
batch no 166 loss =  0.9172695279121399
batch no 167 loss =  0.9977843165397644
batch no 168 loss =  0.9188544750213623
batch no 169 loss =  0.9683499932289124
batch no 170 loss =  1.0196045637130737
batch no 171 loss =  1.0112369060516357
batch no 172 loss =  1.0482044219970703
batch no 173 loss =  1.0166860818862915
batch no 174 loss =  1.0707794427871704
batch no 175 loss =  1.0426247119903564
batch no 176 loss =  1.1214118003845215
batch no 177 loss =  1.141948938369751
batch no 178 loss =  1.169199824333191
batch no 179 loss =  1.1278570890426636
batch no 180 loss =  1.177224040031433
batch no 181 loss =  1.1449683904647827
batch no 182 loss =  1.2180325984954834
batch no 183 loss =  1.2685843706130981
batch no 184 loss =  1.1946467161178589
batch no 185 loss =  1.2430262565612793
batch no 186 loss =  1.3517838716506958
batch no 187 loss =  1.3116604089736938
batch no 188 loss =  1.3319159746170044
batch no 189 loss =  1.331680178642273
batch no 190 loss =  1.3623106479644775
batch no 191 loss =  1.3105368614196777
batch no 192 loss =  1.3695563077926636
batch no 193 loss =  1.5167524814605713