Thanks for your reply!
I found that the accuracies are unusual. Not sure the underlying behavior.
train batch loss: 0.6967262625694275: 100%|██████████| 282/282 [00:32<00:00,  8.73it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 16.44it/s]
train batch loss: 0.6877152919769287: 100%|██████████| 282/282 [00:30<00:00,  9.12it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.53it/s]
train batch loss: 0.6955962181091309: 100%|██████████| 282/282 [00:33<00:00,  8.53it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.29it/s]
train batch loss: 0.6605843901634216: 100%|██████████| 282/282 [00:31<00:00,  9.09it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.15it/s]
train batch loss: 0.6882678270339966: 100%|██████████| 282/282 [00:32<00:00,  8.80it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.23it/s]
train batch loss: 0.6663910746574402: 100%|██████████| 282/282 [00:31<00:00,  9.01it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.27it/s]
train batch loss: 0.6218530535697937: 100%|██████████| 282/282 [00:31<00:00,  8.97it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.37it/s]
train batch loss: 0.6685017943382263: 100%|██████████| 282/282 [00:31<00:00,  9.01it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.32it/s]
train batch loss: 0.6893523335456848: 100%|██████████| 282/282 [00:31<00:00,  9.02it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.29it/s]
train batch loss: 0.6964542865753174: 100%|██████████| 282/282 [00:31<00:00,  8.93it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.28it/s]
train batch loss: 0.6830093264579773: 100%|██████████| 282/282 [00:31<00:00,  9.09it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.53it/s]
train batch loss: 0.6640526652336121: 100%|██████████| 282/282 [00:31<00:00,  9.03it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.26it/s]
train batch loss: 0.6821056604385376: 100%|██████████| 282/282 [00:32<00:00,  8.63it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.35it/s]
train batch loss: 0.6885587573051453: 100%|██████████| 282/282 [00:31<00:00,  8.84it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.14it/s]
train batch loss: 0.6852195858955383: 100%|██████████| 282/282 [00:31<00:00,  8.83it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 16.99it/s]
train batch loss: 0.6919611692428589: 100%|██████████| 282/282 [00:32<00:00,  8.55it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 16.14it/s]
train batch loss: 0.682405412197113: 100%|██████████| 282/282 [00:33<00:00,  8.49it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.14it/s]
train batch loss: 0.6876314282417297: 100%|██████████| 282/282 [00:34<00:00,  8.19it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.23it/s]
train batch loss: 0.6916389465332031: 100%|██████████| 282/282 [00:31<00:00,  9.08it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.35it/s]
train batch loss: 0.695244312286377: 100%|██████████| 282/282 [00:30<00:00,  9.13it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.61it/s]
train batch loss: 0.6778611540794373: 100%|██████████| 282/282 [00:30<00:00,  9.13it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.69it/s]
train batch loss: 0.7030342817306519: 100%|██████████| 282/282 [00:30<00:00,  9.12it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.61it/s]
train batch loss: 0.6811032891273499: 100%|██████████| 282/282 [00:30<00:00,  9.20it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:03<00:00, 17.82it/s]
train batch loss: 0.7325484156608582: 100%|██████████| 282/282 [00:30<00:00,  9.19it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.52it/s]
train batch loss: 0.6639286279678345: 100%|██████████| 282/282 [00:30<00:00,  9.20it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.74it/s]
train batch loss: 0.6750177145004272: 100%|██████████| 282/282 [00:30<00:00,  9.18it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.71it/s]
train batch loss: 0.7046581506729126: 100%|██████████| 282/282 [00:31<00:00,  8.96it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.24it/s]
train batch loss: 0.7280897498130798: 100%|██████████| 282/282 [00:31<00:00,  8.99it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.30it/s]
train batch loss: 0.6575214266777039: 100%|██████████| 282/282 [00:31<00:00,  9.04it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.48it/s]
train batch loss: 0.6623131632804871: 100%|██████████| 282/282 [00:31<00:00,  9.09it/s]
test acc: 0.54924578527063: 100%|██████████| 71/71 [00:04<00:00, 17.44it/s]
train batch loss: 0.7028350234031677: 100%|██████████| 282/282 [00:31<00:00,  9.07it/s]
test acc: 0.4010647737355812: 100%|██████████| 71/71 [00:04<00:00, 17.61it/s]
train batch loss: 0.7099341750144958: 100%|██████████| 282/282 [00:31<00:00,  8.96it/s]
test acc: 0.4010647737355812: 100%|██████████| 71/71 [00:04<00:00, 17.39it/s]
train batch loss: 0.7120646238327026: 100%|██████████| 282/282 [00:31<00:00,  9.06it/s]
test acc: 0.4010647737355812: 100%|██████████| 71/71 [00:04<00:00, 17.45it/s]
train batch loss: 0.6877050399780273: 100%|██████████| 282/282 [00:31<00:00,  9.08it/s]
test acc: 0.4010647737355812: 100%|██████████| 71/71 [00:04<00:00, 17.53it/s]
train batch loss: 0.6763968467712402: 100%|██████████| 282/282 [00:31<00:00,  9.04it/s]
test acc: 0.4010647737355812: 100%|██████████| 71/71 [00:04<00:00, 17.28it/s]