That’s a bit strange, as both models yield the same output and get the same gradients, if you initialize them with the same values and set the seed before calling the forward method:
model1 = A2C_model(input_dim, act_size)
model2 = A2C_model2(input_dim, act_size)

with torch.no_grad():
    for param1, param2 in zip(model1.parameters(), model2.parameters()):
        param1.copy_(param2)


x = torch.randn(1, input_dim)
torch.manual_seed(2809)
output1 = model1(x)
torch.manual_seed(2809)
output2 = model2(x)
print(output1)
print(output2)

output1[2].backward()
output2[2].backward()

for param1, param2 in zip(model1.parameters(), model2.parameters()):
    if param1.grad is None and param2.grad is None:
        print('both none')
    else:
        print((param1.grad == param2.grad).all())

Could you use my code snippet to copy the parameters of your “good” model to the bad one and train the bad model for a bit just to see, if it achieves a similar accuracy then?