When backward with  loss.backward() , will gradient accumulate at every step of the RNN?

yes

And won’t this accumulating leads to gradient explode?

depending on your arch, data, etc.

will the gradient be different from
the sum version?

yes because you scaled the loss