Looks like my assumptions were right. Can you replace your method for computing accuracy with mine and obtain the curves? I don’t think my code will solve the jumpiness. However, it is advisable to use this method.
Also, your validation accuracy is jumping corresponding to jumps in validation loss which means it is not the accuracy computation that is wrong. Since the scale of the validation loss is in the 1 - 10^-1, you see a smaller oscillation. This means there is something funny happening in the loss itself. I suspect it has got something to do with the validation data.