I’d recommend starting with freezing all but the last layer.
In terms of debugging your fluctuating accuracy there are a few things to consider, first what is your training accuracy doing during those epochs? Is it continuously going up? Also, just to clarify, when you say one epoch, do you see all of your training and testing data in an epoch? If not, that could be a symptom of the error. Another thing I’d recommend doing for now is setting momentum = .0 to help debug the issue.
To help debug further could you add your full source code?