If it’s still low, I’d suggest toying around with hyperparameters and architectures. As a sanity check, you might want to:

Make sure you’ve defined data loaders properly (image-label mapping) by inspecting a few examples.
Start with a pre-trained feature extractor and see how a model on top of those features performs.