It’s not a bad accuracy at all, but you could try increasing the number of kernels (out_channels) in conv layers in a standard way like [1,32,64,128,256,512] . Also you could try training a ResNet18 and compare the results.
from torchvision import models
net = models.resnet18(pretrained=False)