This drop in accuracy is because your models use Batch Normalization. Batch Normalization has been shown to work badly when the batch size is small. From this SO answer 1:

unless you can explicitly justify it, I advise against using BatchNormalization with batch_size=1; there are strong theoretical reasons against it, and multiple publications have shown BN performance degrade for batch_size under 32, and severely for <=8. In a nutshell, batch statistics “averaged” over a single sample vary greatly sample-to-sample (high variance), and BN mechanisms don’t work as intended.