By “don’t use batches at all” I assume you are using a batch size of 1?
If so, you should be careful about layers, which depend on the batch size, such as batchnorm layers (which compute the mean and stddev from the input batch), as they might create noisy running stats.
Also, the convergence of the model would change and you might generally see a slower epoch time.