To complement apaszke’s answer, in case you are using Jupyter and you ‘run’ multiple times the cell which creates the network, put always a delete net on that cell, otherwise you might be creating multiple neural networks which will cause similar problems.
If not, just try to find the maximum batch size which works for your hardware.