Could you try to lower the number of filter kernels in your layers and just check, how much memory will be used then? The intermediate activations might take a lot of memory when you are training your model.