Hi,
In fact you are applying GRU to the output of the second convolution if I am not wrong. And I think this is maybe a heavy operation (depending on the number of kernels in you convolution).
If you can’t fit this load in memory, the only possibilities for you are to have better hardware (you can use colab, it may provide you with a GPU of 15 GB), or you can change the architecture, or reduce the input size, (if you change the parameters that you use to compute MFCC features, you can get denser features)