I just use rnn and memory network.
It does not only use a lot of memory, I  monitor the memory usage, it gradually grows and after a short period of time, all the memory is eaten and I have to reboot my machine.