So you are having 64bit (float), seq_length=36, hidden_size=4800, num_layers=1, batch_size=(lets say 4)
mem_needed = 64 * 36 * 4800 * 1 * 4 = 44236800bit = 0.0442368GigaBit = 5.5296MegaByte of total memory just for the RNN If I did the math right, you should not have any problems…mhhhh.
What size are u passing into the other RNN?