Could you post the shapes of a dummy input and target tensor?
Based on your model, I assume you are passing the input as [seq, batch_size, nb_features], while the sequence lengths doesn’t seem to be defined.
I just tried with an input tensor of [64, 64, 65] and am using ~965MB with the CUDA context.