Thanks for the reply.
I tried doing final_fm = Variable(torch.tensor(5,N,256*2), require_grad=True)
That gives me an error about ‘in place modification’. Apperently, you are not allowed to do that. If set require_grad=False then forward pass is fine but I am concerned about the gradients. Will this allow gradients to flow back properly at backword pass?