where is your recurrence step defined?
Your code explotes because of loss_avg+=loss If you do not free the buffer (retain_graph=True, but you have to set it to True because you need it to compute the recurrence gradient), then all is stored in loss_avg. Take in account that loss, in your case, is not only the crossentropy or whatever, it is everything you use to compute it. If you want to keep track of the scalar value that represents your accumulate loss you can do loss_avg+=loss.data (though the use of .data is deprecated, for this cases, I still find it useful, clean and simple). This will only store the actual scalar value.
Anyway I think your code should look something like:
for e in range(epochs):
      for idx,(x,t) in enumerate(data_loader):#x should be 3 dimensional (recurrence,samples,dimension) if your network is fully connected, else 4 dimensional (time_step,batch,rows,cols)
          for t in range(Time_steps):
                 ...