Solved by ptrblck in post #4 
              
              
                One possibility of an increased memory usage might be the storage of the computation graph. 
embed, cluster_size, and ema_embed are created as buffers, which would register the tensors without making them trainable (their requires_grad attribute would be False). 
However, in the forward method you a…