I even printed out what happens after each of them (memory cleaning attempts) occurs and it seems it never get’s cleaned.
optimizer.zero_grad()
    output, loss, _ = model(data, target)

    gc.collect()

    loss.backward()
    optimizer.step()

    losses.append(loss.item())
    # print("Cleaning after loss")
    # optimizer.zero_grad()


    torch.cuda.empty_cache()

    if not quiet and batch_idx % print_every == 0:
        # print the average loss of the last 100 batches
        print("Train epoch: {} [batch #{}, batch_size {}, seq length {}]\tLoss: {:.6f}".format(
            epoch+1, batch_idx, data.size()[0], data.size()[1], np.mean(losses[-100:])))
        gc.collect()
        print("memory before cuda.emptycache")
        print(torch.cuda.memory_allocated())
        print(torch.cuda.memory_cached())
        torch.cuda.empty_cache()
        optimizer.zero_grad()
        del loss, data, target
        print("memory after cuda.emptycache")
        print(torch.cuda.memory_allocated())
        print(torch.cuda.memory_cached())

Train epoch: 1 [batch #0, batch_size 2, seq length 234] Loss: 0.693175
memory before cuda.emptycache
288670720
319029248
memory after cuda.emptycache
288626176
319029248
999it [01:35, 10.82it/s]Train epoch: 1 [batch #1000, batch_size 2, seq length 491]      Loss: 0.001480
memory before cuda.emptycache
298898432
329252864
memory after cuda.emptycache
298849792
329252864
1999it [03:14, 10.23it/s]Train epoch: 1 [batch #2000, batch_size 2, seq length 560]     Loss: 0.001411
memory before cuda.emptycache
301651968
332005376
memory after cuda.emptycache
301602304
332005376
2444it [03:57,  9.56it/s]Traceback (most recent call last):----------here it dies