So any variable that is no longer reference is freed in the sense that its memory can be used to create new tensors, but this memory is not released to the os (so will still look like it’s used using nvidia-smi).
empty_cache forces the allocator that pytorch uses to release to the os any memory that it kept to allocate new tensors, so it will make a visible change while looking at nvidia-smi, but in reality, this memory was already available to allocate new tensors.
Your code look good, I would double check that things that you send to your logger are not Variables but just python numbers using .item() as necessary.
Also, if you’re using 0.4 (I assume current master), then you should remove Variable and .data from your code and replace loss.data[0] by loss.item()