Good news ! I seems able to avoid OOM by the following loading strategy:
checkpoint = torch.load(ckpt_file, map_location=lambda storage, loc: storage)
net.load_state_dict(checkpoint['model'])
optimizer.load_state_dict(checkpoint['optimizer'])  # pytorch 0.3.1 has a bug on this, it's fix in master
del checkpoint  # dereference seems crucial
torch.cuda.empty_cache()