Thanks for the additional information!
Could you run the following code and check the memory usage:
import torch
import torch.nn as nn

torch.cuda.synchronize()  # Create CUDA context (947MB in my case)

model = nn.Linear(6000, 6000).to('cuda')
expected_mem = (model.in_features * model.out_features + model.out_features) * 4 / 1024**2
print('Max allocated {:.3f}MB, expected {:.3f}MB'.format(
    torch.cuda.max_memory_allocated() / 1024**2,
    expected_mem))

model = nn.Linear(30000, 30000).to('cuda')
expected_mem = (model.in_features * model.out_features + model.out_features) * 4 / 1024**2
print('Max allocated {:.3f}MB, expected {:.3f}MB'.format(
    torch.cuda.max_memory_allocated() / 1024**2,
    expected_mem))

> Max allocated 138.023MB, expected 137.352MB
> Max allocated 3572.138MB, expected 3433.342MB

On my system, the CUDA context uses a constant memory of ~950MB, while the model parameters fill up additional memory.