Hi,
i have the following code
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.output_layer = nn.Linear(1, 1)

net = Net()
# normal memory usage
net = net.cuda()
# python consumes about 2gb of host memory

# now waste time so we can look at the memory usage
b = 23
for i in range(1000000000):
    for j in range(1000000000):
        b = b + i % 3434 + j % 3553
        b = b % 2354

print(f"aha, {b}")

when running this with linux (kubuntu 19.10), python 3.7 and pytorch 1.3, host memory usage is about 2gb and the gpu memory usage increases by about 700mb.
I understand that there is some sort of memory pool when using cuda (so 700mb gpu memory sounds ok). I can imagine that it might be necessary to have a copy of all gpu memory on the host. but the host memory is almost triple the gpu memory, which seems odd to me.
would be cool if somebody could explain that