Thank you for the suggestion.
I found that 273 MB was used for initialized CUDA on my side.
After running this script, the program consumes 277 MB, which matches well with my calculation 273MB + 4MB (for input).

net = nn.Conv3d(1, 16, 3, 1, 1)
net.cuda()


input = torch.FloatTensor(1, 1, 64, 128, 128).cuda()
input = Variable(input)
# out = net(input)

However, if I remove the comment of the last line, the program consumes 436 MB. There are 159 MB more memory consumed. However, if I calculate the size of output (16x64x128x128 x 4 bytes x 2) = 128 MB. There is still a not small gap there. Does anybody know why? Where are these additional memory consumed?