Hi, what I understood from your answer is that the number of parameters (weights and biases) are stored twice in pytorch as per your Memory for network parameters. However, I didn’t quite get the Memory for data part. Shouldn’t the total calculation for a generic network in Pytorch be something like this so that it takes care of both the output features/activations at each layer of the network and the network parameters, each of which is stored twice?
total_gpu_usage = 2 x batch_size x (input_data_size + \
		sum(feature_size_at_each_network_layer)) + 2 x parameter_size