Thanks for your reply.
Before employing the running average, my code occupied only half of video memory.
But, when I tried as you suggested, it didn’t proceed after 2 iterations due to ‘out of memory’.
The error is shown below.
CompleteTHCudaCheck FAIL file=/data/users/soumith/miniconda2/conda-bld/pytorch-cuda80-0.1.9_1487349287443/work/torch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
Traceback (most recent call last):
  File "inception_ae_resume_ra.py", line 251, in <module>
train_iter_loss, avg_param = train(config, epoch, avg_param)
  File "inception_ae_resume_ra.py", line 166, in train
avg_param = 0.9*avg_param + 0.1*flatten_params()
  File "/home/sypark/anaconda2/envs/py36/lib/python3.6/site-packages/torch/tensor.py", line 320, in __mul__
return self.mul(other)
RuntimeError: cuda runtime error (2) : out of memory at /data/users/soumith/miniconda2/conda-bld/pytorch-cuda80-0.1.9_1487349287443/work/torch/lib/THC/generic/THCStorage.cu:66