Hello,
I am training a model called EDVR using pytorch, after some arbitrary iterations(900,6800, etc) it stops logging in the terminal without any error. However, the GPU shows memory usage. the details are below:

I am training EDVR model using train.py  from this github: https://github.com/xinntao/EDVR 3  and the model dependencies are:


Python 3
[PyTorch >= 1.1]
NVIDIA GPU + [CUDA]
[Deformable Convolution]


I use one NVIDIA-SMI  GPU with about 11GiB memory, CUDA version 10.1
Pytorch version 1.3.0
The number of workers is set to 3
The code supports distributed training too using this command for example:
python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 train.py -opt training-log.yml --launcher pytorch
I have only one GPU so I run it using this command in terminal:
python train.py -opt training-log.yml --launcher none
-I connected to the ubuntu system which I run the cone on using Teamviewer app

I will be grateful if you help me to solve this problem.