i am training my model(contains conv layers, fc layers ,bigru layers）on two gpus. at the beginning, every batch takes around 5s， but as time goes by , it takes 20s for every batch. and i don’t know why the training time for the same size data will increase.
i use model=nn.DataParallel(model,device_ids=[0,1]) to train my model on two gpus.
and i use torch.cuda.empty_cache() after every batch, but it doesn’t work out.