Processing speed is slow increasingly, perhaps GPU Memory problem.
Except first process time, early processing time is pretty fast.
However, suddenly processing time is slow after 13th run.
After 30 iteration, program was sleep 3 sec to show that processing time is recovery during 12 iteration.

image708×924 7.68 KB

how can resolve this problem?
Enviroment.
GPU : gtx1060 6gb
Image size : 800x800x1
Code.
torch::jit::script::Module module= torch::jit::load(NetworkPath, torch::kCUDA);
torch::NoGradGuard no_grad;
torch::Tensor outputArr[100];
std::cout << "Start Iteration run" << std::endl;
for (int i = 0; i < 100; i++)	{
	if (i != 0 && i % 30 == 0)	{
		std::cout << "------ each 30 iteration forced Sleep 3sec -----" << std::endl;
		Sleep(3000);
	}
	t = clock();
	outputArr[i] = module.forward({ aa_cu }).toTensor();
	std::cout << "[Time] Iter : " << i+1 << " // Time(ms) : " << clock() - t << std::endl;
}