Му current input size is [1, 138, 256, 256] in float32,
which is, i belive, about 40Mb.
The rest part of network is (for testing purposes) is just one small Conv2D.
So i have about 18k of params, in total.
My current best model working fine on same input, having about 2.5m params (Titan Black, 6Gb RAM).
So, as far as i understand, event tens of copies of x is not even enough to fill the RAM.
I’ve tried this implementation:
	def forward( self, x ):

		n2, c2, h2, w2 = x.size()

		x = x.permute( 0, 2, 3, 1 )
		x = x.contiguous()
		x = x.view( n2 * h2 * w2, 1, c2 )
		x = x.contiguous()

		x = self.body( x )

		x = x.view( n2, h2, w2, -1 )
		x = x.contiguous()
		x = x.permute( 0, 3, 1, 2 )
		x = x.contiguous()

		return x

And even this:
		n2, c2, h2, w2 = x.size()

		x = x.cpu()

		x = x.permute( 0, 2, 3, 1 )
		x = x.contiguous()
		x = x.view( n2 * h2 * w2, 1, c2 )
		x = x.contiguous()

		x = x.cuda()
		x = self.body( x )
		x = x.cpu()

		x = x.view( n2, h2, w2, -1 )
		x = x.contiguous()
		x = x.permute( 0, 3, 1, 2 )
		x = x.contiguous()

		x = x.cuda()

		return x

The result is stable.
With big body - OOM.
With small body - CUDNN_STATUS_NOT_SUPPORTED.
My last thoughts about CUDNN_STATUS_NOT_SUPPORTED is that Conv1D is not supports such large batch size.
Found this issue: https://github.com/pytorch/pytorch/issues/4107 8
Is it makes any sesnse to try manualy split whole array in small batches?