Finally I changed my method : every time I call net(image), I make sure to interpolate the image to the desired size with torch.nn.functionnal.interpolate. This way I only have one dataset.
Thank you !
`