Learning discrete values (size in your case) is very hard and this leads to non-continuous problems and so gradients cannot be used.

is that the reason, that when I do this,
a = torch.arange(-5, 5)
x = nn.Parameter(a)

I get error,
RuntimeError: Only Tensors of floating point dtype can require gradients

and we need to find someway to make an integer parameter, that is updated to have only discrete integer values, but updating parameters by gradient based approach is not the way to do this.
so, if I want my neural network to learn things like, image size, image crop size, an integer threshold, an integer batch_size, an integer number of layers in the network, then none of this could be done by gradient based approach, as all of these are discrete integers?