If you have static input shapes, that would be possible, but that would also be kind of the “standard approach” initializing the conv layer just a bit later.
I guess you are dealing with variable sized inputs and would like to use the corresponding conv kernel “on the fly”.
In that case it’s a bit trickier, since there are probably some approaches, which would have some shortcomings.


You could pass your new input to your model and the shape would be checked in forward. If the corresponding kernel was already created, just use it for processing the input and training. Else create this kernel and do the same.
Depending on the variety of your input shapes this approach would create a lot of kernels, which are maybe hardly being trained. E.g. if a specific size occurs only once in your training data, this kernel will just be used once in the training. Also what should happen, if you’ll get new shapes in your validation/test dataset?


Use a single kernel and manipulate it, i.e. increase/decrease somehow its spatial size. In that case I’m not sure how the kernel will behave if you have large differences between the input sizes. The filter will not only see the features in a completely other resolution, but also if you just append some random values to the filter or cut the filter, the weights will have some “discrepancy”, e.g. while the center was already trained, the borders are completely random weights. I’m not sure, if it would be a good idea to interpolate the kernel weights to another size based on the input size, but that seems to me like a valid approach to try out.


If you just need global pooling, have a look at the adaptive pooling layers 10.