Using constant input sizes is not a hard requirement, but rather a recommendation.
For each new input shape cuDNN will profile available kernels internally and select the fastest one, which is then added to a cache. In the next iteration, this cache is checked for the currently used shapes and the already selected kernels will be picked.
The profiling itself will add an overhead and thus your first iteration for each new shape would be slower.
Depending on your use case and how many different input shapes you are expecting, using cudnn.benchmark=True could still work.
Your particular concern would rerun the profiling for the last batch if it’s smaller, so I wouldn’t bother dropping the last batch.