For variable input shapes I would recommend to start with the largest one and reduce the size if this would be possible. This would allow you to reuse the freed memory without running into memory fragmentation issues.