Solved by rwightman in post #4 
              
              
                @vibe2 A researcher submitted a solution to this issue to my codebase a while back, apparently it has also been discussed in a pytorch issue/PR for sometime 


And the PR that was submitted with some info about the solution from the author: https://github.com/rwightman/pytorch-image-models/pull/140