nbingo:

So far, the NVIDIA community seems to think this is a pytorch/Azure issue and the Azure kubernetes community thinks it’s a pytorch/CUDA issue, so this is the last place I have left to ask in this dependency triangle 


Let me close the loop and claim it’s an Azure/Kubernetes issue 
Back to the issue: I’m not familiar with AKS, but could you try to run either the pytorch docker container 2 or the NGC one to check, if the node setup might not be able to run any containers?
If these containers are running fine, I would guess that your custom docker container might create some issues and we would need to check it.