What are the benefits of using Vulkan over NNAPI and vice-versa for running inference on GPU?

The main difference is that with Vulkan, the code that runs on the GPU (shaders) ships as part of the PyTorch runtime.  With NNAPI, we give a high-level description of the model to the OS, which then sends it to the vendor backend for execution.  Right now, any updates to the backed require Android OS updates.

Benefits of Vulkan

Runs on Vulkan-enabled devices that don’t support NNAPI.
Shaders can be updated by updating PyTorch runtime, which means that bug fixes and performance improvements can roll out quickly.
New ops can be added in PyTorch updates, and soon custom ops will be addable by applications using PyTorch, without waiting for an OS update.
Soon, we’ll support an API for using Vulkan GPU tensors directly in your own shaders, so model output can be sent to rendering code without returning to CPU memory.


Benefits of NNAPI on GPU

Vendor backend might be more mature and/or faster than the PyTorch Vulkan backend.
Might be a stepping stone to using NNAPI on DSP or NPU.