mohit7:

But you should explicitly put a restriction on no of threads user is setting because after a certain limit instead of decreasing run-time it is increasing.


Yes, performance degrades when that thread number is more than cpu cores number as more thread switches and thread contention.
But additional capping may introduce some non-transparency for this API, we will think about it.
Our plan is to revise our default thread pool number to be optimal for inference time by default for as much as possible devices.


One more doubt-
Just like in pytorch can we pass as batch of images in case of pytorch android. ?


Vision models is the same, the input shape is N_images * N_channels (e.g. 3) * IMAGE_HEIGHT * IMAGE_WIDTH.
So if you prepare Tensor with N_images > 1 that should work as in desktop version.
But  org.pytorch.torchvisio.TensorImageUtils has api how to prepare tensors only with N_images==1,  so it needs some additional code to prepare it with  N_images > 1.
Do you think that will be useful to have in TensorImageUtils api some helper methods to prepare Tensors for image batches?