In average tensors are about 2000 * 1000 * 100 elements, sometimes they are could be about 15000 * 8000 * 100 elements. I split on chunks by dim=0.
Now I have a guess that they are calculating consequentially instead of in parallel (I thought that ThreadPoolExecutor.map will start first GPU thread, return to the main CPU thread, start second one GPU thread etc, then will await for results from any device that finished. But as I can see it waits until first GPU will finish and then start calculations on second one)
So what is the best practice to start my calculations asynchronous? (I could not use asyncio)