Even creating a local python virtualenv wouldn’t help because in this case I would install it only on the master node, while I need pytorch on each machine that I am using with Pyspark. That’s why I need to send the whl file to each cluster.
Don’t you think that the problem is the same as in here 2?