Hello everyone,
This is a related issue to this one 3.
I am trying to use Pytorch with Pyspark.
In order to deploy my model on a large dataset on HDFS I need to add the Pytorch wheel with
sc.addPyFile(torch-0.4.1-cp27-cp27mu-linux_x86_64.zip)

or directly when I submit the spark job with
py-files torch-0.4.1-cp27-cp27mu-linux_x86_64.whl

I don’t have privileges to install Pytorch on the machines of cluster, so I really need to use the whl file. When I try to import torch, this error happens:
File "/tmp/spark-940d3edb-efdf-4ceb-8955-f1f1e1c59939/userFiles-f88461bb-66de-4d0d-97ad-30a6716c3339/torch-0.4.1-cp27-cp27mu-linux_x86_64.zip/torch/__init__.py", line 80, in <module> ImportError: No module named _C.
I know I should call import torch from another directory than the root one, but in this case I have no clue on how to do it.
Thanks for your help!