Okay, somehow I did it from fresh and it looks like it works!!
I ran:
git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
sudo python3 setup.py develop

and it ended with the following
.
.
.
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/torch/csrc/dl.o -o build/lib.linux-x86_64-3.6/torch/_dl.cpython-36m-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.6/torch/_C.cpython-36m-x86_64-linux-gnu.so -> torch
copying build/lib.linux-x86_64-3.6/torch/_dl.cpython-36m-x86_64-linux-gnu.so -> torch
Creating /usr/local/lib/python3.6/dist-packages/torch.egg-link (link to .)
Adding torch 1.1.0a0+90b8552 to easy-install.pth file
Installing convert-caffe2-to-onnx script to /usr/local/bin
Installing convert-onnx-to-caffe2 script to /usr/local/bin

Installed /home/arunava/Projects/Cloned/pytorch
Processing dependencies for torch==1.1.0a0+90b8552
Finished processing dependencies for torch==1.1.0a0+90b8552

But
when I run
python3 test/test_torch.py

It gives me
Traceback (most recent call last):
  File "test/test_torch.py", line 8, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

I think this is simple linking issue or something like that, some inputs will be invaluable!