I also found this 2 which says where we can change this in cmake  :
if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
  set(TORCH_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=0")
endif()

for me though I had to change the /cmake/TorchConfig.cmake.in which is :
# When we build libtorch with the old GCC ABI, dependent libraries must too.
if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
  set(TORCH_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=@GLIBCXX_USE_CXX11_ABI@")
endif()

but is there another way to set the variable instead of hardcoding  this into cmake file?
Update:
Will doing something like  this suffice? 
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
export TORCH_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=1"
python setup.py install

OK I did this and the build process finished successfully 
Update 2:
However, upon importing the torch now I get 
In [1]: import torch                                                            
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-eb42ca6e4af3> in <module>
----> 1 import torch

~/Desktop/Pytorch1_6/pytorch-1.6.0/torch/__init__.py in <module>
    333 
    334 # If you edit these imports, please update torch/__init__.py.in as well
--> 335 from .random import set_rng_state, get_rng_state, manual_seed, initial_seed, seed
    336 from .serialization import save, load
    337 from ._tensor_str import set_printoptions

~/Desktop/Pytorch1_6/pytorch-1.6.0/torch/random.py in <module>
      2 import warnings
      3 
----> 4 from torch._C import default_generator
      5 
      6 

ImportError: cannot import name 'default_generator' from 'torch._C' (unknown location)


Whats wrong?
OK found the cause :

If you try to import torch while inside the top level pytorch source tree it will import the torch directory and not the installed torch module. Try changing your current working directory and see if the issue resolves.

With this now fixed, mission is now accomplished 
In [1]: import torch                                                            

In [2]: torch._C._GLIBCXX_USE_CXX11_ABI                                         
Out[2]: True


final note, to create the whl file, simply repalce python setup.py with pip wheel . or simply do:
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
export TORCH_CXX_FLAGS="-D_GLIBCXX_USE_CXX11_ABI=1"
pip wheel .