I got it working…
Multiple things:

I was using Jupyter Lab and somehow it was not showing me correct error. I had to change my image to RGB to get it working. Since I am new to pytorch I am still searching why this was required.
Next I got _RuntimeError: cat_out is not implemented for type torch.HalfTensor which got fixed by changing float16 to just float