Thanks a lot, there was a problem in the architecture which I resolved.
Yes, I removed TF.to_tensor on the mask though yet to figure out how to do selective transformations on it.
Although I am encountering the following error on running the current network-
RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at/aten/src/THNN/generic/SpatialClassNLLCriterion.c:109

Now, I am not able to figure out where I am doing wrong?