I have tested my code just now and I think the issue (amongst the other ones in the code I posted) is the dataloader I created in Pytorch:
class Dataset(data.Dataset):

	'''
    This class is a simple data loader that is supposed to allow one to
    load data
	'''
	def __init__(self, filename, height, width, types_):
		self.data = np.load(filename)
		self.feats, self.labels = self.data[types_+'_x'], self.data[types_+'_l']
		self.height, self.width = height, width
		self.transform = transforms.ToTensor()

	def __len__(self):
		return len(self.labels)

	def __getitem__(self, index):
		img_as_np  = np.asarray(self.feats[index]).reshape((self.height, self.width))
		img_as_img = Image.fromarray(img_as_np)
		img_as_img = img_as_img.convert('L')
		return self.transform(img_as_img), self.labels[index]

which I think I did right but appears to have some issue since when I load the batches by simply creating variable like:
rad = np.random.randint(len(X_train), size=32)
local_batch  = Variable(torch.from_numpy((X_train[rad].reshape((32,1,ndim,ndim))).astype(np.float32))) 
local_labels = Variable(torch.from_numpy(y_train[rad].astype(np.float32)))

I get for each epoch:
epoch: 0 | train loss: 0.7168
epoch: 1 | train loss: 0.2158
which I think is now in the right ballpark, and I also see the trend in the results over two epochs.
One last thing which I would like to mention is that each epoch in Keras takes about 3 mins whereas
in Pytorch it is 4-5 mins. Is this expected or is it just due to my implementation? I currently use cpu for these tests for now.