Mi_Rak:

h1 = th.cat((h, y), dim = 1)
        print(h1.size())
#torch.Size([512, 517])



This tells us that y=(no_samples,5) simple maths.



 Mi_Rak:

My objective is to condition also CVAE encoder and decoder by y


You don’t have to do this



 Mi_Rak:

h2 = self.relu(self.hidden(h1))


That’s my whole point.
You need to understand your forward function.
Did you tried this, this will condition both your input and target



 Usama_Hasan:

def encoder(self,x, y):
        h = self.relu(self.fc1(x))
        h1 = torch.cat((h, y), dim = 1)
        temp_h1 = self.relu(self.hidden(h1))
        return self.mu(temp_h1),self.sigma(temp_h1)