Thank you!! I needed to permute dimensions before the .view() call
This is correct code:
x_emb = self.encoder(phrase)
x_emb = x_emb.permute(1,0,2).contiguous().view(batch_size, 1, self.hidden*2)
y_emb = self.encoder2(response)
y_emb = y_emb.permute(1,0,2).contiguous().view(batch_size, self.hidden*2, 1)