I think your problem is on the line:
raw_blend = encoder_blend + decoder_blend                              # BxD
You forgot the activation function and end up learning only linearity. should be:
raw_blend = torch.nn.functional.elu(encoder_blend + decoder_blend)