How is your loss function defined? If the parameters of the decoder are fixed from scratch, it will be very difficult to reconstruct input data solely based on encoder.
Also, you may consider removing the last ReLU activation in the decoder. The ReLU is removing all the negative values in the output, and setting them to zero, so that automatically adds a constant value to the loss value.