Thank you for answering!
I figured out I was not using enough training data, only around 10,000 images. After training on 100,000 images, my output started outputting sprite-like objects. Using a sigmoid on the decoder output, not as part of the network, makes sense for forcing the output in between 0 and 1. However I think since the weights of the output layer are positive after further training, adding a sigmoid afterwards would change the weights but not the output.