Good call! Originally I only used the one optimizer, but I got sidetracked by my troubleshooting and forgot to revisit that.
The weights of Encoder do in fact get updated with just the model optimizer. This is what I was hoping for and had assumed that the model graph would include anything through which my input tensors were passed.