Hmm, that change does yield the same results and there are fewer additional tensors being added on each backwards pass but looks like there are still some memory leaks.  I made the following change to the apply code.
#original
definedTagger=tagger()
...
applier = definedTagger.apply
outs = applier(outs, Values)
#new
outs = Tagger(outs, Values)

Is it possible things are being added when the tensors in Values are used?  Maybe I need more detach() or other things like that?