As far as I know, nn.Transformer remains unchanged since its release. Nonetheless, I found a post 2 that might answer your question. Or, another option is to dive into the source code of huggingface transformer to learn how they implement the cache and other optimization tricks (which is tough but highly worth!!). Moreover, after implementing these tricks, we can pass a custom_decoder also custom_encoder as args to instantiate a nn.Transformer.