Well, its not what easy.
My real example looks like this
self.transformer_decoder = TransformerDecoder(
	[TransformerDecoderLayer(
		self_attention=AttentionLayer(FullAttention(), hidden, nhead),
		cross_attention=AttentionLayer(FullAttention(), hidden, nhead), d_model=hidden) for _ in range(1)])

and
self.transformer_decoder = RecurrentTransformerDecoder(
	[RecurrentTransformerDecoderLayer(
		self_attention=RecurrentAttentionLayer(RecurrentFullAttention(), hidden, nhead),
		cross_attention=RecurrentAttentionLayer(RecurrentCrossFullAttention(), hidden, nhead), d_model=hidden) for _ in range(1)])

I don’t want to change layers’ definitions.
But it has the same variables names, just different math.
So I guess I can init both modules, get dict name-param of first, iterate the second module, and replace its params with params of the first one.
But not sure if its the best way.