I first divided the model into two parts; 1st half are the layers before the layer you want to add and the 2nd half contains the layers after your layer. So something like this:
encoder = nn.Sequential(*list(m.children())[:8])
decoder  = nn.Sequential(*list(m.children())[8:])

Then I added the layers I need, get a list of the layers and append each layer in the 2nd layer into the 1st one. Like this:
tmp_1 = list(encoder.children())
tmp_2 = list(decoder.children())
for i in tmp_2:
    tmp_1.append(i)

And then model = nn.Sequential(*list(tmp_1))