You could create e.g. a dict inside the custom model and store the output activations of the conv layers there (during the forward pass). In the second iteration you could then create the new activations by replacing parts of the conv output with the activations stored inside the model.