Yes, this is the case, only with AdaIn-like res-blocks instead of f.c.’s. Interestingly approach 1 consumes more memory during training.
How can I implement approach 1 more efficiently? Note again, that I need access to all mid-outputs
Thank you very much, Jonathan