I thought it would maybe cause a problem because of the ReLU registering multiple times with the model, and then maybe having certain operations happen on it too often or too little during backprop - I really don’t know because I don’t know details of how nn.module/autograd/etc is implemented.
I had a model that was working well so I cleaned up the code and added a few extra options. A few days later I ran more tests and started noticing that the training was more ‘chaotic’ (the losses would behave more erratically during training, and less consistently towards the target), and the training results never matched the best results I had seen until then, despite many attempts. I was surprised and kept digging until I noticed that changing purely the reuse of ReLUs seemed to make the difference.
In the meantime I must have trained about 50 times, half with reuse, half without, and for the last 36 of those (most of them done since posting the question - all happened on the same machine in essentially direct succession), I have the detailed results.

Screenshot_20200326_1419101216×647 261 KB

The maximum achievable training results seem to be comparable, but for ReLU reuse (‘shared’) the results are less predictable (more frequently has noticeably worse training results than expected). I use a custom version of ReduceLRonPlateau as a learning rate scheduler, and I observe that this regularly ‘saves’ runs with ReLU reuse from getting stuck on higher losses by waiting around long enough with a higher learning rate that it can “by chance” dig its way out of the hole it’s in. So even though the final results of ‘shared’ runs are sometimes okay, the path there is long and difficult.
I’m not sure how to quantify my observations from the 50 different training logs, and the observed differences, with cold hard comparable numbers… But in any case, the worst case performance of FullSize for Individual (no ReLU reuse) was 20.47% (lower is better), compared to three (of 9) Shared runs that were worse (up to 21.51%). And the worst case performance of HalfSize for Individual was 21.98%, compared to 3 of 9 Shared runs that were worse, up to 23.81%. This is far from the consistent 20-22% that Individual brings.
Any ideas? Or just statistical noise and bad luck?