ptrblck:

How are the results indicating that the training is being continued?


I am looking at estimates of Wasserstein distances between different classes of training data and generated data. For copies of the same generator, those estimates should be almost identical before training. However, the distances at the beginning of the second training of a generator look nearly identical as the distance estimates after training the first generator with 5000 batches of data.



 ptrblck:

If you checked, that the models have different parameters, could you just pass a tensor with all ones through the trained model and the randomly initialized model and compare the outputs?


At the moment I do not have an initialized model in memory. I could run this experiment later. What I can do is to compare the results of the four trained models still in memory. Those are (averaged over 1000 outputs due to dropout in both evaluation and training mode):
tensor([ 1.4081,  0.6046, -0.5007,  0.1663, -0.5247, -0.4271, -0.0108,
        -0.2019,  0.0682, -0.9449,  0.0687, -0.0462, -0.0376], device='cuda:0')
tensor([ 1.2530,  0.3588, -0.2415,  0.0276,  0.1169, -0.5837, -0.2653,
        -0.7568,  0.1672, -0.4251, -0.1818, -0.0518, -0.5182], device='cuda:0')
tensor([ 1.3222, -0.5203, -0.0297,  0.7092,  0.2500, -0.6458, -0.5713,
         0.1600, -0.4600, -0.9698, -0.4975, -0.4153, -0.2564], device='cuda:0')
tensor([ 1.5271,  0.0421, -0.4180,  0.3413, -0.4186, -0.0325, -0.3692,
         0.4058,  0.1961, -0.9888, -0.1224,  0.5514,  0.0256], device='cuda:0')




 ptrblck:

I assume both the generator and discriminator are copied?


Yes.



 ptrblck:

Do you also create new optimizers for each GAN?


Yes.
I also make sure that the gradients towards the inputs are not saved over training batches.