ptrblck:

If you checked, that the models have different parameters, could you just pass a tensor with all ones through the trained model and the randomly initialized model and compare the outputs?


I ran the experiment again and checked the outputs of each generator before, between and after training. These are the results:
Before training:
 tensor([-0.0532,  0.4151,  0.3573, -0.2433,  0.1307, -0.0836,  0.2847,
        -0.1178,  0.1347,  0.1278, -0.1246, -0.0588,  0.3419], device='cuda:0')
tensor([-0.0565,  0.4017,  0.3659, -0.2477,  0.1339, -0.0836,  0.2784,
        -0.1184,  0.1386,  0.1262, -0.1121, -0.0488,  0.3315], device='cuda:0')
tensor([-0.0594,  0.3994,  0.3677, -0.2517,  0.1434, -0.0990,  0.2837,
        -0.1150,  0.1401,  0.1330, -0.1092, -0.0620,  0.3439], device='cuda:0')
tensor([-0.0522,  0.4125,  0.3546, -0.2467,  0.1256, -0.0851,  0.2884,
        -0.1105,  0.1383,  0.1332, -0.1343, -0.0499,  0.3382], device='cuda:0')

After training the first generator:
tensor([ 1.2326,  0.1330, -0.2058, -0.2644, -0.1080,  0.1575,  0.2618,
         0.0960,  0.3211,  0.4803, -0.0785,  0.2011, -0.3083], device='cuda:0')
tensor([-0.0563,  0.4148,  0.3497, -0.2479,  0.1295, -0.0838,  0.2905,
        -0.0990,  0.1363,  0.1258, -0.1307, -0.0527,  0.3313], device='cuda:0')
tensor([-0.0562,  0.3983,  0.3653, -0.2451,  0.1365, -0.0950,  0.2755,
        -0.1169,  0.1376,  0.1209, -0.1104, -0.0573,  0.3361], device='cuda:0')
tensor([-0.0639,  0.3964,  0.3724, -0.2654,  0.1279, -0.1069,  0.2877,
        -0.1119,  0.1249,  0.1338, -0.1262, -0.0422,  0.3374], device='cuda:0')

After training the second generator:
tensor([ 1.2509,  0.1938, -0.1883, -0.2593, -0.0736,  0.0829,  0.1862,
         0.0384,  0.3301,  0.5389, -0.0262,  0.2488, -0.3501], device='cuda:0')
tensor([ 1.4068, -0.4611, -0.4623,  0.2457,  0.0790, -0.7879, -0.1040,
        -0.0594,  0.0446,  0.0345, -0.2796, -0.0319, -0.4630], device='cuda:0')
tensor([-0.0517,  0.4005,  0.3747, -0.2566,  0.1396, -0.0952,  0.2775,
        -0.1243,  0.1364,  0.1313, -0.0991, -0.0516,  0.3421], device='cuda:0')
tensor([-0.0709,  0.3985,  0.3652, -0.2412,  0.1333, -0.0953,  0.2782,
        -0.1052,  0.1356,  0.1420, -0.1292, -0.0532,  0.3499], device='cuda:0')

After training the third generator:
tensor([ 1.2610,  0.2675, -0.2377, -0.3138, -0.1379,  0.1104,  0.2437,
         0.0204,  0.3022,  0.5029,  0.0027,  0.2320, -0.3861], device='cuda:0')
tensor([ 1.4057, -0.4849, -0.5250,  0.3343,  0.1568, -0.8417, -0.0839,
        -0.0798,  0.0577,  0.0663, -0.3039, -0.0474, -0.4607], device='cuda:0')
tensor([ 1.4200,  0.1538,  0.4493,  1.2378,  0.8067, -1.0643, -0.2824,
         0.6857, -0.3737, -0.0873, -0.2991,  0.3936, -0.4183], device='cuda:0')
tensor([-0.0647,  0.4080,  0.3546, -0.2470,  0.1262, -0.0913,  0.2868,
        -0.0972,  0.1294,  0.1270, -0.1409, -0.0572,  0.3390], device='cuda:0')

After training all four generators:
tensor([ 1.2381,  0.1970, -0.2328, -0.3427, -0.0975,  0.1496,  0.2497,
         0.0587,  0.2780,  0.5047,  0.0048,  0.2039, -0.3336], device='cuda:0')
tensor([ 1.3951, -0.4659, -0.5315,  0.2548,  0.0990, -0.7395, -0.0883,
        -0.0192,  0.1091,  0.0959, -0.2537,  0.0104, -0.4816], device='cuda:0')
tensor([ 1.4325,  0.1591,  0.4397,  1.3557,  0.8864, -1.1142, -0.3061,
         0.6200, -0.3217, -0.0575, -0.3192,  0.4867, -0.3381], device='cuda:0')
tensor([ 1.0634, -0.4836,  0.4042, -0.5154,  0.0880, -0.2594,  0.0965,
         0.2983,  0.6161, -0.2963,  0.7156,  0.2674, -0.5459], device='cuda:0')