Ok here is what I now tried:
I have exact 4 same models which are playing against each other and trained using PPO.
I now tried to calculate the mean of the parameters and update the models like this:
            for [dict1, dict2, dict3, dict4] in zip(ppo[0].policy.state_dict().items(), ppo[1].policy.state_dict().items(), ppo[2].policy.state_dict().items(), ppo[3].policy.state_dict().items()):
                val1, val2, val3, val4 = dict1[1], dict2[1], dict3[1], dict4[1]
                final_dict[dict1[0]] = (val1+val2+val3+val4)/4
            for i in range(4):
                ppo[i].policy.load_state_dict(final_dict)

However this is not working correctly… (it does not learn faster…)
Any ideas?