@Atcold you have reached the limit of my knowledge  can’t help you sorry
Just about the nn.Sequential point, I implemented a network with 3 branches in parallel, I am not using nn.Sequential, I just end up summing the variables of my branches, this is pure autograd. Hope it can help