I find the previous solution has problems with DataParallel, which complains RuntimeError: tensors are on different GPUs, even in the bleeding-edge version.
It might be related to https://github.com/pytorch/pytorch/issues/689 72. A temporary solution is to create a member variable for each branch (e.g., self.b1 = nn.Sequential(...)), instead of grouping them into a list.