Thanks. Very helpful.
Actually, I did not hard code gpu devices to each sub-module. I just apply do
torch.nn.DataParallel(net, device_ids=[0, 1, 2, 3]).cuda(). I check module1.weight and module2.weight and it shows cuda:0 and cuda:3 respectively.  (This never happen before).
I am suspecting something wrong here. I doubt that a BNInception(module1) + two residual block(module2) would go over 12GB ?