Hi,
I tried to reproduce according to your instruction, but couldn’t get OOM with the following code
MB = 1024*1024
N = int(10.5 * MB * 1024 / 4)

x = torch.zeros(N, device='cuda')
x1 = torch.zeros(1, device='cuda')
x = nn.Parameter(x)
x1 = nn.Parameter(x1)
ts = torch.cuda.comm.broadcast_coalesced([x, x1], [0, 1])

This is on 2 GPUs each with 12G memory. OOM triggers if I broadcast a 11GB tensor, but not when I broadcast a 10.5GB tensor and a 4 byte tensor.
There are two possibilities: 1. the bug was somehow fixed because I am using a master build. 2. the bug only happens with NCCL 1 because I’m using NCCL 2.
I’ll find a NCCL 1 build on Monday and see if I can reproduce.