I haven’t checked the shapes, but I assume the memory might blow up, since the flatten operation is missing, which is defined in TwoMLPHead in this line of code 14.
Replace it with:
model.roi_heads.box_head = nn.Sequential(nn.Flatten(start_dim=1), model.roi_heads.box_head.fc6)

and it should work fine.