Solved by ptrblck in post #2 
              
              
                yolo3_learner is using a wrong layer config, i.e. the third nn.Conv2d layer expects 384 input channels, while the preceding one outputs 256 channels. 
The same issue is in the 5th conv layer.