Solved this by replacing with a tensor of shape [0,4]. So, for example:
    if np.isnan((target['boxes']).numpy()).any() or target['boxes'].shape == torch.Size([0]):
        target['boxes'] = torch.zeros((0,4),dtype=torch.float32)