I changed forward() to this to debug:
    def forward(self, images, labels):
        x = self.main_module(images)
        x = x.view(x.shape[0], -1)
        x = torch.cat([x, labels], dim=1)
        y = self.head(x)
        print(x[0])
        return y

And tensors generally look good:
tensor([-0.0892, -0.0673, -0.1182,  ...,  0.0000,  0.0000,  0.0000], device='cuda:0', grad_fn=<SelectBackward>)
But then the first part in x (i.e. the convolved images) turn into inf values:
tensor([inf, inf, inf,  ..., 0., 0., 0.], device='cuda:0', grad_fn=<SelectBackward>)
Those values generally hovered around 0.1 otherwise but just spiral in a single batch.