It doesn’t seem you are mixing up the variable names, so I’m currently unsure what might be wrong.
Could you print the shapes of xv_cat, xv_cont, preds, and y_val?
Are you using some view operation inside your model with a fixed batch size?
Unrelated to this issue, but some minor issues:

it looks like you are using the Variable class to wrap your tensors, which is deprecated since PyTorch 0.4. If you are using a newer version, you can just use tensors now
you are appending the losses in the training and validation loss, which are attached to the computation graph. This will increase your memory usage, so that you might run out of memory after a while. If you don’t need to call backward on these losses anymore and just store them for debugging purposes, use losses.append(ls.item()) instead.
to save some more memory, warp your validation loop in torch.no_grad(), which will avoid to store the intermediate activations, which are needed for backpropagation:

with torch.no_grad():
    net.eval()
    ...