Ok, I’m starting to get confused. I got this to run on my GPU yesterday and today’s same notebook, with the same model gave me the below again:
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-d7f3d5975bb8> in <module>
     13             break
     14 
---> 15         y_pred = d_o_model(num, cat)
     16         single_loss = criterion(y_pred, label)
     17         aggregated_losses.append(single_loss.cpu().data.numpy())

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

<ipython-input-28-a1d9e4e91784> in forward(self, x_categorical, x_numerical)
     49             embeddings.append(e(x_categorical[:,i]))
     50 
---> 51         x = torch.cat(embeddings, 1)
     52         x = self.embedding_dropout(x)
     53 

RuntimeError: cuda runtime error (710) : device-side assert triggered at ..\aten\src\THC\THCCachingHostAllocator.cpp:278

I’m not finding anything on this device-side assert triggered on the forum.
Here is my training loop"
aggregated_losses = []
max_batch = 6450

for i in range(epochs):
    
    for b, (label, cat, num) in enumerate(train_loader):
        num = num.cuda()
        cat = cat.cuda()
        label = label.cuda()
        b += 1
        if b == max_batch:
            break
            
        y_pred = d_o_model(num, cat)
        single_loss = criterion(y_pred, label)
        aggregated_losses.append(single_loss.cpu().data.numpy())

        print(f'epoch: {i:3}, batch: {b:3}, loss: {single_loss.item():10.8f}')

        optimizer.zero_grad()
        single_loss.backward()
        optimizer.step()
        
    scheduler.step(single_loss)


print(f'epoch: {i:3} loss: {single_loss.item():10.10f}')

The model is the same as the original post.