kchalk:

In particular I’d like to make a confusion matrix, but can’t seem to get my outputs into a single structure (list/array/tensor).


For the confusion matrix you don’t necessarily need to store all predictions and targets. Just indexing at the corresponding row and column should be enough.



 kchalk:

I’m also interested in if there is a good way to include the datum id in here somewhere, so I can compare accuracy within some of the other fields I’m not using in training my model.


Do you mean the index by “datum id”? If so, you could just return the index with the data and target in your Dataset.



 kchalk:

I’d also be happy to know what I’m making harder than it needs to be and whats just bad code.


Your code looks alright. It seems you are using pandas to map some IDs to the data, which makes the code a bit harder to read, but if your need this mapping it’s totally fine. I’m not sure what you’re doing with Vaccuracies, but it you plan on taking the mean, your validation accuracy might have a small error, since the last batch might be smaller than all others. I would rather store the correct predictions and calculate the accuracy later using the length of your Dataset.
Here is a small example to create the confusion matrix (using sklearn’s row-column logic) and storing the predictions as well as the ids:
class MyDataset(Dataset):
    def __init__(self, nb_samples, nb_features, nb_classes):
        self.data = torch.randn(nb_samples, nb_features)
        self.target = torch.randint(0, nb_classes, (nb_samples,))
        
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        
        return x, y, index
    
    def __len__(self):
        return len(self.data)


nb_samples = 100
batch_size = 10
nb_features = 20
nb_classes = 10

dataset = MyDataset(
    nb_samples,
    nb_features,
    nb_classes
)

loader = DataLoader(
    dataset,
    batch_size=batch_size,
    shuffle=True,
)

model = nn.Linear(nb_features, nb_classes)
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

model.eval()
conf_mat = torch.zeros(nb_classes, nb_classes)
losses = 0.
preds = []
pred_ids = []
with torch.no_grad():
    for data, target, data_idx in loader:  # Assuming this is the val loader
        output = model(data)
        loss = criterion(output, target)
        losses += loss.item()
        pred = torch.argmax(output, 1)
        
        conf_mat[target, pred] += 1
        preds.append(pred)
        pred_ids.append(data_idx)

preds = torch.cat(preds)
pred_ids = torch.cat(pred_ids)

Let me know, if this helps or if I misunderstood your issue.



 kchalk:

I’m about to finish a masters degree and this shouldn’t be this hard…


Cheer up! Almost done, so don’t let these minor issues bug you.