I would like to correct my answer here. Detaching shouldn’t be the issue. I tried a simple dummy example. Consider the following snippet. This code can overfit on random input.
import torch
import torch.nn as nn

def train_model(model, data, actual_labels):
    model.train()
    optimizer.zero_grad()
    output = model(data)
    # detaching here
    probe = output.squeeze(0).squeeze(0)
    probe = torch.detach(probe).numpy()
    print(probe.max())

    loss = loss_function(output, actual_labels)
    loss.backward()
    optimizer.step()
    
    return loss.item()

data  = torch.randn(1000, 2048)
labels = torch.empty(1000, dtype=torch.long).random_(3)
loss_function = nn.CrossEntropyLoss()
model = nn.Linear(2048, 3)
optimizer = torch.optim.Adam(model.parameters())

for epoch in range(100):
    loss = train_model(model, data, labels)
    print(f"Epoch: {epoch}, loss: {loss}")


So, I think it should be some problem in your data processing pipeline or modeling. The first suggestion would be to try and overfit just single example and see if your network could learn this single example. If not, I would try to inspect some data/label mismatch in data processing code. And maybe some reshapes on the batch level in the model code which could introduce some mixing of data points. There  are for sure a lot of other possibilities on how we can introduce silent bugs into our neural network related code. For example, I tend to reread this 1 great blog post from Andrej Karpaty about neural networks being such a “leaky abstraction”.