After further investigation, I did not have the correct class. I ended up removing everything relating to numpy and used pure pytorch methods. I fixed the loss using CEL by calling:
loss = nn.CrossEntropyLoss()(out, torch.max(target,1)[1])

However, I’m not quite sure what changing the targets argument actually did.