model.classifier[6] = nn.Sequential(
        nn.Linear(4096, 256), # in features from prevous out of model.classifier
        nn.ReLU(),
        nn.Dropout(0.4),
        nn.Linear(256, 2),
        nn.LogSoftmax(dim=1)
    ).to(device)

Missed that .to(device) on the end. Thank you very much, ptrblck.