Thanks for the code.
Could you remove the sigmoid as the last non-linearity, as nn.CrossEntropyLoss expects raw logits, and rerun the code?
EDIT: Also note that you are passing vgg_comp_model.parameters() to the optimizer, which is undefined in your code.