But it seems my loss function is not improving the network. My accuracy don’t seem to improve as the epochs pass by.

How many epochs did you run? Is the loss decreasing?
Btw classic vgg’s classifier should look like this:
(classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )

You have replaced only 6th layer of classifier, so it looks like this:
(classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Sequential(
      (0): Linear(in_features=4096, out_features=1000, bias=True)
      (1): ReLU()
      (2): Dropout(p=0.4)
      (3): Linear(in_features=1000, out_features=101, bias=True)
      (4): LogSoftmax()
    )
  )

which is probably not as you wanted. If you wanted classic vgg classifier try:
self.vgg16.classifier[6] = nn.Linear(in_features=4096, out_features=101, bias=True)

For fine tuning you can also freeze weights of feature extractor, and retrain only the classifier. You can also experiment with retraining only some layers of classifier, or whole classifier and part of feature extractor. You can do it like this:
    for param in self.vgg16.parameters():
        param.requires_grad = False

    for param in self.vgg16.classifier.parameters():
        param.requires_grad = True