Thanks for your answer. Yes, I’m training from scratch, using the following net definition (inputs are faces of size 96x96x3):
class Net(nn.Module):
def init(self):
super(Net, self).init()
self.pool = nn.MaxPool2d(2, 2)
self.conv1 = nn.Conv2d(3, 6, 5)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1 = nn.Linear(16 * 21 * 21, 512)
self.fc2 = nn.Linear(512, 128)
self.fc3 = nn.Linear(128, 70)
def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    
    x = x.view(-1, 16 * 21 * 21)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

The problem is that when training accuracy is 100%, testing accuracy is just 59%.
Modifying the net layers (increasing or decreasing number of parameters) could improve testing accuracy, or just regularization or retraining work in this case?