Hello Nehemia -



 nehemias:

Hi everyone,
I used a pre-trained ResNet50 as my base model and I have 50 classes at the last layer (fully connected layer). The training was already performed and worked well.
However, I want to add 5 new nodes (=adding new tasks) into the last layer. Is it possible to add the new nodes to the last layer without creating a new layer as I want to keep the existing nodes which have been trained well?
Anyone can help me? Many thanks, everyone


I am hardly an expert, so take this as a not-very-well-informed
suggestion:
It would seem reasonable to me to simply add your new output
nodes with zero or small (or random) values for their weights.
In more detail, let’s assume (just to have some numbers) that
your next to last layer has 100 outputs.  Then your last layer
(with 50 outputs) – being fully connected, and assuming bias
weights – will have 5050 weights.
Make a new model that is the same except that its last layer
has 55 outputs.  This new last layer still has 100 inputs, so it
will now have 5555 weights (including biases).  Set the weights
that connect to your original 50 outputs to their pre-trained
values, but initialize the weights to the 5 new outputs with
some sort of “random” values.
Experts should be able to give you best practices for initializing
the new weights, but I would probably try something like random
values of maybe 20% of the size of the typical pre-trained weights.
Initially your five new task prediction values will come out random,
but should get better as you further train your model.  And the
“feature extraction” encoded in the earlier layers should still
work well.
Best regards.
K. Frank.