Hi @ptrblck. Thanks for your reply!
I don’t really need the output of the classification layer in the fine-tuning stage. I only need it when pre-training the first model for supervision (loss calculation) purposes. After the first model has been trained, I only want to use the output of the LSTM and not the classification layer, but I need to fine-tune it as well (fine-tune everything except the classification layer). My case is similar to when fine-tuning a pre-trained CNN. We remove the classification layer and fine-tune the other CNN layers.