I assume the number represent the accuracy?
Do you have a classification use case? If so, it’s a bit strange to see 0% accuracy, because that would be worse than random.
I’ve never heard of an approach to combine two pre-trained networks together in different parts of the model, but it sounds interesting.
Have you tried to fine-tune the last part first, while freezing the first part? Then maybe after a while you could train the model end-to-end.