I have applied the same technique with a single dataset, everything worked well.
With 20% of the amount of the two datasets, the frequency is low but the results still un-stable.
With 100% of the full datasets the validation loss jumping up in each epoch.
I have concluded that this issue is related to how the model dealing two datasets.
I’m still confused, why this happened and how I can fix it?