I have corresponding text and images that have a label too between 1-9. Using transfer learning on BERT (size 768), I dropped text feature vectors. Using transfer learning on top of ResNet50 I dropped image feature vectors (size 2048).
How can I design a transfer learning network (or another reasonable network) that combines these feature vectors? Which network should be used for this?
I have a 5 fold experiment and I have test and train txt files that on each line contains the corresponding feature vectors.
Say, for test set belonging to fold 0 I have the following two text files:
fold0_test_resnet_9class.txt
fold0_test_BERT_9class.txt