Not an expert (and far from being one) but I’m interested in what you have tried so far.
If I wouldn’t get good results using simple methods (let’s say by just concatenating the two columns) I’d try having an ensemble of two BERT models where one receives column_a and the other receives column_b.
Let’s say one column is an expert opinion about the medical condition of a patient, and the other is the patient’s opinion on his medical condition.
Then to me it makes sense to have two models one fine-tuned to the expert opinion and the other fine-tuned to the patient’s opinions.
Hopefully I didn’t say too much nonsense