It might be a good idea to use a pre-trained BERT model and apply fine-tuning on top of it. The BERT model is pretrained on a large corpus (some of which are finance related), so knowledge transfer should be possible. Also, BERT has performed really well on SuperGLUE for QnA.
You definitely need a dataset (larger the better) for transfer learning to work. Weak supervision may be helpful to create a decent quality dataset. Using crowdsourcing is another way (will cost some money). But, even with a small manually labeled dataset you may be able to achieve decent results.