as far as i know, you can’t use models that are fine-tuned for summarisation directly for question-answering as the tasks are quite different:

in the summarisation case, the model is provided with (document, summary) tuples
in the question answering case, the model is provided with (question, document, answers) tuples (with potentially more subfields like answer_start if we’re dealing with the extractive case where we need to identify a span of text where the answer exists)

so the main problem i see with naively using a summarisation model for QA is that the model does not expect a question / query in its inputs and so will not generate an answer.
btw if you’re looking to build an end-to-end QA system, there’s a neat library called haystack that is based on transformers and provides a lot of nice functionality to store documents, query them etc: https://haystack.deepset.ai/