Hi @gruffgoran, your use cases sound like a perfect match for the ONNX 1 format 
Having said that, you might be able to get a quick win by trying something like the following (see docs 1):
tf_model = TFBertForSequenceClassification.from_pretrained("KB/bert-base-swedish-cased", from_pt=True)

From here you can then run inference / fine-tune etc using TensorFlow.
If you want to go the ONNX route, the idea would be to convert PyTorch → ONNX and then load the ONNX model in TensorFlow. Details on doing the conversion can be found here: Exporting transformers models — transformers 4.3.0 documentation 2