Hi @ptrblck!
I am converting this to ONNX then to *.pb (GraphDef using onnx-tensorflow project, which is OK in execution) and then to TFlite which converter dies.
I am using CUDA_VISIBLE_DEVICES=-1 and #613 fix for onnx_tensorflow project (for Dilation2d issue)
ONNX : 1.6.0
TF: 2.1.0
troch: 1.5.0
class SimpleNet(torch.nn.Module):
    def __init__(self):
        super(SimpleNet, self).__init__()

    def forward(self, x):
        x = F.interpolate(x, scale_factor=2, mode='nearest')#x.view(1, 3, 672, 672)
        return x

Most interesting part of output:
2020-05-13 10:47:51.019359: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: If
2020-05-13 10:47:51.019406: F tensorflow/lite/toco/import_tensorflow.cc:114] Check failed: attr.value_case() == AttrValue::kType (1 vs. 6)
Fatal Python error: Aborted

Whole output:
tensorflow.lite.python.convert.ConverterError: See console for info.
2020-05-13 10:47:50.562338: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/home/marat/anaconda3/lib
2020-05-13 10:47:50.562423: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/home/marat/anaconda3/lib
2020-05-13 10:47:50.562430: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2020-05-13 10:47:51.003834: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-05-13 10:47:51.009672: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3407930000 Hz
2020-05-13 10:47:51.010213: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a180ac8c00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-13 10:47:51.010228: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-13 10:47:51.011847: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-13 10:47:51.014197: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2020-05-13 10:47:51.014232: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: saturn
2020-05-13 10:47:51.014237: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: saturn
2020-05-13 10:47:51.014283: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 440.64.0
2020-05-13 10:47:51.014316: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 440.64.0
2020-05-13 10:47:51.014321: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 440.64.0
2020-05-13 10:47:51.019359: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: If
2020-05-13 10:47:51.019406: F tensorflow/lite/toco/import_tensorflow.cc:114] Check failed: attr.value_case() == AttrValue::kType (1 vs. 6)
Fatal Python error: Aborted

Current thread 0x00007f68c1c05700 (most recent call first):
  File "/home/marat/anaconda3/envs/cexp/lib/python3.7/site-packages/tensorflow_core/lite/toco/python/toco_from_protos.py", line 56 in execute
  File "/home/marat/anaconda3/envs/cexp/lib/python3.7/site-packages/absl/app.py", line 250 in _run_main
  File "/home/marat/anaconda3/envs/cexp/lib/python3.7/site-packages/absl/app.py", line 299 in run
  File "/home/marat/anaconda3/envs/cexp/lib/python3.7/site-packages/tensorflow_core/python/platform/app.py", line 40 in run
  File "/home/marat/anaconda3/envs/cexp/lib/python3.7/site-packages/tensorflow_core/lite/toco/python/toco_from_protos.py", line 93 in main
  File "/home/marat/anaconda3/envs/cexp/bin/toco_from_protos", line 8 in <module>
Aborted (core dumped)