Are you able to post a repro for the internal assert failure?
It sounds like you’re asking for structural typing 2 instead of nominal typing 2. TorchScript is nominally typed, so we don’t have a blessed way to do exactly what you’re asking. (It does look like some version of structural typing is supported, but that is likely an accident 1).
detectron2.structures.box should be included with the serialized code of your TorchScript model, so you should be able to reconstruct it inside the JIT (though the API to do this may not exist/be pretty). Once we fix the above bug we can probably figure out some hack to make this still work.