Yes I think that is a good point - if all the computations are assigned to GPU and all are being executed on a single stream then you won’t get a speedup from additional parallelism. What I’d do here normally is a bit low level (that’s the area I’m normally  ) and dump the graph to see if there are any unexpected edges that are inhibiting control flow (these days I’d dump the Graphdef, convert to TFG (tfg-translate tool) and then look at the output as it is readable, before TFG I’d pipe it through to Graphviz file) and then run with vmodule=executor=1 to see exactly what’s being run and where (it can produce a lot of output even for small graphs).
I don’t know who from TFRT team is on here to ping in for comment, let me check