Thanks for suggestion, really appreciate it.
Actually this implementation that you posted is more like Mixture of Experts than CondConv: see discussion github prstrive/CondConv-tensorflow/issues/1
And instead of doing the CondConv on previous convolution outputs (the number of outputs is always the same), we need to do it on instances of detected objects (the number of objects is different for every image).