When I make a model which requires KL, I usually run it once while testing without the KL to make sure everything is hooked up and working correctly. What I have seen there is that without the KL, the variance of whatever distribution that is being reparameterized collapses to look more deterministic.
It probably depends on the specific data and problem as to whether this would help or hurt. But if the variance does go to zero and it helps, then you probably don’t need a reparamterized model in the first place would be my guess.