I’m trying to run many small models in parallel. I noticed with my first implementation (just calling each of the models in a loop), that GPU utilization was very low, about 25%. So I did some research and it sounds like what I want are cuda streams. So I initialized two streams and broke my models into two groups and I ran the models now wrapped in a with block. The code ran fine, but GPU utilization remained low at about 25%.
Am I misunderstanding the purpose or function of streams?