There might be a “fix”, but it would be interesting to see the performance improvements for some typical use cases first.
If the speedup is marginal (besides extreme cases, where an epochs use 1 sample), I’m not sure it’s worth the effort.
How big is the slowdown in each epoch for your current use case?