Generally, multiple workers should speedup your data loading, although the sweet spot depends on your setup.
You could have a look at this post 6 for more information about potential bottlenecks.
Also, you might want to use DALI 5 in case it fits your use case.