By having more processes simulatenously doing random access IO, good chance you’ll start overloading whatever IO device you’re reading from, it’s not a friendly read pattern and you’ll likely have a lot of processes blocked on IO. There will be a number of workers beyond which there is no point in increasing for your system, start small and increase until it stops improving.
More worker processes will also increase the memory utilization.