This is an excellent area to look for optimizations, but my instinct is that a &quot;pre-runner&quot; approach is going to be of very limited use.  If it&#x27;s 1:1 and the problem is bound by memory latency, the pre-runner is going to go no faster than the working thread.   If it&#x27;s N:1, why not simplify the architecture by doing the computation in the pre-runner?<p>I think the only case it would make sense is with a heterogenous architecture where you have unlimited small cores available, but in this case there may be better solutions.  There&#x27;s nothing wrong with spreading this work across multiple cores, but in most cases every core will have underutilized memory operation ports.  What we need are compilers (and languages) that are smart enough to make effective use of these.<p>I think this is much better solved by small batch processing and more effective use of software prefetch.   Instead of having each core handling one &quot;request&quot; at a time, batch them so that it&#x27;s handling about 5 at a time (which happens to be half the per core limit of outstanding memory accesses).   As a batch, issue all the lookups (with prefetch or directly to registers), then as a batch process the now in-cache results of the previous requests.  Magic --- by keeping the request queue full, you&#x27;ve increased your throughput 5x[+] at the cost of a small additional static latency while you build your batches!<p>[+] This would be theoretical for a memory-latency-bound pointer-chasing scenario.  There are caveats once you get away from this to the real world, but 2x-4x is achievable if you can parallelize your latencies.