Generally we dont have control over which cores thread get assigned to. XNNPACK for fp32 compute has the smarts where it picks different implementation depending on which core thread is mapped to but that still wont address the issue we see here.  In terms of latency I think sticking to 2 threads will be the best. However in terms of utilization it is not the best. Addressing this is slightly more complex.
I have another set of followups:

Is your end goal maximizing resource utilization in such a way that it reduces latency?
Can you open issue on pytorch github and assign it to me.

I do think this is an interesting problem that we may need to look into.