There are 10 processes sharing a socket in my application.
They all wait for it to become readable using select.
But I notice in the application log that only 2 of these 10 processes any time the socket become readable.
What could be the reason?
I suspect what's happening is that the first process is waking up, returning from select(), and calling accept() before the subsequent context switch to the other processes can occur.
I'm not sure what select() actually blocks on or how it wakes up. I suspect when it does wake up from it's waiting on, it re-checks the queue to see if data is still available. If not, it goes back to waiting.
I'll double-down on my hypothesis as well. The fact that 2 processes are waking up is indicative of the fact that you have a dual-core processor. If you had a quad-core, you might see up to 4 processes wake up simultaneously.
One simple way to prove this theory: Put a 2 second sleep() call just prior to calling accept(). I suspect you'll see all 10 processes waking up and logging an attempt to call accept.
If your goal is to have N processes (or threads) servicing incoming connections, your approach is probably still good.  You could probably switch from doing a select() call on a non-blocking socket, to just using a blocking socket that calls accept() directly.  When an incoming connection comes in, one of the processes will return from accept() with a valid client socket handle.  The others will still remain blocked.
Why is only 2 of 10 selecting process notified when a socket become readable?