Epoll is broken – part 2/2
I am making a new post because I feel like this is getting out of hand. There is a lot of misinformation that needs to be cleared up.<p>When you are working with system calls you use fds to tell the kernel what file you are working with. These are simply numbers, an int to be exact. When you call a system call that takes in a fd you pass the int that references the file you want to work with and it is looked up in a table to find the kernel structure that represents the file. When you dup a fd all that does is add a entry to the lookup table to point to the same kernel structure -- increasing the refcount at the same time. The ref count lives on the kernel structure, not with the int. The fd system is real simple. For open, dupe, dup2… will add an entry. While close will remove the entry. If the entry is removed the fd is now useless for any communication to the kernel over system calls. This is because the lookup table will not be able to translate your fd to a kernel structure.
Now the question is epoll broken or have a bug? I don’t think so. You just have to keep in mind you are using a system call and not a library. This is controlled access to kernel level functions.
In the example adding a fd to epoll, then duping it followed by a closing is providing some surprising results. Understanding how userspace talks to kernel space and how system calls translate into running kernel code is key to understanding why this is not a bug but just how things work. You might also be surprised that other system calls behave the same way!
When you tell epoll to wait you use a fd to tell it which file you want to wait on. You are not telling a library, or a userspace program to do this. You are telling the kernel. So the fd is translated into a kernel object and that is what epoll is working with. Only when adding and removing is this translation done. The kernel subsystem that provides epoll will continue to work with kernel objects -- not fds.<p>When you call dup you are adding a new entry to your programs fd lookup table for a new fd but pointing to the same kernel structure -- at the same time you are also increment the refcount of the kernel structure. The file is not actually closed until both the original fd and the duped fd have had closed called on it, or more correctly, until the refcount of the kernel structure is zero. This means that if you tell epoll to wait on a file and do so by an fd, and then close the fd -- you will only see a close event if the refcount is zero. By calling dup before closing your original fd you have incremented the ref count and thus the file remains open -- no close event.<p>If you then try to remove your file from epoll using the now invalid (closed) fd it will fail. This is because the fd is no longer in the fd lookup table and the system call can not translate your fd into a kernel object.
Now you might say this is not how other system calls work. As in another thread pointed out that “read” does not behave this way. To the contrary -- read actually works the same way. If you dup your socket fd, and use a thread to close your socket fd while in an active read -- your read will not fail. It will finish out delivering the data to you. This is because the fd is not the actual object you are working with. It is just a reference to the object. Onces you enter the read function you are in kernel land. And the kernel only knows about the kernel object. Because you duped the fd before closing it the kernel object has a ref count of 2 and your close while read leaves the ref count at 1 thus there is nothing for the kernel to do other than finish out its read. If you attempt to use the now closed and invalid fd for a subsequent read, or close you will get a EBADF error -- much like you do when you attempt to use it with epolls functions to remove the file.