On Linux we could use the non-portable accept4() with a SOCK_NONBLOCK option to do this with one system call (accept4) instead of 3 (1 accept and 2 fcntl).
Cons: Linux only checks i.e. complexity. We still need to maintain the current path for non-Linux machines. Pros: Increased speed on Linux which is probably the biggest deployment target.
I am not alleging that this is a bottleneck at all. If there is interest from the maintainers, I could attempt to write a patch.