§ 34 — epoll Deep Dive: Data Structures · ep_poll_callback · LT/ET · EPOLLONESHOT · io_uring · Reactor/Proactor

1. § 34.1 — epoll Data Structures

Three kernel structs form the backbone of every epoll instance. Understanding them is the prerequisite for every epoll interview question.

struct eventpoll — the top-level container

Created by epoll_create() and backed by an anonymous inode. The returned epfd is a regular file descriptor pointing to this struct.

struct epitem — one per registered fd

Created by epoll_ctl(EPOLL_CTL_ADD). Lives in the RB-tree and optionally in the rdllist when its fd is ready.

struct eppoll_entry — the wait queue hook

The bridge between the monitored socket and the epoll instance. Its wait.func = ep_poll_callback is what fires when the socket becomes ready.

Full Data Structure Chain

The complete ownership chain from eventpoll down to the socket wait queue. This is what makes epoll O(1) — the callback is installed at registration time, not at wait time.

Interactive: epoll Data Structure Explorer

Add and remove fds, trigger I/O events, and call epoll_wait(). Watch the RB-tree rebalance on insert and epitems move into the rdllist on I/O events.

2. § 34.2 — ep_poll_callback: The Key to O(1)

The fundamental difference between epoll and select/poll is when the kernel identifies ready fds. select/poll finds them at wakeup by scanning everything. epoll uses a callback installed at registration time — ready fds are delivered directly, without a scan.

Registration: epoll_ctl(ADD) installs the callback

ep_poll_callback() — called from softirq context

This function runs in softirq/interrupt context — not in any process context. It must be fast and non-blocking.

epoll_wait() flow

Full flow: NIC interrupt → epoll_wait returns

Why O(1)? — epoll vs select wakeup comparison

The critical difference: select wakes all waiting processes and each re-scans all n fds. epoll fires the callback only for the ready fd, appends to rdllist in O(1), and wakes only epoll_wait — which then drains only the k ready entries.

Interactive: ep_poll_callback Trace — 1000 fds, 3 ready

Step through the full path: NIC interrupt → softirq → ep_poll_callback fires on exactly 3 fds → rdllist gets 3 entries → epoll_wait returns 3. 997 fds are never touched.

3. § 34.3 — Level Trigger (LT) vs Edge Trigger (ET)

LT and ET describe when epoll reports a fd as ready. The difference is consequential: ET requires a mandatory drain loop, and getting it wrong silently loses data.

Level Trigger (LT) — default

Edge Trigger (ET) — EPOLLET flag

ET drain loop — mandatory pattern

Interactive: LT vs ET Simulator

Switch between LT and ET. Send 100 bytes, read a partial amount with the slider, then call epoll_wait again — see exactly what each mode reports and what gets lost.

4. § 34.4 — EPOLLONESHOT and EPOLLEXCLUSIVE

Two flags that solve multi-threaded epoll problems: EPOLLONESHOT prevents duplicate fd delivery across threads; EPOLLEXCLUSIVE prevents thundering herd on a shared listen socket.

EPOLLONESHOT

After epoll_wait() returns an fd registered with EPOLLONESHOT, the kernel automatically disables that fd — as if EPOLL_CTL_DEL was called. No other thread can get the same fd on a subsequent epoll_wait() until you explicitly re-arm it.

EPOLLEXCLUSIVE (Linux 4.5+)

When multiple threads each have their own epoll fd, all watching the same listen socket, a new connection wakes all of them — the classic thundering herd. EPOLLEXCLUSIVE uses WQ_FLAG_EXCLUSIVE in the wait queue entry so that only one epoll fd's epoll_wait() is woken per event.

Comparison: EPOLLONESHOT vs EPOLLEXCLUSIVE vs SO_REUSEPORT

Interactive: Thundering Herd Demonstration

Toggle EPOLLEXCLUSIVE and send connections to see the difference. Without it, all 4 threads wake and 3 waste a context switch on EAGAIN.

5. § 34.5 — ET Drain Rule and Common Pitfalls

ET mode is faster than LT in steady state, but it has sharp edges. All of these pitfalls cause silent data loss or hangs — no error is returned.

The Drain Rule

Pitfall 1 — Read pitfall (most common)

Pitfall 2 — EPOLLOUT write readiness

EPOLLOUT in ET mode fires when the send buffer transitions from full → not full (state change). If you register EPOLLOUT and the buffer is never full, it fires on everyepoll_wait() — a busy loop. Pattern: enable EPOLLOUT only when a write returns EAGAIN (buffer full); disable it once writes succeed.

Pitfall 3 — Accept loop on ET listen socket

Pitfall 4 — ET + blocking fd = hang

ET requires O_NONBLOCK. If you use a blocking fd with ET, the drain loop calls read() when the buffer is empty — the call blocks indefinitely instead of returning EAGAIN. The thread hangs. Always set fcntl(fd, F_SETFL, O_NONBLOCK) before registering with EPOLLET.

Pitfall 5 — Self-pipe trick (pre-eventfd)

Before eventfd() and timerfd(), waking epoll_wait() from another thread required a pipe: register pipefd[0] with EPOLLIN, write one byte to pipefd[1] to trigger the wakeup. Now use eventfd(0, EFD_NONBLOCK) instead — lighter and correct with both LT and ET.

Interactive: ET Drain Pitfall — 3 data bursts

Step through 3 data bursts in WRONG mode (read once per event) vs CORRECT mode (drain loop). Watch the buffer state and see exactly how much data is lost.

6. § 34.6 — epoll vs kqueue vs IOCP vs io_uring

epoll is Linux-only and readiness-based (reactor). Other platforms and newer Linux APIs offer different trade-offs — from the general-purpose kqueue to the truly async io_uring.

epoll vs kqueue

IOCP (Windows) — Proactor

IOCP is fundamentally different: the kernel completes I/O into your buffer, then posts a completion packet. You never call read() — the data is already there when your handler runs. This is the proactor pattern.

io_uring (Linux 5.1+) — Async Ring

io_uring uses two shared memory rings between kernel and userspace. Submit I/O without a syscall; collect completions without a syscall. With SQPOLL, even submission is zero-syscall — a kernel thread polls the SQ ring continuously.

Quick API Comparison

Demo 3509 — Syscall Count: epoll vs io_uring

Run the simulation to see exact syscall counts at various connection scales. The SQPOLL column drops to near-zero — that is the fundamental advantage of the shared ring model.

7. § 34.7 — Reactor vs Proactor Pattern

epoll implements the reactor pattern: the kernel tells you a fd is ready, then your code calls read(). io_uring and IOCP implement the proactor pattern: the kernel completes I/O into your buffer and notifies you when it's done — no read() needed.

Reactor Pattern (epoll)

Proactor Pattern (io_uring)

Side-by-side Loop Comparison

Reactor Event Loop — Step by Step

Demo 3510 — Reactor Event Loop Simulator

Add connections, ARM them into epoll, then run epoll_wait to watch the dispatch cycle. Toggle EPOLLONESHOT to see the re-arm step appear after each write.

8. § 34.8 — Echo Server Evolution

The echo server is the canonical benchmark for I/O multiplexing. Each stage removes a bottleneck from the previous one — from blocking single-client all the way to io_uring's zero-syscall proactor.

Stage Explorer

Click a stage in the table above or use the tabs below to explore each implementation.

/* Stage 1: blocking single-client echo server */
int main(void) {
    int lfd = tcp_listen(8080);

    for (;;) {
        /* blocks here — only ONE client at a time */
        int cfd = accept(lfd, NULL, NULL);

        char buf[4096];
        ssize_t n;
        /* blocks in read() — server frozen for all others */
        while ((n = read(cfd, buf, sizeof buf)) > 0)
            write(cfd, buf, n);

        close(cfd);
        /* next accept() — previous client must be gone */
    }
}
/* syscalls per request: accept(1) + read(1) + write(1) = 3
   concurrency: 1 (second client waits in kernel accept queue) */

Demo 3511 — Echo Server Evolution: Syscall & Capacity Comparison

Run to see the quantitative breakdown: syscalls per second at C10K scale and the O(n) cost of poll() vs O(k) of epoll at 10,000 connections.

Interview Prep — Must-Know Questions