§ 35 — libev Internals: Loop Phases · Watcher Types · 4-Heap Timer · ANFD Map · Fork Safety · Threading

1. § 35.1 — Why libev? From Raw epoll to an Event Library

epoll_wait() answers one question: which fds are readable or writable right now? A real server also needs timers, signal handling, child-process exit notification, cross-thread wakeup, and periodic callbacks — all wired together correctly, fork-safely, and with minimal overhead. libev provides this through a single abstraction: a watcher that attaches a callback to any event source.

Full Syscall Path: Application → libev → Kernel → Hardware

When your application calls ev_io_start(loop, w) for a socket fd, here is the exact path through the kernel before your callback fires:

1. ev_io_start() links the watcher into anfds[fd].head and adds fd to fdchanges[] (no syscall yet — deferred).
2. On the next ev_run() iteration Phase 2: epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &ev) — kernel inserts an epitem into the eventpoll RB-tree. It also installs ep_poll_callback as a wait-queue entry in the socket's sk_wq.
3. Phase 4: epoll_wait(epfd, events, 64, timeout_ms) — process sleeps on ep->wq.
4. NIC receives a packet → DMA → sk_data_ready() →ep_poll_callback() fires in softirq context → appends epitem to rdllist → wake_up(&ep->wq).
5. epoll_wait returns → libev looks up anfds[fd] → walks the ev_io watcher list → adds matching watchers to pendings[priority].
6. Phase 6: watcher callback fires in the event loop thread.

libev vs libevent vs libuv

Interactive: Backend Selection

2. § 35.2 — The Loop: struct ev_loop Internals

Every libev application creates one ev_loop per thread (or uses the default loop). The struct is the central hub connecting all watcher types to the backend. Its key fields fall into four groups:

ev_loop Field Map

ANFD Array — the fd → watcher Index

anfds[fd] is libev's merge point for all watchers on the same fd. When multiple ev_io watchers monitor the same fd, only ONE epitem exists in the kernel — the union of all interest masks is passed to epoll_ctl(MOD).

Interactive: ev_loop Struct Explorer

3. § 35.3 — The Loop Iteration: ev_run() Phase-by-Phase

Each call to ev_run(loop, 0) enters a tight loop. Every iteration has exactly 6 phases in strict order. Understanding each phase tells you exactly when a syscall is made and why.

Why fdchanges Defers epoll_ctl

A naive implementation would call epoll_ctl() immediately inside ev_io_start(). libev instead queues the fd into fdchanges[] and flushes them all in Phase 2. This batches multiple watcher changes on the same fd into a single epoll_ctl call. If a callback calls ev_io_stop then ev_io_start on the same fd, only one epoll_ctl(MOD) is issued — not a DEL followed by an ADD.

Interactive: ev_run() Phase Stepper

4. § 35.4 — ev_io Watcher: From Registration to Callback

ev_io is the most commonly used watcher. It bridges application fd interest to the backend epoll. The complete path from ev_io_start() to callback invocation touches three data structures: the anfds[] array, fdchanges[], and pendings[].

ev_io_start() → epoll_ctl → Dispatch

Interactive: ev_io Lifecycle (Start → epoll_ctl → Event → Stop)

Interactive: ANFD Merger — Multiple Watchers, One epitem

5. § 35.5 — Timer Internals: The 4-Heap Algorithm

libev uses a quaternary min-heap (4-heap) rather than the more common binary heap. The root always holds the earliest deadline; its value is peeked in Phase 3 to compute the epoll_wait timeout.

4-Heap Tree Structure

Each node at index i has children at indices 4i+1, 4i+2, 4i+3, 4i+4. Its parent is at (i-1)/4. Compared to a binary heap:

Heap Operations

• ev_timer_start(loop, w): compute w->at = ev_now() + w->after; append to timers[timercnt++];upheap(timercnt - 1) — O(log₄ n)
• ev_timer_stop(loop, w): swap timers[w->active] with timers[--timercnt]; call downheap() or upheap() on the moved element — O(log₄ n)
• Phase 5 expiry: while (timers[0]->at <= ev_now()) → pop root, mark pending, if w->repeat > 0: w->at = ev_now() + w->repeat → re-heap

Interactive: 4-Heap Visualizer

6. § 35.6 — ev_timer vs ev_periodic: Monotonic vs Wall Clock

Interactive: Timer Drift and NTP Correction

7. § 35.7 — Signal Watchers: ev_signal and the Signal Pipe

Signal handlers can interrupt any code — including malloc, epoll bookkeeping, and your own data structures. Calling libev dispatch functions from a signal handler is not async-signal-safe. libev's solution: the signal handler only writes 1 byte to a self-pipe (or increments an eventfd), and the actual ev_signal callback fires safely in the event loop thread.

Signal Delivery Path Step-by-Step

1. ev_signal_start(loop, w, SIGINT) → sigaction(SIGINT, &sa, NULL) installs handler → creates self-pipe with pipe2(O_NONBLOCK|O_CLOEXEC) → internal ev_io on pipefd[0] registered with epoll.
2. SIGINT arrives → kernel delivers signal → handler runs: signals[SIGINT].pending = 1; write(pipefd[1], "\0", 1) — async-signal-safe.
3. epoll_wait returns EPOLLIN on pipefd[0].
4. libev reads and drains the pipe; scans signals[]; dispatches all pending ev_signal callbacks.

Interactive: Signal Pipe Delivery

8. § 35.8 — All Watcher Types and Their Syscalls

ev_child — waitpid in the Event Loop

ev_child reuses the signal pipe: SIGCHLD fires → pipe wakes epoll → libev calls waitpid(-1, &status, WNOHANG) in a loop until it returns 0 → dispatches matching ev_child watchers by PID. The watcher's rstatus field contains the exit status for inspection in the callback.

ev_async — the Only Thread-Safe libev Call

ev_async_send(loop, w) may be called from any thread or signal handler. It calls write(loop->async_fd, &one, 8) on an eventfd, which is specified as atomic. The main loop's epoll_wait returns EPOLLIN on async_fd; libev reads to reset the counter and dispatches all pending ev_async watchers.

9. § 35.10 — Fork Safety: Why epoll Breaks After fork()

fork() copies the parent's address space, including ev_loop and its backend_fd (the epoll fd). Both parent and child now hold file descriptors pointing to the same kernel eventpoll object. The child's new sockets are not registered, and calling epoll_wait in the child monitors the parent's file descriptors — a serious bug.

ev_loop_fork() — What It Does

1. close(old_epfd) — only closes this process's fd; the parent's epoll is unaffected.
2. epfd = epoll_create1(EPOLL_CLOEXEC) — fresh kernel eventpoll for the child.
3. Iterate all active anfds[]: for each entry with non-zero emask, call epoll_ctl(new_epfd, ADD, fd, mask) — O(active fds).
4. Re-initialize signal pipe and async_fd.
5. curpid = getpid() — resets fork detection for the child's own future forks.

libev detects fork automatically: at the start of each ev_run()iteration it compares getpid() against loop->curpid. If they differ, it calls ev_loop_fork() before proceeding. You can also call it explicitly right after fork().

Interactive: fork() + epoll Bug and Fix

10. § 35.11 — Threading Model and ev_async

libev's threading rule is simple: one event loop per thread; never call ev_io_start/stop or ev_run from a different thread. The only cross-thread primitive is ev_async_send().

Correct Multi-Thread Pattern

1. Main thread creates ev_async watcher, starts it, then calls ev_run().
2. Worker threads do CPU-intensive work (compute, compress, encrypt).
3. Worker completion: ev_async_send(loop, &async_w) — writes to eventfd, thread-safe.
4. Main thread wakes from epoll_wait → dispatchesev_async callback → safely calls ev_io_start/stop, enqueues response, modifies loop state.

Interactive: Cross-Thread Wakeup with ev_async

11. § 35.14 — Worked Example: Echo Server O(k) Dispatch

Sequence: Client Connect → Data → Echo

Why O(k), Not O(N)

With 10,000 connected clients and only 3 sending data simultaneously, epoll_wait returns 3 events. libev looks up anfds[fd_A], anfds[fd_B], anfds[fd_C] — the other 9,997 clients are never touched. Compare with select/poll which scans all 10,000 fds every wakeup.

Interactive: O(k) Dispatch Demo

12. § 35.15 — libev vs libevent vs libuv

Interactive: Timer Heap and Memory Comparison

13. Interview Prep