Linux Kernel

Deep Dive — Study Catalog

19 parts · 180+ sections · interview-ready depth on networking, DPDK, memory, storage, virtualization, the C toolchain, threads/processes, and Unix file I/O.

Reference Books

CodeBookFocus
LKDLinux Kernel Development (3rd ed.) — Robert LoveCore concepts, scheduling, memory, sync
ULKUnderstanding the Linux Kernel (3rd ed.) — Bovet & CesatiDeep internals, data structures, boot
LKNLinux Kernel Networking — Rami RosenFull network stack internals
ULNIUnderstanding Linux Network Internals — Benvenutisk_buff, protocols, drivers
LDDLinux Device Drivers (3rd ed.) — Corbet et al.Drivers, PCI, block/char devices
TCPIPTCP/IP Illustrated Vol.1&2 — W. Richard StevensProtocol deep-dive
DPDKDPDK Programmer's Guide — DPDK ProjectOfficial DPDK reference
PPHIs Parallel Programming Hard? — Paul E. McKenneyLocking, RCU, memory ordering
PLKAProfessional Linux Kernel Architecture — MauererComprehensive architecture
K&RThe C Programming Language (2nd ed.) — Kernighan & RitchieC language ground truth
CSAPPComputer Systems: A Programmer's Perspective — Bryant & O'HallaronSystems / linking / ABI / memory hierarchy
APUEAdvanced Programming in the UNIX Environment — Stevens & RagoPOSIX APIs, signals, processes
TLPIThe Linux Programming Interface — Michael KerriskDefinitive Linux syscall reference
LSPLinux System Programming (2nd ed.) — Robert LovePractical syscall + glibc usage
L&LLinkers and Loaders — John R. LevineELF, relocation, dynamic linking
EXPCExpert C Programming: Deep C Secrets — Peter van der LindenPointers, declarations, gotchas
MUELModern Embedded Systems / Mastering Embedded Linux — Chris SimmondsCross-compilation, toolchain, sysroot
PWPTProgramming with POSIX Threads — David R. Butenhofpthread bible: mutex, cond var, cancel, TSD
AOMPThe Art of Multiprocessor Programming — Herlihy & ShavitLock-free, wait-free, transactional memory theory

Part ISystem Boot & Initialization

ULK Ch.1 · LKD Ch.1

Part IIMemory Management

ULK Ch.2,8,9 · LKD Ch.12,15,16

  • §2.1Physical Memory Layout (NUMA / Zones / struct page)
  • §2.2Buddy Allocator
  • §2.3Slab / Slub Allocator
  • §2.4Virtual Memory & Paging (4-level PT, TLB, Huge Pages)
  • §2.5Process Address Space (mm_struct, VMA, mmap, COW)
  • §2.6Page Cache & Buffer Cache (LRU/MRU, writeback)
  • §2.7Memory-Mapped I/O & DMA (ioremap, IOMMU)

Part IIIProcess Management & Scheduling

LKD Ch.3,4,5 · ULK Ch.3,6,11

  • §3.1Process & Thread Data Structures (task_struct)
  • §3.2Process Lifecycle (fork → exec → exit)
  • §3.3CPU Scheduling (CFS, SCHED_FIFO, load balancing)
  • §3.4Preemption & Context Switch
  • §3.5Interrupts & Bottom Halves (IRQ → softirq → NAPI)
  • §3.6Kernel Timers (jiffies, timer wheel, hrtimer, NO_HZ)
  • §3.7Kernel Data Structures (list, hlist, rbtree, xarray, bitmap)

Part IVSystem Calls & User-Kernel Interface

LKD Ch.5 · ULK Ch.10

  • §4.1System Call Mechanism (syscall, vDSO)
  • §4.2Key Networking Syscalls (socket, epoll, io_uring)
  • §4.3ioctl & Netlink
  • §4.4Sync / Async I/O (blocking, O_NONBLOCK, epoll, io_uring, sendfile)
  • §4.5Signals

Part VInterprocess Communication

LKD Ch.17 · ULK Ch.19

  • §5.1Pipes & FIFOs
  • §5.2POSIX IPC (msg queues, shared memory, semaphores)
  • §5.3Futex (fast userspace mutex)
  • §5.4Unix Domain Sockets

Part VIParallel Programming & Synchronization

PPH all · LKD Ch.9,10 · critical interview section

  • §6.1Hardware Foundations (pipeline, store buffer, false sharing, NUMA)
  • §6.2Cache Coherence Protocols (MESI / MOESI)
  • §6.3Memory Barriers & Ordering (smp_mb, acquire/release, C11 model)
  • §6.4Atomic Operations (CAS, fetch-add, ABA problem)
  • §6.5Counting & Per-CPU Counters
  • §6.6Locking Design Patterns (lockdep, ordering, granularity)
  • §6.7Spinlock Internals (ticket → MCS → qspinlock)
  • §6.8Sleeping Locks (mutex, rwsem, semaphore)
  • §6.9Sequence Locks (seqlock_t)
  • §6.10RCU Deep Dive (grace period, call_rcu, SRCU)
  • §6.11Lock-Free & Wait-Free Algorithms (Treiber stack, MS queue, hazard ptrs)
  • §6.12Per-CPU Data & Data Ownership
  • §6.13Deferred Processing (refcount, RCU reclaim, workqueue)
  • §6.14Parallel Design Patterns (batch, sharding, pipeline, map-reduce)
  • §6.15Debugging (lockdep, KCSAN, TSan, perf lock)
  • §6.16Linux Kernel Memory Model (LKMM, litmus tests)

Part VIINetwork Stack Internals

LKN all · ULNI all · TCPIP Vol.1&2

  • §7.1Socket Layer (struct socket → sock → tcp_sock)
  • §7.2sk_buff — Core Data Structure (GSO, GRO, zero-copy)
  • §7.3Network Device Layer (net_device, NAPI, TX/RX path)
  • §7.4Traffic Control & QoS (Qdisc, HTB, cls_flower)
  • §7.5IP Layer (ip_rcv, fragmentation, ICMP)
  • §7.6Routing Subsystem (fib_table, LPM, fib_trie, policy routing)
  • §7.7Neighbor Subsystem (ARP, NDP, proxy ARP, GARP)
  • §7.8TCP/IP Deep Dive (state machine, CUBIC, BBR, SYN Cookie)
  • §7.9UDP & Multicast socket
  • §7.10Netfilter & NAT (5 hooks, conntrack, DNAT/SNAT)
  • §7.11Network Bridge (FDB, MAC learning, STP)
  • §7.12GRE & VXLAN Tunnels
  • §7.13IPv6 (NDP, RA/RS, DHCPv6, extension headers)
  • §7.14Multicast (IGMP, MLD, PIM-SM, RPF)
  • §7.15DHCP Protocol & Proxy DHCP (DORA, DHCPv6)
  • §7.16Routing Protocols (RIP, OSPF, BGP, BIRD)

Part VIIINetwork Drivers & Hardware Interface

LDD Ch.17 · ULNI Ch.8,9 · DPDK docs

Part IXDPDK (Userspace Networking) — Deep Dive

DPDKPG all · critical interview section

  • §9.1Why DPDK: Kernel Bypass Architecture (UIO vs VFIO)
  • §9.2DPDK Startup Flow (rte_eal_init 10 steps)
  • §9.3PMD Poll Mode Driver (descriptor rings, DD bit, doorbell)
  • §9.4NIC Multi-Queue, RSS & Flow Classification (rte_flow)
  • §9.5Memory: Huge Pages, Mempool & Mbuf
  • §9.6NUMA-Aware Programming
  • §9.7Lock-Free Ring Buffer (rte_ring — CAS algorithm)
  • §9.8Cache Optimization (alignment, prefetch, DDIO, WC)
  • §9.9ACL & LPM Classification (DIR-24-8, rte_hash cuckoo)
  • §9.10SR-IOV in DPDK (VF pass-through, VFIO)
  • §9.11Hardware Offload (checksum, TSO, LRO, VXLAN offload)
  • §9.12SIMD in DPDK (SSE4.2, AVX2, AVX-512)
  • §9.13Virtio, vhost-user & vDPA
  • §9.14Pipeline & Run-to-Completion Models
  • §9.15DPDK Hot Upgrade (multi-process, shared memory)
  • §9.16Session Management (fast path / slow path)
  • §9.17QoS in DPDK (rte_meter, rte_sched, distributed rate limit)
  • §9.18DPDK Bonding & Link Aggregation
  • §9.19OVS-DPDK (EMC, dpcls, megaflow, VXLAN)

Part XXDP (eXpress Data Path)

kernel docs · eBPF/XDP reference

Part XIStorage & File Systems

LKD Ch.13,14 · EXT4 docs

Part XIIVirtualization & Containers

LKD Ch.18 · KVM docs · LXC/Docker

Part XIIIHardware Communication & PCIe

PCIe spec · DPDK docs · LDD Ch.15

Part XIVDebugging & Profiling Tools

LKD Ch.18 · perf docs · kernel tracing

Part XVDistributed Systems (Raft)

Ongaro & Ousterhout

Part XVILoad Balancing & Consistent Hashing

Resume projects · DPDK LPM/ACL

Part XVIIC Standard Library, Toolchain & ABI

CSAPP · APUE · TLPI · Linkers and Loaders · Expert C · interview-critical

  • §17.1C libc Landscape (glibc vs musl vs uClibc vs Bionic vs newlib vs dietlibc — size/perf/license trade-offs)
  • §17.2User → Kernel Path: libc Wrapper → syscall Instruction → kernel handler (errno convention, INTERNAL_SYSCALL macros)
  • §17.3vDSO & Pseudo Syscalls (clock_gettime/gettimeofday/time/getcpu run in user space — no ring switch; auxv AT_SYSINFO_EHDR)
  • §17.4Userspace Memory Allocator (ptmalloc2 arenas/bins/fastbins/tcache; brk vs mmap threshold M_MMAP_THRESHOLD; fragmentation)
  • §17.5Allocator Alternatives (jemalloc, tcmalloc, mimalloc, scudo) — per-thread cache, size classes, why DPDK uses rte_malloc instead
  • §17.6stdio Buffering (full/line/unbuffered, FILE*, setvbuf, fflush — fork() double-flush pitfall)
  • §17.7POSIX Thread Library (NPTL, pthread_create → clone(CLONE_VM|...), 1:1 model, futex-backed mutex, pthread_key TSD)
  • §17.8Thread-Local Storage (__thread / _Thread_local, ELF .tdata/.tbss, TLS models GD/LD/IE/LE, FS/GS register)
  • §17.9Dynamic Linker / Loader (ld.so, PLT/GOT, lazy binding via _dl_runtime_resolve, LD_PRELOAD, LD_LIBRARY_PATH, RPATH vs RUNPATH)
  • §17.10Runtime Loading (dlopen/dlsym/dlclose, RTLD_LAZY vs RTLD_NOW, RTLD_GLOBAL, plugin pattern)
  • §17.11ELF File Format (header, PHT vs SHT, segments LOAD/DYNAMIC/INTERP, sections .text/.rodata/.data/.bss/.init_array, symbol/relocation tables)
  • §17.12C Compilation Pipeline (cpp preprocess → cc1 compile → as assemble → ld link; gcc -E/-S/-c/-o; -save-temps; intermediate .i/.s/.o)
  • §17.13Static vs Dynamic Linking (.a archive ar/ranlib, .so, -static, ldconfig cache /etc/ld.so.cache, soname/version, symbol resolution order)
  • §17.14Position Independent Code (PIC -fPIC for .so, PIE -fPIE for ASLR, GOT/PLT indirection cost, copy relocations)
  • §17.15Cross Compilation (target triplet arch-vendor-os-libc e.g. aarch64-linux-gnu, --host/--build/--target, sysroot, multilib)
  • §17.16Cross Toolchains (crosstool-NG, Buildroot, Yocto/OE, Linaro GCC; how to build a glibc/musl cross toolchain from scratch)
  • §17.17Library Inspection Tools (ldd, ldconfig -p, nm, objdump -d/-h/-r, readelf -a/-d/-l, strings, file, strip, ar, c++filt, addr2line, size, ranlib)
  • §17.18Runtime Tracing (strace -e trace=... syscalls, ltrace library calls, LD_DEBUG=symbols/bindings/files, perf trace)
  • §17.19Calling Conventions / ABI (SysV AMD64: rdi/rsi/rdx/rcx/r8/r9 + xmm0–7, red zone 128B, stack 16-byte alignment; x86 cdecl/stdcall; ARM AAPCS r0–r3)
  • §17.20C ↔ C++ ABI (Itanium C++ ABI, name mangling _ZN..., vtable/RTTI layout, extern "C", ABI breakage between GCC versions)
  • §17.21errno & Reentrancy (errno is per-thread macro → *__errno_location(); _r reentrant variants; async-signal-safe set)
  • §17.22Atomics & Memory Model (C11 <stdatomic.h>, atomic_*, memory_order_relaxed/acquire/release/seq_cst, libatomic fallback)
  • §17.23glibc Internals (rtld bootstrap, hidden_def, IFUNC dispatch e.g. memcpy AVX/SSE, symbol versioning GLIBC_2.x, vtable hardening)
  • §17.24Security Hardening (FORTIFY_SOURCE=2/3, -fstack-protector-strong canary, ASLR, RELRO partial/full, NX/DEP, CFI, _GLIBCXX_ASSERTIONS)
  • §17.25Common C Libraries — System: libpthread, librt, libdl, libm, libresolv, libcrypt (now libxcrypt)
  • §17.26Common C Libraries — Crypto/Net: OpenSSL (libssl/libcrypto), libcurl, c-ares, libpcap, libnl (netlink), libnftnl, libnfnetlink
  • §17.27Common C Libraries — Event Loops: libev, libevent, libuv (cross-platform), libaio, liburing (io_uring)
  • §17.28Common C Libraries — DPDK/Kernel-bypass: librte_eal/mempool/mbuf/ring/hash/lpm, libnuma, libhugetlbfs, libpci, liburcu
  • §17.29Common C Libraries — eBPF: libbpf, libxdp, libelf, libdebuginfod, BCC (Python-bound)
  • §17.30Common C Libraries — Compression: zlib, libbz2, liblzma, lz4, libzstd, snappy-c
  • §17.31Common C Libraries — Serialization: protobuf-c, msgpack-c, libcjson, libxml2, libyaml
  • §17.32Common C Libraries — Data Structures: glib (GLib), libavl, libcuckoo, judy arrays
  • §17.33Sanitizers & Static Analysis (ASan/MSan/TSan/UBSan, valgrind memcheck/helgrind/cachegrind, clang-tidy, cppcheck, scan-build)
  • §17.34Build Systems & Package Discovery (Make, CMake, Meson, autotools; pkg-config --cflags/--libs, .pc files)

Part XVIIIThreads, Processes & User-Space Synchronization

TLPI Ch.24-30 · APUE Ch.8,11,12 · Butenhof · interview-critical

Part XIXFile Descriptors, File I/O & Unix Plumbing

TLPI Ch.4-19,44-50,63 · APUE Ch.3,4,14,15,17 · interview-critical

Appendix — Key Kernel Data Structures

StructureHeaderPurpose
struct task_structlinux/sched.hProcess descriptor
struct mm_structlinux/mm_types.hProcess address space
struct pagelinux/mm_types.hPhysical page frame
struct sk_bufflinux/skbuff.hNetwork packet
struct socknet/sock.hSocket state
struct net_devicelinux/netdevice.hNetwork interface
struct Qdiscnet/sch_generic.hTraffic control queue
struct neighbournet/neighbour.hARP/NDP entry
struct rtablenet/route.hIPv4 route cache entry
struct nf_connnet/netfilter/nf_conntrack.hConnection track entry
struct inodelinux/fs.hFile system inode
struct biolinux/bio.hBlock I/O unit
struct vm_area_structlinux/mm_types.hVMA — memory region

Appendix — C libc Implementations

ImplementationTypical UseTrade-off
glibcDesktop / server Linux (Debian, RHEL, Ubuntu)Feature-complete, large (~10 MB), LGPL, IFUNC dispatch, NSS plugin
muslAlpine, static binaries, container imagesTiny (~600 KB), MIT, simpler malloc, stricter POSIX, slower DNS
uClibc / uClibc-ngEmbedded MMU-less or tight MIPS/ARM systemsConfigurable feature subset, smaller than glibc
BionicAndroidBSD-derived, no full POSIX, fortified, integrates with linker64
newlibBare-metal / RTOS (Cortex-M, RISC-V)Provides libc without OS; you supply syscall stubs
dietlibcStatic minimal binariesOptimised for size, GPL, partial coverage
picolibcEmbedded successor to newlib + AVR libcTiny, BSD-licensed, TLS-aware
Cosmopolitan libcCross-OS portable executables (APE)Single binary runs on Linux/Mac/Win/BSD

Appendix — Toolchain & Library Inspection Tools

ToolLayerWhat it answers
fileELF header32/64-bit? statically vs dynamically linked? stripped?
lddDynamic linkerWhich .so files this binary loads, and from where
ldconfig -pld.so cacheList all libraries the loader knows about
readelf -aELF metadataSections, segments, dynamic tags, symbol versions, NEEDED
objdump -dDisassemblyPer-section disassembly, source interleave with -S
objdump -h / -rSections / relocationsSection flags, relocation entries
nmSymbol tableDefined / undefined / weak symbols (T/U/W)
stringsRaw bytesEmbedded literals, version banners, hidden paths
stripSection stripperRemove debug / symbols to shrink binaries
ar / ranlibStatic archiveCreate / inspect / index .a libraries
c++filtDemanglerDecode Itanium-mangled C++ names like _ZN3foo3barEv
addr2lineDebug infoTranslate runtime address back to file:line
sizeSection sizingtext/data/bss bytes per object
pkg-configBuild flagsResolve --cflags / --libs from .pc files
straceSyscall traceEvery syscall a process makes (with -c, -e, -f)
ltraceLibrary traceEvery dynamic library call (printf, malloc...)
LD_DEBUG=...Loader tracesymbols / bindings / files / reloc — see ld.so resolve
LD_PRELOADSymbol overrideInject a .so to replace functions (interposition)
valgrindUserspace VMmemcheck, helgrind (race), cachegrind, massif

Appendix — Common C Libraries Cheat Sheet

LibraryDomainWhy it matters
libpthreadThreadsPOSIX threads, mutex, cond var (now folded into glibc)
librtRealtime POSIXtimer_create, shm_open, mq_open, clock_nanosleep
libdlDynamic loaddlopen / dlsym / dlerror — runtime plugin loading
libmMathsin/cos/exp/log; needs explicit -lm with gcc
libcrypto / libsslCryptoOpenSSL — TLS, AES, RSA, EVP API
libcurlHTTP/FTP clientMulti-protocol transfer, easy + multi interface
c-aresAsync DNSNon-blocking DNS resolver used by curl, Node
libpcapPacket captureBPF filter compile, AF_PACKET / pcap_next_ex
libnl / libnl-3NetlinkUserspace ↔ kernel rtnetlink, nl80211, generic netlink
libnftnl / libmnlnftables / netlinkModern Netfilter userspace
libev / libeventEvent loopepoll/kqueue abstraction, timers, signals
libuvCross-platform asyncNode.js core; epoll on Linux, IOCP on Windows
liburingio_uring APIUser-friendly wrapper around io_uring SQ/CQ rings
liburcuUserspace RCULock-free reads in user space (used by LTTng, DPDK)
libnumaNUMA bindnuma_alloc_onnode, numa_run_on_node, mempolicy
libhugetlbfsHuge pagesBack malloc with 2 MB / 1 GB pages transparently
libbpfeBPF userspaceLoad BPF object, manage maps, CO-RE relocations
libxdpAF_XDP / XDPUMEM, fill/completion ring helpers, multi-prog dispatch
libelf / libdwELF parsingUsed by perf, BPF, debuggers, objcopy
zlib / lz4 / zstdCompressiondeflate / fast / high-ratio compression
protobuf-c / msgpack-cSerializationWire format encode/decode for RPC and storage
libcjson / janssonJSONParse / build JSON with C structs
libxml2 / libyamlXML / YAMLConfiguration parsing, SAX & DOM
jemalloc / tcmalloc / mimallocAllocatorDrop-in malloc replacements with thread-cache
GLibToolkitGArray, GHashTable, GAsyncQueue, main loop (used by GTK, Qemu)
DPDK librte_*Kernel-bypassEAL, mempool, mbuf, ring, hash, lpm, eventdev

Appendix — C Compilation Pipeline & Cross-Build

StageToolInput → Outputgcc flag
Preprocesscpp.c → .i (expand #include / #define / macros)-E
Compilecc1.i → .s (translate to target assembly)-S
Assembleas (binutils).s → .o (ELF relocatable object)-c
Linkld / collect2.o + .a + .so → ELF executable / shared object(default)
Static archivear / ranlib*.o → .a archive (just a bag of objects + index)
Shared librarygcc -shared -fPIC*.o → .so (PIC, has DT_SONAME)-shared -fPIC
Static linkinggcc -staticPulls .a contents into final ELF; no ld.so needed-static
Cross compile<triplet>-gccRun on host, emit binaries for target ISA + libc--sysroot=...
StripstripDrop .symtab / .debug_* sections

Appendix — vDSO Pseudo-Syscalls (no kernel switch)

SymbolReplacesWhy it can stay in user space
__vdso_clock_gettimeclock_gettime(2)Kernel publishes monotonic/realtime counters in a shared page; user reads TSC + offsets
__vdso_gettimeofdaygettimeofday(2)Same shared timekeeping page; arithmetic-only
__vdso_timetime(2)Reads cached unix time from shared page
__vdso_getcpugetcpu(2)Reads CPU id from RDTSCP / TLS, no syscall
__vdso_clock_getresclock_getres(2)Returns hardcoded resolution for known clock_id