§17.1–17.34 libc, Loader, ELF, Toolchain, ABI, and Common C Libraries

1. Overview

libc is the ABI adapter between portable C or POSIX calls and the Linux process model. Some calls are pure userspace library work, some wrap the syscall instruction, and a small set use the kernel-mapped vDSO so the process can read current kernel state without paying for a privilege transition.

The choice of libc is an engineering trade-off: glibc favors compatibility and breadth, musl favors simplicity and static linking, embedded libcs favor size and configurability, and Android or bare-metal libcs adapt the same C surface to very different kernels or no kernel at all.

2. Key Data Structures

A dynamically linked Linux process contains the application, libc, the vDSO mapping, thread-local storage, and the initial auxiliary vector that tells libc where kernel-provided user pages live.

3. Core Mechanism

Background

A C program wants one stable API, but Linux exposes a lower-level syscall ABI where numbers and arguments live in specific CPU registers. libc hides that ABI, adds POSIX behavior such as cancellation points, and converts kernel error returns into the familiar -1 plus errno convention.

Plan

1. Application code calls a libc wrapper such as write(fd, buf, len).
2. libc moves the syscall number and arguments into the architecture syscall registers.
3. The CPU executes syscall on x86_64 or svc #0 on aarch64.
4. The kernel dispatches through its syscall table and returns either a non-negative result or a negative errno value.
5. libc maps negative kernel results to thread-local errno and returns -1.

Walkthrough

Suppose the program calls write(-1, "x", 1). The kernel does not set userspace errno; it returns a negative value in the return register. The wrapper recognizes the reserved negative range, stores the positive code in TLS, and returns -1 so the POSIX API stays uniform across architectures.

vDSO Fast Path

Time calls are different because the kernel can publish read-only timekeeping data into every process. libc finds the vDSO through AT_SYSINFO_EHDR, resolves symbols like __vdso_clock_gettime, and calls them directly when the clock source allows a safe user-mode calculation.

A successful vDSO clock_gettime(CLOCK_MONOTONIC) call never enters ring 0. It reads a sequence counter, base time, and CPU counter; if the sequence changed mid-read or the clock is unsupported, libc can fall back to the real syscall path.

4. Minimal C Demo

This program prints with both the libc write wrapper and the raw syscall(SYS_write) interface, then compares normal clock_gettime against a forced real syscall. On most Linux machines the libc path is faster because it uses vDSO.

5. Kernel Source Pointers

6. Interview Prep

Why does libc exist if Linux already has syscalls?

Syscalls are a kernel ABI, not the full C/POSIX programming environment. libc provides startup code, wrappers, errno handling, cancellation semantics, malloc, stdio, DNS, locale, dynamic loading hooks, and portability across kernel versions and architectures.

What is the difference between glibc and musl in practice?

glibc is broad, highly compatible, and optimized for mainstream Linux distributions. musl is smaller, simpler, MIT-licensed, and strong for static containers, but may differ in extensions, resolver behavior, locale coverage, and performance characteristics.

What happens when a syscall fails?

The kernel returns a negative errno value in the return register. The libc wrapper converts it to -1, stores the positive error number in thread-local errno, and returns to the caller.

Why is vDSO faster than a normal syscall?

vDSO code runs in user mode and reads kernel-maintained shared pages, so it avoids privilege transition, syscall dispatch, register save/restore overhead, and scheduler return-to-user checks.

When does libc use syscall(2) directly?

Programs can call syscall(2) for newer or uncommon syscalls before a stable wrapper exists. The trade-off is that the caller must handle syscall numbers, argument types, restart behavior, and portability details explicitly.

7. §17.4–17.5 Allocator Internals

glibc malloc is ptmalloc2: a dlmalloc-derived allocator with arenas to reduce lock contention, bins to classify free chunks by size, and a per-thread tcache fast path for the hottest small allocations.

Background

A high-throughput C service allocates many short-lived objects but cannot take a global heap lock for every request. ptmalloc2 first tries thread-local state, then arena-local bins, then grows the process with brk or mmap.

Plan

1. Round the request up to an aligned chunk size and choose a size class.
2. Check the calling thread's tcache; no arena lock is needed on a hit.
3. For small objects, consult fastbins or smallbins in the selected arena.
4. For larger reusable chunks, scan the unsorted bin first, then size-sorted largebins.
5. If no reusable chunk exists, split the top chunk or request memory from the kernel.

Walkthrough

Suppose thread A frees seven 32-byte objects and then allocates another 32-byte object. The first allocation after the frees pops directly from A's tcache, so it does not touch another thread, does not lock an arena, and usually returns memory that is still warm in the CPU cache.

Alternative allocators make different bets: jemalloc and tcmalloc emphasize scalable size classes and thread caches, mimalloc emphasizes locality and eager release, and scudo spends more metadata work to catch or contain heap corruption. DPDK's rte_malloc is a separate NUMA-aware hugepage allocator for packet-processing memory, not a drop-in general-purpose libc allocator.

8. §17.6 stdio Buffering

stdio wraps file descriptors with a FILE object and a userspace buffer. That buffer reduces syscall count, but it also means printf, write, fork, fflush, and fsync are different layers of the I/O path.

Background

A program prints a prompt, forks, and then both parent and child exit. If the prompt was still sitting in a userspace FILE buffer when fork copied the address space, both processes can flush the same bytes.

Plan

1. Use setvbuf when a stream needs explicit full, line, or no buffering.
2. Call fflush before fork or before mixing stdio with raw write.
3. Use fsync only when data must reach storage durability, not merely the kernel page cache.

Walkthrough

If stdout is redirected to a file, printf("x") usually stores one byte in the userspace buffer. After fork, parent and child both own a private copy of that byte, so two exits can produce two writes unless the program flushed before forking.

9. §17.7–17.8 NPTL pthread and TLS

NPTL implements POSIX threads as a 1:1 mapping: every pthread_t is backed by a kernel task_struct created with clone, while glibc manages stacks, thread control blocks, TLS, joins, cancellation, and futex-backed synchronization.

Background

A thread library must create independent CPU schedulable contexts without giving each one a separate process address space. Linux solves that with clone flags, and libc layers the POSIX API and TLS model on top.

Plan

1. Allocate a stack, guard page, TCB, and TLS area for the new thread.
2. Call clone with shared VM, fd table, filesystem state, signal handlers, and thread-group membership.
3. Pass the TLS pointer with CLONE_SETTLS so the CPU thread register points at the new TCB.
4. Run a libc trampoline that calls the user's start routine and records the return value for pthread_join.

Walkthrough

When two threads read errno, they are not reading one process-global integer. On x86_64, the FS base points to the current thread's TCB, and libc's errno macro resolves to storage reached through that per-thread base.

10. Minimal C Demo

These demos keep the core runtime mechanisms small enough to trace: allocator accounting after small-object churn, stdio buffer duplication across fork, and NPTL threads with both a racy shared counter and per-thread TLS addresses.

11. Kernel Source Pointers

12. Interview Prep

Why does glibc use multiple malloc arenas?

A single global heap lock would serialize allocation-heavy multi-threaded programs. Multiple arenas let different threads allocate concurrently, while tcache handles many small allocations without taking an arena lock at all.

What is the difference between tcache and fastbins?

tcache is per-thread and lock-free for recently freed small chunks. Fastbins are arena-owned singly linked lists for small chunks; they still belong to allocator arena state and are consolidated later.

Why can printf output duplicate after fork?

fork copies userspace memory, including unflushed FILE buffers. If both parent and child exit normally, both can flush the copied bytes.

Is a pthread a process or a kernel thread?

On modern Linux with NPTL, each pthread is a kernel schedulable task in the same thread group. Threads share address space and many process resources, but each has its own stack, registers, TCB, and TLS.

Why does TLS need several models?

Code linked into the main executable can use direct thread-pointer offsets, while shared libraries loaded later need runtime lookup through module IDs and the DTV. The models trade dynamic-loading flexibility for faster instruction sequences.

13. §17.9 Dynamic Linker and Loader

Dynamically linked executables do not jump straight from the kernel into main. The ELF PT_INTERP segment names ld-linux.so, and that interpreter maps dependencies, resolves relocations, prepares libc startup, and only then enters the application.

Background

A program calls printf, but the final address of printf inside libc is not known when the executable is linked. Position-independent shared libraries can be loaded at different addresses, so ld.so must fix up references at runtime.

Plan

1. The static linker emits PLT stubs, GOT slots, and relocation records for imported symbols.
2. At startup, ld.so maps dependencies listed by DT_NEEDED and applies required relocations.
3. With lazy binding, a PLT call enters the resolver the first time a function is called.
4. The resolver finds the winning symbol, patches the GOT slot, and returns to the function.
5. Later calls use the patched GOT slot directly instead of repeating symbol lookup.

Walkthrough

Suppose main calls malloc for the first time. With lazy binding enabled, the call reaches malloc@plt, which jumps through a GOT slot that still points at the resolver path. ld.so searches the link map, honors interposition rules such as LD_PRELOAD, writes the chosen address into the GOT, and transfers control to the real function.

14. §17.10 Runtime Loading with dlopen

dlopen exposes part of the dynamic loader as a runtime plugin API: the host maps a shared object, asks for symbol addresses with dlsym, calls them through function pointers, and eventually releases the handle with dlclose.

Plugin ABI Shape

Real plugin systems avoid scattering many raw dlsym calls. They usually export one initialization symbol that returns a versioned table of function pointers, so the host can check ABI compatibility before it calls into untrusted extension code.

15. §17.11 ELF File Format

ELF has two views of the same file. The program header table is the loader view used by the kernel and ld.so, while the section header table is the linker and debugger view that names pieces such as .text, .data, .bss, and relocation sections.

Program headers group sections into mappable segments. For example, .text and .rodata often share a read-execute load segment, while .data and .bss live in a writable segment where the file-backed bytes are followed by zero-filled memory.

Background

When execve loads an ELF executable, it does not care about every named section. It uses program headers to map pages with the correct permissions, locate the interpreter, find dynamic linking metadata, and set the initial instruction pointer.

Plan

1. Read the ELF header to validate magic, class, endianness, machine type, and entry point.
2. Use program headers to map PT_LOAD segments with read, write, and execute permissions.
3. Map the interpreter from PT_INTERP when the binary is dynamically linked.
4. Let ld.so process PT_DYNAMIC, symbol tables, and relocation records.
5. Use section headers later for linking, debugging, symbol inspection, and tools like readelf -S.

Walkthrough

A file with a 4 KB zero-initialized global array grows the process memory image, but it does not add 4 KB of zero bytes to the executable. The linker records that storage as .bss with SHT_NOBITS; the loader maps writable memory and guarantees the extra bytes start as zero.

16. Minimal C Demo

These snippets demonstrate the loader surface area: interposing malloc with LD_PRELOAD, loading a plugin function with dlopen, and compiling a tiny ELF that can be inspected with readelf.

17. Kernel Source Pointers

18. Interview Prep

What does ld.so do before main runs?

It maps needed shared libraries, applies relocations, initializes TLS and loader state, runs constructors in the required order, and enters libc startup code that eventually calls main.

Why do PLT and GOT exist?

They let position-independent code call functions whose final addresses are known only after shared libraries are mapped. PLT stubs route calls through GOT slots, and ld.so can patch those slots during relocation or lazy binding.

How does LD_PRELOAD override functions?

The loader inserts preload objects early in the symbol search order. When relocation asks for malloc, the preloaded definition can win, and that wrapper can use dlsym(RTLD_NEXT, ...) to call the next implementation.

What is the difference between RTLD_LAZY and RTLD_NOW?

RTLD_LAZY defers function symbol binding until first call. RTLD_NOW resolves required symbols during load, which makes failures earlier and more deterministic.

Why does .bss not increase file size like .data?

.data stores initialized bytes in the file. .bss records only size and alignment; the loader allocates memory for it and zeros it at load time.

19. §17.12 C Compilation Pipeline

A C build is a staged translation pipeline. The driver named gcc or cc usually orchestrates the preprocessor, compiler proper, assembler, and linker rather than doing every job itself.

Before the linker sees an object file, the assembler has already produced ELF sections, a symbol table, and relocation records. Undefined references such as printf are not bugs at this point; they are promises for the linker to resolve later.

Background

Build failures are much easier to diagnose when you know which stage produced the artifact. A missing header is a preprocessing problem, bad C syntax is a compile problem, an unknown instruction is often assembler-facing, and an undefined external symbol is usually a link problem.

Plan

1. Use -E to inspect macro expansion and included declarations.
2. Use -S to inspect optimized assembly before it becomes ELF bytes.
3. Use -c to create a relocatable object without linking.
4. Link objects and libraries into an executable with final segment layout and dynamic metadata.

Walkthrough

Suppose main.c contains #define TWICE(x) ((x) * 2) and calls printf. The .i file contains the expanded expression, the .s file contains an assembly call, the .o file records an unresolved printf relocation, and the final executable records either linked libc code or a dynamic dependency.

20. §17.13 Static vs Dynamic Linking

Static linking copies selected object files from archives into the final ELF. A .a file is an ar archive of relocatable objects, and the linker pulls only members needed to satisfy currently unresolved symbols.

Dynamic linking records dependency names and relocation work instead of copying all library implementation code. At process start, ld-linux.so maps the named shared objects, applies relocations, and resolves calls through GOT and PLT machinery.

Shared library naming separates the developer-facing linker name from the ABI contract and the real file. The executable normally records the SONAME, not the exact patch-level filename, so compatible library updates can replace the real file.

Background

A deployment wants a single binary, but operations wants library security updates without rebuilding every application. Static and dynamic linking choose different points on that deployment and update trade-off.

Plan

1. For static archives, scan unresolved symbols and pull matching archive members into the output.
2. For shared objects, record DT_NEEDED, symbol references, and relocations in the executable.
3. At runtime, let ld.so locate libraries through rpath, runpath, cache, and default directories.
4. Use SONAME major versions to preserve ABI compatibility across real-file upgrades.

Walkthrough

If an app links against libfoo.so whose SONAME is libfoo.so.1, the executable records NEEDED libfoo.so.1. A distro can move the symlink from libfoo.so.1.2.3 to libfoo.so.1.2.4 without relinking the app, as long as ABI version 1 remains compatible.

21. §17.14 Position Independent Code and PIE

Shared libraries need -fPIC so their text can be mapped at any virtual address. Position-independent executables use -fPIE and link as PIE, giving normal programs the same address-randomization property.

External calls still use indirection. PIC code can call a PLT stub, the stub consults a GOT slot, and the dynamic loader patches that slot when the final callee address is known. With -fno-plt, compilers can emit a direct GOT-indirect call and skip the PLT stub for some calls.

Background

If a shared library contained fixed absolute addresses in its text, the loader would have to modify executable pages for each process. PIC keeps text pages shareable by moving address-specific state into relocatable data such as the GOT.

Plan

1. Compile shared-library code with -fPIC so code references data through position-independent sequences.
2. Keep runtime-patched addresses in writable GOT slots rather than patching executable code.
3. Compile executables as PIE when ASLR should randomize the main program image.
4. Inspect relocations and disassembly to see GOT-relative references and PLT calls.

Walkthrough

A shared library function that reads a global variable does not bake the final address of that variable into the instruction stream. On x86_64 it can use RIP-relative code to reach a GOT entry, and ld.so patches that data slot after choosing the library load address.

22. Minimal C Demo

These snippets create small files under /tmp and run the real compiler tools so the intermediate artifacts are visible: preprocessing, assembly, relocatable objects, archives, shared objects, SONAME metadata, and PIC relocations.

23. Kernel Source Pointers

24. Interview Prep

What are the four main stages from C source to executable?

Preprocessing expands includes and macros, compilation emits assembly, assembly writes relocatable ELF objects, and linking resolves symbols plus writes the final executable or shared object.

Why can an object file contain undefined symbols?

A relocatable .o is not the final program. It can record unresolved references and relocation entries so the linker can bind them to another object, an archive member, or a shared-library import.

What is the practical difference between static and dynamic linking?

Static linking copies library object code into the executable. Dynamic linking records dependencies and lets ld.so map shared libraries at runtime, enabling smaller binaries and library updates at the cost of runtime dependency management.

What problem does SONAME solve?

SONAME gives the runtime loader an ABI-versioned library name such as libfoo.so.1, decoupling the executable's compatibility requirement from the exact real file name installed on disk.

Why is -fPIC required for shared libraries?

A shared library may load at different addresses in different processes. PIC keeps code address-independent and shareable by using RIP-relative instructions and GOT indirection instead of fixed absolute addresses in text.

25. §17.15 Cross Compilation

Cross compilation means the compiler runs on one machine but emits binaries for another. The confusing words are build, host, and target: for normal applications, the host is the machine that will run the result; target matters mainly when the thing being built is itself a compiler.

A cross compiler is not just a different code generator. It must pair target binutils with a target sysroot, otherwise it may accidentally compile against host headers or link host libraries that cannot run on the target CPU or libc.

Background

A common embedded workflow builds on x86_64 CI but deploys to an ARM64 board. The produced ELF must contain AArch64 instructions, use the target dynamic loader path, and link to the target libc, not the CI runner's libc.

Plan

1. Select a triplet such as aarch64-linux-gnu or arm-linux-musleabihf.
2. Point the compiler at a sysroot containing target headers, crt*.o, libc, and dependent libraries.
3. Configure pkg-config with PKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_LIBDIR.
4. Verify the output with file, readelf -h, and optionally qemu-aarch64.

Walkthrough

For aarch64-linux-gnu-gcc -static hello.c -o hello-arm64, the driver invokes an AArch64 assembler and linker, searches the AArch64 sysroot for stdio.h and libc, writes an AArch64 ELF, and produces a binary that the x86_64 build host can inspect but cannot execute directly without qemu-user or real ARM64 hardware.

Sysroot and pkg-config must agree on the same target tree. If pkg-config --libs zlib returns host paths, the final link may mix architectures even when the compiler prefix is correct.

26. §17.16 Cross Toolchains

A complete Linux cross toolchain is built in layers: target binutils first, a bootstrap compiler next, kernel UAPI headers into the sysroot, libc built for that target, then a final compiler that knows how to find the completed runtime.

Background

Toolchain choice depends on whether you need only a compiler, a whole root filesystem, or a product distro with repeatable package metadata. The libc choice is part of that decision because glibc, musl, and uClibc expose different ABI and feature surfaces.

Plan

1. Use crosstool-NG when the deliverable is a pinned compiler and sysroot.
2. Use Buildroot when the compiler and a small root filesystem should be generated together.
3. Use Yocto when product images, package feeds, BSP layers, and SDK export matter more than simplicity.
4. Smoke-test every toolchain by compiling, inspecting ELF metadata, and running under emulator or hardware.

Walkthrough

In a glibc cross bootstrap, the first GCC cannot build normal programs because libc is not available yet. It exists to compile libc itself after kernel headers are installed. The final GCC is rebuilt after libc so its specs can find startup files, libgcc helpers, pthread support, and the target dynamic loader path.

27. Minimal C Demo

This demo tries the real aarch64-linux-gnu-gcc and qemu-aarch64 path when those tools are installed, then creates a tiny sysroot layout and shows how pkg-config is redirected for cross builds.

28. Kernel Source Pointers

29. Interview Prep

What is the difference between build, host, and target?

Build is where the build tools run. Host is where the output program runs. Target is what a compiler being built will emit; for ordinary app cross compilation, build and host are usually the important pair.

Why is a sysroot required?

It gives the cross compiler the target's headers, startup objects, libc, and libraries. Without it, the build can accidentally include or link files from the build machine.

How do you validate a cross-compiled binary?

Use file and readelf -h to verify the ELF machine and interpreter, then run it on target hardware or with qemu-user when the syscall and library surface is compatible.

When would you choose Buildroot over crosstool-NG?

Choose Buildroot when you need a root filesystem and packages in addition to the compiler. Choose crosstool-NG when the main deliverable is a standalone, reproducible cross toolchain.

Why can pkg-config break cross builds?

Host .pc files can return host include and library paths. Cross builds should set PKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_LIBDIR so metadata comes from the target sysroot.

30. §17.17 Library Inspection Tools

Library inspection is the practical loop for answering "what is this binary, what will load it, which symbols does it need, and where did this address come from?" The tools look at different layers of the same artifact: ELF identity, loader metadata, link-time sections, symbol tables, disassembly, archive indexes, and debug line tables.

Key Data Structures

The inspection surface is not one structure; it is a stack of ELF records and tool-specific views. Program headers are the loader's contract, sections and symbols are the linker/debugger's contract, and archive indexes let the static linker pull only the object files that satisfy unresolved symbols.

31. §17.18 Runtime Tracing

Runtime tracing observes the program after the loader and kernel start executing it. strace sees syscall boundaries, ltrace sees dynamically linked library calls, perf trace uses kernel tracing infrastructure, and LD_DEBUG asks ld.so to print its own search and binding decisions.

Dynamic-loader debug output is strongest before main even starts: it shows which files were searched, which relocations were processed, and which library definition won a symbol binding.

Background

A binary works on one machine but fails on another with ENOENT, an unresolved symbol, or a missing shared object. Static inspection tells you what the binary requested; runtime tracing tells you what actually happened on this host.

Plan

1. Start with file and readelf -h/-l/-d to verify architecture, interpreter, and dependencies.
2. Use ldd or LD_TRACE_LOADED_OBJECTS=1 to see the loader's dependency resolution plan.
3. Use nm, readelf -s, and c++filt when symbol names or C++ ABI names are suspect.
4. Use strace -e for failing syscalls and LD_DEBUG=bindings,files for loader failures.

Walkthrough

Suppose ./app prints "error while loading shared libraries: libfoo.so.1". readelf -d ./app confirms a NEEDED entry for libfoo.so.1. LD_DEBUG=files ./app then shows each searched directory. If the file exists but a function is missing, switch to nm -D libfoo.so.1 and LD_DEBUG=bindings to see whether the expected symbol is exported and which object won the binding.

32. Minimal C Demo

This demo builds a tiny shared library and executable under /tmp, then runs the normal inspection commands against them. When installed in the environment, strace and ltrace are used as optional runtime tracers; otherwise the demo prints a skipped message.

33. Kernel Source Pointers

34. Interview Prep

What is the difference between readelf and objdump?

readelf is a direct ELF metadata reader and does not rely on BFD interpretation. objdump is stronger for disassembly and mixed section/code views.

Why can ldd be misleading?

It shows the loader's dependency resolution for this host and environment, not a universal truth. Environment variables, RUNPATH, loader cache, architecture, and container root can change the answer.

How do you debug "undefined symbol" at runtime?

Check readelf -d for dependencies, nm -D or readelf -s for exported symbols, then use LD_DEBUG=symbols,bindings to see the lookup scope and selected definition.

When do you use strace instead of ltrace?

Use strace when the boundary of interest is the kernel: files, sockets, mmap, signals, permissions, and errno. Use ltrace when the question is which dynamically linked library function was called.

What does addr2line need to work well?

It needs an address in the correct binary or shared object plus DWARF debug line information. For shared libraries under ASLR, subtract the library load base before querying the file offset address.

35. §17.19 C Calling Conventions and ABI

A calling convention is the contract that lets separately compiled object files call each other: which registers carry arguments, which registers a callee must preserve, where return values live, how the stack is aligned, and where extra arguments spill.

Background

The compiler can optimize inside one function freely, but a function boundary must be predictable to the linker, debugger, unwinder, assembly code, and foreign-function interfaces. That boundary is why an object built by Clang can call one built by GCC when both obey the platform ABI.

Plan

1. Put the first arguments in the ABI-defined registers and spill the rest to stack slots.
2. Maintain the required stack alignment before call or branch-link instructions.
3. Let the caller assume caller-saved registers may be clobbered after the call.
4. Make the callee save and restore callee-saved registers when it uses them.
5. Return small scalar values in the ABI return registers and larger aggregates through hidden pointers when required.

Walkthrough

For add8(a,b,c,d,e,f,g,h) on SysV AMD64, arguments one through six arrive in rdi, rsi, rdx, rcx, r8, and r9. The seventh and eighth arguments are stack slots above the return address, which is exactly what gcc -S makes visible in assembly.

36. §17.20 C and C++ ABI

Linux C++ compilers usually follow the Itanium C++ ABI for name mangling, object layout, vtables, RTTI, exception metadata, and rules for crossing shared-library boundaries. C interop uses extern "C" because C has no C++ namespace, overload, or class encoding in symbol names.

A virtual call is not a name lookup at runtime. The generated code loads the object's vptr, indexes a fixed vtable slot known at compile time, then indirect-calls the function pointer with the object address passed as the implicit this argument.

37. §17.21 errno and Reentrancy

Modern libc does not expose errno as one writable global integer. The macro expands to a dereference of a function such as __errno_location(), and that function returns the current thread's TLS-backed errno address.

Background

Library functions that keep hidden static state are convenient in single-threaded programs and dangerous in concurrent or signal-heavy programs. Reentrant variants push that state into caller-owned storage, and async-signal-safe functions avoid locks, malloc, and complex libc internals.

Plan

1. Use TLS for state that should be independent per thread, such as errno.
2. Prefer *_r APIs or explicit state objects when two calls must progress independently.
3. Inside signal handlers, restrict work to async-signal-safe calls such as write and _exit.

38. §17.22 C11 Atomics and Memory Model

C11 atomics separate atomicity from ordering. A relaxed atomic counter prevents torn updates but says almost nothing about nearby non-atomic data; release/acquire pairs are the common pattern for publishing initialized data to another thread.

Background

A producer writes a payload and sets a ready flag. If the flag is atomic but the payload writes are not ordered before the flag store, a consumer on a weakly ordered CPU can observe the flag and still read stale payload fields.

Plan

1. Write the payload with ordinary stores while only the producer can see it.
2. Publish readiness with atomic_store_explicit(..., memory_order_release).
3. Poll readiness with atomic_load_explicit(..., memory_order_acquire).
4. Read the payload only after the acquire load observes the released value.

Walkthrough

In a single-producer single-consumer handoff, the producer fills slot.text and then stores ready = 1 with release ordering. The consumer spins with acquire loads; once it sees one, the C memory model guarantees the earlier payload writes happen-before the consumer's payload reads.

39. Minimal C Demo

This program keeps the ABI and memory-model ideas concrete: add8 is suitable for compiling with cc -S to inspect register versus stack arguments, the producer-consumer pair uses release/acquire atomics, and two threads print distinct TLS-backed errno addresses.

40. Kernel Source Pointers

41. Interview Prep

What happens to the seventh integer argument on SysV AMD64?

The first six integer or pointer arguments use registers. The seventh and later arguments are passed in stack slots, while the return address from call also lives on the stack.

Why does C++ need name mangling?

The linker sees flat symbol names, but C++ has namespaces, classes, overloads, templates, and argument types. Mangling encodes that information into unique linker symbols; extern "C" disables it for C ABI boundaries.

Why is errno thread-safe?

It is a macro that resolves to the current thread's TLS storage, commonly through __errno_location(). Two threads can set errno independently because they write different addresses.

What is the difference between reentrant and async-signal-safe?

Reentrant code can be safely called concurrently when each caller owns its state. Async-signal-safe code can run from a signal handler even if the signal interrupted libc while it held locks or was mutating internal state.

When is relaxed atomic ordering enough?

It is enough when only atomicity matters, such as a diagnostic counter whose exact ordering relative to other memory is irrelevant. For publishing data between threads, use release on the producer side and acquire on the consumer side.

42. §17.23 glibc Internals

glibc is not only a collection of exported C functions. Its runtime loader, private symbol namespace, symbol-version database, IFUNC resolvers, NSS modules, malloc state, pthread state, and startup code cooperate before main ever runs.

IFUNC is glibc's hot-path dispatch trick. A symbol can be a resolver instead of a direct implementation; during relocation the resolver inspects hardware capabilities and returns the address of the best implementation for this CPU.

Symbol versioning is glibc's long-term ABI contract. The dynamic symbol is not just realpath or printf; it also carries a version tag, so old binaries can keep binding the old behavior while new links select the default version.

Background

The loader has a bootstrapping problem: it is itself a shared ELF object, but it must execute before the normal relocation machinery is available. glibc solves this with a small self-contained early path that relocates ld.so first, then uses the now-working loader to process the main program and dependencies.

Plan

1. Kernel maps the executable and the PT_INTERP interpreter.
2. ld.so relocates its own GOT, dynamic data, and early function pointers.
3. It builds the link_map for the main executable and each DT_NEEDED object.
4. It resolves normal relocations, IFUNC relocations, symbol versions, and constructor arrays.
5. It transfers into libc startup, which initializes process state and calls main.

Walkthrough

On a Haswell-era x86_64 machine, a relocation for memcpy can point at an IFUNC resolver. During startup, the resolver sees AVX2 and ERMS CPU feature bits, picks the tuned implementation, and ld.so writes that address into the relocation target. The application pays the dispatch cost during relocation, not on every copy.

43. §17.24 Security Hardening

Linux C hardening is layered: the compiler can insert object-size checks and stack canaries, the linker can make relocation tables read-only, the loader and kernel can randomize mappings, and page tables can forbid execution from writable data pages.

RELRO focuses on the dynamic linker's writable relocation targets. Full RELRO combines -z relro with -z now, resolves PLT entries before the program starts, then makes GOT pages read-only so a later memory corruption cannot redirect imported calls through GOT overwrite.

Background

A classic stack overflow writes beyond a local buffer toward saved control data. Without hardening, that bug can overwrite the return address; with stack protector enabled, the overwrite usually corrupts the canary first and the function aborts before returning through attacker-controlled data.

Plan

1. Compile with bounds-aware libc wrappers using -D_FORTIFY_SOURCE=2 or 3 when supported.
2. Add stack canaries for vulnerable frames with -fstack-protector-strong.
3. Build PIE executables so ASLR can randomize the main program as well as shared libraries.
4. Use full RELRO for network-facing programs when startup binding cost is acceptable.
5. Keep NX enabled and use CFI or sanitizers where the toolchain and deployment model allow them.

Walkthrough

Compile a vulnerable function with a local char buf[8] and strcpy. A short input returns normally. A long input overwrites past the buffer; before the function returns, generated epilogue code compares the saved canary against TLS guard state and calls __stack_chk_fail if the value changed.

44. Minimal C Demo

This demo writes a tiny vulnerable program, compiles it with stack protector and fortify flags, runs one safe input, then runs an overflowing input so the abort path is visible. It also prints a few ELF facts that interviewers often ask you to verify with readelf and nm.

45. Kernel Source Pointers

46. Interview Prep

Why does ld.so have to relocate itself?

The kernel maps the interpreter but does not run the full dynamic-link process for it. ld.so starts in a restricted early mode, fixes its own relocation state, then uses that working machinery to relocate the main executable and its libraries.

What problem does IFUNC solve?

It lets one exported symbol choose a CPU-specific implementation at load time. Hot functions such as memcpy can use AVX2, ERMS, or a baseline path without checking CPU features on every call.

What does printf@@GLIBC_2.2.5 mean?

The double at-sign marks the default symbol version used by new links. A single at-sign names a non-default older version retained so old binaries keep their original ABI contract.

What is full RELRO?

Full RELRO uses -z relro -z now: ld.so resolves PLT entries eagerly, then marks GOT-related pages read-only. That blocks later GOT overwrite attacks at the cost of more startup binding work.

What does a stack canary actually protect?

It detects many contiguous overwrites from local buffers toward saved frame data before a function returns. It does not prevent all memory corruption, heap bugs, arbitrary writes, information leaks, or logic bugs.

47. §17.25 System Libraries

Many names that used to feel like separate Unix libraries are now part of the libc surface on modern glibc, but the link names still matter for portability, older distributions, static links, and interview debugging. Know which API family owns the concept before chasing missing symbols.

Background

A build that works on a current Linux laptop can fail on an older production image because glibc moved symbols over time. Before glibc 2.34, pthread, realtime, and dl APIs often required explicit libraries; after 2.34 many of those symbols live directly in libc, while compatibility linker names remain.

Plan

1. Identify the API family: threads, realtime timers, dynamic loading, math, resolver, or password hashing.
2. Check whether the target libc version folds that family into libc or still needs an explicit library.
3. Put dependent libraries after the object files that reference them when using traditional linkers.
4. Use pkg-config, CMake imported targets, or build-system probes instead of hardcoding assumptions for portable projects.

Walkthrough

Suppose a small daemon uses clock_gettime, pthread_create, dlopen, and sqrt. On new glibc, the first three may link through libc alone, but sqrt still commonly needs -lm. On an older glibc build host, the portable command may include -pthread -ldl -lrt -lm, with -pthread preferred over raw -lpthread because it also sets compiler flags.

48. §17.26 Crypto / Network Libraries

Network-facing C programs usually compose several specialized libraries: libcurl owns protocol policy, c-ares can make DNS nonblocking, OpenSSL owns TLS and cryptographic primitives, and libpcap or netlink libraries expose lower-level packet and kernel networking surfaces.

Packet capture follows a different path from socket I/O. libpcap compiles a BPF predicate, asks the kernel capture path to filter frames early, then returns packet metadata and captured bytes to the userspace sniffer.

Background

An HTTPS client looks simple at the call site, but the library stack hides several failure domains: DNS lookup, socket connect, TLS negotiation, certificate validation, HTTP framing, proxy behavior, retries, and timeout accounting. Good C services keep those responsibilities explicit.

Plan

1. Use libcurl when the task is an application protocol client rather than a custom socket protocol.
2. Use OpenSSL's EVP interface for cryptography; avoid direct low-level AES/RSA calls in new code.
3. Use c-ares or the libcurl multi interface when DNS must integrate with an event loop.
4. Use libpcap for passive packet capture, and libnl/libmnl/libnftnl for configuring kernel networking state.

Walkthrough

For https://api.example.com/v1, libcurl asks DNS for addresses, opens a nonblocking TCP socket, hands the connected file descriptor to OpenSSL, verifies the certificate chain against trust anchors, then sends HTTP bytes through TLS records. If capture is needed, libpcap observes packets from the side; it is not part of the client's send/receive path.

49. Minimal C Demo

The first demo shows the practical link behavior around clock_gettime, -lrt, and -lm. The second writes a minimal libpcap sniffer that prints the first 32 bytes of captured packets when libpcap and capture permissions are available.

50. Kernel Source Pointers

51. Interview Prep

Why does -pthread differ from -lpthread?

-lpthread only asks the linker for a library. -pthread is a compiler-driver option that can also define thread-related macros and choose the correct startup/link flags for the platform.

Why did old code use -lrt for clock_gettime?

Older glibc exposed some POSIX realtime APIs through librt. Since glibc 2.34, those symbols are folded into libc, but retaining -lrt is often harmless and keeps old build recipes understandable.

Why prefer OpenSSL EVP APIs?

EVP gives algorithm-independent contexts, provider support, padding/mode handling, and future agility. Direct low-level crypto calls hardcode algorithms and are easier to misuse.

What does libpcap add over a raw socket?

It provides a portable capture API, filter compilation, capture-file support, timestamp/header metadata, and platform-specific backend handling. On Linux it commonly wraps AF_PACKET plus BPF filtering.

When would you use libnl instead of shelling out to ip?

Use libnl when a program must configure routes, links, addresses, wireless state, or generic netlink families directly and needs structured errors, batching, subscriptions, or daemon-grade control.

52. §17.27 Event Loops & Async I/O Libraries

Event-loop libraries turn many blocking-looking resources into one explicit readiness machine: register file descriptors, wait for the kernel to report progress, dispatch short callbacks, then rearm or remove the interest. The design keeps one thread useful while thousands of sockets mostly wait.

io_uring changes the shape from readiness to submission and completion. Userspace fills SQEs in a shared submission ring, the kernel posts CQEs after work finishes, and liburing hides most memory-barrier and ring-index details.

Background

A proxy with 50,000 mostly idle TCP connections cannot afford one blocking thread per connection. The kernel already knows which descriptors are ready, so an event loop asks for readiness notifications and runs only the code that can make progress.

Plan

1. Represent each connection as state plus callbacks, not as a long blocking call stack.
2. Register read, write, timer, and signal interests with the loop backend.
3. Keep callbacks short; move CPU-heavy work to worker threads or a separate queue.
4. Use liburing when completion-style asynchronous kernel operations matter more than portable readiness APIs.

Walkthrough

Suppose connection 42 needs to read a request, connect upstream, and write a response. The loop first registers EPOLLIN. When bytes arrive, the read callback parses headers and registers a writable upstream socket. Later a write-ready event flushes pending bytes. At no point does the loop wait inside that connection; it returns to the kernel after each small state transition.

53. §17.28 DPDK / Kernel-Bypass Libraries

DPDK is a userspace packet-processing stack, but its library split matters for ordinary C systems work: EAL owns process and hardware setup, mbuf/mempool own packet memory, rings connect lcores, and PMDs poll NIC queues without the kernel socket path.

liburcu brings the same read-mostly idea to userspace services. Readers mark a tiny critical section and dereference a protected pointer; writers publish a replacement and wait for a grace period before freeing the old object.

Background

A 100 Gbps packet path has no budget for per-packet malloc, syscalls, interrupts, or remote-NUMA memory misses. Kernel-bypass libraries move those costs to initialization and keep the hot loop to polling descriptors and touching preallocated cache-hot objects.

Plan

1. Reserve hugepage memory and bind lcores close to the NIC NUMA node during startup.
2. Allocate packet buffers from mempools, not from the general-purpose heap.
3. Move packets between stages with rings and keep ownership transfer explicit.
4. Use RCU-style publication for read-mostly tables such as routing or policy snapshots.

Walkthrough

On node 0, EAL reserves hugepages and PMD queue 0 receives packets into mbufs from a node-local mempool. Lcore 2 classifies each mbuf with rte_hash, enqueues work to an rte_ring, and another lcore transmits. If a routing table changes, the writer publishes a new table pointer and frees the old table only after all current RCU readers exit.

54. §17.29 eBPF Userspace Libraries

eBPF userspace libraries are loaders and control planes. They parse BPF ELF objects, create maps, ask the kernel verifier to load programs, attach links to hooks, and then read events or update maps while the actual program runs inside kernel hook points.

AF_XDP uses an XDP program to redirect packets into rings backed by UMEM. libxdp provides setup helpers and dispatcher support so programs can share XDP hooks instead of overwriting each other.

Background

A BPF program is not just a blob of bytecode. It has maps, relocations, type metadata, attach points, and permissions. libbpf packages those details so production loaders can be deterministic instead of a pile of raw bpf() syscalls.

Plan

1. Compile the kernel-side BPF object with BTF information for CO-RE relocations.
2. Open the ELF with libbpf, let it create maps, and load programs through the verifier.
3. Attach with persistent link objects where possible so lifecycle is explicit.
4. Use ring buffers, perf buffers, or maps for communication back to userspace.

Walkthrough

A kprobe tracer starts with trace.bpf.o. libbpf opens the ELF, applies CO-RE field relocations against the running kernel BTF, creates a map for events, loads the program through the verifier, attaches it to the chosen kprobe, and returns a link handle. The userspace process then polls a ring buffer while the kernel invokes the BPF program on each matching event.

55. Minimal C Demo

These demos keep the core ideas runnable without requiring the full libraries in the sandbox: a tiny callback loop, a libnuma allocation build probe, and the minimal libbpf loader sequence a real tracing tool follows.

56. Kernel Source Pointers

57. Interview Prep

How do libev, libevent, and libuv differ?

libev is a small event core, libevent adds higher-level helpers such as bufferevents and HTTP/DNS pieces, and libuv is a cross-platform runtime abstraction with filesystem work queues, process APIs, and Windows IOCP support.

What is the key difference between epoll and io_uring?

epoll reports readiness: the fd can make progress if you call read or write. io_uring submits operations and reports completion: the kernel tells you the requested operation finished with a result.

Why does DPDK care so much about NUMA and huge pages?

Packet rates are high enough that remote socket memory and TLB misses become visible throughput limits. Huge pages reduce TLB pressure, and NUMA placement keeps NIC DMA, CPU polling, and packet buffers close together.

What does libbpf do before a BPF program runs?

It opens the BPF ELF, parses BTF and maps, applies CO-RE relocations, creates maps, loads bytecode through the verifier with bpf(), and attaches the resulting program to a hook.

Why use liburcu instead of a rwlock for read-mostly data?

RCU readers avoid contended locks and usually execute only lightweight bookkeeping. Writers pay the heavier cost by publishing a replacement and waiting for old readers before freeing retired objects.

58. §17.30 Compression Libraries

Compression libraries trade CPU cycles for fewer bytes on disk, network, or cache. The useful question is not which algorithm is best globally; it is whether the workload is latency-bound, storage-bound, decompression-heavy, dictionary-friendly, or locked to a format such as gzip or PNG.

On the same corpus, LZ4 and Snappy optimize throughput, zlib optimizes compatibility, zstd gives the broadest tunable speed/ratio envelope, and xz or bzip2 are usually archive choices rather than hot-path service choices.

Background

A logging pipeline writes repeated symbols, timestamps, and field names. Compressing each batch before disk or network transfer can save I/O, but a slow codec can steal CPU from request handling.

Plan

1. Pick the wire or file format first when compatibility is fixed.
2. Benchmark representative data, not random bytes, because compressors depend on repetition.
3. Measure compression and decompression separately; many systems decompress far more often than they compress.
4. Use streaming APIs for large files and bounded in-memory APIs for request-sized payloads.

Walkthrough

Suppose a service batches 64 KB of JSON logs. LZ4 may shrink it to roughly 25-40 KB with very low latency, zstd level 3 may shrink it further while staying service-friendly, and xz may win ratio but lose the request budget. For an online path, the best answer is usually the fastest codec that hits the network or storage target.

59. §17.31 Serialization Libraries

Serialization libraries define the contract between processes, files, and languages. Schema-based formats move type decisions to build time and make compatibility rules explicit; schemaless formats are easier to inspect but push validation and evolution rules into application code.

Background

An RPC payload must survive version skew: new clients, old servers, missing fields, and unknown fields. The format must make those cases boring instead of turning deploy order into a correctness condition.

Plan

1. Use field numbers or stable names as the compatibility contract.
2. Reserve removed fields so future versions do not accidentally reuse old meaning.
3. Keep unknown fields harmless and optional fields truly optional.
4. Use text formats for human-authored config and binary formats for hot RPC or storage records.

Walkthrough

A Trade message starts with id and symbol. Version 2 adds price as field 3. Old decoders ignore field 3, new decoders treat it as optional when reading old messages, and the field number is never reused for another meaning.

60. §17.32 Data Structure Libraries

C does not ship a standard hash table, vector, queue, or tree library, so production programs either build narrow local containers or adopt a utility library. GLib is the most common broad option; specialized libraries cover balanced trees, concurrent hashes, and compressed sparse arrays.

Background

A daemon needs a map from interface name to runtime state. A hand-rolled table is easy to start but easy to get wrong on resizing, ownership, iteration during deletion, and cleanup after partial initialization.

Plan

1. Decide key and value ownership up front; destroy callbacks should match that decision.
2. Use a stable hash and equality function for the exact key type.
3. Keep mutation rules clear while iterating; remove through iterator-aware APIs where available.
4. Reach for specialized concurrent or sparse structures only when profiling shows the generic container is the bottleneck.

Walkthrough

The daemon inserts eth0 => up and eth1 => down into a GHashTable created with g_str_hash, g_str_equal, and g_free destroy callbacks. Replacing eth1 frees the old value automatically, so the container encodes both lookup policy and memory ownership.

61. Minimal C Demo

These demos write small standalone programs that use the real library APIs when the matching development packages are installed, while still showing the complete compile and run commands in the sandbox output.

62. Kernel Source Pointers

63. Interview Prep

When would you choose LZ4 over zstd?

Choose LZ4 when latency and CPU time dominate and a modest ratio is enough. Choose zstd when you can spend more CPU for a better ratio or need dictionaries and tunable compression levels.

Why is gzip still everywhere if newer codecs are better?

DEFLATE compatibility is universal across HTTP, archives, package formats, and old tooling. In many systems, deployment compatibility matters more than peak compression ratio.

What is the protobuf compatibility rule interviewers expect?

Field numbers are the wire contract. Do not change a field number's meaning, do not reuse removed numbers, add new fields as optional or backward-compatible, and make old readers safely ignore unknown fields.

Why use GLib containers instead of plain arrays and structs?

GLib gives tested resize, lookup, iteration, reference, queue, and cleanup behavior. It is valuable when ownership and error-path cleanup matter more than removing one dependency.

How is a userspace hash table similar to kernel hash tables?

Both start with a hash function, bucket selection, collision handling, and ownership rules. The kernel adds constraints such as RCU-safe traversal, lock granularity, allocation context, and no blocking in atomic paths.

64. §17.33–17.34 Sanitizers, Build Systems, and pkg-config

Sanitizers move memory, thread, and undefined-behavior bugs from production crashes into reproducible test failures by adding compiler checks and runtime shadow state. Build systems decide how those flags, dependency headers, link libraries, and generated backend files reach every translation unit.

65. Key Data Structures

Sanitizers are easiest to compare by bug class, runtime cost, and whether the program must be rebuilt. Use ASan/UBSan in routine CI, add TSan for focused concurrency tests, and keep valgrind useful for binaries you cannot rebuild.

Build tools form a pipeline: a project description becomes concrete compile and link commands, often through a fast backend such as Ninja. Package discovery fills in the non-portable details: include paths, library names, library directories, and sometimes feature versions.

A .pc file is a small build metadata record. pkg-config --cflags --libs openssl turns that metadata into the exact compiler and linker flags needed on this machine or sysroot.

66. Core Mechanism

Background

A test suite passes until one path reads a pointer after free. Without instrumentation the program may print an old value, corrupt a later allocation, or crash far away from the real bug.

Plan

1. Compile with -fsanitize=address -g -fno-omit-frame-pointer so every relevant load and store is checked.
2. Let ASan wrap allocations with redzones and poison freed memory in shadow state.
3. Run the normal test; the stale load consults shadow memory before reading the actual heap byte.
4. Abort immediately with allocation, free, and failing access stack traces.

Walkthrough

A heap allocation returns p, the program writes 123, and then free(p) poisons that address range. When *p is evaluated later, the compiler-inserted check sees poisoned shadow bytes and stops at the exact use-after-free instead of letting execution continue.

67. Minimal C Demo

The first demo creates and runs a deliberate heap use-after-free under AddressSanitizer. The second writes a tiny CMake project that discovers OpenSSL with find_package and also prints the raw pkg-config flags when available.

68. Kernel Source Pointers

69. Interview Prep

What is the difference between ASan and valgrind memcheck?

ASan needs recompilation and uses compiler-inserted checks plus a runtime, so it is much faster and better for CI. Valgrind runs unmodified binaries through dynamic instrumentation, so it is slower but useful when source or rebuilds are unavailable.

Why does TSan complain about code that uses custom atomics?

TSan understands compiler-recognized atomic operations and pthread synchronization. If a project hides synchronization in inline assembly or unsupported primitives, the tool may miss the happens-before edge and report a race.

What does pkg-config actually return?

It reads .pc files and prints compiler flags from Cflags and linker flags from Libs. PKG_CONFIG_PATH adds search directories, which matters for custom prefixes and cross sysroots.

CMake versus Make: what is the real distinction?

Make is a rule executor driven by Makefiles. CMake is a meta-build system: it configures compilers, finds packages, and generates Makefiles, Ninja files, or IDE projects.

Which sanitizer would you enable first in a C service CI job?

Start with ASan plus UBSan on unit and integration tests because they catch common memory safety and UB bugs at tolerable cost. Add TSan as a separate, slower job for concurrency-heavy paths.