Part IX - ISP / SP Design

§ 9 ISP and Service Provider Networks

Provider backbone architecture, peering, IXPs, inter-AS MPLS VPN, IPv6-over-MPLS, Carrier Ethernet, optical transport, 5G xHaul, SD-WAN, NFV, anycast, and DDoS mitigation.

1. Overview

Service-provider design separates customer state, internet edge policy, core forwarding, and transport capacity. The PE owns services, the P router swaps labels at scale, and the ASBR is where business policy becomes BGP policy.

ComponentJobDesign notes
PECustomer-facing provider edgeHolds VRFs, imposes/disposes VPN labels, peers with CE routers
PProvider core transitRuns IGP/MPLS only; keeps customer and internet routes out of the core
ASBRExternal BGP boundaryPeers with transit, IXPs, and other carriers; enforces route policy
Route reflectorBGP scale pointReflects IPv4 unicast, VPNv4/VPNv6, and labeled routes to PE clients
IXP route serverMultilateral peering brokerReduces N-squared sessions and can enforce RPKI filtering
FlowspecBGP-distributed firewall ruleDrops, rate-limits, or redirects attack traffic before it crosses the backbone

2. Transit, Peering, and IXPs

Transit buys reachability to the whole internet. Peering exchanges only each side's customer cone, usually at lower cost and lower latency. LOCAL_PREF expresses that business decision inside BGP.

An IXP is a shared Ethernet fabric. A route server lets every member use one or two BGP sessions instead of building bilateral sessions with every other member, and modern route servers commonly apply RPKI origin validation.

3. L3VPN, Inter-AS, 6PE, and 6VPE

Carrier L3VPN uses RDs to make customer prefixes unique and RTs to decide import/export policy. Hub-and-spoke VPNs manipulate RTs; full-mesh VPNs import the same service RT everywhere.

Inter-AS Option A is operationally simple but scales poorly because ASBRs hold per-customer VRFs. Option B exchanges labeled VPNv4 at ASBRs. Option C scales best by letting PEs run multi-hop MP-BGP while ASBRs provide labeled reachability to PE loopbacks.

6PE and 6VPE let an IPv4 MPLS core carry IPv6 without teaching every P router IPv6. 6PE carries global IPv6; 6VPE adds per-customer IPv6 VRFs and VPNv6 routes.

4. Carrier Ethernet, Optical, and 5G Transport

MEF services package Ethernet into products: E-Line for point-to-point, E-LAN for multipoint, E-Tree for hub-and-spoke, and E-Access for wholesale UNI access. CFM and Y.1731 provide fault and performance OAM.

The optical layer turns packet links into wavelengths. OTN wraps client signals with FEC and OAM, DWDM packs many wavelengths on one fiber, and ROADMs steer wavelengths without manual patching.

5G xHaul splits transport into fronthaul, midhaul, and backhaul. eCPRI replaces rigid CPRI with Ethernet framing, but fronthaul still has tight latency budgets, commonly below 100 microseconds.

5. SD-WAN, NFV, and Anycast

SD-WAN overlays MPLS, internet, and LTE with encrypted tunnels and central policy. OMP distributes routes and policy, while application-aware routing continuously probes latency, jitter, and loss before steering traffic.

NFV turns appliance chains into software chains: firewall, load balancer, CGNAT, and WAN optimization can be instantiated and scaled by MANO systems on OpenStack or Kubernetes infrastructure.

Anycast publishes the same service prefix from many locations. Health checks inject or withdraw the route, and BGP moves users to the nearest healthy point of presence.

6. DDoS Mitigation

RTBH is blunt but fast: advertise a victim route with a blackhole community and drop the traffic at the edge. Flowspec is more selective because BGP carries match conditions and actions such as drop, rate-limit, or redirect.

Scrubbing centers handle attacks that cannot be described safely as a single drop rule. Traffic is diverted to the cleaner, legitimate packets are re-injected, and BCP38 ingress filtering reduces spoofed-source reflection attacks before they start.

7. Core Mechanism Walkthrough

Background: A provider carries a customer VPN across two autonomous systems while also exchanging internet traffic at an IXP. Customer VPN state must stay isolated from internet BGP policy.

Plan: keep customer prefixes in VRFs, exchange VPNv4 or VPNv6 with labels for the L3VPN service, keep the P core free of customer routes, and apply LOCAL_PREF/community policy only at the external BGP edge.

StepPlaneState change
1Service edgePE imports CE routes into a customer VRF and attaches RD/RT metadata.
2MP-BGPRR reflects labeled VPN routes only to PEs importing the matching RT.
3MPLS coreP routers swap transport labels without installing customer prefixes.
4Internet edgeASBR ranks peer, transit, and IXP routes with LOCAL_PREF and communities.
5ProtectionRTBH, Flowspec, and scrubbing policies are pushed at ingress edges during attacks.

8. Minimal C Demo

The BGP policy simulator compares peer and transit routes. Try 0 200 100 1 for peer-customer traffic, or 0 200 100 2 when the peer does not provide a full table.

BGP Peering Policy Simulator — C Demo
stdin (optional)

The SD-WAN demo applies a voice SLA across MPLS, internet, and LTE. Try 150 10 0 1 for normal voice selection, or 150 10 1 1 after MPLS degrades.

SD-WAN Path Selection — C Demo
stdin (optional)

9. Source Pointers

  • RFC 4364: BGP/MPLS IP VPNs and RD/RT design.
  • RFC 4271: BGP path selection and route-policy foundation.
  • RFC 8955: BGP Flowspec traffic-filter distribution.
  • RFC 2827 / BCP38: ingress filtering against spoofed-source traffic.
  • ITU-T G.709 and MEF service definitions for OTN and Carrier Ethernet terminology.

10. Interview Prep

  • Transit vs peering? Transit buys full reachability; peering exchanges customer routes to reduce cost and latency.
  • Why keep customer routes out of P routers? P routers only need IGP/MPLS labels, which keeps the core scalable and service-agnostic.
  • Option B vs Option C? Option B exchanges VPNv4 at ASBRs; Option C moves VPNv4 to PE-to-PE sessions and scales better.
  • RTBH vs Flowspec? RTBH blackholes a prefix; Flowspec distributes granular match/action rules through BGP.
  • What replaces CPRI in 5G fronthaul? eCPRI over Ethernet, with strict latency engineering for DU-to-radio transport.