Self-Controlled CDN Infrastructure: How the Control Plane Actually Works

"Self-controlled infrastructure" usually means one of two things: you rent someone's black box and hope, or you run your own and discover how much of a CDN is not the caching part. This is a description of the second option as actually built — the control plane, how configuration reaches an edge, what happens when the control plane dies, and the parts that are genuinely awkward.

The shape of it

There is one source of truth and three separate distribution channels, chosen by how urgent and how stateful each kind of change is.

The split matters. Declarative state — what an origin is, whether the WAF is on — goes into Redis, where the edge reads it per request. Imperative one-shot actions — provision this zone, add this tunnel peer, reload after a cert rotation — go over MQTT to a node agent. Anything that changes the machine itself goes over SSH from an installer script keyed on the node's role.

Configuration is data, not deployment

The thing that makes the system operable is that turning on a feature does not deploy anything. Each hostname is one JSON document in Redis:

site:example.com -> {
  "origin": "http://203.0.113.10:80",
  "cached": true,
  "geo_mode": "block",  "geo_codes": "RU,CN",
  "ip_mode":  "allow",  "ip_list":  "198.51.100.0/24",
  "rate_mode": "strict", "waf_mode": "strict",
  "cors_mode": "strict", "hotlink_mode": "block",
  "cache_rules": [{"match_type":"ext","match_value":".mp4",
                   "action":"ttl","ttl":86400}],
  "edge_rules":  [{"match_type":"prefix","match_value":"/old/",
                   "action":"redirect","redirect_status":301}]
}
suspended:example.com -> "1"

cdn_router.lua reads that key in the access phase of every request and applies origin selection, caching, geo policy, IP policy, rate limiting, WAF mode, CORS, and hotlink rules from it. No nginx reload, no per-domain config file, no deploy.

Each edge owns its own Redis master

There is no cross-region replication of config. Every edge runs its own master, resolved through its own Sentinel. That is a deliberate rejection of the obvious design: a globally replicated config store means a replication problem in one region becomes a serving problem in another. The cost is reconciliation, handled by two scheduled jobs — cdn:sync-redis every 15 minutes to re-push desired state, and cdn:heal-redis every 5 minutes to repair drift.

What happens when the control plane is down

Edges keep serving. The config they need is already in local Redis, and the Lua's control-plane API call is a fallback for a Redis miss, not the primary path. What stops during an outage is change: no new zones, no purges, no portal. Existing traffic is unaffected. This is the single most important property of the design and it is the reason config is pushed rather than pulled.

Purge: durable by construction

control plane
   |  RPUSH the same job onto EVERY edge's own list
   |  cdn:purge:{region}:{host_name}
   v
per-edge worker (BLPOP, systemd)
   |  {"grep": "KEY:.*example\\.com"}
   v
grep -rlP over /var/cache/nginx/{static,dynamic,api,perma}
   `-> unlink matches

Each edge owns its queue, so a node that is down or restarting still gets every purge — Redis holds the job until that node pops it. A 10,000-message backlog drains in order and nothing is dropped. The honest cost: invalidation is a filesystem scan over as much as 37 GB of cache directories, so it completes in seconds, not the sub-second global purge a larger CDN offers.

Anycast, and the part nobody tells you

Three prefixes, not one: 208.78.78.78 carries authoritative DNS, while 208.78.78.79 and 208.78.79.79 carry the data plane and are deliberately announced through separate transit providers.

The reason is a lesson that cost real debugging time. Buying transit from several providers does not mean traffic arrives over all of them. Every remote router runs its own best-path selection, and a provider reached through a chain of resellers has a longer AS path at every vantage point, so it rarely wins. With unicast that is a peering inefficiency. With anycast the winning path decides which site receives the request, so a large spread in AS path length silently concentrates traffic on one PoP regardless of geography. You cannot see this from inside your own network — it has to be read from public route collectors, which is why there is a BGP upstream checker that pulls the live table from RIPE's RIS collectors and shows the real split per prefix.

The routing layer

Traffic Router (OpenResty, 13 active nodes)

  :80   access_by_lua geo_route.lua
          GeoLite2-City lookup on client IP
          Haversine to every edge -> strictly nearest
          ties within 100 km -> consistent hash on client IP
          balancer_by_lua set_current_peer()

  :443  stream passthrough (L4) -- TLS is NOT terminated here
          upstream pool, least_conn, static distance weights:
             < 500 km -> 10      < 5000 km -> 4
             <2000 km ->  7       beyond   -> 1
          max_fails=3  fail_timeout=30s

Two consequences to be straight about. TLS terminates at the edge, not at the anycast node that accepted the TCP connection, so the handshake pays the distance to the edge. And on 443 the router spreads connections across a weighted pool rather than pinning strictly to the nearest edge — resilient and load-balanced, but it does not guarantee nearest-PoP service the way per-request geo routing on port 80 does.

Operational sharp edges, published rather than hidden

• nginx -t does not compile Lua. A syntax error in a Lua module passes config validation and then 500s every request on that node. The only real gate is luajit -b before pushing.
• A new Lua file must be added to the deployer's copy list or newly provisioned nodes 500 on everything, while existing nodes — which picked the file up during an earlier rollout — look fine. That failure mode is invisible until you build a new node, and a redeploy never fixes it.
• Smoke-test on the alternate HTTPS port, not 443, so a broken candidate config never touches live traffic.
• Redis has no config TTL and relies on write-through invalidation, so any raw or query-builder write that bypasses the model events has to clear cache explicitly.

What you get, and what you give up

Get: config that is inspectable data rather than an opaque API; edges that survive control-plane loss; per-node Redis with no cross-region blast radius; durable purge; anycast you can actually audit; and the ability to read exactly why a request was routed, cached, blocked, or challenged.

Give up: scale. 11 edge PoPs. HTTP/2 since August 2026, but no HTTP/3 or Brotli. No edge compute. No mid-tier or shield cache, so a cold object can miss at up to 11 edges independently. No instant global purge. A single-region control plane. And DDoS defence that is rate limiting, blocklists, and challenges rather than absorption capacity.

Self-controlled infrastructure is not a performance claim. It is a claim about knowing what your infrastructure does and being able to change it. Whether that is worth 11 PoPs instead of 300 depends entirely on what you are running.