Network Services assignment: written report

Scenario summary

Acme Corp (AS 12345) is onboarding its public prefix 203.0.113.0/24 to Cloudflare Magic Transit. Cloudflare announces the /24 from its global anycast network of 300+ locations, absorbs and scrubs DDoS traffic at the nearest edge, applies Magic Firewall policy to ingress traffic, and delivers clean packets to Acme's data center over anycast GRE tunnels. Return traffic leaves Acme's network directly through its own transit providers via direct server return, without transiting Cloudflare. Prefix advertisement is controlled through an API trigger, so the announcement can be turned on and off per prefix during cutover or on demand.

Two GRE tunnels are defined:

Both tunnels share the same Cloudflare-side address because 100.100.100.100 is an anycast address: any of the 300+ edge locations can encapsulate traffic using the same source, which is what makes the tunnel design scale without per-location configuration. Because Cloudflare sees only the ingress half of every conversation, Magic Firewall is stateless by design; that fact shapes several answers below.

Q1: What Acme must supply for Cloudflare to advertise the /24

Plain language: before Cloudflare will announce address space it does not own, Acme has to prove it controls the prefix and has authorized Cloudflare to speak for it, in the specific formats that the internet's routing security infrastructure actually checks.

Concretely, Cloudflare onboarding requires:

1. A Letter of Agency (LOA), as a PDF. It names 203.0.113.0/24 and authorizes Cloudflare (AS 13335) to advertise the prefix from its network. This is the item the question hints at: tier-1 transit providers require an LOA before they will accept and propagate an announcement of customer-owned space from Cloudflare's sessions.
2. The prefix and origin ASN. 203.0.113.0/24 with origin AS 12345, per the scenario, which states the advertisement runs under Acme's own ASN. Magic Transit also supports originating from Cloudflare's ASN instead; whichever origin model is chosen at onboarding, every artifact below has to name that same origin ASN. I am assuming the AS 12345 origin model throughout because the brief specifies it, and I would confirm the choice explicitly during onboarding.
3. An RPKI ROA published at the RIR, authorizing the origin ASN for 203.0.113.0/24 with maxLength set to /24. Strictly speaking RPKI is best practice rather than universally mandatory, but networks that perform route origin validation will discard an announcement whose ROA does not match, so in practice it is required for full reachability.
4. An IRR route object (in RADB or the RIR's IRR) matching the announcement: route 203.0.113.0/24, origin AS 12345. Many networks still auto-generate their prefix filters from IRR data, so a missing route object means silent filtering.

A useful way to keep the three security artifacts straight: the LOA convinces a person, the IRR route object convinces a filter generator, and the RPKI ROA convinces a router.

Q2: External entities to inform, and what to update them with

Plain language: turning up Magic Transit is a routing change the rest of the internet has to be able to validate, so the registries and providers that vouch for Acme's prefix all need matching updates before cutover.

• The RIR (ARIN, RIPE, or equivalent for Acme's region): create or update the RPKI ROA for 203.0.113.0/24 with the chosen origin ASN and maxLength exactly /24. A looser maxLength would let a forged more-specific announcement (for example a /25) validate as RPKI-legitimate, which reopens exactly the hijack class RPKI exists to close.
• The IRR database Acme uses: publish the route object (route: 203.0.113.0/24, origin: AS 12345) so filter generators accept the new announcement path.
• Acme's existing transit and upstream providers: coordinate withdrawal of any current announcement of the /24 and agree the cutover timing. Running the old and new announcements uncoordinated produces a multiple-origin-AS conflict, and a stale ROA during the transition can make the live announcement RPKI-invalid. The upstreams also need to update any prefix filters that referenced the old arrangement, and they remain the egress path for direct server return traffic after cutover.
• PeeringDB: update Acme's records so peers and providers validating the change see consistent data.

Q3: GRE effect on packet size, and the new MSS

Plain language: wrapping every packet inside a GRE envelope costs 24 bytes, and since the wire between networks still caps packets at 1500 bytes, the packet inside the envelope has to shrink. For TCP, the way you enforce that shrinkage is the MSS value exchanged during the connection handshake, and where you enforce it is dictated by the asymmetric return path.

The arithmetic: GRE over IPv4 adds a new outer IPv4 header (20 bytes) plus a GRE header (4 bytes), 24 bytes total. With a 1500-byte physical MTU on the underlay, the tunnel MTU is 1500 - 24 = 1476. The largest TCP payload that fits is the tunnel MTU minus the inner IPv4 and TCP headers (20 + 20), so the new MSS is 1476 - 40 = 1436. (These figures are specific to GRE. IPsec carries more overhead and lands lower, per Cloudflare's per-mode guidance; the 1476/1476-40 pair should not be reused there.)

Where the clamp is applied, and why: the MSS option exists only in the two SYN-flagged segments of the three-way handshake, the SYN and the SYN-ACK. Each side announces the largest segment it can receive, and each sender honors the value its peer announced. The direction that traverses the tunnel is client to server, and the value governing what the client sends is the MSS the server announces in its SYN-ACK. But under direct server return, the SYN-ACK leaves Acme's network directly and never touches Cloudflare, so Cloudflare cannot rewrite it. The only device in the path of both the inbound SYN and the outbound SYN-ACK is Acme's data-center edge router, so that is where MSS clamping to 1436 must be configured (in practice, an adjust-MSS statement covering both the GRE tunnel interface and the WAN egress interface, so every SYN-flagged segment crossing the edge is rewritten). Relying on path MTU discovery instead is fragile because the ICMP fragmentation-needed messages it depends on are widely filtered. Non-TCP traffic has no MSS to clamp, which is why UDP-based protocols crossing the tunnel need application-level datagram sizing.

Q4: Can prefixes smaller than a /24 be routed across the GRE tunnels?

Plain language: yes. The /24 limit is about what the public internet will accept as an announcement, not about what a private tunnel can carry.

The global BGP table (the default-free zone) conventionally filters announcements longer than /24, so /24 is the advertisement floor toward the internet: Cloudflare cannot usefully announce a /25 on Acme's behalf. The GRE tunnels are a different layer entirely: they are point-to-point links carrying statically routed traffic, and Cloudflare's tunnel routing does longest-prefix matching on whatever static routes are configured (Cloudflare's own documentation uses /29 and /30 examples). Acme can therefore steer 203.0.113.0/25 down tunnel 1 to 192.0.2.10 and 203.0.113.128/25 down tunnel 2 to 192.0.3.10, or pin individual /29s for specific services to specific tunnels, while the internet only ever sees the aggregate /24. Advertise coarse, route fine.

Q5: Migrating on-premises firewall rules, given the asymmetric routing

Plain language: Cloudflare's firewall only ever sees half of each conversation, the inbound half. Any rule on the old firewall that depended on remembering a connection has to be rewritten to work without that memory.

Because return traffic uses direct server return, Magic Firewall processes ingress only, which makes it stateless as a direct consequence of the topology rather than a product limitation. There is no connection table, so there is no equivalent of the classic allow-established-and-related rule. Two rule classes need attention:

• Inbound service rules translate nearly one-to-one: allow TCP 443 to the web servers, and so on. These were never stateful in spirit.
• Return traffic for connections Acme initiates outbound is the hard case. When a host inside 203.0.113.0/24 opens a connection out, the operating system picks an ephemeral source port for it (Linux defaults to the 32768-60999 range, which is what the question's hint about source port selection points at). The reply arrives at Cloudflare inbound, addressed to that ephemeral port, and a stateless firewall has no session entry proving it was solicited. The migration pattern is to allow inbound TCP to the ephemeral port range but gate it on TCP flags so only mid-conversation segments qualify: permit packets that are not bare SYNs, for example not (tcp.flags.syn and not tcp.flags.ack). That admits the SYN-ACK and ACK-bearing segments of conversations Acme started while still refusing fresh inbound connection attempts to those ports. It is the stateless approximation of established. UDP has no flags, so solicited UDP replies need tighter source constraints instead, such as permitting UDP with source port 53 from the resolvers Acme actually queries.

Acme's on-premises firewall behind the tunnel can remain stateful for its own enforcement; the migration question is which rules move up into Magic Firewall and how their shape changes when they get there.

Q6: Packet flow, user to web server and back

Plain language: the user's packet is pulled to the nearest Cloudflare location by the anycast announcement, inspected, wrapped in a GRE envelope, and delivered to Acme's edge, where it is unwrapped and handed to the server with its original addresses intact. The reply skips Cloudflare entirely.

 DDoS attack traffic                                      legitimate user
 (floods aimed at 203.0.113.0/24,                         198.51.100.77
  drawn to the nearest edge by the                             |
  same anycast announcement)                                   |  (1) IPv4: 198.51.100.77 -> 203.0.113.80
        \                                                      |      TCP 51514 -> 443, SYN, MSS 1460, DF set
         v                                                     v
   +--------------------------------------------------------------------------+
   |          Cloudflare global anycast edge (300+ locations)                  |
   |  BGP announcement: 203.0.113.0/24, origin AS 12345 (LOA, ROA, IRR)        |
   |  (2) DDoS protection: attack traffic dropped here, at the edge            |
   |  (3) Magic Firewall: stateless ingress policy against the inner packet    |
   +--------------------------------------------------------------------------+
                               |
                               |  (4) anycast GRE tunnel (IP protocol 47)
                               |      outer IPv4: 100.100.100.100 -> 192.0.2.10      (tunnel 1)
                               |      inner IPv4: 198.51.100.77 -> 203.0.113.80      (unchanged)
                               |      tunnel MTU 1476, TCP MSS clamped to 1436
                               |      underlay: internet transit or Cloudflare
                               |      Network Interconnect; tunnel 2 to 192.0.3.10
                               |      is the redundant path (inside 10.10.10.2/31)
                               v
   +--------------------------------------------------------------------------+
   |  Acme data-center edge router                                             |
   |  WAN endpoint 192.0.2.10; GRE decapsulation, tunnel interface 10.10.10.1  |
   |  MSS clamp applied here: the only device that sees SYN and SYN-ACK        |
   +--------------------------------------------------------------------------+
                               |
                               |  (5) inner packet forwarded, original addresses intact
                               v
                      web server 203.0.113.80:443
                               |
                               |  (6) reply: 203.0.113.80 -> 198.51.100.77
                               |      native IPv4, MTU 1500, out via Acme's own transit
                               v
      legitimate user  <====  direct server return: never traverses Cloudflare

The same flow as a per-leg header table:

Two properties worth stating because they surprise people: the web server sees the real client IP (Magic Transit is layer 3; there is no proxy address and no NAT), and the return path's asymmetry is not an anomaly but the design, which is precisely why the firewall reasoning in Q5 and the MSS placement in Q3 look the way they do.

Firewall rule debugging

The exercise supplies a stateful ingress firewall protecting 203.0.113.0/24 with these rules, evaluated first match wins:

1. Does it allow all DNS queries? No. Rule 2 permits DNS over UDP 53 only. DNS also runs over TCP 53: truncated responses fall back to TCP per RFC 7766, and zone transfers are TCP-only. The only TCP allowance is rule 1, which is restricted to destination port 443, so DNS over TCP falls through to rule 6 and is denied. Fix: extend rule 2 to cover both TCP and UDP on destination port 53 (or add a matching TCP 53 rule alongside it).

2. Will it allow pings? No. Rule 4 allows ICMP but with a maximum IP length of 56 bytes. I am reading Max IP len as the total IPv4 packet length, headers included, and stating that assumption because the answer flips if the field means payload only. A default Linux ping is 56 data bytes plus 8 bytes of ICMP header plus 20 bytes of IPv4 header, 84 bytes on the wire; a default Windows ping is 32 + 8 + 20 = 60 bytes. Both exceed 56, fail rule 4's length condition, and fall through to rule 6. The likely intent was the familiar 56-byte ping payload, but as written the rule blocks standard echo requests. Fix: raise the length ceiling to at least 84 (or remove it) so default echo requests match rule 4.

3. Is TFTP available on 203.0.113.8? No. TFTP is UDP on destination port 69. Rule 5 allows all UDP, but only to 203.0.113.0/29, which covers hosts .0 through .7; the .8 address is the first host outside that block. Rule 2 allows UDP only to port 53. So UDP 69 to 203.0.113.8 matches nothing and rule 6 denies it. Fix: either widen rule 5 to 203.0.113.0/28 if the intent is the first sixteen addresses, or better, add a specific rule allowing UDP destination port 69 to 203.0.113.8/32. One operational note: TFTP servers answer from a fresh ephemeral port rather than from port 69, so this rule works together with the firewall's stateful tracking; on a stateless filter such as Magic Firewall the reply legs would need the explicit ephemeral-port treatment from Q5.

4. Are HTTP, HTTPS, and HTTP/3 supported?

• HTTP: no. Plain HTTP is TCP 80, and no rule permits it; rule 6 denies it.
• HTTPS: yes. TCP 443 matches rule 1 for the entire prefix.
• HTTP/3: no, with one accidental exception. HTTP/3 is QUIC, which runs over UDP 443. Rule 2 covers UDP 53 only. Rule 5 does allow any UDP port, but only for 203.0.113.0/29, so servers at .0 through .7 would accept HTTP/3 by side effect while the rest of the /24 cannot. As a deliberate policy, HTTP/3 is unsupported; adding a UDP 443 allowance for the prefix (mirroring rule 1) would enable it.

What I learned

• DDoS protection at this scale is a BGP product before it is a filtering product. The entire mechanism starts with anycast announcement of the customer prefix, which is why the /24 floor, the LOA, and the RPKI and IRR artifacts are onboarding requirements rather than paperwork afterthoughts. The triad stuck with me as: LOA for the humans, IRR for the filter generators, RPKI for the routers.
• Direct server return is the root cause of half of this assignment. Once return traffic bypasses Cloudflare, statelessness stops being a firewall feature choice and becomes topology. The ephemeral-port and TCP-flag reasoning in Q5 is just what established means when nobody is keeping state for you.
• The MSS answer is really a question about who sees both halves of the handshake. MSS lives only in the SYN and SYN-ACK, the SYN-ACK never touches Cloudflare, and therefore the clamp must live at the customer edge. Working that out from the handshake, rather than memorizing clamp-to-1436, is the version that survives contact with a customer conversation.
• Advertisement granularity and routing granularity are different layers. The internet sees an indivisible /24 while the GRE tunnels happily carry static /25s or /29s to different places. Advertise coarse, route fine turns out to be a generally useful mental model for any overlay.
• Firewall tables reward literal reading. The three planted bugs (a length field that blocks default pings, a /29 boundary that quietly excludes .8, and DNS-over-TCP missing entirely) are all the kind of thing that produces confident but wrong support tickets. Quoting the rule number that a packet actually matches is the discipline that catches them.