Rack-scale Networking

Bryan and Adam are joined by a number of members of the Oxide networking team to talk about the networking software that drives the Oxide rack. It turns out that rack-scale networking is hard... and has enormous benefits!

We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from February 27th, 2023. In addition to Bryan Cantrill and Adam Leventhal, speakers included Ryan Goodfellow, Levon Tarver, Ben Naecker, and Arjen Roodselaar.

LinksIntel Tofino SeriesP4 (programming language) - Wikipediap4lang/p4c: P4_16 reference compileroxidecomputer/p4: A P4 compilerThe quote crate: Rust quasi-quotingRIFT WG - Routing In Fat Trees | IETF Community WikiHere's (much of) the live chat from the show:ahl https://github.com/oxidecomputer/oxide-and-friends/blob/master/2021_11_29.mdahl That's the Sidecar switch episodebcantrill https://p4.org/admchl What does "at line rate" mean?Riking Line rate = As fast as the packets could possibly come. 1Gbit, 10Gbit, 100Gbit, etcadmchl Do you need ASICs to hit that speed? I assume x86_64 is not going to be fast enough for these specialised operations?levon Yes, the Tofino 2 is the ASICbcantrill You need ASICsbnaecker Yes, you really can't do these kinds of operations on a general purpose CPU.rng_drizzt Yeah, you need specialized silicon here.JustinAzoff Right, also often across all ports at the same time in both direction. a 48 port 10gbps switch will have a line rate of 960gbps (10 ** 48 ** 2)duckman So the advantage is being able to offload compute to the switch?bnaecker Yes, and specifically that you can separate the data plane (operations on the packets) from the control plane (decisions about what operations to allow or make).tahnok What's TCAM?levon Ternary Content Addressable Memorybnaecker https://en.wikipedia.org/wiki/Content-addressable_memory#Ternary_CAMsryaeng Sure beats logging into a number of Cisco switches and making changes at the console.admchl This is my favourite episode in a long time, this is all really fascinating.rng_drizzt the first Sidecar episode was nearly 1.5 years ago 🤯 , right after we cut the first revlevon That episode blew my mindduckman This sounds like a big deal on the scale of ebpfduckman Or biggerbnaecker It is extremely useful for understanding the processing pipelines. As long as you only run single-packet integration tests 🙂od0 just want to go out and find things to write P4 code forJustinAzoff <@354365572554948608> yeah one way to think about that sort of thing is that xdp can be used to run little programs on a nic, where p4 is kind of like that, but running on effectively a nic with 48+ portsbcantrill https://github.com/oxidecomputer/p4SyntheticGate sidecar is the "codename" of our switch boxSyntheticGate "gimlet" is our server sledbcantrill https://github.com/oxidecomputer/propoliswmf So you have P4 and OPTE in the hypervisor at the same time?bnaecker OPTE is in the host kernel.arjenroodselaar The P4 runtime Ry described only exists in the test bed, where it high level simulates the switches. OPTE is part of the production environment.arjenroodselaar The rough difference between P4 and OPTE is that P4 works on individual packets without much concept of a session (so it can't reason about TCP streams, packet order etc, so no firewall like functionality), while OPTE aims to operate on streams of packets.JustinAzoff So you can run 100 VMs on a test system and wire them up to your virtual switch compiled by x4c?arjenroodselaar Correct.bcantrill OPTE == Oxide Packet Transformation Engineadmchl Gimlet?rng_drizzt Compute serverrng_drizzt The Sidecar switch is actually just a PCIe peripheral to a Gimlet.bnaecker The Gimlet managing the Sidecar is often called a "Scrimlet" for "Sidecar attached Gimlet"Riking and "how do i reconfigure this giant network without hosing my ability to reconfigure this giant network"ShaunO can identify with that - we seriously struggle to keep our own products inter-operating, let alone anyone else'slevon It can feel like a Sisyphean task.a172 Setup a much smaller/simpler network in parallel that is accessible from "not your network" that gets you to the management interface.levon It's a whole new world when you can look at the actual table definitions in P4rng_drizzt Owning all the layers here is immensely beneficiallevon Those DTrace probes have been very helpfulbnaecker Those probes turned out to be everywhere. They are are in: SQL queries, HTTP queries, log messages, Propolis hypervisor state, virtual storage system, networking protocol messages, the P4 emulator, and probably more that I'm forgetting about.levon For those unfamiliar with the DTrace tool, or the rationale behind leveraging DTrace over other tracing / debugging tools: https://www.cs.princeton.edu/courses/archive/fall05/cos518/papers/dtrace.pdfbcantrill https://github.com/oxidecomputer/progenitorahl some notes on rust codegen: https://github.com/ahl/codegen-templatearjenroodselaar DDM! Bring us home!a172 it astonishes me how many "cloud" type architectures are built on v4 only or v4 first.a172 IPv6 is older than Wi-Fia172 It solves real problems. PLEASE use it.nyanotech yessss fina...