Interview with Thomas Graf of Cisco, regarding the Cilium project.

Cilium is a “science project” that Thomas and others at Cisco and

elsewhere are hacking on, to address the question of how to address

policy in a legacy-free container environment that scales to millions of

endpoints. It's an experiment because the outcome isn't yet certain, and

it's a question that hasn't seen much work outside of hyperscale

providers.

Cilium is based on eBPF, a

Linux kernel technology that introduces the ability for userspace to

inject custom programs into the kernel using a bytecode analogous to Java

virtual machine bytecode. Cilium uses eBPF-based hooks can intercept

packets at various places in their path through the kernel to implement a

flexible policy engine.

Topics include:

How Chris Wright encouraged Thomas to become involved with Open

vSwitch, when Thomas was at Red Hat.

The important differences between containers and VMs (quantity,

duration of workloads, and frequency of state changes) and how Cilium

addresses these issues.

The toolchain that Cilium uses to generate a eBPF program customized

for the policy of each container in a minimal and complete way.

The potential to bypass kernel sk_buff overhead by

intercepting packets before these data structures are constructed, via

Express Data Path (XDP), leading to the potential for DPDK-like packet

forwarding performance without ever leaving the kernel. What's the

drawback? We don't know yet how it will play out.

The main benefit of getting XDP-based early access to packets is to

avoid the main Linux networking code paths, which are optimized for

delivery to socket buffers (taking advantage of segmentation offload)

as opposed to forwarding.

Dropping packets quickly is important because many operators are under

attack all the time.

Importance of performance for small and large packets.

Languages available for writing eBPF: C via LLVM/Clang and Python

(among others?).

Limitations due to the verifier (primarily restrictions on loops), and

how to work around them.

How Cilium generates its eBPF programs: a base C program, plus an agent

in Go that generates a C header file. Cilium compiles the C program to

eBPF bytecode, using LLVM, then loads it into the running kernel with

the tc utility.

Potential for difficulty in getting a compiler toolchain into

production deployments. Is the simplicity of supplying Cilium as a

container image that builds in the toolchain an advantage?

How often eBPF programs need to be recompiled in Cilium.

eBPF “map” data structures that can be shared among eBPF programs,

the rest of the kernel, and userspace.

Policy in Cilium. The basic idea is that whoever specifies Cilium

policy should not have to understand traditional networking concepts

like IP addresses ands port. Instead, abstract labels specify which

classes of containers can talk to each other.

Lessons learned from policy in Cilium.

How Cilium does datapath packet processing, how it passes labels from

source to destination, and where it applies policy.

The direction in which Cilium points for eBPF support in Open vSwitch:

first, it shows that it is possible; second, it shows that the tracing

buffer mechanism available from eBPF is a potential replacement for

Open vSwitch “upcalls” currently implemented via Netlink; third, it

points out an alternative for the flow-based model. (Does it makes

sense to implement OVN directly via code generation?)

Connection tracking in eBPF.

eBPF helper functions in Linux, and the limitations of the current

ones.

Potential for applying eBPF to other targets such as DPDK or the OVS

port to Hyper-V.

Performance penalty for eBPF versus native code.

Early controversy in the kernel community over eBPF when it was

introduced.

What's next for Cilium: load balancing, IPsec, IPv4 .

More information about Cilium: slides

and the code repository.

You can find Thomas on the ovs-dev mailing

list, @tgraf__ on Twitter,

or on Facebook.

OVS Orbit is produced by Ben Pfaff. The

intro and bumper music is Electro

Deluxe, featuring Gurdonack, copyright 2014 by My Free Mickey. The

outro music is Girls like

you, featuring Thespinwires, copyright 2014 by Stefan Kartenberg.

All content is licensed under a Creative Commons Attribution 3.0

Unported (CC BY 3.0) license.