OVS Orbit by Ben Pfaff

February 16, 2017 Trumpet: Timely and Precise Triggers in Data Center Networks, with Minlan Yu from Yale

Minlan Yu is an

associate professor of computer science at Yale and before that at USC.

She works in several areas that are relevant to Open vSwitch, including

network virtualization and software-defined networking, enterprise and

data center networks, and distributed systems.

We begin by talking about ACM

SIGCOMM 2017 1st International Workshop on Hot Topics in Container

Networking and Networked Systems, which Minlan is co-chairing.

“HotConNet” is a new workshop at SIGCOMM this year focusing on

networking for containers, which is far from a mature field. The

workshop is accepting 6-page papers until March 24. Workshop papers will

be presented as part of SIGCOMM 2017 in Los Angeles, August 21 to 23.

The main topic of discussion is Trumpet:

Timely and Precise Triggers in Data Centers, published at SIGCOMM in

August 2016. The abstract for this paper is:

As data centers grow larger and strive to provide tight performance and

availability SLAs, their monitoring infrastructure must move from passive

systems that provide aggregated inputs to human operators, to active

systems that enable programmed control. In this paper, we propose

Trumpet, an event monitoring system that leverages CPU resources and

end-host programmability, to monitor every packet and report events at

millisecond timescales. Trumpet users can express many network-wide

events, and the system efficiently detects these events using

triggers at end-hosts. Using careful design, Trumpet can

evaluate triggers by inspecting every packet at full line rate even on

future generations of NICs, scale to thousands of triggers per end-host

while bounding packet processing delay to a few microseconds, and report

events to a controller within 10 milliseconds, even in the presence of

attacks. We demonstrate these properties using an implementation of

Trumpet, and also show that it allows operators to describe new network

events such as detecting correlated bursts and loss, identifying the root

cause of transient congestion, and detecting short-term anomalies at the

scale of a data center tenant.

Afterward, Minlan provides a sneak peak at a paper to be presented at

NSDI in March, “CherryPick: Adaptively Unearthing the Best Cloud

Configurations.” This paper will present an efficient way to choose the

best machine type and parameters in a public from among the hundreds of

configurations offered by large public clouds.

Minlan may be contacted via email: minlan.yu (at) yale (dot) edu. You

can also follow her on

Twitter.

OVS Orbit is produced by Ben Pfaff. The

intro music in this episode is Drive,

music is Yeah Ant

music is Space

content is licensed under a Creative Commons Attribution 3.0

Unported (CC BY 3.0) license.

37 min

January 29, 2017 Software Synthesis for Networks, with Nate Foster from Cornell

Nate Foster is an Associate Professor of Computer Science at Cornell

University and a Visiting Researcher at Barefoot Networks. The goal of

his research is developing programming languages and tools for building

reliable systems. He received a PhD in Computer Science from the

University of Pennsylvania in 2009, an MPhil in History and Philosophy of

Science from Cambridge University in 2008, and a BA in Computer Science

from Williams College in 2001. His awards include a Sloan Research

Fellowship, an NSF CAREER Award, a Most Influential POPL Paper Award, a

Tien ’72 Teaching Award, a Google Research Award, a Yahoo! Academic

Career Enhancement Award, and the Morris and Dorothy Rubinoff Award.

This is a recording of a talk that Nate gave at the Stanford Networking Seminar,

organized by Lavanya Jose

and Eyal Cidon, on December 8,

2016. Slides

and video (recorded by the

organizers) are also available.

Nate describes this talk as:

Software synthesis is a powerful technique that can dramatically

increase the productivity of programmers by automating the construction

of complex code. One area where synthesis seems particularly promising

is in computer networks. Although SDN architectures make it possible to

build rich applications in software, programmers today are forced to

deal with numerous low-level details such as encoding high-level

policies using low-level hardware primitives, processing asynchronous

events, dealing with unexpected failures, etc.

This talk will present highlights from recent work using synthesis to

generate correct-by-construction network programs. In the first part of

the talk, I will describe an approach for generating configuration

updates that are guaranteed to preserve specified invariants. In the

second part of the talk, I will present an extension that supports

finer-grained updates triggered by data-plane events.

Joint work with Pavol Cerny (University of Colorado at Boulder),

Jedidiah McClurg (University of Colorado at Boulder), Hossein Hojjat

(Rochester Institute of Technology), Andrew Noyes (Google), and Todd

Warszawski (Stanford University).

Nick McKeown introduces

the talk.

To get the most from this talk, listen to it along while viewing the slides.

OVS Orbit is produced by Ben Pfaff. The

intro music in this episode is Drive,

music is Yeah Ant

music is Space

content is licensed under a Creative Commons Attribution 3.0

Unported (CC BY 3.0) license.

1 hr 1 min

January 16, 2017 The IO Visor Project, with Brenden Blanco from VMware

Brenden Blanco is one of the most prolific developers working on the IO Visor Project, a Linux Foundation

Collaborative Project. Brenden was an employee at PLUMgrid, the startup behind IO

Visor, until it was acquired by VMware.

The interview begins with a history of the layers that stacked up to form

IO Visor. The history begins with the publication of The

BSD Packet Filter: A New Architecture for User-Level Packet Capture

at the 1993 Winter USENIX Conference, in January 1993. This paper

introduced BPF, short for Berkeley Packet Filter, for selecting packets

to be copied to userspace for analysis. BPF was a register-based,

RISC-like virtual machine (analogous to the Java virtual machine). In

Brenden's words:

Two of the major points really seemed very wise to me. It must be

protocol-independent: the kernel should not have to be modified to add

more protocol support... It must be general: the instruction set should

be rich enough to handle unforeseen uses. They have a few more points

about efficiency and generality, but those two first ones have really

ended up creating kind of a platform that has stood the test of time.

Safety was also critical:

Any operating system developer has an inherent distrust of the userspace,

and the system calls that they're defining need to be secure against

attack, as well as ignorance or bad programming. The API they expose

lets a code be uploaded but the kernel has to protect itself against bad

code. It doesn't allow loops, so you can't create an infinite loop in

this virtual machine, and it doesn't allow access to memory that's out of

bounds... That produces something that's limited, but safe. You can't do

everything that a programmer would want, but you can do so efficiently.

BPF came to Linux during the development of the 2.5 release series, according to LWN.

A second fork of the history comes from academic research on operating

system extensibility, which was a theme that grew to prominence during

the 1990s with extensible research operating systems such as SPIN and

the Exokernel.

Many different approaches were proposed, including those based on safe

languages like Modula-3 and Java. In addition, Brenden draws an analogy

between BPF and the development of instruction set-based GPUs.

Brenden says that the movement to software-defined networking has not

been able to speed up operating system evolution. Part of the goal of IO

Visor is to help speed up this evolution, by allowing the networking

subsystem to evolve independent of the rest of the operating system. In

Brenden's words:

Five years ago or so, when we started seeing a move toward

software-defined networking, the hope was that some of this paradigm

would change a little bit. But what I've seen in some of the solutions

is that some of this actually isn't the case. You have a movement of

network functionality from hardware to software but it's still locked

into kind of the operating system life cycle, which is maybe faster than

hardware but isn't in data centers where we've been trying to address

this, it's still slower than the applications change, so the use cases

still change faster than the infrastructure.

Around 2010, Alexei Starovoitov started bringing together these two

branches by extending BPF to form eBPF, which added numerous features

such as extending registers to 32 to 64 bits, increasing the number of

registers from 2 to 10, additional instructions, and the ability to jump

forward and backward (with some restrictions) and most importantly, the

ability to call a restricted set of kernel helper functions and the

ability to work with data structures (“maps”). These changes made the

platform a better target for compiling high-level language code and

better able to interact with its environments. In addition, compilers

for eBPF started being integrated into the kernel, to make eBPF execution

faster on key architectures.

Brenden introduces the BCC, or BPF Compiler

Collection, one of the primary sub-projects within IO Visor. BCC

provides a set of tools for building software in high-level languages,

such as C and Python, into BPF program objects, loading those programs

into running kernels, and interacting with them once they are loaded.

BCC also includes a large suite of example programs.

BPF programs are loaded into a kernel by attaching them to “hook

points.” Typically, the programs attached to a hook point are invoked

when some particular event occurs. For example, in networking, a BPF

program might be invoked whenever a packet is received on a particular

device, or for performance monitoring a BPF program might be attached to

a “kprobe” point.

Brenden explains how to use BCC to load a BPF program into the kernel

using a 1-line Python program. A more sophisticated program can retain

handles and use them to interact with the program at runtime. Brenden

describes how restrictions on BPF are reflected in what a C programmer

can include in a program. Keeping the in-kernel safety verifier simple

and correct is paramount, which tends to reduce the maximum complexity of

programs that can be loaded. The verifier is a limiting factor that

continues to evolve to make BPF more useful over time.

Currently the Open vSwitch community is considering whether to replace

the use of the Open vSwitch kernel module by BPF. Brenden is in favor of

the idea and offers some of his thoughts. First, his experience at

PLUMgrid shows that BPF is flexible enough to support a wide variety of

network applications, including those that Open vSwitch implements.

Second, it's an enjoyable experience for a single developer to be able to

cover the entire infrastructure for an application.

In the future, Brenden is looking forward to BPF usage becoming

ubiquitous for Linux performance monitoring and other non-networking use

cases, such as storage acceleration and security. BPF is already, for

example, used

for sandboxing in Chrome. He's also looking forward to BPF for

hardware offload; it can already be used for hardware offload on

Netronome NICs.

For more information on IO Visor, please visit iovisor.org. To talk to Brenden and

other IO Visor developers, visit the #iovisor channel on the oftc.net IRC

network. Brenden's nick is bblanco. You can also tweet to Brenden at @BrendenBlanco.

More about BPF:

In episode 11, John Fastabend from Intel talks about

BPF on network edge nodes.

In episode 4, Thomas Graf from Cisco talks about

Cilium, which uses BPF to address the question of how to address policy

in a legacy-free container environment that scales to millions of

endpoints.

Packet

Pushers PQ Show 60, from 2015, interviewed Pere Monclus from

PLUMgrid about the IO Visor Project and Linux networking.

OVS Orbit is produced by Ben Pfaff. The

intro music in this episode is Drive,

music is Yeah Ant

music is Space

content is licensed under a Creative Commons Attribution 3.0

Unported (CC BY 3.0) license.

48 min

December 24, 2016 Benefits of Intent Based Interfaces to IT Infrastructure and Design Problems for Deploying at Scale, with Dave Lenrow from Huawei

This episode is a recording of a talk that Dave Lenrow gave at the Stanford

Platform Lab Fall Seminar on Dec. 6. Dave is Chief Architect for

Next Generation Data Center and Distinguished Engineer at Huawei. He

participates in a wide variety of open source networking projects

including ONOS, Open Daylight, OPNFV, opensourcesdn.org, and Open-O. He

is Chairman of the ONF North Bound Interface (NBI) WG, and a leader in

the OSSDN Boulder open source intent reference software project. He has

spent more than 20 years driving innovation in digital technology with an

emphasis on networks, storage and media.

You may want to follow along with Dave's slides, but the talk makes sense without

them.

The talk's abstract is:

The ONF’s North Bound Interface Work Group has been advocating a new

interface to infrastructure controllers based on describing what is

needed from the system, rather than how the system should provide it.

We’ve been trying to create a common NBI that would be used by diverse

developers and operators to create a network effect and resulting strong

platform and ecosystem. We have been describing this approach as Intent

Based networking (IBN) and recently published a consensus based document

describing the operating principles of such a system. We define an intent

based network system as one with a strict separation of

application/workload communication intent, from details about how a

network system implements and fulfills such intent (mappings). It turns

out that the implementation independent part of the description (the

intent) has several valuable properties and the promise to share state in

a way that allows massive scale and high performance. To fulfill the

promise suggested by this new interface and system architecture, we need

to get broad collaboration, and we need to solve some difficult system

and platform design problems. The goal of this talk is to get feedback

and solicit help from the platform lab in solving some of these problems

in an open collaboration. We hope to discuss the role that intent based

interfaces could serve in building a big control platform in the coming

years.

Keith Winstein introduces

the talk.

The first part of the talk is an introduction to intent-based

software-defined networking, which Dave summarizes as, “Don't tell me

what to do, tell me what you need.” Intent describes the problem,

whereas traditional CLI commands describe the solution to the problem.

In intent, you build a model of the type of workloads you have and let a

piece of software decide how to implement it. Dave gives an analogy to

health care, comparing a request for an aspirin (the status quo in

networking) to a request to cure a headache (intent-based networking).

An intent-based system breaks networking into “intent” and “mapping”

layers, where the mapping layer is the one that resolves the intent into

protocols and other implementation-level details.

One of the big benefits of intent comes from the ability to run more than

one service. A lot of existing SDN controllers can support a single

application. Attempting to run more than one application tends to cause

them to interfere with each other, since each one tries to modify switch

state, such as flow tables, without respecting the other's needs, a

“multiple writers” problem.

The second part of the talk, which starts about half an hour in,

describes a set of open problems. Many of these reference the Big

Control Platform, an ongoing Platform Lab project that seeks to build

infrastructure for controlling collaborative device swarms.

The talk ends with over 20 minutes of questions, beginning at 38:04, from

Keith Winstein, Mendel

Rosenblum, Bob

Lantz, and others.

OVS Orbit is produced by Ben Pfaff. The

intro music in this episode is Drive,

Yeah Ant

music is Space

Unported (CC BY 3.0) license.

1 hr 1 min

December 13, 2016 Container Integration with OVS, with Gurucharan Shetty from VMware

Guru Shetty has been working at VMware, and before that at Nicira, since

about 2011. Guru has done a lot of work to integrate Open vSwitch with

various platforms. This episode is a conversation with Guru primarily

about integrating Open vSwitch and OVN with multiple container systems,

including Docker and Kubernetes.

Guru begins by reiterating the rationale for containers from an

application developer point of view: to capture and encapsulate

dependencies into usable pieces and thereby make the software easier to

consume. This is not so different from a VM, conceptually, but

containers have some advantages in terms of size and life-cycle

management. Microservices, on the other hand, are a way of building an

application from many independently deployed and upgradeable components

that interact through well-defined APIs. The components, which are the

microservices themselves, could be housed on physical machines or in VMs

but are usually in containers.

To introduce the topic of how Open vSwitch integrates with container

systems, Guru begins by describing how Open vSwitch integrates with any

kind of virtualization. This is divided into three parts. First, the

virtualization system must attach its virtual interfaces to Open vSwitch.

This is usually simple: for a system that already integrates with the

Linux bridge, for example, it might entail calling into the OVS

ovs-vsctl program instead of the bridge's

brctl. Second, the virtualization system needs to provide

some metadata to allow a network controller to identify the relationship

between VIFs and VMs. These first two pieces do not differ much between

VM and container virtualization.

The third part of integrating a virtualization system with Open vSwitch

is a matter of matching up the virtualization system's networking

concepts with those understood by the network controller. OpenStack, for

example, has switch and router concepts and supports various ways to

connect virtual networks to external physical ones. Container systems,

though, generally have much simpler ideas of how to interact with

networks, perhaps because they are designed by and for application

developers who have little interest in operating a network as opposed to

simply using one. This makes container systems harder to integrate with

featureful virtual networking systems such as OVN, since they simply lack

many of the concepts. Guru makes the counterpoint, though, that

OpenStack and similar ask application developers to work with a lot of

low-level concepts that really have no direct importance to them.

Rather, networking should “just work.”

OVS integration with Docker evolved through a few steps. The first step

was a form of integration without assistance from Docker itself, through

an external wrapper that built a network namespace and configured Docker

to use it through a veth pair. This was not ideal, so

Guru proposed a native integration framework for Docker. At the same

time, there were about a dozen startups and companies all trying to do

something similar, which led to a lot of fighting over the approach.

Eventually Docker elected to provide a pluggable architecture, although

its form is still unsatisfactory for many purposes and thus a number of

vendors continue to use a “side-band” or wrapper approach. A form of

the latter has been standardized as CNI, which is what

Kubernetes and CoreOS and Mesos use.

Guru has also implemented integration between Open vSwitch and

Kubernetes, the wildly popular container orchestration system from

Google. It approaches networking problems from an application deployer's

perspective. For example, an application developer does not care about

IP addresses, so Kubernetes hides those details, instead exposing

services through higher-level names that are more meaningful to the

developers. Kubernetes also exposes every application through a load

balancer that allows the application to scale by increasing or decreasing

the number of containers behind the load balancer, in a manner hidden

from the application's clients. Guru implemented Kubernetes load

balancing support for OVN through the NAT feature added to recent

versions of Open vSwitch.

The OVN support for Kubernetes is implemented in a repository separate

from the main Open vSwitch repository, which is unusual for OVN and Open

vSwitch. This was done because Kubernetes changes much faster than Open

vSwitch, so that OVN-Kubernetes integration needs to change much more

quickly too.

One possibility for the future of the OVN-Kubernetes integration is to

take advantage of the Open vSwitch support for DPDK to build a very fast

north-south gateway for Kubernetes.

Guru talks briefly about the possibilities for Open vSwitch integration

with Mesos, Pivotal Cloud Foundry, and Rocket.

Guru talks about how OVN might be a useful component for service function

chaining in container environments, where its general concept of network

virtualization gives a lot of value versus ad hoc techniques.

Sometimes containers are portrayed as having, compared to VMs, huge

numbers per host and high rates of change. Guru has not seen this in

practice. While containers are shorter-lived than VMs, the scale is not

much higher.

OVS Orbit is produced by Ben Pfaff. The

intro music in this episode is Drive,

Yeah Ant

music is Space

Unported (CC BY 3.0) license.

45 min

November 29, 2016 Protocol-Independent FIB Architecture, with Ryo Nakamura from University of Tokyo

Ryo Nakamura is a PhD

student at The University of Tokyo, studying IP networking, overlay

networking, and network operation. This episode is a recording I made of

his talk during APSys 2016, the

Asia-Pacific Workshop on Systems, on Aug. 5, based on Protocol-Independent FIB

Architecture for Network Overlays, written with co-authors Yohei Kuga

(from Keio University), Yuji Sekiya, and Hiroshi Esaki.

The abstract for this paper says:

We introduce a new forwarding information base architecture into the

stacked layering model for network overlays. In recent data center

networks, network overlay built upon tunneling protocols becomes an

essential technology for virtualized environments. However, the tunneling

stacks network layers twice in the host OS, so that processing to

transmit packets increases and throughput will degrade. First, this paper

shows the measurement result of the degradation on a Linux kernel, in

which throughputs in 5 tunneling protocols degrade by over 30%. Then, we

describe the proposed architecture that enables the shortcut for the

second protocol processing for network overlays. In the evaluation with a

dummy interface and a modified Intel 10-Gbps NIC driver, transmitting

throughput is improved in 5 tunneling protocols and the throughput of the

Linux kernel is approximately doubled in particular protocols.

Before the talk, session chair Florin Dinu introduces the

speaker. Following the talk, the questions come from Ben Pfaff, Sorav Bansal, and

Florin, respectively. Sorav's question refers to my own talk from

earlier the same day at the conference, which is published as OVS Orbit Episode 14.

OVS Orbit is produced by Ben Pfaff. The

intro music in this episode is Drive,

Yeah Ant

music is Space

Unported (CC BY 3.0) license.

25 min

November 13, 2016 The Faucet SDN Controller, with Josh Bailey from Google and Shivaram Mysore from ONF

Faucet is an open source SDN controller developed by a community that

includes engineers at Google's New Zealand office, the Open Networking

Foundation (ONF), and others. This episode is an interview with Josh Bailey

from Google and Shivaram Mysore from the ONF. It was recorded on Nov. 7,

at Open vSwitch 2016

Fall Conference.

The episode begins with a description of Faucet's goals. Unlike the

higher profile Open Daylight

and ONOS controllers, which focus

on performance at high scale, Faucet places simplicity, ease of

development, and small code size as higher purposes.

Also in contrast to most controllers, Faucet does not contain code

specific to individual vendors or models of OpenFlow switch. Rather, it

targets any OpenFlow 1.3 switches that fulfill its minimum multi-table

and other requirements, using a pipeline of tables designed to be

suitable for many purposes. In Josh's words, “The most important one

was tables. Once you have tables, you can say `if-then'. If you don't

have tables, you can only go `if-and-and-and-and'.”

Faucet development has focused on deployments. Several Faucet users have

come forward to publicly talk about their use, with the highest profile

of those being the Open

Networking Foundation deployment at their own offices. See also a map

of public deployments. Shiva describes a temporary deployment at the

ONF Member Workdays for conference wi-fi use.

Performance is not a focus for Faucet. Instead, developers encourage

users to experiment with deployments and find out whether there is an

actual performance in practice. Shivaram reports that this has worked

out well.

Faucet can control even very low-end switches, such as the Zodiac, a

4-port switch from Northbound Networks that costs AUD $99 (about USD

$75). Faucet itself has low memory and CPU requirements, which mean that

it can run on low-end hardware such as Raspberry Pi (about $30), which

has actually been deployed as a production controller for enterprise use.

Last summer, the ONF hosted a Faucet hackfest in

Bangalore, where each team was supplied its own “Pizod,” a combination

of a Zodiac and Raspberry Pi, for development. Hackers at the hackfest

were required to have Python experience, but not networking or OpenFlow

experience. Each team of 4, which included a documentation and a UX

person, chose a project from an assigned list of possibilities.

Faucet records the state of the system, over time, to an InfluxDB

database and exposes that for inspection through a Grafana dashboard.

The Faucet code is small, about 2,500 lines of code. About this size,

Josh says, “I'd be surprised if it gets about four times the size,

because we've got quite a clear idea of its scope... Think of Faucet as

your autonomic nervous system, a small important part of your brain but

it keeps you breathing and it reacts to high-priority threats before your

conscious mind sets in. You keep that code small and you test the heck

out of it.”

Josh is working on extending support for distributed switching within

Faucet. Troubleshooting large L2 fabrics is especially frustrating, and

Josh aims to make it easier. Shiva is encouraging deployments,

especially feedback from deployments, and control over wi-fi. Other

priorities are better dashboards and better IPv6 support.

For more information on Faucet, visit the Faucet blog, read the ACM Queue article on

Faucet, dive into the

Faucet Github repo, or search for “Faucet SDN” on Youtube.

OVS Orbit is produced by Ben Pfaff. The

intro music in this episode is Drive,

Yeah Ant

music is Space

Unported (CC BY 3.0) license.

47 min

October 28, 2016 OVN Launch, with Russell Bryant from Red Hat

OVN is a network virtualization system that has been under development as

part of the Open vSwitch project over about the last two years. On this

podcast, Ben Pfaff and Russell Bryant, two major contributors to OVN,

describe OVN, its architecture, its features, focusing on features that

were added in the recent release of Open vSwitch 2.6, and some future

directions.

This episode is based on the material presented at the OpenStack Summit

in the session titled “OVN - Moving to Production.” The

summit talk was recorded and video

and slides

are available. This podcast follows the structure of the slides pretty

closely, making them a useful companion document to look at while

listening, but the episode is meant to stand alone.

Resources mentioned in this episode:

Discussion of databases for OVN: ovs-dev

list archive.

Continuous integration keynote, “Demoing the World's Largest

Multi-Cloud CI Application”: video.

OVN SFC talk, “Delivering OpenStack NFV Service Chaining at Scale with

Networking-SFC and Networking-OVN”: video.

Open vSwitch/OVN Git repository at Github.

Open vSwitch development

and general

discussion mailing lists. (You can also reach the list archives

from these pages.)

ovn-kubernetes integration Git repository at Github.

Russell Bryant's blog.

Open vSwitch 2016 Fall Conference, to be held Nov. 7 and 8 in San Jose

California, details,

agenda,

and (until Nov. 7) online

registration.

Russell Bryant is a software developer in Red Hat's Office of the CTO.

You can find him at russellbryant.net or on

Twitter as @russellbryant.

OVS Orbit is produced by Ben Pfaff. The

intro music in this episode is Drive,

Yeah Ant

music is Space

Unported (CC BY 3.0) license.

36 min

October 13, 2016 Debugging OpenStack Problems using a State Graph Approach, with Yong Xiang from Tsinghua University

Yong Xiang is a PhD student at Tsinghua University. This episode is a

recording of his talk during APSys 2016, the

Asia-Pacific Workshop on Systems, on Aug. 5, based on Debugging OpenStack

Problems Using a State Graph Approach, written with co-authors Hu Li,

Sen Wang, Charley Peter Chen, and Wei Xu, which was awarded “best

paper” at the conference. A preprint of the paper is also available at arxiv.org.

Slides

from the talk are available. It is probably easier to follow the talk if

you have the slides available, but it is certainly not necessary.

This is a very practical paper that seeks ways to make it easier for

non-experts to troubleshoot and debug an OpenStack deployment. Its

abstract is:

It is hard to operate and debug systems like OpenStack that integrate

many independently developed modules with multiple levels of

abstractions. A major challenge is to navigate through the complex

dependencies and relationships of the states in different modules or

subsystems, to ensure the correctness and consistency of these states. We

present a system that captures the runtime states and events from the

entire OpenStack-Ceph stack, and automatically organizes these data into

a graph that we call system operation state graph (SOSG). With SOSG we

can use intuitive graph traversal techniques to solve problems like

reasoning about the state of a virtual machine. Also, using a graph-based

anomaly detection, we can automatically discover hidden problems in

OpenStack. We have a scalable implementation of SOSG, and evaluate the

approach on a 125-node production OpenStack cluster, finding a number of

interesting problems.

The first question at the end of the talk comes from me, with an answer

assisted by the paper's coauthor Wei Xu, and the second one from Sorav Bansal.

OVS Orbit is produced by Ben Pfaff. The

intro music in this episode is Drive,

Yeah Ant

music is Space

Unported (CC BY 3.0) license.

20 min

September 26, 2016 Tunneling and Encapsulation, with Jesse Gross from VMware

Tunneling and encapsulation, with protocols from GRE to Geneve, have been

a key part of Open vSwitch since the early days. Jesse Gross, an early

employee at Nicira and major contributor to Open vSwitch, and perhaps

most importantly the maintainer of the Open vSwitch kernel module, joins

this episode of the podcast to talk about this aspect of OVS.

The conversation begins with a discussion of the reasons for L2-in-L3

tunnels. Jesse's reasons for such tunnels include adding a layer of

indirection between physical and virtual networks. VLANs can provide a

solution for partitioning networks, but they don't provide the same layer

of indirection.

Jesse describes the motivation for designing and implementing STT

encapsulation in Open vSwitch. The biggest reason was performance,

primarily the cost of losing the network card hardware support for

various kinds of offloads, such as checksum and TCP segmentation offload

support. Most network cards can only implement these for specific

protocols, so that using an encapsulation not specifically supported by

the card caused performance degradation. STT worked around this by using

(abusing?) TCP as an encapsulation. Since most network cards can offload

TCP processing, this allowed STT encapsulation to be very fast on both

the send and receive sides. Jesse also describes the challenges in

implementing STT and his view of STT's future.

Whereas STT was designed as a performance hack for existing network

cards, Geneve, the second encapsulation that Jesse designed and

implemented in Open vSwitch, addresses the paucity of metadata that the

existing tunneling protocols supported. GRE, for example, supports a

32-bit key, VXLAN supports a 24-bit VNI, STT a 64-bit key, and so on.

None of them supported a large or flexible amount. Geneve, on the other

hand, supports an almost arbitrary number of type-length-value (TLV)

options, intended to be future-proof. Geneve has been working its way

through the IETF for about 3 1/2 years.

Jesse talks about NSH (Network Service Header), which is often mentioned

in conjunction with Geneve. NSH has some specialization for service

function changing, whereas Geneve takes a more general-purpose stance.

NSH does support TLVs, but its primary focus is on a fixed number of

fixed-size headers that it keeps in the packet, and that is what most

implementations of Geneve actually implement. NSH can used inside L2 or

L3, whereas Geneve as currently runs only inside L3. Jesse discusses

pros and cons to each design.

Jesse discusses MTU issues in tunneling and encapsulation, which come up

because they techniques add bytes to each packet, making packets that are

maximum length before encapsulation exceed the MTU. Jesse says that the

solution to MTU problems depends on the use case: for example, in data

center use cases, a simple solution can be to increase the MTU of the

physical network. In the pre-1.10 era, Open vSwitch supported path MTU

discovery for tunnels, and Jesse describes why it was dropped and what it

would take to reintroduce it.

Jesse describes CAPWAP tunneling, why OVS implemented it, and why OVS

dropped support.

Jesse describes GTP tunneling and the potential for including it in OVS,

as well as ERSPAN encapsulation.

Jesse describes the challenges of encapsulations for IP (as opposed to

encapsulations for Ethernet).

Jesse lays out some thoughts on the future of tunneling in Open vSwitch.

OVS Orbit is produced by Ben Pfaff. The

intro music in this episode is Drive,

Yeah Ant

music is Space

Unported (CC BY 3.0) license.

35 min

Share OVS Orbit

Sign up to save your podcasts

OVS Orbit

OVS Orbit episodes

About OVS Orbit