Storage Developer Conference

In this tutorial, we will introduce the audience to the lunatic fringe of extreme high-performance computing and its storage systems. The most difficult challenge in HPC storage is caused by millions (soon to be billions) of simultaneously writing threads. Although cloud providers handle workloads of comparable, or larger, aggregate scale, the HPC challenge is unique because the concurrent writers are modifying shared data.

We will begin with a brief history of HPC computing covering the previous few decades, bringing us into the petaflop era which started in 2009. Then we will discuss the unique computational science in HPC so that the audience can understand the unavoidability of its unique storage challenges. We will then move into a discussion of archival storage and the hardware and software technologies needed to store today’s exabytes of data forever. From archive we will move into the parallel file systems of today and will end the lecture portion of the tutorial with a discussion of anticipated HPC storage systems of tomorrow. Of particular focus will be namespaces handling concurrent modifications to billions of entries as this is what we believe will be the largest challenge in the exascale era.

Caches in modern storage systems lack the ability to adapt automatically and optimize for dynamic workload mixes. Despite the potential for huge improvements in cost, performance, and predictability, such adaptability is extremely challenging, due to inherently complex, non-linear, and workload-dependent behavior. Even when manually-tunable controls are provided to support dynamic cache sizing, partitioning, and parameter tuning, administrators simply don’t have the information required to make good decisions.

In this talk, we will present an overview of the significant opportunity for self-optimizing caches by examining several examples from production systems. We will review recently-published research in this area, including robust, general methods for efficient cache modeling. Optimizations that leverage these models promise to improve the performance of most workloads and cache policies automatically.

The Storage Performance Development Kit (SPDK) is an open source set of tools and libraries for writing high performance, scalable, user-mode storage applications. It achieves high performance by moving all of the necessary drivers into userspace and operating in a polled mode instead of relying on interrupts. The Blobstore is a relative newcomer to SPDK and provides local, persistent, power-fail safe block allocator designed to replace filesystem usage in many popular databases. Most importantly, the Blobstore has been designed for the properties of flash and next-generation media from the start and directly leverages NVMe features. The team has already ported a popular embedded key/value database, RocksDB, to use the Blobstore which demonstrated a significant improvement for database queries under common workloads. In this session we will explore the basics of the Blobstore and review some of the latest exciting performance data!

The Windows Server 2016 release contains support for Persistent Memory including the introduction of DAX volumes. This presentation will discuss the PM improvemenetns in Windows since this release. We will also review Windows support of the NVML library.

Open Channel describes a new interface to Solid State Drives (SSDs) which promises to increase usage of SSDs’ raw bandwidth from 40% to 95%, increase user-visible flash capacity from 50%-70% to 99%, increase I/O bandwidth by 3x and reduce per-GB hardware cost by 50%. Despite many proposals and implementations proving these benefits, industry has seen limited adoption and no standards body has integrated the concept. One of the largest hurdles to adoption is that the proposed changes permeate every layer in the storage stack, from device firmware to application. To reap the benefits, we need not only an end goal, but a pragmatic approach to introducing these changes to one or two layers at a time.

We present relevant information about host and drive architecture, the expected use cases for Open Channel, and a general-purpose, maintainable end target for Open Channel. The final architecture refactors Flash Translation Layer into Log-Management, handled in the host, and NAND Management, handled in the drive. This consolidates repeated software algorithms, and retains media-specific functionality in the drive.

The results of this design set up the storage community to innovate in traditionally independent areas. With a new interface in place, host-side software developers can apply a wide variety of software and technologies to further optimize their storage logs, and firmware and drive designers can focus on improvements in media management and density improvements.

PCIe devices such as GPGPUs, FPGA accerlators, RDMA enabled NICs and NVM Express SSDs are placing a huge strain on the IO subsystem of the CPUs they are connected too. With the emergence of NVMe over Fabrics and heterogeneous compute there is often a desire to move large stream of data between these endpoints without CPU intervention.

In this presentation we give an overview of the latest work we have done to enable this PCIe Peer-2-Peer (P2P) communication in the Linux kernel. We also cover a detailed performance comparison between normal data flows and those that avail of P2P. We show how the P2P data-flows offload the IO subsystem of the CPU and lead to better throughput, latency and Quality of Service.

RDMA-enabled storage protocols (including SMB3, NFSv4, NVMe over Fabrics, etc), and RDMA protocols (RoCE, iWARP, Infiniband, etc) are being extended to enable remote access to Persistent Memory. Data access at ultra-low remote latency, with high fidelity to local semantics, are goals which are well within reach. Beyond the data itself, can other aspects of storage, such as security and data integrity, be addressed with similar network-based solutions, without upper-layer or CPU interaction? What additional types of protocol extensions might be envisioned to provide similar latency and offload benefits from storage and network adapters, for these other important storage attributes?

This talk will explore the landscape of existing protocol-based access to Persistent Memory, and prioritize additional requirements which merit consideration. It will present a set of architectural tools, with practical protocol implementation approaches, to extend the benefits of RDMA to a broader range of storage solutions based on emerging memory storage technologies.

In this presentation, we introduce accelerated NVMe over fabrics (NVMe-oF) target and host solutions with SPDK (storage performance development kit) library. SPDK vhost is a userspace target designed to extend the performance efficiencies of SPDK into QEMU/KVM virtualization environments. Compared with existing kernel based vhost solutions, SPDK vhost target provides up to 10x better performance and/or efficiency while enabling significantly lower latency when used with Intel Optane media. Using both SPDK based NVMe-oF target and host, the round trip latency of single I/O can be reduced to 50% compared the total solution provided by Linux kernel. This session covers the performance of SPDK vhost target, design principles for CPU efficiency, usage models and future plans.

Linux gained the first publicly release driver for the then new NVM Express standard in early 2011. It went through a lot of changes since then, including the move the new blk-mq block driver infrastructure, the addition of NVMe over Fabrics support including the split of the driver

into a common core and transport drivers and the addition of lots of feature since then.

This talk starts by explaining the historic development of the Linux NVMe driver including it’s influence on common Linux code and then presents the current new features and developments including power management, host memory buffer support, adaptive I/O polling support and the latest and greatest in Fabrics including multipathing support.

This talk will explore new capabilities that are proposed to be added to SMB3 for supporting emerging scenarios like containers, direct-access filesystem (DAX ) and large scale-out clusters with affinitized storage.

Containerized workloads accessing data hosted on a remote share typically authenticate to the remote resource using the identity of the container or a pre-plumbed credential setup by an administrator. We explore “identity tunneling” extensions to the protocol to allow applications to tunnel their identity to the server on top of an existing authenticated SMB session.

Direct access storage (Storage Class Memory (SCM) /Persistent Memory (PM)) is now being deployed in more and more scenarios where low-latency IO is required. SMBDirect/RDMA provides a mechanism for clients to directly access DAX storage on a server. We will look at various approaches to accessing DAX storage – first via the existing SMB3 protocol and potential enhancements to mostly bypass the software stack on the client and server.

Lastly in the context of a large scale-out cluster, we’ll take a look at enhancements to the SMB3 protocol to automatically direct the cluster-node hosting the data need by the client. This complements some of the functionality provided by the Witness protocol – and is a step towards scaling out data access via the SMB3 protocol.