Technology:Unpacked

Storage Unpacked 253 – Building Virtual SANs in the Cloud (Revisited)


Listen Later

In this episode, Chris chats with Abel Gordon, Chief System Architect at Lightbits Labs, discussing the challenges and benefits of building a virtual storage area network (SAN) on public cloud infrastructure. Lightbits originally developed the NVMe/TCP protocol and uses this feature to build virtual SANs using public cloud instances. This is a topic we first looked at in episode #210, so it’s good to get a practitioner’s experience.

Modern public cloud now features fast networking, low-latency NVMe and high-performance virtual and physical instances. Unfortunately, NVMe devices are ephemeral and any provisioned storage in the cloud is charged at full capacity. For users of on-premises SANs, the lack of thin provisioning may be an unwelcome surprise.

Why build a virtual SAN, other than to save storage costs? There’s a lot more involved, including delivering resiliency, scalability, targeted performance and capacity. Abel discusses the benefits, then goes on to enumerate the challenges involved when building on vendor-owned infrastructure. Finally, the discussion moves on to how Lightbits’ software is deployed and operated, including the managed application capability in Microsoft Azure.

For more information on Lightbits Labs, visit the company website at https://www.lightbitslabs.com/

As Abel, suggests you can contact him on LinkedIn or email him directly at [email protected].

Elapsed Time: 00:51:15

Timeline
  • 00:00:00 – Intros
  • 00:02:07 – Why build a virtual SAN in the public cloud
  • 00:04:30 – SANs balance out and fully exploit available performance resources
  • 00:06:36 – Public cloud charges for performance and capacity
  • 00:08:12 – On-premises SANs offered full flexibility to manage all metrics
  • 00:09:25 – Cloud autoscaling combined with software gives much more flexible storage
  • 00:13:27 – The on-demand nature of cloud works well for scaling SANs
  • 00:14:40 – New cloud features – NVMe, fast networking and NVMe/TCP have enabled solutions
  • 00:17:19 – What is NVMe/TCP?
  • 00:20:50 – What challenges are there in delivering a SAN on public cloud instances?
  • 00:24:03 – Cloud providers optimise for their system, not for your application
  • 00:25:02 – What operating system issues exist when building a virtual SAN?
  • 00:28:27 – Userspace operation requires a different programming strategy
  • 00:31:00 – NUMA awareness is essential, even in the public cloud
  • 00:32:46 – Each new instance type requires retesting and validation
  • 00:35:53 – What is the Lightbits solution and how is it deployed?
  • 00:37:00 – NVMe cloud drives are ephemeral
  • 00:41:35 – Snapshots work differently in the public cloud
  • 00:44:12 – Is Lightbits dedicated or HCI?
  • 00:46:32 – How is the solution consumed?
  • 00:47:29 – Azure offers management application capability
  • 00:50:36 – Wrap up
  • Related Podcasts & Blogs
    • Is the Public Cloud Becoming More Reliable?
    • Zesty Optimises AWS EC2 EBS Storage
    • Storage QoS In The Cloud
    • #97 – Building Storage Using NVMe/TCP with Kam Eshghi from Lightbits Labs
    • #121 – NVMe 1.4 Deep Dive Part II with J Metz
    • #210 – Building SANs in the Cloud
    • Copyright (c) 2016-2023 Unpacked Network. No reproduction or re-use without permission. Podcast episode #cv54.

      ...more
      View all episodesView all episodes
      Download on the App Store

      Technology:UnpackedBy Technology:Unpacked

      • 4.8
      • 4.8
      • 4.8
      • 4.8
      • 4.8

      4.8

      21 ratings


      More shows like Technology:Unpacked

      View all
      The Joe Rogan Experience by Joe Rogan

      The Joe Rogan Experience

      227,740 Listeners

      Planet Money by NPR

      Planet Money

      30,670 Listeners

      Hidden Brain by Hidden Brain, Shankar Vedantam

      Hidden Brain

      43,540 Listeners

      The Ben Shapiro Show by The Daily Wire

      The Ben Shapiro Show

      154,314 Listeners

      Motley Fool Money by The Motley Fool

      Motley Fool Money

      3,220 Listeners

      Grey Beards on Systems by Ray Lucchesi and others

      Grey Beards on Systems

      18 Listeners

      TechONTAPPodcast by NetApp

      TechONTAPPodcast

      33 Listeners

      Virtually Speaking Podcast by Virtually Speaking Podcast

      Virtually Speaking Podcast

      42 Listeners

      The Daily by The New York Times

      The Daily

      112,454 Listeners

      The Indicator from Planet Money by NPR

      The Indicator from Planet Money

      9,516 Listeners

      The Pure Report by Pure Storage

      The Pure Report

      34 Listeners

      The Journal. by The Wall Street Journal & Spotify Studios

      The Journal.

      6,053 Listeners

      Fiction - Comedy Fiction by The Sunset Explorers

      Fiction - Comedy Fiction

      6,448 Listeners

      Nothing Major by Sift Creative

      Nothing Major

      524 Listeners

      The Wirecutter Show by The New York Times

      The Wirecutter Show

      1,157 Listeners