This week, Chris Mellor is back and the team take a dive into the subject of storage vendor hero numbers.  What are hero numbers?  We’ve all seen them, they’re the huge performance figures quoted by all-flash storage vendors aimed at putting their products forward in the best light possible.

Are hero numbers believable or should we be looking at certified vendor benchmark testing as a guide to capability?  Do users even look at benchmark or hero numbers in the first place?  Could the whole exercise be a waste of time?  The conversation moves to talk about STAC (Securities Technologies Analysis Center) and whether more application specific benchmarks have greater value.  This includes the approach Pure Storage appears to have taken with FlashArray and FlashBlade, only quoting application-specific testing like that of AIRI.

As the discussion moves on, the team talk about more consistent deterministic latency (more on that here) and whether public cloud service providers will ever subject their service offerings to benchmark testing (TL;DR, probably not).  As a final wrap up, the team asks, what can customers actually do to ensure they are seeing the real truth behind benchmark claims.

Elapsed Time: 00:20:48

Timeline

* 00:00:00 – Intro

* 00:01:00 – What are vendor hero numbers – ten meeelion IOPS!!

* 00:02:00 – But certified benchmarks are expensive to perform.

* 00:04:00 – Storage arrays aren’t interesting, do hero numbers help?

* 00:05:00 – Do users even care about industry standard benchmarks?

* 00:06:00 – STAC – financial application benchmarks

* 00:07:48 – Should vendors be more realistic about hero numbers?

* 00:09:50 – Why has Pure Storage stopped producing FlashArray performance data?

* 00:11:40 – However Pure do produce numbers for CI like AIRI (see Soundbytes #008).

* 00:14:30 – Do people do proofs of concept any more?

* 00:16:00 – Maybe arrays are massively over specified anyway?

* 00:16:30 – Or actually, it’s about consistent deterministic performance.

* 00:17:44 – Do we need performance benchmarks for cloud vendors?

* 00:19:00 – So what should customers do?

* 00:20:08 – Wrap Up

Related Podcasts & Blogs

* Pure Storage AIRI – CI for AI

* Why Deterministic Storage Performance is Important

* Avoiding the Storage Performance Bathtub Curve

* Avoiding All-Flash Missing IOPS

* Are DataCore’s SPC Benchmarks Unfair?

* NFS Vendor Benchmarks – Caveat Emptor

* Storage Performance: Why I/O Profiling Matters

Vendors referenced in this post: Violin Systems, Dell EMC (PowerMax), Pure Storage (FlashBlade & FlashArray), NetApp.

Copyright (c) 2016-2018 Storage Unpacked.  No reproduction or re-use without permission. Podcast Episode UGE3.