The Backend Engineering Show with Hussein Nasser

Postgres is combining IO in version 17


Listen Later


Learn more about database and OS internals, check out my courses 

Fundamentals of database engineering https://databases.win 

Fundamentals of operating systems https://oscourse.win



This new PostgreSQL 17 feature is game changer.


You see, postgres like most databases work with fixed size pages. Pretty much everything is in this format, indexes, table data, etc. Those pages are 8K in size, each page will have the rows, or index tuples and a fixed header. The pages are just bytes in files and they are read and cached in the buffer pool.


To read page 0, for example, you would call read on offset 0 for 8192 bytes, To read page 1 that is another read system call from offset 8193 for 8192, page 7 is offset 57,345 for 8192 and so on. 


If table is 100 pages stored a file, to do a full table scan, we would be making 100 system calls, each system call had an overhead (I talk about all of that in my OS course). 


The enhancement in Postgres 17 is to combine I/Os you can specify how much IO to combine, so technically while possible you can scan that entire table in one system call doesn’t mean its always a good idea of course and Ill talk about that. 


This also seems to included a vectorized I/O, with preadv system call which takes an array of offsets and lengths for random reads. 


The challenge will become how to not read too much, say I’m doing a seq scan to find something, I read page 0 and found it and quit I don’t need to read any more pages. With this feature I might read 10 pages in one I/O and pull all its content, put in shared buffers only to find my result in the first page (essentially wasting disk bandwidth, memory etc) 


It is going to be interesting to balance this out. 


...more
View all episodesView all episodes
Download on the App Store

The Backend Engineering Show with Hussein NasserBy Hussein Nasser

  • 4.9
  • 4.9
  • 4.9
  • 4.9
  • 4.9

4.9

40 ratings


More shows like The Backend Engineering Show with Hussein Nasser

View all
Freakonomics Radio by Freakonomics Radio + Stitcher

Freakonomics Radio

32,245 Listeners

Software Engineering Radio - the podcast for professional software developers by team@se-radio.net (SE-Radio Team)

Software Engineering Radio - the podcast for professional software developers

273 Listeners

Risky Business by Patrick Gray

Risky Business

374 Listeners

Science Vs by Spotify Studios

Science Vs

12,172 Listeners

Syntax - Tasty Web Development Treats by Wes Bos & Scott Tolinski - Full Stack JavaScript Web Developers

Syntax - Tasty Web Development Treats

987 Listeners

Darknet Diaries by Jack Rhysider

Darknet Diaries

8,107 Listeners

Practical AI by Practical AI LLC

Practical AI

210 Listeners

Within Reason by Alex J O'Connor

Within Reason

1,658 Listeners

All-In with Chamath, Jason, Sacks & Friedberg by All-In Podcast, LLC

All-In with Chamath, Jason, Sacks & Friedberg

10,228 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

541 Listeners

Big Technology Podcast by Alex Kantrowitz

Big Technology Podcast

513 Listeners

Hard Fork by The New York Times

Hard Fork

5,548 Listeners

The AI Daily Brief: Artificial Intelligence News and Analysis by Nathaniel Whittemore

The AI Daily Brief: Artificial Intelligence News and Analysis

655 Listeners

Prof G Markets by Vox Media Podcast Network

Prof G Markets

1,474 Listeners

The Pragmatic Engineer by Gergely Orosz

The Pragmatic Engineer

74 Listeners