RTFM: High-Performance Computing Edition

Hey! I'd love to hear your thoughts, send me a voice note.

#SLURM101: Why Jobs Wait (and Why They Sometimes Don't)

You submit a job. Then someone else submits a job after you, and it starts first. Is Slurm broken? Is somebody cheating the queue? No. This is Slurm working exactly as designed.

In this episode we follow one job, job 42, through its life in Slurm and look at the quiet decisions the scheduler makes along the way. No command syntax, no config files: just the concepts, so the commands and options stop looking like magic spells.

What you will learn:

- Backfill scheduling: why a small job can start before a big one

- Partitions: separate pools of nodes with their own rules

- Features (constraints): labels on nodes, like the GPU model or local storage

- Topology-aware scheduling: keeping a job's nodes close together on the network

- TRES (trackable resources): GPUs, memory, licenses and other countable things

- QoS (quality of service): fair sharing when one group floods the queue

- Plugins: adding your own logic, for example Pyxis for containers

More info about HPC in the book https://a.co/d/b03YNBn

Hey! I'd love to hear your thoughts, send me a voice note.

What storage is used in #HPC? What are network filesystems and parallel filesystems? How do they work with a #supercomputer?

In this #HPC101 episode we talk about HPC storage, NFS and distributed filesystems, and what you should take into account once your #supercomputing #cluster grows large enough. We also discuss: - how the data is synced in large clusters - how Lustre distributes data across servers - redundancy and failover.

If you are a Linux sysadmin curious about HPC, cluster architecture, AI infrastructure, or distributed systems, this episode is for you.

More info about HPC in the book https://a.co/d/b03YNBn

Hey! I'd love to hear your thoughts, send me a voice note.

Someone hits the "Train" button… and somewhere behind the scenes, hundreds or thousands of GPUs are supposed to start processing a new #AI model. But what actually happens between that click and a new model showing up a week later?

In this first episode of RTFM: High-Performance Computing Edition, we walk through what's really going on under the hood of an AI training job on a real #HPC cluster: resource management, regular storage and parallel file systems, why #RDMA exists, and why #GPU utilization can quietly drop to 10-20% if your network and storage aren't pulling their weight.

This is a #HPC101 level overview, future episodes will dig into each piece (job scheduling internals, parallel file systems, #RDMA, #InfiniBand, and more) in much more depth.

More info about HPC in the book https://a.co/d/b03YNBn

#HPC #Supercomputing #SLURM #AI #DeepLearning #Linux #SysAdmin #DataCenter #HPC101 #supercomputer #supercomputers #lustre