As a new parent, you get exposed to a whole new world. There is this new toddler song “Five little monkeys jumping on the bed” where it starts with five monkeys jumping on the bed and one falls off the bed and the mom calls the doctor and then another monkey falls and the mom calls the doctor again. This keeps going until all monkeys are no longer on the bed.

This clearly looks like a software engineering problem. In the face of it is extreme inefficiency in IO with the mom calling the doctor 5 times in span of 3 minutes. The mom could decide to wait for all monkeys to fall and call the doctor once. A Method that is known as batching.

Two problems with this approach; the first is latency the mom can’t wait, she is worried, she needs immediate answer from the doctor for the first monkey that fell.

The second problem is we don’t know if more monkeys are going to fall off the bed, so while the mom waiting, the first monkey will be in pain and will “starve”.

We makes decisions like this constantly in software engineering. In some database systems for example commits from transactions are grouped and batched for few microseconds/milliseconds so more commits can arrive and the database can flush/fsync the WAL once. This is specially important in highly concurrent system because disk fsync is costly and bypasses the operating system cache. However, this comes at a cost of slight delay in transaction commits but less IO in general.

To batch or not to batch. The five little monkey problem doesn’t have one right solution it all depends.