Clay is a UI layout library written in C, and this video gives a high level explanation of how the flexbox-like layout algorithm in Clay works.

The library can be found at https://github.com/nicbarker/clay

If you're not familiar with Clay, a detailed introductory video can be found here:

If you would like to learn a bit more about how trees are implemented, see this video:

The paper on constraint solvers: https://arxiv.org/pdf/1401.1031

Facebook's "Yoga" layout library which was mentioned: https://github.com/facebook/yoga

If you're interested, I also produce a paid programming fundamentals course at https://thesimplejoyofprogramming.com