Code Conversations

This video is a free preview from the 2h course "Platform as a Product" available on the Team Topologies Academy: https://bit.ly/3o5aLmt

Manuel Pais discusses how many organizations see Kubernetes as “the” platform, rather than just a technical foundation for a true internal platform. Successful Kubernetes adoption requires thinking about what a platform really means and learning which team structures and interactions work well. And evolve them over time.

https://www.youtube.com/watch?v=Iu_T3X-bPqE

Recent research summarised in the book Accelerate points to a set of practices that lead to high software development organisation performance. Simultaneously, research from the Santa Fe institute on Complex Adaptive Systems over the last 20 years seems to point to a grand unified theory of organisational design. So have we cracked it? Do we now have the answer to the question: how do we create and scale high performing software and organisations?

https://www.youtube.com/watch?v=uAwJEFLJunk

Matthew Skelton and Manuel Pais - co-authors of the book Team Topologies - discuss the concept of a Thinnest Viable Platform (TVP) and what this means for organizations building and running modern software systems.

Savvy organisations are discovering the value of treating their internal platforms as products. But what does it mean to treat a “platform as a product”? What benefits does this give, and why would an organisation adopt this approach?

https://www.youtube.com/watch?v=b8YHCDMxqfg

The "Engineering Room" is a monthly series of conversations with people who are influential in the software industry. In this episode Dave Farley, author of "Continuous Delivery", "Modern Software Engineering" and others, talks to Matthew Skelton co-author of one of the most significant software books of the last 10 years - “Team Topologies”, about the ingredients for long-term, viable, sustainable and understandable software development.

https://www.youtube.com/watch?v=vIcp9zDs4EI

What is a platform team, and how do you build effective platforms? Platforms are often an important part of the strategy to scale software development beyond small single teams. Dividing work up so that common behaviours and services can be shared, rather than every service team implementing their own version, is a big gain, but it can also be a big cost. Coupling between teams is one of the biggest causes of problems when trying to build software on larger scales.

https://www.youtube.com/watch?v=_zH7TIXcjEs

Matthew and Manuel's excellent book Team Topologies described the challenges and proposed patterns for organizing teams effectively.

https://www.youtube.com/watch?v=ckGjOkMTGKo

AI agents are a type of compound AI system that uses a large language model (LLM) to control its logic. Compound AI systems combine models and other components to solve problems, and they are more adaptable and effective than models alone.

Here are some key points about AI agents from the sources:

The sources suggest that AI agents are a rapidly developing technology that combines system design and agentic behavior.

https://youtu.be/F8NKVhkZZWI?si=q0dVZ1juEjPLi5s3

https://youtu.be/85RfazjDPwA?si=TM2RugT9QEd1UOZj

Comprehensive Overview of PyTorch Tools for Scaling AI Models

Scaling AI models often involves adding more layers to neural networks to enhance their ability to capture data nuances and execute complex tasks. However, this scaling process demands increased memory and computational power. To address these challenges, PyTorch offers tools like Distributed Data Parallel (DDP) that distribute the training workload across multiple GPUs, enabling faster model training.

Distributed Data Parallel (DDP) comprises three key steps:

A crucial advantage of DDP lies in its ability to overlap computation and communication, enabling back propagation to occur concurrently with gradient communication, maximizing GPU engagement. This efficient process involves dividing the model into segments referred to as "buckets". As the gradients for each bucket are calculated, the gradients of the preceding buckets are simultaneously synchronized.

While DDP proves effective for models that fit on a single GPU, larger models, like the 30 billion or 70 billion parameter Llama models, necessitate a different approach. Fully Sharded Data Parallel (FSDP) tackles this challenge by fragmenting the model into smaller units, called "shards," and distributing these shards across multiple GPUs.

FSDP employs a mechanism similar to DDP, but its operations are performed at the unit level rather than the entire model level. During the forward pass, units are gathered, computations are performed, and memory is released before proceeding to the next unit, ensuring optimal resource utilization. In the backward pass, units are gathered again, back propagation is computed, and gradients are synchronized across the GPUs responsible for specific portions of the model. Like DDP, FSDP leverages the overlap of computation and communication to maintain continuous GPU activity, thereby maximizing efficiency.

Training these large-scale models typically necessitates high-performance computing (HPC) systems equipped with high-speed interconnects like InfiniBand. However, training can also be effectively conducted on more prevalent Ethernet networks using a technique called "rate limiting," developed through a collaborative effort between IBM and the PyTorch community. Rate limiters optimize GPU memory management, striking a balance between communication and computation overlap. This optimization reduces communication demands per computation step, enabling increased computation with consistent communication.

PyTorch's widespread adoption is largely attributed to its "eager mode," which provides a flexible and dynamic programming environment closely aligned with Python's structure. However, this flexibility can lead to GPU idle time, especially when handling larger models. This inefficiency arises because instructions are queued separately on the CPU and GPU, causing delays as the GPU waits for instructions from the CPU.