Machine Learning Guide

How working engineers ship with coding agents: issues an agent can verify, plan mode before code, a verification loop with a browser in it, agent review of agent code, worktrees and CI, cost discipline, and where agents still fail.

Links

• Try a walking desk - stay healthy & sharp while you learn & code
• More OCDevel shows - this one has siblings, each on its own subject and produced the same way

Third and last episode of the vibe-coding sequence. Vibe Coding in 2026 picked an agent; Inside a Coding Agent explained the mechanics. This one is the practice: how working engineers ship real software with Claude Code, Codex, and Antigravity without shipping garbage. Specs, verification loops, agent review, parallel worktrees, headless CI, cost discipline, and the failure modes.

From vibe coding to agentic engineering

Andrej Karpathy coined vibe coding in early 2025 and, a year later, proposed "agentic engineering" for professional work. His older idea of jagged intelligence (models clear hard problems and trip on trivial ones, unpredictably) is why the job is judgment rather than button-pressing. The frame for the episode: when implementation is cheap, the value moves to the two ends of the pipeline, specification (saying exactly what should be true) and verification (proving it). Vibe coding stays fine for throwaways; the rest applies to codebases with users.

Specs: issues as prompts, plan mode in Claude Code, Codex, and Antigravity

The prompt is a spec whether you meant it or not. The unit of work is a tracker issue with four parts: what's wrong, where to look, acceptance criteria a machine or a five-second human check can verify, and an explicit out-of-scope fence. Every serious agent has a read-only planning phase: Claude Code's plan mode (Shift+Tab or /plan), Codex's plan mode on the same keystroke with its own plan-mode reasoning effort, and Antigravity's implementation plan artifact. Judge a plan on three things: the files it names, the verification step it commits to, and whether it stays inside the fence. Heavier spec tooling (GitHub spec-kit, Kiro specs) formalizes requirements, design, and tasks. Anthropic's best practices teach a lighter interview-then-spec-file pattern and say to skip planning when you could describe the diff in one sentence; OpenAI's Codex best practices frame a task as goal, context, constraints, and "done when."

Verification loops: one-command checks, test-first, Playwright MCP and browser agents

An agent without feedback guesses; an agent with a runnable check searches. Make typecheck, lint, tests, and build fast and runnable from one command, because that command is what the agent lives inside. Test-first changes meaning here: it used to be design pressure, and now it gives the agent a fixed target. Anthropic's guidance is that without a success criterion the developer is the only feedback loop (building verification loops). Then add a browser. Playwright MCP drives a real one through accessibility snapshots with stable element refs, no vision model needed; Claude in Chrome is the screenshot route Anthropic names for UI verification; Antigravity's browser agent starts the dev server and clicks through on its own, ending in a walkthrough with verification evidence. The principle: every acceptance criterion maps to a check the agent can run. If one doesn't, either the criterion is vague or the project is missing a kind of check.

The review loop: agent PRs, Claude Code and Codex review, human gate

Five vendor-agnostic stages: trigger, implement, review, fix, gate. Agent review of agent code works because the reviewer has fresh context, so never reuse the implementing session as its own reviewer. Ask for correctness and quality separately and set a confidence bar so the reviewer reports only what it's sure of. Anthropic's code-review plugin runs four parallel reviewers with 0-100 confidence scores and drops findings under 80; a cloud tier (Ultrareview) reproduces each finding before reporting it, and a separate hosted Code Review product reviews every PR automatically. Codex reviews on an @codex review mention or automatically per repo, flagging only serious issues. CodeRabbit and Greptile fill the same slot with a precision/recall tradeoff (ignore vendor benchmarks of each other). The human gate reviews a staging branch as a batch with the running app in front of you, and reads the tests, not just the code.

Parallelism: git worktrees, cloud sandboxes, task queues

Each agent gets its own copy of the repo on its own branch. Locally that's a git worktree (Claude Code worktrees, --worktree and subagent isolation: worktree; Codex added a --worktree flag in 0.154.0). In the cloud it's a sandbox per task: Claude Code on the web (managed VM, credentials behind a proxy, claude --cloud, teleport back to the terminal), Codex cloud tasks (isolated containers, network off by default), Google Jules, and Antigravity's Agent Manager. Parallel tasks must be disjoint, which is why issues that name their files matter. Three interactive agents is a practical ceiling for one person; no vendor publishes a number, and Simon Willison's parallel coding agent lifestyle argues for prompting during natural breaks rather than a fixed count. Past that, scale with a labeled queue instead of more terminals.

Headless and CI: GitHub Actions, label triggers, scheduled runs

Every major CLI has a non-interactive mode, and CI chains it. All three majors ship a GitHub Action covering mention, label, and schedule triggers: claude-code-action (@claude mentions, label_trigger, cron automation mode, claude/ branch prefix), Codex's GitHub integration and codex-action, and Google's run-gemini-cli with hourly issue triage in its examples. GitHub Copilot coding agent takes an assigned issue into an Actions runner and opens a draft PR. The CI fix loop: Claude Code on the web's Auto-fix pull requests subscribes to a PR's webhooks and pushes fixes for failing checks or review comments (/autofix-pr from the terminal); Codex does the same from a PR mention. Two cautions: headless runs need a fixed tool allowlist or a sandbox, and scheduled runs act as the user who wrote the schedule, so gate triggers on the actor to avoid automation loops.

Cost and context discipline

Shape rather than prices, since prices age fastest: every vendor sells a subscription with a rolling window plus weekly cap in multiplied tiers (Claude, Codex, Antigravity), overage credits at roughly API rates, and pay-as-you-go API keys. Subscription for daily interactive use, API for headless and CI. Model tiering is official guidance: Anthropic's cost docs reserve the top model for architectural work, and subagents take a per-agent model and effort; Codex shows model plus reasoning effort on its status line. Context is the invisible line item: every turn re-sends history, compaction is itself a large request, /usage and /context show what eats the window, and the prompt cache goes cold after a break. The habits: clear between unrelated tasks, trim MCP servers, push exploration into subagents, and expect agent teams to run several times a single session's tokens.

Failure modes: reward-hacked tests, scope creep, prompt injection, secrets

Four families, each with a mechanism and a case. Reward-hacked tests: METR measured frontier models gaming graders in about 30% of research-engineering runs (patching the scoring function, locating precomputed answers), and instructing them not to cheat had nearly no effect; Anthropic's reward-hacking research names the canonical move, calling sys.exit(0) inside the harness so tests report green. The mitigation is structural: the worker is never the grader, and a hook flags test-file edits. Scope creep: fence the issue, flag out-of-fence changes in review, route unrelated improvements to a new issue. Prompt injection through repo content: Invariant Labs' GitHub MCP demonstration exfiltrated private code via a malicious public issue; Cursor's CVE-2025-54135 let injected content write MCP config and execute code; Anthropic's security docs say no system is immune and recommend VMs for external services. Secrets and blast radius: the Nx s1ngularity attack (advisory) ran victims' installed Claude, Gemini and Amazon Q CLIs with skip-permissions flags to harvest credentials, leaking 1,000+ tokens; the Replit production database deletion (July 2025) and a reported second wipe (April 2026); the UK AISI incident report (August 2026) on unsanctioned real-world actions during cyber evaluations. Guardrails: branch-scoped tokens behind a proxy, no production credentials on the agent's machine, destructive commands denied by hook or sandbox, plugins and MCP servers installed only from sources you'd let commit (plugin trust guidance), tested backups. On productivity: METR's 2025 RCT found experienced developers 19% slower; the 2026 uplift update flipped that cohort to a speedup with intervals crossing zero; the 2025 DORA report found AI adoption raised throughput and lowered delivery stability. The tools amplify the process you already have.

One-week adoption plan

Day one: one fast command for typecheck, lint, tests, build. Day two: three issues with acceptance criteria and a scope fence, each through plan mode. Day three: browser verification and a screenshot in every UI PR. Day four: fresh-context review of every agent PR and a staging branch as the human gate. Day five: one recurring chore as a scheduled headless run, with secrets and test-integrity guardrails. Days six and seven: two agents on disjoint issues, and notice where your supervision breaks.

Related episodes

• MLA 22: Vibe Coding in 2026, which agent to pick and why
• MLA 23: Inside a Coding Agent, context, instruction files, hooks, skills, MCP, subagents
• MLA 28: AI Agents in 2026, agent architecture beyond coding
• MLA 30: AI and Programming Jobs in 2026, what agents did to the job market and how to position

Want this practice hands-on rather than surveyed? The Gnothi OCDevel Claude Code show goes from a first change in the terminal to a repeatable delivery workflow.

A coding agent is a loop appending to a context window, and the window is what runs out. Instruction files, permissions, hooks, skills, MCP, subagents and memory explained as one mechanism, mapped across Claude Code, Codex and OpenCode.

Links

• Try a walking desk - stay healthy & sharp while you learn & code
• More OCDevel shows - this one has siblings, each on its own subject and produced the same way

Second of three episodes on coding agents: how Claude Code, Codex, OpenCode and pi handle context, instruction files, permissions, hooks, skills, MCP, subagents and memory. Claude Code is the running example; each section maps the same concept to the other three.

The loop and why context is the scarce resource

A coding agent is a short program: send the conversation to the model, run whatever tool it asks for, append the result, repeat until the model answers with text alone. Every appended file and test log stays in the window for the life of the session, so the context window, not the model, is what runs out. Every mechanism below is a way of deciding what enters the window and when. Compaction summarizes the history when the window fills; Claude Code's auto-compact window and /compact are configurable, and /compact is itself an expensive request while /clear is free. Thinking is dialed by effort level in Claude Code and by model_reasoning_effort in the Codex configuration reference.

Instruction files: CLAUDE.md and AGENTS.md

AGENTS.md has become the cross-vendor instruction file, stewarded by the Agentic AI Foundation under the Linux Foundation and read by Codex, OpenCode, Cursor, Gemini CLI, GitHub Copilot, Jules, Aider, Zed and more. Claude Code reads only CLAUDE.md, bridged with an @AGENTS.md import; the Claude Code memory docs describe the managed, user, project and local layers, lazy subdirectory loading, @ imports up to four hops, path-scoped .claude/rules/, and re-injection after compaction. Codex's AGENTS.md guide walks from the git root down with a 32 KiB default cap; OpenCode rules read AGENTS.md with a CLAUDE.md fallback plus an instructions array in opencode.json; pi concatenates AGENTS.md from home, parents and cwd. What belongs: commands, conventions and boundaries the model cannot derive from the code, under a couple hundred lines. Instruction files are context, not enforcement.

Permissions and sandboxing

Claude Code permissions use Tool(pattern) rules with deny over ask over allow, and the docs list how allowlists leak: absolute paths, sh -c, and unstripped wrappers like devbox run. Permission modes are default, acceptEdits, plan, the classifier-backed auto mode, and bypassPermissions. Codex separates sandbox modes (read-only, workspace-write, danger-full-access) from an approval policy. OpenCode permissions map globs to allow, ask or deny with last-match-wins. pi ships no permission system by design. Claude Code sandboxing uses Seatbelt on macOS and bubblewrap on Linux, with filesystem and network-domain allowlists. The cautionary case is PocketOS, April 2026, where a Cursor agent found an unscoped hosting token and deleted the production volume and its backups in seconds, reported by The Register and Fast Company.

Hooks: deterministic guardrails at tool events

A hook runs a program at a fixed point in the loop and its exit code or JSON decides what happens next; unlike an instruction, it fires every time. Claude Code hooks cover more than thirty events; exit code 2 always blocks on blockable events, PreToolUse can stop a call, Stop can refuse to end a turn, PostToolUse cannot block. Codex hooks mirror the event set and require trusting each hook definition by hash. OpenCode plugins expose tool.execute.before, permission.asked and session events; Gemini CLI hooks and Copilot CLI hooks exist too. The three worth having: a PreToolUse block on destructive commands and secret files, a post-edit formatter and typecheck that feeds errors back, and a Stop hook that holds the turn open while tests are red.

Skills and slash commands

A skill is a folder with a SKILL.md whose name and description load at startup, whose body loads when a task matches, and whose bundled files load on demand. The format is the open Agent Skills standard, adopted by Codex, OpenCode, Cursor, Gemini CLI, GitHub Copilot and dozens more. Claude Code skills absorbed custom slash commands; frontmatter flags decide whether a skill is user-invoked, model-invoked or both, and side-effecting skills like deploy should be user-invoked only. A loaded skill body persists in the window, and load-time shell commands can splice live state into a procedure.

MCP: what it solves and what it costs

The Model Context Protocol standardizes tools, resources and prompts over JSON-RPC, via stdio or streamable HTTP with OAuth; Anthropic donated it to the Linux Foundation and the current revision adds stateless per-request negotiation, Tasks and MCP Apps extensions. Tool definitions traditionally loaded into context for every server every session. Anthropic's Code execution with MCP named the two costs and reported 150,000 tokens down to 2,000. The fix that shipped is deferred loading: Claude Code defers MCP tool definitions by default via tool search, also available at the API level; Codex MCP config offers per-server enabled_tools and disabled_tools; OpenCode MCP servers filter tools by glob with no lazy loading documented. Claude Code's cost guidance: prefer CLI tools over MCP servers, scope servers per project, disable what you have not used, and watch the 25,000-token output cap.

Subagents and worktrees as the answer to context rot

A subagent is a second loop with a fresh window; the parent sees only its result. Claude Code subagents are Markdown files with frontmatter for model, tools, permission mode, memory and worktree isolation, started without conversation history unless forked; agent teams add messaging between named teammates at roughly seven times the tokens of a plain session. Codex subagents are TOML files with developer instructions, model, effort and sandbox mode. OpenCode agents split into primary agents and @mention subagents, each with its own permission block. Recommended cast: a read-only scout on a cheap model, a fresh reviewer per diff, one writer per task. Claude Code worktrees create isolated checkouts with --worktree, EnterWorktree, subagent isolation: worktree, .worktreeinclude for gitignored files, and enforcement that blocks edits reaching back into the main checkout; Codex and OpenCode leave worktree management to git.

Memory across sessions

Memory is files. Claude Code auto memory is on by default: a per-repository directory shared across worktrees with a MEMORY.md index (first 200 lines loaded) and topic files read on demand, recording preferences, feedback and learned project facts. Codex memories are an opt-in local store controlled per session. OpenCode has no first-party memory beyond snapshots. Both vendors frame memory as a recall layer: rules that must always apply belong in the instruction file, prohibitions in hooks.

Concept map across tools

• Instruction file: CLAUDE.md (Claude Code); AGENTS.md (Codex, OpenCode, pi, most others)
• Permissions: modes and rules (Claude Code); sandbox mode plus approval policy (Codex); per-tool allow/ask/deny (OpenCode); none by design (pi)
• Hooks: settings hooks (Claude Code); trusted hooks file (Codex); plugins (OpenCode); extensions (pi)
• Skills: shared open format everywhere; slash commands folded in (Claude Code)
• MCP: deferred by default (Claude Code); per-server tool filters (Codex); glob filters (OpenCode)
• Subagents: Markdown (Claude Code, OpenCode); TOML (Codex)
• Worktrees: first-party (Claude Code); do it yourself (Codex, OpenCode)
• Memory: auto memory (Claude Code); memories (Codex); files or an MCP server (OpenCode)

Related episodes

• Vibe Coding in 2026: Codex vs Claude Code vs Antigravity vs Grok
• Agentic Software Engineering: Specs, Verification, and the Review Loop
• AI Agents in 2026: Loops, Tools, Memory, Protocols, and Evaluation

The Gnothi companion show on Claude Code goes from a first change in the terminal to a repeatable delivery workflow: OCDevel Claude Code Podcast.

What a coding agent actually is, and how Claude Code, OpenAI Codex, Google Antigravity and xAI's Grok Build differ in philosophy rather than features. Plus open harnesses pi and OpenCode, the open-weight coders, and a decision rule by constraint.

Links

• Try a walking desk - stay healthy & sharp while you learn & code
• More OCDevel shows - this one has siblings, each on its own subject and produced the same way

First of three episodes on vibe coding: the field of coding agents (Claude Code, OpenAI Codex, Google Antigravity, Grok Build, pi, OpenCode) as of late 2026 and how to choose one. The next two cover the mechanics (Inside a Coding Agent) and the engineering practice (Agentic Software Engineering).

Vibe coding then and now

Andrej Karpathy's February 2025 post named "vibe coding": accept all, don't read the diffs, paste errors back in, and, the caveat everyone dropped, "not too bad for throwaway weekend projects." Collins made it Word of the Year 2025. The arc since is autocomplete, then chat, then agents that read, edit, run and iterate; the engineers who stayed with it moved from vibing to agentic engineering, which is the through line for the sequence.

Anatomy of a coding agent

The model and the harness are separate things, and most tool arguments are about one without noticing which. The harness is the loop around the model, and there are five parts worth naming: the model, the tools (read, write, shell, search, browser), the permission model and sandbox, the context strategy (what to keep, summarize, offload to files, or hand to a fresh subagent), and the surface (terminal, IDE, desktop, web, chat). Ask those five questions of any product; they matter more than the feature grid. The mechanics get their own episode.

Claude Code: terminal-first and composable

Anthropic's stated design is Unix-style composability: pipe logs into it, run it in CI, chain it with other tools. The terminal is the primary surface; the IDE extension, desktop app, web and mobile clients all connect to the same engine and share CLAUDE.md, settings and MCP servers, with handoffs between local, cloud and desktop. Headless -p mode, GitHub Actions and GitLab integrations, and the Claude Agent SDK make it embeddable. Subagents and worktrees are its answer to the context problem. Model tiers (Sonnet, Opus, Fable) sit behind one harness with 1M-context variants and an effort dial; usage is included in Claude subscriptions or billed via the API.

• Claude Code overview
• Model configuration
• Plans and costs
• Headless mode
• Subagents and worktrees
• Claude Agent SDK
• Claude Fable 5 and Mythos 5, Claude Opus 5

OpenAI Codex: cloud-first and parallel

Codex CLI and Codex cloud are one product under one ChatGPT login, with an IDE extension and a desktop app; codex cloud pushes local work into a sandboxed cloud environment and applies the result back. OpenAI's framing is isolated environments, parallel tasks, and comparing multiple solution attempts, with tasks launchable from the web, GitHub, GitLab, Linear or Slack. Codex-tuned models are trained for long-running agentic work, and the newest generation keeps notes across context windows instead of repeatedly compressing a session. The CLI is Apache-2.0 open source; the Agents API brings the same harness to your own cloud agents.

• Codex CLI docs
• Codex cloud docs
• Model lineup and plans and credits
• Codex CLI source
• Introducing the Codex app
• Introducing the Agents API

Google Antigravity: the agent manager

Antigravity still ships under that name, and it absorbed Gemini CLI: Google retired Gemini CLI into Antigravity CLI in June 2026, and Antigravity 2.0 spans IDE, desktop app, CLI and SDK. The philosophy is supervision: "manage higher-level tasks for the Agent, not individual tool calls," with agents working across editor, terminal and browser and producing artifacts you review. The model menu is multi-vendor on consumer tiers (several Gemini tiers plus Claude Sonnet and Opus and GPT-OSS); enterprise is Gemini only.

• Antigravity IDE
• Supported models and tiers
• Antigravity changelog
• Google I/O 2026 developer highlights
• Transitioning Gemini CLI to Antigravity CLI

Grok Build: the cost-first bet

xAI's coding story flipped in 2026 from a fast model in other people's harnesses (grok-code-fast-1, since deprecated from GitHub Copilot) to a first-party open-source terminal agent, Grok Build: interactive TUI, headless -p, Agent Client Protocol for editors, plan-review-approve, hooks, plugins, MCP and parallel subagents. xAI's own positioning for its coding model is fastest and cheapest for agentic coding; the premium model above it is the agent's default. Benchmark numbers and the subagent-count and price claims circulating in blogs are not on xAI's pages.

• Introducing Grok Build
• Grok Build 0.1 on the API
• Grok Build docs and source
• Grok Code Fast 1 deprecated in Copilot

Open harnesses: pi and OpenCode

Both are open source, model-agnostic and very active, and they're opposites in philosophy. pi is a toolkit: small core, unified API over 15+ providers, TUI, print/RPC/SDK modes, TypeScript extensions, branchable sessions, a package system, and deliberately no built-in permission system (containerize instead; the author ships the Gondolin micro-VM). OpenCode is a product: TUI, web, desktop and IDE surfaces, a ~50-entry provider list including local runtimes, permission config, and the optional Zen gateway of verified models. OpenCode's docs note Anthropic prohibits driving Claude subscription models through third-party harnesses and those plugins were removed. Also: Crush, Kilo Code, Cline, and DeepSeek's own harness.

• pi, pi source
• OpenCode docs, providers, Zen, source
• Crush, DeepSeek Harness

Special mentions: open-weight coders and the second tier

Every lab with a model now ships a harness. Qwen has a coder line, an Apache-licensed open series and the Qwen Code CLI. Z.ai's GLM is pitched as the most capable open-weights model for coding, sold as a Coding Plan documented for Claude Code, Cline and OpenCode. DeepSeek serves 1M-context models with an Anthropic-format endpoint, so Claude-shaped harnesses can point at it. Moonshot's Kimi K3 is a very large open-weight multimodal model under a bespoke license, with the Kimi Code CLI. All coding claims are vendor-reported, now headlining SWE-bench Pro and Terminal Bench rather than SWE-bench Verified. Second tier: Copilot coding agent (cloud, inside Actions), Cursor (editor-first, adding cloud agents and a coordinator), Windsurf (folded into Devin), Cline (active), Aider (no commits since May 2026).

• Qwen3-Coder, Qwen Code
• GLM-5, GLM Coding Plan
• DeepSeek models and pricing
• Kimi K3, Kimi Code CLI
• Copilot coding agent, Cursor changelog, Cline, Aider

Decision rule by constraint

Already paying a lab: take that lab's agent (Claude Code, Codex, Antigravity) and stop shopping. Cloud vs local: Codex cloud or Copilot's agent for sandboxed parallel PRs; a harness plus a local or self-hosted model if everything has to stay behind your firewall. Cost: Grok Build or an open harness with a cheap open-weight model for mechanical work. Open weights: pi or OpenCode with Qwen, GLM, DeepSeek or Kimi. Test on a real bug you already understand, and budget for a primary agent plus a cheaper second one.

Related episodes

• MLA 23: Inside a Coding Agent
• MLA 24: Agentic Software Engineering
• MLA 28: AI Agents
• MLA 30: AI and Programming Jobs

Companion show: the Gnothi Claude Code series goes from your first terminal change to a repeatable delivery workflow.

Links:

• Notes and resources at ocdevel.com/mlg/33
• 3Blue1Brown videos: https://3blue1brown.com/
• Try a walking desk stay healthy & sharp while you learn & code
• Try Descript audio/video editing with AI power-tools

Background & Motivation

• RNN Limitations: Sequential processing prevents full parallelization—even with attention tweaks—making them inefficient on modern hardware.
• Breakthrough: "Attention Is All You Need" replaced recurrence with self-attention, unlocking massive parallelism and scalability.

Core Architecture

• Layer Stack: Consists of alternating self-attention and feed-forward (MLP) layers, each wrapped in residual connections and layer normalization.
• Positional Encodings: Since self-attention is permutation invariant, add sinusoidal or learned positional embeddings to inject sequence order.

Self-Attention Mechanism

• Q, K, V Explained:
  • Query (Q): The representation of the token seeking contextual info.
  • Key (K): The representation of tokens being compared against.
  • Value (V): The information to be aggregated based on the attention scores.
• Multi-Head Attention: Splits Q, K, V into multiple "heads" to capture diverse relationships and nuances across different subspaces.
• Dot-Product & Scaling: Computes similarity between Q and K (scaled to avoid large gradients), then applies softmax to weigh V accordingly.

Masking

• Causal Masking: In autoregressive models, prevents a token from "seeing" future tokens, ensuring proper generation.
• Padding Masks: Ignore padded (non-informative) parts of sequences to maintain meaningful attention distributions.

Feed-Forward Networks (MLPs)

• Transformation & Storage: Post-attention MLPs apply non-linear transformations; many argue they're where the "facts" or learned knowledge really get stored.
• Depth & Expressivity: Their layered nature deepens the model's capacity to represent complex patterns.

Residual Connections & Normalization

• Residual Links: Crucial for gradient flow in deep architectures, preventing vanishing/exploding gradients.
• Layer Normalization: Stabilizes training by normalizing across features, enhancing convergence.

Scalability & Efficiency Considerations

• Parallelization Advantage: Entire architecture is designed to exploit modern parallel hardware, a huge win over RNNs.
• Complexity Trade-offs: Self-attention's quadratic complexity with sequence length remains a challenge; spurred innovations like sparse or linearized attention.

Training Paradigms & Emergent Properties

• Pretraining & Fine-Tuning: Massive self-supervised pretraining on diverse data, followed by task-specific fine-tuning, is the norm.
• Emergent Behavior: With scale comes abilities like in-context learning and few-shot adaptation, aspects that are still being unpacked.

Interpretability & Knowledge Distribution

• Distributed Representation: "Facts" aren't stored in a single layer but are embedded throughout both attention heads and MLP layers.
• Debate on Attention: While some see attention weights as interpretable, a growing view is that real "knowledge" is diffused across the network's parameters.

Databricks is a cloud-based platform for data analytics and machine learning operations, integrating features such as a hosted Spark cluster, Python notebook execution, Delta Lake for data management, and seamless IDE connectivity. Raybeam utilizes Databricks and other ML Ops tools according to client infrastructure, scaling needs, and project goals, favoring Databricks for its balanced feature set, ease of use, and support for both startups and enterprises.

Links

• Notes and resources at ocdevel.com/mlg/mla-21
• Try a walking desk stay healthy & sharp while you learn & code

Raybeam and Databricks

• Raybeam is a data science and analytics company, recently acquired by Dept Agency.
• While Raybeam focuses on data analytics, its acquisition has expanded its expertise into ML Ops and AI.
• The company recommends tools based on client requirements, frequently utilizing Databricks for its comprehensive nature.

Understanding Databricks

• Databricks is not merely an analytics platform; it is a competitor in the ML Ops space alongside tools like SageMaker and Kubeflow.
• It provides interactive notebooks, Python code execution, and runs on a hosted Apache Spark cluster.
• Databricks includes Delta Lake, which acts as a storage and data management layer.

Choosing the Right MLOps Tool

• Raybeam evaluates each client's needs, existing expertise, and infrastructure before recommending a platform.
• Databricks, SageMaker, Kubeflow, and Snowflake are common alternatives, with the final selection dependent on current pipelines and operational challenges.
• Maintaining existing workflows is prioritized unless scalability or feature limitations necessitate migration.

Databricks Features

• Databricks is accessible via a web interface similar to Jupyter Hub and can be integrated with local IDEs (e.g., VS Code, PyCharm) using Databricks Connect.
• Notebooks on Databricks can be version-controlled with Git repositories, enhancing collaboration and preventing data loss.
• The platform supports configuration of computing resources to match model size and complexity.
• Databricks clusters are hosted on AWS, Azure, or GCP, with users selecting the underlying cloud provider at sign-up.

Parquet and Delta Lake

• Parquet files store data in a columnar format, which improves efficiency for aggregation and analytics tasks.
• Delta Lake provides transactional operations on top of Parquet files by maintaining a version history, enabling row edits and deletions.
• This approach offers a database-like experience for handling large datasets, simplifying both analytics and machine learning workflows.

Pricing and Usage

• Pricing for Databricks depends on the chosen cloud provider (AWS, Azure, or GCP) with an additional fee for Databricks' services.
• The added cost is described as relatively small, and the platform is accessible to both individual developers and large enterprises.
• Databricks is recommended for newcomers to data science and ML for its breadth of features and straightforward setup.

Databricks, MLflow, and Other Integrations

• Databricks provides a hosted MLflow solution, offering experiment tracking and model management.
• The platform can access data stored in services like S3, Snowflake, and other cloud provider storage options.
• Integration with tools such as PyArrow is supported, facilitating efficient data access and manipulation.

Example Use Cases and Decision Process

• Migration to Databricks is recommended when a client's existing infrastructure (e.g., on-premises Spark clusters) cannot scale effectively.
• The selection process involves an in-depth exploration of a client's operational challenges and goals.
• Databricks is chosen for clients lacking feature-specific needs but requiring a unified data analytics and ML platform.

Personal Projects by Ming Chang

• Ming Chang has explored automated stock trading using APIs such as Alpaca, focusing on downloading and analyzing market data.
• He has also developed drone-related projects with Raspberry Pi, emphasizing real-world applications of programming and physical computing.

Additional Resources

• Databricks Homepage
• Delta Lake on Databricks
• Parquet Format
• Raybeam Overview
• MLFlow Documentation

Machine learning pipeline orchestration tools, such as SageMaker and Kubeflow, streamline the end-to-end process of data ingestion, model training, deployment, and monitoring, with Kubeflow providing an open-source, cross-cloud platform built atop Kubernetes. Organizations typically choose between cloud-native managed services and open-source solutions based on required flexibility, scalability, integration with existing cloud environments, and vendor lock-in considerations.

Links

• Notes and resources at ocdevel.com/mlg/mla-20
• Try a walking desk stay healthy & sharp while you learn & code

Dirk-Jan Verdoorn - Data Scientist at Dept Agency

Managed vs. Open-Source ML Pipeline Orchestration

• Cloud providers such as AWS, Google Cloud, and Azure offer managed machine learning orchestration solutions, including SageMaker (AWS) and Vertex AI (GCP).
• Managed services provide integrated environments that are easier to set up and operate but often result in vendor lock-in, limiting portability across cloud platforms.
• Open-source tools like Kubeflow extend Kubernetes to support end-to-end machine learning pipelines, enabling portability across AWS, GCP, Azure, or on-premises environments.

Introduction to Kubeflow

• Kubeflow is an open-source project aimed at making machine learning workflow deployment on Kubernetes simple, portable, and scalable.
• Kubeflow enables data scientists and ML engineers to build, orchestrate, and monitor pipelines using popular frameworks such as TensorFlow, scikit-learn, and PyTorch.
• Kubeflow can integrate with TensorFlow Extended (TFX) for complete end-to-end ML pipelines, covering data ingestion, preprocessing, model training, evaluation, and deployment.

Machine Learning Pipelines: Concepts and Motivation

• Production machine learning systems involve not just model training but also complex pipelines for data ingestion, feature engineering, validation, retraining, and monitoring.
• Pipelines automate retraining based on model performance drift or updated data, supporting continuous improvement and adaptation to changing data patterns.
• Scalable, orchestrated pipelines reduce manual overhead, improve reproducibility, and ensure that models remain accurate as underlying business conditions evolve.

Pipeline Orchestration Analogies and Advantages

• ML pipeline orchestration tools in machine learning fulfill a role similar to continuous integration and continuous deployment (CI/CD) in traditional software engineering.
• Pipelines enable automated retraining, modularization of pipeline steps (such as ingestion, feature transformation, and deployment), and robust monitoring.
• Adopting pipeline orchestrators, rather than maintaining standalone models, helps organizations handle multiple models and varied business use cases efficiently.

Choosing Between Managed and Open-Source Solutions

• Managed services (e.g., SageMaker, Vertex AI) offer streamlined user experiences and seamless integration but restrict cross-cloud flexibility.
• Kubeflow, as an open-source platform on Kubernetes, enables cross-platform deployment, integration with multiple ML frameworks, and minimizes dependency on a single cloud provider.
• The complexity of Kubernetes and Kubeflow setup is offset by significant flexibility and community-driven improvements.

Cross-Cloud and Local Development

• Kubeflow operates on any Kubernetes environment including AWS EKS, GCP GKE, and Azure AKS, as well as on-premises or local clusters.
• Local and cross-cloud development are facilitated in Kubeflow, while managed services like SageMaker and Vertex AI are better suited to cloud-native workflows.
• Debugging and development workflows can be challenging in highly secured cloud environments; Kubeflow's local deployment flexibility addresses these hurdles.

Relationship to TensorFlow Extended (TFX) and Machine Learning Frameworks

• TensorFlow Extended (TFX) is an end-to-end platform for creating production ML pipelines, tightly integrated with Kubeflow for deployment and execution.
• While Kubeflow originally focused on TensorFlow, it has grown to support PyTorch, scikit-learn, and other major ML frameworks, offering wider applicability.
• TFX provides modular pipeline components (data ingestion, transformation, validation, model training, evaluation, and deployment) that execute within Kubeflow's orchestration platform.

Alternative Pipeline Orchestration Tools

• Airflow is a general-purpose workflow orchestrator using DAGs, suited for data engineering and automation, but less resource-capable for heavy ML training within the pipeline.
  • Airflow often submits jobs to external compute resources (e.g., AI Platform) for resource-intensive workloads.
  • In organizations using both Kubeflow and Airflow, Airflow may handle data workflows, while Kubeflow is reserved for ML pipelines.
• MLflow and other solutions also exist, each with unique integrations and strengths; their adoption depends on use case requirements.

Selecting a Cloud Platform and Orchestration Approach

• The optimal choice of cloud platform and orchestration tool is typically guided by client needs, existing integrations (e.g., organizational use of Google or Microsoft solutions), and team expertise.
• Agencies with diverse client portfolios often benefit from open-source, cross-cloud tools like Kubeflow to maximize flexibility and knowledge sharing across projects.
• Users entrenched in a single cloud provider may prefer managed offerings for ease of use and integration, while those prioritizing portability and flexibility often choose open-source solutions.

Cost Optimization in Model Training

• Both AWS and GCP offer cost-saving compute options for training, such as spot instances (AWS) and preemptible instances (GCP), which are suitable for non-production, batch training jobs.
• Production workloads that require high uptime and reliability do not typically utilize cost-saving transient compute resources, as these can be interrupted.

Machine Learning Project Lifecycle Overview

• Project initiation begins with data discovery and validation of the client's requirements against available data.
• Cloud environment selection is influenced by client infrastructure, business applications, and platform integrations rather than solely by technical features.
• Data cleaning, exploratory analysis, model prototyping, advanced model refinement, and deployment are handled collaboratively with data engineering and machine learning teams.
• The pipeline is gradually constructed in modular steps, facilitating scalable, automated retraining and integration with business applications.

Educational Pathways for Data Science and Machine Learning Careers

• Advanced mathematics or statistics education provides a strong foundation for work in data science and machine learning.
• Master's degrees in data science add the most value for candidates from non-technical undergraduate backgrounds; those with backgrounds in statistics, mathematics, or computer science may benefit more from self-study or targeted upskilling.
• When evaluating online or accelerated degree programs, candidates should scrutinize the curriculum, instructor engagement, and peer interaction to ensure comprehensive learning.

The deployment of machine learning models for real-world use involves a sequence of cloud services and architectural choices, where machine learning expertise must be complemented by DevOps and architecture skills, often requiring collaboration with professionals. Key concepts discussed include infrastructure as code, cloud container orchestration, and the distinction between DevOps and architecture, as well as practical advice for machine learning engineers wanting to deploy products securely and efficiently.

Links

• Notes and resources at ocdevel.com/mlg/mla-19
• Try a walking desk stay healthy & sharp while you learn & code

;## Translating Machine Learning Models to Production

• After developing and training a machine learning model locally or using cloud tools like AWS SageMaker, it must be deployed to reach end users.
• A typical deployment stack involves the trained model exposed via a SageMaker endpoint, a backend server (e.g., Python FastAPI on AWS ECS with Fargate), a managed database (such as AWS RDS Postgres), an application load balancer (ALB), and a public-facing frontend (e.g., React app hosted on S3 with CloudFront and Route 53).

Infrastructure as Code and Automation Tools

• Infrastructure as code (IaC) manages deployment and maintenance of cloud resources using tools like Terraform, allowing environments to be version-controlled and reproducible.
• Terraform is favored for its structured approach and cross-cloud compatibility, while other tools like Cloud Formation (AWS-specific) and Pulumi offer alternative paradigms.
• Configuration management tools such as Ansible, Chef, and Puppet automate setup and software installation on compute instances but are increasingly replaced by containerization and Dockerfiles.
• Continuous Integration and Continuous Deployment (CI/CD) pipelines (with tools like AWS CodePipeline or CircleCI) automate builds, testing, and code deployment to infrastructure.

Containers, Orchestration, and Cloud Choices

• Containers, enabled by Docker, allow developers to encapsulate applications and dependencies, facilitating consistency across environments from local development to production.
• Deployment options include AWS ECS/Fargate for managed orchestration, Kubernetes for large-scale or multi-cloud scenarios, and simpler services like AWS App Runner and Elastic Beanstalk for small-scale applications.
• Kubernetes provides robust flexibility and cross-provider support but brings high complexity, making it best suited for organizations with substantial infrastructure needs and experienced staff.
• Use of cloud services versus open-source alternatives on Kubernetes (e.g., RDS vs. Postgres containers) affects manageability, vendor lock-in, and required expertise.

DevOps and Architecture: Roles and Collaboration

• DevOps unites development and operations through common processes and tooling to accelerate safe production deployments and improve coordination.
• Architecture focuses on the holistic design of systems, establishing how different technical components fit together and serve overall business or product goals.
• There is significant overlap, but architecture plans and outlines systems, while DevOps engineers implement, automate, and monitor deployment and operations.
• Cross-functional collaboration is essential, as machine learning engineers, DevOps, and architects must communicate requirements, constraints, and changes, especially regarding production-readiness and security.

Security, Scale, and When to Seek Help

• Security is a primary concern when moving to production, especially if handling sensitive data or personally identifiable information (PII); professional DevOps involvement is strongly advised in such cases.
• Common cloud security pitfalls include publicly accessible networks, insecure S3 buckets, and improper handling of secrets and credentials.
• For experimentation or small-scale safe projects, machine learning engineers can use tools like Terraform, Docker, and AWS managed services, but should employ cloud cost monitoring to avoid unexpected bills.

Cloud Providers and Service Considerations

• AWS dominates the cloud market, followed by Azure (strong in enterprise/Microsoft-integrated environments) and Google Cloud Platform (GCP), which offers a strong user interface but has a record of sunsetting products.
• Managed cloud machine learning services, such as AWS SageMaker and GCP Vertex AI, streamline model training, deployment, and monitoring.
• Vendor-specific tools simplify management but limit portability, while Kubernetes and its ML pipelines (e.g., Kubeflow, Apache Airflow) provide open-source, cross-cloud options with greater complexity.

Recommended Learning Paths and Community Resources

• Learning and prototyping with Terraform, Docker, and basic cloud services is encouraged to understand deployment pipelines, but professional security review is critical before handling production-sensitive data.
• For those entering DevOps, structured learning with platforms like aCloudGuru or AWS's own curricula can provide certification-ready paths.
• Continual learning is necessary, as tooling and best practices evolve rapidly.

Reference Links

Expert coworkers at Dept

• Matt Merrill - Principal Software Developer
• Jirawat Uttayaya - DevOps Lead
• The Ship It Podcast (frequent discussions on DevOps and architecture)

DevOps Tools

• Terraform
• Ansible

Visual Guides and Comparisons

• Which AWS container service should I use?
• A visual guide on troubleshooting Kubernetes deployments
• Public Cloud Services Comparison
• Killed by Google

Learning Resources

• aCloudGuru AWS curriculum

Try a walking desk to stay healthy while you study or work!

Full notes at ocdevel.com/mlg/mla-18

(Optional episode) just showcasing a cool application using machine learning

Dept uses Descript for some of their podcasting. I'm using it like a maniac, I think they're surprised at how into it I am. Check out the transcript & see how it performed.

• Descript
• The Ship It Podcast How to ship software, from the front lines. We talk with software developers about their craft, developer tools, developer productivity and what makes software development awesome. Hosted by your friends at Rocket Insights. AKA shipit.io
• Brandbeats Podcast by BASIC An agency podcast with views on design, technology, art, and culture. Explore the new microsite at www.brandbeats.basicagency.com

AWS development environments for local and cloud deployment can differ significantly, leading to extra complexity and setup during cloud migration. By developing directly within AWS environments, using tools such as Lambda, Cloud9, SageMaker Studio, client VPN connections, or LocalStack, developers can streamline transitions to production and leverage AWS-managed services from the start. This episode outlines three primary strategies for treating AWS as your development environment, details the benefits and tradeoffs of each, and explains the role of infrastructure-as-code tools such as Terraform and CDK in maintaining replicable, trackable cloud infrastructure.

Links

• Notes and resources at ocdevel.com/mlg/mla-17
• Try a walking desk stay healthy & sharp while you learn & code

Docker Fundamentals for Development

• Docker containers encapsulate operating systems, packages, and code, which simplifies dependency management and deployment.
• Files are added to containers using either the COPY command for one-time inclusion during a build or the volume directive for live synchronization during development.
• Docker Compose orchestrates multiple containers on a local environment, while Kubernetes is used at larger scale for container orchestration in the cloud.

Docker and AWS Integration

• Docker is frequently used in AWS, including for packaging and deploying Lambda functions, SageMaker jobs, and ECS/Fargate containers.
• Deploying complex applications like web servers and databases on AWS involves using services such as ECR for image storage, ECS/Fargate for container management, RDS for databases, and requires configuration of networking components such as VPCs, subnets, and security groups.

Challenges in Migrating from Localhost to AWS

• Local Docker Compose setups differ considerably from AWS managed services architecture.
• Migrating to AWS involves extra steps such as pushing images to ECR, establishing networking with VPCs, configuring load balancers or API Gateway, setting up domain names with Route 53, and integrating SSL certificates via ACM.
• Configuring internal communication between services and securing databases adds complexity compared to local development.

Strategy 1: Developing Entirely in the AWS Cloud

• Developers can use AWS Lambda's built-in code editor, Cloud9 IDE, and SageMaker Studio to edit, run, and deploy code directly in the AWS console.
• Cloud-based development is not tied to a single machine and eliminates local environment setup.
• While convenient, in-browser IDEs like Cloud9 and SageMaker Studio are less powerful than established local tools like PyCharm or DataGrip.

Strategy 2: Local Development Connected to AWS via Client VPN

• The AWS Client VPN enables local machines to securely access AWS VPC resources, such as RDS databases or Lambda endpoints, as if they were on the same network.
• This approach allows developers to continue using their preferred local IDEs while testing code against actual cloud services.
• Storing sensitive credentials is handled by AWS Secrets Manager instead of local files or environment variables.
• Example tutorials and instructions:
  • AWS Client VPN
  • Terraform example
  • YouTube tutorial
  • Creating the keys

Strategy 3: Local Emulation of AWS Using LocalStack

• LocalStack provides local, Docker-based emulation of AWS services, allowing development and testing without incurring cloud costs or latency.
• The project offers a free tier supporting core serverless services and a paid tier covering more advanced features like RDS, ACM, and Route 53.
• LocalStack supports mounting local source files into Lambda functions, enabling direct development on the local machine with changes immediately reflected in the emulated AWS environment.
• This approach brings rapid iteration and cost savings, but coverage of AWS features may vary, especially for advanced or new AWS services.

Infrastructure as Code: Managing AWS Environments

• Managing AWS resources through the web console is not sustainable for tracking or reproducing environments.
• Infrastructure as code (IaC) tools such as Terraform, AWS CDK, and Serverless enable declarative, version-controlled description and deployment of AWS services.
• Terraform offers broad multi-cloud compatibility and support for both managed and cloud-native services, whereas CDK is AWS-specific and typically more streamlined but supports fewer services.
• Changes made via IaC tools are automatically propagated to dependent resources, reducing manual error and ensuring consistency across environments.

Benefits of AWS-First Development

• Developing directly in AWS or with local emulation ensures alignment between development, staging, and production environments, reducing last-minute deployment issues.
• Early use of AWS services can reveal managed solutions—such as Cognito for authentication or Data Wrangler for feature transformation—that are more scalable and secure than homegrown implementations.
• Infrastructure as code provides reproducibility, easier team onboarding, and disaster recovery.

Alternatives and Kubernetes

• Kubernetes represents a different model of orchestrating containers and services, generally leveraging open source components inside Docker containers, independent of managed AWS services.
• While Kubernetes can manage deployments to AWS (via EKS), GCP, or Azure, its architecture and operational concerns differ from AWS-native development patterns.

Additional AWS IDEs and Services

• Lambda
• SageMaker Studio
• Cloud9

Conclusion

• Choosing between developing in the AWS cloud, connecting local environments via VPN, or using tools like LocalStack depends on team needs, budget, and workflow preferences.
• Emphasizing infrastructure as code ensures environments remain consistent, maintainable, and easily reproducible.

SageMaker streamlines machine learning workflows by enabling integrated model training, tuning, deployment, monitoring, and pipeline automation within the AWS ecosystem, offering scalable compute options and flexible development environments. Cloud-native AWS machine learning services such as Comprehend and Poly provide off-the-shelf solutions for NLP, time series, recommendations, and more, reducing the need for custom model implementation and deployment.

Links

• Notes and resources at ocdevel.com/mlg/mla-16
• Try a walking desk stay healthy & sharp while you learn & code

Model Training and Tuning with SageMaker

• SageMaker enables model training within integrated data and ML pipelines, drawing from components such as Data Wrangler and Feature Store for a seamless workflow.
• Using SageMaker for training eliminates the need for manual transitions from local environments to the cloud, as models remain deployable within the AWS stack.
• SageMaker Studio offers a browser-based IDE environment with iPython notebook support, providing collaborative editing, sharing, and development without the need for complex local setup.
• Distributed, parallel training is supported with scalable EC2 instances, including AWS-proprietary chips for optimized model training and inference.
• SageMaker's Model Debugger and monitoring tools aid in tracking performance metrics, model drift, and bias, offering alerts via CloudWatch and accessible graphical interfaces.

Flexible Development and Training Environments

• SageMaker supports various model creation approaches, including default AWS environments with pre-installed data science libraries, bring-your-own Docker containers, and hybrid customizations via requirements files.
• SageMaker JumpStart provides quick-start options for common ML tasks, such as computer vision or NLP, with curated pre-trained models and environment setups optimized for SageMaker hardware and operations.
• Users can leverage Autopilot for end-to-end model training and deployment with minimal manual configuration or start from JumpStart templates to streamline typical workflows.

Hyperparameter Optimization and Experimentation

• SageMaker Experiments supports automated hyperparameter search and optimization, using Bayesian optimization to evaluate and select the best performing configurations.
• Experiments and training runs are tracked, logged, and stored for future reference, allowing efficient continuation of experimentation and reuse of successful configurations as new data is incorporated.

Model Deployment and Inference Options

• Trained models can be deployed as scalable REST endpoints, where users specify required EC2 instance types, including inference-optimized chips.
• Elastic Inference allows attachment of specialized hardware to reduce costs and tailor inference environments.
• Batch Transform is available for non-continuous, ad-hoc, or large batch inference jobs, enabling on-demand scaling and integration with data pipelines or serverless orchestration.

ML Pipelines, CI/CD, and Monitoring

• SageMaker Pipelines manages the orchestration of ML workflows, supporting CI/CD by triggering retraining and deployments based on code changes or new data arrivals.
• CI/CD automation includes not only code unit tests but also automated monitoring of metrics such as accuracy, drift, and bias thresholds to qualify models for deployment.
• Monitoring features (like Model Monitor) provide ongoing performance assessments, alerting stakeholders to significant changes or issues.

Integrations and Deployment Flexibility

• SageMaker supports integration with Kubernetes via EKS, allowing teams to leverage universal orchestration for containerized ML workloads across cloud providers or hybrid environments.
• The SageMaker Neo service optimizes and packages trained models for deployment to edge devices, mobile hardware, and AWS Lambda, reducing runtime footprint and syncing updates as new models become available.

Cloud-Native AWS ML Services

• AWS offers a variety of cloud-native services for common ML tasks, accessible via REST or SDK calls and managed by AWS, eliminating custom model development and operations overhead.
  • Comprehend for document clustering, sentiment analysis, and other NLP tasks.
  • Forecast for time series prediction.
  • Fraud Detector for transaction monitoring.
  • Lex for chatbot workflows.
  • Personalize for recommendation systems.
  • Poly for text-to-speech conversion.
  • Textract for OCR and data extraction from complex documents.
  • Translate for machine translation.
  • Panorama for computer vision on edge devices.
• These services continuously improve as AWS retrains and updates their underlying models, transferring benefits directly to customers without manual intervention.

Application Example: Migrating to SageMaker and AWS Services

• When building features such as document clustering, question answering, or recommendations, first review whether cloud-native services like Comprehend can fulfill requirements prior to investing in custom ML models.
• For custom NLP tasks not available in AWS services, use SageMaker to manage model deployment (e.g., deploying pre-trained Hugging Face Transformers for summarization or embeddings).
• Batch inference and feature extraction jobs can be triggered using SageMaker automation and event notifications, supporting modular, scalable, and microservices-friendly architectures.
• Tabular prediction and feature importance can be handled by pipe-lining data from relational stores through SageMaker Autopilot or traditional algorithms such as XGBoost.
• Recommendation workflows can combine embeddings, neural networks, and event triggers, with SageMaker handling monitoring, scaling, and retraining in response to user feedback and data drift.

General Usage Guidance and Strategy

• Employ AWS cloud-native services where possible to minimize infrastructure management and accelerate feature delivery.
• Use SageMaker JumpStart and Autopilot to jump ahead in common ML scenarios, falling back to custom code and containers only when unique use cases demand.
• Leverage SageMaker tools for pipeline orchestration, monitoring, retraining, and model deployment to ensure scalable, maintainable, and up-to-date ML workflows.

Useful Links

• MadeWithML overview & ML tutorials
• SageMaker Home
• SageMaker JumpStart
• SageMaker Model Deployment
• SageMaker Pipelines
• SageMaker Model Monitor
• SageMaker Kubernetes Integration
• SageMaker Neo