Local Model Aspirations

For all of the folks out there who want to embrace AI but also have issues with the energy consumption and cost, I’m right there with you. When AI started to become mainstream and then eventually mandatory in enterprise work, I was very reluctant to use it and train these models that my company will eventually replace me with.

Well…….that happened verbatim 😅. My previous place of work has been going through rounds of layoffs after a new set of C-suite execs strolled in to make “improvements” to the business, aka, “the company isn’t making enough money so, hire offshore, fire employees with IP knowledge, and go heavy on using AI.” I’m obviously exaggerating (kinda), but the point I’m trying to make is that AI is great when you have a solid plan on how to use it and try not to rush the process. Most companies didn’t take that approach, though, and got shot in the foot because of it.

Does that mean we should oppose AI? Absolutely not. I actually love what AI can be used for, and I think there isn’t enough fine-tuning out there for specific use cases. Now, with Chinese models becoming much more competitive in intelligence, the cost difference is a no-brainer, and many companies are switching to these cheaper yet very capable models.

(Image Source: https://techstartups.com/2026/06/29/western-companies-are-quietly-switching-to-chinese-ai-models-as-u-s-frontier-ai-prices-rise/)

Then, when you use platforms like DeepInfra as your provider, scaling becomes less of an issue, and you only pay for usage and not a monthly subscription. Add the OpenRouter API in the mix, and you have the ability to swap providers at will. If costs ever change, if one provider hits you with rate limits, or if a provider goes down completely, OpenRouter is the failsafe that will forward your request to a different provider automatically (with the proper configurations) so your clients won’t have issues with the products/software you built for them.

This just goes to show how far we’ve come with AI development, and like it or not, it’s a tool almost as important as the internet itself but potentially much more dangerous. While that danger may include physical harm, I’m more worried about the mental and intellectual harm that it’s doing and will continue to do if we don’t make some major changes in how the average person should approach AI.

This is why I started diving into local models, with the understanding that they are not as capable as the frontier models, but that’s the point. They are SLMs and MLMs that you can leverage, but it will still require you to do some sort of critical thinking, which is a good thing. There is absolutely no reason for people who are using AI to just chat or ask a simple question, to be using the compute power and energy consumption of the frontier models. That’s one reason why I marveled over Google’s Gemma 4 series because, in theory, they have a model that should serve on just about any device. In practice, though, I’m learning some major limitations with these models, but most of them have to do with local compute power (as well as my current lack of knowledge on how to fine-tune these systems). That isn’t to say local models will never work; we have come so far with local models already that I'm sure in the next year or so, there will be a better generation of local 1st models that will work as smoothly as using a frontier cloud model. And “smoothly” is the keyword here, because in terms of intelligence, that’s not the main selling point of these local and more ethical models.

Local Model Experimentation

So that leads us to this lil ol’ experiment with me trying to build a “lower third” animation for my podcast intro. A “lower third” is just an animation on the lower left or right of your screen with the host name and title, or whatever you’d like to pop up there briefly (or at least that’s how I understand it 😄). So you’d think that a local model would be able to knock this out no problem, right??

If you have a beefy enough local machine, you’re golden. However, most people do not have a $5k+ home lab that can run these local models without major lag and bottlenecks. When the Gemma4:12b model was released, I was ecstatic because it was small enough for my Mac Mini (M4) not to hit memory limits, and it allowed enough headroom for me to record on OBS without OBS crashing. But reality hit me hard when I needed this model to run efficiently in any capacity for the task at hand.

So I crafted 3 separate prompts for this task since, given the small context window limit of 32k I had to set, we needed to make sure to break up the tasks into individual prompts and sessions so the 12B model can stay as focused as possible. The original prompts are below:

[1/3] Create src/LowerThird.tsx. Transparent background. Static layout only,

[2/3] Animate the entrance (frames 0-21): the gradient bar scales in from the

[3/3] Animate the exit: frames 129-150 reverse the entrance. Hold frames 21-129.

But these evolved greatly throughout my multiple rounds. Before we get into what the final prompts were, we need to make sure that folks who want to follow along can get set up with the tools I used during this process.

Setup

0. Scaffold the Remotion project

# Prereqs: Node 18+ and pnpm

Then install the skills:

npx skills add remotion-dev/skills

That drops remotion-best-practices into the agent folders (.opencode/skills/, .claude/skills/, .agents/skills/, …) and records it in skills-lock.json, so OpenCode — and any other agent you point at the repo — picks it up automatically.

1. Serve the model with a real context window

Ollama’s default context (~4k tokens) is far too small for agentic coding — OpenCode stuffs system prompt + skill rules + file contents into context, and when it overflows, Ollama silently truncates the oldest tokens. The model “forgets” your instructions mid-task and the failure looks like model stupidity. It’s a config problem.

Note for context

We will be setting 2 different contexts for Gemma. One will allow me to record at the same time but will reduce my context window. The other will give a better context window for a lesser chance of hallucination.

You will also need to create Modfiles for our customized models to reference. Each modfile will be above the cat call. You can name the files whatever you want, but I’ve named the models according to their context window.

To bake it in permanently via a Modelfile:

Smaller Context

cat > /tmp/gemma4-12b-16k.Modelfile <<'EOF'

Larger Context

cat > /tmp/gemma4-12b-32k.Modelfile <<'EOF'

16k is the sweet spot on 24 GB: roughly 7.6 GB weights + a few GB of KV cache ≈ 11–12 GB, leaving headroom for Remotion Studio (Chrome) and OBS. 32k works too if you’re not recording at the same time.

You can also serve whatever model you have with a bigger context by running it with that flag.

# Quit the Ollama menu-bar app first, then serve with a bigger window:

But it’s better to set a standard so that you don’t have to serve a bigger context window with each launch of the model.

2. Point OpenCode at Ollama

Create opencode.json in the project root (~/code/remotion_animations):

{

Three notes:

* Model ids must match ollama list exactly — these are the two Modelfile variants built in step 1. If the id doesn’t match a local tag, OpenCode’s requests just 404 against Ollama.

* Top-level "model" sets the default, so OpenCode launches straight into the 16k local model — no /models dance every session. Swap to the 32k variant with /models when you’re not recording.

* "agent": { "build": { "temperature": 0.1 } } turns the sampling temperature way down for the build agent. For codegen you want boring and deterministic, not creative — a 12B at default temperature wanders; at 0.1 it follows the skill rules much more reliably.

You can also just make this update in the OpenCode root opencode.json file, which would typically be located at /Users/{user-name}/.config/opencode/opencode.json. So in my case, it would be /Users/kdleo/.config/opencode/opencode.json

3. Launch and verify

cd ~/code/remotion_animations

LowerThird Build Rounds

I found out quickly that name collisions stall on 12B. At least it did for me. Even though I had a different path listed for the creation, because the actual file name was the same as what I created during my dry run with a different model, it kept telling itself it needed to check that file, but then went back to saying “wait, the path is code/user/file-name.txt, I should just create a new file at the path specified,” but then it would go back to thinking it needed to check the existing file, and do this same loop over and over again. Apparently, the 12B model burns context and confidence on ambiguity a big model shrugs off.

This led me to add some fixes baked into the prompts that I had scripted for this build. I made notes of the technicals too so I can try to paint a better picture of what failed.

Round 1 results

Prompt 1 took 37 min: static layout came out right — bar, plate, fonts, colors, zod schema all to spec. But it hallucinated an AnnotationData import from remotion (lint caught it), left its own deliberation as inline comments, and never did the Root.tsx registration. It stalled asking about the name collision.

Prompts 2 and 3: zero surviving edits. The model went in circles and never wrote the animation to disk. git status after “finishing” showed the file was byte-identical to the prompt-1 state. Fail. Made some adjustments to the prompts and drilled down on explicit file paths and naming conventions for round 2.

Round 2 results

Prompt 1 & 2 took about 30 min combined: The one-file structure worked: Gemma never wandered into Root.tsx this time.

What killed it was prompt 3/3 itself — it said “reverse the entrance… using spring,” and what I didn’t know is that spring can’t run backwards: it’s driven by the raw frame number, not a 0–1 progress value. Gemma tried to comply literally — spring({ frame: interpolate(...) }), a progress value fed into a frame parameter — and circled until the session was abandoned. That one is a prompt bug, not a model bug. Along the way it also used the neighboring src/compositions/LowerThird.tsx as a template, copying its entire forbidden import header and renaming the export to LowerThird, which silently broke the Root.tsx registration; As well as botched an edit that duplicated the whole file top-to-bottom, leaving it uncompilable. Even the manual revert landed on a half-mutated state (junk imports + wrong export name) rather than the clean stub. Fail #2 lol.

Round 3 results that got me 95% there.

Round 3 was a bit different than the 1st 2 rounds in that I created some template files before running the prompts. Since I successfully created these animations in the past and had the files to reference, I decided to do the following as a pre-requisite before running any prompts:

Create src/compositions/LowerThirdGemma.tsx:

import React from "react";

Register it in src/Root.tsx — add with the other imports at the top:

import {

Add the expected LowerThirdGemma and reuse LOWER_THIRD_DEFAULTS:

pnpm lint passes and LowerThirdGemma-Wide shows up in Studio as an empty comp. From here, the model is expected not to touch Root.tsx, can’t create files, and shouldn’t collide with anything.

The final 3 prompts i used are below:

[1/3] Open src/compositions/LowerThirdGemma.tsx. The component is a stub and

[2/3] Open src/compositions/LowerThirdGemma.tsx and edit only this file. It

[3/3] Open src/compositions/LowerThirdGemma.tsx and edit only this file. It

And things were moving pretty smoothly….until prompt 3 AGAIN. For some reason this model just has the hardest time understanding our transitions even when i explicitly tell it how to do it. After it kept going in circles yet again (and i let it run for about an hr on prompt 3 😭) i gave up and decided to use the 31B cloud model, which is still less intelligent than frontier models but it was in the cloud so my aim was to just compare how long it would take to try and resolve the issue. It was a matter of seconds 😐. Below is the correction it did to fix what the 12B model (on my hardware) just could NOT do:

…… That’s it……it literally just had a few misplaced brackets and some missing math logic. Over an hr, it couldn't resolve that issue, and again, I partly blame my hardware, but I’m also aware that it could very well be a limitation in this area of intelligence for this model. I doubt these models have been trained on much code related to Remotion, so I can understand why this Local experiment was a failure, but that doesn’t mean I'm not still a bit salty about it lol.

Local Model Reality

At the end of the day, it was technically a failure. Regardless if I got 90% there, the model could not cross the finish line without manual intervention. And when you compare it to the likes of Opus or Fable, there is no comparison when it comes to intelligence whatsoever. It only took me 1 go using Claude, and the prompts were not nearly as precise or had as much context and setup that was in place for Gemma4:12B. While the 31B model might actually be more than capable at running through these prompts (and maybe even the 26B), you’re limited by the Machine you have to work with. So I will forever be stuck with the 12B local model until i upgrade, or they make a more efficient and intelligent local-first model. The results for the lower third animations are also different in quality.

Claude’s 1st attempt with general and somewhat vague instructions:

Gemma4:12B with multiple rounds of prompt revisions, ultimately needing Gemma4:31b (cloud) to finish the animation:

Moral of the story here is to understand what you’re looking to use AI for and then choose the proper models and tools to try and use it as efficiently as possible, without relying on it to do all of the work for you.

Learning how to build curated RAG Systems is my most pressing task for upskilling, because I want to be able to provide reasonable options for small businesses to choose when it comes to getting involved in AI. I know there are plenty of small businesses in my community that have been underserved for decades when it comes to new advanced technology, and my goal is to assist with bridging that gap.

Conclusion

That about wraps it up for this episode. Hopefully y’all learned a bit about local models, their benefits, and also their bottlenecks when it comes to building using local AI only. I’m certain they will only get better and better as time goes on, but for decisions that need to be made today, take caution and choose wisely ✌🏾.

If you want to keep up with my work or want to connect as peers, check out my social links below and give me a follow!

* 🦋 Bluesky

* 📸 Instagram

* ▶️ Youtube

* 💻 Github

* 👾 Discord