Local AI Agents Failed Us: What 5 Days Taught

Here's the deal: local AI agents were supposed to save us from Anthropic's billing cap. Five days later, we'd reverted every single engineering agent back to Claude and written off the experiment as a cautionary tale. Not a partial win. Not "promising with caveats." A full reversal.

I want to be specific about what broke, because the AI discourse is full of success theater and I'm tired of it.

Why We Tried This in the First Place

We hit Anthropic's usage cap mid-sprint. That's the whole story. We run agentic workflows for several client projects, including work adjacent to KRAIN and Crowdia, and when you're running multi-step code review agents, documentation agents, and architecture feedback loops simultaneously, you burn through API credits faster than you'd expect. The cap hit on a Wednesday afternoon and we needed to make a decision: throttle the workflows, pay for a higher tier immediately, or try going local.

We chose local. We had the hardware for it. The Flow Z13 with a Strix Halo chip has been sitting on my desk doing serious ML inference work for months. On paper, the specs support running quantized 70B models at reasonable throughput. We spun up Ollama, pulled a few models, and started routing our agents through localhost instead of api.anthropic.com.

What followed was instructive in the worst way.

What the LLM Failure Actually Looked Like

The first thing that broke wasn't inference speed. Speed was fine. The first thing that broke was instruction following on multi-step agentic tasks.

Here's what most guides miss about local models and agent loops: benchmarks measure single-turn quality. They ask the model a question, score the answer, move on. Agents don't work that way. Agents run in loops. They call tools, get results, reason about those results, call more tools, and eventually synthesize an output. Every loop iteration is another chance for the model to drift from the original instruction.

Cloud Claude holds context and intent across those loops with a consistency that, after this week, I now recognize as genuinely remarkable engineering. The local models we tested, including strong quantized variants of Llama 3 and Mistral-class models, would start a task correctly and then subtly reinterpret the goal by iteration three or four. Not hallucinate wildly. Just... drift. The agent would decide it had completed a task when it hadn't. Or it would start solving a slightly different problem than the one specified. The outputs looked reasonable in isolation. In a workflow, they were poison.

The code review agent was the clearest failure case. It would identify real issues in the first pass, then in subsequent reasoning steps, it would start rationalizing why those issues weren't actually problems, and produce a final report that contradicted its own earlier analysis. The model was arguing with itself and losing.

The Machine Learning Gap Nobody Talks About

There's a capability gap between frontier models and open-weight local models that the AI integration community consistently underestimates. The gap isn't raw intelligence on simple tasks. It's reliability on complex, multi-constraint tasks under ambiguous conditions.

Linear's recent data on AI usage patterns in software teams is worth reading here. Their findings show that AI tools are deeply embedded in engineering workflows now, which means the failure modes matter more than ever. When an AI agent fails silently in a loop, it doesn't just waste time. It can corrupt the work downstream. A code review agent that produces a confident-but-wrong analysis gets committed to a PR, reviewed by a human who trusts the AI summary, and then shipped. The failure propagates.

We saw this. We caught it before anything shipped, but only because we had a human in the loop specifically watching for it. That human oversight cost us more time than the billing cap would have.

Where the Real Line Sits

After five days, here's my actual assessment of where local AI agents are useful versus where they fall apart in 2026.

Where local works: Single-turn generation tasks with well-defined outputs. Summarization of a specific document. Generating boilerplate from a clear template. Classifying items against a fixed taxonomy. Answering questions about a local codebase when the context window is managed carefully. These tasks have a clear success state and don't require the model to maintain goal coherence across multiple reasoning steps.

Where local fails: Anything with a feedback loop. Anything requiring the model to evaluate its own prior outputs critically. Anything where the task involves navigating ambiguity over multiple steps. Anything where "close enough" produces downstream damage rather than a recoverable miss.

The honest framing is this: local AI agents are good at tasks that didn't really need an agent in the first place. The tasks that benefit most from agentic architecture, the ones where you want autonomous multi-step reasoning, are exactly the tasks where local models degrade fastest.

The Hardware Wasn't the Problem

I want to be clear about this because people always jump to hardware as the excuse. The Strix Halo rig handled inference fine. Latency was acceptable. The models loaded cleanly. Ollama and llama.cpp both performed as advertised.

The hardware gap between "can run a 70B model" and "can run a 70B model well enough for production agentic work" is real, but we weren't bottlenecked by hardware. We were bottlenecked by model quality on the specific task profile that matters for engineering agents.

This is also why I'm watching the Mojo language going open-source under Qualcomm's stewardship with interest. Mojo's pitch has always been that Python-level ergonomics shouldn't require Python-level performance tradeoffs, especially for ML workloads. If that materializes at the systems level, it could eventually change what's possible for on-device inference optimization. But "eventually" is doing a lot of work in that sentence. That's not a 2026 solution to a 2026 problem.

What We Actually Learned About AI Integration

The failure taught us something more valuable than the experiment would have if it had succeeded: we learned that our agentic workflows have a real dependency on frontier model reliability that we'd been treating as a commodity.

When you run cloud Claude for six months without major failures, you start assuming that reliability is table stakes for any sufficiently capable model. It's not. That reliability is a product of training choices, RLHF tuning, and safety work that frontier labs have spent enormous resources on, specifically targeting the kinds of instruction-following failures that destroy agentic reliability. The open-weight models are catching up on benchmarks. They are not catching up on this.

The practical implication is that AI integration architecture needs to account for model-specific reliability profiles, not just capability profiles. A model that scores well on MMLU might still be a disaster in a ReAct loop. These are different things and we need to evaluate them separately.

The Reversion Decision

On day five, we stopped the experiment and went back to cloud Claude. The decision was straightforward once we framed it correctly. The billing cap cost us a few hours of productivity. The local agent failures were costing us hours of review time per day to catch and correct outputs that looked right but weren't. The math wasn't close.

We also made a structural change: we now treat our API dependency on frontier models as infrastructure cost, not a variable to optimize away. It goes in the budget the same way compute and storage do. Trying to eliminate it with local AI agents, at the current state of the technology for our specific use cases, is like trying to replace your database with a spreadsheet to save on licensing. You can do it. You'll regret it.

The right place to use local models in our stack, and we are using them, is in preprocessing pipelines where the task is simple and latency matters more than reliability. Embedding generation. Classification at scale. Summarization as a preprocessing step before a human or frontier model handles the actual decision. Local models as a component in a larger system, with appropriate reliability expectations, is a completely different proposition than local AI agents running autonomously.

Where This Leaves Us

The honest recommendation: if you're running production agentic workflows in 2026, you are paying for frontier model access. Plan for it. Build your pricing around it. If a billing cap is disrupting your sprint, the answer is a higher tier or a better rate negotiation, not a five-day detour into local model infrastructure.

If you want to use local models, use them for the right jobs. Single-turn, well-defined, failure-tolerant tasks. Keep humans in the loop where the task profile doesn't fit that description. And for the love of everything, don't let a local AI agent autonomously touch anything that propagates downstream before you've built serious evaluation tooling around it.

The technology will get there. The open-source model ecosystem is moving fast, and the AI usage patterns Linear documented suggest the demand signal for reliable local inference is only growing. But "will get there" and "is there now" are different things, and five days of real failure data is worth more than any benchmark comparison I could have read instead.

We learned it the hard way. You don't have to.