Oliver: AI Agent That Runs Prod While We Sleep
Here's the deal: autonomous AI agent server management is no longer a conference talk concept. We are running it in production right now, and it has changed how our on-call rotation works in ways I did not fully anticipate when we started building.
This is the honest walkthrough of Oliver, the AI agent we built internally at Bedda.tech that handles production incidents, triages alerts, and in some cases executes remediations without a human in the loop. I will cover the real architecture, the failure modes we hit, and the moments Oliver got things embarrassingly wrong. No marketing spin.
Why We Built It Instead of Buying It
The honest answer is that nothing on the market fit our operational profile. We run a mix of client infrastructure (KRAIN's logistics platform, Crowdia's multi-tenant environment, Nozio's hospitality APIs) alongside our own services, and each of those has different runbook conventions, alert thresholds, and blast radius tolerance. Commercial AIOps tools assume a degree of homogeneity we do not have.
The second reason is cost opacity. The tools that do handle heterogeneous environments price on alert volume, which is exactly the wrong incentive. You want your monitoring to be noisy and catch everything. You do not want to pay more for that.
So we built Oliver. The name came from a junior engineer on the team who said the agent reminded her of a very calm, very literal-minded butler. It stuck.
The Architecture, Plainly Described
Oliver sits on top of the Claude API using tool use (what Anthropic calls function calling in their older docs). The core loop is simple:
- An alert fires from our observability stack (we use a combination of Prometheus, Grafana alerting, and custom webhook handlers).
- The alert payload hits Oliver's ingest endpoint.
- Oliver assembles a context window: the raw alert, the last several lines of relevant logs, recent deployment history pulled from our internal deploy ledger, and the runbook for that service if one exists in our SQLite-backed knowledge base.
- That full context goes to Claude with a set of available tools: read logs, query metrics, restart a service, scale a deployment, post to Slack, page a human, and do nothing.
- Claude reasons through the context, calls tools to gather more information, and either resolves the incident or escalates with a written summary.
The SQLite knowledge base is the part people underestimate. Oliver does not make good decisions without it. Early versions had no memory of prior incidents, so it would repeatedly "discover" the same class of problem as if it were novel. The knowledge base stores incident history, resolution outcomes, and a running record of which runbook steps actually worked versus which ones Oliver attempted and had to roll back. That feedback loop is what makes the autonomous AI agent server management piece actually work over time rather than just for the first week.
We keep the state machine deliberately shallow. Oliver has three modes: observe, act, and escalate. It cannot be in two modes simultaneously, and every state transition is logged to an append-only table. This was a hard lesson from an early incident I will get to shortly.
What the LLM Actually Does Well
The surprising wins have been in triage, not remediation.
Before Oliver, our on-call engineers spent a meaningful portion of incident time just figuring out what was actually wrong. An alert like "p95 latency spiked on the payments service" could mean a dozen different things. Oliver is very good at correlating that spike with a deployment that happened 40 minutes earlier, a database connection pool that is sitting at 98% utilization, and a single slow query that started appearing in the logs. It writes a plain-English summary of that correlation chain before it does anything else.
That summary alone has cut our mean time to understand (MTTU, a metric I care about more than MTTR) significantly. Even when Oliver hands off to a human, that human starts from a much better position.
The other area where it performs well is overnight low-severity incidents. Disk filling up on a non-critical service at 2am. A cron job that failed and needs a manual retry. A certificate that is 10 days from expiry and needs a renewal trigger. These are the things that used to generate a Slack notification that someone would see at 9am and handle in five minutes. Oliver handles them at 2am and posts a summary in the morning. No one loses sleep, the system stays healthy.
The Failure Modes Are Real and Specific
I want to be direct about where Oliver has gotten things wrong, because the pattern of failures tells you something important about the limits of autonomous AI agent server management as a category.
The confidence problem. Claude, like most frontier models, does not naturally express calibrated uncertainty when the context is ambiguous. In one incident, Oliver was looking at a memory leak pattern on a Node service. The runbook had two remediation paths: a soft restart (low risk) and a full redeploy with a config flag change (higher risk, required to fix the underlying cause). Oliver chose the full redeploy. It was correct that the leak needed the config change. It was incorrect that this particular moment, during a period of elevated traffic, was the right time to do it. The redeploy caused a brief outage during a window we would normally have protected.
The fix was not to make Oliver less autonomous. The fix was to add a traffic-awareness check to the tool schema. Oliver now queries current RPS and active session count before any action that involves a restart, and has a hard-coded rule that certain action types require human approval if traffic is above a threshold. The LLM does not override that rule because the rule is enforced at the tool layer, not in the prompt.
The runbook drift problem. Our knowledge base only knows what we put in it. When a service gets refactored and the old runbook steps no longer apply, Oliver will still try them. We had a case where Oliver attempted to restart a service using a systemd command that had been replaced by a Kubernetes rollout restart months earlier. The systemd command failed silently on the new infrastructure, Oliver saw no error, reported the incident resolved, and the service stayed degraded for another 20 minutes until a human noticed.
The lesson: runbook freshness is an operational discipline problem, not an AI problem. But the AI makes the consequences of stale runbooks worse because it acts with confidence on bad information. We now have a scheduled job that flags any runbook entry older than 90 days for human review. Low-tech solution, but it works.
The tool call loop problem. Early on, we had an incident where Oliver got stuck in a diagnostic loop. It would call the log-reading tool, get back a large payload, summarize it, realize it still did not have enough information, call the log-reading tool again with a slightly different query, and repeat. This burned through tokens fast and did not resolve the incident. We added a maximum tool call count per incident (currently capped at 12 sequential calls) after which Oliver is forced to escalate to a human regardless of its confidence level. The cap is arbitrary but it stops the runaway case.
What Most Guides Miss About AI Integration in DevOps
The framing you see most often is "give the AI access to your tools and it will figure things out." That framing skips the part where you have to design your tools to be AI-legible.
A tool that returns a 500-line log dump is not AI-legible. A tool that returns a structured summary of the last 20 error events with timestamps, error codes, and affected request IDs is AI-legible. The difference in Oliver's decision quality between those two tool designs is not marginal. It is the difference between Oliver making a reasonable call and Oliver hallucinating a cause because the signal was buried in noise.
The same principle applies to your alert payloads. Raw Prometheus alerts carry a lot of metadata that means something to a human who knows the system but is confusing to an LLM that is seeing it for the first time every time. We wrote a small normalization layer that translates alert payloads into a consistent schema before they hit Oliver. That normalization layer is probably the highest-leverage thing we built.
There is also a cultural piece that the architecture posts do not cover. The team has to trust Oliver enough to let it act, but not so much that they stop reviewing its decisions. We do a weekly incident review where we look at every action Oliver took, whether it was correct, and whether we would have done the same thing. That review is where most of our prompt and tool improvements come from. Without it, Oliver would have calcified at whatever quality level it shipped at.
The Current State and Where This Goes
Oliver is not a product. It is an internal tool that is deeply coupled to our specific infrastructure, our runbook conventions, and our alert taxonomy. I mention this because I see a lot of energy in the market right now around autonomous AI agent server management platforms that promise to work out of the box. I am skeptical. The value in this kind of system is in the fit between the agent's knowledge and your specific operational reality. That fit takes time to build.
What I do think generalizes is the architecture pattern: a shallow state machine, an append-only audit log, tool schemas designed for AI legibility, a knowledge base with freshness enforcement, and hard-coded guardrails at the tool layer that the LLM cannot reason its way around. Those are the pieces that make autonomous operation safe enough to actually run.
The Anthropic tool use documentation is the right starting point if you want to build something similar. The Claude system prompt guidelines are worth reading carefully for the parts about uncertainty and refusal behavior, because how you frame the agent's operating context shapes its risk tolerance more than any individual instruction.
There is a thread on Hacker News right now about running a Java Spring Boot app on a 512 MB VPS with lightweight monitoring, and the comments are full of people hand-crafting monitoring scripts and cron jobs to manage resource constraints. That is exactly the class of problem where a well-scoped AI agent adds real value: the low-severity, high-frequency operational work that does not need a human but currently gets one anyway because there is no better option.
My Actual Recommendation
If you are running production infrastructure with a small team and you are tired of being paged for things that have known fixes, build something like Oliver. Start small: pick one category of alert (disk space, certificate expiry, something with a clear and safe remediation), build the tool schema for just that category, and run it in observe-only mode for a few weeks before you let it act. The observe-only period is how you build trust in the system and catch the cases where it would have done something wrong.
Do not start with incident response for your most critical services. Start with the boring overnight stuff where the cost of a wrong call is low. Get the audit log right from day one so you can review everything. Add the traffic-awareness guardrails before you give it restart permissions.
The goal is not to remove humans from operations. The goal is to make sure humans are involved in the decisions that actually require human judgment, and not burning out on the ones that do not. Oliver has moved us meaningfully closer to that. It is not magic, it breaks in specific and predictable ways, and maintaining it is real work. But it runs prod while we sleep, and that is worth something.