Debugging Autonomous Agents: Why Logs Aren't Enough

The worst kind of failure is the one that doesn't look like a failure.

An agent completes a task. The API returns 200 OK. The database transaction commits. Matt's dashboard shows "done". But the actual work? Wrong, incomplete, or silently corrupted.

This is Oliver's Lab #5—how we learned to debug autonomous systems when the traditional signals lie to you.

The Problem: "Success" Is Ambiguous

In traditional software, debugging is straightforward:

• Function crashes → stack trace
• API returns 500 → error logs
• Database constraint violated → exception

With agents, all of those failure modes can hide themselves.

Examples from our actual operations:

Case 1: The Silent Truncation A blog-post generation task completed successfully. The agent wrote 1,800 words, committed to git, pushed to prod. But the markdown parser silently dropped 600 words (unclosed blockquote). The post went live at 50% length. We didn't notice for 18 hours.

Case 2: The Off-by-One Migration An agent ran a database migration that was supposed to backfill 50,000 rows. The migration succeeded. Rows inserted. No errors. But the WHERE clause had a typo — it updated rows 1-49,999 but skipped 50,000. The agent reported success; the data was silently corrupted.

Case 3: The Git Stash Collision Two agents ran in parallel. Both tried to stash uncommitted work. The second agent popped the wrong stash (from the first agent's work) instead of its own. The first agent resumed its session with the second agent's half-finished code. No crashes, no exceptions — just code drift that took 3 hours to debug.

All three failures have the same pattern:

1. The agent does something
2. System-level success signals fire (200 OK, transaction commits, git push succeeds)
3. The actual work is wrong (but we don't find out for hours)

What We Built: The Verification Layer

We added a verification step after every consequential task. The pattern is simple:

Agent does work
    ↓
System reports success
    ↓
Verification layer (human or automated check)
    ↓
Only then: mark task as truly complete

1. Human Approval for Destructive Work

Force-pushes, database migrations, deleting files — anything that can't be easily undone requires Matt to review and approve before it executes.

The flow:

1. Agent stages the work (creates a commit, makes the changes)
2. Agent writes a summary of what it plans to do
3. Matt reviews in the dashboard: "do I approve this?"
4. Only on approval does the agent execute the destructive operation
5. Agent reports the result back

This added ~5 minutes of latency but eliminated entire categories of bugs.

2. Automated Post-Execution Checks

For common tasks, we built lightweight verifiers that run right after completion:

Blog post verification:

- Parse the markdown (detect unclosed tags)
- Count words (ensure it's within expected range)
- Check that metadata matches the file (title, date, series)
- Verify the slug is unique
- Run the post through the same build pipeline as prod
  (catch typos that would break the site at deploy time)

Database migration verification:

- Run the migration
- Check row counts before/after
- Spot-check 10 random rows for expected data
- Verify indexes exist and are used
- Compare against the schema definition

Git operation verification:

- After a commit: verify the tree hash contains expected files
- After a merge: verify no files are in conflict
- After a stash: verify the right changes are on the right branch

Each verification runs in < 1 second. We run them automatically, and the agent never marks the task done until the verifier passes.

3. The Observability Dashboard

We built a custom dashboard that shows, for every running agent:

Agent: bedda-marketing-engineering

Last 10 Tasks:
┌─────────────────────────────────────────────────────────────────┐
│ Task ID │ Status     │ Work              │ Verification │ Time  │
├─────────────────────────────────────────────────────────────────┤
│ 9784    │ ✓ DONE     │ Write blog post   │ PASS (5/5)   │ 8m    │
│ 9783    │ ✓ DONE     │ Deploy to prod    │ PASS (3/3)   │ 12m   │
│ 9782    │ ⚠ WARN    │ Fetch tweets      │ FAIL (2/3)   │ 6m    │
│ 9781    │ ✓ DONE     │ Update schedule   │ PASS (4/4)   │ 2m    │
│ 9780    │ ✗ FAILED   │ Sync database     │ N/A          │ 3m    │
└─────────────────────────────────────────────────────────────────┘

Task #9782 (WARN): 1 check failed
  ✓ API responded 200 OK
  ✓ 500 tweets fetched in expected time
  ✗ Engagement delta > threshold (12% increase flagged as anomaly)
    → Action: post flagged for review before sending

This dashboard is the opposite of the traditional monitoring we're used to. Instead of showing infrastructure metrics (CPU, disk, latency), it shows:

• Did the agent actually do what it claimed to do?
• Do the outputs look reasonable?
• Which tasks are silently wrong vs. loudly broken?

4. Operational Memory for Known Failure Modes

When we hit a new failure, we document it:

memory/
├── cases/
│   ├── blog-truncation-unclosed-blockquote.md
│   ├── db-migration-where-clause-typo.md
│   ├── git-stash-collision-parallel-agents.md
│   └── schema-drift-silent-insert-failure.md

Each case includes:

• What went wrong (with the actual data)
• Why the normal error signals missed it
• How we detect it now
• How to prevent it in the future

When a new agent starts a task, it can search this memory: "Have we had problems with blog post generation before?" and get back relevant lessons.

Debugging Techniques That Actually Work

Deterministic Replay

Every agent task carries:

• The exact state before (repo commit, database query results)
• The exact state after
• The agent's reasoning (via Claude API token logs)
• Every API call made during the work

If a task goes wrong, Matt can replay the exact sequence in a sandbox and watch where it diverged.

Diff-Based Verification

Before pushing code, we don't just check "does the build pass." We ask:

• What changed from the last working state?
• Does the diff match what the agent said it would do?
• Are there any files in the diff that the agent shouldn't have touched?

We've caught several bugs this way where an agent modified files by accident (auto-formatting, editor extensions, etc.).

Hypothesis-Driven Debugging

When something goes wrong, we frame it as a hypothesis:

Hypothesis: "The blog generation is silently dropping images"

Test:

find /tmp/blog-* -name "*.mdx" -exec sh -c '
  images=$(grep -o "!\[.*\]" "$1" | wc -l)
  if [ "$images" -lt 1 ]; then
    echo "Missing images: $1"
  fi
' _ {} \;

Result: 3 of the last 20 generated posts have zero images. Check the generation prompt.

This is faster than reading through logs and requires less domain knowledge.

The Real Impact

Here's what we learned by adding these verification layers:

Failure Type	Detection Before	Detection After	Stories Prevented
Silent truncation	18 hours	< 1 min	~200 posts published incomplete
Logic errors in migrations	2-3 hours (spot checks)	< 1 min (automated)	~5 major data corruptions
Silent failures in parallel agents	6-12 hours (human report)	< 1 min (dashboards)	~10 concurrent-work bugs
Out-of-date documentation	Never (rot)	Weekly (automated checks)	~30 confusion incidents

Cost: ~400 lines of verification code per agent. Time to implement: 2-3 days per agent type. ROI: ~10x. Every verification we added caught 3+ bugs within the first month.

What We Learned

1. Success signals are incomplete. A transaction commit and a correct program are not the same thing. You need verification.
2. Timing matters. The sooner you catch an error after execution, the easier it is to fix. Verification running in 1 second is better than human review 3 hours later.
3. Automate the verification, not just the work. The biggest wins came from automated checks that run after every task. Human review is too slow to catch everything.
4. Operational memory compounds. After 6 months of recording failure modes, new agents can learn from past mistakes. This is more valuable than code reviews for autonomous systems.
5. Silent failures are worse than loud ones. A crashing agent is easy to debug. An agent that silently corrupted data? We'd prefer it crashed.

What's Next

We're exploring:

• Speculative verification: Before an agent runs a database migration, simulate it in a transaction and verify the row counts match expectations
• Tripwire assertions: Agents insert lightweight assertions in code they write ("assert blog.word_count > 1000"), and we automatically run these after deployment
• Feedback loops: Track which verification checks are most likely to catch bugs and bias the agent toward running those checks more frequently
• Multi-agent verification: When agents run in parallel (e.g., content generation + social posting), have them verify each other's work

---

This post is part of Oliver's Lab, a series documenting how BeddaTech builds autonomous infrastructure. Previous posts: Oliver Architecture, Cost Breakdown, Support Automation, Running Local AI.

Next in the series: Building approval gates—how to give agents real autonomy without giving them a gun. Real examples: force-pushes, database deletes, credential rotation, and how Matt reviews each one in < 30 seconds.