Oliver's Lab: The Analytics Outage Nobody Noticed for 31 Days
Target: engineers/founders running their own infra | Publish: bedda.tech/blog | Series: Oliver's Lab
On September 2nd, one of our own automation agents repointed a Tailscale port to share an APK file. That port happened to be the one our analytics ingress used. For the next 31 days, every tracker request from our production apps 404'd silently. Umami recorded zero traffic data for a month, and nothing paged anyone, because every health check we had was checking the wrong thing.
Here is what happened, why our monitoring missed it, and what we changed.
The setup
Our analytics stack runs self-hosted Umami behind a Tailscale ingress at a fixed port. Two production apps (a marketplace app and its landing page) proxy their tracking script and event calls through that port via a Next.js rewrite rule. It's a small, boring piece of infrastructure that had been stable for months, which is exactly why nobody was watching it closely.
On 2026-09-02 at 11:30 UTC, an agent session needed to share a large APK file and grabbed that same port to serve it, overwriting the Tailscale serve config that pointed to Umami. The rebind succeeded. From that second on, every /a/script.js and /a/api/send request from either app hit a directory of APKs instead of the analytics server and came back 404.
Why three separate checks all said "fine"
This is the part worth dwelling on, because each failure was individually reasonable.
Liveness checked the wrong layer. Our internal services dashboard probed the Umami container directly on its own port, and that container was healthy and answering 200s the entire time. The container was never down. The route to the container was gone. Liveness and correctness are different questions, and we were only asking the first one.
The alert that did exist couldn't deliver. We did have an hourly healthcheck that correctly flagged the data as stale, every single hour, for 31 days straight. But the task it filed to alert a human was missing a required field, so the API rejected it with a 400 and the alert was silently dropped. The check worked. The escalation path it depended on did not.
The manual fallback link was dead on arrival. The one remaining way a human might have caught this, clicking through from the dashboard, pointed at a plain HTTP URL on the analytics port. Tailscale's .ts.net domains are HSTS-preloaded, so every modern browser forces HTTPS on that domain regardless of what the link says, and nothing was serving TLS on that port. It never worked.
Three checks, three different blind spots, one month of missing data.
What fixed it
The repoint itself was a one-line fix once we found it: point the port back at the real Umami ingress, move the APK sharing to its own port, done. First real event landed within two minutes of the fix. The actual work was making sure "done" meant something more durable than "it's fixed for now."
What changed
- Port registry + a hard guard. Every Tailscale port we use is now in a tracked registry. A pre-tool-use hook blocks any agent session from rebinding, disabling, or funneling a port that is already registered as live. The exact action that caused this now fails before it runs. A new service gets a new free port and gets registered; changing an existing port is a decision that routes to a human, not an agent's judgment call.
- The healthcheck now probes the real path. It no longer asks "is the container up." It asks "did a real event actually arrive through the real public URL in the last two hours." That question would have caught this on day one.
- Alert delivery got a guaranteed path, not a best-effort one. The escalation on a failed check now forces through regardless of normal priority caps, re-pages on a fixed interval (every six hours), and pages again if the escalation mechanism itself fails to fire. No single dropped field silently eats an alert again.
- The underlying rule got written down and generalized: observability and analytics outages are now treated as emergencies, the same severity class as a production outage, not a "fix when convenient" ticket, because by the time you notice a data gap by eye, you've usually already lost the data.
The lesson has little to do with Tailscale or Umami. "The service responds" and "the service is doing its job" are two different claims. A monitor that only checks the first will report everything fine until someone needs the data it was not collecting.