The 97% Number: What Happened When We Automated Security Triage

We've automated security alert triage on engagements like this one. The headline number — 97% of alerts resolved without a human touching them — is the kind of thing that looks great on a slide deck.

Six weeks later, that automation nearly let a known deserialization vulnerability run undetected in production. Not because the system broke. Because a suppression rule outlived the context it was written for, and nobody noticed until a human happened to look.

That's the story. Not the 97%. The part where we almost trusted our own system too much.

A 97% automation rate isn't a flex — it's a claim about the 3% you're still catching.

(The company below is a composite — details anonymized and combined across engagements. The numbers are illustrative; the failure mode is real, and we've watched it happen.)

---

Why they called us

Picture a mid-market SaaS company. Modern security stack — a SIEM, dependency scanning, cloud audit logging — generating hundreds of alerts a day. Two senior engineers splitting time between security and platform work. They weren't triaging. They were scrolling.

A few months earlier, a legitimate credential exposure — a service account key committed to a public repo — sat in the queue for the better part of a week. Secret scanning caught it. The alert surfaced. And it drowned. The key got rotated only after an outside security team flagged it during a vendor assessment.

That's the moment a leadership team stops treating alert volume as a tooling inconvenience and starts treating it as a risk-management failure.

When your triage process is "scroll until something looks scary," you don't have a triage process.

---

What we built

We didn't deploy a product. We built a triage pipeline — suppression rules, enrichment logic, and a confidence-scored escalation path — designed around their specific environment.

The first layer handled the obvious noise: dependency advisories on dev-only packages with no production path, cloud-audit events from their own CI, monitoring anomalies triggered by weekly load tests. Every suppression rule had explicit match criteria, a rationale, an owner, and an expiration date.

The second layer was context stitching. A "suspicious login" that matches a known VPN egress IP. A "new IAM role" that correlates with an infrastructure-as-code change merged twenty minutes earlier. Alerts the system could resolve by looking at information that was available but never attached.

Everything else went through a confidence scorer — not an LLM, but a decision tree built on six months of their historical triage data. High confidence got escalated immediately. Low confidence went to a review queue checked twice daily.

The result: a manageable stream of alerts that two engineers could actually evaluate with judgment instead of reflexes.

Automation without evidence is just faster negligence.

---

The part that almost failed

Six weeks in, we caught something — by luck, not by process.

One of the suppression rules was filtering dependency advisories by package name — if it was a dev dependency, the alert got suppressed. Straightforward enough. Except three weeks earlier, a developer had promoted that library from dev to production. The manifest changed. The import graph changed. The suppression rule didn't.

For three weeks, alerts for a known deserialization vulnerability in a production dependency were being silently eaten by a rule that assumed the library would never leave dev. The system wasn't broken. It was working exactly as configured. The configuration just no longer matched reality.

We caught it because a human recognized a dependency name in the suppression log from a recent PR. Not because the system flagged it. Not because a test caught it.

Every suppression rule is a bet that the context it was written for hasn't changed.

This is the part of automation success stories that doesn't make vendor pitch decks. Suppression rules are security controls. They drift, go stale, and fail silently — always in the direction of permissiveness. Just like firewall rules, IAM policies, and allow-lists.

---

What we changed after

Three additions, all low-tech:

Suppression rules expire. Every rule gets a TTL — default ninety days. When it expires, the alert resurfaces until a human re-evaluates and re-authorizes the suppression.

Dependency promotion triggers review. When a dependency moves from dev to production, any suppression rules referencing it automatically flag for re-evaluation.

Weekly suppression audits. Not automated. Fifteen minutes of a human reviewing the suppression log, checking for pattern drift, and signing off. Fifteen minutes a week that protects the integrity of everything else.

The best thing an automated system can do is tell you when it doesn't know.

---

The ROI that actually matters

Here's the uncomfortable part of triage automation economics: the ROI that sells the project is not the ROI that sustains it.

The selling pitch multiplies alert volume by analyst cost per hour and produces a number large enough to justify anything. But those alerts were never getting analyst-hours — they were getting a fast scroll and a prayer. The real pre-automation cost wasn't money. It was a credential exposure sitting unnoticed for the better part of a week.

After automation, the value isn't "hours saved." It's two engineers who now spend their security time reviewing genuine events and investigating uncertainty — instead of drowning in noise and hoping pattern recognition holds up at 3pm on a Friday.

Automation doesn't save time — it saves judgment.

---

What this means for your team

If you're evaluating triage automation — vendor product, internal build, or an AI copilot that promises to "handle" your alerts — here's what we'd tell you after living with it:

Start with suppression, not intelligence. The highest-value automation is the dumbest: known-safe patterns with explicit match criteria. Get the obvious noise off the board before you touch anything fancy. If you can't write down why an alert class is safe to suppress, you can't automate it.

Treat every rule as a security control. It gets an owner, a rationale, an expiration date, and a review cycle. The moment you stop auditing your suppression rules is the moment they start degrading your security posture.

Measure decisions, not volume. A high automation rate with a six-day miss is worse than a lower rate with same-day response on every escalation. The number that matters is how long genuine events sit before a qualified human sees them.

Budget for the human layer. Automation changes what security engineers do — instead of triaging noise, they're auditing the automation's judgment. That's higher-value work, but it's still work, and it requires people who understand both the security domain and the automation itself.

If your automation strategy doesn't include a line item for humans auditing the automation, you don't have an automation strategy. You have a liability.

---

The uncomfortable conclusion

We built an automation system that works. Their security posture measurably improved. Their engineers make better decisions because they're not drowning in noise.

And the whole thing almost failed silently because one suppression rule outlived its context by three weeks.

That's the real 97% story. Not "machines replaced humans," but "automation created a new class of security control that needs its own governance, its own audit cycle, and its own failure-mode analysis." The number is real. So is the work required to keep it honest.

The number on the dashboard is never the whole story — it's in the logs nobody checks.

---

If your team is drowning in alert noise and considering automation — or if you've already automated and aren't sure what you're missing — we should talk. We've been on both sides of this and we know where the failure modes hide.