Are Your Alerts Actually Shit? AI SOC Reality With Dennis Chow

Alex (00:03)

Dennis… I’ve been hearing “AI SOC” like it’s a magic spell everywhere I go.
Everyone thinks it’s going to clean up their alert queues, fix their detections, save their analysts.
But none of that matters if the alerts themselves are trash.

And today, we’re breaking that open — for real.
Joining me is Dennis Chow, Director of SecOps Engineering at UKG and co-author of Automating Security Detection Engineering.
Dennis, what have you been building?

Dennis Chow (01:20)

We’re aiming for full L1/L2 automation.
Multi-agent pipeline, running on GCP, clustered, orchestrated, context-aware.
We’re getting 7-minute end-to-end investigations and scaling alert consumption per analyst by a massive factor.
We’re basically replacing the low-level ticket grinding.

Alert Quality: Are Your Alerts Actually Shit?

Spoiler: Yes.

Alex (02:20)

Let’s just say it: most alerts are garbage.
The precision is off, the recall is off, the triage notes aren’t helpful, the logic is brittle.
You can’t build an AI SOC on top of garbage.

Are your alerts at UKG where you want them?

Dennis (02:50)

To my standards? They still need work.
Indicator-based detections break way too easily.
Payloads mutate. Regex dies.
Everybody says “behavioral detections,” but almost nobody has the engineering for it.

That’s why the alerts feel unmanageable — not because the SOC is bad, but because detections are shallow.

When Does Alert Chaos Become Manageable?

The “automation readiness” moment.

Alex (04:10)

There’s a point where alert chaos becomes… containable.
Where you have enough signal that automation won’t just accelerate bad decisions.
When does that happen?

Dennis (04:32)

When you can hit something like 50–70% high-fidelity alerts, you’re in the zone.
If you can’t measure TP/FP/Benign Positive, you’re not ready.
If you don’t have data normalization, visibility, or correlation, you’re definitely not ready.

AI doesn’t fix coverage gaps. It just makes them louder.

What Dennis Has Actually Automated

This is the part everyone wants to copy.

Dennis (05:50)

Here’s what we’ve automated safely:

• L1 triage
• Most of L2 investigation
• Context gathering
• Enrichment
• Scoring
• Narrative building
• Recommendations

All in 7 minutes, at scale.
Costs us roughly $2–3k/month.
We freed analysts for DFIR, reverse engineering, and hunting.

The Skill Gap Nobody Likes Admitting

SOC work now demands 10 jobs in one.

Alex (07:12)

I feel like part of this is skill pressure. SOC analysts get expected to know cloud, endpoint, scripting, ETL, ML, identity, threat intel…

Dennis (07:22)

Exactly.
“Entry-level SOC” doesn’t exist anymore.
AI becomes the crutch because people don’t have enough detection engineering capability in-house.
That’s how you end up with thousands of low-value alerts.

AI SOC Vendors: GPT Wrappers vs Real Platforms

Not all AI SOCs are created equal.

Alex (08:50)

Let’s talk vendors. Because some are real, and some are… UI with a temperature setting.

Dennis (09:01)

Yep. A lot of these products are just GPT triage wrappers.
Pretty summaries. Nice colors.
But no evals, no memory control, no context management, no tool orchestration.

If the vendor can’t talk about those four things?
It’s not an AI SOC.

Two Paths to Fixing Bad Detections

You don’t get AI magic until you fix detection debt.

Dennis (10:33)

Two real options:

1 — Rewrite detections as behavioral logic

Hard, long, engineering-heavy.

2 — Correlate signals before alerting

Use informational events, ML clustering, risk scoring, composite detections.
Most orgs should start here.

Where ML Actually Works

Identity and deception signals are still gold.

Dennis (12:15)

ML still shines on:

• Identity CRUD anomalies
• Authorization paths
• Canary token triggers
• Detecting agent drift in multi-agent systems

UBA baselines? Mostly dead.
Identity patterns? Very alive.

Metrics That Matter Before You Automate

This is the real checklist.

Dennis (13:42)

If you don’t measure:

• Signal-to-noise
• TP/FP/BP ratios
• Deprecation cycles
• Threat-informed coverage
• BAS validation
• Analyst efficiency

…then you’re not ready for AI-driven triage.

RAG Isn’t the Savior

When you don’t have enough true positives, RAG hallucinates confidence.

Alex (15:20)

There aren’t enough true positives in half of these scenarios to train a RAG system.

Dennis (15:30)

Exactly.
Some detection types only have one TP a year.
You can’t build RAG on that.
Unless you build the entire pipeline yourself, you don’t have control anyway.

When to Build vs When to Buy

Practical, not religious.

Alex (17:00)

So when should people build, and when should they buy?

Dennis (17:15)

Build when:

• You have real engineers
• You need control
• You can manage context + memory + orchestration

Buy when:

• You lack engineering bandwidth
• You want enterprise models
• You need fast deployment

Just don’t buy a GPT wrapper.

The Future SOC Analyst

Specialize or get automated.

Dennis (18:50)

Entry-level triage disappears.
AI handles that.
The roles that remain are:

• Detection engineering
• DFIR
• Threat intel
• Reverse engineering
• Research

If you stay generalist, AI will replace you.

Closing

Alex (20:40)

If your alerts are shit, AI won’t save you.
Fix the detection fundamentals.
Then automate — not before.

Dennis (21:05)

Thanks for having me. Always ready to stir the pot.

‍