Building AI for Medical Device Alarm Fatigue — And Managing What Actually Matters

The Problem No One Talks About

In modern hospitals, alarms are everywhere. Monitors beep constantly — but most of them don’t matter. Over time, clinicians become desensitized, a phenomenon known as alarm fatigue.

The real risk isn’t noise — it’s missing the one alarm that actually signals danger.

Solving this with AI sounds straightforward: train a model to distinguish real alerts from false positives. In reality, it’s anything but simple.

You’re dealing with:

Real-time telemetry from multiple device types
Inconsistent data formats across manufacturers
Strict requirements around PHI (Protected Health Information)
Regulatory expectations from the FDA for AI/ML systems

This isn’t just a machine learning problem. It’s a coordination problem between engineering, data, and compliance — where mistakes have real-world consequences.

My Role: Bridging Engineering and Compliance

As both Delivery Manager and Data Engineer on this project, my responsibility went beyond building pipelines or tracking tasks.

I needed a way to:

Manage data engineering work (Snowflake ingestion, schema design, model training)
Track compliance requirements (audit trails, de-identification, FDA documentation)
Keep everything aligned in one system without conflicts or blind spots

Most teams separate these into parallel tracks. That’s where things break.

Why We Chose Linear

We needed a system where priorities were explicit and “done” was not subjective.

Linear stood out for a few reasons:

Clear issue hierarchy and prioritization
Structured issue templates with acceptance criteria
Fast, minimal interface that teams actually use

Each issue wasn’t just a task — it was a complete unit of work:

Context and overview
Known issues
Reproduction steps
Acceptance criteria (as checkboxes)

This made expectations unambiguous across engineering, data, and compliance teams.

Designing the Backlog: One Queue, Not Two

We structured the Healthcare Systems (HEA) backlog as a single prioritized list — not separate engineering and compliance boards.

This decision mattered.

Here’s what sat in the backlog:

Data drift detection issues
PHI de-identification tasks
FDA audit trail requirements
Data ingestion pipelines
Model training work

Only 8 issues at a time — intentionally constrained.

Bugs surfaced above features. Urgent items were clearly marked with defined impact.

Because in a regulated system, a “small” bug can be more critical than a “major” feature.

The Issue That Changed Priorities

One issue stood out: HEA-11 — Data Drift from New Firmware

A ventilator firmware update introduced new alarm codes (9000–9999) that the model had never seen.

On paper, this looks like a data issue.

In reality, it meant:

The model would silently fail on ~12% of real-world alarms
No obvious errors — just degraded performance
A direct patient safety risk

This wasn’t something to fix “later.” It became the top priority.

What the Issue Actually Contained

Instead of a vague bug ticket, the issue was structured and actionable.

Known Problems

New firmware alarm codes missing from training data
Unit inconsistencies across devices (ml/hr vs ml/min)
Missing metadata for ~12% of patient records

Reproduction Steps

Run inference on recent production data
Compare results against training baseline
Analyze drift using MLflow

Acceptance Criteria

Root cause identified per device class
Retraining triggered with updated datasets
Drift monitoring alerts configured (PSI threshold)
Mapping tables updated for unit/code consistency

No ambiguity. No assumptions.

A Key Decision: Bugs Above Features

In many teams, feature development continues while bugs are “tracked.”

That doesn’t work in healthcare AI.

I made the call to prioritize:

Data drift (HEA-11)
PHI de-identification (HEA-6)
FDA audit trail & PCCP documentation (HEA-10)

All as Urgent, in the same queue.

This meant:

Model training (HEA-9) was blocked
No new features shipped on unstable foundations
Compliance wasn’t treated as “documentation later”

It forced the team to address risk before progress.

Acceptance Criteria as a Contract

One thing that made the system work: every issue had explicit acceptance criteria.

Not vague definitions like “pipeline complete” or “model ready.”

But clear, testable conditions:

Alerts configured
Data updated
Documentation completed

In a regulated environment, this does two things:

Acts as a QA checklist
Becomes audit-ready documentation

“Done” is no longer a discussion — it’s verifiable.

The Outcome

By keeping engineering and compliance in the same prioritized system:

We avoided silent model failures
Reduced risk of regulatory gaps
Improved cross-team clarity
Prevented feature delivery on unstable foundations

More importantly, the team always knew what mattered most — and why.

Final Thought

Building AI for healthcare isn’t just about models.

It’s about managing risk, alignment, and accountability at every step.

The real challenge isn’t training the model.
It’s making sure nothing important gets missed while you do.

Stack

Linear · Snowflake · Azure Data Factory · Python / Snowpark · MLflow · FDA SaMD Framework