Anomaly Detection at Line Scale

The problem

Manufacturing lines produce an enormous amount of process data and almost none of it is looked at. The classic response is a dashboard nobody opens, or fixed thresholds that either scream constantly or never fire at all.

The real question is narrower and harder: which of these signals, right now, is behaving in a way that predicts a problem? Answering it at a volume of up to five million datapoints per day, with low enough latency to matter, is as much an engineering problem as a modelling one.

Data

• High-frequency process signals from sensors and machine controllers across multiple lines
• Heterogeneous sampling rates, gaps, clock drift and occasional sensor dropouts — all of it normal, none of it clean
• Contextual data: part type, batch, tool state, machine, shift
• Ground truth that is sparse, delayed and partially subjective, which is the defining constraint of the whole project

The evaluation framework

The core of this project is not a model. It is the framework that decides which model is worth having.

Standard anomaly detection metrics answer the wrong question. Point-wise F1 rewards a detector that fires on every sample of a long event and punishes one that fires once, early — which is exactly backwards for production, where early and once is the ideal behaviour. So I designed metrics around what the plant actually cares about:

• How early is the event detected, relative to when it became visible to a human?
• How many separate false alarms per week does an operator have to absorb?
• Is a detected event actionable, or does it arrive after the damage is done?

With those metrics defined, the search became meaningful: 48 000 model variants across algorithm families, feature sets, window lengths, preprocessing choices and thresholds, evaluated systematically rather than by intuition.

   raw signals ─→ windowing / features ─→ candidate models ─→ custom metrics
                                                                  │
                                        48 000 variants ranked ───┘

The result that surprised me most: several simple, cheap methods with good features outperformed heavier models with naive ones. That is not a novel insight in the abstract, but having 48 000 measured comparisons makes it an argument you can actually win in a meeting.

Deployment

• Streaming ingestion and scoring, engineered for the full daily volume rather than a sampled subset
• Alerts routed to the people who can act on them, with the driving signal attached
• Model and feature versions pinned per deployment, every score reproducible after the fact
• Drift monitoring on inputs, plus a periodic re-run of the evaluation framework as ground truth accumulates

What I would do differently

I would spend even more time upfront with the people who receive the alerts. Every hour spent defining what counts as a useful detection paid for itself many times over in the search — and the projects around me that skipped that step produced models that scored well and were ignored in practice.