The Bug That Reported Success

This article was originally published on LinkedIn and is archived here on texlytics.com.

One of our customers didn’t exist.

Not deleted, not archived — simply absent. A real company, with real invoices, that should have been sitting in our analytics layer like everyone else. It wasn’t. And the system responsible for bringing it in had run cleanly every night for weeks. Green checkmark after green checkmark. No errors, no warnings, no failed jobs. The whole time, it was quietly wrong.

This is a story about why the bugs that hurt are almost never the ones that crash.

The false comfort of a clean run

Most of the data world runs on incremental loads. You don’t recopy your entire source system every night — that’s slow, expensive, and pointless when 99% of it hasn’t changed. Instead you ask a simpler question: what changed since last time? You keep a marker — a watermark — usually a “last updated” timestamp. Each run, you grab everything newer than your marker, load it, and move the marker forward.

It’s elegant. It’s efficient. And it has a quiet assumption sitting right at the center of it: that the timestamp moves whenever the data changes.

When that assumption holds, incremental loads are wonderful. When it doesn’t, they fail in the worst possible way — silently. Nothing errors. The job succeeds. The marker advances. And a row you needed slips through the gap, invisibly, because as far as the pipeline can tell, nothing happened.

The hunt

Tracking down a missing record is its own special frustration, because there’s no stack trace pointing at the problem. The system isn’t broken. It did exactly what you told it to. You’re not debugging code — you’re debugging an assumption.

So you start asking uncomfortable questions. Is the record actually in the source? Yes. Did our pipeline ever pull it? No. Why not? Because it was never “new.” And that last answer is the one that stops you cold — because the record was obviously new to us. We’d never seen it. How could the system think otherwise?

The answer was in how the record came to exist.

The mechanism

In a lot of business systems, the thing you care about isn’t a standalone object — it’s a role layered onto something that already exists. A “customer” might really be a general business-partner record that someone flagged as a customer at some point. The customer relationship was brand new. The underlying record was old.

And the timestamp we were watching belonged to the old record.

So when the customer relationship was created, the field we trusted to say “this changed” never moved. The data changed. The signal didn’t. Our watermark wasn’t broken — it was faithfully, obediently doing exactly what we asked, watching a field that simply wasn’t telling the truth about this particular kind of change.

That’s the whole bug. Not a crash. Not a typo. A perfectly reasonable assumption that happened to be wrong in one specific, invisible way.

The fix

The immediate problem was easy to state and faintly embarrassing to act on: go get the record we’d missed. A targeted backfill pulled the customer back in, along with anything else hiding behind the same pattern. One-time cleanup, done.

The structural fix was the part that mattered, and it came in two layers.

First, we stopped trusting a single timestamp. Instead of watching only the business-partner record’s last-updated field, the watermark now takes the latest of several candidate signals — including the one that actually moves when a role like “customer” gets attached to a partner that already exists. If any of them changes, the record is in scope. The pipeline no longer assumes one field tells the whole truth about change; it asks all of them and trusts the most recent.

Second, we added a reconciliation pass — the part I’d keep even if the first fix were flawless. On a schedule, it counts what exists in the CMiC source against what we’ve actually landed in the Fabric lakehouse, and when the two drift apart it flags the gap and pulls the difference. It isn’t glamorous. It’s a safety net that assumes the incremental logic will eventually be wrong about something, and catches it before a stakeholder does.

The first fix closes the specific hole. The second accepts that there will always be another hole you haven’t found yet — which, conveniently, is where the lesson starts.

The lesson that travels

Here’s the part that has nothing to do with my particular stack and everything to do with anyone who has ever trusted a system to keep itself up to date:

An incremental process is only as honest as the signal you trust to move.

Full loads are dumb but honest — they recheck everything, so they can’t quietly miss something. Incremental loads are smart but trusting — they’re fast precisely because they don’t recheck, which means a bad assumption never announces itself. The efficiency and the blind spot are the same feature.

So the defenses aren’t clever. They’re mostly cultural:

Treat “zero errors” as a question, not an answer. A clean run tells you nothing failed loudly. It says nothing about what failed quietly.

Reconcile against the source periodically. Count the rows in the source, count the rows you loaded, and watch for drift. You want to find your blind spot before a stakeholder does.

Know exactly which signal you’re trusting — and ask what kinds of change can happen without moving it. That one question would have caught this months earlier.

The reframe

We tend to picture failure as something loud: the red alert, the 3 a.m. page, the crash. But the failures that actually cost you are the calm ones. The job that succeeds while doing the wrong thing. The number that’s confidently incorrect. The customer who isn’t there, in a report that swears everything is fine. Build for the silent failures. The loud ones can take care of themselves.