Stop Writing One Pipeline Per Source

This article was originally published on LinkedIn and is archived here on texlytics.com.

Every data engineer hits this moment. You’ve integrated three systems, you’re proud of it, and then a fourth source shows up. So you do what you did the last three times: copy the previous notebook, find-and-replace the table names, tweak the date logic, adjust the pagination, ship it. It works. You now have four notebooks that are 80% identical and 20% subtly different — and that 20% is where every future bug will live.

Then comes the fifth source. And the sixth. And one day you go to fix a single piece of retry logic and realize you have to fix it in a dozen places, and you will miss one.

I lived in that world for a while. I decided to change things up from the way I USED to do things. It was time for something new, unconventional from my world.

The reframe

The insight is almost embarrassingly simple: the differences between my sources weren’t logic differences. They were parameter differences. This endpoint paginates by date; that one by a unique key. This one I can pull incrementally; that one I have to snapshot in full. This one needs a special lookup before it will filter; that one doesn’t. None of that requires different code. It requires the same code, told different things.

So the question flipped. Instead of “how do I write a pipeline for this source?” it became “what does my pipeline need to know about a source in order to ingest it?” Write that list down, put it in a table, give it one column per decision. Now the notebook doesn’t contain knowledge about any specific source. It contains the machinery. The knowledge lives in the table.

In my world that table is a single control table — an endpoint registry — and every row is one source. The notebook loops the rows, reads each one’s configuration, and does what the row tells it to. Adding a new source stops being a coding task. It becomes a new row.

The most important design decision

If I could hand you only one thing from this pattern, it’s this: split your configuration columns into two kinds, and never blur the line.

Some columns are things a human sets and owns — the source name, the mode (incremental or snapshot), the key fields, the date boundaries, whether a special lookup applies. In my schema these all carry a UF_ prefix: user-managed (field). They’re the contract. A teammate can open the table, read a row, and understand exactly how that source will be treated without reading a line of Python.

The other columns are things the system sets and owns — the last watermark, the timestamps of the last successful fetch, the run state. These carry an SF_ prefix: system-managed (field). Humans don’t touch them; the notebook writes them as it runs.

That one boundary — who owns this column, the human or the machine — is what keeps a config-driven system from rotting. The moment someone hand-edits a system-managed watermark “just to fix something,” you’ve lost the guarantee that the table reflects reality. Keeping the two halves visually and conceptually separate means anyone can glance at a row and instantly know which part is the instruction and which part is the receipt.

How one notebook handles sources that look nothing alike

The fear with this pattern is that you’ll end up with a notebook full of branching — that “one notebook” is really twelve notebooks wearing a trench coat. The way out is to find the small number of shapes your sources actually come in, and branch on those, not on the sources themselves.

For me it came down to a handful of scenarios defined by what each endpoint accepts: a unique key plus a date range, a key only, dates only, or neither. Four shapes covered everything. The notebook asks the row which shape it is and builds the request accordingly. Incremental versus snapshot is one column. Whether to apply a lookup — and I only wire one up when it’s actually confirmed in the endpoint’s documentation — is another. The branching is real, but it’s bounded: it tracks the handful of ways sources can differ, not the unbounded list of sources.

And because everything routes through one path, I get one place to test. A single override lets me point the whole notebook at exactly one source and run it in isolation, so I can validate a new row without disturbing the ones that already work.

Where it bites — because it does

This pattern is not free, and I’d distrust anyone who sells it as free.

You trade code complexity for configuration complexity. The logic gets simpler; the table gets more load-bearing. That table is now critical infrastructure — if a row is wrong, the failure can be quieter and harder to trace than a bug in named code, because nothing looks broken. The machine just faithfully did what the row said. Debugging gains a layer of indirection: the answer is rarely “the code is wrong” and usually “the configuration is wrong,” and you have to retrain your instincts to look there first.

And some sources genuinely don’t fit. There’s always one endpoint weird enough that forcing it into the shared shapes costs more than just writing it bespoke. Knowing which sources belong in the registry and which earn their own code is the actual skill. Config-driven is a default, not a religion.

The part that isn’t about Fabric

Strip away my stack and the principle is portable to almost any integration work: configuration is data, not code. The things that vary between similar jobs belong in a table you can read, not scattered across files you have to diff. Separate what humans own from what the system owns. Find the shapes, not the instances.

I didn’t invent this — table-driven and registry patterns are old. But a lot of teams reinvent it badly, usually by letting the config and the code bleed into each other until neither is trustworthy. The discipline was never the table. It’s the boundaries around it.