How I Solved CMiC’s 429 Rate Limit Challenge

• Gravatar

This article was originally published on LinkedIn and is archived here on texlytics.com.

If you have ever built an integration against the CMiC Cloud APIs, you already know where this story is going. HTTP 429. Too Many Requests. Over and over.

I pull CMiC ERP data into our Microsoft Fabric Lakehouse on a schedule, Bronze through Gold. The pipeline ran clean for months. Then one morning I walked in to find it already in full meltdown, runs failing on throttling across the board. Nothing had changed on our side. No new endpoints, no extra load, just business as usual the day before. And it was not just our pipeline. Failed runs meant the data I push into QuickBase came through incomplete, and every report built on it inherited the problem. I had about a week to make the numbers trustworthy again. If you work with APIs, you have probably lived some version of this.

Know the limits, including the ones nobody wrote down

Here is what the CMiC docs tell you: 5 concurrent connections per IP, 15 requests per second. Reasonable. Plannable. Except those numbers are not the whole story. Sitting on top of the published limits is an undocumented nginx-level cumulative limit that only shows itself under sustained load. You will not find it in any spec. I found it the way everyone finds it: in production, at the worst possible time. Rate limit awareness has to start with the real ceiling, not the documented one.

Here is the part that reframed everything. I was told CMiC determined that the limit was effectively shared. Our instance is on a multi-tenant cloud instance, and the ceiling I kept slamming into was not really ours alone. A handful of other clients running heavy, unregulated API jobs were consuming the common headroom and grinding the rest of us to a halt. A classic noisy-neighbor problem. Picture an apartment building where everyone shares the same water pressure: when a few units run every faucet and sprinkler at once, everybody else is left with a trickle. The uncomfortable truth is that I could not fix the neighbors. I could only control our own side of the wire. That is exactly why everything that follows is about pacing and resilience, not about asking for a bigger number.

Handle the requests before you ever need to retry

The mistake is treating this as a retry problem. Retry is the safety net. The real fix is pacing the work so you rarely fall into the net at all. I implemented three things that did the heavy lifting:

Stay inside the envelope on purpose. Concurrency is now capped to the 5-connection ceiling, down from the 15 I was running, and requests are paced under the 15-per-second line, rather than firing everything the moment a run starts and praying. Slower on paper, faster in practice, because you stop paying the penalty of rejected calls. Think of merging onto a packed highway: flooring it just causes a wreck, while easing into the flow of traffic actually gets you there sooner.

Queue the heavy endpoints instead of fanning out blindly. Some CMiC endpoints only answer through per-record finders, which means hundreds of calls in a single run. Those get their own pacing and their own slot window so one greedy endpoint cannot starve the rest of the schedule. It is the grocery store express lane: Open a separate checkout for the one cart piled high, so everyone with a basket still flies through the express lane.

Throttle by the clock, not just by the error. A schedule guard checks the active window in Central time (for our environment) before any CMiC connection is even opened. If a run does not belong in this slot, it never dials out. Dynamic throttling that is proactive beats throttling that only kicks in after the server has already said no. Same idea as running the dishwasher overnight instead of during the dinner-hour rush: you do the heavy work when the system has room for it.

Then make the retry layer actually intelligent

My first instinct was the same one everybody has. Catch the 429, sleep, retry. That is not a solution, that is a louder failure. A blind retry loop hammers an already-throttled API harder, and it spends your retry attempts on the wrong failures. So I tore it out and rebuilt the resilience layer properly.

Honor “Retry-After.” When CMiC tells you how long to back off, that header is ground truth. It beats your own backoff math every single time, because the server is telling you exactly when it will be ready. Reach for it first, fall back to computed delay only when it is absent. If a busy restaurant says the wait is twenty minutes, you wait twenty minutes. You do not walk up to the host every thirty seconds asking if the table is ready.

Capped exponential backoff with jitter. The fallback delay is roughly min(cap, base * 2^attempt) plus a randomized jitter term. In plain language, that formula means each retry waits about double the one before it (say one second, then two, then four, then eight), up to a hard ceiling you set in advance. The cap matters because uncapped exponential backoff turns a 30-second blip into a 10-minute stall. The jitter matters more than it looks: without it, a fleet of parallel calls all wake up and retry on the exact same tick, and you have just rebuilt the thundering herd you were trying to escape. And stagger the timing so a whole crowd is not knocking on the same door at the same second.

Separate retry budgets per failure class. This was the real unlock. A transient 503 and a genuine 429 are not the same event and should not draw from the same pool. If they share one budget, a burst of throttling exhausts the attempts you needed for an unrelated blip, and vice versa. Splitting the budgets meant each failure mode gets handled on its own terms. Think of it as separate envelopes of cash for rent and for groceries: a bad week of grocery spending should never leave you unable to make rent.

Cover the whole failure surface, not just the obvious one. CMiC sits on Oracle, and under pressure it does not only return clean HTTP codes. The layer handles 429, 500, 502, 503, and 504, plus the Oracle errors that leak through when the backend is strained, ORA-06511 and ORA-01013. If you only catch the 429, the others will find you. It is the difference between locking the front door and locking every door and window: whatever you leave open is exactly where trouble gets in.

The same resilience logic now lives in both runtime contexts, the standalone Python scripts and the Fabric incremental notebook, so the pipeline behaves identically whether it runs as a manual job or inside Fabric. One mental model, not two.

What this adds up to

Resilient integration. The pipeline is built to bend, not break. Capped concurrency, intelligent retries, and a failure surface that covers HTTP and Oracle errors alike mean a bad minute on the vendor side is a pause, not an outage. It is built for scale because the next ten endpoints will not change the model, only the load.

Optimized throughput. The goal was never raw speed, it was moving the most data I can while staying inside CMiC’s limits. Pacing under the line genuinely moves more data per run than firing blindly and eating rejections. Maximum flow within the limits, not against them.

Reduced errors. The 429 storms that used to define our morning runs are gone. Runs that failed outright now absorb the backpressure and finish, and nobody is restarting a dead notebook by hand. Goodbye 429s, hello reliability.

Business impact. All of that exists for one reason: clean, on-time data landing in the silver layer and flowing out to QuickBase, the Power BI dashboards, and analytics the business actually runs on. A flaky feed becomes a wrong number in a report someone is deciding from. Reliable ingestion is what lets people trust the numbers, on time.

What I would tell my past self

• Pace first, retry second. The cheapest 429 is the one you never trigger.
• Trust the docs, but verify the limits. Vendors publish the happy path. Production owns the rest.
• “Retry-After” is a gift. Use the server’s answer before you invent your own.
• Budget retries by failure type. One shared pool is a trap.
• Catch more than 429. The errors you ignore are the ones that page you at 2 a.m.

If you have fought the CMiC APIs and come out the other side, I would genuinely like to hear how you handled the throttling. We are all comparing notes on the same opponent.