We Built the Grader Before We Built the Agent

Anyone can build a chat window that answers questions about invoices. Connect a language model to the data, ask, “What is our outstanding AR for this company?” and it’ll give you something. In a demonstration, it can look like magic.

The problem is the word something.

In accounts receivable, an answer that is close isn’t a smaller version of the right answer. It’s a wrong answer wearing a suit. If someone carries that number into a collections call, financial report, or board presentation, “approximately right” isn’t helpful. It’s a liability.

That’s the gap between an impressive demonstration and a tool finance can actually trust. When we began building a natural-language AR agent delivered through Microsoft Teams, we made one decision early that shaped everything that followed: we built the grader before we built the agent.

Grading Something That Didn’t Exist Yet

The usual order is to build the agent, demonstrate it, and wait for someone to ask, “But is it right?” By then, the project has momentum, expectations, and probably a launch date. Testing becomes an attempt to validate decisions already made.

We reversed the order.

Before configuring the agent, we created an evaluation harness with over 20 real AR questions across four areas finance regularly reviews: outstanding balances, aging, retainage, and customer-level detail. Each question was paired with a known answer drawn from historical CMiC reports that finance had already reviewed and reconciled.

AR was the planned scope from the beginning. The questions and categories were written before the agent was configured; we didn’t build a broad test, discard the failures, and call what remained the approved scope. AP, GL, and job cost were separate domains for later work, not questions removed to protect the score.

Those reports were our starting benchmark, not unquestionable truth. When different paths to the same answer disagreed, neither received the benefit of the doubt. We traced the source, determined which definition and field were correct, fixed the issue, and ran the tests again.

Our required pass rate was 100 percent, and the agent reached it across the predefined AR suite.

This gave us a scoreboard on day one. Every change could be tested against the same questions, so a fix that broke another answer was caught immediately.

The question, reporting period, expected result, and comparison rule all had to be defined before a pass or failure meant anything.

The Language Can Be Flexible. The Number Cannot.

We graded financial answers using strict numeric equality. Currency symbols, commas, and explanatory wording could vary, but the value had to match exactly. Partial credit or semantic similarity can make sense for written responses. It’s dangerous when the answer is a financial number.

During development, one retainage question produced two different eight-figure totals depending on the path used to retrieve it. Both looked believable. The harness failed the result and forced us to trace the discrepancy to a report section mapped to the wrong source field. The CMiC detail behind the post-date report established the correct retainage mapping, and we corrected the other path.

The harness didn’t merely confirm the historical report. It gave us a way to challenge, correct, and then preserve the business definition.

Strict grading means the agent can’t pass by being convincing. It has to be correct.

Pinning the Answer to a Point in Time

AR changes every day, so an answer key without a date is useless. Each evaluation was tied to the post date used by the corresponding historical report. The expected answer and the agent’s answer were compared using the same accounting cutoff, which let us rerun the test without comparing two different periods.

Not every answer was a single number. A question such as “Which customers are more than 90 days past due?” returns a list. We compared those results against the fixed CMiC detail behind the same historical post-date report, checking exact customer membership and associated values. Matching only the grand total wasn’t enough.

Why the Agent Answers From a Lakehouse

Our ERP returns transactional rows. It doesn’t always calculate the requested business metric, translate internal codes, or apply the same rules as the financial report. Some endpoints also used pagination that wasn’t reliably ordered. If records moved between pages or a page was missed, two pulls could produce different totals.

You can’t build dependable regression tests on a source that doesn’t reliably return the same dataset twice.

The agent therefore answers from a curated Gold layer in Microsoft Fabric rather than querying the ERP directly. That layer contains the definitions needed to reproduce the report: accepted aging logic, resolved company codes, correct precision, explicit intercompany identification, and defined treatment of voided invoices.

The agent translates the question into a query. The Gold layer supplies the governed data and business rules. We log the question, generated query, and answer so every number can be reproduced and traced.

The Work Nobody Sees in the Demo

Making the pipeline trustworthy was the least visible and most consequential part of the project. We diagnosed an undocumented ERP API throttle that caused ingestion runs to hang for hours. A caching rewrite reduced one path from roughly 6,600 API calls per run to about 2,500. Backoff and retry logic stopped partial pulls from silently skipping data, while multi-pass fetching and deduplication replaced unstable pagination.

None of that looks as impressive as an immediate answer in Teams. It’s still the work that determines whether the answer deserves to be believed.

Most of “building the AI” was data engineering: stabilizing ingestion, defining the business rules, and proving that the same question produced the same answer. The agent was the last mile.

Knowing When Not to Answer

The first release was deliberately limited to AR, where the agent had passed the complete evaluation suite. Future domains will need their own predefined questions and the same standard before they’re added.

If someone asks a question outside the available data or established business definitions, the agent declines instead of filling the gap with an assumption. That isn’t a weakness. A confident wrong number can damage more credibility than an honest refusal.

The result shipped in August, four months ahead of its December target, without pretending it could answer everything. It’s now in an early pilot, with users asking supported AR questions through Teams while we learn where the next useful boundaries should be.

The chat window was the easy part. Everything that made the chat window trustworthy was the actual product.