Imagine a review queue filling with drafted replies, extracted fields, and proposed actions. Generating the next item is easy. Deciding whether it should be approved takes more care.
I want to build interfaces that help with that decision. Showing every field with the same emphasis leaves the reviewer to discover which details matter. A source conflict can sit beside a routine label and look equally ordinary.
The design question is where to ask for attention, what evidence to put there, and how to tell whether the interface actually helps.
Not every check deserves the same attention
Consider an illustrative supplier request with a long form. A uniform review screen gives the company name, copied from an existing record, the same visual weight as a start date inferred from "early next month". The interface does little to explain which value needs a decision.
Those values have different evidence behind them. Even the copied name needs to belong to the right supplier, but the date has an additional problem: the source never specified it.
I would put guesses, conflicting sources, and missing required checks at the top. Lower-risk fields with supporting evidence can be grouped, while remaining available to inspect. A matching quotation proves the words exist; it does not prove the value is correct or that the source is current.
The intended benefit is to spend less effort finding the problem and more effort deciding what to do about it. That needs to be tested: measure review time and missed errors together. A faster approval is not an improvement if reviewers overlook more mistakes.
Route by what a mistake costs, not by confidence alone
The usual way to decide what a person must check is a confidence threshold. If the model is 90% sure, let it through. If it is less sure, ask a human.
Confidence alone is a bad router, for two reasons. First, models are often confidently wrong, especially when a source is stale. Second, and more important, a 95% sure guess about an email tag and a 95% sure guess about a supplier's bank account are not the same thing. The first mistake costs a minute. The second can cost real money and a very bad week.
What works better is to look at two things together: how unsure the model is, and what it costs if it is wrong.
In practice this turns into a short list of rules, written down where everyone can read them:
- Low impact, low uncertainty: do it and log it. For example, applying a reversible internal label under a tested policy. A matching quotation alone is not sufficient.
- Low impact, more uncertainty: do it, but put a sample in front of a person every week.
- Medium impact: propose it. A person approves with one click.
- High impact: a person decides. The AI prepares everything the person needs, but never acts on its own.
The interesting part is that these rules are about meaning, not about code paths or confidence numbers. "A new supplier outside our usual country goes to a person." "Any field where somebody wrote 'other' goes to a person." "Everything else can be drafted automatically." A manager can read that and argue with it. A threshold alone does not explain which consequence the business is willing to accept.
Review the intent, not the diff
Engineers know this problem from code review. When a reviewer gets a two thousand line diff, they skim it. When they get two lines with a clear sentence about why, they actually think.
For business work, I would start each review with the request, what is unusual, and what needs a decision. For example: "New supplier, standard payment terms, start date inferred from the email, bank details differ between two sources." The summary guides attention; the underlying fields and source evidence remain accessible.
That is the experience I want from an assistant: preparation that leads to a clear question. The summary itself also needs checking, because a model can omit the very issue a reviewer needed to see.
What this means if you are building with AI
A few practical things I would build into this design:
Measure review time, not only accuracy. If the model gets better but reviewers take just as long, you have not saved anything. Record how long each review takes and how many fields people changed. Pair those measures with missed errors and downstream corrections.
Make "nothing to decide" visible. The best review screen is one where a person can see, in two seconds, that there is nothing here for them. That needs explicit acceptance checks, source access, and monitoring; a clickable source alone cannot prove that no review is needed.
Write the routing rules in plain language. Keep them in a place where the people who carry the risk can read them and change them. If only engineers can see why something went to review, nobody else will trust the system.
Never hide the thing the model did not check. If a required source was not available, say so on the screen. A missing check that is shown lets a person decide what to do. If the check is required, the draft should stay blocked until it is done. A missing check that is hidden is how a wrong value slips through quietly.
The model will keep getting faster and cheaper. People will not. The tools that win will be the ones that treat a person's attention as the most expensive thing in the room, and spend it only where it matters.
Sources and further reading
How to do a code review, by Google engineering practices team. Its guidance covers design, correctness, complexity, and tests. It is relevant to the code-review example; the broader attention-routing policy here is a design proposal.
Saleema Amershi and colleagues, Guidelines for Human-AI Interaction, CHI 2019, provides evaluated guidance for initial interaction, errors, and adaptation over time. It is a research foundation for making AI behavior understandable and correctable, rather than evidence that this proposed interface improves review speed.