7 things to do before your first AI build
Not a checklist — a funnel. Generate candidate jobs from your goals and the constraints blocking them, work out what harness each would need, spot-check which ones you can actually reach in your systems of record, qualify the survivors on evaluability and a real metric, and only then design the one that's left. Carried end to end through one worked example, including what gets deferred, what gets disqualified, and the difference.
The fastest way to a working AI system is not to start building. It's to spend about a week on seven things, most of which have nothing to do with technology, and then build against what you learn.
But it matters that these seven are a funnel, not a checklist. They run in order for a reason: the early steps are cheap and eliminate most candidates, and the late steps — human-in-the-loop design, exception handling, data governance — are genuinely expensive to think through. You want to spend that effort on two surviving candidates, not twelve hopeful ones. Most companies invert this. They fall in love with the most exciting job in the building, then burn a quarter on governance for something that would have failed step three in an afternoon.
Call the exercise a scoping pass, and distinguish it from the thing that gave discovery a bad name — the open-ended engagement that runs three months of workshops and produces a roadmap with fourteen use cases on it, by which point the sponsor has moved on. The difference isn't rigour, and it isn't duration. It's what the work is required to output. A roadmap is a list of things somebody might do. A scoping pass has to terminate in one job you are going to build, with the evidence that it's buildable attached to it. If it ends in a list, it failed.
Every step below resolves to an artifact somebody who already works at your company can produce in a day or two.
To keep it concrete, one example runs the whole way through, from a company objective down to a single job with a design around it. It's a composite — a mid-size contract manufacturer, the shape of plant we write about often — not a specific client, and the numbers are illustrative. What's real is the sequence, and which candidates die at which step.
Stage one — Generate and spot-check candidates
1. Start from the goal and the constraints, not from the tool
The question "where could we use AI?" produces a bad list. It surfaces whatever is fashionable, whatever someone read about, and whatever the loudest person is annoyed by. Nothing on that list has a number attached, so nothing on it can be defended in a budget review six months later.
Ask two questions instead. What is the company actually trying to do this year? And what operational constraints are standing in the way? The second question is the generative one. Constraints are where the company already knows it hurts — they're discussed in operations meetings, they have owners, and they usually already have a metric, because somebody is already being held to account for them.
This inverts the usual order and it's the whole trick. You aren't looking for places to apply AI. You're looking at what's blocking the business and asking which of those blockages are shaped like work an agent can do.
The company's largest customer has put them on notice about late shipments. The objective for the year is on-time delivery from 82% to 95%, without adding headcount.
The constraints in the way, as the operations team would list them: purchasing doesn't find out a supplier will miss until the material doesn't show up, because promise dates live in email rather than the ERP; two bottleneck cells go down unpredictably and every investigation starts from scratch; the scheduler rebuilds the week by hand each Monday and it's stale by Tuesday; a quality escape triggers a customer report that takes two days to assemble; and the one person who understands the old press retires in fourteen months.
The objective in one sentence, and the constraints blocking it as a short list — written by operations, not by IT.
2. Turn the constraints into candidate jobs, and work out what harness each needs
Now convert each constraint into a job — a specific, repeated unit of work somebody does — and ask what an agent would need around it to do that job.
There's a zone where agents need almost no scaffolding, and it's worth knowing where it is: work that's high volume (it happens hundreds of times, not five), language-heavy (the input is documents, emails, calls, notes), and rule-bound (there's a right answer and an experienced person could explain why). In that zone a capable model, decent retrieval and a clear spec get you most of the way, which is why it's where first projects should live.
The mistake is reading that zone as the boundary of what agents can do. It isn't. It's the boundary of what they can do cheaply. Outside it, agents work fine — they just need a harness. A numeric optimization needs a solver the agent calls, explains, and re-runs rather than replaces. Eliciting what's in somebody's head needs a structured interview loop that runs over weeks, not a prompt. A plan that affects physical things needs a way to check it before it ships. None of that is exotic; all of it is real engineering that has to exist.
So this step doesn't produce a kill list. It produces a sequenced list. The question is what each candidate's harness would be, whether you already have the pieces, and whether building them belongs in a first project — because the first build is buying you a working system and the knowledge of how your company ships one, and you want to pay for that lesson on a job where the harness is thin.
Six candidate jobs come out of those constraints:
- A. Chase supplier promise dates out of email and record them in the ERP.
- B. Predict which suppliers are about to slip.
- C. Draft downtime write-ups from CMMS notes and machine logs.
- D. Rebuild the weekly production schedule.
- E. Assemble the customer quality report.
- F. Capture what the retiring press operator knows.
Two get deferred here, and neither because an agent couldn't do them.
D, the scheduling rebuild, needs a solver. An agent is genuinely useful around one: pulling constraints out of email and floor notes, calling the optimizer, explaining why the schedule changed, re-running it when a cell goes down at 2am. That's a real system. But this company doesn't have the optimizer, so the work is two builds rather than one, and the riskier half isn't the agent. Worth doing. Wrong thing to do first.
F, capturing what the retiring operator knows, needs an elicitation loop. An agent can absolutely run this — a months-long structured interview that drafts, asks the follow-up a human interviewer wouldn't think of, and assembles a corpus somebody can actually search. The reason it's not the first build is that its output is an asset rather than a transaction: there's no repeated cycle to measure this quarter, so it can't teach you what shipping looks like. It's also the only candidate with a hard deadline, which argues for starting a cheap version in parallel rather than waiting.
Four candidates continue into the filter: A, B, C, E.
A candidate list, each written as a job with its volume and who does it now, plus the harness it would need. Keep the deferred ones on the list with the reason and the missing piece named — they're a roadmap, not rejects, and somebody will raise them again in three months.
3. Spot-check what you can actually reach
Before anything else technical, and before you spend a day on baselines or benchmarks: can software read from — and write to — your ERP, EHR, CMMS, or case management system? Under what credentials, with what audit trail, and who controls that decision?
Ask in writing, from the person who actually knows, which is usually not the vendor's website. Answers land in one of four places. A modern API, and you're in good shape. A legacy API that exists but needs a license tier, a partner agreement, or middleware. No API but a supported path — flat-file drops, a staging table, an RPA seat — workable, and it shapes the whole design. Or genuinely nothing, in which case a candidate is read-only by necessity and you should decide that deliberately rather than discover it late.
This belongs in the initial spot check, and not because it's the cheapest question — defining the job is cheaper. It's because this is the one answer that can arrive with a procurement clock attached. If the write path needs a license tier or a partner agreement, that's weeks of purchasing which can run in parallel with everything else — but only if you asked in week one. Ask in month three and the clock starts then, with a finished model sitting idle waiting on it.
Two things to notice as you check. Read access is usually easy; write access is what decides how much a candidate can ever save, because a system that drafts while a human carries the result into the ERP has a hard ceiling on it. And reachability is mostly a system-level fact rather than a per-candidate one — establish it once and it prunes whole classes of candidates at a stroke. This single question kills or reshapes more AI projects than model quality ever has, and it's answerable in a day.
C, the downtime write-ups, dies here. It needs two inputs: CMMS notes and machine logs. The CMMS is readable. The machine logs sit on a historian nobody has ever connected to anything, so half the input doesn't exist in reachable form, and connecting it is its own project with its own budget. (The reachable half wouldn't have carried the job either, for reasons that turn up at step 5.)
A needs to write a promise date into a PO, and that turns out to sit behind a licensed integration module the company doesn't own — roughly six weeks to procure. That doesn't kill A. It starts the clock in week one, which is the whole difference between six weeks running in parallel with the build and six weeks of stall discovered after the model already works.
E is read-only: it assembles a document out of four systems and writes nothing back, which makes it the easier build by a wide margin. B's inputs are reachable as well.
Three candidates continue: A, B, E.
A named person's written answer on read and write access per system, plus the timeline and cost of anything you don't already have.
Stage two — Qualify the survivors
Two questions decide whether an agent can do a job well enough to rely on. By now you're asking them about a handful of candidates instead of a wish list, which is the entire point of the ordering.
4. Is it clearly defined, and can you tell whether it's working from a quantitative metric?
Two halves, both hard requirements.
Clearly defined means two experienced people would agree on what the output should be for a given input. If they wouldn't, you don't have a specification, you have a debate — and the model will surface that debate as apparent errors until somebody settles it. Settle it first.
A quantitative metric means there's a number that moves if this works, it already exists or can be measured this week, and it's a number somebody cares about independent of the AI project. Beware the job whose only metric is "people will feel less frustrated." That may be true and it will not survive a budget review.
Take the baseline now, before anything is built. Four numbers: how long the work takes per unit and in staff hours per week, how often it's wrong and how you'd know, the end-to-end cycle time including waiting, and what a single error costs when it escapes. That last one sets the accuracy bar — "98% accurate" means something entirely different for internal cost coding than for a claim submitted to a payer. And take these numbers now because once the system exists, measuring the old process honestly becomes politically impossible.
B, predicting supplier slips, fails here — and it's worth being precise about why, because it's the candidate with the most obvious appeal. It isn't clearly defined: nobody can say what threshold counts as "about to slip." And its metric is a counterfactual — you'd be measuring shortages that didn't happen, against a baseline that doesn't exist. There may be a real system here eventually. It is a terrible first project.
A and E both pass. A's metric is the strongest, and notably it's a number nobody had been tracking: days between a supplier slipping and purchasing finding out. Currently about nine. That number sits upstream of the OTD objective, which makes improving it legible to the person who set the objective.
For each surviving candidate, the metric and its current value, dated, with the method noted.
5. Can you evaluate the output — either from history, or by QA?
You need a way to know whether a given output is correct. There are two acceptable answers and you only need one.
A benchmark from your own history. Pull 100 to 200 real cases — real invoices, real emails, real charts — spanning the easy middle and the genuinely hard edges, and record the correct answer for each. Where "correct" is arguable, capture the disagreement; that tells you something important before the model does. Build it before the system, for two reasons beyond measurement: a benchmark assembled afterward tends to consist of cases the system handles, and building it forces your domain expert to articulate their reasoning on hard cases, which is the requirements document you'd otherwise never get.
Or reliable QA before the output is consumed. Some jobs have no usable history — the work was never recorded, or was recorded so thinly that there's nothing to score against. That's survivable if a competent human can check each output quickly before it matters, because then your review process generates the labeled data you didn't have. This is the honest path for a lot of real work, and it's a fine way to start as long as you're explicit that you're buying accuracy with human attention until the record accumulates.
What's not acceptable is neither: no history to score against, and no practical way to check the output before it's relied on. That's not a project, it's a hope.
A and E both pass, and A's case is unusually strong: the company has years of supplier emails paired with the delivery date that actually happened. The ground truth is simply the past, which is the cleanest benchmark you can get. E is easier still — its output is a report that already exists in a hand-assembled version, so every historical month is a labeled example.
This is also, in retrospect, the second gate C would have failed. The CMMS free-text fields average about four words — "belt broke, replaced" — so even the reachable half of its input couldn't support a benchmark, and the maintenance leads don't agree on what a good write-up contains. Two independent disqualifications is a useful signal: the honest conclusion isn't "do it anyway," it's that better capture is the prerequisite. That's its own small project with its own payoff, and it makes this candidate viable next year rather than never.
For each survivor, either a spreadsheet of real cases with expected outputs and a human baseline score, or a written QA procedure naming who checks what, how fast.
Choosing between the survivors
Two candidates cleared the filter. Pick using proximity to the objective, not ease — but knowing which is which.
E is easier, safer, and read-only. A is harder, needs a procurement cycle, and is tied directly to the metric the CEO committed to.
A is the first build. It's the one whose number the objective is made of, and the first build should teach you how your company ships an AI system on a job that's worth having shipped. E goes second — and because it's read-only and shares most of the same data plumbing, it can run slightly behind A rather than after it.
Stage three — Design the survivor
Only now is it worth spending real thought on the following, because now you're spending it on one job you have good reason to believe is buildable.
6. Design the human-in-the-loop path before the happy path
Decide in advance: what fraction of cases the system handles autonomously, how it signals that it's unsure, where uncertain cases go, who works that queue, and how that person gets enough context to resolve one quickly.
This comes before the build because a system handling 90% of the volume saves nothing if nobody can tell which 90% it was. If exceptions don't surface cleanly to a named human in a place they already look, your team reviews everything to find them, and you've added a step instead of removing one. Confidence signalling and escalation routing are architecture, not features you bolt on at the end.
Then climb, don't leap. Start with a human approving every action, treat each approval as a labeled example — accepted, corrected, discarded — and promote to autonomy per case type as that record earns it. Narrow and boring classes first, ambiguous ones last or never. The reasoning is here: starting cautious and getting bolder is a project that gains trust, while starting bold and pulling back loses it permanently, usually in a single incident.
The agent reads supplier email, extracts the promise date with a confidence signal, and updates the PO. It writes autonomously only when the email is unambiguous and the date falls inside the existing need-by window. Everything else routes to the buyer's existing daily queue — no new tab — with the source email and the schedule impact attached: ambiguous language, a date that slips past need-by, or any supplier on the watch list.
For the first two weeks the buyer approves every write, which builds the labeled set. Autonomy is then granted per supplier, starting with the dozen whose emails are most formulaic.
The autonomy rule, the escalation triggers, the queue it lands in, the person who works it, and the starting threshold.
7. Settle ownership and governance in writing
Two things, and both are cheap now and very expensive later.
The owner. Someone whose existing job gets easier or harder based on whether this works — the purchasing lead, the billing manager, the maintenance supervisor. Not IT, not innovation, not whoever is most excited about AI. Their name goes next to the metric from step 4, they get a real say in the exception design, and their objections are requirements rather than resistance. If nobody in the operating org will take that role, the pain isn't as expensive as the pitch deck said, and you should revisit the candidate rather than proceed with a sponsor who'll be reorganized in six months.
The data posture. Where the data goes, which providers process it, what's retained and for how long, what's logged, who can see the logs, and which agreements cover it. Answer before the architecture exists, because the answers are the architecture: if your position requires zero retention, a specific hosting region, self-hosted inference, or request-level audit logging, that's foundational, and discovering it at the production gate means rebuilding. Draw the data path on one page; every arrow leaving your control needs an agreement behind it. In a regulated industry, get whoever owns compliance to look at that page in week two rather than week twelve — they are almost always more pragmatic when consulted early than when presented with a finished system.
The owner is the purchasing lead, whose team's number this is. Governance raises one real question nobody had considered: supplier email contains pricing and contract terms, so the decision about whether that content leaves the building is a genuine one with a contractual dimension. And the ERP write runs under a named service account with its own audit trail, so that a wrong promise date is always traceable to the agent rather than to a buyer who didn't make it.
A name against a number, a one-page data flow diagram, and the list of agreements covering each hop.
Why the order is the point
Look back at what the funnel did. Six candidates in, one first build out. Two were deferred with their missing piece named, two were genuinely disqualified, and the whole sort took about an hour of thought per candidate. Nothing expensive was spent on any of the five that didn't win. The stage-three work happened once, on a job with a metric, a benchmark, a write path, and an owner.
Notice in particular what the step-three spot check bought. It killed one candidate outright on a fact nobody had checked — that half its input lives on an unconnected historian — and it turned A's licensed integration module from a month-four crisis into a week-one purchase order. Reachability is the cheapest question with the longest lead time attached, which is exactly why it belongs in the initial pass rather than after you've grown attached to something.
Run the funnel backwards and you get the common outcome: three months of governance review and integration architecture for the scheduling rebuild, discovered in month four to be blocked on an optimizer nobody scoped and a definition of "working" nobody agreed. The candidate wasn't wrong. Its harness was never counted, and it was picked first anyway.
Every step resolves to an artifact somebody at your company can produce in a day or two: a sentence, a candidate list, four numbers, a spreadsheet, an email from IT, a rule, a diagram. None of it requires an AI expert. All of it has to happen eventually — the only question is whether it happens in week one, when it's cheap and shapes the design, or month eight, when each answer invalidates something already built. That gap is most of the difference between a first project that ships and one that becomes a pilot that never shipped.
And a side benefit that's easy to miss: run a scoping pass and you become a much harder customer to sell to. You'll know in one meeting whether a vendor or partner is serious, because you'll have the metric, the benchmark, and the constraints in hand — and anyone who wants to skip past them to talk about their platform has told you what they are. That cuts both ways, and it should: the partner worth hiring is the one who wants to run this pass with you before quoting a build. Which is the next question worth asking — whether to buy, build, or partner at all.