Running AI Operations With a Human Review Layer: Where Judgment Belongs
Most conversations about AI operations focus on what a model can produce. A more useful question for anyone building a business or a body of work is where a person still needs to look at the output before it reaches a customer, a publication, or a decision. Todd Michael's public writing on systems, automation, and building in public returns often to the idea that a system is only as trustworthy as the checks around it. This article lays out a way to map your AI workflows, decide where human review belongs, and ask the right questions of any tool you adopt. It is general guidance, not a prescription for any particular setup.
Start by mapping the workflow, not the tool
Teams often begin with a list of AI tools they have tried: a writing assistant, a summarizer, an image generator, a support chatbot. A list of tools tells you very little about risk. What matters is the workflow each tool sits inside. Write out the steps a piece of work passes through, from the first input to the final thing someone else sees or relies on. Mark each step where a model produces text, a number, a classification, a recommendation, or an action.
Once the map exists, label each AI-touched step by its consequence. A draft that only you read has low consequence. A reply that goes to a paying customer, a figure that feeds a financial statement, or a command that changes a live system has high consequence. The label drives everything else. You do not need the same level of review for a brainstorm and for a published claim.
Keep the map short enough to revisit. A one-page diagram or a table with columns for step, tool, output type, and consequence is usually enough. The point is to make the implicit explicit, so that the places where errors would hurt are visible before something goes wrong.
Decide where a human must look
A workable rule is to require human review wherever an output is hard to reverse, hard to detect as wrong, or seen by people outside the team. A published article, an outbound email, a contract clause, and a change to production settings all meet at least one of those conditions. An internal outline or a list of possible headline variants usually does not.
Review should be specific. Telling someone to 'check the AI's work' produces inconsistent results because reviewers do not know what to look for. Instead, define what the reviewer is checking: factual claims against a source, tone against a style guide, numbers against the original data, promised features against what the product actually offers. A short review checklist attached to each high-consequence step turns vague oversight into a repeatable act.
Decide, too, who has the authority to approve. If the person who prompted the model is also the only person who reviews the output, you have a single point of failure. Where the team is small, a simple second read by a different person, even on a delayed schedule, catches a surprising number of errors.
Questions to ask before adopting an AI tool
Before a tool enters a workflow, ask the vendor or your own testing what it is designed to do and what it is not. Look for plain statements about its limits. Ask what data it receives, where that data goes, and whether your inputs are used for anything beyond providing the service. Ask how you can export your outputs and your configuration if you later decide to leave. If the answers are unclear, treat that unclarity as information.
Test with your own material rather than demo examples. Feed the tool a set of realistic inputs, including awkward edge cases, and score the outputs against what a competent person would produce. Record the failures. A tool that performs well on clean examples and poorly on messy real ones needs more review, not less.
Ask what happens when the tool is wrong. Does it signal uncertainty, or does it state errors with the same confidence as correct answers? Does it refuse, or does it invent? A tool that fails loudly is often easier to operate safely than one that fails quietly.
A simple comparison of review approaches
Three common approaches appear in small teams. The first is full pre-review, where every output is checked before anything happens. It is safest and slowest, and it suits high-consequence work. The second is sampled review, where a fraction of outputs is checked on a schedule. It suits lower-consequence work where patterns matter more than individual items, but it requires someone to notice drift over time. The third is exception review, where outputs pass automatically unless a rule flags them, such as a number outside an expected range or a message containing a sensitive term. It scales well but is only as good as the rules you write.
Most workflows benefit from combining these. Use full pre-review for public and irreversible steps, sampled review for internal drafts, and exception rules for routine processing. Revisit the mix whenever a tool changes, a new output type appears, or an error gets through that the current setup should have caught.
A practical checklist to run this month
List every place an AI tool touches your work and write down its consequence level in one line each. Mark any step that is public, irreversible, or financially or legally meaningful. For each marked step, write a three-item review checklist that names exactly what a reviewer checks. Assign a named reviewer who is not the person who generated the output. Choose one tool you currently use and run your own test set of ten realistic inputs, including at least three hard ones, and record where it fails. Decide what failure rate would make you remove the tool from a high-consequence step. Finally, schedule a review of this map after thirty days, when you will have real data about where errors actually occurred.
This process will not make AI output perfect, and it is not meant to. Its purpose is to put human judgment where it does the most good and to keep your operations understandable enough that you can see when something has gone wrong. Systems that are written down, tested against real material, and reviewed on a schedule tend to remain under control longer than systems that exist only in someone's memory. For readers following Todd Michael's work on building systems in public, the same discipline applies: make the workflow visible, name who is responsible for each check, and keep the record honest enough to learn from.