Services · AI data and output review
Check what your AI produces before your users see it
If your product uses AI to create records or draft messages, we review what it actually produces, report the errors and the patterns behind them, and then improve the prompts, validation rules and data pipelines so it's ready to launch.
Request an AI output reviewRead the extraction demonstration
- Starts with
- A written scope of what is in and out
- You get
- A sampled review, the error patterns, and rules in code
- Price
- Set by the written scope, agreed before work starts
If your product uses AI to create records, answer users or draft messages, we read a sample of real outputs against the source data. We log every error with an example and group the errors into patterns. Anything with a fixed shape, like totals, IDs, dates and allowed values, gets a validation rule in code, so bad output is caught before it’s saved or sent. We rerun the same examples after each prompt or rule change, and we check what the AI can reach for each user role. Passwords, API keys and bulk data never go into AI tools. Our document extraction review is a demonstration on invented data, not client work.
Is this a fit?
Who it is for
Startups and small software teams with an AI feature that generates data or text, close to launch or already in front of early users.
Problems it solves
- AI-created records nobody has checked in bulk.
- Drafted messages that sound right but sometimes state things that aren’t true.
- The same mistake again and again, with no clear cause, and no validation before it’s saved.
How to request it
- Open the request form; this service is already selected.
- Say what the AI feature produces, roughly how much of it there is, and what a bad output looks like for you.
- Thomas replies by email. After a short call or a few written questions, you get a written scope and price before any work starts.
Free checklist. Pre-launch QA checklist: the checks we run, written so your team can run them on its own product before launch. No email needed.
For a checklist to use when judging an AI feature or proposal, read How to evaluate an AI implementation before you hire anyone.
Proof from our own demonstration
Numbers from a demonstration on 120 invented documents. It is labelled as a demonstration, not client work, and every failure is published.
- 0 of 34auto-accepted documents with wrong data, on 60 held-out documents
- 54 of 60documents with every field correct
- 6 of 6documents with a wrong or missing field sent to a person
Situations this fits
AI output can read well and still be wrong. Situations like these:
- Records an AI step created or filled in, such as profiles, summaries or categories, that nobody has checked in bulk.
- Drafted messages that sound right but sometimes state things that aren’t true.
- The same kind of mistake showing up again and again, with nobody sure whether the cause is the prompt, the input data or the code around it.
- No validation between the model’s output and your database.
- A launch date, and no clear answer to how often the output is wrong.
What the review covers
- A sample review of AI-produced records and drafted messages, read against the source data they came from.
- An error report: each error with an example, and the recurring patterns grouped so the biggest causes are clear.
- Prompt changes, rerun on the same examples so you can see what changed.
- Validation rules in code that catch bad output before it’s saved or sent.
- Data pipeline fixes where the problem starts in the input data rather than the model.
When a rule beats a better prompt
Not every problem needs a new prompt. A plain rule in code is usually the better fix when:
- The output has to follow a fixed shape: the same fields, formats or allowed values every time.
- The check can be written down as a clear if-this-then-that.
- The value must be exactly right every time, such as totals, IDs, dates or amounts.
Changing the prompt makes sense when the problem is wording, tone or judgment that a rule can’t capture. Either way, the change is rerun on the same examples before it ships.
What you receive
- A written review with examples, error patterns and their likely causes.
- Changes as pull requests to prompts, validation rules and pipeline code, with tests, reviewed by your team before they’re merged.
- Tickets in your tracker for each problem we find.
- A rerun of the same examples after each change, so you can see what moved and what didn’t.
- A weekly written progress note.
What this does not include
So the scope is clear from the start:
- A promise of a particular error rate. We report what we measure, on the sample we agree.
- Changes made directly in production, unless you grant production access in writing.
- Passwords, API keys or bulk data pasted into AI tools.
- IT support, hosting or marketing. We focus on auditing, testing and fixing software.
How the work runs
- Scope. We agree what gets reviewed, how big the sample is, and what counts as an error.
- Review. We read the output against its source data and log each error.
- Report. You get the errors, the patterns behind them, and what we’d change first.
- Improve. We change prompts, rules and pipeline code as reviewed pull requests.
- Recheck. We rerun the same examples and report what moved.
How we work with your team
The same rules apply to every engagement:
- Staging only. We work in a staging environment. We have no production access unless you grant it in writing.
- Every change is a pull request. Your team reviews it before it’s merged.
- Tickets in your tracker. Findings and fixes live where your team already works.
- A weekly written progress note: what’s done, what’s next, and anything blocked.
- AI tools only on accounts that don’t train on your data. Credentials and bulk data never go into AI tools.
Ownership and handover
The aim is code and tests your team can maintain without us.
- Prompts and rules in your repository, not in a tool only we can open.
- Running costs named up front. If a change affects model usage costs, we say so before it ships.
- Ownership. Code written specifically for you becomes yours once it is fully paid for, subject to the signed agreement. Our preexisting tools and code, and third-party materials, are not transferred.
- After launch. There is no automatic fix period. Support after launch is available only under an agreed contract.
What moves the cost
There is no fixed price list, because cost follows scope. These are the things that usually make a review larger or smaller:
- How much output there is to review, and how varied it is.
- How many prompts, models and pipeline steps are involved.
- Whether the source data the AI works from is available to compare against.
- How many rounds of change and recheck you want before launch.
Who’s accountable
Thomas Wagenberg, founder. He scopes and leads every engagement and signs off every finding. Testing and fixes use AI-assisted engineering under his direction, and every change reaches you as a pull request your team reviews.
Wagenberg Technology Group serves startups and small software teams remotely across the United States, with a focus on Cleveland and Fort Lauderdale.
Where and how we work
We work remotely with teams across the United States, including teams in Cleveland and Fort Lauderdale. Pricing depends on the scope of the project, which we agree in writing before work starts.
For teams in Cleveland and Northeast Ohio: software audits and launch-readiness reviews for Cleveland and Northeast Ohio.
Tell us what you’re launching
Describe the product and what worries you about it. Thomas will reply by email.
In your request, say what the AI feature produces, roughly how much of it there is, and what a bad output looks like for you.