Guide

How to evaluate an AI implementation before you hire anyone

General guidance for comparing proposals and reviewing an AI project already under way. It is not a measured result or a proven method of Wagenberg Technology Group.

The short answer

Before you commit money to an AI implementation, you should be able to answer eight questions in writing. If a proposal cannot answer the first three, you cannot judge its cost. If it cannot answer the last four, it is not ready to launch.

1. What problem is this meant to solve?

Name one specific task or decision. “Use AI for support tickets” is not a task. “Read each new support ticket, pull out the account, the product area and the urgency, and send it to a person to confirm” is. Also name who owns that task today. An AI system with no owner will drift.

2. What will you measure?

Choose two or three measures you can count: time per task, error rate, cost per item, or response time. Decide in advance what value would count as success. If a proposal cannot say how success will be measured, the result will be a demonstration, not an improvement.

3. What is the baseline?

Measure the task as it is done today, over a fair period, before any change. Without a baseline you cannot show an improvement, and you cannot tell whether a problem came from the AI or from the work itself. Many failed projects skipped this step.

4. What will it cost?

Count more than the build. Include the setup cost; recurring model usage, hosting and storage; maintenance and updates; and the staff time needed to review output, which is often the largest hidden cost.

5. How is your data handled?

List the kinds of data the system will receive: public text, internal documents, customer records, personal data. For each kind, record who is authorized to send it, which processor handles it and where, what the provider’s current written terms allow (including whether data may be stored or used for training), how long it is kept and when it is deleted, and who in your organization can access it.

Check the provider’s terms yourself on the day you decide. Do not rely on a vendor’s marketing summary or on a setting you have not verified. Passwords and API keys should never be pasted into an AI tool. Other sensitive information may be processed only with authorization, appropriate safeguards, verified provider terms, and controlled access.

6. Who reviews the output?

Name the outputs a person must approve before anyone uses them, and the person responsible for that review. “The AI checks itself” is not a review. A useful test: would a wrong output reach a customer or inform a decision without a human reading it first?

7. What happens when it fails?

Ask what happens when the model is wrong, slow or unavailable. Ask whether failures are logged, who is alerted, and how errors are caught before they reach customers. An automation that fails silently is the most common way AI projects cause damage.

8. Who owns it after launch?

Ask who maintains the system, who owns the code, the data and the accounts, and what happens if the provider changes its price or product. You should hold the keys to the system you pay for. If a vendor controls the only copy of your prompts, data or configuration, you do not own the result.

How to use these questions

Send the eight questions to every bidder and ask for written answers. Compare the answers, not the demonstrations. A polished demo that cannot answer questions 1 to 3 is a sales tool, not an evaluation.

When custom software makes sense

Custom software is usually worth considering when the process is specific to your business and happens often; when off-the-shelf tools would need awkward workarounds to fit it; and when the cost of the current manual process is known or can be estimated. It is usually not worth building when a well-known tool already does the job at an acceptable cost, or when the process has not yet been defined. Before any build, write a scope: what will be built, what will not, and how you will know it works.

What AI workflow automation can and cannot do

It can draft, sort and extract information from text, and move data between tools on a schedule. It can save time on repetitive steps when each output is checked by a person.

It cannot be trusted to be right on its own. Its output can be wrong, out of date, or confidently written and still false. It cannot decide what your business should do, and it does not remove the need for review. A workflow that runs without logging or a review step is a risk, not a saving.

How Wagenberg Technology Group approaches this work

Wagenberg Technology Group reviews the records and drafted messages that AI features produce, reports the errors and patterns, and improves the prompts, validation rules and data pipelines behind them. These are capabilities we can provide; this guide does not describe completed client engagements, customer results or certifications. The approach has four stages, for software and AI work alike:

1. Map. Learn how the work is done today and where it breaks, then write a scope: what is built, what is not, and how success is judged.
2. Build on a staging copy. Changes run on a copy of your system first, not on your live system. Code lands through reviewed changes that are recorded.
3. Test by role and by device. Each change is checked against the people who will use it, on the devices they use, and each fix is retested.
4. Hand over. You receive the code, the documentation and access in your own accounts, so you keep ownership.

For AI steps, a person reviews the output before it is used. Passwords and API keys are never pasted into AI tools.

What to ask before you hire a software or AI consultant

• What is the problem, and how will you know when it is solved?
• What will you measure, and what is the baseline?
• What will it cost in total, including ongoing costs and review time?
• Where will my data go, who can access it, and what do the provider’s written terms allow?
• Which outputs will a person check before use, and who is responsible?
• What happens when it fails, and how will I find out?
• Who will own the code, data and accounts when the project ends?
• Can I see a written scope, and the staging environment, before anything goes live?
• Which parts have you done before, and which are new to you? Ask for evidence you can check, not testimonials.

For when to bring in outside help, and the red flags to watch for, read Hiring an AI automation consultant: when it makes sense and what to ask.

Questions this guide does not answer

It does not give prices. Pricing depends on the scope, which we agree in writing before work starts. It does not promise results and does not name customers. If your team is about to launch a product that uses AI to create records or draft messages, tell us about it and Thomas will reply by email.