How to Evaluate an AI Vendor

Most AI vendor selection happens in a meeting room watching a demo. The demo is polished, the use case is recognisable, and the room nods. What the room is evaluating is the packaging — and packaging is the one thing the vendor has fully controlled.

You don't need to be technical to evaluate an AI vendor well. You need to ask a small set of questions the demo doesn't answer, and to notice which vendors answer them cleanly versus which ones reach for adjectives.

The questions a demo never answers

• What happens to our data? Where is it stored, who can access it, is it used to train the vendor's models, and can we get it back? For APAC firms this is also a compliance question under the PDPA, the Privacy Act, or PCPD guidance — not a technical footnote.
• What is the demo running on? Demos run on clean, curated data. Ask what happens when it meets your data — the inconsistent, half-complete, real version. The gap between the two is where most deployments disappoint.
• How does this integrate? "Integrates with everything" usually means "integrates with nothing without a project." Ask specifically how it connects to the systems you actually run, and who does that work.
• What does it cost to leave? Switching cost is the real price. If your data, workflows, and institutional memory become locked inside the vendor's platform, the renewal conversation is not a negotiation.
• What's the total cost, not the licence? Implementation, integration, training, and the internal time to run it routinely dwarf the sticker price. A cheap licence on an expensive system is not cheap.

Signals worth trusting

Beyond the answers, watch how they're given. Vendors who deal in specifics — named integrations, concrete data-handling commitments, reference clients you can actually call — are describing something real. Vendors who deflect to vision, roadmap, and "enterprise-grade everything" are describing something they hope to build. Both can be fine; you just need to know which one you're buying.

The most useful reference is not the logo wall. It is a client at roughly your size, in roughly your situation, who went live more than six months ago and is still using the thing. Anyone can launch. The question is what survives contact with daily operations.

The independence problem

Here is the structural issue: the people most able to evaluate a vendor are often the people selling one. Resellers, implementation partners, and platform-aligned advisors all carry an incentive that may not match yours. That doesn't make their advice wrong — but it means the evaluation should sit with someone whose only stake is your outcome.

Vanguard Advisory™ reviews vendor proposals and contracts as part of ongoing governance — assessing fit, data handling, and lock-in without the conflict of a reseller relationship. We don't sell the tools. We help you choose them well.