Everyone’s selling AI now. Your accounting software has AI. Your email platform has AI. The person you met at a networking event last week definitely has AI. The term has become so broadly applied that it’s nearly meaningless as a differentiator.

Which creates a real problem for businesses that actually need AI capabilities. How do you sort genuine expertise from marketing fluff? How do you evaluate vendors when half of them are wrapping a ChatGPT API call in a nice interface and charging enterprise prices?

Here’s a practical framework.

Step 1: Define the Problem Before You Talk to Anyone

This sounds basic because it is. But most businesses get it backward. They see an impressive demo, get excited, and then try to retrofit the technology to their operations.

Before you speak to a single vendor, write down — in plain language — what business problem you’re trying to solve. Be specific. “We want to use AI” is not a problem statement. “Our customer support team takes an average of 4.2 days to resolve tickets, and we want to get that under 24 hours” is.

If you can’t articulate the problem clearly, you’re not ready to evaluate solutions. And any vendor worth working with will tell you the same thing.

Step 2: Ask What’s Under the Hood

When a vendor says “AI-powered,” ask what that actually means. You don’t need to be a data scientist to ask useful questions:

• What type of model are you using? Is it a proprietary model, a fine-tuned version of an existing one, or just API calls to OpenAI or Anthropic? None of these are inherently bad, but they have different implications for cost, customisation, and data privacy.

• Where does the data go? This is critical. If your business data is being sent to a third-party model, you need to understand the data handling policies. Ask where data is processed, whether it’s stored, and whether it’s used to train models. The Australian Cyber Security Centre has guidance on data sovereignty that’s worth reviewing before these conversations.

• What happens when the model is wrong? Every AI system makes mistakes. Good vendors will be upfront about accuracy rates and have clear processes for handling errors. If a vendor tells you their system is 99.9% accurate and never makes mistakes, that’s your cue to leave the room.

Step 3: Demand Proof, Not Promises

Demos are designed to impress. They show the best-case scenario with clean data and perfect conditions. Real-world performance is usually messier.

Ask for case studies from businesses similar to yours. Not just testimonials — actual metrics. What was the baseline? What improved? Over what timeframe? If the vendor can’t provide specifics, they either don’t have successful implementations or don’t measure outcomes. Neither is encouraging.

Better yet, ask for a reference call with an existing client. Any vendor confident in their work will facilitate this. If they won’t, consider that a red flag.

Step 4: Understand the Full Cost

AI vendor pricing can be opaque. The subscription fee is often just the starting point. Ask about:

• Implementation costs. How much does it cost to set up, integrate with your existing systems, and configure for your use case?
• Data preparation. Does your data need cleaning, formatting, or migration? Who does that work, and what does it cost?
• Training. Will your team need training to use the system? How is that delivered, and is it included?
• Ongoing support. What’s included in the subscription versus what’s extra? Is there a dedicated account manager or just a help desk?
• Scaling costs. If usage increases, how does pricing change? Some platforms charge per API call, per user, or per data volume. Get clarity early.

A Sydney-based firm we’ve worked with on AI evaluations makes the point that total cost of ownership over twelve months is typically 2-3x the initial quote when you account for integration, training, and change management. That’s not because vendors are dishonest — it’s because buyers don’t ask enough questions upfront.

Step 5: Evaluate the Team, Not Just the Product

AI products change fast. What matters more than today’s feature set is the team behind it. Are they investing in R&D? Do they understand your industry? Can they explain complex concepts in plain language?

Pay attention to how the vendor handles your questions during the sales process. If they’re evasive, overly technical, or dismissive of concerns, that’s how they’ll behave after you’ve signed the contract.

The best vendor relationships feel like partnerships. You should feel like they’re as invested in your success as you are. If it feels purely transactional, it probably will be.

Step 6: Start With a Pilot

Never sign a multi-year contract based on a demo. Push for a pilot phase — typically 60-90 days — where you can test the solution with real data, real users, and real conditions. Define success criteria upfront and evaluate honestly at the end.

A good pilot should answer three questions: Does it work with our data? Can our team use it? Does it deliver measurable value?

If the answer to any of those is no, walk away. The sunk cost of a pilot is nothing compared to the sunk cost of a failed enterprise rollout.

Trust Your Instincts

The AI vendor market will mature over the next few years. Standards will emerge, benchmarks will become more reliable, and the snake oil sellers will get filtered out. Until then, your best defence is clarity about your own needs, persistence in asking hard questions, and a healthy scepticism toward anything that sounds too good to be true.

Because in AI, as in everything else, it usually is.