What Most Firms Get Wrong When They Evaluate AI

Over the past 2 years I have watched more than 1000 firms evaluate AI tools. A clear pattern has emerged: most start by testing meeting notes. It is the path of least resistance. Record a meeting, see what comes back, tick the box. But meeting notes rarely move the needle on capacity. The firms that actually transform their output start somewhere different.

Why meeting notes feel like the right test but are not

Meeting notes are table stakes. Every tool on the market produces acceptable notes from a recording. The quality differences are marginal. But notes do not address the real bottleneck, which is the hours your team spends writing suitability reports, annual reviews, and compliance documentation.

The numbers have not changed: 40% of firms spend more than an hour per suitability letter, and 30% spend up to 7 hours on complex cases. That is where the capacity sits. That is where the evaluation should start.

The question that separates the tools

When you sit down with any AI provider, ask this: does your tool use my templates or its own?

This is not a cosmetic difference. A suitability report written in your firm’s template, with your structure and wording, needs about 30 minutes of review. A report written in someone else’s format needs rewriting from scratch, which defeats the purpose entirely.

Some providers have tried multiple iterations of suitability reports and landed on standardised templates because it is technically simpler for them to build. Others build from your documents. The architectural choice matters because it determines whether the output fits into your existing workflow or creates a new one.

We have built more than 5,000 templates across firms over the past 2 years. 52% of firms that chose to work with us did so after seeing a report built from their own documents. Not a polished demo with perfect inputs. Their real templates, their real client data.

3 things to ask any provider

Before you commit to any tool, put these questions to every vendor on your shortlist:

1. “Can I trial this on my own data, with my own templates?” Not a controlled demo. Your real meetings, your real templates, your real client complexity. If a provider cannot show you output from your own documents, ask why.

2. “What is your guarantee?” If a provider will not put a number on time saved, that tells you something about their confidence in the product. We guarantee 50% time saving or your money back, but the principle applies to anyone you are evaluating.

3. “What does pricing look like before I sit through a demo?” If you cannot see the price on the website, that tells you something about the sales process you are about to enter. Monthly billing with no annual lock-in means the vendor has to keep earning your business every month.

What happens when firms get this right

Once the tool matches your way of working, the adviser’s role changes. You stop writing reports from scratch and start reviewing drafts that are already built from your template, in your tone, against your compliance requirements. The shift from author to editor is where the real capacity unlock happens.

Trust builds over weeks. In the early days most advisers check every line. After a month the review becomes lighter. One paraplanner we work with went from producing 1 suitability report a day to 3, without working longer hours. Emma, our suitability report assistant, generates the first draft from your template. You review and refine.

The freed-up time does not just mean more reports. It means paraplanners spending time speaking to clients, creating video walkthroughs, and improving the overall client experience.

A practical test before you evaluate anything

Before your firm looks at any tool, run this exercise: time how long it takes your team to produce 5 suitability reports this week. That is your baseline. Then ask every provider to produce 1 report from your template, with your data. Compare the output against what your team wrote, not against a demo deck.

The firms that get this right are not the ones that picked the flashiest tool. They are the ones that tested against their own work and measured the result before scaling.