Three months ago, we decided to properly test AI meeting summarizers. Not a quick trial where someone uses it once and declares it amazing or useless. An actual test — multiple tools, across different meeting types, with feedback from the whole team.

We tested four tools: Otter.ai, Fireflies.ai, Fathom, and Microsoft Copilot (through our existing 365 subscription). Each got used for at least three weeks across a mix of internal standups, client calls, strategy sessions, and one-on-ones.

Here’s what we found.

What We Were Hoping For

Our meeting problem wasn’t unique. We have too many meetings, notes are inconsistent, action items get lost, and people who miss a meeting have no reliable way to catch up. We wanted a tool that could:

1. Produce an accurate summary of what was discussed
2. Correctly identify action items and who they were assigned to
3. Be searchable after the fact
4. Not make people uncomfortable about being recorded

That last point matters more than you’d think.

Otter.ai

Otter had the best transcription accuracy of the four — roughly 92 to 95 percent on our internal calls, dropping to about 85 percent on calls with external participants (different accents, worse microphones, more crosstalk). The summaries were decent. They captured the main topics and produced reasonable bullet points.

Where Otter fell short was action item extraction. It would identify things that sounded like action items but frequently misattributed them or pulled out statements that were actually hypothetical. “We could probably update the pricing page” is not the same as “Alex will update the pricing page by Friday.” Otter often couldn’t tell the difference.

The interface is clean and search works well. At $16.99 USD per user per month for the business plan, it’s mid-range. Our verdict: good for transcription and general summaries, unreliable for action tracking.

Fireflies.ai

Fireflies was the most feature-rich tool we tested. Transcription, summaries, topic detection, sentiment analysis, speaker analytics — it does a lot. The summaries were structured by topic, which our team generally preferred over chronological formats.

Accuracy was comparable to Otter — slightly lower on transcription (around 90 percent), but the summaries were arguably better organised. The AI-generated “key takeaways” felt more useful than Otter’s equivalent.

The integration options were strong. Fireflies pushes summaries to Slack, HubSpot, Notion, and various CRMs. For us, the Slack integration was the most valuable — summaries appeared in the relevant channel within minutes of the meeting ending.

Downsides: the free tier is very limited, and the business plan ($19 USD per user per month) adds up quickly for a team our size. Also, the bot joining external calls confused several clients. One asked if they were being recorded for legal purposes, which was an awkward conversation.

Fathom

Fathom surprised us. It’s the most focused of the four — it does transcription, summaries, and action items, and that’s basically it. No sentiment analysis, no speaker analytics, no extensive integrations.

But it does those core things well. The free tier is genuinely usable (unlimited recording and transcription for Zoom). The summaries were concise — sometimes too concise for complex strategy discussions, but consistently capturing the essential points.

What stood out was the action item accuracy. Fathom was the best of the four at correctly identifying who committed to what. It’s not perfect — we estimated about 75 percent accuracy — but it was noticeably better than the others. The interface for reviewing and editing action items after the meeting is clean and fast.

For a small team that wants the basics done well without paying per seat, Fathom is hard to beat. We’ve kept it in our stack.

Microsoft Copilot

We had high hopes for Copilot since we already pay for Microsoft 365 Business Premium. The meeting summarisation is built into Teams, so there’s no additional bot joining calls — it’s native.

The transcription was adequate but not best-in-class. Australian accents tripped it up more than the dedicated tools. Summaries were functional but generic. They read like a textbook summary rather than a useful recap. Action items were basic and frequently missed nuance.

The advantage is zero additional cost and zero friction — it’s just there in Teams. For teams already deep in the Microsoft ecosystem who want “good enough” without adding another subscription, it works. But if meeting summaries are genuinely important to your workflow, the dedicated tools are meaningfully better.

What We Actually Learned

The tools work. Imperfectly, but they work. Here’s what we didn’t expect:

The summary isn’t the most useful part — the recording is. Being able to search for a specific moment in a meeting and replay the actual conversation has been more valuable than any AI-generated summary. When someone says “didn’t we discuss that?” we can find the exact moment in 30 seconds.

People behave differently when they know they’re being recorded. Some people speak more carefully, which arguably improves meetings. Others speak less, which doesn’t. We made recording opt-in for one-on-ones after feedback that people felt less comfortable being candid.

Action item extraction isn’t reliable enough to replace a human. None of the tools got action items right more than about 75 percent of the time. We still have someone manually confirm action items at the end of each meeting. The AI summary is a useful starting point, but it’s not authoritative.

The real ROI is for people who weren’t in the meeting. The people who benefited most from summaries weren’t attendees — they were team members who needed to know what was discussed but didn’t need to be there. This has actually helped us reduce meeting attendance, which might be the biggest win.

Our Recommendation

If you’re considering AI meeting summarizers, start with Fathom’s free tier. It’ll give you a realistic sense of what the technology can and can’t do without costing anything. If you find it valuable and want better integrations, look at Fireflies or Otter.

Don’t expect these tools to fix bad meetings. They’ll faithfully summarise a meeting that should have been an email. The summary will just make that fact more obvious.