The experiment
Nine months ago we shipped a feature most of our competitors already had: AI auto-reply. Flip a toggle and NextFlow drafts a reply to every new lead, then sends it automatically after a 3-minute delay. We thought users would love it. They did not.
After a surge of early adopters, churn on the auto-send cohort ran 18% higher than users who kept approving replies manually. So we ran a proper A/B test — 1,400 users over 6 months, two variants.
What the data showed
Variant A got AI auto-send: NextFlow drafted and sent a reply within 3 minutes of a new lead arriving. Variant B got what we now call 'AI-assisted': the draft appeared in the inbox, the user reviewed it with one click to send.
The leads that received an immediate AI reply were 31% less likely to convert than those who waited an extra 8 minutes for a human-reviewed one.
NextFlow internal data, Nov 2025 - Apr 2026
The reason, when we surveyed lost leads: the auto-replies felt generic. They were technically correct but missed the specific detail the prospect had mentioned. A human glancing at the draft for 10 seconds would have added that detail. The AI, sending blind, did not.
Speed matters, but accuracy matters more. Leads could feel the difference between a reply that read their message and one that pattern-matched to it.
Our recommendation
We removed the auto-send toggle entirely in v2.4. Every AI reply now lands in the inbox as a draft. The user sees it, taps send or edits one line, and the lead gets a response in under 2 minutes on average — fast enough to win the speed game, personal enough to win the trust game.
- Keep the human in the loop — one tap to review is not friction
- Train AI drafts on your past accepted messages, not generic templates
- Show the AI confidence score so users know when to edit vs. send
- Track draft acceptance rate as a product health metric
Reply time dropped 47% across the platform within 30 days of shipping the revised flow. Win rate on quoted leads went up 12 points. The lesson: AI should make humans faster, not replace them.
What we are building next
We are experimenting with a confidence threshold: if the AI scores a draft above 94% confidence match to the lead's query, it flags it as 'safe to send without reading.' Early data looks promising — 8% of drafts qualify. The other 92% still need a human eye.
The broader point is that AI in a sales context is an amplifier, not a replacement. The businesses winning with NextFlow are the ones where the owner still reads every lead — the AI just makes sure they never have to start from a blank page.