Human in the loop AI: lessons from building Handle with Nathan Adler

Most AI products are pitched on what they can do without you. Nathan Adler has spent two and a half years building one on the opposite bet. As co-founder of Handle, an AI agent platform for scaling e-commerce businesses, he thinks human in the loop AI is the only kind that holds up in a real business, because the context behind good work rarely lives in a document an agent can read.

On Now or Never, Nathan sat down with Kaga and Fra from UntilNow to talk about what AI is genuinely good at, where it still falls over, and why the hardest part of building an AI company might not be the AI at all. Here's what founders and product leaders can take from it.

Why human in the loop AI beats "set and forget"

Handle started life under a different name, Cupcake, with a first use case most people will recognise: an AI executive assistant that booked meetings. The product has changed a lot since, but the thesis hasn't. As Nathan puts it, the team believed:

"AI needed to be better than just a solo problem solver, it needed to be a really good interpersonal team coordinator"

His reasoning is practical. The nuance behind how someone wants a task done is "seldom defined in a document somewhere upfront". An engineer might be happy to encode every rule. A busy e-commerce operator isn't. So instead of asking people to build complex workflows before the AI can help, Handle's agents coordinate with the humans around them, whether that's warehouse staff, sales people or customer support reps, plus the tools in their stack.

That's the core of a human in the loop approach. The person isn't a fallback for when the AI breaks. They're part of how the work gets defined in the first place.

What AI agents for business are good at (and where they still fail)

Asked by Kaga for an honest scorecard, Nathan named three strengths without hesitation. AI writes code well, especially off known patterns. It can get you most of the way to a solid answer on general knowledge. And, thanks to more recent models, it can reason through problems methodically without getting tired, which humans, being efficiently lazy, often don't.

The weaknesses matter more if you're designing a product around AI agents for business.

It tells you what you want to hear

Nathan's first point was blunt: AI is "absolutely a people pleaser and it remains so". It'll agree with your framing and run with it, which means a slightly wrong question can send both you and the model down the wrong track.

It doesn't ask the questions a good colleague would

The second gap is subtler. A mid-level team member handed an unfamiliar task will clarify the intent, the effort expected and what success looks like before diving in. AI tends to jump at the first solution that matches the request. In Nathan's words, it's:

"Really good at writing software, but not good at what you'd call soft skills."

Better prompting helps at the edges, he says, but it doesn't fix the underlying issue. That's exactly why keeping a human in the loop isn't a nice-to-have. It's the part of the system that supplies judgement, intent and the questions the model won't think to ask.

Designing AI that learns like a new hire

Handle's first attempt was to encode what the best executive assistant in the world would do. It hit a wall immediately. Everyone's definition of good was different, and the process had to fit the environment the AI was working in.

So the team shifted focus to learning. The model starts from a sensible baseline, then adapts to each business through feedback, the same way a good hire would. Nathan's comparison is a useful design brief for anyone building in this space:

"if you hired a really good team member on day one they're not going to just apply what they learnt in the last job at a completely different company"

Instead, they ask how you want things done, show a first attempt and ask for feedback. Today, Handle has hundreds if not thousands of assistants serving real requests. When one lets a customer down, the team unpacks what went wrong, almost like a psychologist: what information was missing, and why did the AI jump ahead?

From a UX point of view, this is the interesting bit. If the product's value depends on people correcting and teaching it, then the moments of asking, checking and giving feedback are the product. We've written more about this in our guide to designing AI products that actually work.

Founder led sales: start selling before you feel ready

When Fra asked what he'd tell founders still searching for product market fit, Nathan didn't hedge:

"Probably the only suggestion I have is try to start selling something."

His point is that experience on how to execute only comes from putting something in front of someone who doesn't care about what you've built. It's easy to delay that moment because the product isn't ready, a feature needs tweaking or the stealth launch isn't planned yet. None of that matters much until a stranger, not a friend who thinks the idea sounds cool, wants to buy it.

This is the case for founder led sales. Nobody else can learn what the market thinks of your idea on your behalf, at least not early on.

Customer number one changes everything

According to Nathan, things move much faster after the first paying customer. You can ask how many others look like them, and you can learn why they actually bought, because:

"chances are the reason you thought your first customer was going to buy your product is not actually the reason your first customer bought the product."

Once a small handful of customers find the product useful, the problem shifts from building something useful to talking about it in a way that brings in more of the right people.

Positioning is never finished

Nathan was refreshingly candid here. Two and a half years in, Handle is working on exactly that messaging problem, and he called it "kind of embarrassing to admit". The team are technologists and product people, and talking to customers in their language didn't come naturally.

Their fix has been to use their first few customers to build case studies, shape messaging and test what resonates, plus turning up at events where their customers are. Asked how many times they'd revised their positioning, he said "a dozen more", most of it small internal changes to how they talk to customers. He doesn't expect to ever fully settle on it.

His bigger lesson is about narrowing. Most products can serve more purposes than their target market needs, and the trap is saying your product does everything. That blurs the message for the people who'd benefit most. Founders should intentionally narrow their audience, and if a few customers in you still can't describe that audience clearly, your thesis probably isn't strong enough yet.

This is where brand earns its keep in a new category. When there's no "we're like X" to lean on, clear positioning does the explaining. Handle is an UntilNow client (we worked with the team on their brand story and identity early on), and their experience matches a pattern we see often, which we unpack in the less interface you have, the more brand you need.

When building is cheap, judgement is the bottleneck

On the old worry that building too much for one big customer will trap you in enterprise features, Nathan thinks that's largely a problem of 15 or 20 years ago:

"The speed with which you can build product is no longer the bottleneck to launching product into market."

What matters now, he says, is your speed of thought, your feedback loops and your accuracy of execution on the right strategy. Custom features are cheap. Learning which ones matter is not.

We'd go one step further. If AI makes building fast and cheap, the things that separate products are the things AI is still bad at: understanding intent, asking the right questions, and making a product feel trustworthy and clear. That's UX, taste and brand. It's why we think human judgement in digital product design is getting more valuable, not less.

What to take from Nathan's story

• Design AI to collaborate, not just complete tasks. Keep people in the loop where context and intent live.
• Treat feedback as a core feature. AI that learns like a new hire beats AI that guesses like a stranger.
• Sell early. Founder led sales teaches you what no amount of first principles thinking will.
• Narrow your audience and keep refining. Positioning is a living thing, especially in a category that doesn't exist yet.

There's plenty more in the conversation, including how Handle thinks about personal and team assistants with shared and private knowledge. Listen to the full episode with Nathan Adler on Now or Never.