AI agents still need humans for real-world tasks

The next time someone’s super-powered AI agent tries to make a phone call to negotiate a cable bill, it might be a real human haggling with customer service rather than the software.

Indeed, AI agents, powered by cutting-edge core models developed by companies like Meta And OpenAIstill cannot reliably take actions over the web, especially those involving phone calls. Former Yahoo CEO Marissa Mayer, who now runs agent startup Dazzle AI, says today’s top AI models excel at presenting home improvement services online to consumers, but might falter when they need to book and schedule a roof upgrade.

“We rely on the frontier models to give us good suggestions and a good list of candidates, but we check that to make sure that we’re actually providing the kind of service that we can support,” Mayer said, referring to how Dazzle farms out some tasks to human contractors.

AI-driven digital assistants are currently seeing renewed interest thanks to Meta, OpenAI and others touting the technology as capable of performing tasks such as searching for shoes on the Internet.

Meta’s Muse, OpenAI’s Dots, and a host of other agents can accomplish a multitude of web browsing feats, but the technology is still in its infancy and often faces digital hurdles. These obstacles typically involve tasks that would require an AI agent to make a phone call, schedule a doctor’s appointment on someone’s behalf or book a couch cleaning with a local cleaning service, said Meghan Joyce, CEO of digital agent startup Duckbill.

For tasks that require real-world judgment, Joyce said humans still trump AI, and companies building consumer digital assistants will need to integrate real people to take over. That’s why Duckbill recently launched an application programming interface, or API, aimed at companies that create personal agents to connect with human contract workers.

“We are working on a model which is reminiscent of Uberwhere we break down tasks into little atomic pieces,” Joyce said, referring to the ride-sharing company where she previously worked.

Companies creating agents can use Duckbill’s API to delegate tasks that the agent may perform. Duckbill translates these tasks into bite-sized chunks that the average adult can complete consistently and reliably.

“And then we follow what happens,” Joyce said.

Mayer said AI models may still lack common sense. She said a Dazzle AI super user attempted to get the technology to simultaneously handle a series of home improvement projects involving a roof replacement, a broken window and the purchase of a glass top.

But the AI ​​agent, Mayer said, mistook window glass for table glass, leading the digital assistant to “try to figure out if it could fit the table top in the window.”

The incident didn’t last long, Mayer said, because Dazzle’s technology flagged the task and assigned it to a person to complete.

“In a matter of seconds, the human fixed things. Whereas an agent would have burned through a ridiculous number of chips,” Mayer said. “Or maybe they never found a good solution.”

Shivani Poddar, CEO of personal AI agent startup Wajo AI, is also integrating human workers to handle phone calls and even CAPTCHA tests meant to fend off bots from sites.

“I think AI plus humans, collaborative AI, is what you need to actually get the job done,” Poddar said, emphasizing that people will stop using technology if agents can’t accomplish their tasks.

Interactions between chatbots and agents can be ultra-personal, and users have good reason to be concerned about human intervention, especially if the company hasn’t explained to them how it works.

Meta employees have previously raised privacy concerns over the company’s internal testing using human contractors to handle phone calls on behalf of its Muse AI agent. A Meta spokesperson said in a statement at the time that this type of testing was “central to the product development process.”

Meta currently does not use humans to respond to users’ Muse requests.

Duckbill’s Joyce said her startup does a “full background check” on the 1,099 contractors it uses, comparing them to Uber and DoorDash drivers, “people who live here in the United States and want an additional source of income.” Duckbill also uses contract workers from companies that typically offer contact center work to businesses.

Mayer said Dazzle runs background checks and looks for contractors with the same qualities someone might look for in a human personal assistant.

“We generally look for people who we think are extremely discreet and who have good taste and good judgment, both in terms of how they interact with our users, but also just in terms of their ability to forget and compartmentalize that information afterward,” Mayer said. “I understand it’s a matter of privilege, but I’ve had personal assistants for a while, and I know how I want them, and I know how I guide them.”

Mayer said that while basic models continue to improve, the market is in a “transition period” until technology can perform more complex web tasks and phone calls as efficiently as individuals.

“We don’t even try to hide it, we think human participation is a benefit,” Mayer said.

WATCH: Who is responsible when AI agents break the rules?