Microsoft patents AI capable of literally putting words into people’s mouths in real time

Microsoft has just filed a patent for an AI product that can transform live video in real time to change speakers’ words and faces.

The patent, first reported by patent tracking system Patentlyze, is described by its inventors as a “near real-time artificial intelligence (AI) enhancement of video or augmented reality (AR) presentations of live events.”

Using AI agents, the system would identify and change or remove certain words, while changing the speaker’s face and mouth accordingly. Agents would rely on defined settings or user profiles to make changes, and would also edit video and audio just before and after a deleted or replaced word to allow for “smooth transitions.”

In an age of media and the Internet increasingly defined by AI-powered deepfakes, misinformation, and information confusion, the system described in this patent may represent a new world of filtering tools that subtly alter reality as it unfolds, in this case on a highly personalized level. Viewers, the patent suggests, could attend any live event without hearing what they don’t want to hear.

The potential use cases discussed in the patent vary. One possible application is real-time translation, the system for hearing and seeing broadcasts and other live footage in the viewer’s preferred language.

The patent also posits that such a tool could be used to remove filler words like “um” or “ah” from live speech, or to erase stutters or speech disorders.

However, other possible use cases go much further. According to the patent, the system could be used to reduce “personal words or expressions” like “made-up terms,” ​​as well as “colloquial terms” like the term “y’all” or the phrase “bite the bullet.” The patent also claims that the system could eliminate “mumbled words, mispronounced words, misused words, mispronounced words, offensive words, insensitive words, or inappropriate words.” Replacement words, he adds, may come from an “Internet search.”

In other words, live streams may not be live streams after all.

Microsoft did not immediately respond to a request for comment.

Dubbing has been around for ages and video footage is constantly edited, for normal or insidious reasons. However, automatically editing live videos in this way with the help of AI agents could fundamentally change the way audiences interact with live footage. As it stands, if people don’t like what they hear, they can just turn something off; The system described by Microsoft describes an environment in which people can adjust someone else’s speech to their personal tastes. Language translation aside, does using a tool that eliminates errors or erases words and phrases associated with a speaker’s local dialect move us further into personalized bubbles? What’s so bad about hearing someone from the South say “y’all?”

To this end, the patent mentions that the original live video and audio of an event can be recorded and stored for the sake of “ground truth”, which hints at the potential abuse and confusion that could arise from such a tool.

The types of live footage that could be transformed by the system include lectures, presentations, a range of television productions – award ceremonies, games and cooking shows – and live sports, among others.

Learn more about Big Tech patents: Meta Patent technology to detect your mood by constantly analyzing the tone of your voice