Apple wants to bring Zoom-like background replacements to Apple Glasses

Apple has been working on ways to allow "Apple Glass" style headsets to perform chroma keying, or replacing solid color backgrounds with Apple AR images. An Apple U.S. patent application titled "Low Latency Chroma Keying Embedded Head-Mounted Display for Mixed Reality" was filed in February 2020 but was only revealed this week. We are all familiar by now with replacing the backgrounds of offices and homes with other images, such as logos or beach scenes. We are also familiar with the fact that these things rarely work well, especially when the speaker moves.

That's partly because this kind of background replacement takes a lot of processing power, even when done very effectively. Before the July 2020 update, Microsoft Skype for iOS couldn't even blur its background like the Mac or PC versions. This blurring is a complicated process, but part of it involves determining where a person's head ends and the background begins. Chroma key makes this very simple because the person is placed in front of a single, solid-color background. It's the same green or blue screen process used in movies.

Systems like Zoom or Skype make it easier to replace backgrounds when it's obvious which part of a video conference is the person speaking and which part is their background. Those are 2D systems, though, and the goal with Apple Glasses is to do it in 3D so that the background can be replaced as the wearer moves around, rather than just looking at a flat screen. As a result, Apple's patent application is both about making 3D stereoscopic background replacements for Apple AR, and about making it work fast enough. Part of that has to do with headset processing power, but it's also about how fast that data can be returned to the user's field of view. Apple says it requires a wearable immersive head display (HMD) that captures the environment using embedded stereo RGB cameras, does real-time color keying, and a display system to show real-world images augmented with virtual content. The system runs at a high frame rate (typically 75 frames per second or more), and by capturing images and formatting them for display in the headset itself, it achieves latency of less than one frame.

So, what Apple's proposal says is that the headset has a camera sensor that performs Chroma key in a mixed reality environment, and that it does so with minimal latency to the wearer. Low latency is achieved by embedding processing in the headset itself, specifically, formatting the camera image, detecting the selected color range, and compositing it with the virtual content. The patent application does not address where the replacement image comes from. In a Zoom or Skype meeting, this would be selected by the presenter themselves.