Can project panoramic images of video games

According to foreign media reports, a newly published patent shows that Microsoft may plan to expand console games beyond TV screens by projecting "external images" across the room to provide a 360-degree view of the virtual scene.

Microsoft's patent, titled "immersive display experience," was published by the U.S. Patent and Trademark Office (USPTO) last week and was originally filed in early 2011. The patent describes a standard video game system with an external "ambient display" that projects panoramic images in a way that "appears to surround the user."

Such a projector would not replace the TV display used for current game consoles, but would provide "external images" as an "extension" of the main display, with the aim of expanding the gaming environment beyond the TV display so that players can "turn around to observe an enemy sneaking up from behind."

Microsoft's latest patent describes how to project panoramic images around a room

The display device described in the patent is tightly coupled with a depth-sensing camera system that could be packaged in the same box as an ambient projector. This device could be a standard dual-camera structured light system like the current Kinect, or it could be something more complex that incorporates a "multiple image capture device" that "uses multiple captured images to 'stitch' a panoramic image" that can then be projected across the room.

Regardless of its form, the depth-sensing camera system described in this patent could assist environmental projectors by sensing the layout and shape of a room, allowing the projector to provide features such as color and distortion correction so that the image does not appear distorted even when projected onto different wall and furniture surfaces.

In addition, the ambient display can use the player's position information provided by the camera system to "adjust the appearance of the external image display" to ensure that the environment still looks natural when the player moves around the room. The patent even describes creating a "protection" zone where the player stands, so that the light of the projected image does not shine directly into the player's eyes, even when the player is looking in the direction of the projector.

The projected image described in the patent "may be displayed at a lower resolution than the main image", but the patent document points out that this will not "adversely affect the user experience". The projected image does not necessarily appear flat on the wall, and players can wear stereoscopic 3D glasses to gain additional depth and reality. This part of the content is similar to the concept of "Fortaleza glasses" smart glasses mentioned in the Microsoft document leaked in June this year, which is an augmented reality technology.

How projectors detect user location and create “protected” zones

The gaming system described in this patent may never become a fully functional product, but this kind of environment system could bring real changes to the way people play games and think about video games. In my opinion, one of the main factors that limit the current motion control is that the player needs to be facing the TV screen to know what is happening in the game. As a result, in many games, the player can only lean and move to the side instead of turning around or moving parallel to the screen.

But if the game could be projected onto all four walls of a room, the user could actually move around the room and look in any direction to avoid missing what's happening in the game. Imagine a new version of Dance Central that shows Arthur Murray-style footprints on the floor, or an adventure game where you physically explore a room to find hidden treasure, or a first-person shooter where you can physically turn around to face the muzzle flash projected onto the wall to the right of your body, rather than just twisting in response to a red flash on the right side of your display.

If this environmental display works as advertised, it would eliminate the need for head-mounted virtual reality devices like the Oculus Rift, which use sophisticated head tracking technology to simulate the environment around the player.

But I'm a little concerned about how much a 360-degree projector can actually transform a room. Unless the projector described in the patent is mounted on the ceiling, I can't understand how it can avoid the significant problem of shadows obscuring the image - not just by the player's body, but also by furniture and other objects in the room. Color and image distortion correction are good ideas, but it's always a little disconcerting if part of the scene that should be in the corner of the room appears in front of that corner.

Still, it's always interesting to see how Microsoft's R&D department is thinking about revolutionary ways to extend video gaming beyond the black borders of your television.