How do mobile GPUs create realistic light and shadow games?

As smartphone performance continues to upgrade, the mobile game market is booming, and more and more realistic games are loved by gamers. It can be said that creating a real game world is a goal that humans are constantly pursuing. With the popularity of the concept of the metaverse, if we want to reproduce the real environment in the virtual world of the future metaverse, we need to find a way to achieve the intuitive perception of these objects by the human eye. Real-time ray tracing technology may be one of the key supporting technologies.

Ray tracing technology has been mainly used in the gaming field. It simulates the light changes, reflections, and inversions in the real world to display 3D scenes in the game world that are highly similar to the light and shadow in the real world. As the scene continues to grow and the complexity of the picture increases, the effect of ray tracing will become more and more obvious.

In real life, a virtual beam of light from a light source hits an object. The light then interacts with the object and reflects off another surface based on the surface properties of the object. The light then continues to reflect, creating light and shadows. Ray tracing in a computer, or more accurately "path tracing," is the reverse process of how light travels in the real world. Light is actually emitted from the camera's point of view and hits an object in the scene, and an algorithm calculates how the light will interact with that surface based on the properties of the surface it hits. The path of each ray of light is then traced back to each object until it returns to the light source. The result is a scene that is illuminated as if it were lit by the sun in the real world: with realistic reflections and shadows.

Due to the high computational complexity required by this technology, it was first used in animations and movies such as Avatar, Ready Player One, and Gamer, where scenes need to be rendered on a dedicated server cluster for months, which is not applicable to games. In game scenes, they must be generated in real time at a speed of at least 30 frames per second, preferably twice as fast or even higher. It was not until 2018 that NVIDIA's RTX series graphics cards with Turing architecture and AI algorithms such as DLSS appeared, which really pushed ray tracing technology to a peak in PC and console games.

Imagination recently introduced ray tracing technology into the IMG CXT series, the first PowerVR Photon architecture optimized for mobile, which means that mobile gamers and developers will have the experience of desktop-level ray tracing capabilities.

Imagination GPU IP takes another leap forward with the introduction of ray tracing on mobile devices

Before the advent of ray tracing, the images in games were mainly rendered using "rasterization" technology, which can achieve some local lighting effects, and the details of the images are mainly based on textures. If ray tracing technology wants to achieve more realistic images, it requires extremely high graphics computing capabilities of the graphics card. Currently, many platforms in the industry have claimed to support ray tracing technology.

Shi Xin, Vice President of Strategic Marketing and Ecosystem at Imagination China, said that ray tracing can be performed at different performance and efficiency levels. To clarify this, Imagination established a ray tracing level system (RTLS), which determined the requirements for ray tracing functions at six levels, from Level 0 to Level 5. "Level 0 represents traditional solutions, referring to early fragmented ray tracing implementations, including Intel Embree, Caustic OpenRL, etc.; Level 1 represents software solutions on traditional GPUs; Level 2 begins to add dedicated hardware units for ray tracing, including processing issues for the intersection of ray-box and ray-triangle; Level 3 represents hardware BVH processing capabilities; Level 4 represents specific hardware BVH processing capabilities and coherence classification capabilities; Level 5 represents coherence BVH processing functions with BVH hardware generators." He explained, "From Level 1 to Level 2, with the addition of hardware, the performance is improved by dozens of times, and Level 3 to Level 5 is the gradual enhancement of fixed-function hardware units."

"In 2014, Imagination created the industry's first real-time ray tracing silicon architecture and demonstrated the GR6500 test chip code-named 'Plato'. Since then, Imagination has been continuously evolving, improving and optimizing the architecture to achieve optimal performance and power efficiency. But the entire ecosystem was not ready at the time. The graphics API had no standards related to ray tracing, and the related development tool software, game engines, etc. did not have ray tracing capabilities." Shi Xin pointed out, "Since NVIDIA began to launch ray tracing on PCs in 2018, the entire industry ecosystem began to invest. In 2020, the Khronos Graphics Standardization Organization also launched the ray tracing Extension of the Vulkan API. We believe that the time has come to launch chips with ray tracing capabilities together with our partners."

He pointed out that IMG CXT has reached the L4 level of ray tracing, is "the industry's first RTLS Level 4 ray tracing architecture", "the world's first to achieve L4 hardware processing, and on mobile devices", and has a power consumption that is 2.5 times higher than existing RTLS Level 2/3 solutions.

In the picture below, after ray tracing is turned on, the hard-edged shadows are replaced with beautiful, realistic, soft ray traced shadows. All objects have a soft light reflection effect. Some shadows become very soft and almost invisible, but they cleverly increase the realism of the lighting.

According to official information, the IMG CXT-48-1536 RT3 core has three ray acceleration clusters (RACs) that can provide an overall performance of up to 1.3 GRay/s. This can provide realistic ray tracing shadows, reflections, global illumination, and ambient occlusion effects at high frame rates within the power budget of mobile devices. It has also taken an important step forward in rasterization graphics processing performance, with a 50% increase in computing, texture, and geometry performance compared to Imagination's previous generation GPU IP. Its low-power superscalar architecture provides high performance at low clock frequencies, resulting in excellent frame rate power consumption (FPS/W) efficiency, while Imagination image compression (IMGIC) technology can significantly reduce bandwidth requirements.

RAC includes Ray Store, Ray Task Scheduler and Coherency Gatherer, and is tightly coupled with two 128-wide unified shader clusters (USCs), which have high-speed dedicated data paths to achieve the most efficient and lowest power ray tracing deployment. Ray Store can save the ray data structure on the chip during processing and provide high-bandwidth read and write access to all units in RAC, thus avoiding the speed reduction or power increase caused by storing or reading ray data in dynamic random access memory (DRAM). Ray Task Scheduler can offload tasks from shader clusters and deploy and trace ray workloads through dedicated hardware while maintaining high ray throughput and low power consumption. The unique Coherency Gatherer unit can analyze all propagating rays and bind the rays in the entire scene into multiple coherent groups so that they can be processed with higher efficiency.

At present, Imagination has fully supported mainstream API interfaces and has carried out in-depth cooperation with application manufacturers, supporting rendering engines such as OpenGL. The company revealed that it is in in-depth communication and cooperation with major game companies such as Perfect World, Tencent Games, and NetEase Games to explore the application of ray tracing technology in games and other scenarios.

Shi Xin emphasized that the C series of GPU IP products built based on IMG A series and B series GPUs will become the fastest mobile GPU IP core ever available for all games. At the same time, due to its open nature, IMG CXT supports multi-core expansion and can support the powerful computing power required for the Internet and cloud games.

The Metaverse and Ray Tracing

Kristof Beets, Imagination's vice president of technology foresight, emphasized that ray tracing is a large technical concept, and each platform has different technologies to implement it. Some platforms use software simulation to achieve the effect of ray tracing, but Imagination does it through hardware. The efficiency of the two is not at the same level, and Imagination's efficiency is much higher.

"Ray tracing effects can also be achieved on mobile devices using software solutions, but that can only be done in DEMOs. If applied to games, the effect will definitely not be very good. This is because this solution cannot achieve a good balance between power consumption and effects, and the software still has to use a lot of computing resources for shadows and polygonal maps." Kristof Beets pointed out, "In the past, the lighting effects in complex scenes, including reflections, shadows, global illumination, etc., all relied on shaders to simulate and calculate one by one. IMG CXT transfers the rendering load to more dedicated hardware, which is more efficient and achieves lower power consumption."

The launch of IMG CXT and its Photon architecture means that the application of ray tracing technology will be extended to a wider range of fields such as games, mobile, AR (augmented reality), data centers and autonomous driving. For example, in the human-machine interface (HMI) platform of the car, IMG CXT can enable applications and devices such as surround view, satellite navigation and head-up display (HUD) to achieve more realistic images and regional displays, and can also provide users with in-car gaming and entertainment experiences.

As for the recently popular concept of the Metaverse, Kristof Beets believes that it is still evolving, just as he is also part of the Metaverse by accessing a live meeting via video. Even VR/AR, which has been developed for many years, still faces many technical difficulties and has not yet achieved the ideal state of this application. "Ray tracing is likely to be a way to fully realize VR/AR. It is a technology required for VR/AR and the corresponding Metaverse. It also makes us believe that making ray tracing more efficient and less power-consuming is absolutely the right direction. In the future, any scene that needs to be rendered can add ray tracing technology to make the entire rendering look more realistic and have more natural light and shadow effects."