Next-generation image processing technology makes Fujifilm’s X-Trans sensor even more powerful

How do X-Trans cameras differ from other cameras on the market, and how is machine learning revolutionizing the way raw files are processed? DxO Chief Scientist Wolf Hauser discusses the pros and cons of X-Trans, and the processing methods DxO uses to significantly improve image quality.

Fujifilm has never been shy about breaking new ground, and in 2012 they introduced the X-Trans sensor. This was a bold move given that the entire camera industry was using Bayer sensors. Over the past decade, there has been a lot of discussion about X-Trans, but does it offer tangible benefits to photographers, or is it just an elaborate marketing trick? As we'll explore, X-Trans has both advantages and disadvantages, and the algorithms used to interpret the sensor's raw data are crucial to getting the desired results. Fans of Fuji cameras have long been searching for the best software to process their images, and DxO PhotoLab 5 and DxO PureRAW 2 now support processing of X-Trans raw files, bringing out the outstanding detail in sharp images captured by Fuji cameras.

Bayer or X-Trans, Pepsi or Coca-Cola?

You may have noticed that discussing the minutiae of camera technology can stir up a lot of controversy, and since its introduction, debates have raged on message boards and social media about the pros and cons of X-Trans. Yet the color filter array is just one of many features that go into defining a camera, and few customers prioritize it when buying one. Photographers who use Fujifilm cameras love the unique look and feel of their cameras, and enjoy their ergonomics and ease of use, not to mention the stunning results straight out of the camera. Fujifilm engineers are experts in color, with more than 70 years of experience in color photography, and brand enthusiasts love the film simulations that Fujifilm has created based on the company’s long history of film production.

Names like Astia and Velvia also lend a sense of realism to their cameras.

The Bayer vs. X-Trans debate is very similar to the showdown between ARM processors and those designed by Intel. Apple's marketing department claims that their iPad is better because it has their new ARM chip, while Microsoft convinces the public that the Surface is better because it uses the latest Intel chip. Fans of both brands will spend countless nights arguing fiercely on forums about the advantages of ARM vs. Intel, RISC vs. CISC - but in reality 99% of users don't care. They choose a tablet based on the look and feel, user experience, and brand recognition that they prefer.

Aside from some technical differences within the systems, for most users, choosing Fuji vs. Canon, Nikon, Sony, etc. is probably not much different than choosing Pepsi vs. Coca-Cola.

The consequences of complexity

Whatever the criteria for choosing a camera, it’s still important to squeeze the best performance out of its sensor possible. It’s also worth looking at what happens when you blow up the image to 100% or more.

We need to remember that the final image is not just the result of the sensor itself. It goes through various processes, either in the camera or through software, notably demosaicing, which is the algorithm that fills in the gaps between the red, green and blue channels. These algorithms, combined with the color filter array, determine the final result.

One reason the Bayer filter has endured is that engineers are used to processing data from it. It took researchers four years to find the best way to demosaic a Bayer sensor, and the highly complex algorithms they developed over the years allowed them to relax many of the fundamental constraints on their design. Even fairly simple algorithms, such as those embedded in the first generation of digital cameras, can produce good results.

On the other hand, the added complexity of the X-Trans mode also brings with it a much more complex demosaicing process. It is said that it took Fuji engineers five years to catch up with the processing capabilities of their cameras, and they were able to introduce X-Trans in the X-Pro1 in 2012. At the same time, the research community has published far fewer papers on X-Trans demosaicing compared to Bayer. Not only is it a more complex problem, but there are also fewer studies devoted to solving it. In all fairness, the current X-Trans demosaicing algorithm is still some distance away from achieving the theoretically optimal solution. This is one of the reasons why Fuji fans often wander between different software packages to find a solution to get the best results.

Overcoming complexity with machine learning

Today, machine learning—particularly a technique called convolutional neural networks—is revolutionizing image processing. These new algorithms, which are no longer calculated by hand by researchers and engineers but learned empirically from millions of training examples, are eclipsing decades of research in just a few years. In Bayer’s demosaicing technique, for example, neural networks now easily beat the best algorithms designed by humans.

This revolution, while frustrating for researchers who have spent their lives working on demosaicing algorithms, is actually a huge opportunity. Not only do the results get better, but productivity is also improved: a computer can find a state-of-the-art demosaicing algorithm in days or weeks, not years or decades. Machine learning is particularly useful when there is a clear input and expected output, but the mapping between the two is too complex to be expressed by a classical algorithm. Image and speech recognition are the earliest examples. Machine learning is indeed a very powerful tool, and it continues to prove useful in areas where classical algorithms work well, such as demosaicing.

X-Trans demosaicing is a great candidate for machine learning. As a more complex demosaicing process than Bayer, the advantages of machine learning over traditional engineering should be greater than those seen with Bayer demosaicing. This was clearly demonstrated by our peers at Adobe when they launched the machine learning-driven Enhance Details feature in early 2020. Reviewers concluded that the Bayer image showed only subtle differences, but the X-Trans image was significantly improved.

At DxO, we used machine learning in DxO PhotoLab to solve another highly complex task: our RAW conversion technology, DxO DeepPRIME. It uses a single, huge convolutional neural network to apply demosaicing and noise reduction simultaneously. After 10 days of intensive work, our computers developed a highly sophisticated algorithm that outperformed our traditional demosaicing techniques at low ISOs, and our traditional demosaicing and noise reduction techniques at high ISOs.

DxO PhotoLab5 and DxO PureRAW2 offer DxO DeepPRIME for X-Trans

Once the work on the Bayer sensor images was done, making changes to accommodate the X-Trans raw files was no longer a daunting task, as the process of generating the training data could be reused with only minor modifications. There were still difficulties to overcome, as we had to radically change the shape of the network to accommodate the complex X-Trans patterns, but it was doable and promising, and the results were exciting. Let’s look at an example.

This low-light, indoor action shot was shot at ISO 6400 using a Fujifilm X-T3. The original image was underexposed, so we pushed it up two stops in post-processing — to the equivalent of ISO 25600 (above). Such drastic exposure adjustments aren’t possible on JPEG images, so the comparison here isn’t with the camera, but with a well-known raw converter, Adobe Lightroom with the Enhance Details feature enabled (below left). When we zoom in on the face, we can see that the DxO DeepPRIME (below right, using DxO PhotoLab) image is noticeably sharper. Because DeepPRIME uses a neural network to run both demosaicing and noise reduction, it does a better job of reducing noise while preserving more detail in brightness and color.

More than just machine learning

Obviously, it took more than just DxO DeepPRIME to fully support X-Trans in software as sophisticated as DxO PhotoLab and DxO PureRAW 2. Many of the internal tools used in our lab to calibrate the color and noise models for each camera body had to be adjusted as well. Several other processing blocks also had to be designed from scratch, such as the demosaicing algorithm used to display a preview when the user is making adjustments.

Let your photos also benefit from technological advances

After a period of intensive development, DxO PhotoLab 5 and DxO PureRAW 2 are now ready to bring significant improvements to your RAW files. We think photographers will love our DxO DeepPRIME technology, which successfully reproduces previously missing color details, breathes new life into old photos, and improves the quality of high ISO images. Download a free trial and find out how DxO DeepPRIME can improve the quality of your photos.