Comparison between software and hardware decoding of videos under H.265 standard

    1. Concept

    Since the birth of digital video, the terms hard decoding and soft decoding have always accompanied its development. Soft decoding of digital video refers to decoding and playing the video through special playback software on a general-purpose CPU; while hard decoding refers to decoding the video in a dedicated hardware decoding module, and then sending the decoded data to the display screen for playback. Strictly speaking, there is no pure hard decoding, because even if there is a hardware co-processing module, the entire decoding process still needs to be controlled by software. So this article discusses the commonly known soft decoding and hard decoding.

    2. History

    As early as the 1980s, digital video commercial applications (Sony's D1 format, http://en.wikipedia.org/wiki/Digital_video#History ) had already appeared. At that time, digital video was stored on magnetic tapes without compression, which took up a lot of space. The related products were extremely expensive and ordinary consumers could not afford them, so the concept of digital video was not popular. The release of the digital video compression standard MPEG-1 in 1992, especially the subsequent release of the VCD standard based on this standard and the advent of related products, greatly promoted the development of digital video in the consumer field, and also popularized the concept of digital video among consumers. The subsequent popularity of MPEG-2 and DVD and digital television based on it allowed consumers to enjoy higher quality video programs. The new century's H264 standard and other private video standards such as RM, WMV, VP8 format, etc., have greatly promoted the rapid development of network video. The launch of the new generation standard H.265 last year marked that digital video technology and industry have reached a new height.

    As digital compression standards (techniques) continue to improve, its coding and decoding technologies and methods are also constantly innovating. In the early 1990s, when the digital video compression standard was just born, the computing power of general-purpose CPUs was far from meeting the decoding needs, so all decoding methods at that time used special hardware decoding chips for decoding, such as VCD players and MPEG-1 decompression cards (which can be inserted into 386 or 486 computers to play VCD movies). Around 2000, the performance of computers was enough to play DVD programs, and many excellent DVD playback software emerged at this time, which could smoothly soft-decode DVD data on personal computers. During this period, both soft decoding and hard decoding were very popular. Since then, with the development of the Internet, network video has developed rapidly. There are many compression formats for network videos, such as RM, VP8, MPEG4, H.264, WMV, and MPEG2. Due to the variety of formats, it is relatively complicated for hardware implementation. In addition, the network terminals of this period are personal computers, and their computing power is sufficient to decode these video contents, so the advantages of soft decoding are more obvious. At present, the general-purpose CPU of ARM architecture has achieved the same status in the field of mobile devices as the X86 architecture in the field of personal computers. The CPU frequency of ARM architecture can reach above 1.5GHZ, and dual-core, quad-core and even octa-core CPUs are widely used, and their computing power has reached or exceeded that of previous PCs. Video software decoding solutions also appear in large numbers in mobile devices.

    3. Decoding process and the difference between soft and hard decoding

    Digital video soft decoding means that the video stream data is played completely through the general-purpose CPU. The general process is as follows: first, the video playback software reads the video stream file from the storage medium into the memory, and then separates the video data and audio data in the file through an encapsulation format parsing module; then the playback software is responsible for transmitting the video data to the video decoding module, and the audio data to the video decoding module; the video decoding module is also a pure software module running on a general-purpose CPU, which uses complex algorithms to decompress the video data; the decompressed data is sent to the video display module, and the display module drives the hardware to display the video content. The processing of audio data is similar.

    The hard decoding of digital video is that the CPU plays the video stream data with the help of a dedicated decoding hardware module. Its process is as follows: first, the general CPU is responsible for reading the video stream data into the memory, and the data encapsulation format parsing module running on the general CPU separates the video and audio in the data. The audio data is sent to the dedicated hardware decoding module, which is responsible for decompressing the video data and sending the decompressed data back to the general CPU, which uses the decompressed data to drive the hardware to display the video. The audio may be soft-decoded by the general CPU, or there may be a dedicated hardware module for audio decoding.

    From the above description, it can be seen that no matter hardware decoding or software decoding, the participation of general-purpose CPU is indispensable, and the general-purpose CPU is responsible for the video stream playback process. The difference is whether the decoding algorithm with a huge amount of calculation is completed in the CPU or in a dedicated hardware decoding module.

    4. Application of soft decoding and hard decoding

    Currently, both soft decoding and hard decoding of compressed video are widely used, but their application fields and scopes are somewhat different.

    Soft decoding is widely used in the following devices:

    (1) Personal computers. Currently, all videos on personal computers are played through software decoding. This is related to the large number of compression formats used in personal and network applications, and is also closely related to the current super computing power of personal computers.

    (2) Smartphones. Today’s smartphone chips have powerful computing capabilities and are equipped with video hardware decoding modules. However, the hardware decoding modules support a limited number of video formats and are also limited by the dedicated playback software that is compatible with them. Therefore, when playing online videos, smartphones mostly use software decoding.

    (3) Tablet computers. Tablet computers generally use the same chipsets as smartphones and are used in a similar way to smartphones, so they also mainly use software decoding when playing online videos.

    Hardware decoding is mainly used in the following devices:

    (1) Smartphones and tablets: As mentioned above, these devices have video acceleration modules, and when playing local high-definition files, hard decoding is still used in many cases.

    (2) Set-top boxes. Early set-top boxes were basically a MIPS-based CPU + hardware decoding module. Therefore, only hard decoding existed on such devices. However, recently, newer set-top boxes use chips of the same grade as smartphones, but due to cost considerations, most set-top boxes still use hard decoding solutions. However, as chip performance improves and prices decrease, the proportion of set-top boxes using soft decoding will gradually increase.

    (3) Home cameras (video and playback functions). Cameras with video functions also have playback functions. The video decoding of these devices is implemented using hardware decoding.

    (4) Video conferencing. Video conferencing emphasizes real-time performance. The video encoding format is unique, and decoding is generally hard decoding.

    5. Trends and Outlook

    The existence of hardware decoding is to make up for the lack of computing power of general-purpose CPUs. With the improvement of the computing power of general-purpose CPUs, decoding tasks that could not be completed on general-purpose CPUs can gradually be completed, and hardware decoding solutions are no longer important. At present, the soft decoding of MPEG-1 and MPEG-2 video data by smartphone chips is very smooth, and some mobile phones can also soft-decode H.264 video data very smoothly, so the use of soft decoding is becoming more and more common in the field of smartphones. However, this does not mean that the hard decoding method will gradually fade out, because people's video compression ratio and video resolution requirements are also constantly increasing, which requires the device to have more powerful computing power. When the general-purpose CPU cannot meet the requirements, the hardware decoding module is also indispensable.

    The launch of the H.265 standard last year brought about some debate between hardware decoding and software decoding. The focus of the debate is whether H.265 hardware decoding or software decoding will become mainstream on smartphones.

    Supporters of hardware decoding believe that the computing power of smartphone chips is still barely enough to support H.265 high-definition playback, and only hardware decoding can achieve smooth playback. Considering the cost, it is not necessary to require higher performance for the main chip just to support H.265. Supporters of soft decoding consider the maturity of the standard and the performance development of general-purpose CPUs. They believe that H.265, as a relatively new standard, will undergo continuous revisions. If a hardware decoding solution is used, the hardware may not be able to keep changing with the revisions, resulting in the hardware being unable to decode the relevant videos. In addition, the performance improvement of general-purpose CPUs is roughly in line with Moore's Law, and the performance improvement is very fast. It was originally thought that soft decoding of H.265 videos under the ARM architecture was unlikely, but now it seems to be possible. Using the Redmi phone (a low-cost smartphone produced by Xiaomi, using the MTK6589 chip, quad-core cotrtex-A71.5G) to play 1080p videos in H.265 compression format is already very smooth (the playback software uses the H.265 playback apk developed by Beijing Ruiputu Technology Co., Ltd., and the video playback can reach more than 30fps). Currently, high-end mobile phones generally use Qualcomm Snapdragon 800 quad-core processors with stronger computing power, and there is no problem playing H.265 videos.