HTTP ABR video transmission quality measurement description

1. Introduction

HTTP ABR (Adaptive Bit Rate) is currently the most popular OTT (Over-The-Top) transmission technology. Typical examples include Apple HLS (HTTP Live Streaming), Microsoft Smooth Streaming, Adobe Zeri Streaming, and DASH (Dynamic Adaptive Streaming over HTTP).

HTTP ABR is a video technology that uses HTTP/TCP protocol for lossless transmission and automatically adjusts the video bit rate according to the network bandwidth. It is very different from the traditional UDP-bearing or radio and television broadcasting network-bearing lossy transmission video services. When network performance changes, such as router congestion and packet loss, traditional image quality indicators such as MOS-V remain unchanged for HTTP ABR, and lose their meaning. Therefore, HTTP ABR services require a new set of measurement systems to measure video transmission quality. Spirent's AS Score-based indicator system designed for this service has become a new benchmark for measuring this service and is about to become an IETF standard.

2. Why are traditional IPTV video quality analysis methods not applicable to HTTP ABR services?

Comparison between lossy transmission video and HTTP ABR video

Traditional network video IPTV services are mainly based on UDP-bearing video streams. The characteristic of UDP bearing is good real-time performance, but if packet loss occurs, it will not be retransmitted, and packets with jitter and excessive delay will be discarded, which is a lossy transmission for video streams. Therefore, when network damage occurs, the quality of the decoded video will deteriorate, resulting in mosaics, blurred images and other problems, as shown in Figure 1 below.

Figure 1: Mosaic and blurred images appear in UDP-borne video stream

HTTP ABR video service is based on TCP to carry video streams. The characteristics of TCP are reliable connection and lossless transmission. After packet loss, it will be retransmitted. Jitter and delay will be absorbed by the client's download buffer. Generally, the client will not feel it. Only when the video in the buffer is played and no new video clips are downloaded in time, the picture will wait and buffer, as shown in Figure 2 below.

Figure 2: TCP carries video stream

Traditional network video quality analysis indicators are used to evaluate video quality when the video image is damaged. However, when network performance deteriorates, such as when router congestion causes packet loss, HTTP-borne video services will not lose media packets, and the image quality is completely consistent with that of the sender. Are some of the original analysis indicators still applicable?

Are the commonly used measurement indicators for lossy transmission video quality applicable to HTTP ABR services?

The following indicators are commonly used to measure video quality for UDP-based IPTV video services or video services on radio and television broadcasting networks, and Spirent VQA video quality measurement solutions support them all:

MOS-V

MOS-V originally refers to the subjective scoring of 1-5 points by observing the video quality through the human eye, see ITU-T P.910 (04/2008). The MOS-V indicator currently widely used in video quality testing uses an algorithm to analyze the video encoding, frame rate, packet loss distribution, and image group structure received by the client, and converts the algorithm to obtain a MOS-V score equivalent to the subjective evaluation measurement of the human eye. [page] Is

MOS-V applicable to HTTP ABR services?

It is only applicable to the real-time video encoding stage, and is meaningless for network transmission.

As analyzed above, ABR services use TCP lossless transmission. After the encoded video stream (such as H.264 stream) enters the network (such as CDN), the media segments sent by the sender and the segments received by the receiver are completely consistent. TCP packet loss will be retransmitted during the transmission process, and there is no packet loss for the video stream, so the packet loss distribution calculated by the MOS algorithm is meaningless. That is, when packet loss occurs at the network level, the MOS value will not change for the video service carried by TCP.
Therefore, in ABR services, MOS can only be applied to the video encoding stage before the video is sent, that is, to make a preliminary comparison of the encoding quality of the encoder.

In some special occasions, when there are network elements for real-time video transcoding in the transmission network, MOS can also be used to measure the encoding quality of the transcoding device alone. However, for HTTP ABR services, it has the ability to provide a variety of different bitrate streams to adapt to different user situations. The client automatically selects the download bitrate. It is not economical to perform real-time transcoding on the network, so this scenario is not common in HTTP ABR services.

It should be pointed out that the purpose of video transmission quality measurement is to simulate a large number of user accesses with instruments and measure the service quality of the network under high traffic conditions. The encoding quality depends on the encoding algorithm and has nothing to do with the number of users or network status. For example, VOD services are provided by the network to users after the encoding software encodes the files offline and sends them to the network storage (such as CDN) in a non-real-time manner.

The key is that for the service quality of each network element on the transmission network that operators are most concerned about, such as packet loss, jitter, delay, etc., the MOS indicator remains unchanged due to the absence of video damage. That is, when the network quality changes and the user perception changes, the MOS indicator cannot respond and loses its meaning.

MDI

MDI: DF delay factor indicator, indicating the delay and jitter status of the tested video stream. DF is in ms. DF converts the change of video stream jitter into the demand for video transmission and decoding equipment buffering.

MDI: MLR media packet loss rate indicator, the number of media packets lost per second during network transmission, indicating the media packet loss situation.

Is MDI applicable to HTTP ABR services?

Not applicable at all.

MDI:DF was originally intended to indicate the need for buffering on decoding devices, especially TV set-top boxes, which have limited buffering and a buffering time of milliseconds. However, for HTTP ABR services, decoding devices are mainly smart terminals such as PCs and mobile phones, which download media segments. The terminals themselves require buffering space to accommodate a large number of files, and the buffering time is at least in minutes. The MDI:DF indicator has lost its meaning.

The TCP retransmission mechanism itself ensures that there will be no packet loss at the media level, so MDI:MLR must be 0, which is meaningless.

VSTQ

video service transmission quality indicator. It appeared along with MOS, focusing on the video quality in network transmission, which is not applicable to TCP lossless transmission.

There is also PSNR peak signal-to-noise ratio, which is the same and will not be repeated.

I/B/P frame statistics

The original intention is to count the reception and loss of I/B/P frames of video encoding under network damage. Also due to the TCP retransmission mechanism, the I/B/P frames of video encoding are 100% transmitted and will not be lost, so the statistics are meaningless.

Summary

Traditional video quality analysis is based on lossy transmission. MOS and other indicators are intended to provide preliminary and comprehensive video quality indications for service quality comparison, and then further in-depth indicator analysis, such as analyzing media stream damage, network layer packet loss, jitter, delay and other issues, and finally find the reasons that affect user experience and solve them.

However, due to the particularity of HTTP ABR, there is no image damage, and network problems such as packet loss, jitter, and delay cannot affect the MOS index. In HTTP ABR services, the main problems that really affect user experience due to network damage, such as buffer waiting time, waiting times, and video bit rate reduction, cannot be reflected.

So what kind of video quality measurement system does HTTP ABR service need?

3. What kind of indicator system is needed to measure HTTP ABR services?

The HTTP ABR video transmission quality measurement system is divided into three levels. Spirent's test solution provides the test methods and indicators accordingly:

User perception level

Adaptive Streaming Score

Spirent provides an indicator Adaptive Streaming (AS) score designed specifically for HTTP ABR to comprehensively evaluate user experience. The AS score indicates the proportion of users who receive the highest bit rate stream and continue to play. The range of AS score is 0-100. In extreme cases, "0" means that all users are at the lowest bit rate, and "100" means that all users are at the highest bit rate that the server can provide.

This indicator comprehensively indicates the actual user perception: bitrate includes image details such as resolution, frame rate, color scale, and clarity, and whether the video is played continuously also reflects the transmission conditions such as delay, packet loss, and jitter caused by network and server reasons. AS reflects the QoE of users in HTTP ABR services. This indicator is convenient for testers to use as an entry point for test analysis. It is also convenient to compare different test results. [page]

In the example below, the video is encoded into multiple bitrates, with the lowest bitrate being 64K and the highest bitrate being 1.5M. At the beginning, users were concentrated on the lowest bitrate of 64K. As time went on, more users jumped from low bitrate to high bitrate videos. After one minute of playback, all users were using 1.5Mbps bitrate videos, and the corresponding Adaptive Streaming Score also rose from 0 to 100.

Figure 3. Adaptive Streaming Score

Media service layer

Adaptive Streaming Buffering Wait Times

Online HTTP ABR media stream Buffer waiting time, Buffer waiting time refers to the time during which the video is in a loading state with a static image.

Adaptive Streaming Avg. Fragment Response & Download Time

The average response time (from sending GET to receiving the first data byte) and download time (from receiving the first byte of data to the last byte of data) of the media file fragment. Statistics show the sum of the two times, and check which video bit rate segment the file fragment belongs to, and take the average of all responses and download times of the bit rate segment. This indicator indicates the response and download time of file fragments in a certain bit rate segment.

Adaptive Streaming Active Video Channels

displays the distribution of online HTTP ABR media streams in various bitrate segments in real time

Figure 4. Bitrate distribution of HTTP ABR media streams

Fragment Run Statistic

Abort Fragment Request Number of interruptions in downloading file fragments

Buffer Underrun Fragment The number of times the user waits for the video to be downloaded before it can be played. Except when the user has just initiated a new video request to play, this indicator should be 0 under ideal network conditions during playback. Additional Underruns indicate lag.

Pre-Cached Fragment Number of pre-downloaded file fragments

Bitrate Shift

Number of bitrate upshifts, number of bitrate downshifts, number of bitrate maintaining Total Rate Maintaining

Other statistics
Sessions, Channels, Http Requests, Manifest Requests, Fragment Requests Count Statistics

Network Level

Network traffic, TCP connection statistics, TCP SYN/ACK time statistics, Round Trip time statistics, TCP retransmission timeout statistics, TCP first data packet received time statistics, estimated server response time statistics, TCP Checksum fail, Bad header length, Bad data length, Duplicate, Out of sequence, Timeout statistics and other network parameters to analyze jitter, delay, packet loss, error packets and other problems at the network level. [page]

ABR Scores measurement system is becoming an IETF standard .

The complete set of ABR measurement indicators designed by Spirent for HTTP ABR services is an industry-leading measurement system and has become a new benchmark for the measurement of this service. It has been submitted to IETF and will soon become an IETF standard.

Note 1: Spirent is an important member of the Internet Engineering Task Force (IETF Internet Engineering Group) and has developed many important standard documents in the measurement field, such as RFC 2544.

The existing measurement standards of standard organizations such as ITU are mainly aimed at lossy transmission application scenarios. There is currently no published standard for OTT Internet services such as HTTP ABR.

Appendix A: HTTP ABR transmission mechanism description

Figure 8. HTTP ABR video distribution mechanism

The video source content is encoded by the encoder to form video files with different bit rates. A video file contains a string of file segments and corresponding lists. The client selects the video file with a bit rate to download based on the download rate. The following takes a video file named sample played on the Apple HTTP Live Streaming server as an example to illustrate its bit rate selection mechanism. See Figure 9.


Figure 9, Bit rate selection mechanism

The client sends a GET request to the server to obtain the sample.m3u8 file list. The server replies with 200 OK and sends the file list to the client. The file list contains several playback bit rates that the sample video can provide.

The client decides whether to start with the minimum bit rate, the maximum bit rate, or the middle bit rate to obtain the video based on its own set strategy. In this example, the minimum bit rate is set to start, so the client requests the server for a file list with a 64K bit rate. The server replies with a file string list of a 64K bit rate video.

The client requests the first file segment TS file based on the received file string list.

After a certain time interval, the client automatically calculates the download rate of the first file segment, and finds that the current download rate is high, for example, the download rate reaches 500Kbps. Then, based on the bit rate list obtained for the first time, it changes to requesting a 256K file list from the server, and the server returns a file string list of 256K bit rate videos.

The client requests the second file segment TS file in the 256K file string list .

After a certain time interval, the client calculates the download rate of the file segment again and decides whether to change the bit rate.

Note: Standard versions referenced by the client
• Microsoft IIS Smooth Streaming Client 1.1
• Apple HTTP Live Streaming draft-pantos-http-live-streaming-06, IETF
• Adobe Flash Video Specification 10.1