Introduction to ABX Blind Listening Test

When we listen to music, various factors affect our actual listening experience. These factors include cables, speakers and even the software decoder itself. So how can we tell whether these factors have a positive or negative impact?

ABX testing can give us the answer. ABX testing is a common method used in audio applications to measure or compare non-quantifiable characteristics (such as sound quality) between objects of the same nature.

In this test, the tester will get three sound sources marked as A, B and X. A and B are used as references, one of them is processed and the other is not processed, for example, one is the original WAV file and the other is the mp3 file compressed from the WAV file. X is an unknown sound source, it may be A or B. The tester must compare whether it is A or B by listening. The picture on the right is a typical interface of ABX test software.

Since there are only two choices, either A or B, there is a 50% chance of choosing the right answer, so a single test cannot explain the problem. Statistical analysis is needed, that is, more than one test is needed. The usual practice is to require the tester to repeat a test 15 times, and each time a test is performed, one of A and B is randomly selected as X. Analyzing all the test results can greatly reduce the possibility of choosing the right answer by chance. From a probability perspective, the probability of choosing the right answer in one test is 50%, while the probability of choosing the right answer in 16 repeated tests is 50% to the 16th power, that is, one in 65536. In other words, if the difference between A and B is very small and the tester cannot hear the difference at all, the average result of the tester should be a value tending towards 50%. If there is a significant difference between A and B, the average result of the tester should have a significant tendency.

For example, we need to compare the sound quality of two pairs of speakers. Obviously, they should have different performances. After we conduct the test 7 times, there is a 1 in 128 chance of obtaining a false correct answer, that is, the summary result obtained after statistical analysis based on the test results is wrong compared with the actual situation. In statistics, this situation is called "type I error". The possibility of this type of error decreases with the increase in the number of tests. For every increase in the number of tests, the possibility of an error decreases by 1/2. .

Now, let's put a bracelet next to the CD player and do the test again. It can be said that this should not have any effect on the sound quality. We repeated the test 40 times, and the probability of successfully distinguishing the difference in sound quality is 1 in 2 to the 40th power. It can be said that the chance of error is very small. But if the test results show that there is indeed a difference in sound quality when the bracelet is placed and when it is not, there must be an explanation: perhaps the tester heard the sound made by the operator when he moved or put down the bracelet; or the operator's movements slowed down every time the bracelet was placed next to the player; or various other non-auditory factors caused such a result.

As for the statistics, I guess no one is interested in discussing them in depth, so let’s stop here. But from the simple explanation above, we can extract some rules for the ABX blind listening test:

1. It is impossible to verify something that does not actually exist. If you have doubts about something, such as whether a certain audio codec affects the sound quality, you should verify it through testing rather than taking it for granted based on any assumptions.

2. The test should be conducted under double-blind conditions. Single-blind testing means that you cannot tell whether X is A or B unless you actually listen. Double-blind testing means that neither the operator nor the tester knows whether X is A or B during the test to avoid possible intentional or unintentional operator hints. In other words, a third person is required to be responsible for switching. There are quite a few software on the computer that can perform ABX testing.

3. Ensure that the tester knows the results only after the test is completed, including the trial test that may be conducted before the formal test. Otherwise, the tester should know the results after each test and can decide to terminate the test at any time.

4. The test must be conducted for the first time. If not, all previous test results should be considered together.

5. The tester can reserve his opinion, that is, during the test, he can reserve his opinion if he is not sure about his judgment and ask for a pause and continue later.

6. The test process must be reproducible, and the test results obtained according to the same test process should also be consistent.

7. The timing of the test process is also an aspect to be considered. For example, in the continuous test process, the same time interval should be given to avoid the tester's judgment being affected by different timings and making guesses.

Of course, the test results are not necessarily as simple as "X is A" or "X is B". For example, in the sound quality test of MP3 encoding, the tester also needs to select the similarity of X to A and B, and calculate the sound quality effect of MP3 encoding. Other test methods can be extended from this. The figure below is an ABX test result chart of the sound quality comparison of different MP3 encoders.