Who is the “behind-the-scenes hero” who helps Sichuan Telecom optimize its ABM storage system?

As we all know, Sichuan is not only a province with a large population, a large economy, but also a large information consumption province. Take Sichuan Telecom as an example. As of now, the number of mobile phone users of the company has exceeded 28 million; at the same time, the number of fiber broadband and IPTV users has also exceeded 14 million, making it the world's largest single IPTV operator.

However, the rapid growth of business has also put unprecedented tests and challenges on Sichuan Telecom's various support systems, especially the core billing system. Among them, the daily REDO log data archiving volume of the core network element balance management center (ABM) system in the billing system is as high as 3TB2. Such frequent and intensive data submission has undoubtedly put pressure on the storage platform of Sichuan Telecom's old ABM system.

In this context, Sichuan Telecom took the initiative and teamed up with Intel to not only reconstruct the storage platform of its old ABM system with a new Server SAN architecture, but also introduced the Intel® Optane™ SSD DC P4800X for data centers based on Intel® 3D XPoint™ technology to provide acceleration for it. The combination of this new architecture and new products has achieved a breakthrough in the storage performance of the ABM system while effectively reducing Sichuan Telecom's TCO costs.

From this perspective, the in-depth cooperation between Sichuan Telecom and Intel not only provides important reference and lessons for the upgrade and construction of the traditional ABM system infrastructure architecture, but also serves as a good benchmark for more telecom companies to explore and innovate their future business.

As the "data steward" of the telecom billing system, the ABM system plays a vital role in Sichuan Telecom's business processing. On the one hand, it stores all of Sichuan Telecom's important data such as call balance, data usage, user information, package pricing, user consumption history, etc.; on the other hand, it is also the basic platform for Sichuan Telecom to implement various cross-discounts and business integration.

After years of development, Sichuan Telecom has seen a continuous increase in both the number of users and the types of services it provides. This has also brought great challenges to the system, which we can observe from several dimensions:

First, from the perspective of user scale, Sichuan Telecom now has a total of nearly 90 million users of all types. The monthly call bill processing volume of these users is at the level of hundreds of billions, which requires the ABM system to provide stable and reliable support; at the same time, these businesses have increasingly higher requirements for timeliness, and the entire system's call billing must be completed within 1 day.

Second, from the perspective of business types, Sichuan Telecom has launched innovative products and services such as "Douyin Wireless Data Card" and "9 Yuan Free Card" to the market in the past few years, bringing more discounts and more diverse consumption choices to users. However, these new services often have the characteristics of cross-network, multi-product bundling and combined marketing, and involve a large amount of data such as data on free data, call fee sharing, and targeted exemptions generated by account discounts, which also have very strict requirements on the real-time and accuracy of the data.

Third, from the perspective of system architecture, Sichuan Telecom's ABM system was built on Oracle Real Application Clusters (RAC) in the past, but its storage environment was deployed on traditional SAN architecture storage servers and connected to disk arrays via the Fibre Channel (FC) storage protocol.

This traditional architecture is now unable to support the business development needs of Sichuan Telecom, resulting in the ABM system also facing great challenges. On the one hand, it is restricted by the traditional SAN architecture of the old system. The submission of REDO data relies on the FC storage protocol, and the maximum transmission rate provided between the storage server and the disk array is limited to 16Gbps; on the other hand, this indicator is also limited by the IO capability of the storage device. In the old system, REDO and data files are stored in the previous generation of solid-state disk products, and their read and write performance obviously needs further optimization.

Moreover, for the telecom-grade billing database application ABM system that has “five nines” requirements for security and availability, whether the REDO data can be operated efficiently, stably and securely has also become a key point in considering the capabilities of the entire ABM system.

It is not difficult to see that as the "data steward" of the telecommunications billing system, the ABM system plays a vital role in Sichuan Telecom's business processing procedures. They are not only interrelated and complex, but also in the process of program upgrades and optimizations, if the measures are inappropriate, it may "affect the entire body" and cause a system-level disaster, and the serious consequences it brings are unimaginable.

Based on this, Sichuan Telecom, after conducting rigorous technical solution demonstration with Intel's expert team, boldly and cutting-edgely adopted the "new architecture + new product" approach to complete the comprehensive optimization of the ABM system storage system.

First of all, in this optimization and upgrade plan, Sichuan Telecom adhered to the concept of innovation and took the lead in trying architectural changes, building its new generation of storage system through the Server SAN architecture based on the software-defined storage (SDS) concept.

It is understood that the new Sichuan Telecom ABM system storage system adopts a brand-new Server SAN architecture and is interconnected through the iSER storage protocol based on the Infiniband standard. The transmission bandwidth reaches 56Gbps4, far exceeding the old system.

At the same time, the new Server SAN architecture also brings excellent expansion capabilities and convenient operation and maintenance management to the Sichuan Telecom operation and maintenance team. After the system is launched, Sichuan Telecom only needs to calculate the corresponding storage and computing module requirements according to the business development steps and add them, without the need to perform tedious upgrades on the storage system. This can not only greatly reduce costs, but also effectively reduce the complexity of operation and maintenance.

Secondly, in terms of storage hardware selection, Sichuan Telecom chose the Intel® Optane™ SSD DC P4800X to ensure the safe, efficient and stable operation of REDO data. We know that REDO data is the lifeline of the ABM system storage system. In the daily work of the ABM system, REDO data will submit all kinds of transaction logs to the storage system for retention, so it also has the characteristics of high concurrency, large data volume, and high response speed requirements.

In Sichuan Telecom's deployment practice, the performance of Intel® Optane™ SSD DC P4800X far exceeded users' expectations. Data shows that compared with the Intel® SSD DC P4500 (based on 3D NAND media) also deployed in Sichuan Telecom's ABM system, its IOPS is about 10 times that of the latter. At this time, from the storage node side, its disk utilization rate only increased by about 1%. In the test of FIO (an IO testing tool), when the DC P4500 disk utilization rate reached 40%, the DC P4800X disk utilization rate still hovered at a low value of 20%.

In terms of the log file sync wait event indicator that Sichuan Telecom is concerned about, the Intel® Optane™ SSD DC P4800X has broken through the 1 millisecond mark and reached an astonishing 0.7 milliseconds, which is only half of the old system. At the same time, when facing the pressure of massive data submission, the DC P4800X not only leads the way in IOPS indicators, but also has a much smoother jitter rate than the DC P4500, which is exactly what the OLTP business attaches great importance to, and has become one of the cornerstones of the stable and efficient operation of Sichuan Telecom's ABM system.

In addition, another "killer feature" of Intel® Optane™ SSD is that it provides Sichuan Telecom's ABM system with excellent drive write times per day (DWPD) performance. Currently, the daily REDO data of Sichuan Telecom's ABM system is 6TB after adopting the ASM NORMAL redundancy solution. If the capacity of a single drive is 350GB, the DWPD is 5.7. The DWPD design of Intel® Optane™ SSD DC P4800X up to 30 allows users to use the SSD with confidence throughout its life cycle, completely eliminating worries.

In this regard, Li Hua, supervisor of the Enterprise Information Department of Sichuan Branch of China Telecom Co., Ltd., said: "The billing system, including the ABM system, is an important cornerstone for China Telecom to meet the full business operation needs, respond to the intelligent operation model, and build a unified and integrated billing strategy. The performance of its core storage system will directly affect the work efficiency of the entire billing system. The introduction of the new Intel® Optane™ SSD DC P4800X not only significantly improved the storage performance of our system, but also performed well in stability, which also made us very satisfied."

It is worth mentioning that with the official issuance of 5G commercial licenses, Sichuan Telecom also stated that it will actively explore in the fields of smart networks, smart platforms, smart applications, smart devices, smart home appliances, etc. in the future, promote the innovative development of digital core industries such as big data, artificial intelligence, 5G, ultra-high-definition video, electronic information, and digital cultural and creative industries, promote the digital transformation of Sichuan industries, and fully welcome the arrival of the "5G Intelligence +" era.

Obviously, this also brings a new round of challenges to Sichuan Telecom's ABM system. Therefore, Sichuan Telecom and Intel's expert team will continue to conduct more in-depth research on the further application and optimization of new architectures and new products in Sichuan Telecom's system.

Specifically, in terms of hardware configuration and deployment, since test data shows that the Intel® Optane™ SSD DC P4800X will cause certain performance losses when used remotely, the Intel team will help Sichuan Telecom deploy this product for local use to improve performance.

At the same time, given that the NVOF (NVMe over Fibre) storage protocol theoretically has lower system overhead and better performance than the iSER storage protocol, the two parties will also try to apply the NVOF protocol to the Sichuan Telecom ABM system production environment.

In terms of software optimization, the two teams will also plan to introduce the polling mode in the Intel Storage Performance Development Kit (Intel SPDK) to eliminate the performance impact caused by the interrupt mechanism and further improve system performance. Currently, the two teams are planning to combine the NVOF storage protocol to carry out exploration and practice in this regard.

In summary, from the case of Sichuan Telecom's ABM system, we can see that Intel® Optane™ technology combines high capacity, economic practicality and sustainability in an unprecedented way, thus breaking the bottleneck of traditional storage, effectively supporting the stability, availability and security of Sichuan Telecom's core billing business system, and allowing its business to maintain continuous innovation, allowing Sichuan Telecom to achieve new breakthroughs and leaps in the new era of digital transformation.