This year's Double Eleven finally doesn't need to "fight" anymore

Author | Li Yangxia

Editor | Lin Juemin

In the past, “Double Eleven” was an era of competing for traffic, but now it is moving towards an era of competing for experience.

You may have noticed that the "good news-style real-time transaction volume" on Double Eleven has disappeared, and major e-commerce platforms have not released real-time transaction data. Despite this, Tmall's transaction volume on Double Eleven this year was still 540.3 billion.

From the longer purchasing time, we can also see that this year, merchants are no longer competing for traffic but services that enhance consumers’ experience, satisfaction, and happiness.

The most interesting thing is not only that merchants don’t have to compete on Double Eleven Day, consumers are no longer crazy and have become more rational, but also that the OceanBase database no longer has to compete.

Yang Chuanhui said: "In the past, all employees had to work on Double 11, including R&D staff, but now everyone can sit down and drink tea."

On the eve of Double Eleven 2021, Leifeng.com had an in-depth exchange with OceanBase CEO Yang Bing and CTO Yang Chuanhui on OceanBase's development history, capability advantages, future core strategic goals, and the overall development trends of the database industry.

At the same time, we can also see why OceanBase can go from "all staff prepared" to "unattended".

It took OceanBase nearly four years to go from 0 to 1.

"Until 2015 it was a very difficult period for us," Yang Chuanhui couldn't help but sigh.

In 2010, OceanBase was established, and the first project was Taobao's "Favorites". But it was not easy to do this.

OceanBase CTO Yang Chuanhui said, "At that time, a user's favorites had a lot of items. According to the past tradition, a user could collect up to 4,000 pieces of information, and each piece of information needed to obtain the latest dynamics in real time, such as product offline, price, popularity changes, etc. Traditional databases need to associate the two information into two tables, but it is difficult to achieve if the data volume is very large, and even MySQL cannot solve it."

It can be said that in the first few years of version iteration, OceanBase was designed to solve specific problems that other databases could not solve.

The OceanBase database requires a six-month trial by the financial system to determine whether data is lost, and the financial business asks, “How can I let you use it if you don’t prove that you haven’t lost data?” This is equivalent to “How can you prove that your mother is your mother?”

On Double Eleven in 2014, OceanBase got a rare "opportunity". At that time, Alipay already had billions of user transactions a year. Before that, Alipay only dared to hand over 1% of the transaction traffic to OceanBase. But Oracle could only meet 90% of Alipay's traffic. What to do? This required OceanBase to take over 10%, and Lu Su, the CTO of Ant Group at the time, also supported this. Facts have proved that OceanBase has withstood the test.

"The head of R&D jumped up with excitement," Yang Chuanhui said proudly. "Before, people thought you were bragging and your team was not good enough, but now they look up to you and think you can do anything."

Journey: 2015 to 2017

2015: OceanBase 0.5 was launched in Alipay’s transaction and payment libraries. In the same year, OceanBase was launched in MYbank, becoming the world’s first distributed relational database used in a core financial business system. OceanBase processed 100% of transaction data chains and 50% of payment data chains during the Double 11 Shopping Festival that year.

2016: OceanBase 1.0 was launched on the Alipay accounting system, supporting the peak payment volume of 120,000 transactions per second during the Double 11 shopping festival of that year.

2017: OceanBase released version 1.4, adding dump and hierarchical query functions. During the Double 11 Shopping Festival that year, all core systems, including the entire accounting database, were 100% run on OceanBase, setting a record of 42 million times/second database processing peak.

It is worth mentioning that by 2017, OceanBase had replaced Alipay’s core transaction and payment scenarios, as well as payment tools within the ecosystem such as Huabei, Yu’ebao, and recharge.

3.0 era: maximize performance

2018: OceanBase 2.0 was released, supporting MySQL/Oracle dual compatibility mode. It is the first native distributed database that supports stored procedures. In OLTP scenarios, the performance is improved by more than 50% compared to version 1.4, and the storage cost is reduced by 30%.

2019: OceanBase 2.2 was released, outperforming Oracle in the TPC-C test and achieving a peak data processing speed of 61 million transactions per second during the Double 11 Shopping Festival.

So far, OceanBase covers the core business segments of Ant Group and has supported 100% of the group's core links. Billions of different SQLs run in Ant Group's OceanBase cluster every day, carrying hundreds of PB of data and over one million server cores.

Yang Bing pointed out a key point, "A good database is created by trial and error."

The important development history of OceanBase since its inception in 2010 is reflected in the data impact of Alibaba Taobao, Tmall, Alipay and other business scenarios every Double 11. This is not only a review of OceanBase database performance, but also the best engine for rapid database technology innovation.

There is a saying in the payment field: "Accounting is the jewel in the crown of the payment system."

If one system can be applied to the accounting system, it means that your system can be applied to all systems.

The accounting system requires two steps: one is to record the operation, that is, who transfers money to whom, and the other is to record the actual value of the account, that is, how much money was transferred, one party should add and the other party should subtract. It seems simple, but it is extremely difficult to efficiently handle massive amounts of data and the database requirements are extremely high.

The challenges that Double Eleven poses to databases have also gradually evolved from the initial challenge of storage and computing capacity to challenges in performance, stability, and efficiency intelligence. In summary, OceanBase currently has the following core capabilities:

Data consistency: also known as data correctness. The core competitiveness of OceanBase is stability and reliability. The premise of ensuring stability and reliability is a completely correct system, which is very difficult. OceanBase has been applied to core scenarios such as "Double Eleven" since the earliest days. Therefore, it not only does database-related things, but also does many database-related non-functional things, which other databases do not have. OceanBase will verify the data during the operation of the database, so once a problem occurs in the system, it can be discovered quickly.

Continuous and stable operation: OceanBase can run continuously for 8 hours without jitter in system performance. TPC-C is the world's most credible test standard for the combination of functions and performance of online transaction processing databases. TPC-C requires continuous and stable operation without jitter for eight hours, and OceanBase has achieved this, running stably for 8 hours continuously with a jitter of less than 0.5. In addition, OceanBase broke the world record with an online transaction processing performance of 707 million (707,351,007) tpmC. One of the tables has 20 trillion data volumes. OceanBase must ensure that all databases are consistent and all machines are running to ensure the correctness of the transfer. "There cannot be a situation where one side transfers money but the other side does not receive it." The advantage of continuous and stable operation allows users to shop smoothly in extreme scenarios such as "Double Eleven".

Lossless disaster recovery: It is reported that when a machine, city, or computer room fails, OceanBase can ensure that no data is lost. Because the Paxos protocol is used, it can achieve disaster recovery without any problems, and proposes three locations and five centers, and then four locations and eight centers, and eight locations and nine centers. Especially in the financial system, no data can be lost. Only when commercial databases in other industries are fully certified will they be used in the financial industry.

Unlimited capacity expansion and elastic expansion: The capacity on the "Double 11" day will be dozens of times that of normal days, and the distributed database can be infinitely expanded. On that day, the team will quickly deploy servers and quickly transfer data and services to the cloud. After the big promotion, the data and services will be moved down from the cloud. OceanBase can be quickly added and removed every day, and the elastic expansion and reduction process involves nearly 500,000 change operations. Yang Chuanhui said: "Alipay's million-level replica capability means that it can support 1 million peak payments per second, and 1 million payments is close to 100 million QPS (query rate) for the database." The key point of OceanBase's ability to support such a large query rate is through two-level elasticity. The first level is sharding, and the second level is to use OceanBase's partition table capability for automatic expansion. First, the first-level sharding is divided into 100 copies, and then the partition table capability of OceanBase is used to split another 100 copies, a total of 10,000 copies, so that each copy can be automatically elastic on a single machine.

High performance and low cost: OceanBase sets up a project every year to specifically improve performance and optimize costs based on the needs of "Double Eleven". The read and write scenario performance in 2021 increased by 61% compared with the previous year. At the same time, the storage cost was greatly reduced to only about one-third of MySQL.

"Double Eleven" has accumulated many key features for OceanBase, allowing it to be applied to more scenarios and laying a solid foundation for OceanBase.

OceanBase has now been updated to the latest version 3.2, but for "Double Eleven", even if it is not the latest version, it can already support the entire "Double Eleven" traffic.

In the past one or two years, more and more domestic databases have begun to gain favor with government and enterprise customers.

This is a positive phenomenon. Facing powerful competitors like Oracle on the international stage, it is trying to justify the efforts made by domestic database companies in the past 40 years.

On June 8, 2020, Ant Group's self-developed database OceanBase announced that it would operate independently as a company. In the more than one year since its independence, OceanBase has made rapid progress in both technical research and business advancement: it broke its own world record in the second TPC-C benchmark test, and was launched in the core business systems of institutions such as Industrial and Commercial Bank of China, Bank of Nanjing, PICC Health Insurance, and Bank of Xi'an...

But this is just a small step. For OceanBase, financial-grade business scenarios are the Everest for verifying database performance, and it is necessary to start with financial benchmarks, but its goal is to expand to a more open and broader general database.

At present, replacing the core database system of traditional industries is a need of the times under the background of domestic substitution, and it is also an urgent demand for new business forms of government and enterprise customers.

Traditional industries are in urgent need of upgrading their core systems and require a stable, secure, cost-effective database that is highly compatible, highly scalable, highly available, easy to manage, and flexible to deploy.

According to OceanBase CEO Yang Bing, the difficulties faced by enterprises are mainly the following:

Technology cannot be changed: Core business systems were invested and built early, and most of them are deeply bound to database capabilities. Migration and transformation work is large, risky, and time-consuming.

Service cannot hold up: In the era of mobile Internet, users’ consumption habits have changed. The core business systems of banks have changed from meeting 5*8 consumption demands to meeting 7*24 consumption demands, which poses a challenge to high availability. At the same time, institutions of different sizes have different infrastructure conditions and transformation ideas, so targeted solutions are needed.

Multiple management links: The types of business systems are increasing. 50% of users believe that operation and maintenance management becomes more difficult as the number of database instances continues to increase.

Business cannot run: The initial investment and construction scale of the core Internet system is small, but it develops rapidly. Once it grows explosively, there is not enough time to reconstruct the application and design the architecture. Therefore, the underlying database capacity and computing power are difficult to keep up with the speed of business development. The system cannot run, and high concurrency scenarios are required.

In fact, it can be seen that the above problems have been a headache for enterprises for a long time. How to reduce the risk of core system transformation and reduce the cost of transformation has become a problem that most enterprises will encounter. The solution to the above problems is nothing more than migrating to other databases or expanding capacity, which is not only extremely risky but also very costly. Once all migrations are completed, all software must be completely replaced, which is subversive in terms of technology requirements and is also undesirable.

This task is extremely difficult and challenging, with high initial investment and high risk. How can OceanBase do this?

In Yang Bing's words: "OceanBase is just standing on the shoulders of traditional databases, subverting the old market and replacing the existing market, while creating new tracks and spaces."

OceanBase has invested a lot of money and effort to be compatible with MySQL and Oracle traditional databases. Due to historical reasons, Oracle is the most popular database in the world and is widely used. As the slogan of domestic independence has been raised, many enterprises and institutions have begun to use open source MySQL, which has led to the current situation.

OceanBase has helped a leading insurance company to smoothly transition to OceanBase in its core system. A user lamented: China has no shortage of various database companies, whether commercial or open source, but what it lacks is a database that is independently controlled and developed to truly replace MySQL and Oracle.

1. "Integrated architecture" is the best choice for OceanBase

For OceanBase, being a general-purpose database is the ultimate goal. To this end, the team will look at this from several technical perspectives: one is "integrated architecture".

The general trend of the world is that after a long period of unity, there will be division, and after a long period of division, there will be

For a long time, OLTP (transaction processing/online transactions) and OLAP (online analytics) have each dominated the market. In many enterprises, these two types of tasks are handled by different teams, which results in high management and maintenance costs. However, as more and more enterprises move to the cloud, a large amount of data is pouring in, and the processing capacity of a single-machine database is far from meeting the needs for real-time data updates and queries. The demand for both capabilities is growing, so HTAP databases came into being.

With the influx of huge traffic during the "Double 11" shopping festival, the transaction data must flow into the offline data warehouse, and analysis may take a long time to produce results, at least several hours, and merchants cannot respond quickly. The HTAP architecture can solve both OLAP and OLTP business scenarios, allowing merchants to better adjust promotional strategies for activities in real time on Double 11.

Therefore, OceanBase proposed the "integrated architecture", opening up a new model for enterprise-level databases, combining the technical advantages of native distributed frameworks and classic centralized databases (HTAP), so that the database can handle more complex scenarios, leaving "complexity to the database and simplicity to the user".

"Although the underlying architecture of the 'integrated architecture' is still natively distributed, it can take into account both centralized and distributed systems. That is, it can retain the advantages of strong single-machine performance of centralized databases while also taking into account the scalability, expandability, cost-effectiveness, and high availability of distributed databases. Therefore, it is compatible with Oracle and MySQL. This is good news for many users, because it means that the databases that have been used for many years are gradually facing a series of problems such as capacity and price. If they can be smoothly migrated from Oracle or MySQL to OceanBase, this must be a better choice."

The HTAP technology architecture helps OceanBase be applied to more scenarios, and OceanBase will not only be used in edge scenarios but also in more core scenarios, thus opening up a broader ecosystem for the database.

2. Adhere to ecology and open source

Yang Chuanhui told Leifeng.com: "The later the database is, the more it competes with the ecosystem. Therefore, in June this year, OceanBase decided to open source" and open source 3 million lines of code for free. Only by building the community together, improving tools together, and embracing users can we establish a database ecosystem."

Only when more people work together can we test the system in more scenarios. A good database must be polished in different scenarios.

The HTAP capability of the current enterprise version OceanBase 3.2 has been greatly enhanced, with TP performance increased by 30% and AP performance increased by 6 times. It also supports hosting, releases the KVREDO interface, is compatible with the MySQL 5.7 protocol, and opens the operation and maintenance management openAPI......

OceanBase needs to not only continuously optimize core scenario capabilities, but also continuously optimize HTAP performance and continue to open source and build an open source community; it needs to not only serve Alipay's payment transaction system, but also gradually empower more scenarios, covering financial scenarios and even the entire industry, opening up a broader ecosystem for the database.

In the future, OceanBase's entire deployment architecture will move from private cloud to hybrid cloud and even multi-cloud deployment, from financial scenarios to broader general scenarios, and from domestic to foreign. OceanBase will become more and more a universal, neutral, multi-cloud database and become more open.

3. Native distributed databases are definitely the future

OceanBase CEO Yang Bing said: "Native distributed databases will definitely lead the trend of future databases. We predicted this trend as early as 10 years ago, and now we have verified the correctness of this prediction step by step."

There was no precedent for native distributed databases in China 10 years ago. At that time, integrating distribution into the database was a groundbreaking attempt and innovation.

As digital transformation enters the deep water zone, enterprises are accelerating their pace of cloud migration. In the future, the amount of data in the cloud will surge. IDC predicts that the amount of global data is experiencing an explosion, growing from 33ZB in 2018 to 175ZB in 2025. One of the important factors driving this growth is that cloud computing has the advantages of elastic expansion, on-demand use, and pay-as-you-go. Cloud data centers are becoming new data repositories for enterprises. The surge in data volume in the future will bring great challenges to traditional databases, and the advantages of native distributed databases can solve these problems well.

In addition, changes in application technology architecture have given rise to the need for capacity expansion. Applications have gradually evolved from single machines to clusters and then to microservices and clusters. At this time, the pressure of capacity expansion will be transmitted to the storage layer and database layer. The distributed architecture has good scalability and can solve this problem well. This is why OceanBase has three locations and five centers, which can be globally expanded and easily cope with Double Eleven.

At the same time, the hardware architecture has changed from a large, vertical architecture to a small, parallel architecture, and the hardware aspect has also begun to extend to distributed.

In the past, people believed that native distributed databases could only be applied to edge scenarios, and only breakthroughs in core scenarios could be considered real success. Native distributed architecture must be the direction of future database development.

OceanBase has been applied in core scenarios such as the financial industry during the "Double Eleven" event, which means that the native distributed architecture can be applied to both edge scenarios and core scenarios.

Therefore, OceanBase is determined to move towards a native distributed database.

"Applications are the primary driving force of databases." After eight years of experience in the "Double Eleven" payment scenario and the Alipay financial scenario, OceanBase urgently needs to be tested in more core scenarios and move towards a general database. Since its open source, companies such as Ctrip, Meituan, bilibili, and Kuaishou have conducted in-depth practices. OceanBase is also constantly optimizing and upgrading new versions. It has now been upgraded from the open source 3.0 version to the 3.1.1 version.

At that time, OceanBase founder Yang Zhenkun’s words on Double Eleven in 2014: "If something goes wrong later, we will jump from here!" were still ringing in my ears.

Now that times have changed, OceanBase no longer needs to work so hard.

Yang Chuanhui told Leifeng.com: "The best situation in the future would be to have no one on duty."

It can be foreseen that OceanBase's future direction will be intelligence, automated intelligent database operation and maintenance, and the use of AI for automated analysis and decision-making of problems.

The future is still unknown, so let’s wait and see.