Storage architecture competition: Who will be the choice for high-definition surveillance?

With the evolution of the IT trend of security, the digitalization ratio of video surveillance systems, which is an important part of security, is getting higher and higher. At the same time, the involvement of network equipment manufacturers, video surveillance storage equipment manufacturers, and digital codec manufacturers has promoted the development of various components of video surveillance systems (acquisition, encoding, transmission, storage) towards "intelligentization, grid, and IP".

It should be said that the birth of network monitoring has solved the problem of traditional monitoring effects, changed the traditional connection method, and made it simpler and more convenient, but at the same time it has derived a larger storage demand and higher bandwidth satisfaction. For video monitoring data, since network video monitoring is based on the transmission of audio and video streams on the network, the performance of the video monitoring management server will be greatly affected in the case of a large amount of audio and video stream transmission. The more data, the more complicated the scheduling and cache management of the video monitoring management server will become. There will often be multiple cameras on a single server, and multiple servers on a back-end storage device. Frequent access to storage devices and the storage requirements of multiple cameras and multiple servers require a large amount of resources, affecting its processing capacity.

According to a survey in the security industry, the market share of DVR hard disk storage, which once occupied the mainstream security market, has now dropped to 45%. Traditional IT storage methods (including DAS storage, NAS storage, and SAN storage) share the market share with it, also occupying 45%.

High-definition surveillance requirements for data storage

However, with the development of high-definition surveillance, the substantial improvement of front-end image quality has also led to a rapid growth in back-end storage surveillance data. Taking a 720P high-definition camera as an example, its hourly video data can be compressed to about 3GB. Generally, there are dozens or even hundreds of surveillance systems on the road. According to relevant regulations, surveillance videos must be saved for one month (30 days). Assuming that a surveillance system has 100 channels, it can be calculated that:

3GB/hour*24 hours*30 days*100 channels = 216,000GB

In other words, the minimum capacity required for a high-definition video surveillance system with hundreds of channels is 216TB. Taking into account global data redundancy, future data expansion, and automatic data recovery, the system design requires at least 220TB of storage space.

In addition to large capacity, video surveillance storage systems also have the following requirements:

During video acquisition, the video file format generally does not change, and the bit rate remains constant. The frame rate of the video image is generally 25 frames/S. That is to say, in the storage read and write operations, high storage bandwidth must be guaranteed, otherwise frame loss will occur during image acquisition or playback. Therefore, the video surveillance system requires the storage system to have not only large bandwidth but also stable bandwidth under high concurrent IO conditions.

The duration of data read and write operations is long. Since cameras generally work 24/7, the duration of data write operations is also 24/7. The data needs to be obtained in time during later playback. Therefore, the storage system is required to maintain 24/7 uninterrupted operation even when the hard disk is damaged or some storage nodes are down.

Video surveillance systems generally have many cameras and long video image storage time, so the storage capacity requirement is huge, and as the image storage time increases, the storage capacity requirement increases linearly and explosively. Therefore, the video surveillance system storage must support large capacity and have high scalability to meet the needs of long-term large-capacity video image storage.

The shortcomings and defects of traditional IT storage architecture

High-definition video surveillance storage systems need to meet a series of requirements such as efficient management of massive data, efficient retrieval, and high reliability. However, traditional IT storage methods (including DAS storage, NAS storage, and SAN storage) seem to be incompetent in these aspects for various reasons:

DAS: This solution uses HBA cards to directly connect application servers and fiber optic disk arrays, and does not provide data sharing capabilities. If multiple applications need to share the same data, it often takes a lot of time to migrate the data, resulting in multiple copies of the same data in the environment, and the synchronization between multiple copies of data is difficult, wasting a lot of manpower and material resources.

SAN: This solution uses a storage fiber network to connect application servers and fiber disk arrays, which is more flexible and scalable than DAS. However, SAN still does not have the ability to share data, and due to the high price of SAN systems and equipment compatibility issues between different manufacturers, its use in the HPC field is limited.

NAS: NAS systems that provide NFS or CIFS protocol access can provide a unified file system image to application servers, meeting the needs of data sharing between multiple application servers. However, since traditional NAS often provides file-level storage space in the form of a single server, its limited IO bandwidth and scalability make it difficult to meet the large number of concurrent read and write requirements of HPC systems. In addition, NAS has poor support for file system locks, and multiple clients cannot read and write the same file at the same time, resulting in poor collaboration between clients.

SAN+NAS: This solution has the scalability of SAN and the data sharing capability of NAS to a certain extent, but it still has obvious defects. All data I/O of the application must be performed through a single NAS server, so the I/O bandwidth of the entire system is limited to the bandwidth of the NAS server, which is still difficult to meet the needs of the HPC environment.

Cluster storage: the dawn of high-definition surveillance storage?

Faced with such a situation, although everyone places their hopes on cloud storage, it is still too early for cloud computing to be implemented, and the time for cloud storage to be applied to households is still to be discussed. How can video surveillance users store large amounts of high-definition surveillance data? Cluster storage seems to be the answer for storage manufacturers to solve the large-capacity and high-stability problems of video surveillance systems.

Let's first explain the concept of cluster storage. Cluster storage refers to aggregating the storage space of multiple storage devices into a storage pool that can provide a unified access interface and management interface to the application server. Applications can transparently access and use the disks on all storage devices through this access interface, which can give full play to the performance and disk utilization of storage devices. Data will be stored and read from multiple storage devices according to certain rules to obtain higher concurrent access performance.

The advantages of cluster storage are mainly reflected in improving the overall performance of parallel or partitioned I/O, especially workflow, read-intensive and large file access, and reducing overall costs by using lower-cost servers.

In October 2008, a company conducted performance tests on cluster storage architecture and traditional storage architecture using the same hardware environment in the computer room of China Film Group Corporation, and used the iozone tool to test the concurrent IO performance of the storage architecture.

The results show that compared with the traditional storage architecture, the cluster storage architecture has a concurrent write performance that is about 30% higher and a concurrent read performance that is about 10% higher.

In traditional storage architecture, the average I/O read and write speed of each 1TB hard disk can reach up to 15-20MB/s. Taking 60TB as an example, the I/O concurrent read and write speed of the entire storage architecture can only reach 0.9-1.5GB/s, which cannot meet the requirements of high concurrency. The CZSS storage system breaks the bottleneck of I/O read and write speed based on hard disks in the traditional sense, adopts an efficient file management mechanism, and uses nodes as the calculation unit of I/O read and write of the entire system. For the same 60TB, the CZSS system requires about 10 nodes (10 storage units), and the read and write speed of each node is more than 300MB/s, so the concurrent read and write speed can reach more than 3GB/s, which greatly meets the user's requirements for high concurrency of the system.

Moreover, during the traditional storage test, when concurrent reading and writing are performed, the client cannot perform ls (view files in the directory) operations on the traditional storage system mounted directory, that is, the traditional storage system cannot view the data in it when it is under load, so when the system has tasks, it is impossible to retrieve the result data stored in the traditional storage architecture. This phenomenon shows that the metadata processing capability of the traditional storage architecture is very low, and the system cannot provide services when there are a large number of files.

When the number of client requests is large, metadata operations of traditional storage architecture software will become unresponsive. At this time, all clients will be unable to perform operations such as opening folders and querying files. In this way, all desktop clients will become unresponsive and production work will be completely impossible. However, CZSS storage software does not have this problem. Under the same hardware conditions, the IO bandwidth of cluster system software is 30% higher than that of traditional storage architecture software. The investment cost has been reduced by more than 40%.

Conclusion

At present, the development of the domestic security market is mainly driven by some government-led projects, such as safe city construction and traffic monitoring system construction. These security systems have requirements for multiple channels and high image quality, and at the same time require the system to have the ability to process information in real time. The resulting large data storage problem is inevitable.

The security industry has transitioned from the analog era to the digital era, and the surveillance storage architecture has transitioned from DVR to NAS and SAN. It can be said that this is an era of progress. With the advent of high-definition surveillance, when traditional IT storage architecture cannot meet the requirements of high-definition surveillance storage, cluster storage - for the security industry, can this new storage method be accepted by us? True gold is not afraid of fire. We can only say that for any new thing, it takes a certain amount of time and transition period for users to accept it.