LabVIEW Data Logging and Storage (V)—TDMS Files

TDMS (Technical Data Management Streaming) files are a type of binary record file that NI mainly promotes. They combine multiple advantages such as high speed, easy access, and convenience. They can interact seamlessly between various data analysis or mining software of NI, and can also provide a series of API functions for other applications to call.
The logical structure of TDMS is divided into three layers: file, channel group, and channel, as shown in Figure 25. Specific properties can be attached to each layer. Programmers can use these three logical layers to define test data very conveniently, and can also retrieve data at any logical layer, which makes data retrieval orderly and convenient to access. http://zone.ni.com/devzone/cda/tut/p/id/5696 .

• ToC Bitmask: This is a 32-bit integer data segment, which indicates whether the segment contains meta data or raw data.
• Version number: Indicates the version of the segment, which ensures compatibility with some older TDMS file versions.
• Next segment offset: indicates the offset byte of the next segment.
• Raw data offset: indicates the offset byte of Raw data.
• Meta Data: Three levels of attribute storage fields.
• Raw data: The actual raw data storage field.

This physical structure determines the random access characteristics of the TDMS file. When the programmer only wants to read the raw data and does not care about the attributes, the raw data offset can be used to directly obtain the raw data information without paying attention to the Meta Data information. As mentioned earlier, when adding information to TDMS, a new segment is actually added. Therefore, there is no need to care about the content before the segment, which ensures that the speed of writing the TDMS file has nothing to do with the size of the TDMS file, ensuring high-speed access. It is worth noting that the segments in the TDMS physical structure have nothing to do with its three-level logical structure. It is possible that a channel corresponds to multiple segments, or that a segment contains multiple channels. When data is written to the hard disk, a segment is generated, and when data is read from the hard disk, the content of each segment is read out. Figure 25 Physical structure of TDMS file The TDMS file format can be used in LabVIEW, LabWindows/CVI, Signal Express, and DIAdem, and can also be called in Excel or Matlab. In LabVIEW, the TDMS file operation functions are in Programming>>File I/O>>TDM Streaming palette, as shown in Figure 25. Its usage is relatively simple, and only a brief explanation is given here. For specific examples, please refer to the Examples that come with LabVIEW.

• TDMS Open/Write/Read/Close: Similar to the functions of text files and binary files, programmers only need to remember the three-level logical structure of TDMS files.
• TDMS List Contents: Lists the names of the Groups and Channels in the TDMS file, or lists the names of the Channels contained in the specified Group.
• TDMS Set Properties: Set the properties of a TDMS file, channel group, or channel.
• TDMS Get Properties: Gets the properties of a TDMS file, channel group, or channel.
• TDMS Flush: Refreshes the data of all TDMS files in memory, that is, immediately writes all TDMS data to the hard disk (creates new segments).
• TDMS Defragment: Defragments the storage in the TDMS file. When the TDMS file is large, this function can improve the performance of data access.
• TDMS Create Scaling Information: Creates scaling information for data in TDMS. When the collected raw data needs to use linear, polynomial, or various sensor conversion relationships such as y=ax+b, this function can be used to directly change the data values ​​of certain channels in the TDMS file (the change is irreversible and the scaling information cannot be deleted).
• TDMS File Viewer: This VI can list the properties and data in a TDMS file in a tree format, making it easy for users to view.

Figure 25 TDMS function palette
After writing the TDMS file, LabVIEW will automatically generate two files: *.tdms file and *.tdms_index file. The former is the data file (or master file), and the latter is the index file (or header file). The biggest difference between the two is that the index file does not contain raw data information, but only contains information such as properties, which can increase the speed of data retrieval and facilitate the search of TDMS files. This file is automatically generated and does not require programmer intervention.
LabVIEW 2009 further improves the support for the TDMS format, upgrading from version 1.0 to version 2.0, and improves the support for high-speed streaming applications and DAQmx application support, which is at least 4 times faster than the previous version of TDMS files. Of course, TDMS also has some disadvantages. For example, the speed is not as fast as the win32 streaming API, it does not support deleting a channel or channel group, and it only supports Windows operating system and real-time platforms such as VxWorks and Phar Lap. In general, the TDMS file format takes into account multiple factors such as speed, logical organization, and ease of use, and is a very good choice for data storage.