As configuration software becomes more widely used in various industries, more and more people are learning about and using it, and at the same time, their requirements for the various functions of configuration software are also getting higher and higher.
Generally, configuration software uses several basic functions in practical applications, such as data acquisition, real-time data display, alarms, and reports, while historical data storage is indispensable. By automatically recording data changes during system operation, it can provide necessary data sources for future accident analysis and trend statistics, greatly improving the stability of system operation and reducing the chance of production accidents.
Currently, there are many types of configuration software on the market, employing different storage strategies. Some use external databases as historical data stores, which offers advantages such as powerful functionality and stable operation, but also significant drawbacks, including slow speed, large space consumption, and poor software integration. Therefore, many companies develop their own historical data storage functions. These custom-developed historical databases not only offer advantages such as high speed and small space consumption but also better security, unlike general-purpose databases which can be easily accessed.
CenturyStar configuration software also uses an independently developed historical database. To ensure fast data storage speed and small space footprint, CenturyStar configuration software mainly adopts the following three solutions.
The first approach is to minimize data storage, thereby fundamentally reducing the requirements for historical databases.
The configuration software records various parameters collected from the field, such as temperature and pressure. The information that needs to be saved includes variable identifiers, values, and timestamps accurate to milliseconds. To reduce storage requirements, CenturyStar employs two methods:
First, we should use the variable ID instead of the variable name. This is obvious, as the variable ID is represented by 4 bytes, while the variable name usually exceeds 4 bytes. Therefore, given the need to minimize storage space, it's the obvious choice to use the variable ID instead of the variable name for storage.
Secondly, there's the method of recording time. Each variable needs to record a time accurate to milliseconds. Time can be stored in various ways, such as as a string or as a long integer. Some people may not know how to store time as a long integer; a specific conversion method is all that's needed. Of course, storing time as a long integer reduces storage space. However, there's another issue: a long integer cannot store all the values including year, month, day, hour, minute, second, and millisecond. The method used in CenturyStar is to use the date as the name of the historical data file. This way, the time in the file only needs to represent hours, minutes, seconds, and milliseconds.
The second method is to create an index. This method is also widely used in general databases, but due to the special characteristics of the industrial control industry, such as the fact that data is stored periodically and the order of data in the file is regular, this method of creating an index can play a greater role.
CenturyStar configuration software stores two types of historical files in a project: an index file ending in IDX and a data file ending in LGH. The IDX file records the precise location of historical data within the LGH file, with minutes as the smallest unit. When querying data, based on the query time provided by the user, the exact location of the data in the IDX file can be quickly determined, regardless of the size of the LGH data file, achieving the goal of fast querying.
The following diagram illustrates a data query process:
The final strategy is data compression. Industrial control systems sometimes need to store large amounts of data, sometimes with a change rate within 100 milliseconds, and the storage time needs to be long. To preserve this data while ensuring minimal space usage, data compression is essential. CenturyStar uses compression algorithms provided by the zlib library, a free compression library with good compression performance and fast decompression speed. It is currently a relatively mature data compression method.
The effects of using these three methods are obvious. Tests showed that in a project with 10,000 variables, where 4,000 variables were recorded every 200 milliseconds, querying 10 minutes of data took no more than 200 milliseconds; querying 4 hours of data took no more than 2 seconds.
In practical applications, some shortcomings are also observed. For example, in projects with many variables, the required amount of stored data can be very large, resulting in large historical files. Even with index files, query speed will decrease. An improved approach is to split a single file into multiple files. For instance, data with variable IDs less than 5000 can be stored in one file, data with variable IDs between 5000 and 10000 can be stored in another file, and so on. This reduces the file size and thus improves query speed.
