Intelligent data processing tools for small and medium-sized enterprises

summary :

This paper analyzes the current status of MIS applications in domestic enterprises, analyzes the application requirements of enterprise information systems in the future, and introduces the design of a business intelligence tool software based on the current situation and requirements.

Keywords : MIS DSS Business Intelligence OLAP Data Warehouse

1. Current status of enterprise MIS application

With the continuous deepening of domestic enterprise reform, enterprise management has also deepened. Enterprise leaders, managers and professional and technical personnel have become more fully aware that information resources can only be more effectively utilized through the processing of information systems. The construction of information systems plays an increasingly important role in social and economic activities. After decades of exploration and practice, the construction of information systems in various enterprises and institutions in my country has also developed from single applications and small system applications to large-scale and networked directions.

However, we also clearly see that there are still many unsatisfactory aspects in the current MIS construction, such as: many MIS are often limited to statistical analysis of data, and there are few works related to prediction, decision-making and optimization. As a result, they can only replace part of the heavy manual labor and fail to give full play to the benefits of MIS. Some common problems that the current MIS must solve are:

1. The data involved are of many types and large amounts, and the data are often scattered across various business departments. Each business department has its own unique processing requirements for information collection methods, analysis methods, statistical caliber, report output formats, etc.
2. It has obvious time-varying characteristics. The continuous reform of management system and the continuous change of economic form have caused frequent changes in information processing methods. The data collected by various business departments, the analysis and statistical methods of basic data, and the report output format will have new changes and emphases as the situation develops. These modifications must be completed by professional programmers in a long time, which is far from meeting the requirements of flexible and random query of system data, wasting the intrinsic value of precious data. Moreover, due to the constant modification of the system, the stability is greatly reduced.
3. Inheritance and development of new and old systems. Some companies already have MIS, but the old system cannot meet the business needs of each business department. In addition, most early systems use a stand-alone environment, and the application platforms and database systems used are also different. How to inherit these existing, scattered, and independent information and use them effectively is also an important topic in the construction of enterprise informatization.
4. Lack of auxiliary decision support functions. Many MIS are often limited to solving some specific business processing problems, with less statistical analysis of data, and even less work involving multidimensional analysis, decision support and data mining, which makes it difficult to fully utilize the benefits of large amounts of data.

2. Current Status of General Enterprise Management Software

Small and medium-sized enterprises in my country account for 94% of the total number of domestic enterprises, and they are the most dynamic in the market. When the demand for computers from large enterprises has stabilized, small and medium-sized enterprise users are particularly prominent. Due to the limited size of the enterprises themselves, small and medium-sized enterprises do not have the ability to independently develop internal management software, and the imperfect internal management system urgently needs systematic management, which lays a solid foundation for the development of enterprise management software.

At present, establishing competitive advantages and improving market competitiveness have become the core of enterprise management, and the original management software has become increasingly unable to adapt to the requirements of management practice. The new era requires a new generation of management software. After experiencing the single management system and MRP/MRPⅡ application, the development of enterprise management software in my country is moving towards a more advanced ERP model. ERP (Enterprise Resource Planning) is the latest development of management information systems in the 1990s. After being introduced to China in recent years, it has received widespread attention in China. ERP not only integrates the main parts of enterprise operations and management (finance, distribution, manufacturing, human resources, decision support), but also represents advanced management ideas and management methods. We can foresee that ERP will be an important tool for management modernization and will be more and more widely used in enterprises and other organizations in my country.

However, the data analysis capabilities of current domestic ERP systems are mostly not strong enough, and after the ERP system is started and operated, its data analysis capability is an important indicator that determines its performance. The data analysis function for ERP software is called "business intelligence", and its main function is to analyze and process the data accumulated by the ERP system. Figuratively speaking, it helps users discover the potential value of the data accumulated by the ERP system. It can be said that ordinary ERP systems can help users standardize the management of enterprises, while ERP systems with powerful data analysis functions can enable users to obtain greater benefits from such standardized management. Therefore, if an enterprise wants to adopt an ERP system, it generally needs a powerful data analysis component. This data analysis component can be integrated with the ERP system, collect data for analysis during the operation of the ERP system, and store sample data and analysis results in the data warehouse for users to query and use.

3. Enterprise Information System Application Requirements

1. Enterprises’ demand for data warehouses

Surveys and studies show that most companies are not short of data, but are hindered by excessive redundant data and inconsistent data; and it is becoming increasingly difficult to access, manage and use for decision support; the amount of data is growing at an exponential rate. In this way, information centers are facing growing demand for decision support, but application development is becoming more and more complex and labor-intensive. So how to convert large amounts of data into reliable, commercial information for decision support? Data warehouses are widely recognized as the best solution.

Data warehousing is the process of processing data within the enterprise. It collects and organizes the scattered original operation data within the enterprise and the data from the outside, providing the enterprise with complete, timely, accurate and clear decision-making information, so that the end user can truly use the DSS tool to directly extract and analyze data randomly from the enterprise information pool, and effectively serve the enterprise's all-round decision-making. As a decision support environment, DW (Data Warehouse) collects and stores data from various different data sources. Through data organization, it provides decision supporters with data distributed across the entire enterprise and across platforms.

In the next step, the existing management system and the existing data warehouse will be expanded. First, the data warehouse will be expanded from a local enterprise solution to outside the enterprise and to the enterprise's users, making the data warehouse more widely used by enterprise users. This can be achieved with the help of the Internet/Intranet. This is a point of integration between data warehouses and e-commerce. Second, the data warehouse will be expanded from an enterprise data management tool to an enterprise decision-making support tool. It can make full use of the data resources in the data warehouse, assist in decision-making for the development of the enterprise, and make more in-depth use of the data warehouse of enterprise users. This can be achieved through the tools provided by database manufacturers, or by independent tool packages provided by other software companies. This is a point of integration between data warehouses and business intelligence.

2. Enterprises’ demand for Internet/Intranet

The combination of enterprise decision-making system and Internet is becoming one of the focal issues that enterprises need to solve urgently. Because with the increase of information exchange within and between enterprises, users are no longer satisfied with simple file sharing methods, but pursue a more flexible and convenient data sharing strategy. This is why a large number of enterprise users have changed their MIS systems with database as the core from client/server computing mode to Internet/Intranet system architecture. In addition, when enterprises develop to a certain scale, their own internationalization has become inevitable. Some branches may be cross-regional and cross-border. Therefore, when it comes to data transmission, information sharing and publishing, Internet becomes the inevitable choice for these cross-regional enterprises; fierce market competition also requires enterprises to respond quickly to market changes. Users' needs are no longer simple queries on discrete single information, but require a fast and flexible query, analysis and report making method that can summarize a large amount of enterprise data, multi-level and multi-faceted. Therefore, the most demanded thing in the market is how to make the large amount of information in the database meet people's ever-changing business needs and provide services for management decision support in a timely manner. How to organically combine enterprise decision support systems with Internet/Intranet technology to provide a Web-based enterprise-level decision support solution that integrates query, reporting, OLAP (On-Line Analytical Processing) analysis and data mining has become an important topic.

4. Design of Enterprise Intelligent Data Processing Tools

In response to the application needs of enterprises, we developed the DOS version and Windows version of "Keli MIS - Multimedia MIS System Generation Tool Software" in 1995 and 1997 respectively. This tool software mainly solves the needs of users to establish management systems through visual interactive design tools without writing programs. It also supports multimedia applications such as database storage and display of sound, images and AVI moving images. After several years of practice, we believe that enterprises urgently need an intelligent data processing tool, which will solve the following problems:

1. It is built on the basis of the existing MIS system or ERP system of the enterprise and utilizes the existing data. Now many small and medium-sized enterprises already have some MIS systems or ERP systems, but due to changes in the situation, there will soon be further needs, so new software is urgently needed to solve the problems that arise and utilize the existing data.
2. Provide data warehouse-based solutions for some medium-sized enterprises. For small enterprises, due to various considerations, data warehouse-based solutions are generally not adopted. For medium-sized enterprises, they should be given an opportunity to build their own data warehouses. If a data warehouse can be built, the enterprise's data can be better organized, and intelligent decision-making tools can play a greater role.
3. It can provide general MIS system generation tools for small enterprises. For some small enterprises with simpler needs, a MIS system can be realized through visual methods without user programming. Combined with the intelligent decision-making system, it will be able to achieve more flexible and powerful functions.
4. It can realize not only traditional query and statistical report functions, but also advanced functions such as multidimensional data analysis, decision support and data mining. Intelligent decision-making tools can realize extremely flexible query and reporting, and some analysis functions are embedded, which can be called interactive query and reporting. In addition, even in larger MIS systems or even ERP systems, it is difficult to find advanced functions such as multidimensional data analysis, decision support and data mining, but intelligent decision-making tools provide these functions, so that the original data can be well utilized and the capabilities of the original system can be greatly enhanced.
5. Full support for Internet/Intranet. Most outputs can be directed to HTML files, including query results, reports, analysis and data mining results, etc. In addition, some functions are implemented in a browser-based environment, allowing users to perform functions such as querying data, browsing reports, and simple analysis through the browser.

5. System Structure

The system is divided into two parts: analysis and design part and application system part, which are respectively for enterprise IS (information system administrator) personnel and enterprise management personnel. In addition to being responsible for advanced data analysis, IS personnel are also responsible for designing specific application systems for management personnel. Since most of the specific application work can be completed by management personnel, the problem of lack of information talents is alleviated from another aspect. The functional structure diagram of the application system part is shown in Figure 1, and the functional structure diagram of the design part is shown in Figure 2. The most important thing in the system is the design of the "description layer".

When general business users access data, they are most worried about complex database terms and complicated database operations. How can we give these business users the ability to independently access information in databases and data warehouses so that they can forget about those database terms and operations? Here we use a technology called "description layer" to solve this problem. After using the "description layer" technology, not only the problems that general business users are worried about are solved, but also the tools necessary for IS personnel to control and manage data access are provided. This technology describes the complex database structure into easy-to-understand business terms, separating business users from technical database terms and complex SQL access languages. It is like a lens through which users can view the data warehouse. In this way, end users do not need to have computer expertise, let alone database experts, to independently access public data and analyze information, so as to better understand the development trend of the enterprise and make wise decisions.

The "description layer" can be called an interpretation layer covering the internal data objects of the database. It is a code translation layer between the user and the database, that is, the messy and complex data objects in the database (for example, the records of each field stored in the data table) are filtered and converted into actual business objects through pre-defined rules ("description layer"), such as: personnel names, material types, etc. At the same time, the function of the "description layer" is not only filtering and mapping, but also reorganizing the data through pre-defined rules, such as high-level data that does not exist in the database (for example, extracting sales locally through price and sales volume), so we can add some non-existent but meaningful content to the database through the "description layer". In addition, we can also use the "description layer" to increase the amount of information contained in the data in the database (for example, establish a classification rule for a certain field so that the records in the database can belong to different categories. For example, according to the sales performance, they are divided into excellent, good, medium, and poor. Users can directly use conditions such as "sales performance = excellent" to query; there are many other similar rules such as hierarchical rules). Finally, we can also add some predefined conditions in the "description layer", and in the future query or analysis, we can directly extract the conditions from the "description layer". The practical significance of using the "description layer" to reorganize data is that the large amount of precious data resources in the database are no longer "heavenly books" that only database developers can understand. Through the interpretation and organization of the "description layer", most business personnel who do not have computer expertise can directly use this data.

In this part, the main work is two points, namely the definition and interpretation of the "description layer". In the definition part of the "description layer", various types of definitions need to be made, specifically the types mentioned in the previous paragraph, and then the definitions of each type are stored as metadata. In the definition process, it is necessary to access the database or data warehouse through the database connection tool, and then design different "descriptions" according to the structure and content of the database. In addition, when using other main parts of this system in the future, such as query, report, analysis and data mining parts, the "description" defined in the "description layer" will be used, and this requires the interpretation part of the "description layer" to interpret and translate the "description" language into a language that the database can accept.

Another key point in the design of this software is the data mining function that reflects business intelligence. With the continuous development of database technology and the widespread application of database management systems, the amount of data stored in the database has increased dramatically, but there are currently few tools for analyzing and processing this data. What can be done now is only a human-driven analysis of the data already in the database. The amount of information people obtain from this data is only a part of the amount of information contained in the entire database. The more important information hidden behind this data is the description of the overall characteristics of this data and the prediction of its development trend. This information has important reference value in the process of decision-making.

Knowledge discovery in databases (KDD) is the process of extracting valuable knowledge from databases using machine learning methods. It is an interdisciplinary subject between database technology and machine learning. Database technology focuses on the study of efficient methods for data storage and processing, while machine learning focuses on designing new methods to extract knowledge from data. KDD uses database technology to process data at the front end, and uses machine learning methods to extract useful knowledge from the processed data. KDD also has strong connections with other disciplines, such as statistics, mathematics, and visualization technology.

In our system, we will implement a complete KDD tool, which can also be called a data mining tool. Because we are targeting a large number of commercial users, our system pays special attention to the support of user-database interaction. Users select a model based on the data in the database, and then select relevant data for knowledge mining, and continuously adjust and optimize the data of the model. The whole process is divided into the following steps:

Data discovery: Understand the data structure and meaning of the raw data involved in the task, and extract relevant data from the database.

Data cleaning: Clean the user's data to make it suitable for subsequent data processing. This requires the user's background knowledge, and the cleaning rules should also be determined according to the actual task.

Model determination: select an initial model by analyzing the data. Model definition is generally divided into three steps: data separation, model selection and parameter selection. In our system, we mainly introduce association rule models and classification models.

Data analysis: define the selected model in detail, determine the model type and related properties; calculate the relevant parameters of the model through calculation of relevant data, and obtain the attribute values ​​of the model; test and evaluate the obtained model through test data; optimize the model according to the evaluation results.

Output result generation: The results of data analysis are generally complex and difficult for people to understand. It is easier for people to accept the results if they are presented in the form of documents or charts.

In the KDD process, the most important part is the data mining part, that is, the determination of models and related attributes. We plan to use the two most widely used models, namely association rules and classification rules, which are introduced in detail below.

An association rule is a rule in the form of "90% of customers who buy bread and butter also buy milk" (bread + butter => milk). The main object for association rule discovery is transactional databases, and the most typical application is sales data. A transaction generally consists of the following parts: transaction processing time, a set of items purchased by customers, and sometimes a customer identification number (such as a credit card number). If these historical transaction data are analyzed, it can provide extremely valuable information about the customer's purchasing behavior. For example, it can help how to place goods on the shelves (such as putting together goods that customers often buy at the same time) and how to plan the market (how to match purchases with each other). It can be seen that discovering association rules from transaction data is very important for improving decision-making in commercial activities such as retail. With the promotion of applications, association rules have played a role in many fields and have become the most typical data mining application.

Classification is also a very important task in data mining. The purpose of classification is to learn a classification function or classification model (often called a classifier) ​​that can map data items in the database to one of the given categories. There are many application examples of classification, and the most typical one is the premium setting of insurance companies. A key factor for the success of insurance companies is to choose a balance between setting competitive premiums and covering risks. The insurance market is highly competitive. Setting too high premiums means losing the market, while too low premiums will affect the company's profitability. Premiums are usually determined by multiple analyses and intuitive judgments on some major factors (such as the age of the driver, the type of vehicle, etc.). Due to the large number of investment portfolios, the analysis method is usually rough. After using classification for data mining, the ability of computers to process massive data can be used to perform reasonable classification and set reasonable premiums to maximize the benefits of insurance companies.

VI. Conclusion

Intelligent decision support tools for enterprise applications are a promising research and development direction. With the establishment and development of market economy in my country, the demand of small and medium-sized enterprises for enterprise information decision support tools will become increasingly strong. The design scheme of the intelligent decision support tool proposed in this paper aims to communicate with colleagues and jointly promote the development and research of intelligent decision support tools, and finally develop software suitable for the application needs of domestic small and medium-sized enterprises.

The article "Smart Data Processing Tools for Small and Medium Enterprises" was collected and compiled by Blue Rhythm www.21blue.com. The copyright belongs to the author. Please indicate the source when reprinting!