Data Governance Weekly Talk (3): Data Quality Management

The definition and purpose of data quality management

Data quality management refers to the entire life of data from generation, acquisition, storage, sharing, maintenance, application, etc. A series of management activities such as identification, detection, measurement, early warning and processing of various data quality problems that may occur during the cycle.

The purpose of data quality management is to provide enterprises with a solid and reliable data foundation by improving the integrity, accuracy and authenticity of data, enhance the use value of data, and improve the daily operations and precision marketing of enterprises. play an active and effective role in areas such as management decision-making and risk management.

Dimensions of data quality evaluation

How to judge the quality of data? In what ways can data quality be assessed? In practice, we believe that it can generally be assessed through data quality assessment dimensions. Data quality assessment dimensions are one of the characteristics of data quality. They provide a way and standard for measuring and managing data quality. In a specific data quality project, the data quality dimensions most suitable for business needs should be selected for measurement to evaluate the quality of the data.

In "GB/T36344-Information Technology Data Quality Evaluation Indicators", the National Standardization Administration Committee clarified the data quality evaluation indicator framework.

Normativity: The extent to which data conforms to data standards, data models, business rules, metadata, or authoritative reference data.

Completeness: The degree to which data elements are assigned values ??according to the requirements of data rules.

Accuracy: The extent to which data accurately represents the true value of the real entity (actual object) it describes.

Consistency: The extent to which data are not inconsistent with other data used in a specific context.

Timeliness: How accurate the data is over time.

Accessibility: The extent to which data can be accessed.

The International Data Management Association (DAMA) proposed its data quality assessment framework in the "DAMA Data Management Knowledge System Guide" released by it:

The evaluation indicators of data quality are in national standards , there are certain differences in international practice. Enterprises should build appropriate data quality assessment systems, dimensions and indicators based on their own actual business conditions and internal management requirements.

Causes of data quality problems

The consequences of data quality problems are obvious, so what are the root causes of data quality problems? The main factors affecting data quality are technology, business and management. The following will analyze the causes from these three aspects.

Technical aspects

There are data quality problems in data sources. For example: some data are collected from the production system. In the production system, these data are duplicated, incomplete, and inaccurate. and other problems, and whether these problems have been cleaned during the collection process. This situation is also relatively common.

Data collection process quality issues, such as incorrect collection parameters and process settings, inefficient data collection interfaces, resulting in data collection failure, data loss, data mapping and conversion failure.

Problems in the data transmission process, such as: problems with the data interface itself, parameter configuration errors, network unreliability, etc., will cause data quality problems during the data transmission process.

Problems in the data loading process, such as: problems with data cleaning, conversion, and loading rule configuration.

Data storage quality problems, such as: unreasonable storage design, limited storage capacity, artificial background adjustment of data, resulting in data loss, data invalidity, data distortion, and record duplication.

Business systems have data islands and chimney construction, and data inconsistencies between systems are serious.

Business aspect

The data entry on the business side is not standardized, and some common data entry problems are such as upper and lower case, full and half-width, units, etc. When inputting on the business end, the system does not embed relevant data verification rules, resulting in the input being greatly affected by human factors. For example, the contract amount should be entered, such as 100,000 yuan, 100,000 yuan, 100,000 yuan, etc. The quality of manually entered data is closely related to the business personnel who record the data. If the people who record the data work rigorously and conscientiously, the data quality will be relatively good, and vice versa.

Management

Enterprise management thinking level does not realize the importance of data quality. It emphasizes the system but ignores the data. It is believed that the system is omnipotent and the data stored in the system should be Excellent quality.

There is no clear responsibility management system for data within the company, and there is no corresponding centralized management department. If there is a data quality problem, the corresponding person in charge cannot be found.

Data entry specifications are not uniform. When the same business department handles the same business, human factors cause data conflicts or contradictions due to inconsistent specifications.

There is a lack of top-down data planning, no corresponding data quality management goals are set, and no policies, management and assessment systems related to data quality are formulated.

There is a lack of effective data quality problem handling mechanism. There is no unified process and system support from the discovery, assignment, processing and optimization of data quality problems. Data quality problems cannot be managed and assessed in a closed loop.

Data quality management solution

In view of the above analysis of the causes of data quality problems from the three aspects of technology, business and management, it is necessary to carry out preventive control, process monitoring, Data quality monitoring is carried out in three aspects of post-event supervision and management to continuously improve data quality.

Prior control and prevention

Establish data standards covering various business topics within the enterprise, unify indicator definitions, indicator calibers, and input specifications covering each business field. For manually entered data, use non-open input methods as much as possible, such as drop-down menus, single check boxes, time controls, labels (supporting customized learning), etc. The input part must be open and necessary timely corrections must be made. test. In addition, for data quality problems caused by system reasons, we need to establish a data standard system. For production systems that can be transformed, they should be transformed under the guidance of data standards. For systems that cannot be transformed, they should be cleaned and converted through some technical means. When data is generated, The quality of data is controlled in all aspects, so the efficiency must be the highest.

Establishing an internal data accountability system and data quality management department, and formulating data quality monitoring processes and assessment methods will also help improve the prior control and prevention mechanism of data quality.

In-process process monitoring

In-process data quality control is to monitor and process data quality during the maintenance and use of data. By establishing a process-based control system for data quality, we implement process-based control over all aspects of data creation, modification, collection, processing, loading, and application. In this process, relevant modules in the data quality management tool can be used to monitor the data quality of each node of the data flow, which can provide real-time warning of data quality, control the data quality from the source of the data, and support automatic system verification and manual review. managed in a combined manner. In this process, the relevant processes and approval flows of the enterprise's data quality problem handling mechanism can also be embedded in the data quality management tool to effectively assist and monitor data quality.

Post-event supervision and management

For the data that has been stored in the data warehouse, if quality problems are found, data quality control tools must be used. When a data warehouse or data center is established, key fields should be unified in naming, format, precision, etc. according to data standards to eliminate data ambiguity. According to the data standards, a corresponding rule model is established in the data quality management tool. For the imported historical data, data quality problems can be discovered by running the rule model, and the whole process of data quality problems can be tracked in the platform.

Conclusion

Data quality management is an important part of enterprise data governance. All work of enterprise data governance is carried out around the goal of improving data quality. To manage data quality well, we should grasp the key factors that affect data quality, set up quality management points or quality control points, start from the source of data, and fundamentally solve data quality problems.

Data quality issues are already an urgent problem that many companies need to solve, and it is time to carry out data governance. Improving data quality does not happen overnight. A single data rectification can solve all data quality problems. For existing data, verify and clean it through data quality management tools. In addition, it is necessary to establish a complete data quality management and control system through data standards and data quality, monitor every link, regularly check data quality, determine solutions, and implement Improve and continuously improve data quality.