Designing a Framework for Data Quality Validation of Meteorological Data System

Wen-Lung TSAI  Yung-Chun CHAN  

Publication
IEICE TRANSACTIONS on Information and Systems   Vol.E102-D   No.4   pp.800-809
Publication Date: 2019/04/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2018DAP0021
Type of Manuscript: Special Section PAPER (Special Section on Data Engineering and Information Management)
Category: 
Keyword: 
data quality,  meteorological data,  meteorological observatory,  total quality management,  data warehouse,  

Full Text: FreePDF(1.3MB)


Summary: 
In the current era of data science, data quality has a significant and critical impact on business operations. This is no different for the meteorological data encountered in the field of meteorology. However, the conventional methods of meteorological data quality control mainly focus on error detection and null-value detection; that is, they only consider the results of the data output but ignore the quality problems that may also arise in the workflow. To rectify this issue, this paper proposes the Total Meteorological Data Quality (TMDQ) framework based on the Total Quality Management (TQM) perspective, especially considering the systematic nature of data warehousing and process focus needs. In practical applications, this paper uses the proposed framework as the basis for the development of a system to help meteorological observers improve and maintain the quality of meteorological data in a timely and efficient manner. To verify the feasibility of the proposed framework and demonstrate its capabilities and usage, it was implemented in the Tamsui Meteorological Observatory (TMO) in Taiwan. The four quality dimension indicators established through the proposed framework will help meteorological observers grasp the various characteristics of meteorological data from different aspects. The application and research limitations of the proposed framework are discussed and possible directions for future research are presented.