A new method to evaluate overall performance of a climate model

Many climate-related studies, such as detection and attribution of historical climate change, projections of future climate and environments, and adaptation to future climate change, rely heavily on the performance of climate models. Concisely summarizing and evaluating model performance becomes increasingly important for climate model intercomparison and application, especially when more climate models contribute to international model intercomparison projects.

Most current model evaluation metrics, e.g., root mean square error (RMSE), correlation coefficient, and standard deviation measure the model performance in simulating individual variables. However, it is often necessary to evaluate a model's overall performance in simulating multiple variables. To fill this gap, an article published in Geosci. Model Dev. presents a new multivariable integrated evaluation (MVIE) method.

"The MVIE includes three levels of statistical metrics, which can provide a comprehensive and quantitative evaluation on model performance," says XU, the first author of the study from the Institute of Atmospheric Physics, Chinese Academy of Sciences. The first level of metrics, including the commonly used correlation coefficient, RMS value, and RMSE, measures model performance in terms of individual variables. The second level of metrics, including four newly developed statistical quantities, provides an integrated evaluation of model performance in terms of simulating multiple fields. The third level of metrics, multivariable integrated evaluation index (MIEI), further summarizes the three statistical quantities of second level of metrics into a single index and can be used to rank the performances of various climate models. Different from the commonly used RMSE-based metrics, the MIEI satisfies the criterion that a model performance index should vary monotonically as the model performance improves.

According to the study, higher level of metrics is derived from and concisely summarizes the lower level of metrics. "Inevitably, the higher level of metrics loses detailed statistical information in contrast to the lower level of metrics." XU therefore suggests, "To provide a more comprehensive and detailed evaluation of model performance, one can use all three levels of metrics."

More information:
Zhongfeng Xu et al, Multivariable integrated evaluation of model performance with the vector field evaluation diagram, Geoscientific Model Development (2017). DOI: 10.5194/gmd-10-3805-2017

Provided by Chinese Academy of Sciences