Index of Metrics, Statistical Techniques and Data Processing Tools Included in scores

This page lists all the metrics, statistical techniques and data processing tools included in scores.

It is divided into the following sections: continuous, probability, categorical, spatial, statistical tests, processing (tools for preparing data), plotting data, pandas and emerging.

Continuous

Name (Alphabetical order)	API	Tutorial	Reference(s)
Additive Bias (Mean Error)	API	Tutorial	Mean Error (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Consistent Expectile Score	API	Tutorial	Gneiting (2011); Taggart (2022)
Consistent Huber Score	API	Tutorial	Taggart (2022a); Taggart (2022b)
Consistent Quantile Score	API	Tutorial	Gneiting (2011); Taggart (2022)
Flip-Flop Index
Flip-Flop Index	API	Tutorial	Griffiths et al. (2019); Griffiths et al. (2021)
Flip-Flop Index - Proportion Exceeding	API	Tutorial	Griffiths et al. (2019); Griffiths et al. (2021)
Interval Score	API	Tutorial	Gneiting and Raftery (2007) - Section 6.2
Isotonic Fit, see Isotonic Regression	—	—	—
Isotonic Regression (Isotonic Fit, Reliability Diagram)	API	Tutorial	de Leeuw et al. (2009); Dimitriadis et al. (2020); Jordan et al. (2020), version 2
Kling–Gupta Efficiency (KGE)	API	Tutorial	Gupta et al. (2009); Knoben et al. (2019)
Mean Absolute Error (MAE)	API	Tutorial	Wikipedia
Mean Elementary Score, see Murphy Score	—	—	—
Mean Error (Additive Bias)	API	Tutorial	Mean Error (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Mean Squared Error (MSE)	API	Tutorial	Wikipedia
Multiplicative Bias	API	Tutorial	Multiplicative bias (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Murphy Score (Mean Elementary Score)
Murphy Score (Mean Elementary Score)	API	Tutorial	Ehm et al. (2016) - Theorem 1; Taggart (2022) - Theorem 5.3
Murphy Score (Mean Elementary Score) - Theta Values	API	Tutorial	Ehm et al. (2016) - Corollary 2 (p.521); Taggart (2022) - Corollary 5.6
Nash-Sutcliffe Model Efficiency Coefficient (NSE)	API	Tutorial	Nash and Sutcliffe (1970)
Pearson’s Correlation Coefficient	API	Tutorial	Wikipedia
Percent Bias (PBIAS)	API	Tutorial	Percent Bias (CRAN hydroGOF); Sorooshian et al. (1993); Alfieri et al. (2014); Dawson et al. (2007); Moriasi et al. (2007)
Percent Within X	API	Tutorial	—
Pinball Loss, see Quantile Loss	—	—	—
Quantile Interval Score	API	Tutorial	Winkler (1972)
Quantile Loss (Quantile Score, Pinball Loss)	API	Tutorial	Gneiting (2011) - Theorem 9
Quantile-Quantile (QQ) Plots	API	Tutorial	Déqué (2011)
Quantile Score, see Quantile Loss	—	—	—
Reliability Diagram, see Isotonic Regression	—	—	—
Root Mean Squared Error (RMSE)	API	Tutorial	Wikipedia
Spearman’s Correlation Coefficient	API	Tutorial	Spearman (1904)
Threshold Weighted Absolute Error	API	Tutorial	Taggart (2022)
Threshold Weighted Expectile Score	API	Tutorial	Taggart (2022)
Threshold Weighted Huber Loss	API	Tutorial	Taggart (2022)
Threshold Weighted Quantile Score	API	Tutorial	Taggart (2022)
Threshold Weighted Squared Error	API	Tutorial	Taggart (2022)

Probability

Name (Alphabetical order)	API	Tutorial	Reference(s)
Brier Score	API	Tutorial	Brier (1950)
Brier Score for Ensembles	API	Tutorial	Ferro (2013)
Continuous Ranked Probability Score (CRPS) for Cumulative Distribution Functions (CDFs)
Adjust Forecast for CRPS	API	—	—
CRPS CDF Brier Decomposition	API	Tutorial	—
CRPS for CDFs	API	Tutorial	Matheson and Winkler (1976); Gneiting and Ranjan (2011)
CRPS Step Threshold Weights	API	—	—
Continuous Ranked Probability Score (CRPS) for Ensembles
CRPS for Ensembles	API	Tutorial	Ferro (2014); Gneiting And Raftery (2007); Zamo and Naveau (2018)
Threshold-Weighted CRPS (twCRPS) for Ensembles	API	Tutorial	Allen et al. (2023); Allen (2024)
Interval-Threshold-Weighted CRPS (twCRPS) for Ensembles	API	Tutorial	Allen et al. (2023); Allen (2024)
Tail-Threshold-Weighted CRPS (twCRPS) for Ensembles	API	Tutorial	Allen et al. (2023); Allen (2024)
Isotonic Fit, see Isotonic Regression	—	—	—
Isotonic Regression (Isotonic Fit, Reliability Diagram)	API	Tutorial	de Leeuw et al. (2009); Dimitriadis et al. (2020); Jordan et al. (2020), version 2
Mean Elementary Score, see Murphy Score	—	—	—
Murphy Score (Mean Elementary Score)
Murphy Score (Mean Elementary Score)	API	Tutorial	Ehm et al. (2016) - Theorem 1; Taggart (2022) - Theorem 5.3
Murphy Score (Mean Elementary Score) - Theta Values	API	Tutorial	Ehm et al. (2016) - Corollary 2 (p.521); Taggart (2022) - Corollary 5.6
Receiver (Relative) Operating Characteristic (ROC)	API	Tutorial	Fawcett and Niculescu-Mizil (2007); Gneiting and Vogel (2022); Hand (2009); Hand and Anagnostopoulos (2013)); Hand and Anagnostopoulos (2023); Pesce et al. (2010)
Reliability Diagram, see Isotonic Regression	—	—	—

Categorical

Name (Alphabetical order)	API	Tutorial	Reference(s)
Binary Contingency Scores and Binary Contingency Tables	API; API	Tutorial	Methods for dichotomous (yes/no) forecasts
Accuracy (Fraction Correct)	API	Tutorial	Accuracy (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Base Rate	API	Tutorial	Hogan and Mason (2011)
Bias Score (Frequency Bias)	API	Tutorial	Bias Score (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Cohen’s Kappa (Heidke Skill Score)	API	Tutorial	Heidke Skill Score (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Critical Success Index (Threat Score)	API	Tutorial	Threat score (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Equitable Threat Score (Gilbert Skill Score)	API	Tutorial	Hogan et al. (2010); Equitable Threat score (WWRP/WGNE Joint Working Group on Forecast Verification Research)
F1 Score	API	Tutorial	Wikipedia
False Alarm Rate (Probability of False Detection (POFD))	API	Tutorial	Probability of false detection (WWRP/WGNE Joint Working Group on Forecast Verification Research)
False Alarm Ratio (FAR)	API	Tutorial	False alarm ratio (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Forecast Rate	API	Tutorial	Hogan and Mason (2011)
Fraction Correct (Accuracy)	API	Tutorial	Accuracy (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Frequency Bias (Bias Score)	API	Tutorial	Bias Score (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Gilbert Skill Score (Equitable Threat Score)	API	Tutorial	Hogan et al. (2010); Equitable Threat score (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Hanssen and Kuipers’ Discriminant (Peirce’s Skill Score, True Skill Statistic)	API	Tutorial	Hanssen and Kuipers discriminant (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Heidke Skill Score (Cohen’s Kappa)	API	Tutorial	Heidke skill score (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Hit Rate (True Positive Rate, Probability of Detection (POD), Sensitivity, Recall)	API	Tutorial	Probability of detection (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Negative Predictive Value	API	Tutorial	Wikipedia; Monaghan et al. (2021)
Odds Ratio	API	Tutorial	Stephenson (2000)
Odds Ratio Skill Score (Yule’s Q)	API	Tutorial	Stephenson (2000)
Peirce’s Skill Score (True Skill Statistic, Hanssen and Kuipers’ Discriminant)	API	Tutorial	Peirce (1884); Hanssen and Kuipers discriminant (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Positive Predictive Value (Success Ratio, Precision)	API	Tutorial	Wikipedia; Success ratio (WWRP/WGNE Joint Working Group on Forecast Verification Research); Monaghan et al. (2021)
Precision (Success Ratio, Positive Predictive Value)	API	Tutorial	Wikipedia; Success ratio (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Probability of Detection (POD) (Hit Rate, True Positive Rate, Sensitivity, Recall)	API	Tutorial	Probability of detection (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Probability of False Detection (POFD) (False Alarm Rate)	API	Tutorial	Probability of false detection (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Recall (Hit Rate, Probability of Detection (POD), True Positive Rate, Sensitivity)	API	Tutorial	Wikipedia; Probability of detection (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Sensitivity (Hit Rate, Probability of Detection (POD), True Positive Rate, Recall)	API	Tutorial	Wikipedia; Monaghan et al. (2021)
Specificity (True Negative Rate)	API	Tutorial	Wikipedia; Monaghan et al. (2021)
Success Ratio (Precision, Positive Predictive Value)	API	Tutorial	Success ratio (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Symmetric Extremal Dependence Index (SEDI)	API	Tutorial	Ferro and Stephenson (2011)
Threat Score (Critical Success Index)	API	Tutorial	Threat score (WWRP/WGNE Joint Working Group on Forecast Verification Research)
True Negative Rate (Specificity)	API	Tutorial	Wikipedia
True Positive Rate (Hit Rate, Probability of Detection (POD), Sensitivity, Recall)	API	Tutorial	Probability of detection (WWRP/WGNE Joint Working Group on Forecast Verification Research)
True Skill Statistic (Peirce’s Skill Score, Hanssen and Kuipers’ Discriminant)	API	Tutorial	Hanssen and Kuipers discriminant (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Yule’s Q (Odds Ratio Skill Score)	API	Tutorial	Stephenson (2000)
FIxed Risk Multicategorical (FIRM)	API	Tutorial	Taggart et al. (2022)
POD - implementation as used in ROC (NOTE: Please use contingency table classes instead, this API may be removed in future)	API	Tutorial	Probability of detection (WWRP/WGNE Joint Working Group on Forecast Verification Research)
POFD - implementation as used in ROC (NOTE: Please use contingency table classes instead, this API may be removed in future)	API	Tutorial	Probability of false detection (WWRP/WGNE Joint Working Group on Forecast Verification Research)
Risk Matrix Score
Risk Matrix Score	API	Tutorial	Taggart and Wilke (2025)
Risk Matrix Score - Matrix Weights to Array	API	Tutorial	Taggart and Wilke (2025)
Risk Matrix Score - Warning Scaling to Weight Array	API	Tutorial	Taggart and Wilke (2025)
Stable Equitable Error in Probability Space (SEEPS)	API	Tutorial	Rodwell et al. (2010)
Threshold Event Operator	API	Tutorial	—
Make Contingency Manager	API	Tutorial	—
Make Event Tables	API	Tutorial	—

Spatial

Name (Alphabetical order)	API	Tutorial	Reference(s)
Fractions Skill Score (FSS)
FSS - 2D	API	Tutorial	Roberts and Lean (2008); Mittermaier (2021)
FSS - 2D Binary	API	Tutorial	—
FSS - 2D Single Field	API	Tutorial	Roberts and Lean (2008); Faggian et al. (2015)

Statistical Tests

Name (Alphabetical order)	API	Tutorial	Reference(s)
Diebold Mariano (with the Harvey et al. 1997 and the Hering and Genton 2011 modifications)	API	Tutorial	Diebold and Mariano (1995); Harvey et al. (1997); Hering and Genton (2011)

Processing (tools for preparing data)

Name (Alphabetical order)	API	Utilised For
Binary Discretise	API	Receiver (Relative) Operating Characteristic (ROC)
Binary Discretise Proportion	API	Flip-Flop Index
Block Bootstrap	API	Confidence intervals. See tutorial
Broadcast and Match Not-a-Number (NaN)	API	Murphy Score (Mean Elementary Score)
Comparative Discretise	API	Receiver (Relative) Operating Characteristic (ROC)
Cumulative Distribution Functions (CDFs)
Add Thresholds	API	Continuous Ranked Probability Score (CRPS) for CDFs; CRPS CDF Brier Decomposition
CDF Envelope	API	Adjust Forecast for CRPS
Decreasing CDFs	API	Adjust Forecast for CRPS
Fill CDF	API	CRPS for CDFs; CRPS CDF Brier Decomposition
Integrate Square Piecewise Linear	API	CRPS for CDFs
Observed CDF	API	CRPS for CDFs; CRPS CDF Brier Decomposition
Propagate Not-a-Number (NaN)	API	Adjust Forecast for CRPS; CRPS CDF Brier Decomposition; CRPS for CDFs
Round Values	API	CRPS for CDFs; CRPS CDF Brier Decomposition
Isotonic Fit, see Isotonic Regression	—	—
Isotonic Regression (Isotonic Fit, Reliability Diagram)	API	See “Isotonic Regression (Isotonic Fit, Reliability Diagram)” entries in Continuous and Probability
Proportion Exceeding	API	Flip-Flop Index
Reliability Diagram, see Isotonic Regression	—	—

Plotting Data

Name (Alphabetical order)	API	Tutorial	Reference(s)
Murphy Diagrams (Plots of the Murphy Score)
Murphy Score (Mean Elementary Score)	API	Tutorial	Ehm et al. (2016) - Theorem 1; Taggart (2022) - Theorem 5.3
Murphy Score (Mean Elementary Score) - Theta Values	API	Tutorial	Ehm et al. (2016) - Corollary 2 (p.521); Taggart (2022) - Corollary 5.6
Quantile-Quantile (QQ) Plots	API	Tutorial	Déqué (2011)
Receiver (Relative) Operating Characteristic (ROC)	API	Tutorial	Fawcett and Niculescu-Mizil (2007); Gneiting and Vogel (2022); Hand (2009); Hand and Anagnostopoulos (2013)); Hand and Anagnostopoulos (2023); Pesce et al. (2010)

Pandas

Name (Alphabetical order)	API	Tutorial	Reference(s)
Mean Absolute Error	API	Tutorial	Wikipedia
Mean Squared Error	API	Tutorial	Wikipedia
Root Mean Squared Error	API	Tutorial	Wikipedia

Emerging

Caution

This section of the API contains implementations of novel metrics that are still undergoing mathematical peer review. These implementations may change in line with the peer review process.

Name (Alphabetical order)	API	Tutorial	Reference(s)
Risk Matrix Score has moved to the categorical area following peer-reviewed publication	—	—	—