Index of Metrics, Statistical Techniques and Data Processing Tools Included in scores#
Continuous#
Name (Alphabetical order) |
API |
Tutorial |
Reference(s) |
|---|---|---|---|
Additive Bias (Mean Error) |
Mean Error (WWRP/WGNE Joint Working Group on Forecast Verification Research) |
||
Consistent Expectile Score |
|||
Consistent Huber Score |
|||
Consistent Quantile Score |
|||
Flip-Flop Index |
|||
|
|||
|
|||
Interval Score |
|||
Isotonic Fit, see Isotonic Regression |
— |
— |
— |
Isotonic Regression (Isotonic Fit, Reliability Diagram) |
de Leeuw et al. (2009); Dimitriadis et al. (2020); Jordan et al. (2020), version 2 |
||
Kling–Gupta Efficiency (KGE) |
|||
Mean Absolute Error (MAE) |
|||
Mean Elementary Score, see Murphy Score |
— |
— |
— |
Mean Error (Additive Bias) |
Mean Error (WWRP/WGNE Joint Working Group on Forecast Verification Research) |
||
Mean Squared Error (MSE) |
|||
Multiplicative Bias |
Multiplicative bias (WWRP/WGNE Joint Working Group on Forecast Verification Research) |
||
Murphy Score (Mean Elementary Score) |
|||
|
|||
|
Ehm et al. (2016) - Corollary 2 (p.521); Taggart (2022) - Corollary 5.6 |
||
Pearson’s Correlation Coefficient |
|||
Percent Bias (PBIAS) |
Percent Bias (CRAN hydroGOF); Sorooshian et al. (1993); Alfieri et al. (2014); Dawson et al. (2007); Moriasi et al. (2007) |
||
Pinball Loss, see Quantile Loss |
— |
— |
— |
Quantile Interval Score |
|||
Quantile Loss (Quantile Score, Pinball Loss) |
|||
Quantile Score, see Quantile Loss |
— |
— |
— |
Reliability Diagram, see Isotonic Regression |
— |
— |
— |
Root Mean Squared Error (RMSE) |
|||
Threshold Weighted Absolute Error |
|||
Threshold Weighted Expectile Score |
|||
Threshold Weighted Huber Loss |
|||
Threshold Weighted Quantile Score |
|||
Threshold Weighted Squared Error |
Probability#
Name (Alphabetical order) |
API |
Tutorial |
Reference(s) |
|---|---|---|---|
Brier Score |
|||
Brier Score for Ensembles |
|||
Continuous Ranked Probability Score (CRPS) for Cumulative Distribution Functions (CDFs) |
|||
|
— |
— |
|
|
— |
||
|
|||
|
— |
— |
|
Continuous Ranked Probability Score (CRPS) for Ensembles |
|||
|
Ferro (2014); Gneiting And Raftery (2007); Zamo and Naveau (2018) |
||
|
|||
|
|||
|
|||
Isotonic Fit, see Isotonic Regression |
— |
— |
— |
Isotonic Regression (Isotonic Fit, Reliability Diagram) |
de Leeuw et al. (2009); Dimitriadis et al. (2020); Jordan et al. (2020), version 2 |
||
Mean Elementary Score, see Murphy Score |
— |
— |
— |
Murphy Score (Mean Elementary Score) |
|||
|
|||
|
Ehm et al. (2016) - Corollary 2 (p.521); Taggart (2022) - Corollary 5.6 |
||
Receiver (Relative) Operating Characteristic (ROC) |
Fawcett and Niculescu-Mizil (2007); Gneiting and Vogel (2022); Hand (2009); Hand and Anagnostopoulos (2013)); Hand and Anagnostopoulos (2023); Pesce et al. (2010) |
||
Reliability Diagram, see Isotonic Regression |
— |
— |
— |
Categorical#
Spatial#
Name (Alphabetical order) |
API |
Tutorial |
Reference(s) |
|---|---|---|---|
Fractions Skill Score (FSS) |
|||
|
|||
|
— |
||
|
Statistical Tests#
Name (Alphabetical order) |
API |
Tutorial |
Reference(s) |
|---|---|---|---|
Diebold Mariano (with the Harvey et al. 1997 and the Hering and Genton 2011 modifications) |
Diebold and Mariano (1995); Harvey et al. (1997); Hering and Genton (2011) |
Processing (tools for preparing data)#
Name (Alphabetical order) |
API |
Utilised For |
|---|---|---|
Binary Discretise |
Receiver (Relative) Operating Characteristic (ROC) |
|
Binary Discretise Proportion |
Flip-Flop Index |
|
Broadcast and Match Not-a-Number (NaN) |
Murphy Score (Mean Elementary Score) |
|
Comparative Discretise |
Receiver (Relative) Operating Characteristic (ROC) |
|
Cumulative Distribution Functions (CDFs) |
||
|
Continuous Ranked Probability Score (CRPS) for CDFs; CRPS CDF Brier Decomposition |
|
|
Adjust Forecast for CRPS |
|
|
Adjust Forecast for CRPS |
|
|
CRPS for CDFs; CRPS CDF Brier Decomposition |
|
|
CRPS for CDFs |
|
|
CRPS for CDFs; CRPS CDF Brier Decomposition |
|
|
Adjust Forecast for CRPS; CRPS CDF Brier Decomposition; CRPS for CDFs |
|
|
CRPS for CDFs; CRPS CDF Brier Decomposition |
|
Isotonic Fit, see Isotonic Regression |
— |
— |
Isotonic Regression (Isotonic Fit, Reliability Diagram) |
See “Isotonic Regression (Isotonic Fit, Reliability Diagram)” entries in Continuous and Probability |
|
Proportion Exceeding |
Flip-Flop Index |
|
Reliability Diagram, see Isotonic Regression |
— |
— |
Pandas#
Emerging#
Caution
This section of the API contains implementations of novel metrics that are still undergoing mathematical peer review. These implementations may change in line with the peer review process.
Name (Alphabetical order) |
API |
Tutorial |
Reference(s) |
|---|---|---|---|
Risk Matrix Score |
|||
|
Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation. |
||
|
Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation. |
||
|
Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation. |