Key Features of scores

Key Features of scores#

Data Handling#

  • Works with labelled, n-dimensional data (e.g., geospatial, vertical and temporal dimensions) for both point-based and gridded data. scores can effectively handle the dimensionality, data size and data structures commonly used for:

    • gridded Earth system data (e.g., numerical weather prediction models)

    • tabular, point, latitude/longitude or site-based data (e.g., forecasts for specific locations).

  • Handles missing data, masking of data and weighting of results.

  • Supports xarray datatypes, and works with NetCDF4, HDF5, Zarr and GRIB data sources among others.

Usability#

  • A companion Jupyter Notebook tutorial for each metric and statistical test that demonstrates its use in practice.

  • Over 50 metrics, statistical techniques and data processing tools, including:

    • commonly-used metrics (e.g., mean absolute error (MAE), root mean squared error (RMSE))

    • novel scores not commonly found elsewhere (e.g., FIxed Risk Multicategorical (FIRM) score (Taggart et al., 2022), Flip-Flop Index (Griffiths et al., 2019, 2021))

    • complex scores (e.g., threshold-weighted continuous ranked probability score (twCRPS))

    • statistical tests (e.g., the Diebold-Mariano (Diebold & Mariano, 1995) test, with both the Harvey et al. (1997) and Hering & Genton (2011) modifications).

  • All scores and statistical techniques have undergone a thorough scientific and software review.

  • An area specifically to hold emerging scores which are still undergoing research and development. This provides a clear mechanism for people to share, access and collaborate on new scores, and be able to easily re-use versioned implementations of those scores.

Compatibility#

  • Highly modular - provides its own implementations, avoids extensive dependencies and offers a consistent API.

  • Easy to integrate and use in a wide variety of environments. It has been used on workstations, servers and in high performance computing (supercomputing) environments.

  • Maintains 100% automated test coverage.

  • Uses Dask for scaling and performance.

  • Some metrics work with pandas and we aim to expand this capability.

Contents