Interactive online version: . Download notebook.

Additive Bias, Multiplicative Bias and Percent Bias

Introduction

Bias measures how well the mean forecast and mean observation correspond to each other. It can tell us whether there is an over or under-forecast tendency and informs how a forecast system could be easily recalibrated.

scores has multiplicative bias, additive bias and percent bias implementations for use on continuous forecasts.

Additive bias is often called the “mean error”. Since it does not tell us the average magnitude (i.e. average of the absolute value of the error) of the error, it is possible for there to be little or no bias even when there are large positive and negative errors.

Multiplicative bias is well suited for forecasts and observations that have 0 as an upper or lower bound (e.g., significant wave height, or wind magnitude) - or to causes of error which are multiplicative in nature. This could be useful for a forecaster who wants a simple method to bias correct a wind speed forecast as wind speeds can never be negative.

Percent bias is used for evaluating and comparing forecast accuracy across stations or datasets with varying magnitudes. By expressing the error as a percentage of the observed value, it allows for standardised comparisons, enabling assessment of forecast performance regardless of the absolute scale of values. Consider three stations with average streamflow values of 5, 25, and 50 m³/s. Assuming an additive bias of 1 m³/s for each station, the calculated percent bias values are 20%, 4%, and 2%, respectively. Assuming a threshold of 5% for acceptable performance, the first station (flow value of 5) is considered less accurate, while the other stations demonstrate acceptable performance.

Note: In this tutorial we use the forecast and analysis grids that are downloaded or derived in First_Data_Fetching.ipynb. Please run through this tutorial first to fetch data.

import xarray as xr
from scores.continuous import additive_bias, multiplicative_bias, pbias

fcst = xr.open_dataset("forecast_grid.nc")
obs = xr.open_dataset("analysis_grid.nc")

# Let's select the forecast for the same timestamp as the analysis
fcst = fcst.sel(time=obs.time.values)

Additive bias

# Calculate additive bias and preserve the "lat" dimension
bias = additive_bias(fcst.temp_scrn, obs.temp_scrn, preserve_dims="lat")
bias.name = "additive bias"
bias.plot()

[<matplotlib.lines.Line2D at 0x138f3d2e0>]

We can see how biases become larger towards the poles, but are relatively unbiased in the tropics.

Multiplicative bias

Let’s imagine that our data has a lower bound at zero and we want to calculate the multiplicative bias. To avoid downloading more data, we will convert the temperature data to be degrees Celcius and clip the data to have a minimum value of zero. Readers may wish to explore the data available from the NCI server and download a wind grid instead for this example.

# Convert to Celcius and clip data to have a minimum value of zero
fcst_clipped = (fcst.temp_scrn - 273.15).clip(min=0)
obs_clipped = (obs.temp_scrn - 273.15).clip(min=0)

bias = multiplicative_bias(fcst_clipped, obs_clipped, preserve_dims="lat")
bias.name = "multiplicative bias"
bias.plot()

[<matplotlib.lines.Line2D at 0x138fd7fe0>]

It’s worth noting that for a latitude slice, if the mean forecast and mean observation is zero, then the multiplicative bias for that latitude slice will be NaN. If the mean forecast is greater than zero and the mean observation is zero, then the multiplicative bias for that slice is infinite.

We can see below in our dataset that we have np.inf values, and np.nan values close to the poles where values are negative (remember that this data is temperature data converted to degrees Celisus and clipped to have a minimum value of zero).

bias.max()

<xarray.DataArray 'multiplicative bias' ()> Size: 8B
array(inf)

xarray.DataArray

'multiplicative bias'

• inf

  array(inf)

• Coordinates: (0)
• Indexes: (0)
• Attributes: (0)

bias

<xarray.DataArray 'multiplicative bias' (lat: 1536)> Size: 6kB
array([nan, nan, nan, ..., nan, nan, nan], dtype=float32)
Coordinates:
  * lat      (lat) float64 12kB 89.94 89.82 89.71 89.59 ... -89.71 -89.82 -89.94

xarray.DataArray

'multiplicative bias'

• lat: 1536

• nan nan nan nan nan nan nan nan ... nan nan nan nan nan nan nan nan

  array([nan, nan, nan, ..., nan, nan, nan], dtype=float32)

• Coordinates: (1)
  • lat
    (lat)
    float64
    89.94 89.82 89.71 ... -89.82 -89.94

    long_name :

    latitudes

    type :

    uniform

    units :

    degrees_north

    valid_min :

    -90.0

    valid_max :

    90.0

    axis :

    Y

    array([ 89.941406,  89.824219,  89.707031, ..., -89.707031, -89.824219,
           -89.941406])

• Indexes: (1)
  • lat
    PandasIndex

    PandasIndex(Index([         89.941406,  89.82421850032573,  89.70703100065145,
            89.58984350097718,  89.47265600130291,  89.35546850162864,
            89.23828100195436,  89.12109350228009,  89.00390600260582,
            88.88671850293154,
           ...
            -88.8867185029406, -89.00390600261485, -89.12109350228914,
           -89.23828100196343, -89.35546850163769, -89.47265600131195,
           -89.58984350098623, -89.70703100066052, -89.82421850033478,
           -89.94140600000904],
          dtype='float64', name='lat', length=1536))

• Attributes: (0)

Percent bias

Let’s imagine that fcst.temp_scrn represents the forecasted streamflow values and obs.temp_scrn represents the observed streamflow values. Now, we want to create another dataset with significantly higher values by adding 1000 to both the observed and forecasted streamflow values.

import matplotlib.pyplot as plt
fcst_flow = fcst.temp_scrn
obs_flow = obs.temp_scrn
# Calculate percent bias and preserve the "lat" dimension
bias = pbias(fcst_flow, obs_flow, preserve_dims="lat")
bias.name = "percent bias (%)"
# Generate second dataset with large flow values
fcst_flow_large=fcst_flow + 1000
obs_flow_large = obs_flow + 1000
# Calculate percent bias for large flow values and preserve the "lat" dimension
bias_large_flow = pbias(fcst_flow_large, obs_flow_large, preserve_dims="lat")
bias_large_flow.name = "percent bias (%)"
# Compare the percent bias for small and large flow values
bias.plot(label='pbias_small_streamflow')
bias_large_flow.plot(label='pbias_large_streamflow')
plt.legend()
plt.show()

As expected, the percent bias is higher for small forecast and observation values compared to large ones, although the absolute additive bias is the same for both datasets.

What to try next?

Have a look at Isotonic Regression in scores to see how conditional biases can be calculated.

Further Reading

Additive Bias (Mean Error)

• https://www.cawcr.gov.au/projects/verification/#meanerror

Multiplicative Bias

• https://www.cawcr.gov.au/projects/verification/#multiplicative_bias

Percent Bias

• Sorooshian, S., Duan, Q., & Gupta, V. K. (1993). Calibration of rainfall-runoff models: Application of global optimization to the Sacramento Soil Moisture Accounting Model. Water Resources Research, 29(4), 1185-1194. https://doi.org/10.1029/92WR02617

• Alfieri, L., Pappenberger, F., Wetterhall, F., Haiden, T., Richardson, D., & Salamon, P. (2014). Evaluation of ensemble streamflow predictions in Europe. Journal of Hydrology, 517, 913-922. https://doi.org/10.1016/j.jhydrol.2014.06.035

• Dawson, C. W., Abrahart, R. J., & See, L. M. (2007). HydroTest: A web-based toolbox of evaluation metrics for the standardised assessment of hydrological forecasts. Environmental Modelling and Software, 22(7), 1034-1052. https://doi.org/10.1016/j.envsoft.2006.06.008

• Moriasi, D. N., Arnold, J. G., Van Liew, M. W., Bingner, R. L., Harmel, R. D., & Veith, T. L. (2007). Model evaluation guidelines for systematic quantification of accuracy in watershed simulations. Transactions of the ASABE, 50(3), 885-900. https://doi.org/10.13031/2013.23153