Interactive online version:

Introduction to weighting and masking#

When calculating mean verification scores, not all data points should always contribute equally. Depending on the projection, the physical area represented, or the region of interest, you may need to adjust the contribution of each grid cell by applying weights or masks.

To address these issues, scores provides a flexible way to apply weights or masks when calculating the mean scores. Weighting is handled by the function scores.processing.aggregation, which works similarly to xarray’s built-in weighted method.

Examples include:

The gridded forecasts and observations aren’t on an “equal area” map projection. One common map projection is the latitude-longitude projection. On this projection, the grid cells near the equator account for a much larger area than the grid cells near the poles.
You may want to weight the mean score by the relative importance of each point (e.g., by a population density grid, or a vulnerability grid).
You may want to mask data over the ocean to only evaluate a forecast over land

First we will discuss weighting in more detail before moving on to masking. Weighting in the metrics in scores is handled by the function scores.processing.aggregation. To calculate a weighted score, you must supply the weights arg with an xr.Dataset or xr.DataArray. Our approach follows the weighted approach that xarray uses. See the xarray tutorial for further information.

Let’s step through an example of how the weighting works. Suppose we have a 2D grid and we want to calculate the weighted mean with some weights.

First, the squared error \(x_{i, j}\) at the \((i,j)\)th point is calculated. If we have weights \(w\), we can calculate the weighted Mean Squared Error (MSE), \(\bar{x}\) as

\[\bar{x} = \frac{\sum_{i=1}^{M} \sum_{j=1}^{N} w_{i,j} \, x_{i,j}}{\sum_{i=1}^{M} \sum_{j=1}^{N} w_{i,j}}\]

where \(i\) is a latitude for \(M\) latitudes, and \(j\) is the longitude for \(N\) longitudes.

What if I have NaNs in my weights?#

If your weight array contains NaN values, it is not always clear how these should be handled. Because different applications may require different choices, scores will raise an error rather than making an assumption for you.

In most cases, filling missing values with zeros is a sensible approach, since a weight of zero means “this grid cell does not contribute.” You can do this with:

weights = weights.fillna(0)

To demonstrate why this is the case, imagine that we have 3 points and we want to calculate a weighted mean. Let’s say that our errors array is \(x=[1, 3, 5]\). Now suppose that our weights array is \(w=[3, 1, \mathrm{NaN}]\). We expect in this case that the weighted mean should only consist of the first two points and give us a weighted mean of 1.5. Let’s step through this:

Make the NaN value in the weights 0, so that our weights are [3, 1, 0]
Calculate the weighted mean as

\[\mu_w = \frac{\sum_{i=1}^{n} w_i x_i}{\sum_{i=1}^{n} w_i} = \frac{1 \times 3 + 3 \times 1 + 5 \times 0}{3 + 1 + 0}=\frac{6}{4}=1.5\]

Weighting data directly#

Some users may want to apply the weighting function themselves after preserving all dimensions when calculating the score. This could occur when calculating confidence intervals or wanting to calculate the mean one dimension at a time in a specific order when there are NaNs in the data.

scores makes the function used under the hood in each metric (scores.processing.aggregate) available to use. Let’s demonstrate this by repoducing the latitude weighted MSE in two steps.

[13]:

from scores.processing import aggregate

[14]:

# First calculate the squared error at each point
se = mse(fcst, obs, preserve_dims="all")
lat_weighted_mse = aggregate(se, reduce_dims=["time", "latitude", "longitude"], weights=lat_weights)
lat_weighted_mse

[14]:

<xarray.DataArray '2m_temperature' ()> Size: 8B
array(0.45555855)
Coordinates:
    prediction_timedelta  timedelta64[ns] 8B 2 days

We can show that the following two are equal

[15]:

xr.testing.assert_equal(lat_weighted_mse, mse(fcst, obs, weights=lat_weights))

The behaviour of scores.processing.aggregate is as follows:

If reduce_dims is None, no aggregation is performed and the original values are returned unchanged.
If weights is None, an unweighted mean is computed. If weights are provided, negative weights are not allowed and will raise a ValueError.
If weights are provided but reduce_dims is None (i.e., no reduction), a UserWarning is emitted since the weights will be ignored

There is also a method arg which takes "mean" (default) or "sum". In most cases "mean" is appropriate for weighted aggregations. method="sum" can also be used for producing weighted sums. A use case for method="sum" is when calculating a weighted contingency table score such as probability of detection. In this case a weighted sum is applied to the misses and the hits, before calculating the probability of detection.

Things to try next#

Try and calculate weighted scores with methods other than MSE
Download a population density grid and create a mean score weighted by population density.

[ ]:

Introduction to weighting and masking

Contents

Introduction to weighting and masking#

Weighting example#

What if I have NaNs in my weights?#

Masking example#

Weighting data directly#

Things to try next#