Interactive online version: . Download notebook.

Murphy Diagrams

The Murphy diagram is a powerful tool for understanding how forecast performance differs for various user decision thresholds (Ehm et al., 2016).

Given many forecasts, the Murphy diagram shows the mean score for as a function of potential user decision thresholds \(\theta\). It uses the relevant elementary score for the forecast, be it an

• expectile (e.g. mean)

• quantile (e.g. median)

• Huber quantile, or

• probabilistic forecast of a binary outcome.

For information about elementary scores, see the end of this tutorial.

We’ll use scores to create a Murphy diagram comparing two sets of synthetic temperature forecast data.

from scores.continuous import murphy_score, murphy_thetas, mse
import numpy as np
import xarray as xr
from scipy.stats import skewnorm
import matplotlib.pyplot as plt
import plotly.express as px

np.random.seed(100)

# Read the doc string for the murphy_score
# help(murphy_score)

# Read the doc string for murphy_thetas
# help(murphy_thetas)

# Generate some synthetic observations between 0 and 40 repesenting temperature in Celsius
N = 1000
obs = xr.DataArray(data=40 * np.random.random(N), dims=["time"], coords={"time": np.arange(0, N)})

# Generate synthetic forecasts by adding noise to each observation
fcst1 = 0.9 * obs + skewnorm.rvs(4, size=N) # fcst1 has a low bias and is unlikely to forecast outside 0 to 39
fcst2 = 1.1 * obs - skewnorm.rvs(4, size=N) # fcst2 has a high bias and forecasts colder than 0 and warmer than 40

fig = px.scatter(title="First 20 observations and corresponding forecasts")
fig.add_scatter(y=obs[:20], name = "Observations", line=dict(dash='solid'),)
fig.add_scatter(y=fcst1[:20], name = "fcst1 (biased cool)", line=dict(dash='dot'),)
fig.add_scatter(y=fcst2[:20], name = "fcst2 (biased warm)", line=dict(dash='dash'),)
fig.update_layout(xaxis_title="Forecast lead time",)
fig.update_layout(yaxis_title="Temperature (Celcius)",)
fig.show()