Interactive online version:
.
Download notebook.
Murphy Diagrams#
The Murphy diagram is a powerful tool for understanding how forecast performance differs for various user decision thresholds (Ehm et al., 2016).
Given many forecasts, the Murphy diagram shows the mean score for as a function of potential user decision thresholds \(\theta\). It uses the relevant elementary score for the forecast, be it an
expectile (e.g. mean)
quantile (e.g. median)
Huber quantile, or
probabilistic forecast of a binary outcome.
For information about elementary scores, see the end of this tutorial.
We’ll use scores
to create a Murphy diagram comparing two sets of synthetic temperature forecast data.
[1]:
from scores.continuous import murphy_score, murphy_thetas, mse
import numpy as np
import xarray as xr
from scipy.stats import skewnorm
import matplotlib.pyplot as plt
import plotly.express as px
np.random.seed(100)
[2]:
# Read the doc string for the murphy_score
# help(murphy_score)
[3]:
# Read the doc string for murphy_thetas
# help(murphy_thetas)
[4]:
# Generate some synthetic observations between 0 and 40 repesenting temperature in Celsius
N = 1000
obs = xr.DataArray(data=40 * np.random.random(N), dims=["time"], coords={"time": np.arange(0, N)})
# Generate synthetic forecasts by adding noise to each observation
fcst1 = 0.9 * obs + skewnorm.rvs(4, size=N) # fcst1 has a low bias and is unlikely to forecast outside 0 to 39
fcst2 = 1.1 * obs - skewnorm.rvs(4, size=N) # fcst2 has a high bias and forecasts colder than 0 and warmer than 40
[5]:
fig = px.scatter(title="First 20 observations and corresponding forecasts")
fig.add_scatter(y=obs[:20], name = "Observations", line=dict(dash='solid'),)
fig.add_scatter(y=fcst1[:20], name = "fcst1 (biased cool)", line=dict(dash='dot'),)
fig.add_scatter(y=fcst2[:20], name = "fcst2 (biased warm)", line=dict(dash='dash'),)
fig.update_layout(xaxis_title="Forecast lead time",)
fig.update_layout(yaxis_title="Temperature (Celcius)",)
fig.show()