Interactive online version: . Download notebook.

Receiver Operating Characteristic (ROC)

A ROC curve shows the ability of a probabilitic forecast of a binary event (e.g., chance of rainfall) to discriminate between events and non-events. It plots the hit rate or probability of detection (POD) against the false alarm rate or probability of false detection (POFD) using a set of increasing probability thresholds (e.g., 0.1, 0.2, 0.3, …) that convert the probabilistic forecast into a binary forecast.

The area under the ROC curve is often used as a measure of discrimination ability. ROC is insensitive to forecast bias so should be used together with reliablity or some other metric which is measures bias.

from scores.categorical import probability_of_detection, probability_of_false_detection
from scores.probability import roc_curve_data
import matplotlib.pyplot as plt
import numpy as np
import xarray as xr
from scipy.stats import beta, binom

Imagine we have a sequence of binary forecasts and a sequence of binary observations, where 1 indicates a forecast or observed event and 0 indicates a forecast or observed non-event.

fcst = xr.DataArray([1, 0, 1, 0, 1])
obs = xr.DataArray([1, 0, 0, 1, 1])

We can then use scores to calculate the POD (\(\frac{H}{H + M}\)) and POFD (\(\frac{F}{C + F}\)), where \(H, M, F, C\) are the number of hits, misses, false alarms, and correct negatives respectively.

probability_of_detection(fcst, obs)

<xarray.DataArray 'ctable_probability_of_detection' ()>
array(0.66666667)

xarray.DataArray

'ctable_probability_of_detection'

• 0.6667

  array(0.66666667)

• Coordinates: (0)
• Indexes: (0)
• Attributes: (0)

probability_of_false_detection(fcst, obs)

<xarray.DataArray 'ctable_probability_of_false_detection' ()>
array(0.5)

xarray.DataArray

'ctable_probability_of_false_detection'

• 0.5

  array(0.5)

• Coordinates: (0)
• Indexes: (0)
• Attributes: (0)

Now suppose that we have a forecast that is the probability of rain occuring, and corresponding observations where 1 means it rained, and 0 means that it didn’t rain. We generate some synthetic forecasts using a Beta distribution and generate some corresponding observations.

fcst = beta.rvs(2, 1, size=1000)
obs = binom.rvs(1, fcst)
fcst = xr.DataArray(data=fcst, dims="time", coords={"time": np.arange(0, 1000)})
obs = xr.DataArray(data=obs, dims="time", coords={"time": np.arange(0, 1000)})

thresholds = np.arange(0, 1.01, 0.01)

Now calculate the ROC curve.

roc = roc_curve_data(fcst, obs, thresholds)
roc

<xarray.Dataset>
Dimensions:    (threshold: 101)
Coordinates:
  * threshold  (threshold) float64 0.0 0.01 0.02 0.03 ... 0.97 0.98 0.99 1.0
Data variables:
    POD        (threshold) float64 1.0 1.0 1.0 1.0 ... 0.1125 0.0875 0.04063 0.0
    POFD       (threshold) float64 1.0 1.0 1.0 1.0 1.0 ... 0.0 0.0 0.0 0.0 0.0
    AUC        float64 0.8074

xarray.Dataset

• Dimensions:
  • threshold: 101
• Coordinates: (1)
  • threshold
    (threshold)
    float64
    0.0 0.01 0.02 ... 0.98 0.99 1.0

    array([0.  , 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 , 0.11,
           0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2 , 0.21, 0.22, 0.23,
           0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.3 , 0.31, 0.32, 0.33, 0.34, 0.35,
           0.36, 0.37, 0.38, 0.39, 0.4 , 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47,
           0.48, 0.49, 0.5 , 0.51, 0.52, 0.53, 0.54, 0.55, 0.56, 0.57, 0.58, 0.59,
           0.6 , 0.61, 0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.7 , 0.71,
           0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.8 , 0.81, 0.82, 0.83,
           0.84, 0.85, 0.86, 0.87, 0.88, 0.89, 0.9 , 0.91, 0.92, 0.93, 0.94, 0.95,
           0.96, 0.97, 0.98, 0.99, 1.  ])

• Data variables: (3)
  • POD
    (threshold)
    float64
    1.0 1.0 1.0 ... 0.0875 0.04063 0.0

    array([1.       , 1.       , 1.       , 1.       , 1.       , 1.       ,
           1.       , 1.       , 1.       , 1.       , 1.       , 1.       ,
           1.       , 1.       , 0.9984375, 0.9984375, 0.996875 , 0.9953125,
           0.9953125, 0.9921875, 0.9921875, 0.9921875, 0.9921875, 0.9921875,
           0.990625 , 0.9890625, 0.9875   , 0.9875   , 0.984375 , 0.9828125,
           0.9828125, 0.978125 , 0.975    , 0.9734375, 0.9734375, 0.9671875,
           0.9625   , 0.95625  , 0.9515625, 0.9484375, 0.9390625, 0.9328125,
           0.928125 , 0.925    , 0.9203125, 0.9125   , 0.9015625, 0.89375  ,
           0.8890625, 0.8828125, 0.88125  , 0.871875 , 0.8671875, 0.8578125,
           0.853125 , 0.8453125, 0.834375 , 0.83125  , 0.815625 , 0.8046875,
           0.79375  , 0.778125 , 0.765625 , 0.753125 , 0.7421875, 0.725    ,
           0.7140625, 0.7      , 0.690625 , 0.6671875, 0.65625  , 0.6390625,
           0.6203125, 0.596875 , 0.58125  , 0.575    , 0.5671875, 0.5546875,
           0.5328125, 0.51875  , 0.5      , 0.4890625, 0.465625 , 0.446875 ,
           0.43125  , 0.4140625, 0.3921875, 0.3734375, 0.346875 , 0.3328125,
           0.3140625, 0.2828125, 0.259375 , 0.2296875, 0.2046875, 0.1734375,
           0.1375   , 0.1125   , 0.0875   , 0.040625 , 0.       ])

  • POFD
    (threshold)
    float64
    1.0 1.0 1.0 1.0 ... 0.0 0.0 0.0 0.0

    array([1.        , 1.        , 1.        , 1.        , 1.        ,
           0.99722222, 0.99444444, 0.98888889, 0.98611111, 0.98055556,
           0.97777778, 0.97777778, 0.96944444, 0.95555556, 0.95277778,
           0.94444444, 0.93611111, 0.93055556, 0.92222222, 0.91388889,
           0.90277778, 0.89722222, 0.88611111, 0.86666667, 0.85555556,
           0.84444444, 0.84166667, 0.82777778, 0.81111111, 0.8       ,
           0.79444444, 0.78055556, 0.77222222, 0.75277778, 0.74722222,
           0.74166667, 0.725     , 0.70555556, 0.68611111, 0.66944444,
           0.64444444, 0.63055556, 0.61666667, 0.6       , 0.57777778,
           0.56388889, 0.54722222, 0.53333333, 0.51944444, 0.5       ,
           0.49166667, 0.46666667, 0.46111111, 0.45      , 0.42222222,
           0.40277778, 0.39722222, 0.38611111, 0.37222222, 0.35555556,
           0.33055556, 0.325     , 0.31111111, 0.29444444, 0.28333333,
           0.27222222, 0.25833333, 0.24722222, 0.24166667, 0.21666667,
           0.20555556, 0.19722222, 0.18611111, 0.18333333, 0.16666667,
           0.15833333, 0.14444444, 0.13888889, 0.13611111, 0.12222222,
           0.11666667, 0.11111111, 0.09722222, 0.07222222, 0.06388889,
           0.06111111, 0.05555556, 0.05555556, 0.04722222, 0.03888889,
           0.03611111, 0.03055556, 0.02777778, 0.01944444, 0.01111111,
           0.00277778, 0.        , 0.        , 0.        , 0.        ,
           0.        ])

  • AUC
    ()
    float64
    0.8074

    array(0.80740451)

• Indexes: (1)
  • threshold
    PandasIndex

    PandasIndex(Float64Index([               0.0,               0.01,               0.02,
                                0.03,               0.04,               0.05,
                                0.06,               0.07,               0.08,
                                0.09,
                  ...
                                0.91,               0.92,               0.93,
                  0.9400000000000001, 0.9500000000000001,               0.96,
                                0.97,               0.98,               0.99,
                                 1.0],
                 dtype='float64', name='threshold', length=101))

• Attributes: (0)

The output that we have calculated the POD and POFD for each threshold. We also calculated the AUC which is the area under the ROC curve, which we will discuss shortly. Let’s now plot the ROC curve.

plt.title('Receiver Operating Characteristic')
plt.plot(roc.POFD, roc.POD, 'b', label = 'AUC = %0.2f' % roc.AUC.item())
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('POD')
plt.xlabel('POFD')
plt.show()

• A perfect ROC curve would go from the lower left corner, up to the top left corner, and across to the top right corner.

• If the curve follows the dotted line, it means the forecast has no discrimination ability. This synthetic forecast clearly does have discrimination ability.

• The AUC is the area under the curve. 1 is the maximum value that can be achieved. If we picked a random forecast value from our dataset where the event occurred, and a random forecast value when the event did not occur, the AUC tells us the probability that the former forecast value will be greater than the latter.

Cautionary notes

• ROC or AUC is not a suitable overall performance measure for a probabilistic classifier as it is not a proper scoring rule and is ignorant of calibration (Hand 2009; Hand and Anagnostopoulos 2013, 2023). It is however, a useful tool for understanding potential predictive performance subject to recalibration.

• Non-concave ROC curves like the synthetic example above have several problems (Pesce et al. 2010; Geniting and Vogel 2022). To get around these problems, you can apply isotonic regression to the forecast data before you calculate the ROC curves (Fawcett and Niculescu-Mizil 2007).

• Results may be misleading if two ROC curves cross (Hand 2009).

References

Fawcett, T. and Niculescu-Mizil, A., 2007. PAV and the ROC convex hull. Machine Learning, 68, pp.97-106.

Gneiting, T. and Vogel, P., 2022. Receiver operating characteristic (ROC) curves: equivalences, beta model, and minimum distance estimation. Machine learning, 111(6), pp.2147-2159.

Hand, D.J., 2009. Measuring classifier performance: a coherent alternative to the area under the ROC curve. Machine learning, 77(1), pp.103-123.

Hand, D.J. and Anagnostopoulos, C., 2013. When is the area under the receiver operating characteristic curve an appropriate measure of classifier performance?. Pattern Recognition Letters, 34(5), pp.492-495.

Hand, D.J. and Anagnostopoulos, C., 2023. Notes on the H-measure of classifier performance. Advances in Data Analysis and Classification, 17(1), pp.109-124.

Pesce, L.L., Metz, C.E. and Berbaum, K.S., 2010. On the convexity of ROC curves estimated from radiological test results. Academic radiology, 17(8), pp.960-968.