A measure of model reliability

Lynn R. Williams, Richard W. Leggett

Research output: Contribution to journalArticlepeer-review

9 Scopus citations

Abstract

A method for evaluating predictive models is developed by giving a precise and statistically meaningful interpretation to the statement that a model is accurate within a factor of k. This method is applicable to any model for which there is a set {y1,…, yn} of observations corresponding to a set {x1, …, xn} of model predictions. We define a geometrically intuitive measure of model reliability kg in terms of the ratios yi/xi and a statistically rigorous measure ks in terms of ln (yi/xi). For reasonably accurate models, kg and ks are in virtual agreement and thus can be used interchangeably as a reliability index. The index ks estimates a model parameter of the form exp[(V1+ V2)1/2], where V1 describes an observational variance and V2 is related to an uncertainty associated with the model itself. The computed value for ks is not unique but depends on the sample of observations. The probability distribution of ks can be characterized provided the observational distributions are lognormal, independent, and satisfy a homoscedasticity condition. These requirements are often satisfied by quantities of interest in radiation risk analyses. The reliability indices kg and ks may be applied even if the underlying observational distributions are not lognormal, although the probability distribution of ks cannot be characterized in this case. .

Original languageEnglish
Pages (from-to)85-95
Number of pages11
JournalHealth Physics
Volume46
Issue number1
DOIs
StatePublished - Jan 1984

Fingerprint

Dive into the research topics of 'A measure of model reliability'. Together they form a unique fingerprint.

Cite this