Uncertainty Analysis of the Global Burden of Disease Estimates
The 2001 GBD study estimated mortality and the burden of disease for a comprehensive set of disease and injury causes and for all regions of the world, including regions with limited, incomplete, and uncertain data. To allow users of the information to assess whether the information uncertainty range is compatible with the purpose at hand, providing some analysis and guidance on levels of uncertainty is important (Murray, Mathers, and Salomon 2003). This is difficult to do, because apart from the large number and disparate nature of the data sources used (see chapter 3), information or knowledge about the quality of and potential biases in the data is often limited. This and the following sections provide an overview of initial efforts to quantify the uncertainty associated with the estimation of deaths by cause, with disability weights, and with epidemiological estimates of incidence and prevalence for GBD 2001.
Sources of Uncertainty
Uncertainty in estimated disease burden may arise from the following sources:
-
incomplete information, for example, when estimates for a population are based on observations from a sample;
-
potential biases in information, for instance, issues concerning the representativeness for a whole population of estimates from a study of a subgroup or the validity of a survey instrument in addressing the quantity of interest;
-
heterogeneity or from disagreements among information sources, as when several studies give different estimates for the same quantity of interest;
-
model uncertainty, for example, the variables or functional form specified in a regression model;
-
the data generation process itself; for instance, investigators may only infer risks from event counts in a population, which means that they can never know the risks themselves with certainty.
The most familiar and most commonly quantified kind of uncertainty arises from random error in the direct measurement of a quantity. An estimate of an epidemiological quantity for a population will have uncertainty arising from the finite sample used in the study as well as from random measurement error. The standard error of the mean or the confidence interval for such a quantity specifies the distribution of uncertainty in knowledge of the true mean value in the population (assuming no systematic error).
Most measurement involves not only random (stochastic) error, but also systematic error arising from biases in the measurement instrument, for instance, unrepresentativeness of a sampling frame for a survey, or from inaccuracies in the assumptions used to infer the actual quantity from the available data, for example, estimating the prevalence of a disease for a country from studies of representative sub-populations. Examinations of historical measurements reveal a consistent tendency to underestimate systematic error, perhaps because systematic error usually relates to sources of error that are unknown or about which little is known. Ignoring systematic error when estimating uncertainty is common, but this often results in substantial underestimation of the true uncertainty (Morgan and Henrion 1990).
Putting upper and lower bounds on the systematic error component is often possible, for example, where a disease process has biological limits or where evidence from a range of populations provides likely upper and lower limits to an epidemiological parameter such as prevalence or case fatality. In addition, consistency analysis across the various inputs for the DALY calculation (incidence, prevalence, case fatality rates or relative risk of mortality, and remission rates) often helps identify sources of systematic error and provides some basis for quantifying them (Kruijshaar, Barendregt, and Hoeymans 2002; Mathers, Murray, and Lopez 2002). This is discussed further in chapter 3.
Much of the uncertainty in estimates of deaths or DALYs for the 2001 GBD study is associated with the assessment of systematic errors in primary data. Chapter 3 examined primary data sources and their reliability in some detail and provided summary tabulations of the numbers of data sources available across regions and causes. This review clearly indicated that even though most countries have some information about prevalence, incidence, and mortality from some diseases and injuries and about population exposures to risk factors, it is generally fragmented, partial, incomparable, and diagnostically uncertain. One of the explicit aims of the GBD approach is to provide a coherent framework for integrating, validating, analyzing, and disseminating fragmentary information on the health of populations so that it is truly useful for health policy and planning. An important aspect of this framework is to assess the reliability and validity of data, particularly in relation to systematic error, and hence to provide some guide to the uncertainty in the resulting estimates.
Describing and Quantifying Uncertainty
We follow Morgan and Henrion's (1990) approach toward interpreting and using probability to describe and quantify uncertainty. The classical or frequentist view of probability defines the probability of an event occurring in a particular trial or experiment as the frequency with which it would occur during a long sequence of similar experiments. For many quantities of real interest, it is difficult to imagine how to operationalize a long sequence of relevant, similar experiments. An example of such a quantity would be the probability, estimated in late 2005, that avian influenza will cause a major global epidemic with deaths exceeding, say, 1 million in 2006. One approach has been to distinguish events whose probabilities are knowable through a series of experiments from those whose probabilities are unknowable or uncertain because no unique and operationalizable set of similar experiments exists, but this essentially limits the use of probabilities to games of chance.
Alternatively, a Bayesian view of probability defines it as the degree to which a person believes that an event will occur, or that a parameter has a certain value, given all the relevant information currently known to that person. Because different people have different information, they may legitimately assign different probabilities to the same event. These subjective probabilities must obey all the same axioms and rules as frequentist probabilities. These conceptual distinctions do not usually affect the practice of statistical inference, and essentially the same formal inference models of probability may be applied (King, Tomz, and Wittenberg 2000; Morgan and Henrion 1990). Moreover, when an empirical series of data from trials becomes available, the Bayesian assessment of probability should converge to the frequentist assessment, assuming the Bayesian approach uses the data rationally to update the assessments.
Our general approach to describing and estimating uncertainty in quantities of interest is to express them as probability distributions using a Bayesian interpretation of probability as expressing uncertainty of an observed or hypothetical event given a set of assumptions about the world. Probability distributions can therefore be used to express uncertainty about epidemiological quantities, such as the prevalence of depression in a particular population, the population values reflected in health state valuations, or the underlying risk of mortality due to a specific cause in a specific population.
Advances in computer technology have facilitated analytical methods for dealing with uncertainty enormously. One general approach to combining the uncertainties of multiple inputs into estimates relies on numerical simulation methods. The simulation approach uses multiple samples from probability distributions around uncertain inputs to allow estimates of the probability distributions around quantities of interest that may be complicated functions of these inputs, without the need to solve difficult, or in many cases insoluble, mathematical equations (King, Tomz, and Wittenberg 2000; Vose 2000).
