Commentary: An argument against E-values for assessing the plausibility that an association could be explained away by residual confounding

International Journal of Epidemiology, vol. 49, pp. 1501–1503

Abstract

A major source of uncertainty in non-randomized studies is the confounding left after adjustment by a set C of measured variables (uncontrolled, unmeasured or residual confounding).1–9 Imagine a context in which the scientific question is this: Can residual confounding alone ‘plausibly’ account for the reported C-adjusted risk ratio RRFD relating exposure F to disease D? The emphasis is on ‘plausibly’ because this question is not one of algebraic bounds or hypothetical examples, but rather of real data and context. There is grave concern that, despite rigorous, detailed efforts to promote its proper interpretation,1–4 the E-value Eval = RRFD + [RRFD(RRFD−1)]½ is being taken as providing a general, realistic answer to the question, despite the fact that its relevance hinges on conditions that are strong and often highly implausible in light of available data.5–7 This plausibility problem is one of absolute (logical) bounds in general, but Eval is the first bound to gain popularity and bring the problem to the fore. Eval has been defined verbally as ‘the minimum strength of association, on the risk ratio scale, that an unmeasured confounder would need to have with both the treatment and the outcome to fully explain away a specific treatment–outcome association, conditional on the measured covariates’.1 But that definition presumes conditions can be met which may be implausible and even contradicted by available data. In those cases, Eval easily leads to exaggerated impressions of residual confounding risk, and thus appears to be a misleading indicator of ‘likely’ residual confounding.5–7 A full scientific (contextual) justification for using Eval in an answer to the scientific question would empirically address the following conditions, which are implicit in taking Eval as a plausible (as opposed to logical) ‘minimum strength’ for uncontrolled confounders to explain away RRFD. The exposure F is binary, or else accurate accounting has been made for the information lost by its dichotomization (which is difficult and thus rarely done). The plausibility of potential ‘explaining away’ can be evaluated with further adjustments involving just two levels of a single unmeasured confounder U at a time, ignoring the complexities of the full multivariate confounder (C, U) distribution and its effects. The risk ratios relating the uncontrolled confounder U to exposure F and disease D can plausibly approximate each other through a shared value RRU that is also constant across all C-adjustment levels (or at least a certain average RRU of the UF and UD risk ratios is roughly constant). It is plausible that, within every adjustment (C) level, the exposure and confounder distributions are those that maximize confounding by U under condition (iii), to explain away RRFD. Condition (i) is often satisfied, and sometimes a case can be made for condition (ii). But conditions (iii) and (iv) together would rarely if ever be remotely credible: not only are they contrived to maximize confounding, but they imply extreme conditions (e.g. exposure never occurs without the confounder). When combined with the usual regression assumption of a constant RRFD across C-levels, they further suggest that the exposure prevalence stays fairly constant across those C-levels, which would make the adjusted RRFD roughly equal to the unadjusted risk ratio (i.e. RRFD would be approximately collapsible over C). Effectively then, the worst case [conditions (iii) and iv)] can only remain plausible when conventional C-adjustment fails to control much confounding. Such a scenario usually contradicts available data and contextual information, making Eval understate the actual minimum RRU needed to explain away RRFD. Expanding on the information unused by Eval, it is well known1–9 that (a) no matter how strong a confounder’s effects, its ability to bias RRFD is limited by its prevalence (or, more generally, its distribution); (b) very strong risk factors may produce little confounding due to weak association with exposure; (c) very strong exposure indications may produce little confounding because of weak effects on disease; (d) confounding by unmeasured factors may be weakened considerably due to their associations with strong controlled confounders (e.g. age and sex); and (e) the biases from different uncontrolled confounders may cancel. Any bound (including Eval) that ignores these facts can seriously mislead one about the ease with which RRFD could be explained away by further adjustments. Consider a study of in utero serotonergic antidepressant exposure and subsequent autism-spectrum diagnosis, which reported a hazard ratio of 1.59 (95% limits 1.17, 2.17) from a Cox model with 12 potential confounders composing C (Table 2),10 yielding Eval = 1.59 + (1.59·0.59)½ ≈ 2.6 (the outcome is uncommon enough to ignore hazard vs risk ratio distinctions). Many users will interpret this 2.6 as the minimum confounder effect RRU on F and D needed to explain away RRFD ≈ 1.59, with no mention of conditions (i)–(iv).5–7 But the data strongly indicate the study is not anywhere near the worst case assumed by Eval: first, the unadjusted hazard ratio is 2.16 (1.64, 2.86), well above the adjusted RRFD (P < 0.00001 from a collapsibility test11); and second, although the above study could have easily stopped at and published the Cox-model estimate with its E-value, it went on to adjust for several hundred further covariates via propensity scoring, obtaining RRFD ≈ 1.61 (95% limits 0.997, 2.59). Thus, further adjustment did nothing to explain the original adjusted 1.59 in terms of confounding (as opposed to ‘random error’). The study did subset analyses to achieve further control, but those analyses also did little to the point estimate (e.g. a sib-matched adjusted analysis yielded RRFD ≈ 1.60, 95% limits 0.63, 3.74). This example shows how Eval can distract attention toward a purely mathematical possibility with no regard for how much that possibility conflicts with the data or adjustment sensitivity. In practical terms, conditions (ii)–(iv) assert that all relevant available information about multiple uncontrolled covariates (including their intercorrelations, bias reinforcements, and bias cancelations among themselves and the controlled covariates C) can be represented in a model with a single summary U. That may be reasonable when the main confounding concern is failure to adjust for one variable. But if U represents a large collection of uncontrolled factors, the connection of Eval to the underlying reality is opaque and divorced from relevant available data. In particular, the confounding by uncontrolled factors can vary in direction, resulting in cancellation even when experts expect only reinforcement. The lesson is that confounding should be examined using data analyses, not authoritative judgements or worst-case assumptions (of which pure reinforcement is one). Blum et al. state: ‘We worry that E-values may be used to dismiss the threat of residual confounding without sufficient and robust consideration’.7 This is a concern to emphasize if one fears erroneous rejections of the null (false positives) more than erroneous claims of ‘no association’ (false negatives). But to promote scientific neutrality,12,13 we must also worry that Eval may be used to raise the threat of residual confounding without sufficient, robust consideration5—as it would have in the above example had the study reported Eval instead of further adjustments. Such failure to address bias toward the null is already enshrined in ritualized null-hypothesis significance testing (which has caused extensive damage to the scientific literature, resulting in calls for its abandonment while nonetheless retaining P-values14,15), so the bias hardly needs another layer of reinforcement. The general problem with Eval as a minimum strength is its cognitive null-bias from its worst-case conditions (iii)–(iv). By understating the confounder effects needed to ‘plausibly’ produce RRFD in light of the full data and considerations like (a)–(e) above, Eval exaggerates the ease of explaining away RRFD by residual confounding—especially when it is applied to lower confidence limits (where it exaggerates uncertainty due to its failure to properly merge random error with unknown confounding8). The E-value uses no background or data information on confounders or prevalences, and no expectations about unobserved confounders or intercorrelations with controlled confounders. Instead it assumes only a worst-case scenario for the bias parameters, reducing it to nothing more than a transform of the observed risk ratio. In particular, as seen above, Eval suffers from a fatal statistical flaw: it ignores data information that refutes the worst-case conditions needed for it to provide an achievable minimum. How difficult are formulas that instead use that information to exclude implausible conditions when evaluating residual-confounding risk? Well, several such formulas are (like Eval) suitable for pocket calculators and spreadsheets,8,9 yet allow non-extreme and symmetric treatment of unknowns like confounder prevalences. Such formulas facilitate full portrayal of uncertainty about bias toward as well as away from the null—provided the user is acquainted enough with the topic, study design and data to specify plausible values for their inputs. Although divergences remain about the value of E-values, this commentary benefitted immensely from critical comments from Ghassan Hamra, Sebastien Haneuse, John Ioannidis, Jay Kaufman, Richard MacLehose and Tyler VanderWeele. None declared.

Authors 1

1. Sander Greenland corresponding Aachen Department of Clinical Epidemiology

   University of California, Los Angeles · Leiden University Medical Center

   Affiliation as printed

   Department of Epidemiology and Department of Statistics, University of California, Los Angeles, Los Angeles, CA, USA

   Department of Clinical Epidemiology, Leiden University Medical Center, Albinusdreef 2, 2333 ZA Leiden, The Netherlands