Chapter 3

The Daubert Worldview

"If you don't know where you are going, you'll end up somewhere else."

-- Yogi Berra

Chapter 3: Daubert Misinterpreted

The good people at Ecology and Environment, Inc., formerly displayed at their web site a graphic, preserved here for posterity, that tidily if inadvertently encapsulates most of the elementary mistakes it is possible to make about the Daubert decision.

Let us leave to one side any marketing message intended to be conveyed by the menacing figure of the judge, who appears ready to apply his gavel forcefully to the cranium of any expert (or lawyer?) who cannot successfully run the gauntlet represented in the flow chart he bestrides. Let us leave aside, as well, the wisdom of committing one's Daubert fortunes to the care of persons who cannot spell "admissible." It is better to focus on the content of the flow chart itself.

The chart identifies four tests for scientific testimony, and presents each as a necessary condition for admissibility. Therein lies the chart's first error. Some of the criteria listed in the flow chart do bear some resemblance to some of the factors identified in Daubert as bearing on admissibility. But the Daubert decision explicitly warns that its discussion of these factors rises only to the level of "general observations," and that the factors listed are not exhaustive and do not constitute "a definitive checklist or test." The decision also emphasizes that testimony may be admissible even where one or more of the factors are unsatisfied. Publication, for example, "is not a sine qua non of admissibility," according to the Court's opinion, and "does not necessarily correlate with reliability." Likewise, as the chart's authors appear not to have noticed, the entire burden of the Daubert decision is to reject the previously prevailing rule that scientific evidence must necessarily be founded on theories that have won general scientific acceptance. "Nothing in the text of [Rule 702]," the opinion says, "establishes 'general acceptance' as an absolute precondition to admissibility."

In short, the "flow" set forth in the chart is simply wrong. It is, indeed, so spectacularly wrong that the chart would arguably come closer to an accurate depiction if a "NO" were substituted for each "YES," a "YES" substituted for each "NO," and the words "INADMISSABLE" [sic] and "ADMISSABLE" [sic] transposed.

So commonplace is this error (i.e., the "necessary condition" fallacy), and so frequently is its refutation trotted out, that it is easy to forget that Daubert does in fact specify criteria that expert scientific evidence must satisfy. As we saw in Chapter 2, those criteria are two. First, to assist the trier of fact in determining a fact in issue, as required under Rule 702, the evidence must be relevant. This is supposed to be a very weak condition. Under Fed. R. Evid. 401, it is satisfied if the evidence tends to make some fact in issue more probably true than it would be in the absence of the evidence. Second, to constitute scientific knowledge under Rule 702, the evidence must "grounded in the methods and procedures of science" -- or, as the Daubert opinion also puts it, the inferences and assertions embodied in the testimony must be "derived by the scientific method" and "supported by appropriate validation."

Our much derided flow chart, therefore, does contain at least one accurate representation. Only if expert scientific testimony rests on scientific methods will the testimony be admissible under Daubert. Even in capturing this valid point, however, the chart misportrays its logical status, by placing the testimony's grounding in scientific methodology on equal conceptual footing with the factors bearing on whether the testimony is thus grounded. As Daubert makes plain, the decisive question is whether the testimony rests on methods or techniques that fairly count as scientific, and such factors as peer review and rate of error are not the criterial equals of this requirement, but rather subsidiary indicia potentially supporting the conclusion that the requirement is satisfied.

As for the chart's actual recital of the relevant factors, the chart's authors have failed at even the rudimentary task of accurately listing the factors set forth in Daubert. Most conspicuously, the chart fails even to mention what the Daubert opinion describes as a "key question" -- viz., whether the theory or technique upon which the testimony is based has been, or can be, tested. At the same time, the chart elevates to canonic status a factor nowhere appearing in the Daubert opinion -- viz., whether the scientific theory on which the testimony is founded has "specified the reliability of statistical information on which it relies." This may be the authors' stab at characterizing the meaning of the Court's "rate of error" factor, but if so, the authors have assigned to this factor a narrow and specific meaning not adumbrated in the actual text of the Court's opinion (from which the word "statistics" and its derivatives are completely absent in this context).

One would not dwell on these last particular shortcomings of this particular flow chart, were there not a more general lesson to be learned. By now, hundreds of judicial opinions and secondary sources contain putative summaries of the so-called "Daubert factors." Too frequently, these summaries incorporate an overlay of interpretation, more or less controversial, that their authors fail to disclose explicitly. One good example is the question whether experts have developed their theories for purposes of litigation. Whether this is an appropriate consideration in gauging the reliability of expert testimony on any particular occasion is not a question that needs to be addressed or resolved at the moment. What does deserve comment is that this "factor" is sometimes listed as though the Daubert Court had specifically commended it to judicial attention. The truth is that the Daubert opinion does not specifically identify the development of expert theories for litigation as a factor, damning or otherwise, in gauging reliability. The lesson, then, is that glib recitations of "the Daubert factors" should be an occasion for counting the intellectual silverware, by comparing the factors listed in such recitations with the text of the Daubert decision and other governing legal authority.

We may allow our flow chart to rest in peace after one last flaw is noted. Despite Supreme Court's express admonition against treating the Daubert opinion as though it embodied some simplistic "checklist" for evidentiary reliability, and despite the indefatigable repetition of that admonition by subsequent courts and commentators, the factors discussed by the Court in Daubert are very commonly marched forth, in single file, as though they were part of a simple checklist, with each item bearing roughly comparable weight and significance. To reject this way of thinking, one need proceed no further than the Daubert opinion itself, which is not short on hints about the different importance and significance that the listed factors may have. The opinion takes care to note, for example, that peer review and publication are of diminished importance for expert testimonial propositions that are "too particular, too new, or of too limited interest to be published." It especially highlights the "key" importance of testability, in stark contrast to the opinion's much more modest claim that general acceptance "may yet have a role to play." And it emphasizes, famously, that the focus must be on the expert's methods, and not on the conclusions those methods generate. We need not pause here for a detailed analysis of the comparative importance and significance to be assigned to various indicia of reliability, or of the interdependence of at least some such indicia. For the moment, it need be observed only that the inquiry mandated under Daubert is not a mechanical exercise; it is, or should be, richly textured and context-dependent. Lip service is often paid to that truism. But as our whipping-boy chart amply illustrates, the truism's truth is not always heeded in practice.

Coming soon: Chapter 4 -- The Logic of Daubertian Inquiry

Chapter 3