Antipodes

Critics of the American judicial system sometimes urge that experts should be retained by the courts, because that would foster impartial and unbiased testimony.

Now comes a report from Xinhua that experts will leave the employ of the Chinese courts, because that will help foster impartial and unbiased testimony.

11th Circuit Upholds Exclusion of Chemical Engineer's Testimony

In 1997, Mediterranean fruit flies were found in Florida. Government officials wanted to kill them, or at least to encourage them to fly back to the Mediterranean. So they decided to spray Florida with a mixture of protein bait and a pesticide known as malathion. The malathion was initially shipped from a Denmark plant to warehouses in Florida, Georgia, and Texas. From there, it was transported to Tampa, where authorities mixed it with the protein bait and then sprayed the city and surrounding areas. In short order, various Tampa domiciliaries became ill. They blamed the malathion and brought a putative class action. When malathion is stored above certain temperatures, you see, a process of chemical decomposition produces isomalathion, which is toxic to humans. And it can get hot in those southern warehouses in the summer.

But did it get hot enough? To show it did, plaintiffs retained Dr. Jack Matson, who boasted not only a Ph.D. in chemical engineering but also experience with pesticides (though not with malathion). Matson took weather data for the various storage locations and plugged them into a formula. Because of some indirect evidence that the temperature inside the Texas storage facility was actually 18 degrees higher than the ambient outdoor temperatures recorded by the weather service, he bumped up his temperature estimates for the warehouses in all three states, based on their structural similarities and basic latitudinal equivalence.

The district court excluded Matson's testimony, largely because Matson didn't actually investigate conditions at the non-Texas warehouses. Plaintiffs' causation case thereupon collapsed like a house of cards, and the claims were dismissed on summary judgment.

The Eleventh Circuit has released its opinion affirming the trial court's ruling. See Link v. Cheminova, Inc., No. 04-10160 (11th Cir. Feb. 24, 2005) (Birch, Barkett & Cox, JJ.). In its opinion, the panel rejects as unfounded the plaintiffs' argument that the lower court impermissibly evaluated Matson's credibility. According to the appellate opinion, the district court properly focused not on credibility, but on the reliability of Matson's methods. But the opinion does not stop there. It goes on to gloss the credibility/reliability distinction in a footnote, which we reproduce below:

We note that the argument made by the putative class representatives misconstrues the difference between a district court's evaluation of an expert's reliability, which is required by [Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579 (1993)], and an expert's believability or persuasiveness, which is reserved for the trier of fact, see [Quiet Tech. DC-8, Inc. v. Hurel Dubois UK, Ltd., 326 F.3d 1333, 1341 (11th Cir. 2003)]. As we explained in Quiet Technology, a district court's exercise of its gatekeeper function under Daubert "is not intended to supplant the adversary system or the role of the jury." See id. (citation omitted). Accordingly, a district court may not exclude an expert because it believes one expert is more persuasive than another expert. Additionally, a district court cannot exclude an expert because it believes the expert lacks personal credibility because of prior bad acts or other prior instances of untruthfulness. Vigorous cross-examination ensures that these issues of general credibility are properly presented for consideration by the trier of fact. See [Daubert, 509 U.S. at 596]. In evaluating the reliability of an expert's method, however, a district court may properly consider whether the expert's methodology has been contrived to reach a particular result. See General Elec. Co. v. Joiner, 522 U.S. 136, 146 (1997)] (affirming exclusion of testimony where the methodology was called into question because an "analytical gap" existed "between the data and the opinion proffered"); see also Elcock v. Kmart Corp., 233 F.3d 734, 748 (3d Cir. 2000) (questioning the methodology of an expert because his "novel synthesis" of two accepted methodologies allowed the expert to "offer a subjective judgment . . . in the guise of a reliable expert opinion").

Leaving Dr. Matson to one side, how far is it really legitimate for a district judge conducting a Daubert inquiry to "consider" whether "the expert's methodology has been contrived to reach a particular result"? We will not finally resolve this vexing problem in a weblog post. But we would like to discuss a hypothetical.

Suppose then that we have videotape of Dr. Al Sayit, notorious hired gun, who will opine for anybody if the money is right, meeting in a seedy bar with counsel for plaintiffs. Suppose that the tape captures him saying: "Don't worry. There are several different methods and techniques by which this problem can be assessed, and I'll contrive to find one that yields the result you want." Suppose further that defense counsel somehow secures this tape, plays it for Dr. Sayit at his deposition, and asks Dr. Sayit how he can sleep at night. Suppose, finally, that the expert answers as follows: "Yes, it's true. I did exactly what I said I would do. I intentionally chose the one available method that would yield the result favorable to plaintiff, and I chose the method for that very reason. That's how I get paid. But the method has been tested and peer-reviewed, has a low error rate, is governed by standards controlling its application, is generally accepted in the field, and was faithfully applied to the data, all of which is transparently shown in my report. Whatever my motivations, then -- and I'll freely confess they were mainly pecuniary -- my work is scientifically valid."

We incline to the following view. Dr. Sayit's mercenary attitude may be a reason to scrutinize the reliability of his work with special care. But if his methods do indeed satisfy Daubert, then his motivations should be irrelevant to the admissibility of his testimony. There is a role for credibility findings by district courts in determining whether the factual predicates for admissibility are satisfied. If an expert on the standard of care in a medical malpractice case suddenly starts to stutter and perspire when he says he has performed this type of surgery on many occasions, a judge may properly conclude that the witness's testimony on that point is dubious. But the court's legitimate reliance on credibility determinations ends there. That's the way things have to be, we think, under a regime that purports to gauge the admissibility of scientific evidence by reference to the degree of objective epistemic support that the evidence can claim.

Another way to put this point may be to say that the word credibility isn't very precise at capturing what's at stake when an expert chooses among valid alternative methods with an eye to their outcomes. As suggested above, the term motivation is more exact. And when that more exact term is used, the case for excluding "contrived" evidence grows much weaker. All testimony is "contrived" in the sense that various motivations may plausibly be ascribed to the witness. The remedy for "contrived" but scientifically valid testimony should not be exclusion. It should be more testimony (including testimony from the contriving witness on cross).

Expert Perjury Watch

The news from Tuscaloosa is that a private investigator hired to testify as an expert in an estate matter has been arrested for padding his vitae to an allegedly perjurious extent. According to prosecutors, the expert claimed an imaginary medical degree from Harvard and a fictional Ph.D. from MIT. Prosecutors say his claims to possess a Texas medical license aren't checking out either.

We'll say it again. Lawyers, vet your experts. Experts, don't exaggerate your credentials.

"Great Red Flags of Fraud"

The indispensable Legal Reader reports on what Judge Janis Graham Jack, the judge presiding over the multidistrict silicosis litigation in the Southern District of Texas, has called "great red flags of fraud" among the plaintiffs' medical experts. The allegations, explored during three days of Daubert hearings in December, are that the radiologists who have been screening claims for plaintiffs' counsel are running diagnosis mills. One expert testified that he never intended any of his 3700+ "B-reader" reports to indicate a diagnosis of silicosis. Other B-readers have reportedly diagnosed asbestosis or silicosis in the same patients, depending on the litigation occasion. The Legal Reader post links to a LexisOne article reporting on further details and the state of procedural play.

Justice Department Calls for Fingerprint Research

The research branch of the U.S. Department of Justice has issued a call for research on the reliability and appropriate standards for fingerprint identification, according to a report in yesterday's Chicago Tribune.

From the Tribune story:

Four years after scuttling a study into the reliability of fingerprinting, the research arm of the Justice Department is seeking answers to fundamental questions about the grandfather of forensic science.

The National Institute of Justice recently called for researchers to explore such crucial issues as how to measure the quality of fingerprints lifted from crime scenes and the accuracy of comparisons made by law-enforcement examiners.

The research solicitation seeks to "provide juries with increased information about the significance and weight of fingerprint evidence" and also to create tools "to improve the fingerprint examination process," said Catherine Sanders, spokeswoman for the Office of Justice Programs, which includes the institute.

The agency's decision is the latest example of an unmistakable shift in the previously defiant world of fingerprint experts. Until recently, they had pointed to nearly a century of convictions in U.S. courts to dismiss calls for a closer examination of their discipline.

The institute's solicitation is "very significant because it's recognizing that there is an area that needs to have research done, and they're willing to step up to the plate and fund it," said Ronald Singer, president of the American Academy of Forensic Sciences.

The article quotes an FBI forensics instructor who says that the FBI will probably issue standards within a year, including a minimum number of points needed for declaring a match.

Malpractice Claims and Insurance Rates

The Business Section of today's New York Times includes an article analyzing the relationship between malpractice insurance rates and damage awards -- not a very robust relationship, according to the article. The stronger more currently salient factor may be investment returns in the insurance business. The Times says insurers held rates fairly constant in the 1990's, when they were competing for funds to invest in the boom market, and were forced to jack up their rates when the stock market tumbled and their reserves dwindled.

The actual trend from 2003 to 2004, meanwhile, is that the number of malpractice suits and average damage awards both saw modest declines. Such is the putative crisis. Federal legislation to resolve it will be introduced within a month or so, the Times reports.

We all know to expect those damage caps -- the ones that the federal government urgently needs to enact, because they have been adopted through the normal democratic process in only 27 states so far, and voters in the remaining 23 states must be protected from themselves. Those damage caps will probably receive the most press coverage.

But we hope that this time around, the media (including the blogosphere) will also pay some attention to the other restrictive provisions (e.g., on expert testimony) with which the legislation is sure to be larded. The practical significance of the more recondite legislative provisions may be harder than damage caps for people outside the legal profession to understand. But if the press wanted to do its job, it would try to make them understandable, so that the polity could better judge whether the legislation as a whole is reasonably tailored to its ostensible purpose.

Update 2/23/05: Over at Overlawyered.com, Walter Olson has noticed the Times piece and promises a "closer analysis of the Times's reasoning" shortly. For "closer analysis," we read "rebuttal." That should be interesting -- not least because the Times piece is sourced largely to observers from within the insurance industry, so that their reasoning too is at issue.

Update II 2/23/05: A first installment of Olson's promised analysis of the Times story is now posted at Point of Law. It is a more modulated response than some might have anticipated. Maybe that's because Olson is a temperate soul. Or, to give him the benefit of the doubt, maybe it's because he has the flu. In any event, Olson says the Times reportage "wasn't a complete botch." The Times was "generally on target" in its discussion of insurance company investment returns, he says, and the article as a whole mostly practiced "'on the one hand, on the other hand' journalism."

But Olson isn't resting easy, because "[f]oes of malpractice-suit reform" (Olson's words) are "sure to seize" (Olson's words) on two statements from the Times article to the effect that "recent" (the NYT's word) increases in malpractice premiums do not appear to be associated with a corresponding increase in damage payouts. He's right, of course. We must rank ourselves among those who are suspicious of "malpractice reform." And in our original post, we reacted to the Times story just as Olson now prophesies that opponents of "reform" will generally do. The problem with that reaction, says Olson, is that the data in the chart forming what Olson calls the "centerpiece" of the Times story actually demonstrate the opposite of what the Times says. Over the long term, says Olson -- i.e., since 1975 -- both damage awards and malpractice premiums show a sharp general upward trend, in inflation-adjusted dollars, according to the Times chart itself.

For several reasons, this critique of the no-recent-correlation remarks in the Times story seems weak to us. First, the Times itself drew that conclusion only for what it called "recent" trends. It did not draw into question what would be the unsurprising existence of some longer-term relationship between premiums and payouts. This is an important qualification, because much of the political impetus behind the drive for malpractice "reform" depends on the notion that we are experiencing some current "crisis" in the form of runaway malpractice awards, which stand in some allegedly causal relationship to current premiums. Look at the Times graphic for some interval you consider current or "recent" -- say from 1995 or 2000 to date. Whatever those data imply, they do not suggest a current relationship of direct and contemporaneous causation.

We do not doubt that proponents of "reform" could develop hypothetical explanations, broadly consistent with both the data and their "reform" agenda, for the failure of the "recent" trends in damage awards and malpractice premiums to track each other more closely (or even to move in the same direction, for the most recent year reflected in the Times data). Indeed, the explanations tentatively floated by Olson seem plausible enough -- the existence of multiple causal factors affecting premium fluctuations, of which damage awards are only one important influence, and/or the likelihood of some temporal offset between premium adjustments and past (or anticipated future) awards. But absent some testing and empirical support for those hypotheses, they remain exactly that: hypotheses. And the hypotheses are sufficiently broad and ill-defined that it is difficult to imagine what data would tend to confirm or falsify them, let alone measure the relative importance of the multiple causal influences, or the likely effect of damage caps on the observed trends.

Where, moreover, do even these hypotheses leave us? They leave us with this considerably diluted rationale for "reform": Although the data do not show any obvious connection between "recent" malpractice awards and any current crisis in malpractice premiums, and although it is conceded that recent escalations in premiums are probably attributable in part to vicissitudes in the performance of insurance company investments, and although malpractice awards actually dropped in constant dollars by close to 10% for the most recent year studied, it is believed, on intuitively plausible conceptual grounds, that there is some longer term relationship between awards and premiums, and that current premiums are partly attributable in a larger sense to rapidly escalating malpractice awards over a period dating back to 1975.

Perhaps the problem is then portrayed as one that should have been fixed sometime in the Ford Administration, so that it's about time we fix it now, while we have the chance, with Republicans firmly in power. The problem with that explanation is that it tends to cast "reform" in a less sympathetic light -- i.e., as a proposal animated by general philosophical considerations that conservatives have been yearning to implement for decades, and which they now think they have the political opportunity to enact, the only particular urgency being that their window of opportunity may not last very long at the national level.

Or perhaps, as Hegel says somewhere, a quantitative change can eventually precipitate a qualitative change. Perhaps, that is, the story would go that the increase in damage awards has simply accreted over a long period, with matters only recently reaching a "crisis" stage where no play remains in the premium structures, and in which physicians are therefore fleeing their practices (a proposition, however, for which the evidence is both spotty and largely anecdotal).

Don't get us wrong. Wide-ranging federal legislation is frequently enacted on shakier empirical and policy foundations than these. And it does not necessarily refute a "reform" agenda that the arguments in its favor are uncertain, modulated, and complex. What this whole discussion does suggest, however, is that things are far more uncertain and complicated than the more alarmist political rhetoric offered in support of "malpractice reform" tends to acknowledge. And very arguably, the proponents of sweeping legal change should bear the burden of showing empirical evidence that the change (a) is necessary and (b) would have the desired effect. If they cannot do that, then at a minimum, they should follow Olson's lead in tempering their rhetorical claims and acknowledging the existence of some reality-based complexities. Karl Rove notwithstanding, mere lawyer-baiting is not a strong enough rationale for wholesale legal revolutions.

Second, although no one seriously doubts the proposition that malpractice premiums probably incorporate some actuarial component, very little is proved by displaying two curves that both head in a generally upward direction over a time period spanning three decades. Grant the proposition that some significant relationship probably exists, in the long term, between damage payouts and insurance premiums. The central debate might then be over whether the social burdens imposed by higher premiums more than offset the social benefits, if any, associated with the payouts. To answer that question, we would have to know something about the reasons for the increase in payouts. To what extent are they tied to changes in the health-care delivery system over time? (We remember the 1970's as the decade in which HMO's enjoyed their advent.) To what extent are they associated with demographic causes? To what extent, if any, are pain-and-suffering awards actually increasing in real terms over time? To what extent, if any, is the increase in payouts associated with higher real incomes among claimants? To what extent are the increased damage awards the product of a deteriorating quality of care, or higher standards of care? And by the way, to what extent, these days, do damage awards successfully deter negligent health care? All of these things are difficult to measure empirically, and mere uncertainty about them should not forestall reform, if the national welfare truly depends on it. But the world has changed greatly in the past thirty years. That two curves move in the same direction and have comparable slopes over a three-decade interval may be an interesting point of analytical departure, but says little or nothing about the factors that have shaped their direction -- and nothing, directly, about whether that direction is unequivocally the right one or the wrong one. It cannot really be said, then, that the Times data prove the opposite of what the Times article actually says. It can be said, at most, that the data reported by the Times would not necessarily support extrapolation of the article's no-current-correlation observations to a broader historical period.

Third, there actually are potential empirical investigations that might shed light on some of these issues. Rather than rush to nationwide judgment, maybe we should conduct those investigations. Olson noncomittally raises the possibility, for example, that the downward spike in last year's payouts might be attributable to newly passed "tort reform" legislation at the state level. This is a possibility that opponents of "tort reform" may sometimes be reluctant to discuss (because if true, it might suggest that tort reform "works"). But discussion of this issue should also make advocates of federal "reform" a little nervous. Do damage caps at the state level work to reduce premiums, or don't they? If they do, then perhaps any need to enact them at the federal level has been overtaken by events -- i.e., by legislative action in the 27 states that have adopted caps. If they don't, then there may be little reason to suppose that federal caps would work any better. What we do know is that malpractice premiums are rated geographically, so that empirical study should be possible. Are the rates lower in capped states than in uncapped states, or not? Perhaps someone has done this research. If so, we should be discussing the results. If not, or if insufficient time has elapsed to judge, then maybe we should should be good federalists, and await the outcome the grand experiment currently being conducted by the states.

In any event, following Olson's example, we should modulate our own recital of the issues. We originally said, when we first posted on the Times article, that the primary factor driving malpractice premiums may be the return on insurance company investments. At a minimum, we should have added the qualifier "recent." And it might have been wise of us to give more of a conceptual nod to the likely role of other influences -- actuarial influences. No doubt it is the plaintiffs' lawyer in us that originally balked at taking that more balanced view.

The larger problem with balanced views, of course, is that they generally fail to supply sufficiently gratifying support for ideological jihads. But we guess we could live with that.

Update III 2/24/05: Walter Olson has now delivered the second installment of his critique of the Times article, focusing on the research about the efficacy of damage caps in reducing malpractice premiums. As noted above, we aren't familiar with that research. Olson says it shows that caps do reduce premiums significantly, although not dollar-for-dollar. Assuming without deciding that Olson is correct about that research, two observations seem warranted.

First, what problem would then remain for federal caps to address? That some states have set the caps too high? That 23 states have yet to adopt caps at all? What arguments exist that federal law should displace state policy choices on this issue? Indeed, aren't "tort reform" proponents the ones who told the state legislatures that they should make the policy choices and set the caps in the first place? And isn't the political process in the states already working pretty well for the "tort reform" contingent? Should "tort reformers" not content themselves with that success, rather than opening a federal Pandora's Box? What will they think about the federalism question, after all, in twenty years, when woolly-headed lefties hold power in Washington and have the votes to enact national legislation subjecting malpractice-committing physicians to treble damages and public floggings?

Second, the research to which Olson refers (or other research) may incorporate (or provide a method for developing) estimates of the premium savings that caps would effect. That sounds like something policymakers (state or federal) might want to know about, before acting. Yet we haven't noticed any such estimates in the political debate as reported in the press. Why not?

Daubert in Georgia: The Other Shoe Drops

On 2/1/05, we posted on "tort reform" legislation approved by the Georgia Senate. From our learned Georgia correspondent, Charles Beans, we now learn that the legislation has been passed by Georgia's House of Representatives as well, and signed into law by Governor Perdue. The full text of the bill as finally enacted is available here.

Among other measures, the legislation now places Georgia squarely in the ranks of the Daubert states, for civil cases. In civil cases, the bill emphasizes Georgia's insistence that its courts "not be viewed as open to expert evidence that would not be admissible in other states."

Criminal cases, however, are another matter. The bill provides: "In criminal cases, the opinions of experts on any question of science, skill, trade, or like questions shall always be admissible; and such opinions may be given on the facts as proved by other witnesses." (The emphasis is ours.)

There has long been a de facto dichotomy in federal court between Daubert's application in criminal cases (fairly licentious) and civil ones (more strict). Perhaps Georgia is to be applauded for its candor in making the difference in standards explicit and official, though some might feel that the distinction should operate in the opposite direction.

We will await with interest the first case in which the collateral estoppel issue arises. That issue would be whether the verdict from a criminal case estops the defendant (or is even admissible) in a subsequent civil case, if the criminal verdict was dependent on expert proof.

NRC Publication Cites Daubert on the Web

We will never be cited by the National Research Council of the National Academies for doing anything scientifically useful. But our parent site is referenced in the prepublication version of a study by the NRC's Water Science and Technology Board, entitled "Valuing Ecosystem Services: Toward Better Environmental Decision-Making," at page 82, note 4 (2004).

We're thrilled. Can a Nobel be far behind?

10th Circuit Affirms Exclusion of Breast Implant Causation Testimony

The Tenth Circuit has published an opinion upholding the trial court's exclusion of plaintiff's causation experts in a breast implant case. The panel held that the experts' differential diagnosis was flawed because they lacked a basis to "rule in" the implants as the cause of plaintiff's connective tissue / autoimmune disease, in the face of a substantial epidemiological literature disclosing no generic causal link. See Norris v. Baxter Healthcare Corp., No. 03-1471 (10th Cir. Feb. 8, 2005) (Seymour, McKay, & Murphy, JJ.).

9th Circuit Reverses Exclusion of Defense Expert on Hedonic Damages

In overturning a $6.7 million jury verdict, the Ninth Circuit has reversed the trial court's decision excluding testimony from the defendant's damages expert in a wrongful death case. The opinion would appear to stand for three propositions: (1) a ruling that one expert's testimony is reliable does not necessarily imply that an opposing expert's contrary opinion is unreliable; (2) there may be some validity to the "goose and gander" rule; and (3) the sky would not fall if some shaky expert testimony happened to reach a jury's ears. See Dorn v. Burlington N. Santa Fe R.R., No. 03-35071 (9th Cir. Feb. 7, 2005) (Kleinfeld, Callahan, & Bertelsman, JJ.).

The district court in Dorn had overruled a Daubert challenge to the plaintiff's expert on hedonic damages. When it came time for defendant to offer its own expert's critique of the plaintiff's damages analysis, plaintiff's counsel objected on the ground that plaintiff's expert had already been found to be testifying reliably. The trial court sustained the objection.

On appeal, the Ninth Circuit did not reach the admissibility of the plaintiff's expert testimony, though the panel did note its concern that the plaintiff's expert's methodology was not in total harmony with his numerical conclusions (because his putative method was to measure market premiums for safe products, safe occupational conditions, etc., whereas the expert's actual damage estimate commingled those market-based figures with other, non-market numbers derived from government safety expenditures, etc.). The panel did conclude, however, that the defendant's expert testimony should not have been excluded merely because the plaintiff's expert's opinion had been found sufficiently reliable to survive Daubert scrutiny. It was for the trier of fact to resolve the experts' differences, the appellate court said.

The court also made this very intriguing observation (emphasis ours): "The Supreme Court in [Daubert] was not overly concerned about the prospect that some dubious scientific theories may pass the gatekeeper and reach the jury under the liberal standard of admissibility set forth in that opinion; indeed, the Court said, 'Vigorous cross-examination, presentation of contrary evidence, and careful instruction on the burden of proof are the traditional and appropriate means of attacking shaky but admissible evidence.'"

One panel's take? Or a signal to lighten up a little on the gatekeeping? Maybe time will tell.

Meanwhile, we wonder how much damage that defense expert really would have done, if the jury had heard his testimony. The plaintiff has now lost a sizeable verdict and will have to retry the case. The reversal was attributable in part to other errors, but the Daubert error was held not to be harmless. Perhaps the wisest course, sometimes, is to consider whether an unsound objection is worth making, if the net effect of having it sustained will be to unravel what might otherwise have been a victory untainted by error.

Temporal Trends (?) in Appellate Review of Daubert Decisions

In this post, we will: (1) present a chart tracking cumulative federal appellate decisions affirming or reversing rulings on expert evidence since 1/1/2000; (2) explain how the chart was compiled; (3) speculate on what it might mean (perhaps nothing); and (4) invite further (i.e., more serious and systematic) research.

(1) The chart



Here is a PDF version of the same chart.

(2) Explanation of the chart

When we launched our parent site, Daubert on the Web, in early 2002, we began compiling cumulative statistics on affirmance rates for a sample of cases intended to comprise all federal appellate dispositions since January 1, 2000, published or unpublished, addressing the admissibility of expert testimony. In particular, we have tracked the rate at which federal appellate decisions:

• affirm the trial courts' evidentiary rulings in general (the overall affirmance rate, represented in the chart by the yellow line in the middle, labeled "O" in the legend);
• affirm lower court decisions admitting expert testimony (the admissibility affirmance rate, represented by the dark blue line at the top of the chart, denoted "A" in the legend); and
• affirm the trial courts' rulings excluding expert evidence (the exclusion affirmance rate, represented by the pinkish line at the bottom, labeled "E" in the legend).

What the chart appears to show is that the likelihood of appellate affirmance is increasing over time for evidentiary decisions in all three categories.

The methods used to allocate appellate decisions to the listed categories are explained here. Briefly, affirmance rates can be analogized to batting averages. If a district court decision on admissibility is affirmed, we have a "hit." If it is reversed, we have an "out." If the game is called on account of rain during the district court's "at bat" -- e.g., if the appellate court does not reach the lower court's evidentiary decision -- then the decision isn't counted at all. It follows that if the district court's decision is never appealed, it is not in the sample. Note that outcomes are counted expert by expert, not case by case. This matters because one case may involve several experts. Note also that the sample tabulates affirmances or reversals of evidentiary rulings, not judgments. Thus if a trial court's decision admitting expert testimony is held to be erroneous, but harmless error, the outcome is counted as a reversal, not an affirmance.

These figures are maintained for each individual circuit, and also for all circuits taken as a whole. In retrospect, it would have been wise to tabulate subtotals over regular temporal intervals, so that there would be figures, e.g., for each year. Unfortunately, we lacked the foresight to do that. Thus the data points in the chart represent not the affirmance rate prevailing at any fixed point in time, but rather the cumulative affirmance rates as of the relevant date, for all decisions since January 1, 2000. That should mean that the magnitudes of any true temporal trends are understated in the graph, in the following sense. If, e.g., the cumulative overall affirmance rate has climbed from .843 (311/369) as of 4/9/03 to .872 (539/618) as of 2/5/05, or by about 3%, then the affirmance rate for the interval between those two dates will be higher than the terminal cumulative value of .872. In point of fact, for the period between those two dates, the overall affirmance rate will be .916 (228/249), or about 7% higher than in the earlier period.

Because the site is updated every few days to reflect new decisions, and because we are lazy and have a hard disk of only finite size, we do not store backups for every temporal instantiation of the site, and so we do not have daily figures for such cumulative totals. The data points in the graph were obtained by recourse to the Wayback Machine. The graph was charted on an Excel spreadsheet.

(3) Interpretation of the figures

(a) Caveats. Before trying to interpret any temporal trends, we should discuss some of the reasons to doubt their reality. There is a significant element of judgment involved in determinations that an appellate opinion reverses or affirms a district court's evidentiary decision. Some district court decisions exclude testimony only in part; some appellate rulings affirm or reverse only in part. Nor is it always clear whether to count an appellate decision as even pronouncing on lower court evidentiary rulings. One example is presented by the occasional decision involving a petitioner for habeas corpus who claims that his defense counsel's failure to raise a Daubert challenge in the underlying criminal proceedings constituted ineffective assistance. It might seem obvious that such challenges do not necessarily embroil the courts in de novo consideration of whatever Daubert issues might have been posed in the underlying trial, but some district court and appellate decisions have seemed to approach the issue in those terms, perhaps on the theory that if the possibility of a valid Daubert challenge could be ruled out, then no constitutional issues need be reached.

Having done the counting ourselves, we will represent that such uncertain cases constitute a very small fraction of the sample. Although we are not statisticians, we doubt they are of sufficient number that vagaries in their categorization would be responsible for trends of the dimensions suggested by the graph. Nevertheless, it should be noted that one individual has done all the categorizing for this sample, under rules that have not been tested for inter-rater reliability, and it is possible that any apparent trends in the data merely reflect the unconscious evolution of his own tacit methods for resolving ambiguities.

What cannot reasonably be contested is the representativeness of the sample, which includes 100% of all instances of federal appellate review during the relevant time period. (Actually, it is possible we have missed a decision or two, but we will represent that if we have, their number would be very small, and we can think of no reason why there would be any tendency for missed decisions to fall into one category more often than another). The sample, by now, is also of reasonably substantial dimensions (appellate review of evidentiary decisions on 618 experts). To repeat, we're not statisticians. But the sample seems large enough that major trends are not likely the product of random variation.

(b) Hypotheses. The decisions in the sample include both criminal and civil cases, and involve a wide variety of fields of expertise. We have not attempted to evaluate whether the trends toward affirmance are an artifact of the mix of cases, but for example, if criminal cases were found to represent an increasing fraction of the overall total, then that by itself would be one very convincing explanation for a substantial increase in admissibility affirmance rates (since forensic testimony is admitted, and its admissibility is upheld, at a very high rate in criminal cases). It would be a far less plausible explanation, however, for the similar upward trend in exclusion affirmance rates.

We also have not attempted to analyze (because we doubt that these raw data would permit us to analyze) how the trends in affirmance rates may relate to any trends in admissibility rates. One interesting feature of the graph, however, is that the gap appears to be narrowing, over time, between the rates of affirmance for decisions admitting expert evidence and decisions excluding it. Decisions reversing the exclusion of expert evidence have consistently been more numerous, in relative terms, than decisions reversing the admission of such evidence. That trend has been evident since we first started compiling the data, not only for federal appellate decisions as a whole, but also at the individual circuit level. Nevertheless, affirmance rates in these two categories have apparently begun to converge.

One possible explanation for the trends might be that appellate courts have grown more lax in their oversight of Daubert rulings over the past five years. It is inherently difficult to measure something like the prevalent level of stringency in appellate review, and so we may be left with practitioners' intuitions. For ourselves, we have read each decision in the sample, and that process has not left us with the impression that appellate review has been growing more lax.

One other possible explanation might be that as the corpus of appellate decisions under Daubert and Kumho Tire has grown, district courts have enjoyed more guidance on the application of the principles embodied in those decisions, and are therefore better able to judge the outer bounds of their judicial discretion. That hypothesis might suggest that Supreme Court opinions articulating abstract factors pertaining to admissibility are less useful, in guiding specific rulings, than the now substantial body of intermediate appellate decisions mapping recurrent fact patterns into outcomes.

(4) Further research

To urge "further research" is actually something of a misnomer. These data were not originally collected or structured with an eye toward testing any hypothesis. We have simply charted some data that happen, fortuitously, to be at our disposal, at a level of rigor that not only fails to rival that prevailing in the natural sciences, but which is also uninformed by any statistical analysis or techniques going beyond mere tabulation. We have decided to post the data because for all the caveats and uncertainties, we are left with the firm impression, when all is said and done, that they probably do reflect something real about the judicial world, though we are unable to say what.

Deeper analysis would involve a more systematic protocol for data collection, some inquiry into the "mix" of decisions in the corpus, a finer temporal mesh, and perhaps some comparative analysis between circuits, or between federal and state decisions. Such a project would be ambitious. We hope someone undertakes it.

Notes on the "Unborn Child Pain Awareness Act of 2005"

On 6/6/04, we posted some thoughts and questions about the appropriate role of expert testimony in litigation over the constitutionality of abortion prohibitions. We were moved to reflect on this subject by the district court's opinion in Planned Parenthood Ass'n of Am. v. Ashcroft, 320 F. Supp. 2d 957 (N.D. Cal. 2004) (Hamilton, J.) (striking down ban on "partial birth" abortion), which included an extensive discussion of expert testimony on such subjects as the experience of fetal pain at various gestational ages. The expert debate on fetal pain in Planned Parenthood was mostly about whether the capacity to experience pain emerges as early as 22 weeks in fetal development, or as late as 26 weeks. Judge Hamilton felt that the question was not dispositive of the legal issues before her (because it was undisputed that if fetuses experience pain during "partial birth" abortions, then they also experience equal or greater pain during alternative procedures). That may partially explain why she was ultimately agnostic on the point:

The only way that an outside observer can determine whether any entity feels pain is if the entity communicates distress to the observer. The parties agree that fetuses are unable to communicate, so it is impossible to determine conclusively if the stress responses seen in fetuses in fact translate into an actual pain response, and thus no studies on fetal pain suffered during abortions have been conducted. Both parties agreed that as a result, much of the debate on this issue is based on speculation and inference.

320 F. Supp. 2d at 997; see also id. at 1002 (issue is "unsettled in the scientific community").

Anti-abortion legislators are now trying another tactic. The "Unborn Child Pain Awareness Act of 2005" has been introduced by sponsors in both the United States Senate (S. 51) and the House of Representatives (H.R. 356). Under the bill, abortion providers would be required to deliver the following "information" to pregnant women seeking abortion of fetuses whose probable gestational age exceeds 20 weeks:

You are considering having an abortion of an unborn child who will have developed, at the time of the abortion, approximately XX weeks after fertilization. The Congress of the United States has determined that at this stage of development, an unborn child has the physical structures necessary to experience pain. There is substantial evidence that by this point, unborn children draw away from surgical instruments in a manner which in an infant or an adult would be interpreted as a response to pain. Congress finds that there is substantial evidence that the process of being killed in an abortion will cause the unborn child pain, even though you receive a pain-reducing drug or drugs. Under the Federal Unborn Child Pain Awareness Act of 2004 [sic], you have the option of choosing to have anesthesia or other pain-reducing drug or drugs administered directly to the pain-capable unborn child if you so desire. The purpose of administering such drug or drugs would be to reduce or eliminate the capacity of the unborn child to experience pain during the abortion procedure. In some cases, there may be some additional risk to you associated with administering such a drug.

This "information" would have to be delivered orally and also in a to-be-developed governmental brochure. If the woman requested anesthesia for the fetus, its provision would be a condition for performance of the abortion. Because physicians and patients suffer no real burden from governmentally mandated paperwork, the woman would also be required to complete an "Unborn Child Pain Awareness Decision Form," which the provider would be required to retain on file. Because civil penalties of $100,000 (first violation) or $250,000 (subsequent violations) would not afford sufficient incentives for providers to comply, and because one salutary method for deterring physician misbehavior is to promote private lawsuits seeking exorbitant damages, a private right of action would be created for women whose providers did not supply the "information" as prescribed, to include claims for "actual and punitive damages." No word in the legislation on whether the "actual" damages would be limited to pure economic loss or would also encompass the mother's "pain and suffering." If the latter, the legislation imposes no cap.

It is a hallowed strategy, when promulgating legislation of dubious constitutionality, for legislators to deem facts into existence. But if this legislation is passed, judicial scrutiny will not stop at such congressional deeming in the inevitable constitutional challenges (including not only the obvious Roe v. Wade challenges, but also the First Amendment challenges to be brought by physicians who object to being conduits for government-mandated speech). The courts will want to know whether there is actual scientific support for the legislation's premises. It therefore appears that the courts will again face the question that Judge Hamilton found it both unnecessary and impossible to resolve. It will be interesting to see whether the experts themselves are a little less agnostic this time around.

Tort Reform in the States

As the national debate on tort reform intensifies, there are emerging signs that not everyone wants to be rescued from the litigation "crisis" -- not even in the red states. In recent weeks, for example, the Republican-dominated Wyoming legislature has turned back proposed legislation to tighten the rules on expert testimony in malpractice cases, and a proposal to move toward a full-blown Daubert standard in Virginia has reportedly died in committee.

Georgia, however, is showing less trepidation. On Tuesday, a sharply divided Georgia Senate passed a comprehensive and controversial malpractice reform provision, and the Georgia House is expected to follow suit. (The Atlanta Journal-Constitution has the story.) Georgia's Senate Bill 3 is no halfway measure. The bill imposes damage caps, provides for the inadmissibility of apologies, tightens venue rules, and provides for fee-shifting where the plaintiff has rejected a settlement offer substantially more favorable than the jury's eventual award.

Needless to say, it also tightens the reins on expert testimony in various ways, and contains this unusual provision:

It is the intent of the legislature that the courts of the State of Georgia not be viewed as open to expert evidence that would not be admissible in other states. Therefore, in interpreting and applying this Code section, the courts of this state may draw from the opinions of the United States Supreme Court in Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993); General Electric Co. v. Joiner, 522 U.S. 136 (1997); Kumho Tire Co. Ltd. v. Carmichael, 526 U.S. 137 (1999); and other cases in federal courts applying the standards announced by the United States Supreme Court in these cases.

This statutory language is admittedly vague, but on one reading, it comes close, at least in spirit, to enacting a most-favored-nation rule for Georgia malpractice defendants: If some significant body of jurisdictions would reject the plaintiff's evidence on the standard of care or causation, then Georgia's courts should reject it too.

If the states now plan to launch a contest to see which jurisdiction can woo the greatest number of malpractice-prone physicians, we wonder if some litigation-friendly states might decide to fight back. Suppose, for example, that Judge Posner turned out to be correct in thinking that malpractice exposure does provide tangible incentives for health care providers to exercise due care. Suppose further, as seems not totally implausible, that over time, the states began to fall into polarized camps: "traditional" states that eschew damage caps and reject stringent restrictions on expert testimony, versus "reform" states that limit damages and scrutinize expert evidence very strictly. If the quality of care began to suffer markedly in the "reform" states, would some consumers of medical services not begin to consider opting for doctors in other jurisdictions, where the quality of care might be higher, and where full-blooded malpractice remedies would be available in the event of mishap?

Most consumers, confessedly, are not currently in a position to travel to doctors outside the state of their domicile. You're lucky, in America, if you have health insurance permitting you access to any doctor at all. But many persons weigh their decisions about medical treatment very, very carefully, and some persons do possess enough resources to have choices. Should the domiciliary of a "tort reform" state be permitted to travel to another jurisdiction, enjoy the benefits of its superior standards of care, and resort to its more permissive malpractice regime in case things go awry? And if that sort of thing starts, could the "traditional" states consider adopting choice-of-law measures under which the substantive, procedural, and evidentiary rights of malpractice claimants would be derived from the state of their domicile? Would it be constitutional, to do that? And if it were, would such interstate mischief begin to supply a legitimate rationale for a uniform federal standard?

Before some conservative think-tank picks up on this line of thought, we had better be clear. So far as we know, no one has proposed the hypothetical choice-of-law legislation we're imagining, and if somebody did, we suspect there would be serious problems under the Dormant Commerce Clause. No such problems, in fairness, would appear to beset Georgia's Senate Bill 3, which does not discriminate between Georgia residents and nonresidents.

But it is odd, all the same, to see a state so unabashedly determined to lead the Nation in the inadmissibility of expert evidence in malpractice cases. And it does raise the question of how other states should respond, if things were to reach the point where consumers start hopping on airplanes to take vacations from their own states' malpractice regimes.

Update 2/9/05: Law.com has posted an article by Greg Bluestein of the Fulton County Daily Report, entitled "Mother's Death Is Case Study for Tort Reform," that evaluates how Senate Bill 3 might have affected a 2002 malpractice case in which a jury awarded $2.5 million to the family of a 29-year-old homemaker after she died following what should have been routine surgery.