FORENSIC SCIENCE: DAUBERT'S FAILURE. - Free Online Library (2024)

"The man who discovers a new scientific truth has previouslyhad to smash to atoms almost everything he had learnt, and arrives atthe new truth with hands bloodstained from the slaughter of a thousandplatitudes." (1)

CONTENTSINTRODUCTIONA. Daubert and Rule 702B. National Academy of Sciences Forensic Report (2009)I. DISCREDITED TECHNIQUES A. Bite Mark Comparisons 1. Foundational Research 2. Texas Forensic Science Commission (2016) 3. White House PCAST Report (2016) B. Microscopic Hair Analysis 1. FBI Hair Review 2. White House PCAST Report (2016) C. Arson Investigations 1. Willingham Case 2. Han Tak Lee Case 3. National Fire Protection Association Guidelines 4. Dog-Sniff Evidence 5. Post-Daubert Cases D. Comparative Bullet Lead Analysis 1. NAS Bullet Lead Report (2004) 2. Post-Report DevelopmentsII. Misleadingly Presented Techniques A. Firearms & Toolmark Identifications 1. Post-Daubert Cases 2. NAS Ballistic Imaging Report (2008) 3. NAS Forensic Science Report (2009) 4. White House PCAST Report (2016) B. Fingerprint Examinations 1. Post-Daubert Cases 2. Madrid Train Bombing 3. NAS Forensic Science Report (2009) 4. White House PCAST Report (2016) 5. AAAS Fingerprint Report (2017)III. FORENSIC SCIENCE RESEARCH A. National Commission on Forensic Science (2013-17) B. White House PCAST Report (2016)IV. INDEPENDENT SCIENTIFIC REVIEWCONCLUSION

INTRODUCTION

In 2015, Judge Alex Kozinski of the Ninth Circuit Court of Appealsnoted that "[m]any defendants have been convicted and spentcountless years in prison based on evidence by arson experts who werelater shown to be little better than witch doctors." (2) In thesame year, Dr. Jo Handelsman, a White House science advisor, observed:"Suggesting that bite marks [should] still be a seriously usedtechnology is not based on science, on measurement, on something thathas standards, but more of a gut-level reaction." (3) According toJudge Catharine Easterly of the D.C. Court of Appeals, "[a]smatters currently stand, a certainty statement regarding toolmarkpattern matching has the same probative value as the vision of apsychic." (4) A New York Times editorial echoed these sentiments:

 [C]ourts have only made the problem worse by purporting to be scientifically literate, and allowing in all kinds of evidence that would not make it within shouting distance of a peer-reviewed journal. Of the 329 exonerations based on DNA testing since 1989, more than one-quarter involved convictions based on "pattern" evidence--like hair samples, ballistics, tire tracks, and bite marks--testified to by so-called experts. (5)

These criticisms are valid--which raises a puzzling andconsequential question: Why didn't the Supreme Court's"junk science" decision, Daubert v. Merrell DowPharmaceuticals, Inc., (6) prevent or restrict the admissibility oftestimony based on flawed forensic techniques? After all, Daubert wasdecided in 1993, twenty-five years ago.

A. Daubert and Rule 702

Daubert was considered a revolutionary decision. (7) It"radically changed the standard for admissibility of scientifictestimony" (8) by sweeping away the Frye v. United States (9)"general acceptance" test, (10) which had been the majorityrule in both federal and state cases. (11) The Frye standard gave greatdeference to the views of forensic practitioners and not to empiricaltesting. (12) Daubert promised to be different. The Supreme Court heldthat "[p]roposed testimony must be supported by appropriatevalidation--i.e., 'good grounds,' based on what is known. Inshort, the requirement that an expert's testimony pertain to'scientific knowledge' establishes a standard of evidentiaryreliability." (13) In making this reliability determination, theDaubert Court highlighted five factors: (1) empirical testing, (2) peerreview and publication, (3) error rate, (4) maintenance of standards,and (5) general acceptance. (14) The first and most important factor isempirical testing. The other factors are supplementary. (15) Peer reviewand publication are designed to expose defects in testing. Acceptance ofa technique within the scientific community is achieved through thepublication of valid test results. Similarly, both error rates andstandards are derived from testing.

Daubert was followed in 1999 by Kumho Tire Co. v. Carmichael, (16)which held that Daubert's reliability standard applied to allexpert testimony, not only scientific evidence. (17) By 2000, theSupreme Court was describing Daubert as establishing an"exacting" standard. (18) In the same year, Federal Rule ofEvidence 702 was amended to incorporate the Daubert-Kumho standard. (19)Although a handful of jurisdictions continue to apply the Fry e test,about forty jurisdictions have adopted the Daubert standard in one formor another. (20)

During this time, there was no shortage of commentary on the lackof empirical research in forensic science. (21) For example, shortlyafter Daubert was decided, Professor Margaret Berger wrote:"Considerable forensic evidence made its way into the courtroomwithout empirical validation of the underlying theory and/or itsparticular application." (22) After Kumho, two commentators--citingbite mark, hair, and firearm analysis--observed that "littlerigorous, systematic research has been done to validate thediscipline's basic premises and techniques, and in each area therewas no evident reason why such research would be infeasible." (23)

Notwithstanding Daubert's promise, scholars soon discerned itsuneven application in civil and criminal cases: "[T]he heightenedstandards of dependability imposed on expertise proffered in civil caseshas continued to expand, but ... expertise proffered by the prosecutionin criminal cases has been largely insulated from any change inpre-Daubert standards or approach." (24) The title of a 2005article summed up the state of the law--"The (Near) Irrelevance ofDaubert to Criminal Justice. (25) In short, in the criminal context,courts applied Daubert-lite.

B. National Academy of Sciences Forensic Report (2009)

In 2006 Congress entered the picture by authorizing the NationalAcademy of Sciences ("NAS") to conduct a study of forensicscience. After a three-year investigation, NAS issued a landmark report.One of its most riveting passages concluded: "Among existingforensic methods, only nuclear DNA analysis has been rigorously shown tohave the capacity to consistently, and with a high degree of certainty,demonstrate a connection between an evidentiary sample and a specificindividual or source." (26) The report went on to state that"some forensic science disciplines are supported by little rigoroussystematic research to validate the discipline's basic premises andtechniques." (27) Such common forensic techniques as fingerprintexaminations, (28) firearm ("ballistics") and toolmarkidentifications, (29) handwriting examinations, (30) microscopic hairanalysis, (31) and bite mark comparisons (32) fell into this category.

Not only did the NAS report highlight flaws in forensic science, itsharply criticized the judiciary for failing to demand the validationthat Daubert required: "The bottom line is simple: In a number offorensic science disciplines, forensic science professionals have yet toestablish either the validity of their approach or the accuracy of theirconclusions, and the courts have been utterly ineffective in addressingthis problem." (33) In a later passage, the report declared that"Daubert has done little to improve the use of forensic scienceevidence in criminal cases." (34) The disparate treatment of civilactions and criminal prosecutions was also noted. After finding that"trial judges rarely exclude or restrict expert testimony offeredby prosecutors," the report commented: "ironically, theappellate courts appear to be more willing to second-guess trial courtjudgments on the admissibility of purported scientific evidence in civilcases than in criminal cases." (35) Despite the NAS report, courtscontinued to admit the same evidence. Only a handful of courts appliedthe "exacting" standard that the Supreme Court said Daubertdemanded. (36)

This Article examines the justice system's failure byreviewing the status of two categories of forensic techniques. The firstcategory involves discredited techniques: (1) bite mark analysis, (2)microscopic hair comparisons, (3) arson evidence, and (4) comparativebullet lead analysis. The second category involves techniques that havebeen misleadingly presented, including firearm and toolmarkidentifications and fingerprint examinations. Both categories presentDaubert issues. The Article argues that the system's failure can betraced back to its inability to demand and properly evaluatefoundational research--i.e., Daubert's first factor, empiricaltesting. Indeed, the justice system may be institutionally incapable ofapplying Daubert in criminal cases.

A different paradigm is needed, one that assigns an independentagency the responsibility of evaluating foundational research. Asdiscussed in Part IV, this approach was recently recommended by theNational Commission on Forensic Science (37) and the President'sCouncil of Advisors on Science and Technology ("PCAST"). (38)Both recommended that the National Institute of Standards and Technology("NIST") evaluate all forensic disciplines on a continuingbasis, thereby injecting much needed scientific expertise into thecriminal justice system. The recent reports on latent fingerprints (39)and arson investigations, (40) which were published by the AmericanAssociation for the Advancement of Science ("AAAS"), buttressthe need for independent scientific evaluations.

I. DISCREDITED TECHNIQUES

A. Bite Mark Comparisons

For decades, bite-mark evidence has been admitted in hundreds oftrials, (41) many of which were capital prosecutions. (42) No reportedAmerican case has rejected bite mark testimony. Moreover, it is notuncommon for courts to speak of bite mark comparisons as a"science" (43) --even an "exact science." (44)Acceptance of the technique is so deeply entrenched that some courtshave taken judicial notice of its validity, (45) which means itsreliability is indisputable. (46)

Distinctive characteristics of a person's dentition were firstused to identify skeletonized remains and individuals in mass disasterssuch a plane crashes. (47) Courts assumed that these distinctivecharacteristics can be transferred to another person's skin duringa violent crime--e.g., homicides, rapes, and child abuse (48)--anassumption that overlooked some obvious problems. First, bite markstypically involve no more than the edges of six to eight front teeth,not thirty-two teeth with five anatomical surfaces that can be used whencomparing a deceased person's dentition with X-rays. Second, bitemarks do not reveal artifacts such as fillings, crowns, etc., all ofwhich assist in associating human remains with a person's dentalrecords. (49) Moreover, human skin is extremely malleable and thussubject to various types of distortion. (50) In addition, bite markanalysis is a subjective technique with no agreed-upon methodology.

1. Foundational Research

Despite overwhelming judicial approval, bite mark evidence is notsupported by foundational research. (51) Indeed, the only rigorousstudies are recent--and undercut the technique's validity. (52) The2009 NAS forensic report concluded that "the scientific basis isinsufficient to conclude that bite mark comparisons can result in aconclusive match." (53) Despite the NAS Report, courts continued topermit expert testimony on the subject. For example, in State v. Prade,(54) decided in 2014, the expert testified that "bite mark evidenceis generally accepted within the scientific community ...," (55)Similarly, in Coronado v. State, (56) a different expert stated that hedid not "agree with the NAS Report's conclusion that bite markanalysis cannot result in a conclusive match," adding, "you donot have to be a 'rocket scientist' to see that, in somecases, there is a unique and distinct pattern of teeth that can beidentified." (57) In addition, these experts rejected the validresearch mentioned above, (58) and both prosecutors and their expertsattacked researchers without offering any foundational research. (59)

Unfortunately, the American Board of Forensic Odontology("ABFO") has fiercely defended bite mark analysis. To bolsterits position, the ABFO conducted a study that was presented at aforensic conference in 2015. (60) As it turned out, the study undercutthe ABFO's own position. Thirty-nine ABFO-certified bite markexperts--with an average of twenty years' experience--examined onehundred bite mark photographs. (61) Each was asked three questions:

1. Is there sufficient evidence in the presented materials torender an opinion on whether the patterned injury is a human bite mark?

2. Is it a human bite mark, not a human bite mark, or suggestive ofa human bite mark?

3. Does the bite mark have distinct, identifiable arches andindividual tooth marks? (62)

The results to the first question were not reassuring. Thethirty-nine experts agreed unanimously in only four out of the onehundred cases. (63) In only twenty cases was there 90 percent or moreagreement. (64) At the end of question two--whether the mark is a humanbite mark-there were only sixteen cases with 90 percent or moreagreement. (65) At the end of the third question, there were only eightcases in which at least 90 percent of the analysts agreed. (66) Equallydisturbing was the ABFO's decision to postpone publishing theresults "until the organization can tweak the design of the studyand conduct it again, a process that's expected to take at least ayear." (67) In effect, ABFO wanted a do-over. Meanwhile, anAssociated Press analysis reported that at least twenty-four menconvicted or charged with murder or rape based on bite marks have beenexonerated since 2000. (68)

2. Texas Forensic Science Commission (2016)

Steven Chaney spent twenty-eight years in prison for murder basedlargely on bite mark evidence. When his conviction was overturned, (69)the Innocence Project filed a complaint on his behalf with the TexasForensic Science Commission ("TFSC"). (70) In 2016, after asix-month investigation, the TFSC recommended a moratorium on theadmission of bite mark testimony. It found that there is no scientificbasis for claiming that a particular mark can be associated to aperson's dentition: "Any testimony describing human dentitionas 'like a fingerprint' or incorporating similar analogieslacks scientific support." (71) Similarly, "there is noscientific basis for assigning probability or statistical weight to anassociation, regardless of whether such probability or weight isexpressed numerically (e.g., 1 in a million) or using some form ofverbal scale (e.g., highly likely/unlikely)." (72)

TFSC was also alarmed that the ABFO study was not published due to"political and organizational pressures." (73) In theCommission's view, "such a resistance to publish scientificdata contradicts the ethical and professional obligations of theprofession as a whole, and is especially disconcerting when oneconsiders the life and liberty interests at stake in criminalcases." (74)

3. White House PCAST Report (2016)

In September 2016, the White House released its report on forensicscience. (75) Regarding bite mark analysis, it concluded that (1)appropriately designed validation studies are lacking, (2) the fewavailable studies had "very high" false-positive rates, (3)"inappropriate closed-set designs ... are likely to underestimatethe true false positive rate," and (4) the studies show thatexperts "cannot even consistently agree on whether an injury is ahuman bitemark." (76) Numerous cases support the last observation.(77)

In sum, the courts have yet to reject bite mark evidence--asubjective method that is not supported by foundational research andlacks agreed-upon standards. (78) "Perhaps no discredited forensicassay has benefitted more from criminal courts' abdication ofgatekeeper responsibilities than bite mark analysis." (79) Instead,it was the Innocence Project that spearheaded the challenges in thisarea, and in 2016 the Texas Forensic Science Commission became the firstgovernmental body to seriously scrutinize the technique. Notwithstandingthe NAS, PCAST, and TFSC reports, courts continue to admit bite markevidence. (80)

At the April 10, 2017 meeting of the National Commission onForensic Science, Keith Harward described how bite mark evidenceresulted in his thirty-three years of imprisonment before he wasexonerated by DNA evidence. (81) Incredibly, the next day the chairmanof the National District Attorneys Association stated that hisorganization believes that bite mark evidence is a "reliablescience." (82)

B. Microscopic Hair Analysis

In this examination, samples are first analyzed to identifyfeatures visible to the naked eye such as color and form, i.e., whetherit is straight, wavy, or curved. Next, the sample is viewedmicroscopically to determine characteristics such as shaft form, hairdiameter, and pigment size. (83)

Experts have long acknowledged that a positive identification isnot possible with microscopic hair analysis. Instead, examiners testifythat a crime scene exemplar was "consistent with" a hairsample from the de-fendant. The probative value of this conclusionwould, of course, vary if only a hundred people had microscopicallyindistinguishable hair as opposed to several million. Due to a lack ofresearch, no one knows whether the crime scene hair could have come from10 other persons or 100, 10,000, and so forth. (84) This importantqualifying information was often omitted from the experts'testimony, thus making marginal evidence appear misleadingly convincing.(85)

Experts frequently went way beyond the "consistent with"language in their testimony, however, often suggesting a rareassociation. For example, in the Edward Honaker case, the experttestified that the crime scene hair sample "was unlikely to matchanyone" other than the defendant. (86) Honaker spent ten years inprison before DNA proved him innocent. (87) In another case, an experttestified that matching hair samples were "consistentmicroscopically" but then elaborated: "In other words, hairsare not an absolute identification, but they either came from thisindividual or there is--could be another individual somewhere in theworld that would have the same characteristics to their hair." (88)This is an implicit--and extreme--probability statement that lacks anyempirical support.

Although microscopic hair analysis had long been judiciallyaccepted, (89) its validity was suspect. (90) In 1995, a federaldistrict court in Williamson v. Reynolds (91) observed: "Althoughthe hair expert may have followed procedures accepted in the communityof hair experts, the human hair comparison results in this case were,nonetheless, scientifically unreliable." (92) The court also notedthat the "expert did not explain which of the'approximately' 25 characteristics were consistent, anystandards for determining whether the samples were consistent, how manypersons could be expected to share this same combination ofcharacteristics, or how he arrived at his conclusions." (93)Williamson, who was five days from execution when the district courtissued a stay, was subsequently exonerated by DNA testing. (94)

The Williamson opinion--perhaps the only thorough judicial analysisof microscopic hair comparisons--was all but ignored by other courts. In1999 in Johnson v. Commonwealth, (95) the Kentucky Supreme Court upheldthe admissibility of hair evidence, taking "judicial notice"of its reliability (96) and thus implicitly finding its validityindisputable. (97) Other courts echoed Johnson, not Williamson, (98)Indeed, ten years after Williamson was decided, a 2005 decision by theConnecticut Supreme Court observed--correctly--that "[t]heoverwhelming majority of courts have deemed such evidenceadmissible." (99)

Once again, the courts abdicated their responsibility. Indeed, hairevidence only began to be carefully scrutinized after a startling numberof DNA exonerations were reported. (100) A 2008 study of 200 DNAexonerations found that forensic evidence was the second leading type ofevidence, at 57 percent--after eyewitness identifications at 79percent--used in wrongful conviction cases. (101) A subsequentinvestigation of trial transcripts underscored the role of hair analysisin the exoneration cases: "Of the 65 cases involving microscopichair comparison in which transcripts were located, 25 cases, or 38%, hadinvalid forensic science testimony." (102) The 2009 NAS Reportobserved that "testimony linking microscopic hair analysis withparticular defendants is highly unreliable." (103)

1. FBI Hair Review

In April 2013, the Mississippi Supreme Court, in a five-to-fourdecision, rejected Willie Jerome Manning's request for a stay ofexecution to permit DNA testing--"potentially setting up whatexperts said would be a rare case in recent years in which a person isput to death with such requests unmet." (104) A week later, thecourt unexpectedly stayed Manning's execution after the Departmentof Justice ("DOJ") notified state officials that FBI expertshad presented misleading testimony at his trial, including hair andfirearms evidence. (105)

Soon after, the DOJ announced that Manning was but one of 120cases--including twenty-seven death penalty prosecutions--in whichimproper microscopic hair analysis had been introduced in evidence.(106) For example, examiners claimed to connect a hair sample to asingle person "to the exclusion of all others" or stated orsuggested a probability for such a match from past casework. (107) TheFBI review came after three District of Columbia men, who had beenconvicted of rape or murder in the early 1980s, were exonerated throughDNA testing. (108) In one of these cases, the prosecutor claimed that,based on an FBI expert's testimony, the chances that the samplecame from someone else were "one in 10 million." (109)

After further investigation, DOJ reported in 2015 that "FBIexaminers had provided scientifically invalid testimony in more than 95percent of cases where that testimony was used to inculpate a defendantat trial." (110) Commonwealth v. Perrot (111) was one of the firstcases to reach the courtroom as a consequence of the DOJ review. Asuperior court granted Perrot a new trial in 2016, criticizing themisleading use of hair evidence. The court noted: "In discussingthe 'microscopic characteristics' of hair, [the expert] statedthat these characteristics 'make that hair somewhat unique.'He likened the 'subtle' characteristics of hair that'make it somewhat unique' to the subtle differences in a humanface." (112)

2. White House PCAST Report (2016)

In June 2016, the DOJ released proposed guidelines concerning hairtestimony. Documentation purporting to support the validity andreliability of hair evidence accompanied the guidelines. (113) Listingseveral studies, the FBI concluded:

 Based on these and other published studies, microscopic hair comparison has been demonstrated to be a valid and reliable scientific methodology. These studies have also shown that microscopic hair comparisons alone cannot lead to personal identification and it is crucial that this limitation be conveyed both in the written report and in testimony. (114)

The White House PCAST Report, however, challenged the supportingdocumentation, which discussed only a handful of studies from the 1970sand 1980s but did not comment on subsequent studies that found"substantial flaws in the methodology and results of the keypapers." (115) Moreover, "PCAST's own review of the citedpapers [found] that these studies do not establish the foundationalvalidity and reliability of hair analysis." (116)

The bottom line, again, is the judiciary's dereliction infailing to curb the misuse of hair microscopy testimony. The InnocenceProject's track record of DNA exonerations brought this issue tothe fore. Indeed, the three exonerations in the District of Columbiatriggered the FBI review. Yet, DOJ's proposed guidelines were basedon "foundational research" that PCAST questioned.

C. Arson Investigations

For decades, arson investigators came from the "oldschool" of investigators--those who used intuition and a number ofrules of thumb to determine whether a fire was incendiary. Criticscomplained that instead of being rooted in science, the approach wasbased on folklore that had been passed down from generation togeneration--without any empirical testing. (117) A government reportnoted, as early as 1977, that common arson indicators had "receivedlittle or no scientific testing" and that "[t]here appears tobe no published material in the scientific literature to substantiatetheir validity." (118) Proponents of a science-based approach toarson investigations waged an uphill battle through the 1980s, finallywinning a major victory when the National Fire Protection Association("NFPA") published its Guide for Fire and ExplosionInvestigations ("NFPA 921") in 1992. (119)

1. Willingham Case

Although NFPA 921 would later become the bible for arsoninvestigations, (120) Cameron Todd Willingham was convicted for thearson-murders of his young children weeks before the guide waspublished. Willingham, who was executed twelve years later, is theposter boy for junk science in arson investigations. (121)

At trial, Deputy Fire Marshall Vasquez testified that "[t]hefire tells a story. I am just the interpreter.... And the fire does notlie. It tells me the truth." (122) He also testified that he hadfound numerous so-called "indicators" for arson during hispost-fire investigation of Willingham's house. (123) One suchindicator was a low burning fire. (124) He told the jury that"[a]ll fire goes up," (125) and thus, burn patterns on thefloor and lower walls suggested that an accelerant was used. (126) Thisreasonable notion, however, has its limitations--especially when a fireoccurs in a contained area, such as a house with its windows shut:

 Due to buoyancy, a thermal plume initially rises once a fire is ignited. As the fire continues, the plume reaches the ceiling, which causes it to spread outward towards the walls. When it reaches the walls, the combustion products press down from the ceiling creating an upper level, which continues to increase in depth and temperature. Eventually thermal radiation replaces convection as the principal method of heat transfer. (127)

At this point, every combustible surface in the room willspontaneously burst into flames. This transition phenomenon, known asthe onset of "flashover," can occur within minutes. (128)After flashover, the entire room is burning, including the lower wallsand floor. Flashover has been described as the point at which the firetransitions from a "fire in a room" to a "room onfire." (129) At trial, prosecution witnesses acknowledged thatthere was an explosion, which could be explained by flashover. (130)Consequently, a low burning fire is not necessarily indicative of theuse of an accelerant. (131)

Moreover, many of Vasquez's other"indicators"--including what he called "pourpatterns" and "puddle configurations," (132) which appearas splotchy areas on the floor--can appear after flashover in anaccidental fire. (133) Additional indicators, such asalligatoring--large shiny charred blisters on burned wood--can also beexplained by flashover. (134) The flashover phenomenon also accounts foranother fact that Vasquez thought incriminatory. Willingham toldinvestigators that he had attempted to save his daughters, but he wasforced to run from his home without shoes because the heat was toogreat. According to Vasquez, the burn debris on the floor made itimpossible that Willingham would not have had burns on his feet. (135)However, Willingham's feet would not have been burnt if he left hishome before flashover. (136)

Charring under an aluminum threshold of an interior door providedanother clue according to Vasquez. (137) But again, this"indicator" may occur in a flashover. Other perceived"indicators"--melted bed springs, (138) multiple points oforigins, (139) and brown stains on a concrete floor (140)--were alsoconsistent with an accidental fire. (141) Finally, Vasquez relied on thepresence of "crazed glass"--spider-web patterns on thewindows--as an indication of arson. (142) Arson investigators longbelieved that crazed glass resulted from a fire that burned fast and hotand that the presence of crazed glass indicated that the fire was fueledby a liquid accelerant. Yet, subsequent research demonstrated thatcrazing occurs from rapid cooling--rather than rapid heating--when waterfrom fire hoses is sprayed on heated windows. (143)

In retrospect, one of the numerous debris samples submitted forlaboratory analysis contributed the most damning piece of evidence.(144) The debris sample--collected from an area near the front door--wasthe only sample that tested positive for a chemical commonly used incharcoal lighter fluids. [145] This finding can be explained by the factthat a charcoal grill and lighter fluid were on the front porch at thetime of the fire. (146) In fact, that the other samples yielded negativeresults supported Willingham's case. (147)

Numerous nationally recognized experts reviewed the arson testimonypresented at Willingham's trial and found it seriously flawed. Thefirst examination of the record by an independent expert was part ofWillingham's petition for habeas corpus and was also submitted tothe governor and the Board of Pardons and Parole days beforeWillingham's execution. It concluded: "On first reading, acontemporary fire origin and cause analyst might well wonder how anyonecould make so many critical errors in interpreting the evidence."(148) Nevertheless, a stay was denied, and Willingham was put to death.Subsequent evaluators agreed that the trial evidence was junk science.For example, five independent experts prepared a forty-three-pagereport, finding that "each and every one of the indicators reliedupon have since been scientifically proven to be invalid." (149)

In May 2006, the Innocence Project petitioned the Texas ForensicScience Commission to review the arson testimony in Willingham'sand Ernest Ray Willis' cases. (150) The TFSC is not authorized todetermine guilt or innocence. Instead, the Innocence Project argued thatthe State Fire Marshall Office should have reinvestigated arson cases inwhich its experts testified after NFPA 921 was published in 1992--a fulltwelve years before Willingham's execution. (151) TFSC retained itsown independent consultant, Dr. Craig Beyler, anothernationally-recognized expert, to review the arson evidence. Hisfifty-one-page report dissected the expert testimony, concluding:

 The investigations of the Willis and Willingham fires did not comport with either the modern standard of care expressed by NFPA 921, or the standard of care expressed by fire investigation texts and papers in the period 1980-1992. The investigators had poor understandings of fire science and failed to acknowledge or apply the contemporaneous understanding of the limitations of fire indicators. Their methodologies did not comport with the scientific method or the process of elimination. A finding of arson could not be sustained based upon the standard of care expressed by NFPA 921, or the standard of care expressed by fire investigation texts and papers in the period 1980-1992. (152)

Once Beyler's report became public, a political firestormerupted, and the governor, who was in the midst of a reelection battle,abruptly replaced commission members two days before a meeting wasscheduled to consider the Beyler report. (153) The newly appointedchair, a prosecutor, promptly cancelled the meeting, (154) raising thespecter of a cover-up. (155) Next, the Attorney General issued anopinion finding that the TFSC was prohibited "from considering orevaluating specific items of evidence that were tested or offered intoevidence prior to [its creation in 2005]." (156)

The TFSC eventually produced a report--one that did not directlydeal with the Willingham and Willis cases. Nevertheless, theReport's recommendations and statements indicated that theWillingham arson investigation was seriously flawed. Its firstrecommendation was "that fire investigators adhere to the standardsof NFPA 921." (157) In addition, the report reviewed a number ofarson indicators that were used in the Willingham and Willis cases.Citing Vasquez's testimony, the report undermined his opinionsconcerning (1) V-patterns as an indicator of origin, (2) pour patterns,(3) low/deep burning, (4) multiple separate points of origin, (5)spalling, (6) burn intensity, and (7) crazed glass. (158) It alsoobserved that "testimony, such as Vasquez's response to aquestion regarding Willingham's state of mind, is an example of thetype of testimony that experts should avoid as falling outside of theirfield of expertise." (159) The report even encouraged lawyers to"aggressively pursue admissibility hearings in arson cases."(160)

Despite the opinions of all the independent experts, the State FireMarshal vigorously defended its investigation. In a breathtaking letter,the office asserted that "[i]n reviewing documents and standards inplace then and now, we stand by the original investigator's reportand conclusions." (161) This left the TFSC incredulous. (162)

2. Han Tak Lee Case

Unfortunately, Willingham's case was not an outlier. In the1989 trial of Han Tak Lee, (163) the expert also relied on the old"myths" to declare the fire incendiary: (1) greater intensityand heat, (2) burn patterns, (3) alligatoring, (4) melted metal in bedframes, and (5) crazed glass. (164) In addition, the investigation was"hobbled by an incomplete and inaccurate understanding" offlashover. (165) After serving twenty-five years, Lee was released fromprison in 2014. (166)

3. National Fire Protection Association Guidelines

After the publication of NFPA 921 in 1992, the kind of testimonypresented in the Willingham and Lee cases should have vanished from thecourtroom. But arson investigators balked. According to one expert,"[t]he initial response to NFPA 921 in the fire investigationcommunity was overwhelmingly negative." (167) Babick v. Berghuism(168) is illustrative. In that case, Andrew Babick was convicted ofarson-murder for a 1995 house fire and was sentenced to two terms oflife imprisonment without the possibility of parole. (169) He latersought habeas relief, claiming ineffective assistance of counsel andprosecutorial misconduct. In 2010, the Sixth Circuit rejected theseclaims. (170)

In dissent, however, Judge Merritt chastised the defense attorneyfor not contesting the arson evidence in "this strange junk sciencecase." (171) One prosecution expert testified that: (1) char markson the porch were evidence of an accelerant, (2) a "line ofdemarcation" in a burn pattern on a carpet was"suspicious" because "it should not have burned thecarpeting on these jagged edges," and (3) the burns were "notnormal" and were "unnatural." (172) Another prosecutionexpert stated that "low burning" and other"unnatural" patterns indicated the presence of an accelerant.(173) Both experts "testified--in direct contrast to the NFPAguide--that they were so confident in their reading of burn patternsthat the absence of any laboratory confirmation of accelerant had noeffect on their testimony." (174)

4. Dog-Sniff Evidence

More alarming, in Judge Merritt's view, was dog-sniffevidence. (175) The NFPA guide provides: "Research has shown thatcanines have been alerted to pyrolysis products that are not produced byan ignitable liquid" and a positive canine alert without laboratoryconfirmation "should not be considered validated." (176) Thelab tests had not detected accelerants in the house debris. Yet, a doghandler testified that "his dog, Samantha, was '1000times' more effective at detecting fire starters or liquidaccelerants than a laboratory test on burnt material." (177) Inshort, the "jury was misled into trusting Samantha over the arsonforensic lab." (178)

A more recent arson-dog case involved James Hebshie, who wasconvicted of arson and mail fraud in 2006. A federal district courtgranted his habeas petition based on ineffective assistance of counselgrounds. (179) In the court's view, if a Daubert hearing had beenrequested on the canine evidence, there was a "'reasonableprobability' that the Court would have excluded the caninetestimony or severely limited it." (180) Without a challenge fromthe defense, the dog handler testified that his dog, Billy, "was97% accurate." (181) Indeed, the handler testified to "analmost mystical account of Billy's powers and her unique olfactorycapabilities." (182) The court explained:

 [The handler] went on and on about what he understood about Billy, as if his relationship with Billy somehow enhanced the reliability and probative value of the results--that she was unique, that he could "read her face," that he was with her 365 days a year, that he knew her personality, "the way her eyes shifted," the ways her ear shifted, etc. (183)

Moreover, the handler focused on one area as the origin of the fireand testified that the dog had not alerted anywhere else on thepremises. However, the handler had limited the dog's access to thatone area. (184) In addition, a dog's failure to alert has noevidential value: "[T]he scientific literature cast doubt on thesignificance of the dog's failure to alert (false negatives) andeven raised concerns about canine 'proficiency' testing,concerns counsel never raised." (185) Indeed, the term"accelerant-detection" dog was misleading because the dog istrained to alert to many common materials that are not accelerants; thesite of the fire was a convenience store which sold lighter fluid andlighters. (186)

5. Post-Daubert Cases

The courts' response to bogus arson evidence is mixed. (187)It is not hard to find cases citing discredited arson indicators afterDaubert, such as pour patterns or puddle configurations, (188) meltedbedsprings, (189) concrete spalling, (190) fire load, (191) and"fast and hot" burn. (192) Decided in 1998, Michigan MillersMutual Insurance Corp. v. Benfield (193) is considered the "firstserious challenge to the 'old school' of fireinvestigators." (194) In that case, the Eleventh Circuit ruled thatarson testimony "is subject to Daubert's inquiry regarding thereliability of such testimony." (195) Some federal courts afterBenfield cited NFPA 921. (196) Yet, a 2011 article on the subject beganwith the passage: "Fire researchers have shattered dozens of arsonmyths in recent years. So why do American courts still lag behind?"(197) And a 2013 survey of 586 public sector fire investigators foundthat some myths endure: "Nearly 40 percent did not know that crazedglass is caused by rapid cooling, not rapid heating. Twenty-threepercent think puddle-shaped burns indicate the use of an accelerant.Eight percent still believe that alligator blistering implies that afire burned fast and hot." (198)

The TFSC Report did more than the courts to curb flawed arsontestimony. And it took the execution of an innocent man to trigger thatreport. (199) In addition, the resistance to change is all-too-familiar:Rules based on science "were slow to take hold, as veteraninvestigators clung to what now are considered disproven theories. Insome police and fire departments, investigators were openly hostile tothe updated science." (200)

D. Comparative Bullet Lead Analysis

For more than three decades, FBI experts testified aboutComparative Bullet Lead Analysis ("CBLA"), a technique firstused in the investigation into President Kennedy's assassination.(201) CBLA compares trace chemicals found in bullets at crime sceneswith ammunition found in a suspect's possession. This technique wasused when firearms or "ballistics" identification could not beemployed. FBI experts used various analytical techniques, first neutronactivation analysis ("NAA"), and then inductively coupledplasma-atomic emission spectrometry ("ICP-AES"), to determinethe concentration levels of seven elements--arsenic, antimony, tin,copper, bismuth, silver, and cadmium--in the bullet lead alloy of thesuspect's bullets and those recovered from the crime scene.Statistical tests were then used to compare the elements in each bulletand determine whether the fragments and suspect's bullets were"analytically indistinguishable" for each of the elementalconcentration averages. Exactly what the phrase "analyticallyindistinguishable" meant was the main issue--i.e., did such afinding mean that the bullet fragments came from a small or largeuniverse? Obviously, the probative value of the test results woulddiffer if only a hundred bullets had the same chemical composition asopposed to several million.

The published cases revealed disparate and often inconsistentinterpretive conclusions provided by FBI experts. In some, expertstestified that two exhibits were "analyticallyindistinguishable." (202) In other cases, examiners concluded thatsamples could have come from the same "source" or"batch." (203) In still others, they stated that the samplescame from the same source. (204) The testimony in numerous cases wentmuch further and referred to a "box" of ammunition--typicallyfifty loaded cartridges, sometimes twenty. For example, two specimens:

(1) Could have come from the same box; (205)

(2) Could have come from the same box or a box manufactured on thesame day; (206)

(3) Were consistent with their having come from the same box ofammunition; (207)

(4) Probably came from the same box; (208) or

(5) Must have come from the same box or from another box that wouldhave been made by the same company on the same day. (209)

Several other statements that differ appear in the publishedopinions. An early case reported that the specimens "had come fromthe same batch of ammunition: they had been made by the samemanufacturer on the same day and at the same hour." (210) One casereports the expert's conclusion with a statistic. (211) In anothercase, the expert used the expressions "such a finding is rare"(212) and "a very rare finding." (213) In still another case,the expert "opined that the same company produced the bullets atthe same time, using the same lead source. Based upon DOJ records, sheopined that an overseas company called PMC produced the bullets around1982." (214)

1. NAS Bullet Lead Report (2004)

The technique was not seriously challenged until a retired FBIexaminer, William Tobin, began questioning the procedure in scientificand legal journals (215) as well as in court testimony. (216) As aresult, the FBI asked the National Academy of Sciences ("NAS")to review the technique. The 2004 NAS Report undercut the FBI testimony,stating: "The available data do not support any statement that acrime bullet came from a particular box of ammunition. In particular,references to 'boxes' of ammunition in any form should beavoided as misleading under Federal Rule of Evidence 403." (217)Perhaps the most disturbing case is State v. Earhart, (218) a capitalmurder case in which the CBLA evidence apparently played a significantrole. (219) The transcript contains the following expert testimony:"We can--from my 21 years experience of doing bullet lead analysisand doing research on boxes of ammunition down through the years I candetermine if bullets came from the same box of ammunition...."(220) However, the NAS Report found that the amount of bullets that canbe produced from a melt "can range from the equivalent of as few as12,000 to as many as 35 million 40-grain, .22 caliber longriflebullets...." (221) Ear hart was executed before the report waspublished. (222)

2. Post-Report Developments

Much of the FBI testimony rested on a database, which the Bureauhad built up over the span of several years. Although the NAS committeefrequently asked for this data during its year-long investigation, theFBI did not turn it over until it was too late to analyze for itsreport. (223) The two statisticians who served on the NAS committeelater wrote that their subsequent inspection of the data"identified several peculiarities." (224) First, the databasewas incomplete. The FBI claimed to have a "complete data file"of some 71,000+ measurements, but it only turned over 64,869. Moreover,only ICP-AES's measurements were included; a different analyticalmethod--NAA--had been used before 1997. Both techniques measured thesame elements, and thus the results from either technique would havebeen appropriate for comparison. Additionally, the numbering system forthe bullets was "highly inconsistent and rather unexpected,"suggesting that some bullet measurements were deleted. (225)Additionally, "a rough investigation of the measurement errorindicated many measurement errors that exceeded the FBI's claimedanalytical precision of 2-5%." (226) Finally, "only 15% of the1,079 cases listed in these two files had measurements from [NationalInstitute of Standards and Technology] ... making it impossible todetermine the frequency of matches ... in a case." (227)Accordingly, the "missing data and the inconsistentprecisions" undermined the Bureau's public claims. (228) Theseauthors were puzzled by the FBI's failure to disclose data:"The scientific method is important for science generally; forensicscience is no exception.... [T]he evidence in this paper suggests that,at least for [CBLA], forensic science failed in the requirement to sharethe material, methods, and data to reach conclusions with the scientificcommunity." (229)

The FBI's response to the NAS Report was also troubling. TheBureau quickly put out a press release, obscuring the report'sfindings. (230) The release highlighted the committee's conclusionthat the FBI was using appropriate instrumentation and suitable elementsfor comparison. Yet, these aspects of CBLA were never seriouslyquestioned. Rather, the interpretation of the data was disputed. Onlyone sentence in the press release addressed this critical issue:"Recommendations by the [NAS] included suggestions to improve thestatistical analysis, quality control procedures, as well as experttestimony." (231) The news media read the report quitedifferently--e.g., "Study Shoots Holes in Bullet Analyses byFBI," (232) "Report Finds Flaws in FBI Bullet Analysis,"(233) "FBI Lab Under Scrutiny Again," (234) and "ReportQuestions the Reliability of an F.B.I. Ballistics Test." (235)

The Bureau also included the following passage in the pressrelease: "The basis of bullet lead compositional analysis issupported by approximately 50 peer-reviewed articles found in scientificpublications beginning in the early 1970's. Published research andvalidation studies have continued to demonstrate the usefulness of themeasurements of trace elements within bullet lead." (236) Incontrast, the NAS Report pointed out that there were "very fewpeer-reviewed articles on hom*ogeneity and the rate of false positivematches" and "[o]utside reviews have only recently beenpublished." (237)

Over a year later, the FBI discontinued CBLA testing (238) andissued another, similar press release. Once again, the release minimizedthe problems, citing the following reason for its decision: "Whilethe FBI Laboratory still firmly supports the scientific foundation ofbullet lead analysis, given the costs of maintaining the equipment, theresources necessary to do the examination, and its relative probativevalue, the FBI Laboratory has decided that it will no longer conductthis exam." (239) Nevertheless, Dwight Adams, the laboratorydirector, had written a private memorandum to the FBI Director a monthearlier specifying different reasons for abandoning the technique,including the following comments: (1) "We cannot afford to bemisleading to a jury" and (2) "We plan to discourageprosecutors from using our previous results in future cases." (240)The press release did not reflect either concern.

In the wake of the NAS Report, several state courts excluded CBLAevidence. (241) Surprisingly, the FBI supplied affidavits in severalcases supporting prosecutors' efforts to sustain convictions basedon the technique. In one affidavit, the FBI cited the NAS report butignored that the report had faulted the Bureau's statisticalmethods. The chair of the NAS committee criticized the affidavit becauseit did "not discuss the statistical bullet-matching technique,which is key and probably the most significant scientific flaw found bythe committee." (242) The affidavit was also misleading because itestimated that the maximum number of .22-caliber bullets in a batch oflead was 1.3 million, when the NAS committee found that the number couldbe as high as 35 million. (243)

On November 18, 2007, 60 Minutes aired a segment on CBLA. (244) Inan interview, the now-retired FBI lab director acknowledged thattestimony about boxes was "misleading and inappropriate."(245) That broadcast, along with a Washington Post investigation,questioned the FBI's response to the NAS Report. The main problemwas that only the FBI had records of all the cases in which its expertshad testified, and the Bureau had declined to disclose the names ofthose cases. (246) Instead, the Bureau relied on the NAS Report, its ownpress releases, and pro forma letters sent to prosecution and defenseorganizations to notify defendants. This method of communication wasgrossly inadequate because the letters neither highlighted the problem,nor its significance. (247) A few days after the 60 Minutes expose,Senator Patrick Leahy, the Chairman of the Senate Judiciary Committee,sent a letter to the FBI Director noting that the Bureau's lettersgave "the false impression that these discredited tests hadcontinuing reliability." (248)

Here, the flaws are many: Lack of foundational research, failure tomake a database available to outside scientists, and ignoring theFBI's own protocols by presenting inconsistent and misleadingtestimony. Moreover, the reluctance to confess error and take timelycorrective action violated basic scientific norms. After decades of use,a federal district court in 2003 excluded CBLA evidence under theDaubert standard for the first time. (249)

II. MISLEADINGLY PRESENTED TECHNIQUES

A. Firearms & Toolmark Identifications

Firearms identifications, popularly known as"ballistics," is another long-established forensic discipline.It developed in the early part of the last century, and by the 1930scourts were admitting evidence based on this technique. "Subsequentcases have followed these precedents, admitting evidence of bullet,cartridge case, and shot shell identifications." (250) Toolmarkcomparison, a related discipline, was also accepted during this period.(251) At the time Daubert was decided, the FBI's position wasclear: "Firearms identification is the Forensic Science disciplinethat identifies a bullet, cartridge case or other ammunition componentas having been fired by a particular firearm to the exclusion of allother firearms." (252) Yet, the examination, by means of acomparison microscope, is subjective and without a meaningful standard.

1. Post-Daubert Cases

The courts gave short shrift to the initial post-Daubert challengesto firearms and toolmark identifications. (253) In 2005, however, thelegal landscape changed abruptly. In United States v. Green (254) thedistrict judge questioned the foundational basis of firearmsidentifications. The court wrote that the expert "declared thatthis match could be made 'to the exclusion of every other firearmin the world.' ... That conclusion, needless to say, isextraordinary, particularly given [his] data and methods." (255)Moreover, the expert could not cite any reliable error rates andadmitted that he relied mainly on his subjective judgment. In addition,"[t]here were no reference materials of any specificity, nonational or even local database on which he relied. And although herelied on his past experience with these weapons, he had no notes orpictures memorializing his past observations." (256) In the end,the court restricted the expert's testimony; he could only explainthe ways in which the casings were similar but not that they came from aspecific weapon "to the exclusion of every other firearm in theworld." (257) In the court's view, that conclusion"stretches well beyond [the expert's] data andmethodology." (258)

A few weeks later, a different district judge in United States v.Monteiro (259) found that the technique "is largely a subjectivedetermination based on experience and expertise." (260)Importantly, the court also concluded that the theory on which theexpert relied was "tautological." (261) The Association ofFirearm and Toolmark Examiners ("AFTE"), the leadingorganization of examiners, proposed the theory. (262) Under this theory,the examiner may declare a positive identification if (1) there is"sufficient agreement" of marks between the crime scene andtest bullets; and (2) there is "sufficient agreement" when theexaminer says there is. (263) In short, the "sufficientagreement" threshold is "in the minds eye of the examiner andis based largely on training and experience." (264) The court wouldnot admit the evidence unless the expert could better document theexamination.

Together, Green and Monteiro should have served as a shot acrossthe bow. But they did not; courts continued to admit the same evidenceas before. (265)

2. NAS Ballistic Imaging Report (2008)

In 2008, NAS published a report on computer imaging of bullets.(266) Although firearms identification was not the primary focus of theinvestigation, a section of the report commented on the subject. (267)After surveying the literature on uniqueness, reproducibility, andpermanence of individual characteristics, the report noted that"[m]ost of these studies are limited in scale and have beenconducted by firearms examiners (and examiners in training) in state andlocal law enforcement laboratories as adjuncts to their regularcasework." (268) The report found that the "validity of thefundamental assumptions of uniqueness and reproducibility offirearms-related toolmarks has not yet been fully demonstrated."(269) The report went on to caution:

 Conclusions drawn in firearms identification should not be made to imply the presence of a firm statistical basis when none has been demonstrated. Specifically ... examiners tend to cast their assessments in bold absolutes, commonly asserting that a match can be made "to the exclusion of all other firearms in the world." Such comments cloak an inherently subjective assessment of a match with an extreme probability statement that has no firm grounding and unrealistically implies an error rate of zero. (270)

Citing this report, the district court in United States v. Glynn(271) ruled that the expert would only be permitted to testify that itwas "more likely than not" that recovered bullets andcartridge cases came from a particular weapon. (272) The court alsocommented: "Based on the Daubert hearings ... the Court veryquickly concluded that whatever else ballistics identification analysiscould be called, it could not fairly be called'science.'" (273) The court further noted that"[t]he problem is compounded by the tendency of ballistics experts... to make assertions that their matches are certain beyond all doubt,that the error rate of their methodology is 'zero,' and othersuch pretensions." (274)

3. NAS Forensic Science Report (2009)

As noted previously, NAS issued its forensic report the followingyear in 2009. That Report summarized the state of the research asfollows:

 Because not enough is known about the variabilities among individual tools and guns, we are not able to specify how many points of similarity are necessary for a given level of confidence in the result. Sufficient studies have not been done to understand the reliability and repeatability of the methods .... Individual patterns from manufacture or from wear might, in some cases, be distinctive enough to suggest one particular source, but additional studies should be performed to make the process of individualization more precise and repeatable. (275)

In a different passage, the report--citing firearm and toolmarkidentifications--observed that "[m]uch forensic evidence ... isintroduced in criminal trials without any meaningful scientificvalidation, determination of error rates, or reliability testing toexplain the limits of the discipline." (276)

AFTE rejected these findings out of hand, arguing that NAS"ignore[d] extensive research supporting the scientificunderpinnings of the identification of firearm and toolmark evidence...," (277) The court in United States v. Otero (278) accepted theAFTE's position, citing studies which it was ill-equipped toevaluate. (279) A subsequent review of the oft-cited studies by twoscientists concluded:

 Exaggerated and unfounded implications relating to rates of error inferred from even the best of existing experiments in the field of firearms/toolmarks, generally self-described as 'validation studies', typically result from statistical, metallurgical and/or psychological (cognitive) deficiencies in the design and conduct of the experiments, and frequently lead to unjustified inferential extrapolation to universal assumption for the practice domain. (280)

Other courts took an important, but still limited, step ofrestricting examiner testimony by precluding the expert from makinggross overstatements such as declaring a match to the exclusion, eitherpractical or absolute, of all other weapons. (281) Similarly, somecourts forbade experts from testifying that they hold their opinions toa "reasonable degree of scientific certitude." (282) That termhas long been required by courts in many jurisdictions for the admissionof expert testimony. Incredibly, the phrase has no scientific meaningand the claim of certainty is unsupported by empirical research. Thus,it is grossly misleading. Indeed, the National Commission on ForensicScience rejected it. (283) Still other courts went off on a quixotictangent, substituting the phrase "reasonable degree ofballistic" certitude. (284) Changing "scientificcertainty" to "ballistic certainty" merely underscoresthe courts' scientific incompetence.

However, even these modest limitations were rejected by othercourts. (285) For example, in United States v. Casey, (286) the districtcourt declined "to follow sister courts who have limited experttestimony based upon the 2008 and 2009 NAS reports and, instead, remainsfaithful to the long-standing tradition of allowing the unfetteredtestimony of qualified ballistics experts." (287)

4. White House PCAST Report (2016)

The 2016 White House PCAST report agreed with the 2009 NASReport's characterization of the scientific research on firearmsand tool-marks identification: "We find that many of these earlierstudies were inappropriately designed to assess foundational validityand estimate reliability. Indeed, there is internal evidence among thestudies themselves indicating that many previous studies underestimatedthe false positive rate by at least 100-fold." (288) In addition,PCAST found only one of the post-2009 studies sufficiently rigorous. TheDefense Department's Forensic Science Center commissioned thestudy, which was conducted by an independent testing lab--the AmesLaboratory, a Department of Energy national laboratory affiliated withIowa State University. In this study, "[t]he false-positive ratewas estimated at 1 in 66, with a confidence bound indicating that therate could be as high as 1 in 46." (289) The study had not beenpublished in a scientific journal. According to the PCAST Report, morethan one study is required and studies should be published inpeer-reviewed scientific literature. Consequently, "the currentevidence still falls short of the scientific criteria for foundationalvalidity." (290)

AFTE quickly retorted, expressing their "disappointment in thePCAST's choice to ignore the research that has been conducted"and claiming that "[d]ecades of validation and proficiency studieshave demonstrated that firearm and toolmark identification isscientifically valid. ..." (291) However, when PCAST later invitedstakeholders to submit validation studies that it may have overlooked,no studies satisfying PCAST's criteria were offered. (292)

The lessons here are familiar. For years, an entrenched forensicdiscipline vigorously guarded its turf by rejecting the conclusions ofthe outside scientific community. (293) It published a journal which was"peer-reviewed" by other members of its discipline. Thejournal, which is advertised as "the Scientific Journal" ofAFTE, was not generally available until 2016. The discipline claimed tobe a "science" but did not hold itself to the normativestandards of science. The AFTE "Theory of Identification" is"clearly not a scientific theory, which the National Academy ofSciences has defined as 'a comprehensive explanation of some aspectof nature that is supported by a vast body of evidence.' ... Moreimportantly, the stated method is circular." (294) Only recently,after two NAS reports, have some courts begun to limit misleadingtestimony. Many have not. Thus, the courts' competence to deal withflawed research remains extant. (295)

In 2005, the district court in Green cautioned: "The morecourts admit this type of toolmark evidence without requiringdocumentation, proficiency testing, or evidence of reliability, the moresloppy practices will endure; we should require more." (296) Over adecade later, a concurring opinion in Williams v. United States (297)concluded: "As matters currently stand, a certainty statementregarding toolmark pattern matching has the same probative value as thevision of a psychic: it reflects nothing more than the individual'sfoundationless faith in what he believes to be true." (298) Inshort, there is a "lost decade" during which the disciplinesummarily dismissed criticisms when it should have lead the effort formore rigorous research.

B. Fingerprint Examinations

Before DNA analysis, fingerprint identification was the "goldstandard" in forensics. (299) Like many other forensic disciplines,it gained judicial acceptance decades before Daubert was decided. Peoplev. Jennings (300) the first reported fingerprint case, was decided in1911. In 1984, the FBI pronounced the technique "infallible"in its official publication, which also referred to the technique as a"science." (301) Nevertheless, it is a subjective techniquewithout an objective standard and typically involves partial prints withinevitable distortions.

1. Post-Daubert Cases

After Daubert, challenges to fingerprint comparison testimony weredecidedly unsuccessful. (302) One infamous case, United States v.Havvard, (303) illustrates the judiciary's lack of rigor inapplying Daubert. Not only did the district court uphold the fingerprinttestimony's admissibility, it styled the technique as "thevery archetype of reliable expert testimony under [the Daubert/Kumho]standards." (304) According to the court, latent printidentification had been "tested" for nearly one hundred yearsin adversarial proceedings with the highest possible stakes--liberty andsometimes life. Yet, Daubert required scientific, not"adversarial," testing. (305) Next, in citing "peerreview," the court noted that a second fingerprint examiner alsocompared the prints: "In fact, peer review is the standardoperating procedure among latent print examiners." (306) Thisstatement reveals a fundamental misunderstanding of "peerreview" as used in Daubert. In that case, peer review meantrefereed scientific journals in which validation research is published.An amici brief submitted in Daubert by the New England Journal ofMedicine and other scientific publications explained that peerreview's "role is to promote the publication of well-conceivedarticles so that the most important review, the consideration of thereported results by the scientific community, may occur afterpublication." (307)

Moreover, the court accepted the prosecution expert'sastounding claim that the "error rate for the method is zero."(308) Experts argued that, while individual examiners may make mistakes,the method itself is perfect. The dichotomy between"methodological" and "human" error rates in thiscontext, however, is "practically meaningless" (309) becausethe examiner is the method. (310) Finally, the court turned Daubert onits head by requiring the defendant to prove the evidence wasunreliable, a distortion that would be employed in later cases. (311)

Then, United States v. Llera Plaza (312) "sent shock wavesthrough the community of fingerprint analysts." (313) In that 2002case, Judge Pollak ruled that fingerprint experts would not be permittedto testify that two sets of prints "matched"--that is, apositive identification to the exclusion of all other persons. This wasapparently the first time in over ninety years that such a decision hadbeen rendered. (314) On rehearing, however, Judge Pollak reversedhimself, (315) and later cases continued to uphold the admissibility offingerprint evidence. (316) Nevertheless, the case captured theattention of the media with news reports, (317) main streampublications, (318) scientific journals, (319) and television showsgiving it substantial coverage. (320) A spate of legal articlesfollowed, (321) with some commentators believing that Llera Plaza / wasmore faithful to Daubert than Llera Plaza II. (322) In response, the FBIadopted a "circle the wagons" attitude, fiercely defending thetechnique. The head of the FBI fingerprint section told 60 Minutes thatthe error rate was "zero," examiners only testify to"hundred percent certainty," and the FBI had won"forty-one out of forty-one" legal challenges to fingerprintevidence. (323)

The appellate opinion most faithful to Daubert appeared in UnitedStates v. Crisp (324)--unfortunately in dissent. The majority opinionupheld the admissibility of fingerprint evidence by shifting the burdenof proof to the defendant and by grandfathering the technique. (325) Indissent, Judge Michael conscientiously applied the Daubert factors.First, he noted that the "government did not offer any record oftesting on the reliability of fingerprint identification.... [T]herehave not been any studies to establish how likely it is that partialprints taken from a crime scene will be a match for only one set offingerprints in the world." (326) Second, as for peer review:

 [a]gain, the government offered no evidence on this factor at trial. Fingerprint examiners ... have their own professional publications.... But unlike typical scientific journals, the fingerprint publications do not run articles that include or prompt critique or reanalysis by other scientists. Indeed, few of the articles address the principles of fingerprint analysis and identification at all ..., (327)

Third, "an error rate must be demonstrated by reliablescientific studies, not by assumption." (328) Fourth, "thegovernment did not establish that there are objective standards in thefingerprint examination field to guide examiners in making theircomparisons." (329) Fifth, while acknowledging general acceptancein the fingerprint community, the judge remarked that "[n]othing inthe record in this case shows that the fingerprint examination communityhas challenged itself sufficiently or has been challenged in any realsense by outside scientists." (330)

2. Madrid Train Bombing

Llera Plaza was soon eclipsed by a more sensational event--theFBI's misidentification of Brandon Mayfield as the source of thecrime scene prints in the terrorist train bombing in Madrid on March 11,2004. (331) More than any other event, the Mayfield affair exposed themyth of fingerprint infallibility. This debacle resulted ininvestigations by the FBI (332) and the Inspector General of the DOJ("IG"). (333) One of the more troubling aspects of thesereports dealt with the culture in the laboratory. The FBI internalinvestigation found that "[t]o disagree was not an expectedresponse," (334) and the IG reported that "FBI examiners didnot attempt to determine the basis of the [Spanish NationalPolice's] doubts before reiterating that they were 'absolutelyconfident' in the identification on April 15, a full week beforethe FBI Laboratory met with the SNP." (335)

In addition to highlighting the lack of foundational research,these events raised a host of other issues, including: (1) the role ofcognitive bias in subjective techniques,336 (2) the lack of well-definedstandards, (337) (3) the failure to administer rigorous proficiencytests, (338) (4) the manipulation of research, (339) and (5) otherinstances of misidentifications. (340) The FBI did not undertake aserious review of fingerprints until it was compelled to address theissue due to the negative publicity surrounding the Mayfieldmisidentification. Even then, however, the FBI still characterized thetechnique as "scientific." (341)

The scientific community continued to note the lack of research,(342) and the courts continued to ignore this fact. (343) Indeed, inUnited States v. Baines, (344) decided in 2009, the head of the FBIfingerprint section testified: "As to these 'falsepositives' ... the FBI had 'made, on average, about oneerroneous identification every 11 years.' The total number ofidentifications made has been about one million per year ... so that theknown actual error rate was about one per eleven millionidentifications." (345) Problematically, he merely assumed that allthe other identifications were correct, thus disqualifying his analysis.Perhaps the most troubling aspect of this testimony was the lack ofself-awareness for a person who claimed to be a scientist. (346)

3. NAS Forensic Science Report (2009)

Fingerprint examiners follow a procedure known as Analysis,Comparison, Evaluation, and Verification (ACE-V). The 2009 NAS reportobserved that since "the ACE-V method does not specify particularmeasurements or a standard test protocol ... examiners must makesubjective assessments throughout." (347) Thus, the ACE-V method istoo "broadly stated" to "qualify as a validated methodfor this type of analysis." (348) The Report added that "[t]helatent print community in the United States has eschewed numericalscores and corresponding thresholds" and consequently relies"on primarily subjective criteria" in making the ultimateattribution decision. (349) In making the decision, the examiner mustdraw on his or her personal experience to evaluate such factors as"inevitable variations in pressure," but to date those factorshave not been "characterized, quantified, or compared." (350)In addition, the Report gave short shrift to the zero-error-rateargument, finding that "claims that these analyses have zero errorrates are not scientifically plausible." (351) In conclusion, theReport outlined an agenda for the research it considered necessary"[t]o properly underpin the process of friction ridgeidentification...." (352)

Several studies were published after the NAS Report. (353) The mostimportant was a FBI study published in 2011, (354) which is discussedbelow.

4. White House PCAST Report (2016)

According to the White House PCAST report, "latent fingerprintanalysis is a foundationally valid subjective methodology" and theFBI "has lead the way" by conducting the black-box study.(355) Nevertheless,

 [the] false positive rate ... is substantial and is likely to be higher than expected by many jurors based on longstanding claims about the infallibility of fingerprint analysis. The false-positive rate could be as high as 1 error in 306 cases based on the FBI study and 1 error in 18 cases based on a study by another crime laboratory. In reporting results of [a] latent-fingerprint examination, it is important to state the false-positive rates based on properly designed validation studies ..., (356)

Moreover, "testimony asserting any specific level of increasedaccuracy (beyond that measured in the studies) due to blind independentverification would be scientifically inappropriate, as speculationunsupported by empirical evidence." (357)

5. AAAS Fingerprint Report (2017)

In September 2017, the AAAS published an extensive report onfingerprint analysis. (358) An accompanying news release, summarized thereport's findings: "Courtroom testimony and reports stating oreven those implying that fingerprints collected from a crime scenebelong to a single person are indefensible and lack scientificfoundation ...," (359)

The report reached a number of conclusions. First, claims thatexperts can identify the source of a latent print with 100 percentaccuracy, are "clearly overstated and are now widely recognized asindefensible." (360) Second, use of the term"identification" in reports and testimony even withqualifications "fail to deal forthrightly with the level ofuncertainty that exists in latent print examination" and"cannot be justified scientifically." (361) Third, because ofpublic misconceptions, experts:

 should acknowledge: (1) that the conclusions being reported are opinions rather than facts (as in all pattern-matching disciplines), (2) that it is not possible for a latent print examiner to determine that two friction ridge impressions originated from the same source to the exclusion of all others; and (3) that errors have occurred in studies of the accuracy of latent print examination. (362)

The report went on to make several recommendations. Experts should"avoid statements that claim or imply that the pool of possiblesources is limited to a single person. Terms like 'match,''identification,' 'individualization' and theirsynonyms, imply more that the science can sustain." (363) Inaddition, experts should "be prepared to discuss forth rightly theresults of research studies that tested the accuracy of latent printexaminers on realistic known-source samples." (364)

Despite the ruckus created by Llera Plaza and the Mayfield fiasco,examiner testimony remained unchanged. Testimony such as "zeroerror rates," "matches to the exclusion of all otherfingerprints," and "100 percent certainty"--which hadbeen used for decades--continued, while the fingerprint communityremained oblivious that such statements were scientifically implausible.As with firearms identification, there is a "lost decade"during which more research could have been conducted. As one judge notedin a 2003 dissent: "The government has had ten years to comply withDaubert. It should not be given a pass in this case." (365) Thosewords were written fifteen years ago.

On a positive note, the Mayfield incident did trigger theFBI's black box study, which was a significant achievement. Still,this study was released 100 years after the courts first admittedfingerprint evidence. (366) The White House PCAST report found it"distressing" that properly constructed validation studies hadonly been conducted recently and only one study had been published in apeer-reviewed journal. (367) Daubert has had little effect. (368)

III. FORENSIC SCIENCE RESEARCH

By now it is almost a truism that too many forensic disciplines arenot grounded in science--and yet their adherents continue to claim themantle of science. The 2009 NAS Report emphasized the "notabledearth of peer-reviewed, published studies establishing the scientificbases and validity of many forensic methods." (369) Indeed, theco-chair of the NAS committee, Judge Harry Edwards, later stated:"I think that the most important part of our committee'sreport is its call for real science to support the forensicdisciplines." (370) Not surprisingly, the report triggeredextensive commentary. (371) One article cataloged the numerous ways inwhich forensic science has failed to develop a research culture (372)and argued that the "core values" of a scientific culture"are empiricism, transparency, and an ongoing criticalperspective." (373) Another article documented the serious problemsthat have arisen when law enforcement controls forensic research. (374)

A. National Commission on Forensic Science (2013-17)

To its credit, the DOJ, in partnership with the National Instituteof Standards and Technology ("NIST"), established the NationalCommission on Forensic Science in 2013. The commission's task wasto enhance the practice and improve the reliability of forensic science.(375) Early on, the commission created a subcommittee on scientificinquiry and research, which undertook the task of reviewingbibliographies of foundational literature that had been compiled byvarious forensic disciplines. (376) The subcommittee quickly concludedthat even a "cursory review" of the bibliographies raisedserious concerns. One basic problem involved the definition offoundational literature. According to the subcommittee, "[i]n somecases, it was unclear which literature citations are crucial to supportthe foundation of a particular forensic science discipline." (377)This finding led the subcommittee to define the term. Foundational,scientific literature should consist of "original research,substantive reviews of the original research, clinical trial reports, orreports of consensus development conferences." (378) Tellingly, thesubcommittee felt compelled to add: "While other forms ofdissemination of research and practice (e.g., oral and posterpresentations at meetings, workshops, personal communications,editorials, dissertations, theses, and letters to editors) play animportant role in science, the open, peer-reviewed literature is whatendures and forms a foundation for further advancements." (379)

The subcommittee's second concern was that "[s]ome of thecited literature had not undergone a rigorous peer-review process."(380) Peer review by other members of a forensic discipline is notsufficient. (381) Many of the reviewers are not scientists, and there isthe problem with role bias. According to the subcommittee, foundationalresearch should be subjected to "rigorous peer review withindependent external reviewers to validate the accuracy ... [and]overall consistency with scientific norms of practice" (382) and"[p]ublished in a journal that is searchable using free, publiclyavailable search engines ... " (383) With few exceptions, thedisciplines considered above have not satisfied these requirements.(384)

Another recommendation--one on technical merit--provides: "Allforensic science methodologies should be evaluated by an independentscientific body to characterize their capabilities and limitations inorder to accurately and reliably answer a specific and clearly definedforensic question." (385) Significantly, the commission recommendedthat NIST be the independent scientific evaluator within the justicesystem.

B. White House PCAST Report (2016)

Unlike the commission, which had a broad mandate, the White HousePCAST Report focused only on the validation issue. It took pains toexplain the concept of validation, noting that forensic methods must bebased on empirical studies and be "repeatable, reproducible, andaccurate, at levels that have been measured and are appropriate to theintended application." (386) The report recognized that forensicmethods may be either objective or subjective. Foundational validity forobjective methods "can be established by studying [and] measuringthe accuracy, reproducibility, and consistency of each of its individualsteps." (387) By definition, this approach is not possible withsubjective techniques because they involve significant human judgment.Consequently, validity and reliability for these methods must be basedon "black-box studies"--as if a "black box" is inthe examiner's head--in which numerous examiners make decisions onmany independent tests in order to determine error rates. (388)

Importantly, the report also specified what does not qualify asvalidation: "[N]either experience, nor judgment, nor goodprofessional practices (such as certification programs and accreditationprograms, standardized protocols, proficiency testing, and codes ofethics) can substitute for actual evidence of foundational validity andreliability." (389) Moreover, expressions of confidence byindividual examiners or a consensus among practitioners about accuracycannot substitute for "error rates estimated from relevantstudies." (390) In sum, empirical evidence is the "sine quanon" for establishing foundational validity. (391)

PCAST also recommended that NIST conduct scientific evaluations ofthe validity of current and new forensic technologies: " [t]oensure the scientific judgments are unbiased and independent, suchevaluations should be conducted by an agency which has no stake in theoutcome." (392)

In response, DOJ released a statement criticizing the report on theday of its release. According to DOJ, the PCAST Report "does notmention numerous published research studies which seem to meetPCAST's criteria for appropriately designed studies providingsupport for foundational validity. That omission discredits the PCASTreport as a thorough evaluation of scientific validity." (393)PCAST, in turn, invited all stakeholders to identify validity studiesthat it might have overlooked. "DOJ ultimately concluded that ithad no additional studies for PCAST to consider." (394) Nor did themore than 400 papers submitted by twenty-six respondents cause PCAST tochange its positions. The bottom line remained: "In science,empirical testing is the only way to establish the validity and degreeof reliability of such an empirical method. Fortunately, empiricaltesting of empirical methods is feasible. There is no justification foraccepting that a method is valid and reliable in the absence ofappropriate empirical evidence." (395) However, most prior studiesuse "closed-set design." (396) In these studies, "thecorrect source of each questioned sample is always present; studiesusing the closed-set design have underestimated the false-positive andinconclusive rates by more than 100-fold." (397)

IV. INDEPENDENT SCIENTIFIC REVIEW

As discussed above, the courts have too often failed to fulfilltheir "gatekeeper" (398) function under Daubert. However, theDaubert Court also suggested that the adversary system would serve as acomplementary safeguard, noting that "[vigorous cross-examination,presentation of contrary evidence, and careful instruction on the burdenof proof are the traditional and appropriate means of attacking shakybut admissible evidence." (399)

Yet, these "traditional" means have also provedinadequate. After the release of the NAS Report, some commentatorsfocused on defense counsel's incompetence. (400) A 2009 study ofthe cases of 137 convicts exonerated by DNA profiling revealed that"[d]efense counsel rarely made any objections to the invalidforensic science testimony in these trials and rarely effectivelycross-examined forensic analysts who provided invalid sciencetestimony." (401) One commentator summed it up this way:

 Unlike the extremely well-litigated civil challenges, the criminal defendant's challenge is usually perfunctory. Even when the most vulnerable forensic sciences--hair microscopy, bite marks, and handwriting--are attacked, the courts routinely affirm admissibility citing earlier decisions rather than facts established at a hearing. Defense lawyers generally fail to build a challenge with appropriate witnesses and new data. Thus, even if inclined to mount a Daubert challenge, they lack the requisite knowledge and skills, as well as the funds, to succeed. (402)

Although the defense bar bears some responsibility forDaubert's failure, there are limits to what can be expected ofoverburdened and chronically underfunded public defenders when dealingwith expert testimony. Better training for defense counsel--which issorely needed--is not sufficient. Similarly, access to defenseexperts--also sorely needed--may not be adequate. (403) Defense expertscan challenge prosecution experts' methods and opinions but do nothave the funds to conduct foundational research, nor can they act asindependent evaluators of foundational research on an ongoing basis.(404)

An independent scientific review is required. NAS has published themost authoritative and independent reviews of forensic science. Inaddition to the forensic report, NAS issued reports on soundspectrometry ("voiceprints"), (405) DNA profiling, (406)polygraph testing, (407) and bullet lead analysis. (408) But NAS is nota governmental entity, and its work depends on outside funding. Thejustice system needs scientific expertise on a continuing basis--andthus institutionalized.

The National Commission's proposal, endorsed by PCAST, taskedNIST with the responsibility of evaluating forensic disciplines on anongoing basis. (409) It should be adopted. NIST has the expertise andindependence for this task and has been increasingly involved inforensic research. There would be a cost, but litigating validity issuesacross the country at Daubert and Frye hearings also has a cost.Moreover, there is a significant expense associated with rectifying thepast mistakes that occurred with hair, (410) bullet lead, (411) DNA,(412) and arson cases. (413)

Unfortunately, the current Attorney General did not renew thecommission's charter in April 2017. (414) The independentscientists on the commission objected to this action, writing:

 The Justice Department now proposes to improve forensic science by moving its oversight and development to an office within the department. This is precisely the opposite of what was recommended by the National Academy of Sciences report and the NCFS. It is a step backwards, because it reinforces the conditions that contributed to the current problems, namely, placing this discipline within the control of law enforcement and prosecutors. The Justice Department is home to many dedicated public servants including scientists whose passion for justice is unquestioned. However, DOJ is not a scientific body, and it is difficult to see how forensic science can become a true science in that environment. Science flourishes when free and independent; only then can the tools and technology that it creates be truly reliable. (415)

The AAAS concurred, also stressing that independence "cannotbe overstated" and expressing concern about the "inherentconflict of interest in having law enforcement overseeing the work offorensic labs on which police and prosecutors rely to win and defendconvictions." (416) The American Academy of Forensic Science alsoopposed the formation of an Office of Forensic Science within DOJ. (417)Instead of heeding this advice, the Attorney General appointed aprosecutor instead of a scientist to head the working group within theDOJ. (418)

These recent events should be put in context. The 2009 NAS Reportrecommended the creation of an independent federal entity--the NationalInstitute of Forensic Sciences--to oversee the field, including theestablishment of a research agenda. (419) If adopted, this proposalwould have wrest control of forensic science from law enforcement. Thereport provided the following justification: Some federal entities were"too wedded" to the status quo and "have failed to pursuea rigorous research agenda to confirm the evidentiary reliability ofmethodologies used in a number of forensic science disciplines."(420) As a result, these "agencies are not good candidates tooversee the overhaul of the forensic science community...." (421)There is little question that the NAS was referring to NationalInstitute of Justice and the FBI Laboratory. The report noted that,although both had provided "modest leadership" in forensicscience, "neither entity has recognized, let alone articulated, aneed for change or a vision for achieving it." (422) Consequently,"advancing science in the forensic science enterprise is not likelyto be achieved within the confines of DOJ." (423) In fact, lawenforcement had manipulated science in the past by shaping the researchagenda, limiting access to data, attacking experts who disagreed withits positions, and "spinning" negative reports. (424)

When Congress did not authorize the creation of the NationalInstitute of Forensic Sciences, DOJ, to its credit, established theNCFS. Most importantly, independent scientists were appointed to thecommission. (425) Placing science back under the DOJ now is a major andunjustified retreat.

CONCLUSION

This Article explained how the judiciary's failure to fulfillits gatekeeper role can be traced back to its refusal to demand andproperly evaluate foundational research--i.e., Daubert's firstfactor, empirical testing. This failure has been systemic. Flawedforensic techniques such as bite mark analysis, microscopic haircomparisons, arson evidence, and comparative bullet lead analysis wereroutinely admitted into evidence without foundational research. Inaddition, firearms, toolmark, and fingerprint examiners repeatedlypresented overstated and misleading conclusions. This Article alsoargued that the justice system may be institutionally incapable ofapplying Daubert in criminal cases because it does not have access toindependent scientific expertise on an ongoing basis, and endorsed theNCFS and PCAST recommendation that NIST should be tasked with thisresponsibility.

Even if an independent scientific review is not institutionalized,PCAST, NCFS, and AAAS have provided guidance for courts dealing withadmissibility challenges. First, the flawed techniques discussed in thisArticle should be excluded. If used at all, bite mark analysis should belimited to exclusions and perhaps to closed universe situations. (426)For hair analysis, mitochondrial DNA analysis is far superior tomicroscopy. Arson evidence should comport with NFPA 921 and the AAASreport. As noted above, the FBI has abandoned comparative bullet leadanalysis.

Second, courts should focus, as Daubert requires, on foundationalresearch. According to PCAST, "neither experience, nor judgment,nor good professional practices (such as certification programs andaccreditation programs, standardized protocols, proficiency testing, andcodes of ethics) can substitute for actual evidence of foundationalvalidity and reliability." (427) The NCFS concurred. (428)

Third, subjective methods can be empirically tested. Such researchhas been conducted. PCAST identified studies in fingerprint and firearmsidentification that meet stringent standards. (429) These studies showan error rate, which should be presented to the jury. (430) However,more than one study is needed.

Fourth, in ruling on admissibility in firearms, toolmark, andfingerprint examination cases, courts should appreciate that there hasbeen a "lost decade"--or two--during which rigorous researchwas not conducted. (431) Instead, the disciplines examined in thisarticle vigorously resisted the views of independent scientists, andthey were typically supported by prosecutors. For example, the wrongfulexecution of Cameron Todd Willingham, which triggered numerousscientific reviews, was not enough to persuade the Texas Fire MarshalOffice that its evidence was flawed, (432) and it took a serendipitousevent--the Madrid train bombing--to provoke fingerprint research.

In short, forensic evidence is in a "Catch-22" situation:Only the federal government has the resources to fund the neededindependent research, but it has no incentive to do so as long asevidence continues to be admitted without proper limitations. Until morescientifically sound studies are published and peer-reviewed byindependent scientists, courts should follow the approach adopted inUnited States v. Glynn, (433) which permitted the expert to testify onlythat it was "more likely than not" that recovered bullets andcartridge cases came from a particular weapon. (434)

Fifth, the presentation of expert testimony needs to be controlled.Once again, PCAST made several recommendations, including:

 Statements suggesting or implying greater certainty are not scientifically valid and should not be permitted. In particular, courts should never permit scientifically indefensible claims such as: "zero," "vanishingly small," "essentially zero," "negligible," "minimal," or "microscopic" error rates; "100 percent certainty" or proof "to a reasonable degree of scientific certainty;" identification "to the exclusion of all other sources;" or a chance of error so remote as to be a "practical impossibility." (435)

The NCFS also recommended against the use of the phrase"reasonable degree of scientific certainty" (436) and the 2009NAS report criticized the use of "zero error rates" and claimsof infallibility. (437) The recent AAAS fingerprint report found noscientific justification for statements of "identity" or"practical certainty" and cautioned against the use of termssuch as "match," "identification," and"individualization." (438)

Unfortunately, there is little reason to believe that examinerswill give up their claims that there is a scientific foundation fortheir discipline. A subjective method without a meaningful protocol canhardly claim to be a science. This is not a new issue, as an editorialin the prestigious scientific journal, Science, entitled "ForensicScience: Oxymoron?" and written by the editor-in-chief, made thesame point fifteen years ago. (439) Similarly, the 2009 NAS Reportcommented: "The law's greatest dilemma in its heavy relianceon forensic evidence ... concerns the question of whether, and to whatextent, there is science in any given forensic science discipline."(440) After Daubert hearings, one court "very quickly concludedthat whatever else ballistics identification analysis could be called,it could not fairly be called 'science.'" (441) The sameis true of fingerprint examinations.

Courts should also guard against attempts to introduce claims of"science" through the backdoor by means of circumlocutionssuch as statements that firearms and fingerprint identifications aresubjective techniques that are "based on science." This ismisleading. Many things are "based on science"--e.g., riding abike, throwing a curve ball, and flying a kite.

Sixth, proficiency testing issues will continue to be litigated.These tests have long been suspect. They are not conducted blind and arenot challenging. (442) The President of Collaborative Testing Servicestold the NCFS "during its seventh meeting on August 10, 2015 thathe has been under commercial pressure to make proficiency testseasier." (443)

Paul C. Giannelli ([dagger])

([dagger]) Distinguished University Professor Emeritus & AlbertJ. Weatherhead III and Richard W. Weatherhead Professor Emeritus, CaseWestern Reserve University. The author wishes to thank Jules Epstein andJulia Leighton for their insightful comments.

(1.) Jose Ortega y Gasset, The Revolt of the Masses 157 (W.W.Norton & Co. 1960) (1930).

(2.) Alex Kozinski, Preface: Criminal Law 2.0, 44 Geo. L.J. ANN.REV. CRIM. PROC. iii, v (2015); see also Almeciga v. Ctr. forInvestigative Reporting, Inc., 185 F. Supp. 3d 401, 415 (S.D.N.Y. 2016)(Rakoff, J.) ("There have been too many pseudo-scientificdisciplines that have since been exposed as profoundly flawed,unreliable, or baseless for any Court to take this [gatekeeping] rolelightly.").

(3.) See Radley Balko, A High-Ranking Obama Official Just Calledfor the "Eradication" of Bite Mark Evidence, WASH. POST (July22, 2015), https: //www.washingtonpost.com/news/the-watch/wp/2015/07/22/a-high-ranking -obama-official-just-called-for-the-eradication-of-bite-mark-evidence/?utm_term=.449f38b65769 [https://perma.cc/F9ES-B79J](quoting Handelsman's remarks presented at the InternationalSymposium on Forensic Science Error Management: Detection, Measurementand Mitigation, Arlington, Virginia (July 20-24, 2015), organized by theNational Institute of Standards and Technology (NIST)).

(4.) Williams v. United States, 130 A.3d 343, 355 (D.C. 2016)(Easterly, J., concurring).

(5.) Editorial, Junk Science at the F.B.I., N.Y. Times (Apr. 27,2015), https: //www.nytimes.com/2015/04/27/opinion/junk-scieiice-at-the-fbi.html [https: //perma.cc/6PH3-8SPL]; see also Eric S. Lander, Fix theFlaws in Forensic Science, N.Y. Times (Apr. 21, 2015),https://www.nytimes.com/2015/04/21/opinion/fix-the-flaws-in-forensic-science.html[https://perma.cc/ 8LPZ-BWHX] ("No expert should be permitted totestify without showing three things: a public database of patterns frommany representative samples; precise and objective criteria fordeclaring matches; and peer-reviewed published studies that validate themethods.").

(6.) 509 U.S. 579 (1993).

(7.) See David L. Faigman, Is Science Different for Lawyers?, 297SCI. 339, 340 (2002) ("Daubert initiated a scientific revolution inthe law.").

(8.) United States v. Barnette, 211 F.3d 803, 815 (4th Cir. 2000);see also United States v. Alatorre, 222 F.3d 1098, 1100 (9th Cir. 2000)("Daubert has become ubiquitous in federal trial courts.").

(9.) 293 F. 1013 (D.C. Cir. 1923).

(10.) Id. at 1014 (stating that a technique "must besufficiently established to have gained general acceptance in theparticular field in which it belongs").

(11.) See Paul C. Giannelli, The Admissibility of Novel ScientificEvidence: Frye v. United States, a Half-Century Later, 80 COLUM. L. REV.1197, 1228 (1980).

(12.) See Michael J. Saks, Merlin and Solomon: Lessons from theLaw's Formative Encounters with Forensic Identification Science, 49HASTINGS L.J. 1069, 1138 (1998) ("Frye does not work because itsmeasure of validity is the judgment of 'the field,' and thefield may consist of nonsense. For example, the Frye doctrine cannotexclude astrology.").

(13.) Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 590 (1993)(emphasis added).

(14.) Id. at 593-94.

(15.) Id.

(16.) 526 U.S. 137 (1999).

(17.) Id. at 141.

(18.) Weisgram v. Marley Co., 528 U.S. 440, 455 (2000).

(19.) After Daubert, the Court decided General Electric Co. v.Joiner, 522 U.S. 136 (1997). Joiner established the standard forappellate review--abuse of discretion--for applying the Daubert factors.Id. at 139. Daubert, Joiner, and Kumho make up what is known as theDaubert Trilogy.

(20.) See 1 PAUL C. GIANNELLI ET AL., SCIENTIFIC EVIDENCE [section]1.11 (5th ed. 2012).

(21.) A few perceptive scholars had noted the lack of empiricaltesting prior to Daubert. See Randolph N. Jonakait, Forensic Science:The Need for Regulation, 4 HARV. J.L. & TECH. 109, 137 (1991)("Forensic science is supported by almost no research. Thelaboratory practices are based on intuitions and deductions, not onempirical proof."); D. Michael Risinger et al., Exorcism ofIgnorance as a Proxy for Rational Knowledge: The Lessons of HandwritingIdentification "Expertise," 137 U. PA. L. REV. 731, 738 (1989)("Our literature search for empirical evaluation of handwritingidentification turned up one primitive and flawed validity study fromnearly 50 years ago, one 1973 paper that raises the issue of consistencyamong examiners but that presents only uncontrolled impressionistic andanecdotal information not qualifying as data in any rigorous sense, anda summary of one study in a 1978 government report. Beyond this,nothing."); Michael J. Saks & Jonathan J. Koehler, What DNA"Fingerprinting" Can Teach the Law About the Rest of ForensicScience, 13 CARDOZO L. REV. 361, 372 (1991) ("[F]orensicscientists, like scientists in all other fields, should subject theirclaims to methodologically rigorous empirical tests. The results ofthese tests should be published and debated. Until such steps are taken,the strong claims of forensic scientists must be regarded with far morecaution than they traditionally have been.").

(22.) Margaret A. Berger, Procedural Paradigms for Applying theDaubert Test, 78 MINN. L. REV. 1345, 1354 (1994) ("Courts neverrequired some of the most venerable branches of forensic science--suchas fingerprinting, ballistics, and handwriting--to demonstrate theirability to make unique identifications. ").

(23.) Paul Giannelli & Edward Imwinkelried, ScientificEvidence: The Fallout from Supreme Court's Decision in Kumho Tires,CRIM. JUST., Winter 2000, at 12, 40. For an insightful analysis of howidentification science was accepted by the courts, see Saks, supra note12.

(24.) D. Michael Risinger, Navigating Expert Reliability: AreCriminal Standards of Certainty Being Left on the Dock?, 64 ALB. L. REV.99, 149 (2000). In addition, an extensive study of reported criminalcases found that "the Daubert decision did not impact on theadmission rates of expert testimony at either the trial or the appellatecourt levels." Jennifer L. Groscup et al., The Effects of Dauberton the Admissibility of Expert Testimony in State and Federal CriminalCases, 8 PSYCHOL. PUB. POL'Y & L. 339, 364 (2002).

(25.) Peter J. Neufeld, The (Near) Irrelevance of Daubert toCriminal Justice and Some Suggestions for Reform, 95 AM. J. PUB. HEALTHS107, S107 (2005).

(26.) COMM. ON IDENTIFYING THE NEEDS OF THE FORENSIC SCI. CMTY.,NAT'L RESEARCH COUNCIL, STRENGTHENING FORENSIC SCIENCE IN THEUNITED STATES: A PATH FORWARD 100 (2009) [hereinafter NAS FORENSICREPORT].

(27.) Id. at 22. At another point, the Report stated: "Thesimple reality is that the interpretation of forensic evidence is notalways based on scientific studies to determine its validity. This is aserious problem." Id. at 8; see also id. at 6 ("Often thereare no standard protocols governing forensic practice in a givendiscipline. And, even when protocols are in place ..., they often arevague and not enforced in any meaningful way.").

(28.) Id. at 144 (noting that research is needed "[t]oproperly underpin the process of friction ridge [fingerprint]identification").

(29.) Id. at 154 ("Sufficient studies [on firearmsidentification] have not been done to understand the reliability andrepeatability of the methods.").

(30.) Id. at 166 ("The scientific basis for handwritingcomparisons needs to be strengthened.").

(31.) Id. at 161 ("[Testimony linking microscopic hairanalysis with particular defendants is highly unreliable.").

(32.) Id. at 174 ("No thorough study has been conducted oflarge populations to establish the uniqueness of bite marks....").

(33.) Id. at 53 (emphasis added).

(34.) Id. at 106.

(35.) Id. at 11.

(36.) See, e.g., Weisgram v. Marley Co., 528 U.S. 440, 455 (2000).A former federal district judge, Nancy Gertner, noted, "a busytrial judge can rely on the decades of case law to legitimize decisionsrejecting a hearing or motions in limine. And the trial judge can counton the Court of Appeals likely concluding that rejecting the challengewas not an abuse of the judge's discretion." Nancy Gertner,Commentary on The Need for a Research Culture in the Forensic Sciences,58 U.C.L.A. L. REV. 789, 790 (2011).

(37.) In 2013, the Department of Justice ("DOJ"), inpartnership with the National Institute of Standards and Technology("NIST"), established the National Commission on ForensicScience to enhance the practice and improve the reliability of forensicscience. The author served on the Commission. NAT'L COMM'N ONFORENSIC SCI., U.S. DOJ, REFLECTING BACK--LOOKING TOWARD THE FUTURE 1,app. A at 1 (2017).

(38.) PRESIDENT'S COUNCIL OF ADVISORS ON SCI. & TECH.,REPORT TO THE PRESIDENT: FORENSIC SCIENCE IN CRIMINAL COURTS: ENSURINGSCIENTIFIC VALIDITY OF FEATURE-COMPARISON METHODS (2016) [hereinafterWHITE HOUSE PCAST REPORT].

(39.) See WILLIAM THOMPSON ET AL., AM. ASS'N FOR THEADVANCEMENT OF SCI., FORENSIC SCIENCE ASSESSMENTS: A QUALITY AND GAPANALYSIS: LATENT FINGERPRINT ANALYSIS 7-8, 43-44 (2017) [hereinafterAAAS FINGERPRINT REPORT] (discussing cognitive bias, recommendingremedies, and arguing for the need of scientific validation studies).

(40.) See JOSE ALMIRALL ET AL., AM. ASS'N FOR THE ADVANCEMENTOF SCI., FORENSIC SCIENCE ASSESSMENTS: A QUALITY AND GAP ANALYSIS: FIREINVESTIGATION 8-9 (2017) [hereinafter AAAS FIRE REPORT] (discussingcognitive bias and recommending remedies).

(41.) See 1 GIANNELLI ET AL., supra note 20, [section] 13.05(discussing the admissibility of bite mark evidence). In Doyle v. State,263 S.W.2d 779, 779 (Tex. Crim. App. 1954), a bite mark was left in apiece of cheese in a burglary case. Two decades later, in Patterson v.State, 509 S.W.2d 857, 861 (Tex. Crim. App. 1974), a prosecution expertmatched the defendant's teeth to a mark found on a murder victim.

(42.) See, e.g., Tucker Carrington, Mississippi Innocence: TheConvictions and Exonerations of Levon Brooks and Kennedy Brewer and theFailure of the American Promise, 28 Geo. J. LEGAL ETHICS 123, 134-35,142-45, 149-50 (2015).

(43.) See, e.g., People v. Marsh, 441 N.W.2d 33, 35 (Mich. Ct. App.1989) ("[T]he science of bite-mark analysis has been extensivelyreviewed in other jurisdictions.").

(44.) See, e.g., State v. Sager, 600 S.W.2d 541, 569 (Mo. Ct. App.1980).

(45.) See State v. Richards, 804 P.2d 109, 112 (Ariz. 1990)("[B]ite mark evidence is admissible without a preliminarydetermination of reliability. ..."); People v. Middleton, 429N.E.2d 100, 101 (N.Y. 1981) ("The reliability of bite mark evidenceas a means of identification is sufficiently established in thescientific community to make such evidence admissible in a criminalcase, without separately establishing scientific reliability in eachcase. ..."); State v. Armstrong, 369 S.E.2d 870, 877 (W. Va. 1988)(holding that trial court may take judicial notice of generalreliability of bite mark evidence).

(46.) See FED. R. EVID. 201(b) (limiting judicial notice to a"fact that is not subject to reasonable dispute").

(47.) I GIANNELLI ET AL., supra note 20, [section] 13.02(discussing the admissibility of dental identifications).

(48.) See People v. Milone, 356 N.E.2d 1350, 1358 (111. App. Ct.1976) ("The concept of identifying a suspect by matching hisdentition to a bite mark found at the scene of a crime is a logicalextension of the accepted principle that each person's dentition isunique."); People v. Smith, 443 N.Y.S.2d 551, 556-57 (Cty. Ct.1981) ("The basic premise is the unique nature of individualdentition and the virtually infinite number of individual biteconfigurations.") (citations omitted).

(49.) "Restorations alone, with varying shapes, sizes, andrestorative materials, may offer numerous points for comparison. Inaddition to restorations, the number of teeth, prostheses, decay,malposition, malrotation, peculiar shapes, root canal therapy, bonepatterns, bite relationship, and oral pathology may all provideidentifying characteristics." I GLANNELLI ET AL., supra note 20,[section] 13.02.

(50.) See I.A. Pretty & D. Sweet, The Scientific Basis forHuman Bitemark Analyses--A Critical Review, 41 Sci. & Just. 85, 87(2001) ("Skin is a poor registration material since it is highlyvariable in terms of anatomical location, underlying musculature or fat,curvature, and looseness or adherence to underlying tissues. Skin ishighly visco-elastic, which allows stretching to occur during either thebiting process or when evidence is collected.").

One study classified different types of distortion: Primarydistortion occurs at the time of biting and results (1) from thedynamics of the biting process (dynamic distortion) and (2) from thefeatures of the tissue bitten (tissue distortion). Secondary distortionoccurs at a subsequent time. It can be subdivided into three categories.The first is time-related distortion, e.g., caused by subsequent healingor decomposition. "Posture distortion results when the bite mark isviewed or recorded in a position that differs from the position at thetime of biting." Photographic distortion results from the angle ofthe camera and the curvature of the body. D.R. Sheasby & D.G.MacDonald, A Forensic Classification of Distortion in Human Bite Marks,122 FORENSIC SCI. INT'L 75, 75-77 (2001).

(51.) See Saks, supra note 12, at 1120 ("[R]ather than thefield convincing the courts of the sufficiency of its knowledge andskills, admission by the courts apparently convinced the forensicodontology community that, despite their doubts, they really were ableto perform bite mark identifications.").

(52.) Dr. Mary Bush and her colleagues at the Laboratory forForensic Odontology, State University of New York at Buffalo, havepublished over a dozen studies that have undermined the assumptionsunderpinning bite mark evidence. E.g., Mary A. Bush et al., StatisticalEvidence for the Similarity of the Human Dentition, 56 J. FORENSIC SCIS.118, 122 (2011) ("Our results show that given our measurementparameters, statements concerning dental uniqueness with respect tobitemark analysis in an open population are unsupportable. ...Confidence in the notion of dental uniqueness in bitemark analysis hasbeen based on anecdotal knowledge, the use of inappropriate statistics,and precedence of admission in the courtroom."). Bush and her teamalso reported the results of a study where twenty-three bites were madein cadaver skin with the same dentition using an instrumented-bitingmachine. The cadavers were moved and re-photographed in differentpositions. Subsequent measurements showed differences between all bitemarks. In addition, postural distortion was significant. See Mary A.Bush et al., Biomechanical Factors in Human Dermal Bitemarks in aCadaver Model, 54 J. Forensic Scis. 167, 169-170, 174 (2009). One surveyof fifteen odontologists involved their opinions of six images ofsupposed bite marks. The "practitioner agreement was at best fair,with wide-ranging opinions on the origin, circ*mstance, andcharacteristics of the wound given for all six images." Mark Pageet al., Expert Interpretation of Bitemark Injuries--A ContemporaryQualitative Study, (58) J. Forensic Scis. 664, 664 (2013).

(53.) NAS FORENSIC REPORT, supra note 26, at 175.

(54.) 9 N.E.3d 1072 (Ohio Ct. App. 2014).

(55.) Id. at 1097.

(56.) 384 S.W.3d 919 (Tex. Crim. App. 2012).

(57.) Id. at 926.

(58.) See, e.g., Prade, 9 N.E.3d at 1098 ("As to Dr.Bush's cadaver studies, Dr. Wright testified that cadaver skinsimply cannot compare with living skin. Dr. Wright explained thatcadaver skin only distorts after a bite for two to three minutes at mostbecause, unlike live skin, no bruising, contusions, or lacerationsoccur. Dr. Wright also testified that using a mechanical jaw to bite isproblematic because the jaw operates on a fixed hinge that cannot mimicthe wider range of movement that an actual jaw is capable of.").But see Iain A. Pretty & David Sweet, A Paradigm Shift in theAnalysis of Bitemarks, 201 Forensic Sci. Int'l 38, 40 (2010)(noting that, while cadaver models have limitations, "there islittle alternative for researchers to produce bitemarks of knownorigin" and the use of anesthetized pigs to create peri-morteminjuries raises a different issue--i.e., differences between pigskin andhuman skin).

(59.) See Radley Balko, In Angry, Defensive Memo, ManhattanDA's Office Withdraws Bite Mark Evidence, Wash. Post (Jan. 13,2016), https:// www.washingtonpost.com/news/the-watch/wp/2016/01/13/in-angry-defens ive-memo-manhattan-das-office-withdraws-bite-mark-evidence/?utm_term=.48e8ac4cc71e [https://perma.cc/CMD7-6GM4]; Radley Balko,Attack of the Bite Mark Matchers, Wash. Post (Feb. 18, 2015),https://www. washingtonpost.com/news/the-watch/wp/2015/02/18/attack-of-the-bitemark-matchers-2/?utm_term=.3455206afa8d[https://perma.cc/T3R3T7ZH]; Radley Balko, The Path Forward on Bite MarkMatching--and the Rearview Mirror, Wash. Post (Feb. 20, 2015),https://www. washingtonpost.com/news / the-watch /wp/2015/02/20/the-path-forwardon-bite-mark-matching-and-the-rearview-mirror/?utm_term=.41c6a22faa7d[https://perma.cc/9GF W-NAB V].

(60.) The study is known as Construct Validity Bitemark AssessmentsUsing the ABFO Bitemark Decision Tree ("Freeman/PrettyStudy"). See AM. ACAD, OF FORENSIC SCIS., ADVANCE PROGRAM 67THANNUAL SCIENTIFIC MEETING: CELEBRATING THE FORENSIC SCIENCE FAMILY 175(2015).

(61.) Radley Balko, A Bite Mark Matching Advocacy Group JustConducted a Study that Discredits Bite Mark Evidence, WASH. POST (Apr.8, 2015), https://www.washingtonpost.com/news/the-watch/wp/2015/04/08/a-bite mark-matching-advocacy-group-just-conducted-a-study-that-discredits-bite mark-evidence/?utm_term=.b752ad99e635 [https://perma.cc/EB2B-CDYD],

(62.) Id.

(63.) Id.

(64.) Id.

(65.) Id.

(66.) Id.

(67.) Id.

(68.) Amanda Lee Myers, Once Key in Some Cases, Bite-Mark EvidenceNow Derided as Unreliable, Yahoo (June 16, 2013), https://www.yahoo.com/news/ap-impact-bites-derided-unreliable-court-150004412.html [https://perma.cc/875Q-9RB6].

(69.) Sarah Kaplan, Texas Inmate's 1989 Conviction OverturnedAfter Bite Mark Evidence Discredited, Wash. Post (Oct. 13, 2015),https://www. washingtonpost.com/news/morning-mix/wp/2015/10/13/texas-mans-convict ion-overturned-after-bite-mark-evidence-discredited/?utm_term=.d3513f42c6 99 [https://perma.cc/T6UX-XETE].

(70.) Texas Forensic Science Commission Steps Up to InvestigateBite Mark Analysis, INNOCENCE PROJECT (Dec. 14, 2015),https://www.innocenceproject.org/texas-forensic-science-commission-steps-up-to-investigate-bitemark-analysis/ [https://perma.cc/BC95-HGB8]. Texas created the TexasForensic Science Commission (TFSC) in 2005 after a scandal requiredHouston to close its crime lab. Michael Hall, False Impressions, Tex.MONTHLY (Jan. 2016),https://www.texasmonthly.com/articles/falseimpressions/[https://perma.cc/6JJR-GA9Y]. Accordingly, TEX. CODE CRIM. PROC. ANN.art. 38.01, [section] 4(a)(3) (Supp. 2015), provides that the Commissionshould "investigate, in a timely manner, any allegation ofprofessional negligence or misconduct that would substantially affectthe integrity of the results of a forensic analysis conducted by a crimelaboratory."

(71.) TEX. FORENSIC SCI. COMM'N, FORENSIC BITEMARK COMPARISONCOMPLAINT FILED BY NATIONAL INNOCENCE PROJECT ON BEHALF OF STEVEN MARKCHANEY--FINAL REPORT 11-12 (2016).

(72.) Id. at 12.

(73.) Id. at 13

(74.) Id.; see also Brandi Grissom, Arguments Over Bite Marks GetTesty at Texas Forensic Science Commission Meeting, DALLAS MORNING NEWS(Nov. 17, 2015), https://www.dallasnews.com/news/politics/2015/ll/17/argumentsover-bite-marks-get-testy-at-forensic-science-commission-meeting [https://perma.cc/V536-3RB8] (reporting on a 2015 TFSC meetingdiscussing bitemark evidence).

(75.) See generally WHITE HOUSE PCAST REPORT, supra note 38.

(76.) Id. at 9. "PCAST finds that bitemark analysis is farfrom meeting the scientific standards for foundational validity."Id.

(77.) See, e.g., Ege v. Yukins, 380 F. Supp. 2d 852, 878 (E.D.Mich. 2005) ("[T]he defense attempted to rebut Dr. Warnick'stestimony with the testimony of other experts who opined that the markon the victim's cheek was the result of livor mortis and was not abite mark at all."); Czapleski v. Woodward, No. C-90-0847 MHP, 1991WL 639360, at *l-2 (N.D. Cal. Aug. 30, 1991) (noting that, while adentist's initial report concluded that "bite" marksfound on child were consistent with dental impressions of mother,several experts later established that the marks on child's bodywere postmortem abrasion marks and not bite marks); Kinney v. State, 868S.W.2d 463, 46465 (Ark. 1994) (noting disagreement between expertwitnesses about whether injuries were from human bite marks); People v.Noguera, 842 P.2d 1160, 1165 n.l (Cal. 1992) ("At trial, extensivetestimony by forensic ondontologists [sic] was presented by both sides,pro and con, as to whether the wounds were human bite marks and, if so,when they were inflicted."); State v. Duncan, 802 So. 2d 533, 553(La. 2001) ("Both defense experts testified that these marks on thevictim's body were not bite marks."); Stubbs v. State, 845 So.2d 656, 668 (Miss. 2003) ("Dr. Galvez denied the impressions foundon Williams were the results of bite marks.").

(78.) See Michael J. Saks et al., Forensic Bitemark Identification:Weak Foundations, Exaggerated Claims, 3 J.L. &; biosciences 1, 29(2016) ("[R]ecent reviews of the field's claims, as well asrecent empirical findings, have underscored the lack of reliability andvalidity of the most fundamental claims about the ability of forensicdentists to identify the source of bite marks on human skin.").

(79.) M. Chris Fabricant & Tucker Carrington, The ShiftedParadigm: Forensic Science's Overdue Evolution from Magic to Law, 4Va. J. CRIM. L. 1, 38 (2016).

(80.) In Commonwealth v. Ross, No. CP-07-CR 2038-2004, slip op. at5 (Ct. Com. pi. Blair Cty., Pa., filed Mar. 8, 2017), the court admittedbite mark evidence, albeit limited, noting that " [t]heCommonwealth notes that no state or federal court has suppressed experttestimony in a criminal case based upon the NAS Report, and that nocourts have prohibited bite mark evidence based upon the PCAST and TFSCreports." See also Radley Balko, Incredibly, Prosecutors Are StillDefending Bite Mark Evidence, WASH. POST (Jan 30, 2017),https://www.washingtonpost.com/news/thewatchIwp/2017/01/30/incredibly-prosecutors-are-still-defending-bite-markevidence/ [https://perma.cc/V6TF-HTU9].

(81.) See NAT'L COMM'N ON FORENSIC SCI., U.S. DOJ,MEETING #13, at 6163 (2017) [hereinafter Meeting #13]; Frank Green, DNAProves Man Innocent of 1982 Rape and Murder in Famous'Bite-Mark' Case, Lawyers Say, RICHMOND TIMES-DISPATCH (Mar.12, 2016), http://www.richmond.com/news/dna-proves-man-innocent-of-rape-and-murder-in-famous / article_05ab68ce-064c-58bb-b57a-211e2bb51ecd.html [https://perma.cc/WUW43R57](discussing the Keith Harward case); Spencer S. Hsu, Va. ExonerationUnderscores Mounting Challenges to Bite-Mark Evidence, Wash. Post (Apr.8, 2016), https://www.washingtonpost.com/local/public-safety/vaexoneration-underscores-to-mounting-challenges-to-bite-mark-evidence/2016/04/08/55bbfe98-fd9a-11e5-886f-a037dba38301_story.html?utm__term=.262f4d5c302d [https://perma.cc/263G-QK5E] (same).

(82.) Meeting #13, supra note 81, at 111; e.g., Pema Levy,Sessions' New Forensic Science Adviser Has a History of OpposingPro-Science Reforms, Mother Jones (Aug. 10, 2017),http://www.motherjones.com/crimejustice/2017/08/sessions-new-forensic-science-adviser-has-a-history-of-opposing-pro-science-reforms/ [https://perma.cc/R3AR-DJSQ].

(83.) See generally 2 GIANNELLI ET AL., supra note 20, [section]24.02[l] (discussing the techniques used to identify the human source ofa hair sample, including conventional microscopy).

(84.) As one hair examiner wrote, "[i]f a pubic hair from thescene of a crime is found to be similar to those from a known source,[the courts] do not know whether the chances that it could haveoriginated from another source are one in two or one in a billion."B.D. Gaudette, Probabilities and Human Pubic Hair Comparisons, 21 J.FORENSIC SCI. 514, 514 (1976).

(85.) Professor Berger explained the problem:

 We allow eyewitnesses to testify that the person fleeing the scene wore a yellow jacket and permit proof that a defendant owned a yellow jacket without establishing the background rate of yellow jackets in the community. Jurors understand, however, that others than the accused own yellow jackets. When experts testify about samples matching in every respect, the jurors may be oblivious to the probability concerns if no background rate is offered, or may be unduly prejudiced or confused if the probability of a match is confused with the probability of guilt, or if a background rate is offered that does not have an adequate scientific foundation.

Berger, supra note 22, at 1357.

(86.) EDWARD CONNORS ET AL., CONVICTED BY JURIES, EXONERATED BYSCIENCE: CASE STUDIES IN THE USE OF DNA EVIDENCE TO ESTABLISH INNOCENCEAFTER TRIAL 58 (1996).

(87.) Id. at 59.

(88.) Williamson v. Reynolds, 904 F. Supp. 1529, 1554 (E.D. Okla.1995) (emphasis added), rev'd sub nom. Williamson v. Ward, 110 F.3d1508, 1523 (10th Cir. 1997) (holding that due process--notDaubert--controls in federal habeas review).

(89.) See Edward J. Imwinkelried, Forensic Hair Analysis: The CaseAgainst the Underemployment of Scientific Evidence, 39 WASH. & LEEL. REV. 41, 62 (1982) (stating that "[t]he massive body of caselaw, liberally admitting even hair evidence of low probative value,dwarfs the handful of cases excluding hair evidence").

(90.) See Clive A. Stafford Smith & Patrick D. Goodman,Forensic Hair Comparison Analysis: Nineteenth Century Science orTwentieth Century Snake Oil?, 27 COLUM. HUM. RTS. L. REV. 227, 231(1996) ("If the purveyors of this dubious science cannot do abetter job of validating hair analysis than they have done so far,forensic hair comparison analysis should be excluded altogether fromcriminal trials.").

(91.) 904 F. Supp. 1529 (E.D. Okla. 1995), rev'd sub nom.Williamson v. Ward, 110 F.3d 1508, 1523 (10th Cir. 1997).

(92.) Id. at 1558.

(93.) Id. at 1554.

(94.) See JIM DWYER ET AL., ACTUAL INNOCENCE: FIVE DAYS TOEXECUTION AND OTHER DISPATCHES FROM THE WRONGLY CONVICTED 146 (2000)(noting that the hair evidence was shown to be "patentlyunreliable"). See also JOHN GRISHAM, THE INNOCENT MAN: MURDER ANDINJUSTICE IN A SMALL TOWN 166-87 (2006) (examining Williamson'strial, including the role played by hair analysis).

(95.) 12 S.W.3d 258 (Ky. 1999).

(96.) Id. at 263.

(97.) See FED. R. EVID. 201(b) (limiting judicial notice to a"fact that is not subject to reasonable dispute").

(98.) See 2 GIANNELLI ET AL., supra note 20, [section] 24.03, at825 (noting the "limited impact of Daubert").

(99.) State v. West, 877 A.2d 787, 808 (Conn. 2005).

(100.) In 1998, a Canadian judicial inquiry into the wrongfulconviction of Guy Paul Morin was released. Morin's originalconviction was based, in part, on hair evidence. The judge conductingthe inquiry recommended that "[t]rial judges should undertake amore critical analysis of the admissibility of hair comparison evidenceas circ*mstantial evidence of guilt." FRED KAUFMAN, MINISTRY OF THEATTORNEY GENERAL, REPORT OF THE KAUFMAN COMMISSION ON PROCEEDINGSINVOLVING GUY PAUL MORIN 312 (1998); see also EDWARD CONNORS ET AL.,supra note 86, at 3334-76 (discussing cases in which hair samples playeda role in convicting defendants who were later exonerated by DNAevidence).

(101.) See Brandon L. Garrett, Judging Innocence, 108 COLUM. L.REV. 55, 81 (2008).

(102.) Brandon L. Garrett & Peter J. Neufeld, Invalid ForensicScience Testimony and Wrongful Convictions, 95 VA. L. REV. 1, 14-15(2009).

(103.) NAS FORENSIC REPORT, supra note 26, at. 161.

(104.) Campbell Robertson, Mississippi Inmate's Bid for DNATests Is Denied with Tuesday Execution Set, N.Y. Times (May 3, 2013),http://www. nytimes.com /2013/05/04/us /dna-tests-rejected-for-inmate-facing-tuesdayexecution.html[https://perma.cc/58JT-VN35]; see also Andrew Cohen, A Ghost ofMississippi: The Willie Manning Capital Case, ATLANTIC (May 2, 2013),https://www.theatlantic.com/national/archive/2013/05/a-ghost-ofmississippi-the-willie-manning-capital-case/275442/[https://perma.cc/Q72 4-2K7U].

(105.) See Campbell Robertson, With Hours to Go, Execution IsPostponed, N.Y. TIMES (May 7, 2013),http://www.nytimes.com/2013/05/08/us/willie-jmanning-granted-stay-of-execution.html [https: //perma.cc/J84G-LPX2] (noting that the DOJ"disavowed] the degree of certainty expressed by F.B.I, forensicexperts at the man's trial").

(106.) See Jack Nicas, Flawed Evidence Under a Microscope: DisputedForensic Techniques Draw Fresh Scrutiny; FBI Says It Is ReviewingThousands of Convictions, WALL ST. J. (July 18, 2013),https://www.wsj.com/articles/SB10001424127887324263404578614161262653152 [https://perma.cc/37WN-MRCZ],

(107.) Spencer S. Hsu, U.S. Reviewing 27 Death Penalty Convictionsfor FBI Forensic Testimony Errors, Wash. Post (July 17, 2013),https://www. washingtonpost.com/local/crime/us-reviewing-27-death-penalty-convictionsfor-fbi-forensic-testimony-errors/2013/07/17/6c75a0a4-bd9b-11e2-89c9-3be8095fe767_story.html?utm_term=.d9ec6013dda8 [https://perma.cc/53Y4C9MS]("[O]n the witness stand, several agents for years went beyond thescience and testified that their hair analysis was a near-certainmatch.").

(108.) See Editorial, FBI Lab Failures Should Lead to Reform, WASH.POST (Apr. 20, 2012),https://www.washingtonpost.com/opinions/fbi-lab-failuresshould-lead-to-reform/2012/04/20/gIQAe61YWT_story. html?utm_term=.13c701fdc3c6[https://perma.cc/637M-HY97] ("Kirk L. Odom was incarcerated for 20years and Donald E. Gates for nearly 30 for crimes they did not commit.Santae A. Tribble spent 28 years behind bars, even though DNA evidencenow shows he almost undoubtedly was not the culprit.").

(109.) Martin Enserink, Evidence on Trial, 351 Sci. 1129, 1129(Mar. 11, 2016), http://science.sciencemag.org/content/sci/351/6278/1128.full.pdf [https:// perma.ee/6LXB-BR7B]. The prosecutor misstated theexpert's testimony.

(110.) white House PCAST Report, supra note 38, at 3; see alsoEditorial, Junk Science at the F.B.I., N.Y. Times (Apr. 27, 2015),https:// www.nytimes.com/2015/04/27/opinion/junk-science-at-the-fbi.html[https: //perma.cc/L5A3-THN2] (noting "a sweeping post-convictionreview of 2,500 cases in which its hair-sample lab reported amatch"); Hugh B. Kaplan, DOJ Examiners Gave Bad Testimony in 90Percent of Hair Comparison Cases, 97 CRIM. L. RPT. 77, 77 (2015)("[T]he review has revealed that FBI examiners made erroneousstatements in 90 percent of the cases that have been re-examined sofar.").

(111.) Nos. 85-5415, 5416, 5418, 5420, 5425, 2016 WL 380123 (Mass.Super. Ct. Jan. 26, 2016).

(112.) Id. at *32. The court also noted:

 [The expert] asserted that the hairs "matched" and showed a "strong association." In discussing the chance that the hair found on the victim's bed came from someone other than Perrot, [the expert] conceded the possibility, adding that during his ten years of experience "it's extremely rare that I will have known hair samples from two different people that I can't tell apart." [The expert] made these statements of confidence, despite being unable to recall at trial the length or diameter of the one hair found on the bed. Id.

(113.) Press Release, Office of Pub. Affairs, DOJ, JusticeDepartment Issues Draft Guidance Regarding Expert Testimony and LabReports in Forensic Science (June 3, 2016),https://www.justice.gov/opa/pr/justice-department-issuesdraft-guidance-regarding-expert-testimony-and- lab-reports-forensic[https://perma.cc/T3BK-5P9Q]. These documents are known as the UniformLanguage for Testimony and Reports. Id.

(114.) DOJ, SUPPORTING DOCUMENTATION FOR DEPARTMENT OF JUSTICEPROPOSED UNIFORM LANGUAGE FOR TESTIMONY AND REPORTS FOR THE FORENSICHAIR EXAMINATION DISCIPLINE 4 (2016) [HEREINAFTER DOJ SUPPORTINGDOCUMENTATION], https: //www.justice.gov/archives/dag/file/877736/download[https://perma.cc/79FV-GCSK].

(115.) WHITE HOUSE PCAST REPORT, supra note 38, at 13.

(116.) Id. DOJ cited Max M. Houck & Bruce Budowle, Correlationof Microscopic and Mitochondrial DNA Hair Comparisons, 47 J. FORENSICSCIS. 964 (2002). DOJ SUPPORTING DOCUMENTATION, supra note 114, at 8.This FBI study used mitochondrial DNA analysis to re-examine samplesfrom previous FBI microscopic hair examination cases. Houck &Budowle, supra, at 964. The PCAST Report did not accept that this studysupported validity and reliability because the study showed that in nineof eighty cases--11 percent--the microscopic examination found the hairindistinguishable but DNA analysis showed that the hairs came fromdifferent individuals. White House PCAST REPORT, supra note 38, at 28.

(117.) See JOHN J. LENTINI, SCIENTIFIC PROTOCOLS FOR FIREINVESTIGATION 471-505 (2d ed. 2013) (discussing myths of arsoninvestigations); Paul C. Giannelli, Junk Science and the Execution of anInnocent Man, 7 N.Y.U. J.L. & Liberty 221, 225 (2013).

(118.) JOHN F. BOUDREAU ET AL., NAT'L INST, OF LAW ENF'T& CRIMINAL JUSTICE, DOJ, ARSON AND ARSON INVESTIGATION: SURVEY ANDASSESSMENT 88 (1977).

(119.) NAT'L FIRE PROT. ASS'N, NFPA 921: GUIDE FOR FIREAND EXPLOSION INVESTIGATIONS, 2017 EDITION 1 (2016) [hereinafter NFPA921], The NFPA promotes fire prevention and safety. Id.; see alsoGiannelli, supra note 117.

(120.) See United States v. Hebshie, 754 F. Supp. 2d 89, 109 n.39(D. Mass. 2010) ("NFPA 921 ... is widely accepted as the standardguide in the field of fire investigation."); Thomas R. May, FirePattern Analysis, Junk Science, Old Wives Tales, and Ipse Dixit:Emerging Forensic 3D Imaging Technologies to the Rescue?, 16 Rich. J.L.& Tech. 1, 5 (2010) (noting that NFPA 921 has "become the defacto national standard for fire scene examination and analysis").

(121.) See Frontline: Death by Fire (PBS television broadcast Oct.19, 2010) (detailing the case of Cameron Todd Willingham): David Grann,Trial by Fire: Did Texas Execute an Innocent Man?, New Yorker (Sept. 7,2009), https://www.newyorker.com/magazine/2009/09/07/trial-by-fire[https:// perma.cc/Z5PU-P7BS] (noting that after Willingham'sexecution, the Innocence Project commissioned a panel of fire expertsthat, after reviewing the evidence supporting the conviction,"concluded that 'each and every one' of the indicators ofarson had been 'scientifically proven to be invalid'");Hall, supra note 70 ("The 893-page report [on the Willingham case],released in April 2011, was anticlimactic for people looking for proofthat Texas had executed an innocent man."); Steve Mills &Maurice Possley, Texas Man Executed on Disproved Forensics: Fire thatKilled His 3 Children Could Have Been Accidental, Chi. Trib. (Dec. 9,2004), http://articles.chicagotribune.com/2004-12-09/news/0412090169_l_cameron-todd-willingham-arson-fire-fire-scene [https://perma.cc/2CLW-25SK] ("[The fireinvestigators] used rules of thumb that have since been shown to befalse. There was no evidence to support a conclusion that the fire wasintentionally set. Just an unsupported opinion.").

(122.) Statement of Facts at vol. XI, 244, State v. Willingham, No.24,467-CR (Dist. Ct. Navarro Cty., Tex. 1992) [hereinafter WillinghamTranscript], aff'd, 897 S.W.2d 351 (Tex. Crim. App. 1995) (enbanc).

(123.) See id. at 224-68 (recording the direct examination ofManuel Vasquez). A second expert's testimony essentially trackedVasquez's. See id. at 156-85 (recording the testimony of DouglasFogg).

(124.) Id. at 248. Vasquez testified that there was "charburning, like, for example, this is the bottom here. It's burneddown here at the bottom. That is an indicator in my investigation of anorigin of fire because it's the lowest part of the fire." Id.at 239; see also Willingham, 897 S.W.2d at 354 ("An expert witnessfor the State testified that the floors, front threshold, and frontconcrete porch were burned, which only occurs when an accelerant hasbeen used to purposely burn these areas. This witness further testifiedthat this igniting of the floors and thresholds is typically employed toimpede firemen in their rescue attempts.").

(125.) Willingham Transcript, supra note 122, at vol. XI, 232.

(126.) Id. at 256 ("So when I found that the floor is hotterthan the ceiling, that's backwards, upside down. It shouldn'tbe like that. The only reason that the floor is hotter is because therewas an accelerant."); see also Giannelli, supra note 117, at 226.

(127.) Giannelli, supra note 117, at 227.

(128.) Id.

(129.) LENTINI, supra note 117, at 77.

(130.) See Willingham Transcript, supra note 122, at vol. XI, 75(testimony of Mary Diane Barbe) ("The windows, the electricitystarted crackling and popping, and the top of the--well, I was facingthe side of the house, and it just blew out. The flames just blew out.... All the windows and the front room was engulfed."); id. at 96(testimony of Brandy Barbe) ("We was running towards the house, meand my mother, we was fixing to go and try to get in, and that'swhen it was an explosion. ..."). Vasquez mentioned flashover in histestimony, but he did not appear to understand its implications. See id.at vol XII, 47-48.

(131.) Paul C. Giannelli & Kimberly Gawel, Arson Evidence, 47CRIM. L. BULL. 1241, 1250-51 (2011) (identifying "flashover"as an alternative reason for "low burning" fires).

(132.) Willingham Transcript, supra note 122, at vol. XI, 244-45("You can see that on the burnt patterns on this puddleconfiguration on Exhibit No. 36. This is a strong indicator of a liquid.... [The sunlight] just lights up the puddle configurations, the burnttrailers, the pour patterns on that floor.").

(133.) Giannelli & Gawel, supra note 131, at 1242-44.

(134.) Id. at 1246.

(135.) "There was fire on the floor. ... He had no injuries onhis feet." Willingham Transcript, supra note 122, at 267.

(136.) Giannelli, supra note 127, at 228.

(137.) Willingham Transcript, supra note 122, at 251.

(138.) "[T]he springs were burned from underneath. Thisindicates there was a fire under this bed because of the burn underneaththe bed." Id. at 241.

(139.) Id. at 255 ("Multiple areas of originindicate--especially if there is no connecting path, that they wereintentionally set by human hands."). There are two problems here.First, there could have been one origin, according to independentexperts. DOUGLAS J. CARPENTER ET AL., REPORT ON THE PEER REVIEW OF THEEXPERT TESTIMONY IN THE CASES OF STATE OF TEXAS V. CAMERON TODDWILLINGHAM AND STATE OF TEXAS V. ERNEST RAY WILLIS 12 (2006). Second,even if the fire scene had shown multiple points of origin, this wouldnot necessarily indicate an intentional fire. Lentini, supra note 117,at -513-14.

(140.) Willingham Transcript, supra note 122, at vol. XI, 248-49.Fire experts reviewing the evidence from Willingham's trial pointedout that "[t]he behavior of concrete in fires, including thedevelopment of various colors, has been extensively studied."CARPENTER ET AL., supra note 139, at 18. These experts concluded thatthere is simply "no scientific basis for Mr. Vasquez'sstatement about the brown discoloration being an indication of thepresence of accelerants." Id.

(141.) Vasquez's testimony also demonstrated othermisconceptions. A common one is that arson fires burn hotter and fasterthan "normal" fires: "You know, [an accelerant] makes thefire hotter. It's not a normal fire." Willingham Transcript,supra note 122, at vol. XI, 249. However, the temperature of burningwood and burning gasoline are nearly identical, so to claim that a fireusing liquid accelerants burns "hotter" than a wood fire iswrong. LENTINI, supra note 117, at 501-02.

(142.) "The pieces of broken window glass on the ledge of thenorth windows to the northeast bedroom disclosed a crazed 'spiderwebbing' condition. This condition is an indication that the fireburned fast and hot." CARPENTER ET AL., supra note 139, at 18(citing Vasquez's written report on the Willingham fire at 4).

(143.) LENTINI, supra note 117, at 478 ("It is unclear whyanyone ever thought that crazing of glass indicated rapidheating.").

(144.) In closing argument, the defense counsel referred to a"dozen samples." Willingham Transcript supra note 122, at vol.XIII, 20.

(145.) Id. at vol. XI, 220-21 (documenting testimony by expertstating there was "no distinguishing characteristic" in thecharcoal lighter fluid from plastic container on porch "that wasnot present" in the front-door threshold sample tested); id. atvol. XIII, 20-21 ("They sent [the samples] to the lab and what didthey find? Nothing, not a trace of anything at all except on the veryfront of the front porch where the charcoal lighter fluid was.");id. at vol. XIII, 45 ("The [accelerant] is gone, except [on] thethreshold; it burned away. ...").

(146.) Id. at vol. XII, 14-15 (noting that, although photographsshow a grill, Vasquez apparently did not know of the grill'spresence); id. at vol. XII, 16 (acknowledging that a fire-damagedcharcoal lighter fluid container was found on the front porch).

(147.) The prosecutor would later say that he '"never didunderstand why they weren't able to recover' positive tests inthese parts." Grann, supra note 121. At trial, he argued that,except in the threshold, the "liquid ... burned away in thatdestructive madness created by Cameron Todd Willingham." WillinghamTranscript, supra note 122, at vol. XIII, 45.

(148.) Report of Dr. Gerald Hurst at 1, Ex parte Willingham, No.24,4670(B), (Dist. Ct. Navarro Cty., Tex., Feb. 13, 2004).

(149.) CARPENTER ET AL., supra note 139, at 3.

(150.) The expert evidence in both cases was comparable, but Williswas lucky. His death penalty conviction was overturned on proceduralgrounds, and the prosecutor subsequently refused to reindict him afterDr. Hurst wrote the same type of critical report in Willis's casethat he had written in Willingham's. Willis, who had spentseventeen years on death row, was subsequently exonerated on actualinnocence grounds. See Mary Alice Robbins, New-York Based InnocenceProject Attacks Texas Arson Convictions, TEX. LAW., May 8, 2006,Factiva, Doc. No. TEXASL0020060508e25800001.

(151.) See Letter from Innocence Project to Tex. Forensic Sci.Comm'n (Aug. 20, 2010) (on file with Case Western Reserve LawReview).

(152.) Craig L. Beyler, Analysis of the Fire Investigation Methodsand Procedures Used in the Criminal Arson Cases Against Ernest RayWillis and Cameron Todd Willingham 51 (Aug. 17, 2009) (unpublishedreport) (on file with Case Western Reserve Law Review).

(153.) See Christy Hoppe, Perry Defends Removal of 3 Before ArsonHearing, DALL. MORNING NEWS (Oct. 2, 2009),https://eji.org/sites/default/files/dpdallasmorning-perry-defends-removal-of-3- before-arson-hearing-10-02-09.pdf [https://perma.cc/DRP9-ZZE6](detailing the removal of the three members of the Texas ForensicScience Commission); Mary Alice Robbins, Fired Up; Changes Sought forTexas Forensic Science Commission at Center of Heated Controversy, Tex.Law., Nov. 9, 2009, Factiva, Doc. No. TEXASL0020091109e5b900002("[Former Commissioner] Levy says he believes 'things wentsouth' for the commission after [former Chair] Bassett releasedBeyler's report to the public in August 'as he was required bylaw to do."'). The meeting was scheduled for October 2, 2009.Agenda, TEX. FORENSIC SCI. COMM'N (Oct. 2, 2009),http://www.fsc.texas.gov/sites/default/files/D_100209MeetingAgenda_000.pdf [https://perma.cc/UC8Z-HQMY],

(154.) Hoppe, supra note 153 (noting that the new chair was"known as one of the toughest law-and-order prosecutors in thestate").

(155.) See Jennifer Emily, Texas Forensic Science CommissionRefuses to End Inquiry into Willingham Arson Case, DALL. MORNING NEWS(Sept. 18, 2010), https://www.dallasnews.com/news/crime/2010/09/18/Texas-Foren sic-Science-Commission-refuses-to-5315[https://perma.cc/3RP5-GSH6] ("Perry's replacements were seenby some as a political maneuver intended to change the outcome of thecommission's decision."); Christy Hoppe, Perry Ousts OfficialsBefore Arson Hearing: He's Assailed as New Chair Delays Session onFlawed Case that Led to Execution, DALL. MORNING NEWS, Oct. 1, 2009, at1A; Dave Mann, Fire and Innocence, TEX. OBSERVER (Dec. 3, 2009),https://www.texasobserver.org/fire-and-innocence/ [https://perma.cc/SB7T-AGYB] ("Then in late September, Perry booted three membersoff of the Texas Forensic Science Commission, which was investigatingthe Willingham and Willis cases, just three days before a crucialhearing on scientists' findings. Perry's new appointeespromptly canceled the hearing and have yet to reschedule it. Evenconservative commentators cried cover-up, suggesting that Perry, in atough battle for re-election, was trying to subvert an investigationthat might prove he oversaw the execution of an innocent man.").

(156.) Letter from Greg Abbott, Attorney Gen. of Tex., to theHonorable Nizam Peerwani, Presiding Officer, Tex. Forensic Sci.Comm'n (July 29, 2011),https://texasattorneygeneral.gov/opinions/opinions/50abbott/op/2011/pdf/ga0866.pdf [https://perma.cc/9SMK-P5MY].

(157.) REPORT OF THE TEXAS FORENSIC SCIENCE COMMISSION:WILLINGHAM/WILLIS INVESTIGATION 39 (2011) [hereinafter TFSC Report].

(158.) Id. at 22-28.

(159.) Id. at 36.

(160.) Id. at 48.

(161.) Letter from Paul Maldonado, State Fire Marshal, to LeighTomlin, Comm'n Coordinator, Tex. Forensic Sci. Comm'n (Aug.20, 2010) (on file with Case Western Reserve Law Review) (emphasisadded).

(162.) TFSC REPORT, supra note 157, at 16 ("This appears to bean untenable position in light of advances in fire science. The fires inthese cases occurred two decades ago; there are few circ*mstances inwhich an investigation could not be improved with the benefit of twentyyears of controlled scientific experiment and practicalexperience.").

(163.) Lee petitioned for a writ of habeas corpus in 2010, based inpart on "inaccurate and unreliable evidence." Lee v. Tennis,No. 4:CV-08-1972, 2010 WL 3812160, at *2 (M.D. Pa. Sept. 22, 2010).Although the district court denied Lee's petition, the ThirdCircuit reversed and remanded the case to the district court. Lee v.Glunt, 667 F.3d 397, 407-08 (3d Cir. 2012) ("If Lee'sexpert's independent analysis of the fire scene evidence--applyingprinciples from new developments in fire science--shows that the fireexpert testimony at Lee's trial was fundamentally unreliable, thenLee will be entitled to federal habeas relief on his due processclaim.").

(164.) Lee v. Tennis, Civil No. 4:08-CV-1972, 2014 WL 3894306, at*2 (M.D. Pa. June 13, 2014), adopted, 2014 WL 3900230 (M.D. Pa. Aug. 8,2014), aff'd sub nom. Lee v. Superintendent Houtzdale SCI, 798 F.3d159 (3d Cir. 2015).

(165.) Id.

(166.) Mark Hansen, Badly Burned: Long-Held Beliefs About ArsonScience Have Been Debunked After Decades of Misuse and Scores ofWrongful Convictions, A.B.A. J., Dec. 2015, at 37, 37, 43.

(167.) Id. at 40.

(168.) 620 F.3d 571 (6th Cir. 2010). See generally Marc Price Wolf,Habeas Relief from Bad Science: Does Federal Habeas Corpus ProvideRelief for Prisoners Possibly Convicted on Misunderstood Fire Science?,10 MINN. J.L. SCI. & Tech. 213 (2009) (detailing the shift of theprinciples in the field of fire investigation and how past convictionsmay use habeas relief).

(169.) Id. at 574.

(170.) Id. at 580.

(171.) 620 F.3d at 580, 582-83 (Merritt, J., dissenting).

(172.) Id. at 581 (quoting transcript).

(173.) Id.

(174.) Id.

(175.) Id. at 580.

(176.) NFPA 921, supra note 119, [section] 17.5.4.7 (describing therole of canine investigations as "assisting with the location andcollection of samples" for laboratory testing).

(177.) Babick, 620 F.3d at 580.

(178.) Id. at 581. See also United States v. Myers, No. 3:10-00039,2010 WL 2723196, at *3-4 (S.D.W. Va. July 8, 2010) (granting motion inlimine to prohibit expert testimony of a canine handler because thealert had not been confirmed by lab testing, conflicted with the FireGuide, and did not meet the Daubert standards).

(179.) United States v. Hebshie, 754 F. Supp. 2d 89, 128 (D. Mass.2010).

(180.) Id. at 124 (quoting Strickland v. Washington, 466 U.S. 668,694 (1984)).

(181.) Id. at 102; see also Michael E. Kurz et al., Effect ofBackground Interference on Accelerant Detection by Canines, 41 J.Forensic Scis. 868, 872 (1996) (noting the varying levels of reliabilityin accelerant detection depending on the substance in question and thecanine handler); Farm Bureau Mut. Ins. Co. of Ark. v. Foote, 14 S.W.3d512, 518, 520 (Ark. 2000) (affirming the trial court's exclusion ofa canine handler who sought to testify about "the alleged superiorability of his canine partner, Benjamin, to detect the presence ofaccelerants after a fire ... [that he could] discriminate betweendifferent types of chemicals," and that he had an "accuracyrate of 100 percent").

(182.) Hebshie, 754 F. Supp. 2d at 93-94.

(183.) Id. at 120.

(184.) Id. at 94.

(185.) Id.

(186.) Id. at 93, 96-97.

(187.) 2 GIANNELLI ET AL., supra note 20, [section] 26.07[b], at1102-03 ("Many appellate courts continue to routinely acceptinvestigators' testimony about experientially-basedgeneralizations. ...").

(188.) See, e.g., State v. Allen, No. 22835, 2009 WL 2096295,[paragraph] 114 (Ohio Ct. App. July 17, 2009) (noting that investigatortestified to "an irregular burn pattern on the floor which throughall my experience and training it appears to be an irregular pourpatterns [sic], an ignitable liquid pour pattern"); State v. Wolf,891 N.E.2d 358, 360 (Ohio Ct. App. 2008) (noting that a firefightertestified "he observed 'pour patterns' located on thefloor throughout the mobile home; that pour patterns are burnt marksthat look like puddles that result from ignitable liquids ... beingpoured out of containers ..."); Colburn v. State, 990 So. 2d 206,209-10 (Miss. Ct. App. 2008) ("This pour pattern, [the fireinvestigator] explained, was indicative of flammable liquid being pouredin the area. ... On cross-examination [the fire investigator] did admitthat the State Crime Laboratory was unable to identify ignitable liquidsin the three debris samples taken from the pour pattern area.");State v. Henderson, 125 P.3d 1132, 1137 (Mont. 2005) (finding that thetrial court "did not err in allowing [a firefighter] to identify inthe photographs and diagrams the pour patterns he had observed at thescene").

(189.) Simon v. State, 633 So. 2d 407, 409 (Miss. 1993), vacated,513 U.S. 956 (1994) (mem.).

(190.) See, e.g., State v. Amodio, 915 A.2d 569, 576 (N.J. Super.Ct. App. Div. 2007) ("They washed the floor and observed areas ofspalling in the concrete underneath the door. This was an indicationthat a flammable liquid had been employed in that area."); McCordv. Gulf Guar. Life Ins. Co., 698 So. 2d 89, 95 (Miss. 1997) ("Thearson investigator ... testified that he found five different areas ofspalling and concluded arson to be the cause of the fire.").

(191.) See, e.g., Wise v. State, 719 N.E.2d 1192, 1200 (Ind. 1999)(noting that a fire investigator testified that a fire was intentionallyset based on several factors, including that "the fire burned toofast for its fuel load"); Carter v. State, 516 S.E.2d 556, 560 (Ga.Ct. App. 1999) ("[The arson investigator] deduced there must havebeen an accelerant or some kind of extra fuel load.").

(192.) See, e.g., People v. Klait, No. 289522, 2010 WL 2076956, at*5 (Mich. Ct. App. May 25, 2010) ("[B]oth [investigators] testifiedthat they believed, based on the fast and hot nature of the fire, thatit was set intentionally."); State v. Walters, 813 P.2d 857, 858(Idaho 1990) (noting that a fire investigator testified that "itwas a hot, fast fire as opposed to a small or as opposed to a slow,smoldering fire, yes, the evidence suggests to me that it wasdeliberately set"); State v. Cutlip, No. 99-L-149, 2001 WL 687493,at *2 (Ohio Ct. App. 2001) (noting that a fire department lieutenanttestified to a list of factors including that "the fire was fastand hot" and "that such observations are typical of a firestarted by someone pouring an accelerant and lighting it").

(193.) 140 F.3d 915 (11th Cir. 1998).

(194.) John J. Lentini, The Evolution of Fire Investigations andIts Impact on Arson Cases, 27 Crim. Just. 12, 14 (2012).

(195.) Benfield, 140 F.3d at 920.

(196.) See e.g., Fireman's Fund Ins. Co. v. Canon U.S.A.,Inc., 394 F.3d 1054, 1058 (8th Cir. 2005) (finding the district courtdid not abuse its discretion where the court concluded that evidentiarysupport of arson theory advanced by experts was inadequate because theydid not examine their theory "against empirical data obtained fromfire scene analysis and appropriate testing," in violation of NFPA921); Ind. Ins. Co. v. Gen. Elee. Co., 326 F. Supp. 2d 844, 850-51 (N.D.Ohio 2004) (holding that cause-and-origin expert's failure toproperly collect evidence violated NFPA 921).

(197.) Douglas Starr, Up in Smoke, DISCOVER, Nov. 2011, at 36, 37.

(198.) Hansen, supra note 166, at 42-43.

(199.) For a fuller discussion of the Cameron Todd Willingham case,see Giannelli, supra note 117.

(200.) See Steve Mills, Convicted Murderer Hopes Latest FireScience Proves Innocence, Chi. Trib. (May 18, 2015),http://www.chicagotribune.com/news/ct-arson-science-adam-gray-met-20150518-story.html [https://perma.cc/B4XF-FMTN].

(201.) See generally Erik Randich & Patrick M. Grant, ProperAssessment of the JFK Assassination Bullet Lead Evidence fromMetallurgical and Statistical Perspectives, 51 J. Forensic Sci. 717(2006) (discussing the original analysis of the bullet fragments); PaulC. Giannelli, Daubert and Forensic Science: The Pitfalls of LawEnforcement Control of Scientific Research, 2011 U. ILL. L. REV. 53, 81(2011).

(202.) See Wilkerson v. State, 776 A.2d 685, 689 (Md. Ct. Spec.App. 2001).

(203.) See State v. Krummacher, 523 P.2d 1009, 1012-13 (Or. 1974)(en banc).

(204.) See United States v. Davis, 103 F.3d 660, 673-74 (8th Cir.1996); People v. Lane, 628 N.E.2d 682, 689-90 (111. App. Ct. 1993).

(205.) See State v. Strain, 885 P.2d 810, 817 (Utah Ct. App. 1994);Jones v. State, 425 N.E.2d 128, 131 (Ind. 1981).

(206.) See State v. Grube, 883 P.2d 1069, 1078 (Idaho 1994); Peoplev. Johnson, 499 N.E.2d 1355, 1366 (111. 1986).

(207.) See State v. Reynolds, 297 S.E.2d 532, 534 (N.C. 1982).

(208.) See Bryan v. State, 935 P.2d 338, 360 (Okla. Crim. App.1997).

(209.) See Davis, 103 F.3d at 666 ("An expert testified thatsuch a finding is rare and that the bullets must have come from the samebox or from another box that would have been made by the same company onthe same day."); Commonwealth v. Daye, 587 N.E.2d 194, 207 (Mass.1992); State v. King, 546 S.E.2d 575, 584 (N.C. 2001) ("[Theexpert] opined that, based on her lead analysis, the bullets sheexamined either came from the same box of cartridges or came fromdifferent boxes of the same caliber, manufactured at the sametime.").

(210.) Brown v. State, 601 P.2d 221, 224 (Alaska 1979) (emphasisadded).

(211.) Earhart v. State, 823 S.W.2d 607, 614 (Tex. Crim. App. 1991)(en banc); see also Giannelli, supra note 201, at 83 n.200.

(212.) Davis, 103 F.3d at 666.

(213.) Id. at 667.

(214.) People v. Villarta, No. H021354, 2002 WL 66887, at *6 (Cal.Ct. App. Jan, 17, 2002). In later years, the testimony became morelimited. A 2002 FBI publication states the conclusion as follows:"Therefore, they likely originated from the samemanufacturer's source (melt) of lead." Charles A. Peters, TheBasis for Compositional Bullet Lead Comparisons, FORENSIC SCI.COMMC'NS (July 2002) (emphasis added),https://archives.fbi.gov/archives/about-us/lab/forensic-science-communications/fsc/july2002/peters.htm[https ://perma.cc/889Y-4TGL]. Testimony to the same effect has alsobeen proffered. Transcript of Record at 6, Commonwealth v. Wilcox, No.00CR2727 (Ky. Cir. Ct. Jefferson County Feb. 28, 2002) (trial testimonyof Charles Peters, FBI examiner) ("Well, bullets that areanalytically indistinguishable likely come from the same molten leadsources of lead, uh, as opposed to bullets that have differentcomposition come from different, uh, melts of lead.").

(215.) See Edward J. Imwinkelried & William A. Tobin,Comparative Bullet Lead Analysis (CBLA) Evidence: Valid Inference orIpse Dixit?, 28 OKLA. CITY U. L. REV. 43 (2003); Erik Randich, WayneDuefeldt, Wade McLendon & William Tobin, A Metallurgical Review ofthe Interpretation of Bullet Lead Compositional Analysis, 127 ForensicSci. Int'l 174 (2002); William A. Tobin & Wayne Duerfeldt, HowProbative Is Comparative Bullet Lead Analysis?, Crim. Just., Fall 2002,at 26; Paul C. Giannelli, Comparative Bullet Lead Analysis: ARetrospective, 47 CRIM. L. BULL. 306 (2010).

(216.) E.g., Ragland v. Commonwealth, 191 S.W.3d 569, 577 (Ky.2006); Clemons v. State, 896 A.2d 1059, 1068-70 (Md. 2006); State v.Behn, 868 A.2d 329, 339-42 (N.J. Super. Ct. App. Div. 2005) (quotingTobin's affidavit submitted with motion for reconsideration).

(217.) COMM. ON SCI. ASSESSMENT OF BULLET LEAD ELEMENTALCOMPOSITION COMPARISON, NAT'L RESEARCH COUNCIL OF THE NAT'LACADS., FORENSIC ANALYSIS: WEIGHING BULLET LEAD EVIDENCE 7 (2004)[hereinafter NAS Report], The author served on the NAS Committee.

(218.) 823 S.W.2d 607, 614 (Tex. Crim. App. 1991) (en banc)("[The expert] concluded that the likelihood that two .22 caliberbullets came from the same batch, based on all the .22 bullets made inone year, is approximately .000025 percent, 'give or take azero.' He subsequently acknowledged, however, that the numberswhich he used to reach the .000025 percent statistic failed to take intoaccount that there are different types of .22 caliber bullets made eachyear--.22, .22 long, and .22 long rifle. [The expert] ultimatelytestified that there could be several hundred thousand bullets perbatch, but with some variation in the elemental composition within thebatch.").

(219.) See Earhart v. Johnson, 132 F.3d 1062, 1067 (5th Cir. 1998)("Given the significant role the bullet evidence played in theprosecution's case, we shall therefore assume Earhart could havemade a sufficient threshold showing that he was entitled to a defenseexpert under Texas law.").

(220.) Transcript of Record at 5248-49, State v. Earhart, No. 4064,Dist. Ct. Lee County, 21st Judicial Dist., Texas (testimony of JohnRiley); see also id. at 5258 ("Well, bullets that are--that haveanalytically indistinguishable compositions or compositions that aregenerally similar typically are found within the same box of ammunitionand that is the case that we have here. Now, bullets that are the samecomposition can also be found in other boxes of ammunition, butit's most likely those boxes would have been manufactured at thesame place on or about the same date."). But see testimony ofCharles Peters, FBI examiner, Commonwealth v. Wilcox, Kentucky, Feb. 28,2002 (testifying during a Daubert hearing: "We have nevertestified, to my knowledge, that that bullet came from that box.We'd never say that. All we are testifying is that bullet, or thatvictim fragment or something, the bullet, either came from that box orthe many boxes that were produced at the same time." Transcript at1-2 (emphasis added)).

(221.) NAS REPORT, supra note 217, at 6.

(222.) See James Earhart, DEATH PENALTY INFO. CTR.,https://deathpenaltyinfo.org/james-earhart [https://perma.ee/2KUJ-QK7X](last visited Oct. 15, 2017).

(223.) See Clifford H. Spiegelman & Karen Kafadar, DataIntegrity and the Scientific Method: The Case of Bullet Lead Data asForensic Evidence, 19 CHANCE, no. 2, 2006, at 17, 22 ("During theopen sessions of the committee meetings, the FBI claimed to have a'complete data file' of some 71,000+ measurements. Followingrepeated requests from the Committee, the FBI submitted at its lastmeeting a CD-ROM that contained two data files with a combined total of64,869 bullet (not 71,000+) measurement records. This data set could notbe analyzed in time for the release of the report...."); Giannelli,supra note 215.

(224.) Spiegelman & Kafadar, supra note 223, at 17, 22.

(225.) Id. ("[T]he numbering system of the bullets was highlyinconsistent and rather unexpected (e.g., the bullets from a suspect ina particular case might be numbered Q13A, Q13B, Q13C, Q14A, Q14B, Q14C,... leading one to wonder what happened to bullets Q01, Q02, ...Q12)." (omissions in original)). Other illustrations of incompletedata were noted: "[W]hile most of the bullets indicated threemeasurements, about 30 bullets had six or more measurements.... [O]nlyabout 50% of the bullets in this data set were identified as having comefrom one of the four major bullet manufacturers in the United States[i.e., Cascade Cartridge, Inc.; Federal; Remington; Winchester]; the'complete data file' of 71,000 bullets may yield a higherproportion of bullets from these four manufacturers." Id.

(226.) Id.

(227.) Id.

(228.) Id.

(229.) Id. at 24; see also Giannelli, supra note 215.

(230.) FBI Nat'l Press Office, National Academy of SciencesReleases FBI-Commissioned Study on Bullet Lead Analysis, FBI (Feb. 10,2004), https://archives.fbi.gov/archives/news/pressrel/press-releases/national-acade my-of-sciences-releases-fbi-commissioned-study-on-bullet-lead-analysis [https: //perma.cc/8SXR-2PSV].

(231.) Id.

(232.) Maurice Possley, Study Shoots Holes in Bullet Analyses byFBI, Chi. Trib. (Feb. 11, 2004),http://artides.chicagotribune.eom/2004-02-ll/news/0402110356_1_bullet-analysis-bullet-lead-analysis-bullet-comparisons[https://perma.cc/Z59B-UZ5T].

(233.) Charles Piller, Report Finds Flaws in FBI Bullet Analysis:Changes Are Proposed for the Technique Often Cited in Expert Testimonyin Criminal Trials, L.A. TIMES (Feb. 11, 2004),http://articles.latimes.com/2004/feb/ 11/science/sci-bullet 11[https://perma.cc/4QHZ-G4QM].

(234.) Bootie Cosgrove-Mather, FBI Lab Under Scrutiny Again,Associated Press (Feb. 10, 2004, 1:26 PM),https://www.cbsnews.com/news/fbi-labunder-scrutiny-again/[https://perma.cc/PG9K-VGWL].

(235.) Eric Lichtblau, Report Questions the Reliability of anF.B.I. Ballistics Test, N.Y. TIMES (Feb. 11, 2004),http://www.nytimes.com/2004/02/11/us/report-questions-the-reliability-of-an-fbi-ballistics-test.html[https://perma.cc/N7JC-RF6P].

(236.) FBI Nat'l Press Office, supra note 230.

(237.) NAS REPORT, supra note 217, at 100.

(238.) Eric Lichtblau, F.B.I. Abandons Disputed Test for Bulletsfrom Crime Scenes, N.Y. Times (Sept. 2, 2005),http://www.nytimes.com/2005/09/02/politics/fbi-abandons-disputed-test-for-bullets-from-crime-scenes.html[https ://perma.cc/4 JLF-GCWW].

(239.) Press Release, FBI, FBI Laboratory Announces Discontinuationof Bullet Lead Examinations (Sept. 1, 2005),https://archives.fbi.gov/archives/news/pressrel/press-releases/fbi-laboratory-announces-discontinuation-of-bullet-lead -examinations [https://perma.cc/QMY5-7MBF]).

(240.) John Solomon, FBI's Forensic Test Full of Holes, Wash.Post (Nov. 18, 2007),http://www.washingtonpost.com/wp-dyn/content/article/2007/ll/17/AR2007111701681 .html [https://perma.cc/YGJ2-8ZK4].

(241.) See Ragland v. Commonwealth, 191 S.W.3d 569, 580 (Ky. 2006)(noting that "[i]f the FBI Laboratory that produced the CBLAevidence now considers such evidence to be of insufficient reliabilityto justify continuing to produce it, a finding by the trial court thatthe evidence is both scientifically reliable and relevant would beclearly erroneous"); Clemons v. State, 896 A.2d 1059, 1070, 1078(Md. 2006) ("CBLA is not admissible under the Frye-Reed standardbecause it is not generally accepted within the scientific community asvalid and reliable."; "Based on the criticism of the processesand assumptions underlying CBLA, we determine that the trial court erredin admitting expert testimony based on CBLA because of the lack ofgeneral acceptance of the process in the scientific community.");State v. Behn, 868 A.2d 329, 331 (N.J. Super. Ct. 2005) (finding thetechnique was "based on erroneous scientific foundations").But see Commonwealth v. Fisher, 870 A.2d 864, 871 (Pa. 2005) ("TheCBLA evidence, at best, established a possible connection betweenAppellant and the bullets recovered from the victim's body.");see also United States v. Davis, 406 F.3d 505, 509 (8th Cir. 2005)("Davis's trial counsel cannot be said to be ineffective forfailing to challenge the FBI's methodology on a basis that was notadvanced by the scientific community at the time of trial.").

(242.) John Solomon, FBI's Forensic Test Full of Holes, Wash.Post (Nov. 18, 2007),http://www.washingtonpost.eom/wp-dyn/content/article/2007/11/17/AR2007111701681_5.html?sid=ST2007111701983 [https://perma.cc/FKM4-L4WQ] (quoting Kenneth MacFadden).

(243.) Id.

(244.) 60 Minutes: Evidence of Injustice (CBS television broadcast,Nov. 16, 2007), https://www.youtube.com/watch?v=H4g62cpRz7M[https://perma.cc/XC 7E-E5HP],

(245.) Id. at 5:26.

(246.) Solomon, supra note 242 ("Hundreds of defendantssitting in prisons nationwide have been convicted with the help of anFBI forensic tool that was discarded more than two years ago. But theFBI lab has yet to take steps to alert the affected defendants orcourts, even as the window for appealing convictions isclosing....").

(247.) The Innocence Network and the National Association ofCriminal Defense Lawyers formed a task force and worked with the FBI tocontact defense attorneys and convicts. See Vesna Jaksic, FaultyBullet-Test Cases Finding Way to Court, NAT'L L.J. (Feb. 25, 2008),http://lethal-injectionflorida.blogspot.com/2008/02/faulty-bullet-test-cases-finding-way-to. html[https://perma.cc/PH7J-LTY9] ("The task force is lining up pro bonocommitments from several law firms to handle the cases.").

(248.) John Solomon, Leahy Pursues Forensic-Test Answers: AttorneyGeneral Is Told to Prepare For Senate Inquiry, WASH. POST (Nov. 22,2007), https://www.pressreader.com/usa/the-washington-post/20071122/2815394 01609068 [https://perma.cc/BKY5-SFFH] (quoting Leahy). Leahy alsowrote: "The new revelations about bullet-lead analysis are just thelatest examples of the Department's inadequate efforts to ensurethat sound forensic testing is utilized to the maximum extent to findthe guilty rather than merely obtain a conviction. Punishing theinnocent is wrong and allows the guilty party to remain free." Id.;see also Giannelli, supra note 215.

(249.) United States v. Mikos, No. 02 CR 137, 2003 WL 22922197, at*4-5 (N.D. Ill. Dec. 9, 2003).

(250.) 1 GIANNELLI ET AL., supra note 20, [section] 14.06, at 772(citations omitted).

(251.) Id. at [section] 14.12.

(252.) FBI HANDBOOK OF FORENSIC SCIENCE 57 (rev. ed. 1994)(emphasis added).

(253.) See, e.g., United States v. Hicks, 389 F.3d 514, 526 (5thCir. 2004) (stating that "the matching of spent shell casings tothe weapon that fired them has been a recognized method of ballisticstesting in this circuit for decades"); United States v. Foster, 300F. Supp. 2d 375, 376 n.1 (D. Md. 2004) ("Ballistics evidence hasbeen accepted in criminal cases for many years.... In the years sinceDaubert, numerous cases have confirmed the reliability of ballisticsidentification."); United States v. Santiago, 199 F. Supp. 2d 101,111 (S.D.N.Y. 2002) ("The Court has not found a single case in thisCircuit that would suggest that the entire field of ballisticsidentification is unreliable.").

(254.) 405 F. Supp. 2d 104 (D. Mass. 2005).

(255.) Id. at 107.

(256.) Id.

(257.) Id. at 109.

(258.) Id.

(259.) 407 F. Supp. 2d 351 (D. Mass. 2006).

(260.) Id. at 355.

(261.) Id. at 370.

(262.) See Theory of Identification, Association of Firearm andTool-mark Examiners, 30 AFTE J. 86, 86 (1998).

(263.) See Itiel E. Dror, How Can Francis Bacon Help ForensicScience? The Four Idols of Human Biases, 50 Jurimetrics 93, 104 (2009)("The potential problem here is the nonscientific nature of theidentification criteria. If the comparison of toolmarks enablesconclusions about common origin when the unique surface contours of twotoolmarks are in 'sufficient agreement,' what is thescientific definition and measurement of what constitutes such'sufficient agreement'? It seems that it is more in the eye ofthe beholder than strict scientific measures because it is determinedwithout specific quantification and criteria.").

(264.) Monteiro, 407 F. Supp. 2d at 370 (quoting Richard Grzybowskiet al., Firearm,/Toolmark Identification: Passing the Reliability TestUnder Federal and State Evidentiary Standards, 25 AFTE J. 209, 213(2003)).

(265.) See, e.g., United States v. Williams, 506 F.3d 151, 161-62(2d Cir. 2007) (upholding admissibility of firearms identificationevidence); United States v. Natson, 469 F. Supp. 2d 1253, 1261 (M.D. Ga.2007) ("According to his testimony, these toolmarks weresufficiently similar to allow him to identify Defendant's gun asthe gun that fired the cartridge found at the crime scene. He opinedthat he held this opinion to a 100% degree of certainty. ... The Courtalso finds [the expert's] opinions reliable and based upon ascientifically valid methodology. Evidence was presented at the hearingthat the toolmark testing methodology he employed has been tested, hasbeen subjected to peer review, has an ascertainable error rate, and isgenerally accepted in the scientific community.").

(266.) NAT'L RESEARCH COUNCIL OF THE NAT'L ACADS.,BALLISTIC IMAGING (Daniel L. Cork et al. eds., 2008).

(267.) The committee was asked to assess the feasibility, accuracy,reliability, and technical capability of developing and using a nationalballistic database as an aid to criminal investigations. It concluded:(1) "A national reference ballistic image database of all new andimported guns is not advisable at this time;" and (2) the NationalIntegrated Ballistics Information Network (NIBIN) "can and shouldbe made more effective through operational and technologicalimprovements." Id. at 5, 239.

(268.) Id. at 70.

(269.) Id. at 81. The report also stated: "Additional generalresearch on the uniqueness and reproducibility of firearms-relatedtoolmarks would have to be done if the basic premises of firearmsidentification are to be put on a more solid scientific footing."Id. at 82.

(270.) Id. at 82.

(271.) 578 F. Supp. 2d 567 (S.D.N.Y. 2008).

(272.) Id. at 575.

(273.) Id. at 570.

(274.) Id. at 574.

(275.) NAS Forensic Report, supra note 26, at 154.

(276.) Id. at 107-08.

(277.) AFTE Comm. for the Advancement of the Sci. of Firearm andTool Mark Identification, The Response of the Association of Firearmsand Tool Mark Examiners to the February 2009 National Academy of ScienceReport "Strengthening the Forensic Science in the United States: APath Forward", 41 AFTE J. 204, 206 (2009).

(278.) 849 F. Supp. 2d 425, 437-38 (D.N.J. 2012) ("TheCourt's analysis of the proposed testimony according to the Daubertfactors leads it to conclude that [the] expert report and opinion areadmissible under Rule 702.").

(279.) Id. at 438; see also NAS FORENSIC REPORT supra note 26, at153-55.

(280.) Clifford Spiegelman & William A. Tobin, Analysis ofExperiments in Forensic Firearms/Toolmarks Practice Offered as Supportfor Low Rates of Practice Error and Claims of Inferential Certainty, 12LAW, PROB. & RISK 115, 115 (2013).

(281.) See, e.g., United States v. Ashburn, 88 F. Supp. 3d 239, 249(E.D.N.Y. 2015) ("Nor can [the expert] testify that a match heidentified is to 'the exclusion of all other firearms in theworld,' or that there is a 'practical impossibility' thatany other gun could have fired the recovered materials."); UnitedStates v. Taylor, 663 F. Supp. 2d 1170, 1180 (D.N.M. 2009) ("[Theexpert] also will not be allowed to testify that he can conclude thatthere is a match to the exclusion, either practical or absolute, of allother guns.").

(282.) See, e.g., Ashburn, 88 F. Supp. 3d at 249 ("[T]he courtjoins in precluding this expert witness from testifying that he is'certain' or '100%' sure of his conclusions thatcertain items match."); United States v. Willock, 696 F. Supp. 2d536, 549 (D. Md. 2010) ("[The expert] shall state his opinions andconclusions without any characterization as to the degree of certaintywith which he holds them."); People v. Robinson, 2 N.E.3d 383, 402(111. App. Ct. 2013) ("[T]he judicial decisions uniformly concludetoolmark and firearms identification is generally accepted andadmissible at trial. Accordingly, we conclude the trial court did noterr in ruling the testimony in this case was admissible ... particularlywhere the trial judge barred the witnesses from testifying theiropinions were 'within a reasonable degree of scientificcertainty.'").

(283.) NAT'L COMM'N ON FORENSIC SCI., DOJ, VIEWS DOCUMENTON USE OF THE TERM "REASONABLE SCIENTIFIC CERTAINTY" 1 (2016).

(284.) Taylor, 663 F. Supp. 2d at 1180 ("He may only testifythat, in his opinion, the bullet came from the suspect rifle to within areasonable degree of certainty in the firearms examinationfield."); United States v. Cerna, No. CR 08-0730 WHA, 2010 WL3448528, at *4 (N.D. Cal. 2010) (allowing experts "to testify thata particular bullet or cartridge case was fired from a particularfirearm 'to a reasonable degree of certainty in the ballisticsfield'" (quoting United States v. Diaz, No. 05-00167, 2007 WL485967, at *14 (N.D. Cal. 2007)); Commonwealth v. Pytou Heang, 942N.E.2d 927, 945 (Mass. 2011) (stating that "the expert may offerthat opinion to a 'reasonable degree of ballisticcertainty'").

(285.) See, e.g., Fleming v. State, 1 A.3d 572, 590 (Md. Ct. App.2010) ("[Notwithstanding the current debate on the issue, courtshave consistently found the traditional method [of firearmsidentification] to be generally accepted within the scientificcommunity, and to be reliable."); People v. Givens, 912 N.Y.S.2d855, 857 (Sup. Ct. 2010) ("This Court was unable to find any caseswhere firearms and toolmark identification was found to be unreliable orno longer scientifically acceptable.").

(286.) 928 F. Supp. 2d 397 (D.P.R. 2013).

(287.) Id. at 400; see also United States v. Sebbern, No. 10 Cr.87(SLT), 2012 WL 5989813, at *7 (E.D.N.Y. Nov. 30, 2012); State v.Langlois, 2 N.E.3d 936, 950 (Ohio Ct. App. 2014) ("Our conclusionon this issue finds support in the decisions of other appellatedistricts in Ohio, notwithstanding the recent criticisms in scientificreports and the limitations some federal courts have imposed on thetestimony of firearms experts. These decisions hold that the methodologyof comparatively analyzing and testing bullets and shell cases recoveredfrom crime scenes is reliable."); State v. Jones, 303 P.3d 1084, at[paragraph] 75 (Wash. Ct. App. 2013) (expert testimony comparing buntermarks on the base of shell casings found at the crime scene to shellcasings found in Jones's home admissible under the Frye standard).

(288.) WHITE HOUSE PCAST REPORT, supra note 38, at 11.

(289.) Id.

(290.) Id.

(291.) Association of Firearm and Tool Mark Examiners Response toPCAST Report on Forensic Science, 48 AFTE J. 195, 195 (2016).

(292.) PRESIDENT'S COUNCIL OF ADVISORS ON SCI. AND TECH., ANADDENDUM TO THE PCAST REPORT ON FORENSIC SCIENCE IN CRIMINAL COURTS 7(2017) ("Several respondents wrote to PCAST concerning firearmsanalysis. None cited additional appropriately designed black-box studiessimilar to the recent Ames Laboratory study.").

(293.) See William A. Tobin et al., Absence of Statistical andScientific Ethos: The Common Denominator in Deficient ForensicPractices, 4 STAT. & PUB. POL'Y 1, 9 (2016)("[Practitioners remain intractable even after years of criticalscholarly papers, ad hoc committees of the National Academy of Sciences(NAS), position statements from the U.S. Department of Justice....") (citation omitted).

(294.) WHITE HOUSE PCAST REPORT, supra note 38, at 60.

(295.) Tobin et al., supra note 293, at 9 ("[T]he purported'validation studies' typically proffered to courts areseriously flawed [and] have no external validity. ...").

(296.) United States v. Green, 405 F. Supp. 2d 104, 109 (D. Mass.2005).

(297.) 130 A.3d 343 (D.C. 2016).

(298.) Id. at 355 (Easterly, J., concurring).

(299.) See Joseph L. Peterson & Anna s. Leggett, The Evolutionof Forensic Science: Progress Amid the Pitfalls, 36 STETSON L. REV. 621,654 (2007) ("The scientific integrity and reliability of DNAtesting have helped DNA replace fingerprinting and made DNA evidence thenew 'gold standard' of forensic evidence.").

(300.) 96 N.E. 1077 (111. 1911). See generally 1 GIANNELLI ET AL.,supra note 20, [section]16 (discussing the scientific and legal issuesassociated with fingerprint identification).

(301.) FEDERAL BUREAU OF INVESTIGATION, THE SCIENCE OFFINGERPRINTS: CLASSIFICATION AND USES iv (1984).

(302.) See, e.g., United States v. Collins, 340 F.3d 672, 682 (8thCir. 2003) ("Fingerprint evidence and analysis is generallyaccepted."); United States v. Hernandez, 299 F.3d 984, 991 (8thCir. 2002); United States v. Martinez-Cintron, 136 F. Supp. 2d 17, 20(D.P.R. 2001) (discussing how flaws and difficulties surroundingfingerprint evidence do not exclude the evidence outright).

(303.) 117 F. Supp. 2d 848 (S.D. Ind. 2000), aff'd, 260 F.3d597 (7th Cir. 2001).

(304.) Id. at 855; see also Paul C. Giannelli, Daubert Challengesto Fingerprints, 42 CRIM. L. BULL. 624, 628 (2006).

(305.) See Sandy L. Zabell, Fingerprint Evidence, 13 J.L. &POL'Y 143, 170 (2005) ("The argument that no latent print hasever been found to match the rolled print of a different person is ...misleading because no systematic search for such pairs on the entiredatabank of millions of fingerprints has ever been performed.").

(306.) Havvard, 117 F. Supp. 2d at 854.

(307.) Brief of the New England Journal of Medicine, Journal of theAmerican Medical Association, and Annals of Internal Medicine as AmiciCuriae in Support of Respondent, Daubert v. Merrell Dow Pharm., Inc.,509 U.S. 579 (1993) (No. 92-102), 1993 WL 13006287, at *3.

(308.) Havvard, 117 F. Supp. 2d at 854.

(309.) Jennifer L. Mnookin, Fingerprint Evidence in an Age of DNAProfiling, 67 BROOK. L. REV. 13, 60 (2001). Professor Mnookin goes on toprovide this analogy: "The same argument could be made ofeyewitness testimony, a notoriously unreliable form of evidence. Peopleare all distinct from one another in observable ways; therefore thetheoretical error rate of eyewitness identification is zero, though inpractice observers may frequently make errors." Id,; see also SimonA. Cole, More Than Zero: Accounting for Error in Latent FingerprintIdentification, 95 J. CRIM. L. & CRIMINOLOGY 985, 1040 (2005)(stating that while a "distinction can be drawn between'methodological' and 'practitioner' error" inother areas, "in fingerprint practice the concept isvacuous").

(310.) See Zabell, supra note 305, at 172 ("But, given itsunavoidable subjective component, in latent print examination people arethe process.").

(311.) See Michael J. Saks, The Legal and Scientific Evaluation ofForensic Science (Especially Fingerprint Expert Testimony), 33 SetonHall L. REV. 1167, 1173-76 (2003) (discussing the reversal of the burdenof persuasion as one of several judicial responses employed to avoidconfronting the lack of empirical testing); see also Giannelli, supranote 304, at 630.

(312.) 179 F. Supp. 2d 492 (E.D. Pa. 2002), vacated, 188 F. Supp.2d 549 (E.D. Pa. 2002).

(313.) D.H. Kaye, The Nonscience of Fingerprinting-. United Statesv. Llera-Plaza, 21 QUINNIPIAC L. REV. 1073, 1073 (2003).

(314.) As Professor Mnookin has noted, however, "fingerprintswere accepted as an evidentiary tool without a great deal of scrutiny orskepticism." Mnookin, supra note 309, at 17. She elaborated:"Even if no two people had identical sets of fingerprints, this didnot establish that no two people could have a single identical print,much less an identical part of a print. These are necessarily matters ofprobability, but neither the court in Jennings nor subsequent judgesever required that fingerprinting identification be placed on a securestatistical foundation." Id. at 19.

(315.) Llera Plaza, 188 F. Supp. 2d at 572.

(316.) See, e.g., United States v. Abreu, 406 F.3d 1304, 1307 (11thCir. 2005) ("We agree with the decisions of our sister circuits andhold that the fingerprint evidence admitted in this case satisfiedDaubert."); United States v. Janis, 387 F.3d 682, 690 (8th Cir.2004) (finding fingerprint evidence reliable); United States v.Mitchell, 365 F.3d 215, 244 (3d Cir. 2004) (finding that when assessedusing the Daubert factors, the fingerprinting evidence "pass[ed]muster"); United States v. Crisp, 324 F.3d 261, 269 (4th Cir. 2003)(finding that the district court did not abuse its discretion byadmitting fingerprint analysis as evidence); United States v. Sullivan,246 F. Supp. 2d 700, 704 (E.D. Ky. 2003) (finding that the possibilitythat two fingers may have a portion of their print in common speaks toweight and not the admissibility of fingerprint evidence).

(317.) E.g., Joann Loviglio, Trial Judge Reaffirms FingerprintUsability; Hearing Shows Him Science Involved, SAN ANTONIO EXPRESS-NEWS,Mar. 14, 2002;

Andy Newman, Judge Who Ruled Out Matching Fingerprints Changes hisMind, N.Y. Times, Mar. 14, 2002; Richard Willing, Judge ChallengesFingerprint Identification, USA TODAY, Jan. 10, 2002.

(318.) See, e.g., Michael Specter, Do Fingerprints Lie? The GoldStandard of Forensic Evidence Is Now Being Challenged, NEW YORKER (May27, 2002), https://www.newyorker.com/magazine/2002/05/27/do-fingerprints-lie [https: //perma.cc/T873-TJA7].

(319.) See, e.g., David L. Faigman, Is Science Different forLawyers?, 297 Science 339, 339-340 (2002).

(320.) 60 Minutes: Fingerprints: Infallible Evidence? (CBStelevision broadcast Jan. 5, 2003).

(321.) See, e.g., Simon A. Cole, Grandfathering Evidence:Fingerprint Admissibility Rulings from Jennings to Llera Plaza and BackAgain, 41 Am. Crim. L. Rev. 1189, 1189 (2004); Robert Epstein,Fingerprints Meet Daubert: The Myth of Fingerprint "Science"Is Revealed, 75 S. CAL. L. REV. 605, 607 (2002); Kristin Romandetti,Note, Recognizing and Responding to a Problem with the Admissibility ofFingerprint Evidence Under Daubert, 45 Jurimetrics 41, 53-56 (2004).

(322.) See Jennifer L. Mnookin, Fingerprints: Not a Gold Standard,ISSUES IN SCI. & TECH., Fall 2003, at 47 ("Judge Pollak'sfirst opinion [restricting latent fingerprint individualizationtestimony] was the better one."); Recent Case, United States v.Havvard, 260 F.3d 597 (7th Cir. 2001), 115 HARV. L. REV. 2349, 2352(2002) ("Fingerprint expert testimony does not survive applicationof the Daubert factors. ...").

(323.) 60 Minutes, supra note 320.

(324.) 324 F.3d 261 (4th Cir. 2003).

(325.) Id. at 269 ("Put simply, Crisp has provided us noreason today to believe that this general acceptance of the principlesunderlying fingerprint identification has, for decades, been misplaced.Accordingly, the district court was well within its discretion inaccepting at face value the consensus of the expert and judicialcommunities that the fingerprint identification technique isreliable."); see also Giannelli, supra note 304, at 632.

(326.) Id. at 273-74 (Michael, J., dissenting).

(327.) Id. at 274 (citing Epstein, supra note 321, at 644.

(328.) Id. The judge added: "In a 1995 test conducted by acommercial testing service, less than half of the fingerprint examinerswere able to identify correctly all of the matches and eliminate thenon-matches. On a similar test in 1998, less than sixty percent of theexaminers were able to make all identifications and eliminations.... Anerror rate that runs remarkably close to chance can hardly be viewed asacceptable under Daubert." Id. at 275 (citing Epstein, supra note321, at 634-35).

(329.) Id. at 276.

(330.) Id.

(331.) See Sara Kershaw, Spain and U.S. at Odds on Mistaken TerrorArrest, N.Y. Times, June 5, 2004, at Al (Spanish authorities clearedBrandon Mayfield and matched the fingerprints to an Algerian national);Flynn McRoberts & Maurice Possley, Report Blasts FBI Lab: PeerPressure Led to False ID of Madrid Fingerprint, Chi. Trib., Nov. 14,2004, at 1.

(332.) See Robert B. Stacey, A Report on the Erroneous FingerprintIndividualization in the Madrid Train Bombing Case, 54 J. FORENSICIDENTIFICATION 707 (2004).

(333.) See OFFICE OF THE INSPECTOR GEN., DOJ, A REVIEW OF THEFBI'S HANDLING OF THE BRANDON MAYFIELD CASE: UNCLASSIFIED EXECUTIVESUMMARY 7 (2006) ("Having found as many as 10 points of unusualsimilarity, the FBI examiners began to 'find' additionalfeatures in [the print] that were not really there, but rather weresuggested to the examiners by features in the Mayfield prints.").

(334.) Stacey, supra note 332, at 713.

(335.) OFFICE OF THE INSPECTOR GEN., supra note 333, at 10.

(336.) See Itiel E. Dror et al., Contextual Information RendersExperts Vulnerable to Making Erroneous Identifications, 156 FORENSICSCI. INT'L 74, 75-76 (2006) (reporting an experiment that showedfingerprint examiners changed their opinions when provided withirrelevant information); Elizabeth F. Loftus &; Simon A. Cole,Letter to the Editor, Contaminated Evidence, SCIENCE, May 2004, at 959,959("[F]orensic scientists remain stubbornly unwilling to confrontand control the problem of bias, insisting that it can be overcomethrough sheer force of will and good intentions."); Stacey, supranote 332, at 713 ("confirmation bias"). See generally D.Michael Risinger et al., The Daubert/Kumho Implications of ObserverEffects in Forensic Science: Hidden Problems of Expectation andSuggestion, 90 CAL. L. REV. 1, 39 (2002).

(337.) Examiners follow a procedure known as Analysis, Comparison,Evaluation, and Verification (ACE-V). See Zabell, supra note 305, at 178("ACE-V is an acronym, not a methodology. It is merely the commonsense description of what anyone would do if they were examining alatent and a candidate source print.").

(338.) See United States v. Crisp, 324 F.3d 261, 274 (4th Cir.2003) (Michael, J., dissenting) ("Proficiency testing is typicallybased on a study of prints that are far superior to those usuallyretrieved from a crime scene."); United States v. Llera Plaza, 188F. Supp. 2d 549, 565 (E.D. Pa. 2002) (noting that "the FBIexaminers got very high proficiency grades, but the tests they took didnot.... [O]n the present record I conclude that the proficiency testsare less demanding than they should be."); Jennifer L. Mnookin,Editorial, A Blow to the Credibility of Fingerprint Evidence, BOSTONGlobe, Feb. 2, 2004 ("There are no systematic proficiency tests toevaluate examiners' skill. Those tests that exist are not routinelyused and are substandard.").

(339.) See Donald Kennedy, Editorial, Forensic Science: Oxymoron?,SCIENCE, Dec. 5 2003, at 1625, 1625 (discussing the cancellation of aNational Academies project designed to examine various forensic sciencetechniques, including fingerprinting, because the DOJ and Defenseinsisted on a right of review that the Academy had refused to othergrant sponsors); United States v. Mitchell, 365 F.3d 215, 255 (3d Cir.2004) ("We are deeply discomforted by Mitchell'scontention--supported by Dr. Rau's account of events, thoughcontradicted by other witnesses--that a conspiracy within the DOJintentionally delayed the release of the solicitation until afterMitchell's jury reached a verdict. Dr. Rau's story, if true,would be a damning indictment of the ethics of those involved.").See generally Giannelli, supra note 201 (discussing the manipulation offorensic science research, including fingerprint research, by lawenforcement).

(340.) See Simon A. Cole, More Than Zero: Accounting for Error inLatent Fingerprint Identification, 95 J. CRIM. L. & CRIMINOLOGY 985,999 (2005) (discussing twenty-two cases involving misattribution oflatent fingerprints). The misidentification cases include some thatinvolved (1) verification by one or more other examiners, (2) examinerscertified by the International Association of Identification, (3)procedures using a sixteen-point standard, and (4) defense experts whocorroborated misidentifications made by prosecution experts. Id. at1023-25; see also CNN Presents: Reasonable Doubt (CNN televisionbroadcast Jan. 9, 2005) (discussing the misidentification of RikyJackson, who spent two years in prison).

(341.) See Bruce Budowle et al., Review of the Scientific Basis forFriction Ridge Comparisons as a Means of Identification: CommitteeFindings and Recommendations, FORENSIC SCI. COMM., Jan. 2006, at 1,https://www2. fbi.gov/hq/lab/fsc/backissu/jan2006/research/2006_01_research02.htm [https://perma.cc/86NF-XSBQ].

(342.) See Donald Kennedy & Richard A. Merrill, AssessingForensic Science, Issues in SCI. & TECH., Fall 2003, at 33, 34("The increased use of DNA analysis, which has undergone extensivevalidation, has thrown into relief the less firmly credentialed statusof other forensic science identification techniques (fingerprints, fiberanalysis, hair analysis, ballistics, bite marks, and tool marks). Thesehave not undergone the type of extensive testing and verification thatis the hallmark of science elsewhere."); Zabell, supra note 305, at164 ("Although there is a substantial literature on the uniquenessof fingerprints, it is surprising how little true scientific support forthe proposition exists.").

(343.) See, e.g., United States v. Pena, 586 F.3d 105, 110 (1stCir. 2009) ("The district court did not abuse its discretion.Numerous courts have found expert testimony on fingerprintidentification based on the ACE-V method to be sufficiently reliableunder Daubert."); United States v. Abreu, 406 F.3d 1304, 1307 (11thCir. 2005) ("[T]he fingerprint evidence admitted in this casesatisfied Daubert."); United States v. Janis, 387 F.3d 682, 690(8th Cir. 2004) ("We conclude the district court did not err inadmitting the fingerprint expert's testimony."); United Statesv. Mitchell, 365 F.3d 215, 241 (3d Cir. 2004) ("We therefore acceptthat the error rate has been sufficiently identified to count thisfactor as strongly favoring admission of the [fingerprint]evidence.").

(344.) 573 F.3d 979 (10th Cir. 2009).

(345.) Id. at 984.

(346.) See WHITE HOUSE PCAST REPORT, supra note 38, at 53("The fallacy is obvious: the expert simply assumed withoutevidence that every error in casework had come to light.").

(347.) NAS FORENSIC REPORT, supra note 26, at 139.

(348.) Id. at 142.

(349.) Id. at 141.

(350.) Id. at 144. Moreover, examiners lack population frequencydata to quantify the rarity or commonality of a particular type offingerprint characteristic. Id.

(351.) Id. at 142; see also id. at 143 ("Some in the latentprint community argue that the method itself, if followed correctly ...has a zero error rate. Clearly, this assertion is unrealistic.... Themethod, and the performance of those who use it, are inextricablylinked, and both involve multiple sources of error (e.g., errors inexecuting the process steps, as well as errors in humanjudgment).").

(352.) Id. at 144.

(353.) See WHITE HOUSE PCAST REPORT, supra note 38, at 91-95.

(354.) Bradford T. Ulery et al., Accuracy and Reliability ofForensic Latent Fingerprint Decisions, 108 PROC. NAT'L ACAD. SCIS.7733 (2011). The White House PCAST Report described the methodology ofthe report: "To attempt to ensure that the non-mated pairs wererepresentative of the type of matches that might arise when policeidentify a suspect by searching fingerprint databases, the known printswere selected by searching the latent prints against the 58 millionfingerprints in the [Automated Fingerprint Identification System]database and selecting one of the closest matching hits." WHITEHOUSE PCAST REPORT, supra note 38, at 94.

(355.) WHITE HOUSE PCAST REPORT, supra note 38, at 9.

(356.) Id. at 9-10.

(357.) Id. at 96.

(358.) See AAAS Fingerprint Report, supra note 39.

(359.) Anne Q. Hoy, Fingerprint Source Identity Lacks ScientificBasis for Legal Certainty: More Research into Validity of FingerprintComparisons Needed, Forensic Report Says, AAAS (Sept. 15, 2017),https://www.a2ias.org/news/fingerprint-source-identity-lacks-scientific-basis-legal-certainty[https://perma.cc/Z8SP-42KL].

(360.) AAAS FINGERPRINT REPORT, supra note 39, at 9.

(361.) Id. at 10.

(362.) Id. at 11.

(363.) Id. The report suggested that:

 [E]xaminers might say something like the following: "The latent print on Exhibit ## and the record fingerprint bearing the name XXXX have a great deal of corresponding ridge detail with no differences that would indicate they were made by different fingers. There is no way to determine how many other people might have a finger with a corresponding set of ridge features, but this degree of similarity is far greater than I have ever seen in non-matched comparisons." Id. at 11.

(364.) Id.

(365.) United States v. Crisp, 324 F.3d 261, 272 (4th Cir. 2003)(Michael, J., dissenting).

(366.) See supra note 300 and accompanying text.

(367.) WHITE HOUSE PCAST REPORT, supra note 38, at 95.

(368.) Some courts did, however, place limitations on thetestimony. See, e.g., United States v. Mitchell, 365 F.3d 215, 246 (3dCir. 2004) ("Testimony at the Daubert hearing indicated that somelatent fingerprint examiners insist that there is no error rateassociated with their activities or that the examination process isirreducibly subjective. This would be out-of-place under Rule702."); Commonwealth v. Gambora, 933 N.E.2d 50, 61 n.22 (Mass.2010) ("[O]pinions expressing absolute certainty about, or theinfallibility of, an 'individualization' of a print should beavoided.").

(369.) NAS FORENSIC REPORT, supra note 26, at 8.

(370.) Honorable Harry T. Edwards, The National Academy of SciencesReport on Forensic Sciences: What it Means for the Bench and Bar,Address at Conference of Superior Court of the District of Columbia,Washington, D.C. (May 6, 2010), in 51 JURIMETRICS 1, 9 (2010).

(371.) See Paul C. Giannelli, The 2009 NAS Forensic Science Report:A Literature Review, 48 CRIM. L. BULL. 378 (2012) (listing numerousarticles and conferences commenting on the report).

(372.) Jennifer L. Mnookin et al., The Need for a Research Culturein the Forensic Sciences, 58 U.C.L.A. L. Rev. 725 (2011).

(373.) Id. at 742.

(374.) See Giannelli, supra note 201.

(375.) National Commission on Forensic Science, DOJ,https://www.justice.gov/ncfs [https://perma.ee/AGU4-VDR8] (last visitedOct. 19, 2017).

(376.) As a result of the 2009 NAS report, an Interagency WorkingGroup--the Research Development Technology and Evaluation (RDT&E) ofthe National Science and Technology Council's Subcommittee onForensic Science--was tasked with identifying foundational research inthe forensic sciences. The RDT&E committee requested ScientificWorking Groups (SWGs) to address a series of discipline-specificquestions. In response, literature compendiums were submitted to theRDT&E committee by several forensic working groups. NAT'L SCI.& TECH. COUNCIL, COMM. ON SCI., SUBCOMM. ON FORENSIC SCI.,STRENGTHENING THE FORENSIC SCIENCES (2014).

(377.) NAT'L COMM'N ON FORENSIC SCI., DOJ, SCIENTIFIcl*tERATURE IN SUPPORT OF FORENSIC SCIENCE AND PRACTICE 2 (2015).

(378.) Id.

(379.) Id.

(380.) Id.

(381.) See United States v. Crisp, 324 F.3d 261, 274 (4th Cir.2003) (Michael, J., dissenting) ("Fingerprint examiners ... havetheir own professional publications.... But unlike typical scientificjournals, the fingerprint publications do not run articles that includeor prompt critique or reanalysis by other scientists. Indeed, few of thearticles address the principles of fingerprint analysis andidentification at all...."). See also Zabell, supra note 305, at164 ("Although there is a substantial literature on the uniquenessof fingerprints, it is surprising how little true scientific support forthe proposition exists.").

(382.) NAT'L COMM'N ON FORENSIC SCI., supra note 377, at3 ("Published in a journal that maintains a clear and publiclyavailable statement of purpose that encourages ethical conduct such asdisclosure of potential conflicts of interest integral to the peerreview process.").

(383.) Id. at 2. Other publication requirements include being: (1)"[p]ublished in a journal or book that has an InternationalStandard Number (ISSN for journals; ISBN for books) and recognizedexpert(s) as authors (for books) or on its Editorial Board (forjournals);" and (2) "[p]ublished in a journal that is indexedin databases that are available through academic libraries and otherservices (e.g. JSTOR, Web of Science, Academic Search Complete, andSciFinder Scholar)." Id. at 2-3.

(384.) Another commission document provided guidance for evaluatingscientific literature. NAT'L COMM'N. ON FORENSIC SCI., DOJ,VIEWS OF THE COMMISSION REGARDING IDENTIFYING AND EVALUATING LITERATURETHAT SUPPORTS THE BASIC PRINCIPLES OF A FORENSIC SCIENCE METHOD ORFORENSIC SCIENCE DISCIPLINE (2016). This guidance includes:

* Is the problem or hypothesis clearly stated?

* Is the scope of the article clearly stated as appropriate(article, case study, review, technical note, etc.)?

* Is the literature review current, thorough, and relevant to theproblem being studied?

* Does this work fill a clear gap in the literature or is itconfirmatory and/or incremental?

* Are the experimental procedures clear and complete such that thework could be easily reproduced?

* Are the experimental methods appropriate to the problem?

* Are the methods fully validated to the necessary level of rigor(fit for purpose)?

* Are the data analysis and statistical methodology appropriate forthe problem, and explained clearly so it can be reproduced?

* Are the experimental results clearly and completely presented anddiscussed?

* Are omissions and limitations to the study discussed andexplained?

* Are the results and conclusions reasonable and defensible basedon the work and the supporting literature?

* Are the citations and references complete and accurate?

* Are the references original (primary) and not secondary?

* Are funding sources and other potential sources of conflict ofinterest clearly stated?

Id. at 3.

(385.) NAT'L COMM'N. ON FORENSIC SCI., DOJ, VIEWS OF THECOMMISSION: TECHNICAL MERIT EVALUATION OF FORENSIC SCIENCE METHODS ANDPRACTICES 2 (2016); see also NAT'L COMM'N. ON FORENSIC SCI.,DOJ, RECOMMENDATION TO THE ATTORNEY GENERAL TECHNICAL MERIT EVALUATIONOF FORENSIC SCIENCE METHODS AND PRACTICES 1 (2016).

(386.) WHITE HOUSE PCAST REPORT, supra note 38, at 4. Here,"repeatable" means an examiner reaches the same result whenanalyzing the same sample. "Reproducible" means that differentexaminers reach the same result when analyzing the same sample. The term"accurate" means that "an examiner obtains correctresults both (1) for samples from the same source (true positives) and(2) for samples from different sources (true negatives)." Finally,"reliability" means "repeatability, reproducibility, andaccuracy." Id. at 47.

(387.) Id. at 5.

(388.) Id. at 5-6.

(389.) Id. at 6.

(390.) Id.

(391.) Id.

(392.) Id. at 14. The NAS report considered NIST beforerecommending an independent agency but rejected the idea because, atthat time, NIST had limited ties to forensic science. See NAS ForensicReport, supra note 26, at 17.

(393.) FBI, COMMENTS ON PCAST REPORT TO THE PRESIDENT, FORENSICSCIENCE IN FEDERAL CRIMINAL COURTS: ENSURING SCIENTIFIC VALIDITY OFPATTERN COMPARISON METHODS 1 (2016), http://www.crime-sceneinvestigator.net/PDF/fbi-response-to-forensic-science-in-federal-criminalcourts-ensuring-scientific-validity-of-pattern-comparison- methods.pdf[https://perma.cc/DTF5-RJ5M].

(394.) PRESIDENT'S COUNCIL OF ADVISORS ON SCI. AND TECH., ANADDENDUM TO THE PCAST REPORT ON FORENSIC SCIENCE IN CRIMINAL COURTS 3(2017), https://obamawhitehouse.archives.gov/sites/default/files/microsites/ ostp/PCAST/pcast_forensics_addendum_finalv2.pdf[https://perma.cc/68 NX-6C6A]).

(395.) Id. at 4.

(396.) Id. at 7.

(397.) Id.

(398.) Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 597(1993) (describing Daubert as prescribing "a gatekeeping role forthe judge").

(399.) Id. at 596 (citing Rock v. Arkansas, 483 U.S. 44, 61(1987)).

(400.) See Gertner, supra note 36, at 790 ("[T]he NASReport's concerns will not be fully met until advocacychanges."); D. Michael Risinger, The NAS/NRC Report on ForensicScience: A Path Forward Fraught with Pitfalls, 2010 UTAH L. REV. 225,242 (2010) ("Criminal defense lawyers ... are supposed to be thepeople who recognize bogus expert claims, challenge them, move to getthem excluded, and undermine those that survive exclusion byknowledgeable, thorough, and telling cross-examination. On the whole,they don't do any of these things very well.").

(401.) Garrett & Neufeld, supra note 102, at 89.

(402.) Neufeld, supra note 25, at SUO.

(403.) See generally Paul C. Giannelli, Ake v. Oklahoma; The Rightto Expert Assistance in a Posi-Daubert, Post-DNA World, 89 CORNELL L.REV. 1305 (2004) (discussing the legal disputes over the scope ofAke--e.g., whether it applied to non-capital cases and tonon-psychiatric experts).

(404.) Although prosecutors are ethically obligated to avoid theuse of flawed forensic testimony, the National District AttorneysAssociation recently asserted that bite mark evidence is a"reliable science"--an untenable position. See supra note 82and accompanying text. See generally Paul C. Giannelli & Kevin C.McMunigal, Prosecutors, Ethics, and Expert Witnesses, 76 FORDHAM L. REV.1493 (2007).

(405.) NAT'L RESEARCH COUNCIL, NAT'L ACAD, OF SCIS., ONTHE THEORY AND PRACTICE OF VOICE IDENTIFICATION (1979).

(406.) NAT'L RESEARCH COUNCIL, NAT'L ACAD, OF SCIS., THEEVALUATION OF FORENSIC DNA EVIDENCE (1996); NATIONAL RESEARCH COUNCIL,NAT'L ACAD, OF SCIS., DNA TECHNOLOGY IN FORENSIC SCIENCE (1992).

(407.) NAT'L RESEARCH COUNCIL, NAT'L ACAD, OF SCIS., THEPOLYGRAPH AND LIE DETECTION (2003).

(408.) NAT'L RESEARCH COUNCIL, NAT'L ACAD, OF SCIS.,FORENSIC ANALYSIS: WEIGHING BULLET LEAD EVIDENCE (2004).

(409.) In 2005, Peter Neufeld proposed an institute of forensicscience. Neufeld, supra note 25, at SI 13.

(410.) See David R. Cameron, Forum: Review of FBI Lab Suggests HugeNumber of Wrongful Convictions, NEW HAVEN REGISTER (April 26, 2015, 5:24PM), http://www.nhregister.com/opinion/artiele/Forum-Review-of-FBI-lab-sugges ts-huge-number-of-11353007.php [https://perma.cc/3NVM-2UN8]("The FBI review has identified roughly 2,500 cases that fit thosecriteria. The review is still in its early stages; thus far, it hasconsidered 268 trials involving 284 defendants. It has found that labexaminers gave flawed testimony regarding the comparison of hairs in 257of the 268 trials--more than 95 percent. Almost all of the examinersover that period--26 of 28--presented flawed testimony.").

(411.) See supra note 248 and accompanying text.

(412.) See Lauren Kirchner, Traces of Crime: How New York'sDNA Techniques Became Tainted, N.Y. Times (Sept. 4, 2017),https://www.nytimes.com/2017/09/04/nyregion/dna-analysis-evidence-new-york-disputed-techniques,html [https://perma.cc/H46S-9AQF] (explaining that two controversialtechniques have been discontinued); Spencer S. Hsu, FBI Notifies CrimeLabs of Errors Used in DNA Match Calculations Since 1999, wash. Post(May 29, 2015), https://www.washingtonpost.com/local/crime/fbi-notifiescrime-labs-of-errors-used-in-dna-match-calculations-since-1999/2015/05/29/f04234fc-0591-lle5-8bda-c7b4e9a8f7ac_story.html?utm_term=.2eed2704c8 35[https://perma.cc/GUZ3-VC4E] ("The FBI has notified crime labsacross the country that it has discovered errors in data used byforensic scientists in thousands of cases to calculate the chances thatDNA found at a crime scene matches a particular person, several peoplefamiliar with the issue said.").

(413.) See supra Section 2.C.

(414.) See Spencer S. Hsu, Sessions Orders Justice Dept. to EndForensic Science Commission, Suspend Review Policy, Wash. Post (April10, 2017), https://www.washingtonpost.com/local/public-safety/sessions-orders-justice-deptt o-end-forensic-science-commission-suspend-review-policy /2017/04/10/2dadaoca-1c96-11e7-9887-1a5314b56a08_story.html?utm_term=.f2d36117b56d[https: //perma.cc/XH7V-ZF WG].

(415.) Sunita Sah et al., We Must Strengthen the"Science" in Forensic Science, Sci. Am. (May 8, 2017),https://blogs.seientificamerican.com/observations/we-must-strengthen-the-science-in-forensic-science/ [https://perma.cc/47Y F-479M],

(416.) Spencer S. Hsu, Science Organizations Renew Call forIndependent U.S. Committee on Forensics, Wash. Post (June 29, 2017),https://www. washingtonpost.com/local/public-safety/science-organizations-renew-call-forindependent-us-committee-on-forensics/2017/06/28/3ab8cdea-5b6a-11e7-9b7d-14576dc0f39d_story.html?utm_term=.802de0997045[https://perma.cc/X46C-SY6U].

(417.) Betty Layne DesPortes, Message from the AAFS President, Am.Acad. of Forensic Scis. (April 3, 2017),https://news.aafs.org/presidentsmessage/message-from-the-aafs-president-april-2017/ [https://perma.cc/ 4T3K-34HA].

(418.) See Pema Levy, Sessions' New Forensic Science AdviserHas a History of Opposing Pro-Science Reforms, Mother Jones (Aug. 10,2017, 8:38 AM), http://www.motherjones.com/crime-justice/2017/08/sessions-new-forensicscience-adviser-has-a-history-of-opposing-pro-science-reforms/ [https://perma. cc/HAX9-QQDY] ("Attorney GeneralJeff Sessions has resisted efforts to rein in forensic science and holdit to higher standards. And this week, he appointed a senior adviser onforensics who has a history of opposing reforms that would bring moreaccountability and scientific rigor to forensic crime labs and experttestimony.").

(419.) NAS Forensic Report, supra note 26, at 19 (Recommendation1(c): "promoting scholarly, competitive peer-reviewed research andtechnical development in the forensic science disciplines").

(420.) Id. at 18.

(421.) Id.

(422.) Id. at 16. The Report also stated: "Neither has thefull confidence of the larger forensic science community. And becauseboth are part of a prosecutorial department of the government, theycould be subject to subtle contextual biases that should not be allowedto undercut the power of forensic science." Id.

(423.) Id. at 18.

(424.) See Giannelli, supra note 201.

(425.) Having served on the NCFS, the Author believes that thereshould have been more independent scientists on the Commission.

(426.) See, e.g., State v. Lambright, No. M2012-02538-CCA-R3-CD,2014 WL 46839, at *6 (Jan. 7, 2014) ("Dr. Tabor said that,considering the number of teeth that the victim's sister had, shewould not have been capable of producing the bite mark found on thevictim's nose and upper lip. It was Dr. Tabor's expert medicalopinion that a two-year-old was not capable of producing the nature,severity, number, and orientation of bites sustained by thevictim.").

(427.) WHITE HOUSE PCAST Report, supra note 38, at 6.

(428.) See supra note 379 and accompanying text (emphasizing theimportance of published peer review research).

(429.) See supra note 367 and accompanying text (noting thatclosed-set studies are not sufficiently robust).

(430.) If examiners claim that there is no error rate, they shouldbe required to explain why not. See White House PCAST Report, supra note38, at 19 ("In testimony, examiners should always state clearlythat errors can and do occur, due both to similarities between featuresand to human mistakes in the laboratory.").

(431.) This depends on when the clock started ticking. Daubert wasdecided in 1993. In 1995, the first challenge to handwriting testimonywas decided. United States v. Starzecpyzel, 880 F. Supp. 1027 (S.D.N.Y.1995). As noted above, the Williamson case on microscopic hair analysiswas decided the same year. See supra notes 92-93 and accompanying text.Llera Plaza was decided in 2002. See supra notes 312-315 andaccompanying text (addressing the admissibility of fingerprinttestimony). And Green was decided in 2005. See supra notes 254-258 andaccompanying text (addressing firearms identification testimony).Moreover, during this period numerous courts restricted the use ofhandwriting identification. See, e.g., United States v. Hines, 55 F.Supp. 2d 62, 73-74 (D. Mass. 1999) (holding that expert testimonyconcerning the general similarities and differences between adefendant's handwriting exemplar and a stick up note was admissiblebut not the specific conclusion that the defendant was the author).

(432.) See supra note 161 and accompanying text.

(433.) 578 F. Supp. 2d 567 (S.D.N.Y. 2008).

(434.) Id. at 575.

(435.) White House PCAST Report, supra note 38, at 19.

(436.) See supra note 283 and accompanying text.

(437.) See supra Section III.B.3.

(438.) AAAS Fingerprint Report, supra note 39, at 11.

(439.) See Kennedy, supra note 339 (discussing the cancellation ofa National Academies project designed to examine various forensicscience techniques, including fingerprinting, because the Departments ofJustice and Defense insisted on a right of review that the Academy hadrefused to other grant sponsors).

(440.) NAS FORENSICS SCIENCES REPORT, supra note 26, at 9.

(441.) United States v. Glynn, 578 F. Supp. 2d 567, 570 (S.D.N.Y.2008). See also United States v. Starzecpyzel, 880 F. Supp. 1027, 1038(S.D.N.Y. 1995) ("[F]orensic document examination, despite theexistence of a certification program, professional journals and othertrappings of science, cannot, after Daubert, be regarded as'scientific ... knowledge.'"); id. at 1041 ("[W]hilescientific principles may relate to aspects of handwriting analysis,they have little or nothing to do with the day-to-day tasks performed by[Forensic Document Examiners].... [T]his attenuated relationship doesnot transform the FDE into a scientist.").

(442.) For example, a fingerprint examiner from New Scotland Yardtestified in one case that the FBI proficiency tests were deficient:"It's not testing their ability. It doesn't test theirexpertise. I mean I've set these tests to trainees and advancedtechnicians. And if I gave my experts these tests, they'd fallabout laughing." United States v. Llera Plaza, 188 F. Supp. 2d 549,558 (E.D. Pa. 2002). The district court agreed, noting that "theFBI examiners got very high proficiency grades, but the tests they tookdid not.... [O]n the present record I conclude that the proficiencytests are less demanding than they should be." Id. at 565.Similarly, in a trial involving handwriting comparisons, the courtwrote:

 There were aspects of Mr. Cawley's testimony that undermined his credibility. Mr. Cawley testified that he achieved a 100% passage rate on the proficiency tests that he took and that all of his peers always passed their proficiency tests. Mr. Cawley said that his peers always agreed with each others' results and always got it right. Peer review in such a "Lake Woebegone" environment is not meaningful.

United States v. Lewis, 220 F. Supp. 2d 548, 554 (S.D.W. Va. 2002);see supra note 338 (discussing fingerprint proficiency testing).

(443.) NAT'L COMM. ON FORENSIC SCI., VIEWS OF THE COMMISSION:OPTIMIZING HUMAN PERFORMANCE IN CRIME LABORATORIES THROUGH TESTING ANDFEEDBACK (May 27, 2016), https://www.justice.gov/ncfs/file/864776/download [https://perma.cc/8BPS-DEER]).