In Ross v. State, the Supreme Court in Bronx County ruled on a Frye motion by the defense”

“to preclude a ballistics expert from testifying that shell casings found at a crime scene matched a firearm found in a car the defendants occupied. Defendants argued a forensic comparison of the shell casings to test fires from the gun would lack ‘scientific validity and general acceptance in the relevant scientific community.'”

Several experts testified at the Frye hearing in the case.

Defense experts were psychologist Dr. Nicholas Scurich (U.C. Irvine) and Dean David Faigman (Hastings Law). Both emphasized that studies of foundational validity and error rates are key to assessing the reliability of a method like firearms comparison – and they emphasized that the sole such study, conducted by the Ames lab, displayed troublingly high error rates.

The Court explained: “The Ames study adhered to what is commonly referred to as “black box” design, in which the researchers tested and arranged all the ammunition in advance so the correct answers (either a match or no match) were known to them. Volunteer examiners were asked to determine whether a given pair of bullets or cartridges matched, and then to move on to the next set (“pairwise comparison”). Both defense experts found this design more likely to generate accurate data than the numerous studies in which examiners were given a number of known cartridges or bullets and asked whether any of them matched a number of unknowns (“set-to-set” design).”

The firearms examiner retained by the prosecution in the case, “countered that the sheer number of studies with similar results verify the success of trained examiners in matching bullets or casings to firearms.” Further, “[h]e pointed out that DNA analysis–the gold standard forensic discipline–was also somewhat subjective and open to interpretation, particularly where there are multiple contributors or low copy DNA.” He also “defended the studies’ treatment of inconclusive answers as correct answers, due to the examiners’ cautious approach and experience operating in real world conditions; he praised examiners for not jumping to conclusions.”

The Court retained an additional expert: a statistician, Dr. Heike Hoffman, from Iowa State University, who “testified that the determination of a true error rate was very tricky to determine from these semi-field studies because of the multiple sources for error.” Further, “Dr. Hofman expressed particular concern over evidence quality and the lack of controls over the examiners taking the tests. These factors were not taken into account in calculating error rates in most studies, even the Ames Laboratory study. Hofmann also agreed with the defense experts that the studies which had a closed set, where a cartridge case must match or not match one other piece of evidence are known to underestimate the error rate. Finally, she testified that by excluding inconclusive results, the error rates were artificially deflated.”

The Court emphasized:

Courts must be wary when scientific evidence is offered to prove a defendant’s guilt. The problem of a microscopist putting expert conclusions before the jury is not solved by the opportunity to cross examine or present a countervailing expert on methodology. Experts enter upon the jury’s province, since the expert–-and not the jury–draws conclusions from the facts.

The  Court concluded (with emphasis added):

The People may call an expert to testify as to whether there is evidence of class characteristics that would include or exclude the firearm at issue. The ballistics examiner may explain the reasons for an opinion that class characteristics are present or not present to the jury. In addition, the examiner may, if s/he believes the class characteristics are the same, indicate that the firearm cannot be ruled out as the source of the shell casings. The examiner may further explain what is done with instruments, e.g. the process of using a comparison microscope, describe verbally and/or show the jurors photos of the relevant evidence, including shell casings and test fires.

The examiner may not, however, offer qualitative opinions on matters not adequately supported by the defined relevant scientific community. Specifically, the examiner may not opine on the significance of any marks other than class characteristics, as the reliability of that practice in the relevant scientific community as a whole has not been established. Moreover, any opinion based in unproven science and expressed in subjective terms such as “sufficient agreement” or “consistent with” may mislead the jury and will not be permitted.