By Dr. Ken Broda-Bahm:
Our court system is all about law, obviously, but sometimes the legal field intersects with research fields. Two points where the law overlaps with research methods are the use of scientific testimony and the conduct of pretrial research projects like mock trials or focus groups. In both settings, a failure to appreciate basic research norms can lead to misunderstanding and misuse. Problems are most acute when statistics are used by those who aren't statistical experts and don't have statistical experts on their teams. As part of their, "Ten Simple Rules," series, the online academic journal PLOS has published a widely-read essay by Carnegie Mellon University's Robert E. Kass entitled, "Ten Simple Rules to Use Statistics Effectively." The paper is aimed at researchers who, finding themselves with larger and larger amounts of data, are increasingly tempted to "go fishing" to find relationships within the data, but it also carries some important implications for consumers of research. "The paper is an instant 'must-read' for anyone who cares about good and reproducible science," CMU psychologist Michael Tarr says. In this post, I will share Kass's ten rules and discuss the implications for both scientific testimony and for pretrial research.
One, Statistical Methods Should Enable Data to Answer Scientific Questions
Collaborating with statisticians is often most helpful early in an investigation because inexperienced users of statistics often focus on which technique to use to analyze data, rather than considering all of the ways the data may answer the underlying scientific question.
The questions lead, and the statistics follow. That means that any investigation ought to begin with research questions, and only then employ statistics to look for answers to those questions. The alternative is to simply sift through the data looking for any and all correlations there might be. Remembering that a .05 level of statistical significance means that you have a one in twenty chance of a false positive, that means if you're simply trying a large number of correlations, you're bound to find many that won't necessarily be meaningful. Both your expert's analysis as well as your mock trial should begin with research questions which then guide the use of statistics.