Comments on PATTERN
Brandon L. Garrett and Megan T. Stevenson
September 12, 2019
We are grateful to have been requested to provide feedback on the Prisoner Assessment Tool Targeting Estimated Risk and Needs (PATTERN). The Department of Justice was made aware of our work on this issue by the Charles Koch Institute, whose representatives have engaged ineduction on risk assessments and related criminal justice related topics. The institute’s affiliate,the Charles Koch Foundation, is a charity that has provided grants supporting our broader criminal justice research. The views presented here are offered independently and have not been discussed with any of our supporters, nor have we received any compensation for providing them. We are not writing as representatives of any institution, but as scholars who study the use of risk assessment in criminal justice.
We applaud the U.S. Department of Justice (DOJ) for holding a 45-day public comment period, permitting review and comments on the PATTERN system in order to “consider ways in which it may be improved.”i The report states that while operating “under an extremely short timeline,” the tool achieves a “high level of predictive performance and surpasses what is commonly found” in risk assessment tools used in the U.S.ii While we understand the constraints of a short timeline, the report does not provide enough detail for us to evaluate the methodology with anything approaching the rigor of a “peer-review” process, and without access to the underlying data we are unable to verify claims regarding predictive performance or fairness. Our response, therefore, addresses general issues with the tool and its implementation as we understand it from the report. We offer these comments in the spirit of supporting efforts to use risk assessment to reduce incarceration for individuals who could otherwise be in the community.
Issue 1: Selecting Risk Cut-Offs
While the Act states that the federal Bureau of Prisons (BOP) must use a new risk assessment instrument to assign inmates to one of four risk categories—minimum, low, medium, and high—the statute does not define those categories. The DOJ Report that announced the development of the PATTERN describes how the experts who developed it made certain key decisions concerning the risk thresholds that separate these groups. However, no information is provided in that Report about how those thresholds were set. Determining how many individuals are rated minimum or low risk, and therefore get the many benefits associated with this designation, is one of the most influential decisions pertaining to the risk assessment tool.
Crucially, “[t]he ultimate description of a defendant’s risk as low, moderate, or high in a givenjurisdiction is a policy decision, not a scientific one.”iii It is important that this central policy decision be transparent and supported. It also crucially important that the public and policymakers have confidence that the choice made to select those thresholds was a sound one. BOP administrators will be relying on those thresholds to classify every eligible federal prisoner. That is a burdensome, momentous, and important task. The BOP too must have confidence that a considered and fully transparent decision was made to select those cutoffs.
We are concerned that the thresholds currently chosen will unnecessarily limit the opportunities for early release for defendants that pose little risk to the community. As it currently stands, the threshold for being placed in the moderate risk category is a 24% probability of rearrest or technical violation within three years. Therefore, an individual who has a 75% chance of complete success upon release – not even a minor arrest or violation – would not be able to accrue the benefits of a low risk classification.
We recommend that the decision of where to place the risk-thresholds for each risk category be made in a transparent manner, with a clearly specified rationale. We recommend expanding the minimum and low risk category so that more defendants can take advantage of the benefits of the First Step Act.
Issue 2: What Type of Risk?
While the Act calls for the development of a risk assessment tool to predict recidivism, it does not clarify which measure of recidivism should be used. PATTERN currently defines recidivism in two ways: the risk of any sort of arrest or technical violation (“general recidivism”), and the risk of arrest for a new violent crime (“violent recidivism”). No information is provided on why these specific recidivism measures were chosen, although we understand that data limitations may have precluded using conviction as a measure instead of arrest.
We are concerned about the general recidivism measure. The risk of serious or violent crime is of clear concern, but the risk of committing minor offenses or technical violations is much less central to decisions around early release (or to the other proposed uses of PATTERN). Yet the general recidivism risk score does not distinguish between the two. In fact, it is likely capturing much more of the latter than the former. Among federal prisoners, roughly 2/3 of rearrests within 8 years are for nonviolent offenses.iv While some nonviolent offenses are serious, many are not. Nationwide, among arrests for which charges are filed, the vast majority are misdemeanors.v
The inclusion of minor offenses and violations means that the general recidivism score will be less accurate at predicting those at risk of serious crime. Instead, those who score high in the general recidivism score will be disproportionately the type of people who tend to get arrested for misdemeanors or violate technical terms of parole: those who struggle with mental health, homelessness, and substance abuse issues.
A risk assessment tool that includes low-level arrests in its recidivism measure is also expected to be more racially biased. While people of all races jaywalk, appear intoxicated in public, and possess marijuana, not all are arrested for such activities. Arrest for low level offenses is more likely if such behavior occurs in police presence or is revealed by way of stop-and-frisk. Since minority neighborhoods are often more heavily policed, the probability of arrest, conditional on engaging in minor illegal behavior, is particularly high for people of color.
While PATTERN also includes a violent recidivism risk score, the low risk classification that automatically triggers expanded access to early release requires a minimum or low risk ranking in both the general and violent recidivism risk assessments. Based on the thresholds described in the Report, we expect that many people will be blocked from receiving the expanded good-timecredits by their risk of “general recidivism”, even if they are at low risk of violent crime. This adds further opacity to the risk threshold method, since the overall classification depends on the threshold selected for both risk assessments as well as the correlation between the two. And it results in the exclusion of First Step Act benefits to those who pose little risk of engaging in serious crime.
We suggest that recidivism be defined to exclude arrest for minor crime or technical violations. If multiple risk assessments are used, risk thresholds should be adjusted so that the requirement of minimum or low risk scores in both risk assessments does not unnecessarily exclude too many individuals from the advantages outlined in the Act.
Issue 3: Insufficient Resources
An additional question is whether adequate resources will be made available for programming. An inmate cannot change the static factors used to calculate the PATTERN score, by definition. Beyond simply aging, the only way that an inmate can improve a score is through completion of various programs in federal prisons.
Unfortunately, not all inmates will be able to access these programs. Indeed, there may be far greater demand for these programs once they are linked to the possibility for early release.
Resources for rehabilitative programming in federal prisons is a longstanding need. The Inspector General’s audit of the reentry programming in the BOP raised a similar problem,regarding a lack of resources to conduct sufficient reentry programming. During the consideration of the Act, the American Federation of Government Employees, AFL-CIO, and Council of Prison Locals called for sentencing reforms, but raised the concern as to the use ofrisk assessment, that federal prisoners are suffering from “sustained cuts” to staffing levels,which reduces access to rehabilitative programming.vi The Act calls for $75 million in funding per year, including for expanding such rehabilitative programming, as well as an expansion of reentry programming such as halfway housing. vii However, in addition to the funding needed to expand such programming, some part of the available funding may need to be expended on implementing the risk assessment tool itself. The Act passed after a budget had already been finalized, and therefore it is not yet clear whether, in future budgets, the full $75 million will be requested or obtained from Congress. viii
This resource problem has hampered risk assessment efforts at the state level as well. In a study of Virginia sentencing, researchers found that eligible offenders did not receive the alternative sentences for which they were recommended. Many judges explained, in responses to a survey by the same research team, that they had resource constraints on their ability to follow the risk assessment recommendations.ix
Considering the substantial weight that is placed on age in PATTERN, we are particularly concerned that young individuals will not be able to adequately lower their risk score through participation in rehabilitative programming. A male prisoner who is under the age of 25, and whose first conviction was under the age of 18, is at the high end of the moderate risk range even if their criminal history scores are zero and they have no other aggravating factors. They would need to engage in a substantial number of programs in order to reduce their risk score to a lower classification. In fact, they would need to complete greater than 10 programs simply to cancel out the effect of having their first conviction under the age of 18. (A first conviction under the age 18 adds 12 points to the risk score; the maximum number of points that can be deducted from the risk score by participating in programs is 12.)x If their facility does not have the resources to provide such programs, and also to ensure meaningful opportunities to complete programs that are provided, then they would be stuck with the higher risk classification until they age into a lower one.
We recommend that the BOP present a plan for individuals to be able to gain access to the benefits of a low risk classification if resources for programming are inadequate.
While there are many laudable aspects of the First Step Act, it is important to get the details right if the adoption of the risk assessment is going to support reducing incarceration for individuals who pose little public safety risk. We note two cautionary tales based on prior federal implementation of risk assessment:
Consider the use of the federal Pretrial Risk Assessment (PTRA), which was implemented in 2009, and is currently used in most federal districts. Adoption of this instrument does not appear to have increased pretrial release rates; in fact, pre-trial release rates have declined over the past decade. (Researchers at the Probation and Pretrial Services Office of the Administrative Office of the U.S. Courts confirmed that the decline is observed even if one controls for changes in the federal defendant population, such as the increase in immigration filings and the decrease in financial crime filings. Even defendants with light criminal history profiles saw higher detention rates.xi )
Or consider a setting in which the BOP has been involved. The BOP had been using risk and needs assessments to decide whether to place federal prisoners in residential reentry centers (RRC), or halfway houses, as well as direct home confinement, while serving the remainder of their sentences, pursuant to the Second Chance Act of 2007. The DOJ’s Office of Inspector General conducted an audit of that program in 2016, following DOJ efforts to improve reentry programs in the federal system. The auditors concluded that inmate risk and needs were notbeing carefully considered: “contrary to BOP policy, BOP guidance, and relevant research, BOP’s RRC and home confinement placement decisions are not based on inmate risk forrecidivism or need for transitional services.” They also found that the BOP was underutilizing direct home confinement for low-risk prisoners, which also meant that there was not room in RRCs for higher-needs and higher-risk prisoners before their release dates.xii
There is ample evidence that simply adopting a risk assessment tool does not automatically bring benefits. To maximize the likelihood of success, great care needs to be given in developing the tool and designing its implementation. We believe that the DOJ should not adopt an instrument without adequate vetting by the research community. The report released does not permit such an assessment of the PATTERN.
Brandon L. Garrett is the L. Neil Williams Professor of Law at Duke University School of Law, where he has taught since 2018. He was previously the Justice Thurgood Marshall Distinguished Professor of Law and White Burkett Miller Professor of Law and Public Affairs at the University of Virginia School of Law, where he taught since 2005. His research and teaching interests include criminal procedure, wrongful convictions, habeas corpus, corporate crime,scientific evidence, civil rights, and constitutional law. Garrett’s work, including several books,has been widely cited by courts, including the U.S. Supreme Court, lower federal courts, state supreme courts, and courts in other countries. Garrett also frequently speaks about criminal justice matters before legislative and policymaking bodies, groups of practicing lawyers, law enforcement, and to local and national media. Garrett attended Columbia Law School, where he was an articles editor of the Columbia Law Review and a Kent Scholar. After graduating, he clerked for the Hon. Pierre N. Leval of the U.S. Court of Appeals for the Second Circuit. He then worked as an associate at Neufeld, Scheck & Brustin LLP in New York City. Garrett has participated for several years as a researcher, and now principal investigator, in the Center for Statistics and Applications in Forensic Science (CSAFE), as well as a principal investigator in an interdisciplinary project examining eyewitness memory and identification procedures supported by Arnold Ventures. Supported a grant from the Charles Koch Foundation, Garrett directs the Center for Science and Justice at Duke Law, an interdisciplinary empirical research group focused on criminal justice.
Megan T. Stevenson is an Assistant Professor of Law at George Mason University’s Antonin Scalia Law School. Her research uses econometric methods to evaluate criminal law and policy in areas such as bail, pretrial detention, risk assessment, and juvenile justice. Her studies have been published in top journals in both law and economics, such as the Stanford Law Review and the Review of Economics and Statistics. She was the 2019 winner of the Oliver E Williamson prize for best article, chosen from all articles published in the Journal of Law, Economics, and Organization within three years. Her research on bail was cited extensively in a landmark federalcivil rights decision, O’Donnell v. Harris, and has received widespread media coverage. Inaddition to legal scholarship, Professor Stevenson has written a number of op-eds for news outlets such as the Houston Chronicle and the Philadelphia Inquirer. Her research has been funded by the National Science Foundation, the Russell Sage Foundation, and the Laura and John Arnold Foundation. Stevenson has been an Assistant Professor of Law at George Mason University’s Antonin Scalia Law School since 2017. In that capacity she has worked withGeorge Mason’s Law and Economics Center, which is funded by the Charles F. KochFoundation. Prior to joining the law faculty at George Mason, she was a fellow at the Quattrone
Center for the Fair Administration of Justice at the University of Pennsylvania Law School (2015-2017). She holds a BA in Interdisciplinary Studies (2009, with highest distinction) and a PhD in Agricultural and Resource Economics (2016), both from the University of California, Berkeley. She teaches Law & Economics and Criminal Law.
i U.S. Department of Justice, The First Step Act of 2018: Risk and Needs Assessment System (2019).
ii Id. at 43.
iii Sarah L. Desmarais and Evan M. Lowder, Pretrial Risk Assessment Tools: A Primer for Judges, Prosecutors, and Defense Attorneys, SAFETY AND JUSTICE CHALLENGE 4 (Feb. 2019).
iv United States Sentencing Commission, Recidivism among Federal Offenders: A Comprehensive Overview, March 2016
v Megan T. Stevenson & Sandra G. Mayson, The Scale of Misdemeanor Justice, 98 B.U. L. REV. 101 (2018) at 746.vi See The American Federation of Government Employees, AFL-CIO & Council of Prison Locals, Letter to Chairman Grassley, Ranking Member Feinstein, Chairman Goodlatte, Ranking Member Nadler, and Senator Durbin, (May 8, 2018).
vii FIRST STEP Act of 2018, 115 P.L. 391 § 104(a).
viii Maggie Haberman and Annie Karni, Trump Celebrates Criminal Justice Overhaul Amid Doubts it will be Fully Funded, N.Y.Times, April 1, 2019.
ix Brandon L. Garrett, Alexander Jakubow & John Monahan, Judicial Reliance on Risk Assessment in Sentencing Drug and Property Offenders: A Test of the Treatment Resource Hypothesis, Criminal Justice and Behavior,
46 CRIM. JUSTICE & BEHAVIOR 799 (2019).
x x U.S. Department of Justice, The First Step Act of 2018: Risk and Needs Assessment System (2019) at 53, 54.
xi Matthew G. Rowland, The Rising Federal Pretrial Detention Rate, in Context, 82 Fed. Probation 13, 13 (2018).xii See U.S. DEPARTMENT OF JUSTICE OFFICE OF INSPECTOR GENERAL, AUDIT OF FEDERAL BUREAU OF PRISONS’MANAGEMENT OF INMATE PLACEMENTS IN RESIDENTIAL REENTRY CENTERS AND HOME CONFINEMENT ii (Nov. 2016), https:/ /oig.justice.gov/reports/2016/a1701.pdf.