Risk Assessment Protocol Crucial to Success of Juvenile Probationers

Valid implementation of risk assessment and case management procedures improve resource allocation without interfering in youth’s lives or increasing the risk to public safety. This is the bottom line of a recently published article in Law and Human Behavior. Below is a summary of the research and findings as well as a translation of this research into practice.

Featured Article | Law and Human Behavior | 2016, Vol. 40, No. 6, 683-696

Risk Assessment Matters, But Only When Implemented Well: A multisite Study in Juvenile Probation

Authors

Gina M. Vincent, University of Massachusetts Medical School
Laura S. Guy, Simon Fraser University
Rachael T. Perrault, University of Massachusetts Medical School
Bernice Gershenson, University of Massachusetts Medical School

Abstract

There is a strong movement toward juvenile justice agencies’ use of risk assessment and risk-need responsivity approaches to improve case management decisions for young offenders. However, little is known about whether adoption of risk assessment actually effectuates any changes in the way young offenders are handled. This was a multisite study of the impact on case processing of implementation of the Structured Assessment of Violence Risk in Youth (SAVRY) or Youth Level of Service/Case Management Inventory in 6 juvenile probation offices using a prepost design and 1,694 propensity score-matched young offenders. Consistent with the risk principle, there were significant changes to at least some areas of case processing in all but 1 site, most notably with respect to decreases in the amount of supervision youth received and in rates of out-of-home placement. The nature and extent of the impact varied as a function of sites’ characteristics and implementation quality, not as a function of the risk assessment used. No increases in recidivism were observed in any site, and there was a significant reduction in recidivism in 1 site. The key benefits of implementation of valid risk assessment and case management procedures were improved resource allocation and fewer instances of inappropriate interference in youths’ lives without an apparent increased risk to public safety.

Keywords

SAVRY, YLS/CMI, implementation study, juvenile, RNR

Summary of the Research

“Considerable attention has been directed toward reforming juvenile justice over the past decade by integrating research evidence and principles of adolescent development into practice (National Research Council [NAS], 2013). One prominent recommendation for reform has been to base individual programming decisions on risk and criminogenic needs. The NAS (2013) strongly recommended structured risk and need assessment tools (RNAs) be used to identify low-risk youth who could be handled less formally, to match youth to appropriate treatment, and to target high-risk youth for more intensive interventions. Similarly, the Council of State Governments (Seigle, Walsh, & Weber, 2014) listed use of valid RNAs for supervision, service, and resource-allocation as one of the four core principles for reducing recidivism and improving outcomes for youth. Consequently, most states today have instituted a RNA in juvenile probation for use in case planning (Wachter, 2015). These recommendations stem from evidence that individualized case management models, such as risk-need-responsivity (RNR; Andrews & Bonta, 2003, 2010; Andrews, Bonta, & Hoge, 1990), are effective means for reducing recidivism whereas more global approaches toward punishment are not (e.g., Gatti, Tremblay, & Vitaro, 2009; Lipsey, 2009; Lipsey & Cullen, 2007; Loughran et al., 2009; MacKenzie, Wilson, & Kider, 2001; Petrosino, TurpinPetrosino, & Guckenburg, 2010)” (p. 683).

“Studies of probation officers (POs) have found that the actual use of RNAs in decision-making in the justice field is limited…Very few studies have examined actual changes that result in the handling of probation cases after implementation of a RNA” (p. 684).

“The current study focused on the impact of implementing the risk principle in these decisions, anticipating a decrease in use of restrictive monitoring and incarceration once a RNA and RNR practices were implemented. Specifically, we hypothesized decreases in rates of (a) more restrictive dispositions, (b) out-of-home placements, and (c) more intensive supervision levels. We also hypothesized that, in accordance with the risk principle, (d) restrictive dispositions, placements, and levels of supervision would be positively related to risk, and (e) high-risk youth would receive more services than lower risk youth. Lastly, we hypothesized that (f) reoffending would not increase after implementation of the RNA and RNR” (p. 685).

A pre-post, quasi-experimental design was utilized to collect data from 6 county probation offices and 1,694 propensity score-matched offenders. The sample included cases that met the criteria for a RNA to be implemented. Juvenile Probation Officers (JPOs) were trained in risk/need assessment and were told about the implementation procedures. Then, policies for risk/need assessments and case planning were created. Administrators tailored the case plans to be consistent with criminogenic needs identified on the RNA. Finally, JPOs were trained on the fundamentals of the RNR approach to treatment, and completed workshops on the YLS/CMI or SAVRY. Each JPO used information from the juvenile’s file and interviews with the juvenile, parent, and juvenile and parent together. Semi-structured interview scripts were given to JPOs who also administered the SAVRY or YLS/CMI. The study examined four dependent variables: most restrictive disposition, out of home placements, community supervision, number of service referrals, and recidivism.

Results

“There was a significant shift toward less restrictive dispositions in four of the five sites. Risk level was associated with at least some, if not all, disposition decisions in each of these sites, indicating the risk principle was followed” (p. 693).

“As hypothesized, the risk principle influenced placement decisions in all five sites. On a positive note, most sites placed only 50 to 75% of their high-risk youth, suggesting the RNA and RNR training was effective in communicating that many of these youth could be managed safely in the community. A crucial takeaway is that the direction of the impact of the RNA differed as a function of each site’s placement rates before the RNA was used” (p. 693).

“The most consistent area of impact of implementation of a RNA was on level of probation supervision. In every site at which supervision level could be tested (with the exception of YLS/CMI Site 3) there were significant decreases in the use of maximum and moderate levels of supervision, and increases in the use of minimum levels of supervision…Most sites followed the risk principle in their service allocation such that high-risk youth on average received one to three more services than low-risk youth” (p. 693).

“Consistent with the hypotheses, in most sites rates of new petitions [rates of recidivism] did not change [except for site 1]… The stability in recidivism rates after implementation of a valid RNA and RNR approach may be a surprise and a disappointment. After all, the primary benefit of the RNR approach has been touted as recidivism reduction. However, most studies that have reported RNR leads to reductions in recidivism have demonstrated this at the macro level for services that address criminogenic needs (Dowden & Andrews, 1999; Romani, Morgan, Gross, & McDonald, 2012), for young offenders with strong service-to-need matching (Luong & Wormith, 2011; Peterson-Badali et al., 2015; Vieira et al., 2009), or in comparisons between POs with and without intensive training in RNR-related case management (Bonta et al., 2013, 2011). The present study differed in that it examined whether implementation of a valid RNA with RNR-related policies led to reductions in reoffending within a jurisdiction as a whole” (p.694).

Although it is important to implement the RNR approach in juvenile justice agencies, “we stress that focusing on recidivism as the most important or even sole outcome variable of interest when studying RNA implementation could be detrimental. Future studies should examine other outcomes such as reduction in agencies’ human and financial costs and improvement in youths’ educational attainment or employment” (p. 695).

Translating Research into Practice

“A major implication of this study is that quality implementation of risk assessment and RNR will conserve resources and serve more justice-involved youth in the community without an increased risk to public safety…Another implication of this work is that agencies must implement the RNA well to reap its benefits and avoid losses of time and other resources” (p. 694).

“A final implication is that the actual RNA instrument used should not matter as long as the tool is being completed reliably by staff and has been demonstrated to be a valid predictor of delinquent behavior for the type of population and setting where it is being used” (p. 694).

Other Interesting Tidbits for Researchers and Clinicians

The present study did not examine the quality of implementation of the need principle at any of the sites when measuring recidivism. Although the JPOs were trained, there was no standard assessment or evaluation for program delivery or management. Additionally, the authors speculated that sufficient time was not given to see any significant reductions in reoffending. “Good implementation of a new intervention can require 3 years, particularly before any benefits can be realized (Fixsen et al., 2005; Flores et al., 2006)” (p. 695). Finally, developing evaluations for all three stages of the RNR model during the time of implementation would allow future researchers to isolate both the efficient and inefficient aspects of the treatment.

Join the Discussion

As always, please join the discussion below if you have thoughts or comments to add!

Authored By Sarah Hartigan

Sara Hartigan is a second year Forensic Psychology Master’s student at John Jay and hope to obtain a Ph.D. in Clinical Forensic Psychology in the future. My main areas of interest include clinical evaluations and developing treatment interventions within the forensic population.

YLS/CMI and SAVRY helpful for Juvenile Probation Officers’ case planning decisions

Forensic-Training-AcademyJuvenile Probation Officers find the YLS/CMI and SAVRY helpful in guiding case management decisions, particularly when sound implementation strategies are employed, allowing frontline users to benefit from empirically based developments in risk assessment. This is the bottom line of a recently published article in the International Journal of Forensic Mental Health. Below is a summary of the research and findings as well as a translation of this research into practice.

Featured Article | International Journal of Forensic Mental Health | 2014, Vol. 13, No. 3, 227-241

 

What Do Juvenile Probation Officers Think of Using the SAVRY and YLS/CMI for Case Management, and Do They Use the Instruments Properly?

Authors

Laura S. Guy, Psychiatry, University of Massachusetts Medical School, Worcester, Massachusetts, USA
Rebecca J. Nelson, University of Massachusetts Medical School, Worcester, Massachusetts, USA
Samantha L. Fusco-Morin, Psychology, Fordham University, Astoria, New York, USA
Gina M. Vincent, Psychiatry, University of Massachusetts Medical School, Worcester, Massachusetts, USA

Abstract

Juvenile probation officers (JPOs; n = 71) in the United States were interviewed three and ten months after the SAVRY or YLS/CMI was implemented in their office. Quantitative and qualitative analyses were used to explore their experiences using the instruments and adherence to practice guidelines. JPOs typically perceived the instruments as being ‘somewhat’ or ‘very’ helpful for guiding their case planning decisions. A frequently cited barrier to using both instruments in practice related to the increased length of time it took to complete reports; yet, at the same time, some JPOs also acknowledged that use of the measures forced them to gather important information about the youth’s background and current situation that proved useful. Most JPOs (77%, n = 33 of 43) using the SAVRY expressed preference for a risk assessment model that emphasized use of appropriate professional discretion rather than a score-based approach. “Buy-in” for the instruments and the reported difficulties varied across sites. The present findings may inform recommendations specifically for delivering training on the SAVRY and YLS/CMI and, more broadly, strategies to promote their effective implementation in juvenile justice settings.

Keywords

juvenile probation, risk assessment, implementation, SAVRY, YLS/CMI

Summary of the Research

Juvenile probation officers’ (JPOs) “decision-making regarding the appropriate programming for youth can lead to significant reductions in re-offense rates particularly with respect to various intervention programs (e.g., individual counseling, behavioral programs, interpersonal skills) and appropriate matching to services. Despite the availability of instruments in juvenile justice to structure decisions about issues highly relevant to disposition, such as risk for future offending, juvenile justice professionals tend to rely on unstructured professional judgment. Meta-analytic data demonstrate this approach to be substantially less accurate than structured approaches. Therefore, it is noteworthy that, despite the pervasiveness of the use of unstructured professional judgment for assessing risk and formulating intervention and case management plans, jurisdictions that follow the recommendations in the Juvenile Justice and Delinquency Prevention Act (JJDPA, 2002) now have in place some form of structured instrument for assessing risk” (p. 227).

“This study investigates JPOs’ expectations of and experiences using the YLS/CMI and SAVRY for case planning with probationers. It extends earlier work examining the implementation of these instruments in the Risk/Needs Assessment in Juvenile Probation: Implementation Study (RNAJP).” Three study aims were examined, including “JPOs’ broad perceptions about the usefulness and difficulties of the instruments; JPOs’ experiences using the instruments with respect to (1) rating specific items and (2) making overall ratings about risk level; and among SAVRY users only, the degree to which JPOs reported rating the items and making a summary risk rating (SRR) in a manner consistent with the SPJ model’s concepts of manifestation, relevance, and linearity” (pp. 229-230).

Participants were 71 JPOs sampled from six probation offices that participated in the RNAJP study; three offices implemented the YLS/CMI, and three offices implemented the SAVRY. “JPOs participated in two-day training workshops conducted by a co-author of the instrument, typically completed three post-workshop practice cases and received feedback, and participated in booster training six months following the initial workshop. Eight trained researchers interviewed JPOs three times about their case management practices and experiences supervising probationers: prior to implementation of and training on the risk assessment instrument, three months after implementation, and 10 months after implementation. Only data from the two post-implementation interviews are reported here, and subsequently are referred to as the first and second interviews, respectively” (pp. 230).

Aim 1. Perceptions About the Usefulness and Difficulties of the Instruments

“Overall, the majority of users of both instruments perceived them to be ‘somewhat’ or ‘very’ helpful for making recommendations about disposition, services, and level of supervision across both follow-up periods. The most frequently identified theme was use of the [YLS/CMI] to “back up” their opinions about risk level and recommendations regarding services and level of supervision, which they believed they would have reached using only their professional experience” (pp. 229).

“JPOs using the SAVRY provided more comments about usefulness than YLS/CMI users, [primarily] SAVRY’s emphasis on professional judgment or lack of reliance on a total score, enhanced data collection that led to more useful recommendations, and increased knowledge about risk factors. Several JPOs indicated their knowledge about empirically based risk factors increased after completing the SAVRY training workshop and becoming familiar with the manual” (pp. 232)

Aim 2. Experiences Rating Specific Items and Making Overall Risk Ratings

Thirty-six percent of JPOs found specific items [on the YLS/CMI] difficult to rate. “Items in the Attitudes/Orientation domain were identified most frequently, followed by the Substance Abuse and Leisure/Recreation domains.” Fifty-seven percent of JPOs found one or more [SAVRY] items difficult to rate, most frequently among the “Social/Contextual scale: Peer Rejection, Stress and Poor Coping, Lack of Personal/ Social Support, and Community Disorganization. Over 90% of JPOs using the SAVRY reported at the first interview that making the SRR was ‘very easy’ (52%) or ‘somewhat easy’ (39%). By the second interview, the majority (23 of 44, 52%) reported the SRR was ‘very easy’ to generate. Among JPOs who reported some difficulty, 23% indicated they had insufficient information, 21% cited lack of clarity in the manual’s item description, and 11% reported that the source difficulty varied depending on the particular item” (pp. 234-235).

Aim 3. Among SAVRY Users, To What Degree Did JPOs Report Using the Instrument in a Manner Consistent With Principles of the SPJ Model?

“Responses indicated that just over 90% of JPOs’ understood that a single risk factor could be present for two youth for different reasons. Most JPOs (89%) replied in a manner that suggested that they understood the concept of individual relevance well. However, only a minority of respondents indicated (correctly) that youth with the same risk factors present would not necessarily have the same level of overall risk” (p .235).

Translating Research into Practice

“JPOs perceived both the YLS/CMI and SAVRY as being ‘somewhat’ or ‘very’ helpful for guiding case planning decisions. Most SAVRY users demonstrated good understanding of key components of the SPJ model. Both measures generally were experienced as being easy to use, including the aspect of the SAVRY requiring the most discretion – developing the overall summary judgment about the youth’s risk level. The majority of SAVRY users expressed preference for risk assessment procedures that called for professional discretion rather than a score-based approach. Finally, [these] results suggest that many anticipated barriers to putting a risk assessment instrument into routine practice can be overcome when sound implementation procedures are followed, allowed frontline users and agencies more generally to benefit from research based developments in the science of risk assessment” (p. 239).

“High-quality implementation strategies are crucial in probation settings for risk assessment practices to effectuate change in decision-making. Implementation is difficult in organizations in which staff members are cynical about the changes. Thus, quality training with and buy-in from JPOs appear to be necessary pre-requisites to successful implementation of risk assessment systems” (pp. 237).

“Results of the present study offer guidance for areas of curriculum that should be emphasized during training on risk assessment with youth populations.” Some of the difficulties in rating items were related to insufficient information and unclear rating instructions in the manual. “Clarification [on the YLS/CMI] regarding how to rate dichotomous items when users feel the case information is more nuanced appears warranted during training workshops. Regarding the SAVRY, trainers should devote ample time for instruction on the Social/Contextual risk factors. With respect to training about some of the standard SPJ assessment practices, our results indicate the concept of manifestation is relatively easy for JPOs to understand. [These] findings suggest that SAVRY workshops could be improved by including more explicit instruction regarding the need to consider the idiographic relevance of each risk factor for the particular youth being evaluated, as well as guidance about how to apply information about items’ relevance when making the SRR.”

Other Interesting Tidbits for Researchers and Clinicians

“Other minor themes observed related to positive aspects of the SAVRY included the promotion of objectivity and transparency in the risk assessment process, having a research based procedure “back up” their professional opinion and recommendations, the utility of the SAVRY for tracking changes in risk over time, ease of communication between professionals trained in the same instrument (e.g., speaking the “same language”), and increased ability to “pinpoint” the most critical criminogenic needs to be targeted for treatment” (pp. 233).

“Despite the additional guidance of the YLS/CMI for identifying need areas (in the form of scales with explicit labels), there appeared to be little difference in JPOs’ perceived helpfulness of these instruments for making recommendations about disposition, service referrals, or supervision level, albeit this was not the main focus of the study. Developing a case plan with a high likelihood of reducing a youth’s risk requires JPOs to identify and then triage the youth’s most important dynamic risk factors (also known as criminogenic needs) to be targeted for intervention. SAVRY users mentioned a benefit of the instrument was its ability to identify dynamic risk and protective factors; therefore, the instrument ostensibly should lead to quality case planning. How well disposition recommendations and case plans align with youths’ criminogenic needs and whether the alignment differs between the YLS/CMI and the SAVRY are areas in need of further study” (p. 239).

Join the Discussion

As always, please join the discussion below if you have thoughts or comments to add!

Contributing AuthorBanfordMegan-pic

This post was authored by Megan Banford.

Megan is a graduate student in the Forensic Psychology program at John Jay College. She graduated in 2013 from Simon Fraser University with a B.A. (Honors) and hopes to attain her PhD in clinical forensic psychology. Her main research interests include violence risk assessment and management, juvenile offenders and public policy.