New Contributions on
Emphasizing Evidence-Based
Programs for Children and Youth

K r i s t i n  M o o r e ,  P h D
R o s e  A n n  R e n t e r i a ,  P h D C h i l d  T r e n d s

D i a n a  T y s o n ,  P h D
O f f i c e  o f  t h e  A s s i s t a n t  S e c r e t a r y  f o r  P l a n n i n g  a n d  E v a l u a t i o n

P r e s e n t e d   a t   t h e   O A H   a n d  A C Y F   T e e n a g e   P r e g n a n c y   P r e v e n t i o n   G r a n t e e   C o n f e r e n c e
B a l t i m o r e ,  M a r y l a n d ,   M a r c h  1 2 ,   2 0 1 2

Presentation Overview

— Discuss purpose of the Evidence-Based Programs for
Children and Youth Project

— Share themes from the April 2011 Forum

— Describe dissemination plans

— Summarize issue papers

¡  Core Intervention Components: Identifying and
Operationalizing “What Works”

¡  The Importance of Implementation for Research, Practice and
Policy

Overarching Questions

— What are the opportunities and challenges of promoting
evidence-based programs?

— We have identified what we think the evidence-based models are—now what?	How do we reach our goals of improved program quality and accountability?

— What is needed to support successful implementation of evidence-based programs?

— How do we conduct evidence-based (or evidence-
informed) policy work within a growing evidence-base?

Project Goals

— Assemble an advisory group of experts in implementing
EBPs

— Identify the challenges faced by program implementers
as they implement EBPs for children and youth

— Assess the state of the knowledge from research and practice -- across fields -- that helps to address these challenges

— Synthesize lessons learned about replicating and scaling up EBPs and implementing evidence-informed or innovative strategies

Emphasizing Evidence-Based Programs Forum

— Convened experts in April 2011 in research and practice that focus on human service programs to discuss:

¡  Selecting evidence-based programs for replication

¡  Adaptation

¡  Scale-Up

¡  Developing evidence-informed approaches

Key Themes from the Forum

— Implementation quality matters

— Using good criteria for selecting and implementing may be more important that the original evidence alone

— Implementation science can guide program selection

— Research can be used for innovation

¡  When EBPs are not available, there are strategies and bodies of
evidence to inform program development.

Next Steps

— Disseminate products from the Forum in the spring
of 2012 on FindYouthInfo.gov and ChildTrends.org:

¡  Videos

¡  Summaries of the Forum discussion

— Develop a set of issue papers for the field to synthesize the knowledge base and identify knowledge gaps

¡  Experts are creating three issue papers:

÷ Strategies for Identifying Core Components

÷ Importance of Implementation

÷ Developing Evidence-Informed Strategies

Core Intervention Components:

I D E N T I F Y I N G  A N D  O P E R A T I O N A L I Z I N G
“ W H A T  W O R K S ”

How to Identify Core Components

— Authors: Karen Blase and Dean Fixsen, Frank Porter
Graham Child Development Institute at UNC-CH

— Audiences: Designed for the “initial doers”: program developers, researchers (but also practitioners, funders, and policy makers)

— Purpose: Little guidance exists about how to identify
core components of evidence-based programs

¡  Outcome data do not tell the whole story

¡  In addition to outcomes, need to define and understand the
active ingredients that make the programs work

What are Core Components?

— Essential functions or principles that make up a program or intervention and the active ingredients (activities that operationalize each core component)

— Components that are intended to, and have been found through research to, positively affect the proximal outcomes

Benefits of Identifying Core Components

— Focusing on core components and their associated active
ingredients increases our ability:

¡  To focus implementation resources on the right variables

¡  To accurately interpret outcomes and use effective program improvement strategies

¡  To make adaptations that improve fit without decreasing benefits

¡  To replicate and scale-up without program drift

Challenges

— Few programs have adequate and clear descriptions of the core components and their recommended dosage and strength

— Commonly accepted criteria to verify the presence or validation of core components are lacking

— The components named by program developers may not necessarily include the functional core components or all the core components.	More research is needed

Elements of Well-Operationalized Programs

— Clear description of the context for the program
¡  Philosophical principles and values behind the program
¡  Intended population
— Clear description of the core components/active
ingredients
— Description of the activities that operationally define the
core components

— Practical assessment of the performance of the practitioners
¡  Should be highly correlated with intended outcomes
¡  Also called fidelity assessments

Implications

— Greater attention to the definition and validation of core components would be assisted by investments in research agendas to clearly operationalize, measure, and test the efficacy of the core components.

— Asking program developers about core components and fidelity measures helps communities and agencies assess the fit for their community and understand how well specified the program is at this point.

The Importance of Implementation
for Research, Practice and Policy

Importance of Quality Implementation

— Author: Joseph Durlak, Loyola University (emeritus)

— Audiences: Policymakers, funders, researchers, practitioners

— Purpose: To discuss the cost of poor program
implementation and share lessons learned

¡  Poorly implemented programs are a poor allocation of
resources

¡  Inability to achieve impacts for youth

What is quality implementation?

— Quality Implementation = Conducting a program as
intended/planned

— Dimensions of quality implementation:

¡  Fidelity: the extent to which all core components (the critical active ingredients of a program) are replicated faithfully

¡  Adaptation: extent to which program elements other than the
core components are changed

¡  Dosage: how much of the program is delivered

Factors that Affect Implementation

— Community-wide or societal factors
¡  Funding, policies, politics
— Practitioner characteristics
¡  Self-efficacy, skill proficiency, perceived need
— Characteristics of the program
¡  Adaptability, fit with the setting and population

— Characteristics of the organization hosting the program
¡  Positive climate, shared vision, use of data, leadership
— Factors specific to the implementation process
¡  Successful training, technical assistance, feedback to staff

Importance of Program Implementation

— Extensive data demonstrates the link between quality of EBP implementation and effect sizes on outcomes for children and youth
¡  After school literature
¡  Violence prevention programs for youth
¡  Drug prevention programs

— Money, resources, and time spent on poorly
implemented programs is wasted.

— Poor implementation limits the ability to learn from evaluation studies.

Phases of Quality Implementation

— Initial considerations regarding the host setting

¡  Assessment activities (needs assessment)

¡  Decisions about adaptation

¡  Capacity-building strategies

— Creating a structure for implementation

¡  Create teams and an implementation plan

— Ongoing structure support once implementation
begins

¡  TA, process evaluation, feedback mechanism

— Improving future applications

Lessons Learned

— Don’t implement a program on your own.
¡  Draw on the expertise and experience of outside professionals.

¡  Chances of quality implementation increase when multiple stakeholders work collaboratively in a systematic way.
— Implementation is never perfect and varies over
time.
¡  Be patient and anticipate and deal with challenges.
¡  A pilot program is a good idea.
— Practitioners vary in their performance.
¡  It’s important to monitor practitioners’ performance and offer
professional development as needed.

Implications for Practice

— Organizations that host new programs must devote sufficient effort, time, and resources to implementation.

— Investing in evidence-based programs requires investing in quality implementation.

— Funders should allow time and resources for implementation.

A Word from Joseph  Durlak

-----------------------------------------------------------------------------------------------------------0-----------------------------------------------------------------------------------------------------------

Questions

-----------------------------------------------------------------------------------------------------------0-----------------------------------------------------------------------------------------------------------