You are hereBlog / Developmental Evaluation: YPSN style (Part 1)

Developmental Evaluation: YPSN style (Part 1)


By petra - Posted on 12 September 2012

Transformation in people's lives happens all the time. We see it every day in our work. How do we know what role our programs have in those transformations?

In the beginning, here at Young Parents Support Network, we forged our own approach to program evaluation, because pre-existing models just did not serve our community. Square peg: round hole, you know how that is.

Our approach to evaluation evolved over time, to support our primary goal of adaptation to changing conditions to ensure meaningful service delivery to young parents and their children. Our homespun evaluation style also supported our secondary goal of creating an organization that fostered ongoing development of all kinds (professional, personal & transpersonal) for all members of our community (participants, staff & volunteers), as we saw evaluation as an opportunity to continuously and humbly engage in a co-creative learning and development process with the families we served. We considered evaluation to be embedded in our program development and delivery cycle, and found creative ways to meet & exceed reporting requirements for our funders, while still remaining true to our belief that evaluation could be emancipatory, participatory and useful!

Then we discovered Developmental Evaluation, an approach developed by Dr. Michael Quinn Patton, and we cheered aloud because it was exactly what we had been attempting to create by muddling along and devoting whatever time & resources we could, while still keeping front-line service delivery at the heart of everything we do. Dr. Patton suggested we"experiment, pay attention to what happens, be ferocious about getting feedback, and learn by doing”. Yes! We agreed. That's what we do! Every day! Not only did we discover that our oddball approach to evaluation had a name, but it was also rigorously researched, reality-tested, and backed by current research. So here it is. Our first crack at describing Developmental Evaluation: YPSN-style. Our gratitude to Michael Quinn Patton is vast and enduring.

 

Evaluation: What & why

What is it we are trying to evaluate?

Change. Lives. Processes. Practice. Learning. Experimental programs. Established programs. Our ways of thinking. Our ways of behaving. Transformation (or lack thereof).

Why?

For two reasons, both equally important: 1.) we want to ensure that our services are as meaningful as they can be for the young parent families and the communities we serve; and 2.) we are accountable to our funders to provide accurate and insightful information about the change that their dollars are purchasing.

 

Developmental Evaluation

Traditional approaches to program evaluation have been based on a logical view of the world in which inputs (dollars, staff time, 'best' practices) can be linked to outputs (changed lives), and, in fact, some situations are fairly logical & predictable (for example, if you feed hungry people, they will be less hungry). But when programs are participatory; participants lives are complex; and conditions are changing rapidly resulting in a need for almost continual adaptation; we believe it is arrogant (actually) and oppressive (in many cases) to assume that certain inputs will result in particular outputs in people's lives.

And yet transformation in people's lives happens all the time. We see it every day in our work. So how do we know what role our programs have in those transformations? Through an approach to evaluation that is designed to explore, and when possible capture, the complexity of real life. With Developmental Evaluation!

 

What is Developmental Evaluation?

We'll let just Michael Patton describe it:

  • "Traditional evaluation aims to control and predict, to bring order to chaos... Developmental evaluation adapts to the realities of complex linear dynamics rather than trying to impose order and certainty on a disorderly and uncertain world".
  • "Social innovators [that's us!] tend to be so busy engaging that they fail to systematically track what is developing and document the reasons they choose one path over another at critical forks along the innovation road. The developmental evaluation helps identify the dynamics and contextual factors that make the situation complex, then captures the decisions made in the face of complexity, tracks their implications, feeds back data about what's emerging, and pushes for analysis and reflection to inform next steps, and then the cycle repeats".
  • “Developmental evaluation acknowledges uncertainty, expects it, and accepts that evaluation can increase complexity even while attempting to understand it, by being one more factor among the many already operating and interacting. A primary strategy for coping with uncertainty applies to both programs and evaluations and the interactions between the two: shorten and speed up the feedback”.
  • “Developmental evaluators must anticipate emergence, track emergenct interactions among key players, both formal and informal, both planned and unplanned. Map networks, system relationship, and self-organizing subgroups. Track information flows, communications, and emergent issues. Emergence applies to both processes and outcomes. Watch for and assess not only what emerges, but what declines or even disappears. Disappearance is the other side of the phenomenon of emergence”.
  • “The sources of nonlinearity, emergence, and unpredictability are deeply enmeshed in the complex web of relationships that we all experience... on a broader systems change level, the developmental evaluator would also help track and document changes and developments in societal norms and how policy makers and the general public are coming to understand and react to emergent... issues, and how those societal developments interact with and effect local interventions and innovations”.

 

Adaptation & Evaluation

At YPSN, our goal is for evaluation data to be readily & freely available to support adaptations to ensure that our service delivery is as meaningful as it can be.

In Developmental Evaluation, 'real time' responsiveness ensures that information is readily available. As Dr. Patton explains, 'real time' “refers generally to rapid feedback and response, linking data and action as close together in time as possible... in evaluation situations, real time typically means getting results to intended users in a day or two".

Non-hierarchical structures (our board of directors is comprised of young parents), participatory processes (young parents are involved in all major decisions and program development), and a commitment to reflective practice (we talk!) all help to ensure that evaluative information is freely available. At YPSN, we are fairly adapt in this area but we need to continue to work on improving our communication overall and this has been a goal for some time.

Readily & freely available information supports a culture of adaptation. Developmental evaluation provides information, and it also tracks the adaptations that are made as a result of sharing this information, documenting the learning that occurs, so that 'lessons learned' aren't lost and (hopefully) don't need to be re-learned!

Developmental evaluation itself is inherently adaptive, in that any evaluation methods (from ultra-logical-old-school or emergent-new-paradigm) can be used, as long as they are rigorous, appropriate to the context, and allow for rapid responsiveness and adaptivity. Doc Patton says: “Whatever methods are used or data are collected, rapid feedback is essential. Speed matters. Dynamic complexities don't slow down or wait for evaluators to write their reports, get them carefully edited, and then approved by higher authorities. Any method can be used but will have to be adapted to the necessities of speed, real-time reporting, and just-in-time, in-the-moment decision making”. A robust toolbox of evaluation approaches is required! We are adding to our toolbox all the time...

Actually, getting locked-in to any evaluation methods or predetermined outcomes is the antithesis of Developmental Evaluation, as these may need to be adapted as the innovation unfurls. Dr. Patton assures us that "measures can change during the evaluation", which causes us to rejoice, because how can we know in advance precisely what will occur and therefore what to measure?

We once had a respite care program which provided young parents with childcare in their homes so that they could attend to self-care. Radical concept, and it really worked. Almost unanimously, young parent participants reported reduced stress and improved parenting all month long, just knowing that they were going to get a respite session once a month. We anticipated that. What we didn't anticipate were some of the other powerful impacts and outcomes of the program, including that some participants chose to stay home with the respite care provider in order to learn how to interact with their children (it provided mentorship & parenting education); others never booked respite care but told us that they deeply appreciated that their respite care provider would call them every month to see if they needed help and talk to them about their kids (it provided community); others were able to access emergency respite care that kept their children out of foster care, therby disrupting multi-generational cycles of children raised in government care (it provided a safety net). Our evaluation measures changed as the program evolved. Likewise, the program evolved as we worked with participants to learn what was meaningful. The program supported hundreds of families, and then funding for it came to an end, but we learned a lot!


Adaptation & practice

Clearly, adaptation is fundamental to YPSN's practice style, and ironically, this can be one of the most challenging aspects of adapting to our organization for new team members. Adaptation, or 'situational responsiveness' as Michael Q. Patton calls it, necessarily requires a base of experience from which to make spur-of-the-moment assessments to inform seat-of-the-pants adaptations. Building this capacity is one of the reasons why we believe so wholeheartedly in ongoing professional development and provide weekly mentoring meetings for our front-line staff with our Perinatal & Family Support Coordinator, Jacquelin.

Describing how and what it is exactly that we do in practice can be challenging, and we grapple with this all the time at YPSN. “Most front-line staff in successful programs can testify that while they operate on a body of shared knowledge and skills, a significant portion of what they do cannot be standardized... We underestimated the subtleties of effective interventions. Even the best practitioners often can't give usable descriptions of what they do” (Schorr, 1997). Developmental Evaluation should be able to assist with creating 'usable descriptions' of adaptive practice in complex situations through provoking reflective practice and conversation as adaptations are underway, and documenting these conversations.

 

Adaptiveness: the shadow side

It's easy to see the benefits of adaptiveness, both for practice and for evaluation, but adaptiveness has it's shadow side, and developmental evaluation can also track these and their impacts on programs and people.

  • Being in a state of continual change can be stressful.
  • Attention must be paid to keeping communication flows open so that stakeholders are fully included as adaptations occur.
  • Charismatic leaders and other community members need to ensure that they serve the community and do not unduly influence the opinions of others.
  • Attachment in the status quo needs to be attended to as both an important tempering influence and a possible hindrance to innovation.
  • Inexperienced practitioners (as well as experienced ones) will make mistakes when they are 'set free' and encouraged to be adaptive, and this inevitability needs to be considered part of an organizations risk management strategy.
  • Questions emerge: How much adaptiveness is tolerable/desirable? Who decides whether a particular adaptation is in the highest & best interest of the community? How much control do leaders need to have over the adaptiveness that is occurring?

 

A culture of adaptiveness can lead to a misty swamp full of boundary quandries... so we'll need a flashlight for our next exciting installment of 'Developmental Evaluation-YPSN style'. Stay tuned!