Evaluation: Changing the Measure of Success

The Claremont Evaluation Center is helping to determine if a nonprofit’s college-readiness programs are proving effective
How do we measure a student’s probability for success after high school? Should educators consider the traditional benchmarks of academic achievement: grades, test scores, and IQ? Lisa Teachanarong, PhD student in the School of Social Science, Policy & Evaluation (SSSPE), says such measures are only part of the story.

In the course of her research, Teachanarong is broadening how we define high school students’ success and college readiness. While test-taking skills and critical thinking are important, they don’t capture the full picture. As it turns out, skills like mindfulness, emotion regulation, and interpersonal effectiveness are just as, if not more, important, according to Teachanarong.

Teachanarong, a doctoral student in positive developmental psychology, has spent the last three years conducting research for Project GRAD Los Angeles (PGLA). PGLA is a nonprofit organization dedicated to helping low-income minority students in the San Fernando Valley navigate a path to a college degree. They do this primarily through long-term mentorship and classes focused on social-emotional skill development.

Social-emotional skills are noncognitive or “soft skills” that allow us to manage stress and build healthy relationships with others. Though not traditionally taught in public school systems, Teachanarong believes in the ability of institutions like the public school system to promote social-emotional development. She uses physical education as an example.

“Physical education won’t help with students’ math or writing skills, but we don’t leave physical health up to chance,” Teachanarong said. “Public education created a system to make sure that all kids get some kind of exposure to physical health education. Why should social or emotional education be any different?”

PGLA students take after-school classes that teach these social-emotional skills, like mindfulness, distress tolerance, and emotional regulation. Students are also assigned a mentor that follows them throughout the seven-year program and meets with them one-on-one to set personal and academic goals, discuss challenges, and apply the skills learned in the classes.

Students that are at risk don’t always have good role models to rely on when they encounter stressful situations, according to Teachanarong. “Some of them get to college, and maybe they’ve been underprepared academically, so they get their first F. They go through this crisis of thinking they’re not meant to be there and drop out,” she said.

To prepare PGLA students for scenarios like these, PGLA staff teaches their middle and high school students about the social-emotional skill “growth mindset,” which is the idea that your intelligence is malleable.

“The hope is that if we can train students to think with a growth mindset, when they encounter this sort of challenge they will see schoolwork as hard, but their intelligence can be moved. It isn’t this static thing that means they’re not cut out for college,” she added.

Teachanarong conducts her research as a project manager through the Claremont Evaluation Center, under the supervision of Tiffany Berry, associate professor in the Division of Behavioral and Organizational Sciences (DBOS). In order to help PGLA and other organizations like it, the Claremont Evaluation Center spends the first several years in their partnership with a program, assessing the organization’s goals and services, examining its underlying theory, and giving data-driven feedback to help them operate at optimal capacity. Then they take a look at the outcomes, evaluating whether the program is achieving its intended aim. Teachanarong’s job is to determine if PGLA’s programming is proving to be effective in their ultimate goal: getting students to college and helping them stay there until graduation.

But evaluating whether the theories behind the organization’s program can stand up to scientific scrutiny is only part of the job. Another critical part of the project includes finding an appropriate comparison group—students who face similar challenges to those in PGLA but are nonparticipants—and tracking them to approximate PGLA’s impact on its students.

With the help of the Claremont Evaluation Center, PGLA ultimately hopes to narrow the gap in the quality of education that socioeconomic conditions can create.

“These students who are at high risk for low academic achievement also tend be at higher risk for social and emotional difficulties, given the amount of stress they have to face,” Teachanarong said. “Let’s come together as a group of people and make sure that institutions are helping to foster social and emotional growth so that all students have the tools they need to succeed.”

To watch the “15 Years of Project GRAD, 15 Years of Accelerating Equity” video, click on the link below: