Why the ‘Field of Dreams’ Approach to Data Doesn’t Work

If You Build Good Systems, the Answers Will Come … Right?

by Brad C. Phillips and Jordan E. Horowitz

How can educational institutions use data to help students? The answer involves more than just buying the right software package, explain alumni Brad Phillips (PhD, Psychology, ’96) and Jordan Horowitz (MA, Psychology, ’87). In the following excerpt from their new book Creating a Data-Informed Culture in Community Colleges: A New Model for Educators, the authors explain that supporting student success requires more than just generating good data.

Marsha Chapman directs the institutional research office at her college, which has about fifteen-thousand students, is located in a suburban setting, and is considered a “good” community college. Marsha has one research analyst and one part-time administrative aide on her staff. She and her analyst spend much of their time extracting data from the student information system, writing research reports, developing survey items, responding to queries from both internal and outside resources, attending meetings, and updating the research database every semester.

To gather their information, they use the latest tools that enable them to extract data in a more efficient fashion, build tables and charts, and add color and emphasis where they think necessary. They post much of this information on both their internal and external websites in an effort to inform the college faculty and staff, as well as the wider community, about the college’s performance. They are tasked with far more than they can possibly accomplish, and at times the work is simply overwhelming.

But although she and her staff circulate so much information so widely, Marsha has no idea how it is used by the college community or viewed by the community at large. She has some sense that the data must be used because college administrators, faculty, and staff frequently request research on a particular interest they have. Unfortunately, Marsha does not have the time to fill these numerous requests. She also does not have the time to write long reports about the research she conducts. Most of her “output” is in the form of tables and charts. In the end, Marsha has no idea if her work is having an effect on improving student outcomes.

The good news is that this college is interested in data use; the bad news is that there are many missed opportunities by using what we call the “study everything” approach. As noted by Tris Lumley, great organizations “are built around great data. This data [allows] them to understand the needs they address, what activities are likely to best address these needs, what actually happens as a result of these activities, and how to allocate resources and tweak what they do for even greater impact.”

There are those who believe that if we just had the right indicator and the right data, educators would be motivated to make the changes in policy and practice that will address their challenges and lead to improved student success. The problem with this paradigm is the belief that data, in and of itself, is enough to change behavior. This thinking does not take into account human nature. According to Heath and Heath, statistics alone are not good enough. Numbers, tables, and charts in and of themselves are not enough to inform and influence. Rather, it is the story behind the numbers that has the ability to impact educators.

Relying on data alone to impact community colleges is the Field of Dreams approach—faith that if you have enough of the right data, problems will solve themselves but many colleges are signing on. The number of companies touting their latest data visualization software and analytics tools has expanded considerably. However, just because a new software can do something doesn’t mean that it is the right something, and colleges are learning this hard lesson. We often hear from colleagues whose colleges have purchased these packages, admonitions to be cautious and make sure to keep your receipts in case your college needs to return the purchase.

Let us be clear—we are not saying these tools are bad. What we are saying is that these tools are not the panacea for effective data use. That’s because these tools ignore the fact that human beings need to interface with information; just increasing access to information or being able to slice and dice and order data in a variety of ways only gets you part of the way there.

Excerpted from Creating a Data-Informed Culture in Community Colleges: A New Model for Educators (Harvard Education Press), 2017. Reprinted with permission of the authors and publisher.

Brad C. Phillips is president/CEO at the Institute for Evidence-Based Change, a nonprofit organization dedicated to helping education stakeholders use data to make informed decisions, collaborate, and act to increase success for all students.

Jordan E. Horowitz is vice president at the Institute for Evidence-Based Change.