Limitations of Ed Research

Chris at A Blog About School has a post up about “the idea that empirical research in social science can dictate indisputable policy solutions that we just need to impose uniformly on all of our schools.”  I want to follow up with some thoughts of my own, but it makes for too long a post to politely leave as a comment.  Besides, Blogger seems to think that I am very probably a robot; WordPress isn’t worried about whether or not I am human.

The Gold Standard

The gold standard for research is a double-blind, random assignment study using a large sample size.  For obvious reasons, education research isn’t conducted this way but it is important to recognize the possible shortcomings of education research studies and determine whether the researchers have adequately controlled for these shortcomings.  Is the sample size large enough?  Have they established correlation or causation?  If there is no random assignment of students, has that compromised the outcome of the study?  [Insert your own questions here.]

These problems are not insurmountable, but we ought to be conscious of the difficulties in constructing reliable studies.  Lotteries for vouchers and charter school admission, for example, provide an opportunity for random assignment to control or treatment groups.  Research in other disciplines, like psychology, may shed light on education issues.  [See, for example, the writings of Daniel Willingham, Professor of Psychology.]

Lies, Damned Lies, and Statistics

A problem with the use of research- and evidence-based decision-making in education is research can be misused.

Sometimes research is distorted or misrepresented.  Perhaps the research demonstrates correlation, but it is treated as if it has established causation.  Perhaps important details are glossed over.  For example, a research study may show that a small sample of disadvantaged children participating in a very expensive and involved preschool program showed better school outcomes and fewer arrests than disadvantaged children not participating in the preschool program but this result is turned into “research shows that universal, conventional preschool programs will create better outcomes for all children.”

Sometimes the research isn’t research at all.  Research- and evidence-based is sometimes used in a way that suggests that the program or policy has been scientifically-validated when, in fact, the program or policy represents an untested idea based on some other research or evidence (which may or may not have been distorted or misrepresented by the policy or program proponent).

Sometimes “research- and evidence-based” is used to shut down discussion.  Who among us has the authority to question science (even if, as it may often happen, there are education experts on both sides of the issue)?  The suggestion of scientific-validity can lull us into forgoing critical-thinking about important education issues, even when we can observe for ourselves that the programs or policies are not delivering results as promised or desired.

Discoverable Truths

Which leads to an important point (made frequently by Chris at A Blog About School and others, but cannot be repeated too often): many (most?) education policy decisions are about values rather than about scientifically-discoverable truths.  This is not to say that education research cannot be valuable, but that science cannot answer all education policy questions for us.  Science cannot tell us whether the mission of public schools ought to be to create future workers or to introduce children to the liberal arts, for their own sake.  Science cannot tell us whether children should read Shakespeare or learn French or not.  Science cannot tell us that we should prefer online music courses to in-person, group orchestra or band classes.  Science cannot tell us whether we should spend scarce resources on a new theater or an improved athletic field.

Which I guess means, keep your critical-thinking caps handy and don’t hesitate to ask questions, like:  “What does the research really show?” and “Is this a question that ought to be settled solely by research or is it more a matter of values and preferences?”


2 thoughts on “Limitations of Ed Research

  1. Chris

    So many good points here. I’m working on a new post on the same topic, and you’ve taken many of the words out of my mouth.

Comments are closed.