Evaluating evaluations

[Part three in a series on educational reform. See below for parts one and two.]

Update: If you want a good summary of the standardized tests vs. portfolio debate, read this article by Jay Mathews of the Washingon Post. Read it, however, in the context of everything I've been saying so far regarding the underlying goals of education and questioning the oft-unquestioned logic of evaluation.

Let's talk about subjectivity, and let's talk about evaluation -- which is a roundabout way of saying, let's talk about testing.

Pragmatically, schools need some system of evaluation. Ideologically, that's not as clear-cut; any system whose external logic includes a measure on which students will be ranked offers an incentive to do only what it takes to achieve that self-interest rather than just learning because learning makes you a productive, intelligent person. However, we live in a world of supply and demand, and colleges and employers only have so many slots to fill and need some way to discriminate among applicants. Moreover, it would be nice to have an occasional method of check progress in elementary, middle and high schools, if for no other reason than to prove that our proposed paradigm works.

So there's the set-up of the problem: We need some system of evaluation, but it can't be one which detracts from the goal of learning how to think rather than learning things to think.

Option 1: Standardized multiple-choice tests. The bulk of current evaluation is done through this method. Prominent examples include the Virginia SOLs and the SAT (though it's adding an essay section -- more on that later). The supposed advantage of this system is twofold: universality and objectivity. Children in Topeka and San Diego are answering the same questions, and there's only one right answer.

The problem with this, of course, is that standardized multiple-choice tests offer neither universality nor objectivity, and worse yet they encourage teaching rote facts rather than comprehensive understanding. Demographics play a role, much as many people would rather they not; a dilapidated school with 75% of its children on free or reduced lunch and full of refugees or illegal immigrants is not on the same playing field as an affluent counterpart. Results from the two schools are not reflective of the actual learning going on inside.

Then we get to objectivity, and I want to digress for a moment to talk about this in depth. Setting aside for a moment the fact that multiple choice questions rarely have a truly objective answer (If you asked "What is the fastest way to get to the store?" with pictures of a horse, a boat, a car and a person walking, the answer to a kid who came out of a Kenyan refugee camp might be walking -- they don't have roads! Of course, that would be the "wrong" answer), let's talk about whether objectivity is necessarily superior.

Objectivity, of course, stands in contrast to subjectivity. But we often indict subjectivity without thinking about why. Consider the analogy of a judge. Here we have a person who has gone through extensive training, moved up through a highly competitive profession and has a wealth of experience. Each case before our judge has different merits, different circumstances, different people. The entire point of a judge is to be able to do just that -- judge. Yet, over recent years we have seen a huge push for mandatory minimums, a matrix box which pre-determines sentences for certain drug offenses and strips the judge of his discretionary power. So, the 18-year old kid with no criminal record headed for Stanford whose dad just died of a heart attack and is caught with the tiniest amount of crack HAS to be thrown away for 5 years. The judge might speak out passionately against the sentence at the same time he's delivering it, saying that the kid clearly posed no danger to society and that the penalty was brutally excessive -- but his hands are tied. Discretion, and subjectivity, are not necessarily vices. Especially when you're dealing in a realm where every situation, every schoolchild is intrinsically different, how can we reduce their achievement to a matrix box, to a bubble sheet?

Obviously, teachers right now are not vetted to the extent of a judge, but under our reforms the average quality of the profession would skyrocket (that's for a later post). Assuming for a moment that teachers are as qualified as judges, and it seems equally ridiculous to slap mandatory minimums on them; yet, that is precisely what standardized multiple-choice tests are. Both are "objective," both are "consistent," and both are "fair." Neither, of course, is any of those things.

So, what are some other options?

Option 2: Portfolios. A portfolio evaluation is exactly what it sounds like; the student compiles all of his or her work from throughout the year and presents it to a panel usually consisting of his or her teacher, one school administrator, and another teacher from the same grade. There is an emphasis in this system on oral presentation and in-depth reports. Tests are sometimes taken completely out of the equation.

The costs and benefits of portfolios should be obvious; they provide a far more comprehensive review of a student's true understanding, but the results are hard to compare across the board and the panel has an incentive to evaluate highly if that's the metric that will be used to review the school. Also, it is much easier for a parent to cry foul if her or his kid doesn't pass muster. So while this is an ideologically superior method to standardized MC tests, it provides a host of logistical challenges which some proponents have found prohibitive.

Option 3: Hybrid (My proposal). If you can't get rid of standardized tests altogether, at least set them up so that the more you learn how to think, the better you will do. This means an emphasis on short answer and essay questions -- no multiple choice. Moreover, they have to be well-designed questions that truly test understanding.

For instance, one of the best essay questions I've ever had in college was in my history of the Middle East class when I was asked, "Explain why the 1906 Persian revolution failed while the 1979 revolution succeeded." Here was a question which you could take in any of a thousand different angles; it was interesting; it was a topic that there is no consensus on whatsoever; it wasn't something I could bulls*** around -- I had to know my stuff. However, a bad question, and one you're more likely to see on essay questions in high school, would have been "What was the cause of the 1979 Iranian revolution?" That doesn't necessarily demand insight or higher-level understanding, just paying attention in class/to the readings.

If you can line up what tests demand and what you should be learning (i.e. methodologies of thought), then you can circumvent the problem of teaching to the test while maintaining the integrity of a universal testing regime which society demands. Oh, and take demographics into account.

Now, the primary problem with my system is the same as with portfolios -- grading of essays is subjective, and teachers might have incentive to not grade accurately. This is where the SAT is encouraging. There is obviously enough of an understanding that MC is not holistic enough, because they're adding an essay section (which will be subjectively graded!) to the one test that nearly every college-bound high school student in America takes. That hurdle might not be so insurmountable as originally thought.

This post is incredibly long, but I want to make one last point about evaluation: My junior year H.S. physics teacher used a 4-point scale for grading instead of the standard 100-point scale. Why? He explained simply, "I don't know the difference between a 87 problem set and an 88 problem set. I do know the difference between a 3.5 and a 3." One way to dampen the subjectivity "flaw" is to establish broad categories of evaluation that guide -- but don't constrain -- the teacher's opinions.

Evaluation is a fundamental part of modern American education, and it can't be ignored in the midst of reforms because it influences so heavily the direction of classes. It can't be taken away altogether, so it has to be made to work for us.

Find me a bubble sheet that can express all that.

- posted by Elliot H. @ 1:03 PM