Hard Facts: Teaching

More wisdom from Hard Facts:

Merit pay for teachers is an idea that is almost 100 years old ahd has been subject to much research.  In one study conducted in 1918, “48 percent of U.S. school districts sampled used compensation systerms that they called merit pay.” … The evidence shows that merit-pay plans seldom last longer than five years and that merit pay consistently failes to improve student performance.  … [Researchers] also showed that cheating [by teachers] was quite sensitive to the size of the incentives provided for enhancing student scores.  … The same problems emerged when merit-pay systems were implemented in the 1980s. … “It is like policy makers suffer from amnesia.” (pp.22-24) …

The evidence strongly suggests that students learn better when they are not graded and certainly not when they are graded on a curve.  … When drill instructors were tricked into believing that certain randomly selected soldiers would achieve superior performance, those soldiers subsequently performacned far better on tasks like firing weapons and reading maps.  (p.38)

Ending social promotion harms students and schools, and the strongest negative effects are found in the best, most rigorous studies.  At least 55 studies show that when flunked students are compared to socially promoted students, flunked students perform worse and drop out of school at higher rates.  One of the most careful studies found that, after controlling for numberous alternative explanations indlucing race, gender, family income, and school characteristics, students held back one grade were 70 percent more likely to drop out of high school.  (p.51)

GD Star Rating
Tagged as:
Trackback URL:
  • John 4

    Very interesting. Any comments on the methodology? For example, comparing flunked students now to socially promoted students now isn’t really the right comparison – I’m more interested in comparing how flunked students in a society that commonly flunks students compare with socially promoted students now. Perhaps even more importantly, I would like to know what effect social promotion and flunking has on the other students: one of the biggest problems with social promotion is that the kids who are behind hold up all the other kids. Even if it turns out to be better for them (the kids who are behind) to be around more advanced students, it doesn’t follow that it is better overall.

    • Aaron

      If you’re looking at the societal angle than helping that bottom 1% could be very valuable.

      I wonder what the correlation between students who flunked a grade and crime is, does that change with social promotion?

  • “The evidence strongly suggests that students learn better when they are not graded and certainly not when they are graded on a curve. ”

    What? This can’t be right. There are any number of subjects I never would have learned if I didn’t have to cram for an exam. I recall most of my classmates as similar in this regard.

    • How much of what you crammed have you been able to remember or use?

      • I remember about as much of it as I remember of things I chose to learn on my own. I guess I don’t use much of it, but then I don’t really use all the random factoids I read on wikipedia either.

      • Jess Riedel

        Is the claim really that students will suddenly start wanting to learn the material on their own? With no incentives whatsoever? I’ve taken classes which were pass-fail before, and the effort of my classmates plunged.

      • josh

        But “research suggests” it, feministx! What are you? Some kind of creationist?

    • J. Cross

      As someone who teaches at a school with no grades, I believe it.

      Now, I’m not sure what would happen if the students had graded classes and non-graded classes. In that scenario they might not take the non-graded classes seriously. But, put in a school with no grades but with the very clear expectation that they take all of their classes seriously, students work hard.

      • Josh

        As another person who works at a school with no grades, I third it.

        However, we go a bit further and also have no pre-determined curricula, grade levels, power hierarchy or institutional expectations of what they should do or learn other than that they should know that they are individually responsible, now and forever, for what they do and learn.

        And, while I don’t know about “results” in the current test-results sense because we don’t test, I can say that our graduates over the years have gone on to college at a higher rate than your typical traditional school in this region (rural, central WV) and have become doctors, entrepreneurs, and stay-at-home moms. No judgment.

        Just saying, it takes a lot of trust, but humans are self-motivated their whole lives if they are in an environment where they are free and responsible for what they do. Isn’t that what a democracy is supposed to be anyways?

    • Boris

      For me the “end of unit” exams were one last chance to consolidate what I had learned, and the marking of these exams showed me what I (mis)understood. Useful for teachers to know what the classes were struggling with or learning easily.

      These exams were possibly the most educational day of the unit of work we were studying.

    • There’s a difference between being required to learn something and having that learning graded relative to those in your immediate physical vicinity.

  • mattmc

    I had to mostly disregard this part of the book. The problem with the traditional approach to merit pay and measuring progress is that it is done as absolutes, without regard to the “longitudinal” progress of the students. If you tell me that I need to make my students pass the minimum standards for the grade each year, but give me no control over where they start, I am doomed to fail. Adequate yearly progress is insanely defined by No Child Left Behind without regard for the fact that some students arrive in kindergarten reading 100 page books, while others haven’t even been read to once in their life.

    All subjects, if they are worth grading at all, should be studied until they are mastered. The real problem is our traditional schooling method of setting 20-30 kids in a room and giving them all the same level of work. The best progress I ever made in school was for a few years in elementary school in Texas where we were excused from any math concept for which we could pass a pretest. For subjects which you didn’t pass the pretest, you had up to two weeks to study them, and then you took a post test. If you passed, you moved on. If you failed, you took it again. We were all in the same grade together, but by the end of the year some of us were doing math 3 grades beyond where we started while others were just getting to the start of the nominal grade level.

    I agree with the point about social promotion, but when social promotion is combined with the “anti-tracking” bias of our current educational establishment, it forces schools to suck.

  • Unlike in 1910s or 1980s, we can now track long term performance of students really well, making cheating by teachers really really hard. This part of the problem is fully solvable.

    • Jess Riedel

      We can track student progress and prevent teacher cheating now better than the ’80s? I don’t understand how computers change anything.

      • Simon K

        There are tell-tale signs of teacher cheating that show up in statistical analysis of test results. Computers make it possible to automate this analysis.

  • Robin, it is fine to mention/excerpt/summarize studies of varying and unknown methodological rigor. But it would best if you could attach some confidence level to the claims so that readers would know how much faith you had in the claims.

    David Figlio provides the best evidence available which suggests higher grading standards result in more learning. (Ungated summary for practitioners).

    These reports, all based on Chicago data, are some of the richest and most rigorous evidence on ending social promotion. John 4’s comments are right on. They paint a more positive and more complex picture than “hard facts.”

    I don’t have the book but I certainly hope the author isn’t arguing that problems with merit pay systems in the past indicate that we should do no further experimentation and maintain the status quo.

    • That result is about tough vs easy grading. Why can’t both results be true, that grading is worse but tough grading is better than easy grading?

  • kebko

    If merit pay is just some bureaucratic, top-down program, it will fail. The distinction to be made is between top-down coersion & bottom-up markets. The principal at our charter school has full accountability for the teachers there. You can be that the teachers working there are based on merit. But, she isn’t filling out some form to submit to the union. She’s hiring & firing based on student performance, the fit with other teachers in their areas, or for that matter, how they look or smell.

    Merit-promotion is a natural product of market based approaches & it is somewhat indefinable. To say that merit promotion is the key is mistaking the symptom for the disease. It’s the lack of decentralized, accountable competition that’s the problem.

  • Pingback: Overcoming Bias : Hard Facts: Teaching « Beware of Roaming()

  • A system of Merit Based pay that combines today’s technologies has not even been tried yet in education.

    Imagine the very best teachers were paid based on the number of students that chose to attend the class – anywhere in the world!

    With Video Conferencing, students could migrate to the very best of the best lecturers in the world on a given subject. Imagine, your child sits down and listens in to the very best Algebra teacher in the world, then goes on to watch the most advanced teaching techniques applied to learning the parts of speech.

    This type of true market would create Teacher Superstars akin to what we have in professional sports. The increased competition would stretch imagination on how to best teach students different subjects.

    The production value of the classes would rise dramatically as multi-variate testing and competition would be always advancing the way students learn and produce the absolute best courses imaginable.

    That’s the world I want to live in.

  • More about teaching— the first half of the article is about a teacher who independently studied excellent teachers (as shown by improvement in student test scores), and discovered a repetoire of extremely useful learnable techniques for keeping students interested and involved.

    The second half is about making sure that teachers know the subject matter.

  • Chris

    When testing the flunking vs social promotion paradigm, were they comparing flunked to socially promoted students in general or were they comparing flunked to socially promoted students who would have flunked if the same standards for flunking were applied to them?

  • some guy

    My girlfriend worked in a school where students were assigned classes based on teacher favoritism. This was not public knowledge so much as private knowledge that the assigner knew and used as leverage.

    Teachers well-liked by the person in charge of creating the makeup of the classes were given the best behaved students, and disliked (or often just new) teachers were given the worst behaved. In a poor district with serious problems, that sort of thing can make your life miserable.

    And that’s without financial incentives. I can’t imagine how weird things would get if there were serious money on the line.

    Ultimately, you can’t just award someone a lot of power without going through a long pain period afterwards trying to fix the disparity, which, ultimately, will be disrupted once more after everyone forgets about the pain period and decides to “innovate”

  • The evidence strongly suggests that students learn better when they are not graded…

    This sounds a bit like a John Campbell editorial explaining why psychics do better when they aren’t watched by skeptics.

    At least 55 studies show that when flunked students are compared to socially promoted students, flunked students perform worse and drop out of school at higher rates.

    In other words, they compared the bottom of one population to the whole of another population.

    I didn’t think mergers were a great idea until now, but maybe I was wrong…

    • Simon K

      They compared students who failed and were flunked with those who failed and were promoted anyway, not with all students. The latter is so obviously invalid not even an economist could get away with it.

      • Chris

        Unfortunately, blank slate thinking still holds sway in much of social science (even though they now pay lip service to biology) and this can’t be assumed.

      • I don’t know the details of the system – if there are schools which have absolutely rigid criteria, you could compare the people who fell very close to the boundary on either side, whereas if there’s a discretionary component, that could still correlate with a number of factors for success which they haven’t managed to control for.

  • I thought that the sentence “The evidence shows that merit-pay plans seldom last longer than five years and that merit pay consistently failes to improve student performance.” pretty much contradicts itself.

    Some education reforms are programs that operate directly, for example Kumon. On might reasonably demand prompt results.

    Other education reforms are structural. Merit pay is a good example. One imagines that each individual teacher believes that he is above average and carries on as before. There are differences in pupil attainment and some teachers are paid bonuses because of this leading to ill-feeling and quarrels, but we clearly expect the early years of merit pay to involve “throwing money away” as the scheme rewards (or not) teachers for doing what they would have done anyway.

    Some unrewarded teachers are upset about the money. Other unrewarded teachers are upset about the status. They copy the teachers who are being rewarded for “merit”. Years go by, and Sod’s Law tells us what to expect: failure. It is not enought for a teacher to find out what works for other teachers, they have to find out how to make it work for them.

    It is necessary to step back a moment and ask “Why are we even attempting a structural reform? We aren’t we just looking at what the best teachers doing, and telling the others: copy that or be sacked.” Presumably the key point it that teaching is difficult. It is a skilled job. If we expect the pupils to learn more, the teachers will have to improve their skills. This will take years, and cannot be micro-managed. The theory of merit pay is to accept this, and respond shrewdly by setting up the incentives and waiting for the long term to arrive.

    How does it work out in the long term? That is the key empirical question. If one sticks with a merit pay system long enough for the teachers to stop being cross about it and switch to trying to earn merit rises, and one sticks with it a bit longer still to give the teachers two or three tries at trying to earn merit rises, what actually happens?

    That is a scary question because if the empirical result is that education gets worse, one has stuck with a bad idea for a decade or longer, on the basis that you need to give a structural reform time to work through the system. Well, yes, scary, but I don’t see any way avoid the risk.

    Perhaps the book contains a hard fact about politics, that structural reforms are not worth trying because it is not politically possible to hold a line for a decade to see if they actually work. The vested interests keep working to undermine the reform and after five years the public spirited reformers have wearied of endless battles and moved on to other causes. That might be true.

    I do not see how the book can contain a “hard fact” about whether a dictator should embrace merit pay for teachers on the grounds that it will come good after ten years. The quote seems to admit this.

  • One again testing squeezes out education. We need a separation of testing and education. If employers and customers need to know who is most capable, let them find out for themselves. It seems pretty rude to me that we pay schools to educate our children and if our children do not do well the people we paid to educate them blab their poor performance to the world.

    • I was just making this point the other day, with respect to university. If someone counters a rejection of evaluation with “But how will employers know who to hire?” I respond, “That would be their problem.”

      • Robert Wiblin

        It would also be the students’ problem, because they would not have to hire another group to certify their competence and impressiveness. Doesn’t mean it’s a bad idea, but let’s not forget lots of students want a nice transcript and will refuse to go to a university which doesn’t certify them as smarter than other students.

    • It is interesting that we trust schools to evaluate themselves here, rather than having independent evaluators of their effectiveness.

  • John Maxwell IV

    “One of the most careful studies found that, after controlling for numberous alternative explanations indlucing race, gender, family income, and school characteristics, students held back one grade were 70 percent more likely to drop out of high school.”

    That isn’t nearly enough stuff to control for. Both being held back a grade and dropping out of school are strongly caused by not being a very good student. It’s not at all obvious from this result that being held back causes a student to be more likely to drop out (although I do find this plausible myself).

  • John Maxwell IV

    Another point is that awarding or denying status based on academic performance may be desirable if it motivates top students to achieve even more.

  • I’d like to see studies about schools that don’t age segregate classes. It doesn’t make any obvious sense to expect students to make progress at the same rate in all their subjects.

  • Mike Rappaport

    I don’t have direct evidence about merit pay or social promotion, but I do about grading. Consider the incentives to learn the material when a class is graded or pass fail. I KNOW, from experience, that I learn more when it is graded. In fact, grades limited to A, B, C, D, and F (without pluses and minuses), as in my kids’ high school, are also bad. One of my children just got 91s the whole time, since the additional work was not rewarded. He would have learned the material better if he had an incentive to get a 98.

    Sorry, but this claim about grades is close to being a reductio.

  • A special case in which grading was bad for learning: I took an Alexander Technique (a method of improving coordination) class which was ungraded because the teacher had noticed that if students thought about grades, their coordination deteriorated. He graded on attendance, and mentioned anohther AT teacher who graded on the students’ notebooks– but there was no effective way of grading progress in the Alexander Technique.

  • “The evidence shows that merit-pay plans seldom last longer than five years and that merit pay consistently failes to improve student performance.”

    Does it really?

    Michael J. Podgursky, Matthew G. Springer
    Journal of Policy Analysis and Management, Vol. 26, No. 4, 909–949 (2007)

    From conclusion:

    “The evaluation literature on performance-related compensation schemes in education is very diverse in terms of incentive design, population, type of incentive (group versus individual), strength of study design, and duration of the incentive program. While the literature is not sufficiently robust to prescribe how systems should be designed—for example, optimal size of bonuses, mix of individual versus group incentives—it is sufficiently positive to suggest that further experiments and pilot programs by districts and states are very much in order. It is critical that these programs be introduced in a manner amenable to effective evaluation. Moreover, as noted by Courty and Marschke (2003), an overarching lesson seems to be that trial and error is likely required to formulate the right set of performance incentives. Development of massive student longitudinal achievement databases, such as those sponsored by USDoE, further opens prospects for rigorous value-added assessment over time.”


    • “Though we still have no clear results after all this time and money, further study is warranted; send your check to our address below.”

      • Conditional on not making even more dramatic changes to the hiring and compensation of teachers, further experimentation with merit pay is warranted. If Robin disagrees I would really like to hear why.

  • Captain Oblivious

    The evidence strongly suggests that students learn better when they are not graded

    If they’re not graded, how can anyone (especially the student!) ever know if they’re learning what they’re supposed to be learning?

    and certainly not when they are graded on a curve.

    I guess I always thought the ‘curve’ was to protect the students from bad teachers – even if you didn’t do very well, as long as you were in (say) the top 10% you learned about as much as it was possible to learn in that environment… and we can’t demand any more than that from the individual student. Does anyone understand how this practice harms students?

    Ending social promotion harms students and schools,

    ENDING social promotion (we’re talking about the practice of “passing” students who aren’t learning anything, thereby ensuring that they’re not prepared for the next grade either – right?) harms students??? How bizarre! I would have thought that giving students a false sense of achievement, encouraging them to squander their learning years doing anything but learning, and finally sending them out into the real world as young adults, to learn just how badly screwed they truly are, would harm them far worse. But I’m just a parent, not a professional educator, so what could I possibly know about how children learn?

    And what does it mean to ‘harm’ the school, anyway? Does the roof begin to leak or something?

    At least 55 studies show that when flunked students are compared to socially promoted students, flunked students perform worse and drop out of school at higher rates.

    OK, so social promotion keeps students “in school”… but what’s the POINT of being “in school” if you’re not learning? I’m guessing that if there are a lot of socially promoted students in a school, it’s just that much harder for the other students to learn (too many distractions) and that much more tempting to slack off (hey, no one else bothers to study and nothing bad happens to then, so why should I study?)

    Honestly, this whole excerpt reads like something written by people who are very determined not to see the obvious. The fact that this is coming from a .EDU website terrifies me.

    • Captain Oblivious

      OK, my bad – I had scrolled back up and saw an EDU URL, and thought that was the source of the article, but it turns out that it was in someone’s comment.

    • J. Cross

      If they’re not graded, how can anyone (especially the student!) ever know if they’re learning what they’re supposed to be learning?

      You’ve tried to learn things outside of a class, right? Weren’t you able to tell when you were learning and when you weren’t?

      One way to think about this is to ask the question “How would I teach my own kid something I want them to learn?” If you want to teach your kid to play baseball, build a toolbox or learn to program would you give them tests and grades? Would that be the focus with the presumption being that without grades they would have no motivation to learn?

      • Jess Riedel

        Kids like playing baseball. They don’t like learning grammar. If I wanted to teach my kids grammar outside of school, and they weren’t interested, I would have to offer them some sort of incentive (based on some sort of metric) to try.

  • Captain Oblivious

    P.S. Is there a ‘preview’ or ‘delete’ button I’m overlooking?

  • Thanks Robin for sharing this.

    There really is a world of difference between schooling and learning. All the comments along the lines of “how will you tell if they are learning”? are completely unaware of what real learning actually is.

    Most of “education” ranges from a colossally-expensive baby-sitting operation to a stupendously wasteful signalling and credentialling machine.

  • “At least 55 studies show that when flunked students are compared to socially promoted students, flunked students perform worse and drop out of school at higher rates.”

    This is a surprise? Is this supposed to imply that flunking students causes them to perform worse and drop out of school?

    I have an alternate hypothesis: Students who perform worse and know they are likely to drop out of school, are more likely to be flunked.

  • colin

    I haven’t read the book so I don’t know the methodology. What type of schools were they getting their data from? Was it mostly public schools? Did they have a controlled study of just private or even charter+private?

    My point being, if the incentive pay programs are implemented in public schools, which essentially have monopoly control over education, and whose teachers generally belong to a strong union, then I’d expect to see mass grade inflation without any improved learning. There’d be no incentive NOT to just juice up the grades.

    Whereas if incentives were provided in a system where school choice was more open and the disciplining (and firing/hiring process) of teachers weren’t so costly, we might see real benefit.

    If they’re not controlling the data for the vast majority of our ed sector being a public monopoly, then the data isn’t really all that informative.

  • Incentive pay for teachers will do the same thing that the state exams have started doing. Teachers now have to teach to the tests that the students have to take instead of the subjects and alternative thinking that promotes true education. Nothing more than zombie generators.

    School-children are being turned into mini-encyclopedias that can only regurgitate the smallest facts that have been stuffed into them. Kids also now understand that they can google anything and have an answer, or find someplace to ask it.

    Our education system is seriously lacking.

  • Pingback: “Why Schools test Often” | Lode Cossaer()