DailyDirt: Can Computers Grade Written Essays?

from the urls-we-dig-up dept

Technology aimed at education could really benefit an incredible number of students by making classes and learning (potentially) a more pleasant and efficient experience. Computers can't replace a really good human teacher, but they can make it easier for good human teachers to reach a vast audience of students. Massively open online courses (MOOCs) promise to change how education works, but there are some technological tools that might be missing. It's pretty straightforward to test students on math problems in an automated way, but grading essays is a much more daunting problem. There have been some calls for automated grading software from various organizations (like the Hewlett Foundation). But at the same time, the National Council of Teachers of English argues that computers simply can't grade essays. Here are just a few more links on this debate over the use of algorithms over English professors (or grad students). If you'd like to read more awesome and interesting stuff, check out this unrelated (but not entirely random!) Techdirt post via StumbleUpon.


Reader Comments (rss)

(Flattened / Threaded)

  •  
    identicon
    Anonymous Coward, May 9th, 2013 @ 5:20pm

    I imagine they'll start using this for grading bar exams soon...

    And we'll have a flood of new lawyers who can convince computers without using any facts at all... yay!

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 9th, 2013 @ 5:21pm

    I think the teachers are arguing they'd like to keep their jobs. I've honestly seen some teachers who think their job is grading rather than teaching. Anyways a well done grading program could grade just fine. I guess the argument would be that nobody has created a great program for grading written papers.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 9th, 2013 @ 5:50pm

    Yeah, cell phones are really good it, let's apply it to exams.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 9th, 2013 @ 5:51pm

    Here's an idea!

    Lets combine automated grading software with automated essay composition software and leave humans completely out of the loop.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 9th, 2013 @ 7:22pm

    Considering the Zero Tolerance stance on.. er... children... in american schools, I'm surprised we aren't seeing computers designed to impose suspensions.

     

    reply to this | link to this | view in chronology ]

    •  
      identicon
      Anonymous Coward, May 9th, 2013 @ 7:35pm

      Re:

      I have always thought that the way to restore sanity regarding zero tolerance is to threaten to do that.

      Nothing is as influential as a bureaucrat trying to preserve their own job. If it goes through without a return to sanity well you punished the morons behind it by making them unemployed and you aren't worse off. Not to mention the tax revenue saved. It is a heads you lose tails I win situation.

       

      reply to this | link to this | view in chronology ]

    •  
      identicon
      Anonymous Coward, May 10th, 2013 @ 5:04am

      Re:

      It could be like in that movie - Demolition Man.
      Swearing is illegal and you get a printout for each infraction. I'd better hurry and invest in paper companies before its cool.

       

      reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 9th, 2013 @ 7:30pm

    Grading? Easy

    Grading an essay is actually pretty damn easy, just return a value. Grading it right is the challenge.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    artp (profile), May 9th, 2013 @ 7:55pm

    Depends

    Can a student turn in a computer-generated paper that is written to the grading algorithm?

    It would take some smarts to do that, too. Neither would be satisfactory.

     

    reply to this | link to this | view in chronology ]

    •  
      identicon
      Anonymous Anonymous Coward, May 9th, 2013 @ 8:38pm

      Re: Depends

      Along those lines, isn't the current system just a set of 'facts' (loosely describing the current standardized testing system) and that the way to game the system is to know what is on the final test?

      Reminds me of a segment regarding the author of In Search of Excellence. He was a new professor on his first day, and he handed out the final exam to all of the students. A more senior professor was passing by and noticed this atrocity. He cornered his junior and remonstrated him for giving out the final exam. The new to the campus professor retorted, (to the best of my memory) "Not only am I giving them the final exam on the first day, I intend to spend the rest of the semester giving them the answers."

      As an experienced employer, with at times thousands of employees, my philosophy on hiring is: I don't care what you know, I care what you can do! Education provides knowledge. Knowledge, TRAINING, AND EXPERIENCE provide skill. Skill is what one sells on the marketplace. Either to a new employer, or a customer.

      Unfortunately, other employers are not so enlightened. They care what your grades are and what is on your Facebook page, and if those keywords weren't on the resume, we wouldn't even be talk'n.

      For me, teach the student the basics (read'n, rite'n, and rithmitic) then teach them to think, make decisions, take stances, be critical without animosity, resolve issues, expand creativity (I wish art and music were required, but creativity can come elsewhere, even in math), and to 'do it, try it, fix it.

      I care not what the grades were, I care what they can do, and if they might grow with us.

       

      reply to this | link to this | view in chronology ]

      •  
        identicon
        Anonymous Anonymous Coware, May 9th, 2013 @ 8:44pm

        Re: Re: Depends

        Tom Peters, damn this early onset....what was I talking about?

         

        reply to this | link to this | view in chronology ]

      •  
        identicon
        Anonymous Coward, May 9th, 2013 @ 11:47pm

        Re: Re: Depends

        The problem really is that there is a hierarchy of knowledge (use information-gathering mechanism of choice for finding out about Bloom's Taxonomy and related). At the lowest levels, you have things characterised by words like 'recall' and 'recite'. At higher levels, you progress through things like 'understand' and 'apply' through to 'analyse' up to 'create'.

        Evaluating the lower levels ranges from trivially easy to simple. As you look for higher-level skills, you begin to require expert judgment. I'll assert that expert judgment is almost impossible to implement as an algorithm.

        I'm doing a MOOC atm. It is sitting squarely in the lower cognitive domains. It'll replace a bad teacher easily. It won't replace the best a university can do.

         

        reply to this | link to this | view in chronology ]

    •  
      identicon
      Anonymous Coward, May 10th, 2013 @ 2:48am

      Re: Depends

      Yes. Les Perelman, a director of writing at MIT, did an analysis of e-Rater algorithms last year. He came up with a completely nonsensical and factually incorrect essay that got a top score, and a well-argued and well-written essay that scored lower.

      Here's the NYT article about it:
      http://www.nytimes.com/2012/04/23/education/robo-readers-used-to-grade-test-essays.html

      And here's his top-scoring essay:
      http://www.documentcloud.org/documents/346138-essay-awarded-a-top-grade-by-e-rater.html

      I f nothing else, it's worth skimming through the essay. Among other things, it includes a line from Howl and claims that the union of teaching assistants is more powerful than the Freemasons (although less powerful than the Jedi Knights).

      Textual analysis is hard. It can potentially test for writing style, but it can't test for comprehension or content.

       

      reply to this | link to this | view in chronology ]

      •  
        icon
        art guerrilla (profile), May 10th, 2013 @ 6:46am

        Re: Re: Depends

        yep, thanks, was going to post about that...
        but it kind of fits with the times: style over substance...

        art guerrilla
        aka ann archy
        eof

         

        reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 9th, 2013 @ 8:36pm

    Need to worry about people gaming the models

    Here's a quick summary of the most insightful comments I've seen about this issue:

    Yes, computers can apparently grade essays written for humans as well as underpaid and overworked graders do (which is badly). But once you start having computers actually grade the students, the students will start writing their essays to fool the computer graders (likely with help of computer programs)...

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    timmaguire42 (profile), May 11th, 2013 @ 6:02am

    MS Word has an auto-summary feature that is disturbingly good, but just because software can create a summary doesn't mean it can recognize a good one.

    The day they write software that can recognize and appreciate a pun is the day they can dream about writing software that can recognize and appreciate a sound argument.

    This sounds more like the kind of dumbing down we get from standardized testing. And I suppose if they really do want to reduce our children to mindless drones, then this is another step in the right direction

     

    reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
A word from our sponsors...
Essential Reading
Techdirt Reading List
Techdirt Insider Chat
A word from our sponsors...
Recent Stories
A word from our sponsors...

Close

Email This