DailyDirt: Can Computers Grade Written Essays?
from the urls-we-dig-up dept
Technology aimed at education could really benefit an incredible number of students by making classes and learning (potentially) a more pleasant and efficient experience. Computers can’t replace a really good human teacher, but they can make it easier for good human teachers to reach a vast audience of students. Massively open online courses (MOOCs) promise to change how education works, but there are some technological tools that might be missing. It’s pretty straightforward to test students on math problems in an automated way, but grading essays is a much more daunting problem. There have been some calls for automated grading software from various organizations (like the Hewlett Foundation). But at the same time, the National Council of Teachers of English argues that computers simply can’t grade essays. Here are just a few more links on this debate over the use of algorithms over English professors (or grad students).
- EdX, the non-profit started by Harvard and MIT, is releasing some software to automagically grade human-written essays. Some see this software as just another tool for educators to use for more immediate feedback to students, while others are worried that these algorithms will be used incorrectly and lead to disastrous educational policies and outcomes. [url]
- There are studies that show algorithms are statistically comparable to humans when it comes to ranking essays on a 5 point scale. There are things machines can do better and things humans do better — just make sure you know the differences and automated essay grading can be done productively in the right context. [url]
- Automated essay readers can grade 16,000 essays in 20 seconds. The Educational Testing Service is testing out automation, so students may soon be facing algorithmic grading for their college entrance exams. [url]
- Grading a few sentences can be harder than it might look. Professional (human) teachers are obviously better at interpreting the insights and ideas behind the words a student writes, but computers scale much better and never tire of horrible spelling mistakes or misplaced modifiers…. [url]
If you’d like to read more awesome and interesting stuff, check out this unrelated (but not entirely random!) Techdirt post via StumbleUpon.
Filed Under: ai, algorithms, artificial intelligence, automation, education, edx, essays, ets, exams, grading, mooc, nlp, tests
Companies: harvard, hewlett foundation, kaggle, mit
Comments on “DailyDirt: Can Computers Grade Written Essays?”
I imagine they'll start using this for grading bar exams soon...
And we’ll have a flood of new lawyers who can convince computers without using any facts at all… yay!
I think the teachers are arguing they’d like to keep their jobs. I’ve honestly seen some teachers who think their job is grading rather than teaching. Anyways a well done grading program could grade just fine. I guess the argument would be that nobody has created a great program for grading written papers.
Yeah, cell phones are really good it, let’s apply it to exams.
Here's an idea!
Lets combine automated grading software with automated essay composition software and leave humans completely out of the loop.
Considering the Zero Tolerance stance on.. er… children… in american schools, I’m surprised we aren’t seeing computers designed to impose suspensions.
Re: Re:
I have always thought that the way to restore sanity regarding zero tolerance is to threaten to do that.
Nothing is as influential as a bureaucrat trying to preserve their own job. If it goes through without a return to sanity well you punished the morons behind it by making them unemployed and you aren’t worse off. Not to mention the tax revenue saved. It is a heads you lose tails I win situation.
Re: Re:
It could be like in that movie – Demolition Man.
Swearing is illegal and you get a printout for each infraction. I’d better hurry and invest in paper companies before its cool.
Grading? Easy
Grading an essay is actually pretty damn easy, just return a value. Grading it right is the challenge.
Depends
Can a student turn in a computer-generated paper that is written to the grading algorithm?
It would take some smarts to do that, too. Neither would be satisfactory.
Re: Depends
Along those lines, isn’t the current system just a set of ‘facts’ (loosely describing the current standardized testing system) and that the way to game the system is to know what is on the final test?
Reminds me of a segment regarding the author of In Search of Excellence. He was a new professor on his first day, and he handed out the final exam to all of the students. A more senior professor was passing by and noticed this atrocity. He cornered his junior and remonstrated him for giving out the final exam. The new to the campus professor retorted, (to the best of my memory) “Not only am I giving them the final exam on the first day, I intend to spend the rest of the semester giving them the answers.”
As an experienced employer, with at times thousands of employees, my philosophy on hiring is: I don’t care what you know, I care what you can do! Education provides knowledge. Knowledge, TRAINING, AND EXPERIENCE provide skill. Skill is what one sells on the marketplace. Either to a new employer, or a customer.
Unfortunately, other employers are not so enlightened. They care what your grades are and what is on your Facebook page, and if those keywords weren’t on the resume, we wouldn’t even be talk’n.
For me, teach the student the basics (read’n, rite’n, and rithmitic) then teach them to think, make decisions, take stances, be critical without animosity, resolve issues, expand creativity (I wish art and music were required, but creativity can come elsewhere, even in math), and to ‘do it, try it, fix it.
I care not what the grades were, I care what they can do, and if they might grow with us.
Re: Re: Depends
Tom Peters, damn this early onset….what was I talking about?
Re: Re: Depends
The problem really is that there is a hierarchy of knowledge (use information-gathering mechanism of choice for finding out about Bloom’s Taxonomy and related). At the lowest levels, you have things characterised by words like ‘recall’ and ‘recite’. At higher levels, you progress through things like ‘understand’ and ‘apply’ through to ‘analyse’ up to ‘create’.
Evaluating the lower levels ranges from trivially easy to simple. As you look for higher-level skills, you begin to require expert judgment. I’ll assert that expert judgment is almost impossible to implement as an algorithm.
I’m doing a MOOC atm. It is sitting squarely in the lower cognitive domains. It’ll replace a bad teacher easily. It won’t replace the best a university can do.
Re: Depends
Yes. Les Perelman, a director of writing at MIT, did an analysis of e-Rater algorithms last year. He came up with a completely nonsensical and factually incorrect essay that got a top score, and a well-argued and well-written essay that scored lower.
Here’s the NYT article about it:
http://www.nytimes.com/2012/04/23/education/robo-readers-used-to-grade-test-essays.html
And here’s his top-scoring essay:
http://www.documentcloud.org/documents/346138-essay-awarded-a-top-grade-by-e-rater.html
If nothing else, it’s worth skimming through the essay. Among other things, it includes a line from Howl and claims that the union of teaching assistants is more powerful than the Freemasons (although less powerful than the Jedi Knights).
Textual analysis is hard. It can potentially test for writing style, but it can’t test for comprehension or content.
Re: Re: Depends
yep, thanks, was going to post about that…
but it kind of fits with the times: style over substance…
art guerrilla
aka ann archy
eof
Need to worry about people gaming the models
Here’s a quick summary of the most insightful comments I’ve seen about this issue:
Yes, computers can apparently grade essays written for humans as well as underpaid and overworked graders do (which is badly). But once you start having computers actually grade the students, the students will start writing their essays to fool the computer graders (likely with help of computer programs)…
MS Word has an auto-summary feature that is disturbingly good, but just because software can create a summary doesn’t mean it can recognize a good one.
The day they write software that can recognize and appreciate a pun is the day they can dream about writing software that can recognize and appreciate a sound argument.
This sounds more like the kind of dumbing down we get from standardized testing. And I suppose if they really do want to reduce our children to mindless drones, then this is another step in the right direction
Assignment writer uk
This is really nice information which you have shared with us. Please keep sharing that kind of information further as I really like it.
essay
At my university essay test worked very attentive teacher. However, using the http://www.grabmyessays.com/ – service for writing essays I successfully passed the course.
good
hey, The day they write software that can recognize and appreciate a pun is the day they can dream about writing software that can recognize and appreciate a sound argument.
http://www.dreamfilm.ws/forfattare/edgar-wright
Great
Massively open online courses (MOOCs) promise to change how education works, but there are some technological tools that might be missing.
http://www.123movies.biz/423-criminal-full-movie.html
Thloolm
Massively open online courses (MOOCs) promise to change
Thloolm
http://xmovies-8.cc/
Thloolm
https://xmovies-8.ws/movies
Thloolm
Thanx
Thloolm
thxxxxxxxxxxxxxxxxxx
Thloolm
Txxxxxx
Thloolm
ThanxThanx
Thloolm mol
Kooopasi !
Thloolm
Tgabjs
Thanks
Great one !
Thloolm
For movies : http://sefilmdk.ws/
Thloolm
Watch it live : http://www.rainierland-movies.com/the-phenom-rainierland/
Thloolm
War dogs online : http://movies25.biz/41-war-dogs.html
Thloolm
Oh yes !
Thloolm
yES THX §
Thloolm
Thak !
Thloolm
Thxx !
Streama Hundraettåringen som smet från notan och försvann
Hundraettåringen som smet från notan och försvann swefilmer, Hundraettåringen som smet från notan och försvann swesub streama gratis online
www.genvideos.ws
good
www.pubfilm.ws
good post
pubfilm.ws
This is really nice information which you have shared with us. Please keep sharing that kind of information further as I really like it.
www.putlockers.la
good site
Streama Film
http://gratis-filmer.tv Streama Film Online
MS Word has a disturbingly good auto-summary feature, but just because software can build a summary doesn’t mean it can recognize a good one.
The day they’re writing software that recognizes and appreciates a joke is the day they can dream of writing software that can recognize and appreciate a sound statement.
Watch online movies 2020 on 123Movies new site.