"In fact it would count for you because good detail." "So you can write that the War of 1812 began in 1945, and that wouldn't count against you at all," he says. This is a case where the machines are very, very stupid.īecause computers can only count, and cannot actually understand meaning, he says, facts are irrelevant to the algorithm. He then enters three words related to the essay prompt into his Babel Generator, which instantly spits back a 500-word wonder, replete with a plethora of obscure multisyllabic synonyms: To demonstrate, he calls up a practice question for the GRE exam that's graded with the same algorithms that actual tests are. Essay Language") Generator, it works like a computerized Mad Libs, creating essays that make zero sense, but earn top scores from robo-graders. He's designed what you might think of as robo-graders' kryptonite, to expose what he sees as the weakness and absurdity of automated scoring. Turns out that's an easy question to answer, thanks to MIT research affiliate, and longtime-critic of automated scoring, Les Perelman. "Is it going to reward some vapid drivel that happens to be structurally sound?" "What is the computer program going to reward?" Henderson challenges. Marder and Henderson worry robo-graders will just encourage the worst kind of formulaic writing. "What about original ideas? Where is room for creativity of expression? A computer is going to miss all of that," she says. "An art form, a form of expression being evaluated by an algorithm is patently ridiculous."Īnother English teacher, Robyn Marder, nods her head in agreement. "The idea is bananas, as far as I'm concerned," says Kelly Henderson, an English teacher at Newton South High School just outside Boston. But all in all, she says the automated scoring system has been a boon for the state, not only for the cost savings, but also because it enables teachers to get test results back in minutes rather than months.Īn art form, a form of expression being evaluated by an algorithm is patently ridiculous. In about 20 percent of cases, she says, when the computer detects something unusual, or is on the fence between two scores, it flags an essay for human review. But she says the computer scoring has proven "spot-on" and Utah now lets machines be the sole judge of the vast majority of essays. Cyndee Carter, assessment development coordinator for the Utah State Board of Education, says the state began very cautiously, at first making sure every machine-graded essay was also read by a real person. Several states including Utah and Ohio already use automated grading on their standardized tests. The computer also breaks it down in several categories of sub-scores showing, for example, a one on spelling and grammar, and a two on task and focus. "It gives an overall score of two out of four," Foltz explains. To demonstrate, he takes a not-so-stellar sample essay, rife with spelling mistakes and sentence fragments, and runs it by the robo-grader, which instantly spits back a not-so-stellar score. "We've done a number of studies to show that the scoring can be highly accurate," he says. That includes not only basics like spelling and grammar, but also whether a student is on topic, the coherence or the flow of an argument, and the complexity of word choice and sentence structure. "We have artificial intelligence techniques which can judge anywhere from 50 to 100 features," Foltz says. Then, the automated programs score essays themselves by scanning for those same features. Foltz says computers "learn" what's considered good writing by analyzing essays graded by humans.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |