EssayTools - Maple Programming Help

Home : Support : Online Help : Education : EssayTools : EssayTools/BuildScoreModel

EssayTools

 BuildScoreModel
 grade essays based on training data
 Score
 grade essays based on training data

 Calling Sequence BuildScoreModel( essays, scores ) Score( essay, model )

Parameters

 essays - array of strings scores - array of numeric grades questionText - (optional) string essay - string model - module returned by BuildScoreModel

Description

 • The BuildScoreModel command takes existing essays with assigned grades and returns a module that can be used by the Score command.  The Score command gives a rating for a new essay based on attributes of previously scored essays stored in built scoring model.
 • The reliability of the Score commands depends greatly on the amount of training data used to build the model.  At least 150 scored essays are required to start getting minimum performance (on the order of a 50% - 70% correlation with human-graded essays).  Larger sets on the order of 300 - 800 training essays are preferable and have been shown to score QuadraticWeightedKappa values of better than .80 compared to human grading.
 • The reliability of the score also depends on many other factors including the length of the expected answer (longer is better), and the nature of the question.
 • Questions can be wide-open with no set expected answer.  An open-ended question like, "tell a humorous story" has no one expected answer or pattern of keywords that must be common between answers.  In this case the style of writing becomes more important.  The results of computer scoring of such questions has proven to have a high correlation with human scoring.
 • Generally computer scoring is used in conjunction with human scoring. A set of essays will be scored by a human grader and the computer.  Any essays that receive scores that disagree by more than one or two points are then re-scored by an independent human grader.  Computer scoring has also been found useful for immediate feedback in homework and practice.
 • The text of the question can optionally be added to the model using the questionText = option.  It is recommended that this option be used.
 • This function is part of the EssayTools package, so it can be used in the short form SimilarityScore(..) only after executing the command with(EssayTools). However, it can always be accessed through the long form of the command by using EssayTools[SimilarityScore](..).

Examples

 > $\mathrm{Question}≔"What is Archimedes Principle?":$
 > $\mathrm{Answers}≔\left["Archimedes\text{'} principle is a law of physics stating that the upward force \left(buoyancy\right) exerted on a body immersed in a fluid is equal to the weight of the amount of fluid the body displaces.","Archimedes\text{'} principle says that the magnitude of the buoyancy force is also equivalent to the weight of the displaced fluid.","Archimedes\text{'} principle states that a body immersed in a fluid is buoyed up by a force equal to the weight of the displaced fluid.","Archemedes ran through town shouting Eureka because he noticed something about the water in his bath.","Archimedes principle says water weighs more than a block of gold.","Archimedes\text{'} principal is the law that states that a boyancy force is equal to a girlancy force.","Archimedes principle says that a block of gold will displace a matching volume of water.","Archimedes principle is a law that says a boyancy force is equal to the weight of the displaced water","Archimedes principle was discovered when King Hiero suspected that his new crown was not pure gold. Archimedes figured out how to test for a fake. He compared the weight of the crown with the weight of displaced water."\right]:$
 > $\mathrm{Scores}≔\left[10,10,10,3,0,0,4,7,6\right]:$
 > $\mathrm{with}\left(\mathrm{EssayTools}\right):$
 > $\mathrm{model}≔\mathrm{BuildScoreModel}\left(\mathrm{Answers},\mathrm{Scores},\mathrm{questionText}=\mathrm{Question}\right):$
 > $\mathrm{Score}\left("I don\text{'}t know.",\mathrm{model}\right)$
 ${0}$ (1)
 > $\mathrm{Score}\left("Archimedes\text{'} principle is about buoyancy.",\mathrm{model}\right)$
 ${2}$ (2)
 > $\mathrm{Score}\left("Archimedes\text{'} principle is a law that equates the force of buoyancy and the amount of displaced fluid.",\mathrm{model}\right)$
 ${7}$ (3)
 > $\mathrm{Score}\left("Nothing happened. The fish just moved away slowly and the old man could not raise him an inch. His line was strong and made for heavy fish and he held it against his hack until it was so taut that beads of water were jumping from it. Then it began to make a slow hissing sound in the water and he still held it, bracing himself against the thwart and leaning back against the pull. The boat began to move slowly off toward the north-west.",\mathrm{model}\right)$
 ${0}$ (4)

References

 Butcher, Philip G. and Jordan, Sally E. (2010).  A comparison of human and computer marking of short free-text student responses.  Computers and Education, 55(2), pp. 489-499
 Kakkonen, T., Myller, N., Sutinen, E., and Timonen, J. (2008).  Comparison of Dimension Reduction Methods for Automated Essay Grading.  Educational Technology & Society, 11(3), 275-288
 Kaplan, Randy M., Wolff, Susanne E., Burstein, Jill C., Lu, Chi, Rock, Donald A., and Kaplan, Bruce A. (1998), Scoring Essays Automatically Using Surface Features, GRE Board Professional Report No. 94-21P
 Miltsakaki, Eleni and Kukich, Karen, Automated Evaluation of Coherence in Student Essays
 Shermis, Mark D. and Hamner, Ben, (2012) Contrasting State-of-the-Art Automated Scoring of Essays: Analysis

Compatibility

 • The EssayTools[BuildScoreModel] and EssayTools[Score] commands were introduced in Maple 17.