Automated Essay Scoring

It might be difficult, time-consuming and costly to score constructed response items in tests. However, improvements in computer technology have enabled the automated scoring of constructed response items. Yet, the application of automated scoring without making investigation on validity, reliability and test equating can lead to serious problems. In this sense, the aim of this study was to score the constructed response items in mixed format tests automatically and to investigate the effect of this on test equating and reliability. The data examined in this study were the 8th grade Turkish test data of ABİDE research (Education Skills Monitoring and Evaluation) carried out by Ministry of National Education in Turkey in 2016. These tests contained common items. Support vector machine (SVM), logistic regression (LR), multinominal naive bayes (MNB), long-short term memory (LSTM) and bidirectional long-short term memory (BLSTM) were selected as automated scoring methods. During the test equating process, methods based on Classical Test Theory and Item Response Theory were utilized. The results of the study revealed that the most compatible automated scoring method with actual raters is BLSTM. The scores obtained by the BLSTM method were in good agreement with the scores of actual raters. In most of the equating methods, it was observed that errors of equating process done with automated scoring were close to the errors of equating process done by actual raters. It was concluded that automated scoring can be applied since it is compatible with actual raters and convenient in terms of equating.

This paper distinguishes features of automated adaptive guidance used in K-12 instructional settings and recommends directions for design. We use meta-analysis to synthesize 24 independent comparisons between automated adaptive guidance and guidance provided during typical teacher-led instruction, and 29 comparisons that isolate effects of specific adaptive guidance design features in computer-based instruction. We find automated adaptive guidance to be significantly more effective than guidance provided in typical instruction, particularly for students with low prior knowledge. Automated adaptive guidance is most effective when students are generating and integrating ideas (e.g. writing essays, making concept diagrams) as opposed to selecting from given options. Guidance that promoted self-monitoring was more likely to improve learning outcomes than guidance that addressed only content knowledge. Our findings have implications for researchers who investigate K-12 teaching and learning, designers who create and refine instructional materials using automated guidance, and practitioners who deliver or customize instruction featuring automated guidance.

In the last 30 years, numerous scholars have described the possible changes in marking writing assignments. The paper reflects these developments as it charts the paths recently taken in the field, evaluates automated and human essay scoring systems in academic environments and analyzes the implications that both systems offer. In recent years, ways and opportunities for giving feedback have changed as computer programs have been more widely used in assessing students writing. Numerous researchers have studied computerized feedback and its potential. Different problems, such as quality of this type of feedback, validity, and reliability have been analyzed. This critical review examines two major types of academic writing support. The objective of the study based on the literature review is to examine the potential support of human and automated proofreaders for teaching and learning purposes.

In this paper we present first results for the task of Automated Essay Scoring for Norwe-gian learner language. We analyze a number of properties of this task experimentally and assess (i) the formulation of the task as either regression or classification, (ii) the use of various non-neural and neural machine learning architectures with various types of input representations, and (iii) applying multi-task learning for joint prediction of essay scoring and native language identification. We find that a GRU-based attention model trained in a single-task setting performs best at the AES task.

OF THE THESIS EFFICACY AND IMPLEMENTATION OF AUTOMATED ESSAY SCORING IN INSTRUCTION OF LITERACIES TO HIGH LEVEL ELLs by Aaron J. Alvero Florida International University, 2016 Miami, Florida Professor Eric Dwyer, Major Professor This thesis explored the integration of automated essay scoring (AES) software into the writing curriculum for high level ESOL students (levels 3, 4, and 5 on a 1-5 scale) at a high school in Miami, Fl. Issues for Haitian Creole speaking students were also explored. The Spanish and Haitian Creole speaking students were given the option to write notes, outlines, and planning sheets in their L1. After using AES in the middle of the writing process as a revision assistant tool, 24 students responded to a Likert Scale questionnaire. The students responded positively to the AES based on the results of the Likert scale questionnaire: 71% responded “agree” and “strongly agree” to the question “Other students would benefit from using writing software before handing i...

Automated writing evaluation (AWE) software uses artificial intelligence (AI) to score student essays and support revision. We studied how an AWE program called MY Access!® was used in eight middle schools in Southern California over a three-year period. Although many teachers and students considered automated scoring unreliable, and teachers’ use of AWE was limited by the desire to use conventional writing methods, use of the software still brought important benefits. Observations, interviews, and a survey indicated that using AWE simplified classroom management and increased students’ motivation to write and revise.

OF THE THESIS EFFICACY AND IMPLEMENTATION OF AUTOMATED ESSAY SCORING IN INSTRUCTION OF LITERACIES TO HIGH LEVEL ELLs by Aaron J. Alvero Florida International University, 2016 Miami, Florida Professor Eric Dwyer, Major Professor This thesis explored the integration of automated essay scoring (AES) software into the writing curriculum for high level ESOL students (levels 3, 4, and 5 on a 1-5 scale) at a high school in Miami, Fl. Issues for Haitian Creole speaking students were also explored. The Spanish and Haitian Creole speaking students were given the option to write notes, outlines, and planning sheets in their L1. After using AES in the middle of the writing process as a revision assistant tool, 24 students responded to a Likert Scale questionnaire. The students responded positively to the AES based on the results of the Likert scale questionnaire: 71% responded “agree” and “strongly agree” to the question “Other students would benefit from using writing software before handing i...

The use of open-ended items, especially in large-scale tests, created difficulties in scoring open-ended items. However, this problem can be overcome with an approach based on automated scoring of open-ended items. The aim of this study was to examine the reliability of the data obtained by scoring open-ended items automatically. One of the objectives was to compare different algorithms based on machine learning in automated scoring (support vector machines, logistic regression, multinominal Naive Bayes, long-short term memory, and bidirectional long-short term memory). The other objective was to investigate the change in the reliability of automated scoring by differentiating the data rate used in testing the automated scoring system (33%, 20%, and 10%). While examining the reliability of automated scoring, a comparison was made with the reliability of the data obtained from human raters. In this study, which demonstrated the first automated scoring attempt of open-ended items in the Turkish language, Turkish test data of the Academic Skills Monitoring and Evaluation (ABIDE) program administered by the Ministry of National Education were used. Cross-validation was used to test the system. Regarding the coefficients of agreement to show reliability, the percentage of agreement, the quadratic-weighted Kappa, which is frequently used in automated scoring studies, and the Gwet's AC1 coefficient, which is not affected by the prevalence problem in the distribution of data into categories, were used. The results of the study showed that automated scoring algorithms could be utilized. It was found that the best algorithm to be used in automated scoring is bidirectional long-short term memory. Long-short term memory and multinominal Naive Bayes algorithms showed lower performance than support vector machines, logistic regression, and bidirectional long-short term memory algorithms. In automated scoring, it was determined that the coefficients of agreement at 33% test data rate were slightly lower comparing 10% and 20% test data rates, but were within the desired range.

Writing fluently and accurately is a goal for many advanced learners of English. Assigning grades to students’ writing is part of the educational process, and some grades, e.g., in standardized tests, even serve as a qualification for entering universities. This chapter aims to identify factors which correlate strongly with essay grades and English proficiency level. Because we would like to contribute to Automated Essay Scoring (AES), we focus on candidate factors that might easily be automated. The study is based on an existing database containing 81 variables (many of which hand-coded) in spontaneously, short written texts by 481 Dutch high school learners of English. All texts were holistically scored on a proficiency level from 1 to 5 by a team of experts. Another question we investigate is which machine learning algorithm (from a subset of those present in the WEKA 4 data mining/machine learning software) provides the best classification accuracy in terms of predicting the Eng...

Examination is the process by which the ability and the quality of the examinees can be measured. It is necessary to ensure the quality of the examinees. Online examination system is the process by which the participants can appear at the examination irrespective of their locations by connecting to examination site via Internet using desktop computers, laptops or smart phones. Automated score generation is the process by which the answer scripts of the examinations are evaluated automatically to generate scores. Although, there are many existing online examination systems, the main drawback of these systems is that they cannot compute automated score accurately, especially from the text-based answers. Moreover, most of them are unilingual in nature. As a result, examinees can appear at the examination in a particular language. Considering this fact, in this paper, we present a framework that can take Multiple Choice Questions (MCQ) examinations and written examinations in two differ...

Examination is the process by which the ability and the quality of the examinees can be measured. It is necessary to ensure the quality of the examinees. Online examination system is the process by which the participants can appear at the examination irrespective of their locations by connecting to examination site via Internet using desktop computers, laptops or smart phones. Automated score generation is the process by which the answer scripts of the examinations are evaluated automatically to generate scores. Although, there are many existing online examination systems, the main drawback of these systems is that they cannot compute automated score accurately, especially from the text-based answers. Moreover, most of them are unilingual in nature. As a result, examinees can appear at the examination in a particular language. Considering this fact, in this paper, we present a framework that can take Multiple Choice Questions (MCQ) examinations and written examinations in two different languages English and Bangla. We develop a database where the questions and answers are stored. The questions from the database are displayed in the web page with answering options for the MCQ questions and text boxes for the written questions. For generating the scores of the written questions, we performed several types of analysis of the answers of the written questions. However, for generating the scores of the MCQ questions, we simply compared between the database answers and the user's answers. We conducted several experiments to check the accuracy of score generation by our system and found that our system can generate 100% accurate scores for MCQ questions and more than 90% accurate scores from text based questions.

This study evaluated a “substantively driven” method for scoring NAEP writing assessments automatically. The study used variations of an existing commercial program, e-rater®, to compare the performance of three approaches to automated essay scoring: a brute-empirical approach in which variables are selected and weighted solely according to statistical criteria, a hybrid approach in which a fixed set of variables more closely tied to the characteristics of good writing was used but the weights were still statistically determined, and a substantively driven approach in which a fixed set of variables was weighted according to the judgments of two independent committees of writing experts. The research questions concerned (1) the reproducibility of weights across writing experts, (2) the comparison of scores generated by the three automated approaches, and (3) the extent to which models developed for scoring one NAEP prompt generalize to other NAEP prompts of the same genre. Data came ...

Scoring constructed-response items can be highly difficult, time-consuming, and costly in practice. Improvements in computer technology have enabled automated scoring of constructed-response items. However, the application of automated scoring without an investigation of test equating can lead to serious problems. The goal of this study was to score the constructed-response items in mixed-format tests automatically with different test/training data rates and to investigate the indirect effect of these scores on test equating compared with human raters. Bidirectional long-short term memory (BLSTM) was selected as the automated scoring method for the best performance. During the test equating process, methods based on classical test theory and item response theory were utilized. In most of the equating methods, errors of the equating resulting from automated scoring were close to the errors occurring in equating processes conducted by human raters. It was concluded that automated scoring can be applied because it is convenient in terms of equating.

Managing crowded classes in terms of classroom assessment is a difficult task due to the amount of time which needs to be devoted to providing feedback to student products. In this respect, the present study aimed to develop an automated essay scoring environment as a potential means to overcome this problem. Secondarily, the study aimed to test if automatically-given scores would correlate with the scores given by a human rater. A quantitative research design employing a machine learning approach was preferred to meet the aims of the study. The data set to be used for machine learning consisted of 160 scored literary analysis essays written in an English Literature course, each essay analyzing a theme in a given literary work. To train the automated scoring model, LightSide software was used. First, textual features were extracted and filtered. Then, Logistic Regression, SMO, SVO, Logistic Tree and Naïve Bayes text classification algorithms were tested by using 10-Fold Cross-Validation to reach the most accurate model. To see if the scores given by the computer correlated with the scores given by the human rater, Spearman's Rank Order Correlation Coefficient was calculated. The results showed that none of the algorithms were sufficiently accurate in terms of the scores of the essays within the data set. It was also seen that the scores given by the computer were not significantly correlated with the scores given by the human rater. The findings implied that the size of the data collected in an authentic classroom environment was too small for classification algorithms in terms of automated essay scoring for classroom assessment.

Automated Essay Scoring (AES) has been quite popular and is being widely used. However, lack of appropriate methodology for rating nonnative English speakers' essays has meant a lopsided advancement in this field. In this paper, we report initial results of our experiments with nonnative AES that learns from manual evaluation of nonnative essays. For this purpose, we conducted an exercise in which essays written by nonnative English speakers in test environment were rated both manually and by the automated system designed for the experiment. In the process, we experimented with a few features to learn about nuances linked to nonnative evaluation. The proposed methodology of automated essay evaluation has yielded a correlation coefficient of 0.750 with the manual evaluation.

We describe a new representation of the content vocabulary of a text we call word
association profile that captures the proportions of highly associated, mildly associated,
unassociated, and dis-associated pairs of words that co-exist in the given
text. We illustrate the shape of the distribution and observe variation with genre
and target audience. We present a study of the relationship between quality of writing
and word association profiles. For a set of essays written by college graduates
on a number of general topics, we show that the higher scoring essays tend to have
higher percentages of both highly associated and dis-associated pairs, and lower
percentages of mildly associated pairs of words. Finally, we use word association
profiles to improve a system for automated scoring of essays.

Character-based representations in neural models have been claimed to be a tool to overcome spelling variation in word token-based input. We examine this claim in neural models for content scoring. We formulate precise hypotheses about the possible effects of adding character representations to word-based models and test these hypotheses on large-scale real-world content scoring datasets. We find that, while character representations may provide small performance gains in general, their effectiveness in accounting for spelling variation may be limited. We show that spelling correction can provide larger gains than character representations, and that spelling correction improves the performance of models with character representations. With these insights, we report a new state of the art on the ASAP-SAS short content scoring dataset.

Information and communication technologies (ICTs) have been
provided by higher education authorities to ensure that teachers use ICT to enhance their teaching, learning and research outcomes. However, ICT has been found to be underutilized, especially by  teachers. To investigate this issue, the technology adoption and gratification (TAG) model was validated and used to examine Malaysian university teachers' adoption and gratification of ICT for teaching and research purposes and then used to investigate the moderating effect of universities in different regions. Three hundred and ninety-seven teachers from west and east Malaysian universities were surveyed using a stratified random sampling technique. The research data were analysed using three-stage structural equation modelling. There were focused use factors: (a) academic purpose, (b) research purpose, and (c) teaching purpose. The TAG model was useful in explaining differences in ICT uses. West Malaysian teachers' perceived usefulness and ease of use mediated the relationships between computer self-efficacy, gratification and intention to use ICT, respectively. However, for east Malaysian teachers, only perceived ease of use did not mediate the relationship between computer self-efficacy and gratification. Finally, west and east Malaysian teachers' intentions to use ICT mediated the relationships between actual use, perceived usefulness and ease of use. This research confirms the utility of the TAG model for comparing teachers' adoption and gratification of ICT. In addition, this study provides a basis for targeted professional development of teachers in east and west Malaysian universities which were selected based on known differences.

It is important for developers of automated scoring systems to ensure that their systems are as fair and valid as possible. This commitment means evaluating the performance of these systems in light of construct-irrelevant response strategies. The enhancement of systems to detect and deal with these kinds of strategies is often an iterative process, whereby as new strategies come to light they need to be evaluated and effective mechanisms built into the automated scoring systems to handle them. In this paper, we focus on the Babel system, which automatically generates semantically incohesive essays. We expect that these essays may unfairly receive high scores from automated scoring engines despite essentially being nonsense.

We found that the classifier built on Babel-generated essays and good-faith essays and using features from the automated scoring engine can distinguish the Babel-generated essays from the good-faith ones with 100% accuracy. We also found that if we integrated this classifier into the automated scoring engine it flagged very few responses that were submitted as part of operational submissions (76 of 434,656). The responses that were flagged had previously been assigned a score of Null (non-scorable) or a score of 1 by human experts. The measure of lexical-semantic cohesion shows promise in being able to distinguish Babel-generated essays from good-faith essays.

Our results show that it is possible to detect the kind of gaming strategy illustrated by the Babel system and add it to an automated scoring engine without adverse effects on essays seen during real high-stakes tests. We also show that a measure of lexical-semantic cohesion can separate Babel-generated essays from good-faith essays to a certain degree, depending on task. This points to future work that would generalize the capability to detect semantic incoherence in essays.

This thesis explored the integration of automated essay scoring (AES) software into the writing curriculum for high level ESOL students (levels 3, 4, and 5 on a 1-5 scale) at a high school in Miami, Fl. Issues for Haitian Creole speaking students were also explored. The Spanish and Haitian Creole speaking students were given the option to write notes, outlines, and planning sheets in their L1.

After using AES in the middle of the writing process as a revision assistant tool, 24 students responded to a Likert Scale questionnaire. The students responded positively to the AES based on the results of the Likert scale questionnaire: 71% responded “agree” and “strongly agree” to the question “Other students would benefit from using writing software before handing in a final draft.” Also, the majority reported that they valued teacher feedback. None of the students chose to use their L1 to write notes/outlines.

Researchers studying technology development often examine how rhetorical activity contributes to technologies’ design, implementation, and stabilization. This article offers a possible methodology for studying one role of rhetorical activity in technology development: knowledge consolidation analysis. Applying this method to an exemplar case, the author describes how explanations of Project Essay Grade (PEG), the first initiative to computerize student essay assessment, made knowledge available about this technology project. More specifically, technologist Ellis Page and his coauthors reworked a key explanatory argument, a justification of PEG’s functioning, during the course of several decades, refining and clarifying its key contrasts and strengthening its presentation for generalist educators in particular. Analysis suggests that late presentations of the argument reveal that knowledge was successfully consolidated about a technical procedure Page and his coauthors called “rating simulation.” The conclusion discusses the key advantages and limitations of the method.

Özellikle geniş ölçekli testlerde açık uçlu maddelerin kullanılması açık uçlu maddeleri puanlama konusunda zorluk ortaya çıkarmıştır. Ancak açık uçlu maddelerin otomatik puanlanmasına dayalı yaklaşımla bu zorluğun üstesinden gelinebilmektedir. Araştırmanın amacı açık uçlu maddeleri otomatik puanlayarak elde edilen verilerin güvenirliğini incelemektir. Alt amaçlardan birisi otomatik puanlamada makine öğrenmesine dayalı farklı algoritmaları (destek vektör makineleri, lojistik regresyon, çok terimli sade bayes, kısa-uzun süreli bellek ve iki yönlü kısa-uzun süreli bellek) karşılaştırmaktır. Diğer alt amaç ise otomatik puanlamaya özgü sistemin test edilmesinde (%33, %20 ve %10) kullanılan veri oranının farklılaştırılmasıyla otomatik puanlamanın güvenirliğinin değişimini incelemektir. Otomatik puanlamaların güvenirliği incelenirken gerçek puanlayıcılardan elde edilen verilerin güvenirliği ile karşılaştırma yapılmıştır. Türk dilindeki açık uçlu maddelerin ilk otomatik puanlama denemesini gösteren bu araştırmada Millî Eğitim Bakanlığı tarafından uygulanan Akademik Becerilerin İzlenmesi ve Değerlendirilmesi (ABİDE) programı Türkçe test verileri kullanılmıştır. Sistemin test edilmesinde çapraz geçerlikten yararlanılmıştır. Güvenirliği gösterecek uyum katsayılarına yönelik olarak uyum yüzdesi, otomatik puanlama araştırmalarında sıklıkla kullanılan karesel ağırlıklı Kappa ve verinin kategorilere dağılımındaki dengesizlik sorunundan etkilenmeyen Gwet'in AC1 katsayısı kullanılmıştır. Araştırma sonuçları otomatik puanlama algoritmalarından faydalanılabileceğini göstermiştir. Otomatik puanlamada kullanılabilecek en iyi algoritmanın iki yönlü kısa-uzun süreli bellek olduğu bulunmuştur. Kısa-uzun süreli bellek ve çok terimli sade bayes algoritmaları; destek vektör makineleri, lojistik regresyon ve iki yönlü kısa-uzun süreli bellek algoritmalarından daha düşük performans sergilemiştir. Otomatik puanlamada %33 test veri oranında uyum katsayılarının %10 ve %20 test veri oranlarına göre biraz daha düşük olduğu ancak istenilen aralıkta olduğu belirlenmiştir. Anahtar Kelimeler: Açık uçlu madde, makine öğrenme algoritmaları, otomatik puanlama, puanlayıcılar arası güvenirlik, uyum katsayıları. GİRİŞ Bireyler yaşamları boyunca çok sayıda testle karşılaşırlar. Testler, bireylerin bilgi, beceri ve yetenekleri arasındaki farkları gösterir. Böylece bireyler hakkında kararlar alabilir (Geisinger ve Usher-Tate, 2016). Son yıllarda testlerde birden fazla madde formatı kullanımı daha çok rağbet görmektedir. Karma test olarak adlandırılan bu yaklaşımda aşina olunan çoktan seçmeli maddelerin yanında yanıtı sınırlandırılmış ya da sınırlandırılmamış açık uçlu maddeler kullanılmaktadır. Çoktan seçmeli maddelerde bireyler bir problemle ilgili bir doğru, birden fazla yanlış cevapla karşılaşmaktadır. Yanıtı sınırlandırılmış açık uçlu maddelerde bireyler sorulara birkaç kelime, cümle ya da paragrafla cevap verirken yanıtı sınırlandırılmamış maddelerde bireyler sorulara istedikleri uzunlukta cevap vermektedir (Downing, 2009). Madde türlerinin bir arada kullanımı her bir formatın kısıtlı yönlerinin ortadan kaldırılmasını sağlamaktadır (Messick, 1993). Örneğin testlerde sadece çoktan seçmeli maddelerin kullanımı öğretime etki etmekte, bireyler çoktan seçmeli testlere yönelik çalışma yapmaktadır. Bu durum özgün, eleştirel ve üst düzey düşünme becerilerini kısıtlayabilmektedir. Ancak açık uçlu maddelerin kullanımı bu kısıtlılığı ortadan kaldırabilmektedir.