Abstract
In the present study, we tested a computer-based procedure for assessing very concise summaries (50 words long) of two types of text (narrative and expository) using latent semantic analysis (LSA) in comparison with the judgments of four human experts. LSA was used to estimate semantic similarity using six different methods: four holistic (summary-text, summary-summaries, summary-expert summaries, and pregraded-ungraded summary) and two componential (summary-sentence text and summary-main sentence text). A total of 390 Spanish middle and high school students (14–16 years old) and six experts read a narrative or expository text and later summarized it. The results support the viability of developing a computerized assessment tool using human judgments and LSA, although the correlation between human judgments and LSA was higher in the narrative text than in the expository, and LSA correlated more with human content ratings than with human coherence ratings. Finally, the holistic methods were found to be more reliable than the componential methods analyzed in this study.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Brown, A. L., &Day, J. D. (1983). Macrorules for summarizing texts: The development of expertise.Journal of Verbal Learning & Verbal Behavior,22, 1–14.
Bruner, J. (1986).Actual minds, possible worlds. Cambridge, MA: Harvard University Press.
Einstein, G. O., McDaniel, M. A., Owen, P. D., &Coté, N. C. (1990). Encoding and recall of texts: The importance of material appropriate processing.Journal of Memory & Language,29, 566–581.
Escudero, I. (2004).Procesamiento de inferencias elaborativas en la comprensión del discurso y según el tipo de texto [Elaborative inference processing in discourse comprehension and type of text]. Unpublished doctoral dissertation, Universidad Autónoma de Madrid.Fletcher, C. R., Chrysler, S. T., van den Broek, P., Deaton, J. A., & Bloom, C. P. (1995). The role of co-occurrence, coreference, and causality in coherence of conjoined sentences. In R. F. Lorch, Jr. & E. J. O’Brien (Eds.),Sources of coherence in reading (pp. 203–218). Hillsdale, NJ: Erlbaum.
Foltz, P. W. (1996). Latent semantic analysis for text-based research.Behavior Research Methods, Instruments, & Computers,28, 197–202.
Foltz, P. W., Gilliam, S., &Kendall, S. (2000). Supporting contentbased feedback in on-line writing evaluation with LSA.Interactive Learning Environments,8, 111–127.
Foltz, P. W., Kintsch, W., &Landauer, T. K. (1998). The measurement of textual coherence with latent semantic analysis.Discourse Processes,25, 285–307.
Glenberg, A. M., &Robertson, D. A. (2000). Symbol grounding and meaning: A comparison of high-dimensional and embodied theories of meaning.Journal of Memory & Language,43, 379–401.
Goldman, S., &Bisanz, G. (2002). Toward a functional analysis of scientific genres: Implications for understanding and learning processes. In J. Otero, J. A. León, & A. C. Graesser (Eds.),The psychology of science text comprehension (pp. 19–50). Mahwah, NJ: Erlbaum.
Graesser, A. C., Singer, M., &Trabasso, T. (1994). “Constructing inferences during narrative text comprehension.”Psychological Review,101, 371–395.
Graesser, A. C., Wiemer-Hastings, P., Wiemer-Hastings, K., Harter, D., Person, N., &the Tutoring Research Group (2000). Using latent semantic analysis to evaluate the contributions of students in AutoTutor.Interactive Learning Environments,8, 129–147.
Hartley, J., &Trueman, M. (1983). The effects of headings in text on recall, search and retrieval.ritish Journal of Educational Psychology53, 205–214.
Kintsch, E., Steinhart, D., Stahl, G., Matthews, C., Lamb, R., &the LSA Research Group (2000). Developing summarization skills through the use of LSA-based feedback.Interactive Learning Environments,8, 87–109.
Kucan, L., &Beck, I. L. (1996). Four fourth graders thinking aloud: An investigation of genre effects.Journal of Literacy Research,28, 259–287.
Landauer, T. K. (1998). Learning and representing verbal meaning: The latent semantic analysis theory.Current Directions in Psychological Science,7, 161–164.
Landauer, T. K., &Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge.Psychological Review,104, 211–240.
Landauer, T. K., Foltz, P. W., &Laham, D. (1998). An introduction to latent semantic analysis.Discourse Processes,25, 259–284.
Landauer, T. K., Laham, D., & Foltz, P. W. (1998).Computer-based grading of the conceptual content of essays. Unpublished manuscript.
Landauer, T. K., &Psotka, J. (2000). Simulating text understanding for educational applications with Latent Semantic Analysis: Introduction to LSA.Interactive Learning Environments,8, 73–86.
León, J. A., &Carretero, M. (1995). Intervention in comprehension and memory strategies: Knowledge and use of text structure.Learning & Instruction,5, 203–220.
León, J. A., Escudero, I., &van den Broek, P. (2003). La influencia del género del texto en el establecimiento de inferencias elaborativas [The influence of type of text on the establishment of elaborative inferences]. In J. A. León (Ed.),Conocimiento y discurso: Claves para inferir y comprender (pp. 153–170). Madrid: Pirámide.
León, J. A., &Peñalba, G. E. (2002). Understanding causality and temporal sequence in scientific discourse. In J. Otero, J. A. León, & A. C. Graesser (Eds.),The psychology of science text comprehension (pp. 155–178). Mahwah, NJ: Erlbaum.
León, J. A., & the Reading Literacy Research Group (2004).La competencia lectora y los procesos de comprensión: Un proyecto de investigación basado en la evaluación de los tipos de comprensión [Reading literacy and reading processes: A research project on assessment of types of comprehension]. Unpublished manuscript.
León, J. A., &Slisko, J. (2000). La dificultad comprensiva de los textos de ciencias: Nuevas alternativas para un viejo problema educativo [The difficulty of understanding science texts: New alternatives for an old education problem].Psicología Educativa,6, 7–26.
Magliano, J. P., Wiemer-Hastings, K., Millis, K. K., Muñoz, B. D., &McNamara, D. [S.] (2002). Using latent semantic analysis to assess reader strategies.Behavior Research Methods, Instruments, & Computers,34, 181–188.
Martins, I. (2002). Visual imagery in school science texts. In J. Otero, J. A. León, & A. C. Graesser (Eds.),The psychology of science text comprehension (pp. 73–90). Mahwah, NJ: Erlbaum.
Millis, K. K., Kim, H.-J. J., Todaro, S., Magliano, J. P., Wiemer-Hastings, K., &McNamara, D. S. (2004). Identifying reading strategies using latent semantic analysis: Comparing semantic benchmarks.Behavior Research Methods, Instruments, & Computers,36, 213–221.
Palincsar, A. S., &Brown, A. L. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities.Cognition & Instruction,1, 117–175.
Perfetti, C. A. (1998). The limits of co-occurrence: Tools and theories in language research.Discourse Processes,25, 363–377.
Polkinghorne, D. (1988).arrative knowing and the human sciences Albany: State University of New York Press.
Rehder, B., Schreiner, M. E., Wolfe, M. B. W., Laham, D., Landauer, T. K., &Kintsch, W. (1998). Using latent semantic analysis to assess knowledge: Some technical considerations.Discourse Processes,25, 337–354.
Steyvers, M., Shiffrin, R. M., &Nelson, D. L. (2005). Word association spaces for predicting semantic similarity effects in episodic memory. In A. F. Healy (Ed.),Experimental cognitive psychology and its applications (pp. 237–249). Washington, DC: American Psychological Association.
Trabasso, T., Secco, T., &van den Broek, P. (1984). Causal cohesion and story coherence. In H. Mandl, N. L. Stein, & T. Trabasso (Eds.),Learning and comprehension of text (pp. 83–107). Hillsdale, NJ: Erlbaum.
van Dijk, T. A., &Kintsch, W. (1983).Strategies of discourse comprehension. New York: Academic Press.
Wolfe, M. B. W. (2005). Memory for narrative and expository text: Independent influences of semantic associations and text organization.Journal of Experimental Psychology: Learning, Memory, & Cognition,31, 359–364.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
León, J.A., Olmos, R., Escudero, I. et al. Assessing short summaries with human judgments procedure and latent semantic analysis in narrative and expository texts. Behavior Research Methods 38, 616–627 (2006). https://doi.org/10.3758/BF03193894
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03193894