Powers (2005) recommends both correlational and experimental approaches for establishing the relevance of response length in the evaluation of written responses: “the former for ruling out response length (and various other factors) as causes of response quality (by virtue of their lack of relationship) and the latter for establishing more definitive causal links” (p. In all these contexts, the underlying question is a similar one: Should text length be considered when judging students’ writing – or is it a source of judgment bias? The objective of this paper is to investigate to what degree text length is a construct-relevant aspect of writing competence, or to what extent it erroneously influences judgments. In the context of classroom language learning and instruction, studies have found effects of text length on teachers’ diagnostic judgments (e.g., grades Marshall, 1967 Osnes, 1995 Birkel and Birkel, 2002 Pohlmann-Rother et al., 2016). Both in the area of educational measurement and of language technology, text length has been shown to strongly influence text ratings by trained human raters as well as computer algorithms used to score texts automatically ( Chodorow and Burstein, 2004 Powers, 2005 Kobrin et al., 2011 Guo et al., 2013). The relationship between such (superficial) aspects of written responses and the assessment of text quality has been a controversial issue in different areas of educational research. Judgments of students’ writing are influenced by a variety of text characteristics, including text length. The findings are discussed with respect to the role of training and context in writing assessment.
#ETS 5 WRITING EXAMPLES PROFESSIONAL#
Furthermore, an experimental study found that pre-service teachers tended to undervalue text length when compared to professional ratings. Secondary analyses were performed on a large-scale dataset with highly trained raters, showing an effect of text length beyond language proficiency. This paper used both a correlational and an experimental approach to investigate this question. The question of whether text length is a construct-relevant aspect of writing competence or a source of judgment bias has been discussed controversially. Text length has been found to strongly influence human judgment of text quality. The assessment of text quality is a transdisciplinary issue concerning the research areas of educational assessment, language technology, and classroom instruction. 3School of Education, Institute of Secondary Education, University of Applied Sciences and Arts Northwestern Switzerland, Brugg, Switzerland.
2Institute for Psychology of Learning and Instruction, Kiel University, Kiel, Germany.1Department of Educational Research and Educational Psychology, Leibniz Institute for Science and Mathematics Education, Kiel, Germany.Johanna Fleckenstein 1*, Jennifer Meyer 1, Thorben Jansen 2, Stefan Keller 3 and Olaf Köller 1