Meaningful Measurement: Teachers and Test Making

Page No.: 
Jay Wong, Temple University; Takayuki Okazaki, Kyoto Sangyo University; Andrew Atkins, Shiga University


In this paper we describe teacher-led efforts to develop and revise a set of large-scale, university-wide tests for an English language program at a university in Western Japan. We hoped through this work to create a common measurement scale so that student performance on different versions of the tests could reliably be compared. We discuss the challenges of implementing teacher-managed test development in our work context and raise concerns about the lack of attention paid to construct mapping in the initial stages of the test development cycle. Using data from the reading section of our test, we show how we wrote and piloted test passages and items which gave us better information for student placement and end-of-term assessment. Also, we explain how we used Rasch modeling to determine how test items performed. In addition to statistical analysis of test items, we also suggest that qualitative student feedback be used when considering items for test inclusion or exclusion. Finally, we conclude that ideal test construction is an iterative and time-consuming process that offers the most promise to institutions willing to take a long-term perspective. With proper institutional support teachers can play a leading role in test development.