The Language Teacher
September 1999
Program Evaluation
Alister Cumming
The Aoyama University English Department 50th anniversary lecture series
During the week of May 24, Alister Cumming gave a series of lectures and workshops at Aoyama University, Tokyo, as part of the English Department's 50th anniversary celebrations.
"Education is an inexact area of activity, so evaluation is a way of appreciating the art of teaching," said Cumming. "Evaluation creates an awareness of the richness, the creativity, and the philosophies of the people involved."
Best known for his research into second language writing, Cumming heads the Modern Language Centre at the Ontario Institute for Studies in Education (OISE), University of Toronto. Past editor of the journal Language Learning, he is on the Writing Team of the TOEFL 2000 Project, and he has served as a consultant on some 20 different program evaluations, including a recent survey of language development in 25 countries.
"In North America, these surveys are in the newspapers all the time because everyone's concerned about which country is on top. That's kind of misleading, a 'racehorse approach.'
"For the top third, the scores are so similar that the differences don't matter and are often the result of the tests used. The lower third should be concerned about their education but usually these are countries with a lower socioeconomic state of development and they don't have the resources or the values of the upper third."
According to Cumming, the real value of comparative surveys was to describe different approaches in education and to develop descriptive models of the process. He drew one such model on the blackboard, a simple interaction between four different variables. They met like streets at an intersection. From within these terms, he described education as the intersection of teaching, learning, a social context, and a content or subject matter. In the case of EFL, the content was language education.
Cumming distinguished the purposes of evaluators from those of researchers in education: An evaluation may yield valuable insights into the educational process, but its purpose is to gather information in order to make decisions about a program. He then listed seven benefits of evaluation: (a) validating educational innovations, (b) informing program development, (c) ascertaining what students learn, (d) illuminating the perspectives of a particular group, (e) clarifying an educational rationale, (f) proposing ethical criteria, (g) appreciating the art of education.
When asked, "What's the best method of teaching language?" he emphasized the importance to learners of using a language for communication in meaningful, relevant ways, and he stressed that language learning took a very long time and great efforts. But Cumming challenged the assumption of a single approach even within a single curriculum. He differentiated between the intended achievement of a language education program, the implemented curriculum, and the achieved curriculum.
"When you talk about a method, that's at the level of the intended curriculum--what you're supposed to do. When you study teachers, you find out they do a lot of different things. And even if the method or the textbook is supposed to be the same, students attend to different things."
Cumming described how he had been involved in an assessment of a new program in Ontario and found a complete mismatch between expectations and results. "What the teachers thought they were doing was very different from what the program described. The problem was that after the program had been developed, there was no money left for implementation, for teacher training. The teachers were just given the curriculum guides."
In another case, he took part in a four-month evaluation of ESL programs in Vancouver. He reviewed policy and curricular documents, and he interviewed and observed teachers in schools of diverse ESL populations and economic levels. One decision which would emerge from the assessment was a common one in North America: Should the growing ESL student population have a separate program or be mainstreamed into regular classrooms--a cheaper alternative. One part of his study showed that some regular classroom teachers were coping with ESL students in their classes. Although he did not recommend it, one year later, the provincial government used this finding as a rationale to cut ESL programs.
Ideally, Cumming explained, evaluation can be an important tool in improving the quality of education. Among its benefits he reviewed, evaluation could illuminate the perspectives of a particular group: In Japan for example, returnees are functionally bilingual in English but lack the equivalent academic skills. Evaluation can also clarify the educational rationale for a program and make the goals clearer to the staff, freeing teachers to pursue them by drawing on their own knowledge and experiences.
"An English language teacher I studied a few years ago was very musical, and she organized her classes around musical themes. Her students repeated things in choruses and she would orchestrate them. She put them into groups like little ensembles performing for other people."
Noting that individual teachers, consciously, or not, often work in terms of metaphor, Cumming added, "she was getting her students to rehearse so they could perform as university students in their second language."
reported by Gregory Strong
All materials on this site are copyright © by JALT and their respective authors.
For more information on JALT, visit the JALT National Website