The Language Teacher

TJL: Dr. Brown, we appreciate your coming to our college to speak today, and thank you for taking the time to allow me to interview you.

Among foreign and Japanese university educators, you are highly regarded as an authority on language testing, and are an outspoken critic of the Japanese university entrance examination system. How did you first become involved in researching and studying the entrance examination system in Japan?

JDB: The second time I came to Japan I taught a testing course, and the issue of entrance exams always surfaced. A lot of people everywhere are very concerned about them. There is no question about that. There are people, of course, who apologize for them, in the sense that they argue for them. They try to explain to me that I am a gaijin (foreigner) and should just mind my own business.

To be honest, I have often felt the same way--that it really is a Japanese problem, and J.D. Brown can't do anything to contribute to solving the problem. Eventually, I got to know a doctoral student of mine here in Japan. Little by little, I got to know her family, and I saw first hand some of the problems her boys were going through with the exams. It became real to me suddenly, that there are kids out there who are struggling like crazy and who are suffering.

TJL: In order to put the issue of university entrance examinations in better perspective, could you comment upon the US and Japanese systems in a comparative/contrastive manner, emphasizing some (if any) of the more outstanding similarities and a few of the differences?

JDB: Of course, there are similarities. In the US, we have exactly the same problem in that we have to decide (because there are limited resources) who gets to go to what school. As long as there is more demand than there is supply, we have to make decisions about the entrance of students to our universities. That is probably, however, where the similarities stop.

US universities do not have individual entrance exams like in Japan. Instead, we have centralized tests, similar to the Center Exam in Japan, but different as well. Organizations called the Educational Testing Service and the College Boards which do the SAT [Self-Aptitude Test] and the ACT [American College Test] are completely independent, non-profit organizations. Virtually every US college student takes the SAT, and that single score (which they can take over and over again, in any month of the year, almost anywhere, and at a much lower cost), is used by universities to partially gauge acceptance into the institution. All of these factors are big differences between the Japanese and American systems. Heaven help kids here who get sick in the winter because there is just no hope in Japan of them passing the exam that year.

One of the problems is that the exams have been very similar for many, many years. Interestingly, back in March of 1993, Monbusho issued guidelines for communicative language teaching, particularly the listening and speaking skills. These guidelines advocate teaching language that would be very useful for students, but the entrance examinations haven't changed one wit!

Another major difference between the US system and the Japanese system is that in the US, by and large, preparation for the exams is not as big of an industry. Most people in the US see these exams as tests of overall ability. Something that can't be prepared for really. That is certainly not the case in Japan, where the belief system seems to be that you can study for the exams, and indeed you had better or you're not going to make it.

TJL: What do you personally regard as the most pressing issues concerning Japanese entrance examinations currently, and how can these be remedied?

JDB: Perhaps one thing that could be done would be to change the entrance examination question formats to make them more homogeneous. Maybe a format consisting of 4 or 5 different variations that are grouped together so that the student isn't constantly shifting gears in terms of the testing method, would be more appropriate. But more importantly, perhaps, it would be beneficial to allow the tests themselves to focus on proficiency--the ability of the students to do something with the language.

There is something called the "washback effect" that we are talking about a lot in the United States right now. This is the effect of tests on the curriculum that's related to them, or the curriculum that precedes them. There's no question in my mind, that the entrance exams in Japan have a very definite washback effect at the high school level of teaching English. It definitely goes on. Basically, teachers teach to prepare for particular tests. The same is true for the yobiko and juku [cram schools]. In fact, these schools gain customers by having a proven track record with certain exams. There is a really high anxiety level involved with these exams--studying for them and getting ready for them.

TJL: As well as making them. University teachers all over Japan agonize over how to properly create tests that are fair.

JDB: Ahh, well, that is a worrisome aspect of it, too. I suspect that many teachers feel like that. Actually, many people that I've talked to are concerned about the fact that they are doing something for which they are not prepared. Why is it that Japan has 300 exams or more? These exams are being made by people who don't know what they're doing, who say they don't know what they're doing. They are doing the best they can, but ultimately, they don't know what they are doing. They are preparing tests that are haphazard and of unknown reliability and validity.

The sad thing is that these tests are then used to make very, very important decisions about peoples' lives. All of this wouldn't bother me so much, if the people making the tests were looking at them in an effort to improve them. There are many ways which the results of a test can be analyzed statistically and logically to help to make a better product for the next time out. In the US, we've worked out systems whereby we pilot items and actually know what will happen when they are used. But I'm told for thousands of different reasons that this is not possible in Japan. This is something I'm not sure I believe. . .

TJL: J.D., what would you recommend teachers, both Japanese and foreign, do when placed in the awkward position of developing these exams, in order to make the test items fair, unambiguous and scientifically sound?

JDB: Probably the single most important thing to do would be to do an analysis of the results each year, and look at what each question did. There are very simple statistics that one can do. Virtually any testing book will describe them. One of them is called "item facility," which is simply the percent on each item that is answered correctly. This statistic gives some idea of the difficulty level of that item. The other is called the "item discrimination index," which is how well the item separates out the really top students from the bottom students, based on the total scores. You look at each item and compare the item facility for the upper to the item facility for the lower, maybe third. One can simply be subtracted from the other. If all of the students in the upper group, say 100% answered it correctly and none in the lower group answered it correctly, 100% minus 0%, would be 100%. So, that would be a very good discriminating item. After selecting those items that discriminate well, it would be wise to use them in a pilot mode, then use them again in a live testing situation.

TJL: Universities here are very hesitant about piloting exam material.

JDB: Of course, of course, because security is such a big issue. But, alternatively, what can be done is to learn from these results, like which kinds of questions work well. Then use those kinds of questions in the future, similar but not the same. It is important to avoid those kinds of questions that are just not discriminating well, or are discriminating in a negative manner, which may be the case, too. The problem is that items that are not discriminating or are discriminating negatively, are creating noise in the decision making. They are making the results more or less random, and this is why reliability may be low on some of these tests.

Another issue that comes up, that a teacher can actually affect locally, is to try to move for the use of more than one piece of information for admission purposes. The test is a useful piece of information, but we want to know more about the students, so it is necessary to use resources that are available like the grade point average (GPA) from high school; perhaps a school may want to ask students to write things like statements of purpose, to see what the students have to say, and have letters of recommendation submitted. In fact, any other pieces of information that can be put together in making the decision that will help to make the whole decision more reliable is beneficial. Each one has its own reliability problems, but when you look at them together, they should cross-validate each other, which in the end will make the decision much, much better. This is a policy decision that people can sort of move toward.

Administrators are so very aware of the effect the tests have on their financial situation, and how they are viewed on the outside. The issue of prestige is a big one with these exams. If you have a hard exam, then it must be a good university. Hence, there is a very prestige oriented reason for wanting these exams to be so tough. Adding to this is the belief that by adding a listening test or maybe even something as radical as an oral interview, there is a general fear that the students will be scared away. The students haven't prepared for these kinds of components, and they won't want to take the entrance exam at that particular university. So, from an administrator's perspective, they are simply worried that the students will not come. They need lots of students to come to the exam to generate the money they need for buildings and for various bonuses and such.

TJL: Finally, J.D., if you could wave a magic wand and change five aspects of the Japanese university examination system that is currently in place, what would they be, and why?

JDB: I think the first thing I would do would be to centralize the university exam system, maybe or maybe not under the jurisdiction of Monbusho. That would be my number one change.

Second, if English is to be tested in that system, then it should be tested in a variety of different ways, including of course some of the tried and true tested, reliable and valid methods. These should include grammar tests, reading tests and listening tests--but also tests of other skills like speaking and writing for sure. Even though there are some problems with the reliability of those, they can be worked out with adequate resources. This would, in the end, give a much better profile of the students in terms of not only what they understand, but also what they can do with it. So first, centralize and second, broaden the definition of what English proficiency means on any exams that are given in English, nationally.

Third, I would use multiple sources of information in the systems of admission. This is so a variety of different types of information, which we discussed earlier, would be considered and not just the language tests.

Fourth, I would apply the same standards to everybody in Japan. That's not the way it is done now. There are too many other ways to get into these universities. For example, if a student gets into a well-known high school which is affiliated with one of the very well-known universities, the chances are almost guaranteed in some situations that the student will get in. Everybody that I have talked to agrees that many of those students aren't as good as the ones who enter by sitting for the entrance exam.

And fifth, I would make sure that there is some contingency plan in place to help universities find the resources that have been whipped out from under them by having the entrance exam centralized. It is important for universities to find ways to pay for things by the people who benefit from them, rather than depending on sitting fees from failed students to do business as usual. Perhaps the introduction of affiliations such as alumni societies where former, successful alumni donate money for specific projects. In the beginning, however, it would take some government subsidies to get this to happen, but eventually universities could generate their own money to replace the money that they depended on from the entrance exam sitting fee.

So, those would be the five things that I would change, and I have never actually listed them together before now.

TJL: J.D., thank you for taking the time to talk with me today. You have touched upon some very interesting points which certainly need further discussion and deliberation. Hopefully, this interview will be of interest and benefit to those currently involved in the preparation and development of university examinations in Japan.