It is important for teachers and researchers to be able to assess L2 learners’ proficiency through their performance. The measures of complexity, accuracy, and fluency (CAF) have been used for over 30 years in L2 research to analyze language performance (Ellis & Barkhuizen, 2005; Housen & Kuiken, 2009; Wolfe-Quintero, Inagaki, & Kim, 1998). However, there remain unanswered questions about CAF measures. For example, the length measure (number of words per syntactic unit) needs to be investigated because it has been used inconsistently: Some researchers have used it as a syntactic complexity measure, while others have used it as a fluency measure. Koizumi (2005) pointed out this discrepancy as a serious problem because the interpretation of a single measure varies depending on researchers’ orientations. In addition, the results of factor analysis across some studies (e.g., Sheppard, 2004; Tavakoli & Skehan, 2005) showed that the measures of fluency could be divided into two types: speed and disfluency. These discrepancies in the construct are key issues pertaining to the measures and need to be investigated. Moreover, although CAF measures have often been investigated in Indo-European languages, they have not been sufficiently investigated in other languages; thus, it is important to determine whether CAF measures can be applied to the Japanese language in the same way.
Accordingly, this study examined the construct validity of CAF measures in Japanese as a second language (JSL) from the following three perspectives.
- As expected, do the measures represent distinct factors of the three CAF dimensions?
- Does the length measure (the number of words per syntactic unit) have the same construct as the syntactic complexity measure does, rather than having that of the fluency measure?
- Can the speed measure (the number of words per minute) and the disfluency measure (the number of disfluency markers per minute) be explained as one construct?
To investigate these research questions, 10 general measures were calculated from the narrative production of 113 university-level students learning JSL. Their Japanese language proficiency level ranged from intermediate to advanced on the ACTFL–OPI (American Council on the Teaching of Foreign Languages–Oral Proficiency Interview). Subsequently, a factor analysis was conducted to investigate the construct validity of CAF measures. The initial solution was extracted using the principal factor method, followed by Promax rotation. A three-factor solution was adopted using the Kaiser criterion of eigenvalues greater than one.
The results of this factor analysis indicated the following.
- The validity of CAF measures was partially demonstrated in terms of syntactic complexity measures (number of clauses per Analysis of Speech unit [AS-unit], and number of subordinate clauses per AS-unit) and accuracy measures (percentage of error-free AS-units, number of errors per AS-unit, and number of errors per clause) but was not demonstrated in terms of lexical complexity measures (number of word types per 100 words, and the Guiraud index) or fluency measures (number of words per minute, and number of disfluency markers per minute).
- The length measure indicates syntactic complexity because of the high loading (.83) on the same factor with the general syntactic complexity measures.
- The speed measure and the disfluency measure did not have the same factor as one construct of the fluency, which in turn supports the findings of previous studies (Sheppard, 2004; Tavakoli & Skehan, 2005). The results of this study suggest that further research must be conducted to establish the validity of the fluency measure and the validity of the lexical complexity measure, especially for the Japanese language, which has an agglutinating morphology.