Few mixed methods research studies have been published in language learning and teaching. To encourage this methodology among English language teachers and researchers, I will introduce a simple logic model of interconnected steps in this research approach. The model starts with a mixed methods problem and continues with the collection and analysis of quantitative and qualitative data, the combination or integration of the two databases, the framing of integration within a specific type of mixed methods design, analysis of integration within a table of data, and finally, the interpretation or metainferences drawn from the quantitative and qualitative data combination. Through this process, participants will learn the language of mixed methods research, will be introduced to state-of-the-art thinking, and will see the practical value of using this methodology. I will end with a proposed mixed methods study in Japanese language learning based on my own experiences illustrating the steps in the logic model.
Mixed methods research studies are found in many fields in the social and health sciences. For language teaching and learning, authors of the overviews of using this methodology have lamented the lack of research studies. However, the conversation about using mixed methods has begun in language learning. In 2017, an entire book addressed its application in language learning and teaching (Riazi, 2017). A year earlier, my colleagues at Cambridge English and I authored a book titled Second Language Assessment and Mixed Methods Research (Moeller et al., 2016). Our efforts focused on bringing mixed methods in the second language field and encouraging their use. I remember studying carefully and citing a well-written language learning article by Wesely (2010) that addressed the motivation to learn languages in an immersion program. More recent publications in language studies provide a systematic research synthesis in language writing (Park et al., 2021) and explore web-based classroom instruction in language learning (Ebadi & Rahimi, 2018). Still, few articles link mixed methods to language teaching and learning.
My experiences in presenting workshops and lectures in Asian countries have encouraged me to clarify the meaning of mixed methods research for non-English speaking researchers. Consequently, in this paper, I present a simplified logic model that describes the major components of this approach. I will begin with an overview of the model, detail each component, and end by proposing a mixed methods study based on my experiences during the last three years as a language learner of Japanese. This paper reinforces the JALT2022 theme of educating teachers with the latest research practices.
Mixed Methods Logic Model
As mixed methods research has expanded around the world to new countries and disciplines, many researchers are learning about this methodology for the first time. Consequently, I want to make mixed methods as accessible as possible. Recently, I found an article by Curran (2020) titled Implementation Science Made Simple. This brilliant article succinctly summarized the major components of implementation science, a field that researchers often struggle to understand. By adapting this idea to mixed methods research, I have developed a simple way of explaining it.
An Overview of the Model
This overview is presented in Figure 1. The major components are in the boxes, and I present these in the order researchers use to design and conduct a study.
Figure 1
Interconnected Research Phases in a Mixed Method Study
The process of conducting research in this figure may be familiar. It begins with a research problem and a research question posed by the researcher. To answer the research question, the mixed methods researcher collects both quantitative and qualitative data. A key element in mixed methods is the combination of datasets. Although useful information develops from collecting and analyzing the quantitative and qualitative data, more insights emerge from combining the datasets. The researcher plans procedures for combining the data (the use of a design). Then, after collecting and analyzing the results from the two datasets, the researcher examines the results together to see what insights emerge and interprets them. Now I will discuss each component in more detail.
Mixed Methods Problems and Research Questions
What type of mixed methods problem is suitable for a mixed methods study? Unfortunately, the literature in the field does not address this question often enough. The suitable problems come from many disciplines and fields. Certainly, some fields support only quantitative approaches to research (e.g., economics), and some tend to highlight only qualitative approaches (e.g., anthropology). A cursory look at social sciences and health journals reveals the publication of many mixed methods studies across diverse fields. First, the problem needs to be addressed best with data from quantitative and qualitative sources. This means gathering information based on instruments, observations, and documents that will yield numeric scores. It also means obtaining personal perspectives from people and respecting and collecting their viewpoints. Thus, either quantitative or qualitative data alone will not suffice to best understand the problem or question. Second, the problem can best be understood not only with the two sources of data but also with the insight to emerge from combining them. The researcher realizes the importance of looking across the two databases for information about the research problem. Thus, mixed methods are suitable when the researcher (1) has an opportunity to collect both quantitative and qualitative data and (2) realizes that something more—more insights—will emerge by combining the two databases. With this understanding of collecting quantitative and qualitative data and seeking insight from the merging of the two datasets, the research questions for a mixed methods study naturally follow. The researcher raises quantitative questions that probe both descriptive (e.g., frequency of something happening) and inferential questions (e.g., how can the results be inferred from a sample to a population). These inferential questions often address the relationship among variables (e.g., independent and dependent) or comparisons among groups (e.g., how does group 1 differ from group 2?). Qualitative questions allow participants to give responses (e.g., what are the views about X?). In mixed methods research, we have a third question which addresses what insight emerges from combining the two databases. It is easiest to form this question once a researcher identifies a mixed methods design or procedure for the study because the question differs depending on the type of design. Further, this question is unique to mixed methods research because of the combined feature of this methodology. Examples of mixed methods research questions can be found in Creswell and Plano Clark (2018).
A Mixed Methods Data Inventory
With the desire to answer research questions, the researcher needs to plan and collect quantitative and qualitative data. These two types of data differ. They yield distinct types of information (recognizing that some researchers see less distinction than I do). Some say that quantitative research is numbered (numeric) data and qualitative (text) stories. Others report that quantitative research consists of scores and qualitative research of perspectives. My view focuses on the types of questions participants answer. In quantitative research, the investigator asks a question and provides response options. For example, participants respond on a scale from strongly agree to strongly disagree in closed-ended response options. In qualitative research, open-ended responses exist. The researcher asks a broad question and allows the participant to form response options. To me, this distinction—between closed-ended and open-ended questions—provides a clear distinction between the two data sources. Within the category of closed-ended quantitative data, I include instrument data, observation checklists, and documents reporting scores. For open-ended qualitative data, I incorporate interviews, observations, documents (with text), and information from visual and social media. In mixed method research, if a central feature of this form of research consists of bringing two databases together, then identifying the two databases separately is necessary. A data inventory table provides a means for accomplishing this division. This inventory is a table with two columns, one for the quantitative data and the other for qualitative data. The researcher lists the sources of data (e.g., specifically identifying attitudinal instruments used quantitatively or focus groups for qualitative data) in each column in this table. The table also includes information, such as the number of participants and the place or site for data collection. With this data inventory table, we can now see the distinct types of both databases for the project.
Integrating the Two Datasets
How do we combine the two datasets? The answer is not intuitive to most researchers. Researchers traditionally keep the two datasets separate and seldom consider combining them. This combination is a major feature introduced into research by mixed methods writers. They call it integrating the two datasets. Integration has been a confusing subject even within the mixed methods field (Bryman, 2006). Recently, writers have begun to understand the concept better (Lynam et al., 2019; Moseholm & Fetters, 2017).
Helpful ways of viewing integration are to see it as linking the two datasets in different ways and to consider the intent or reasons for this linkage. In terms of linking the two datasets, there are several ways to accomplish it. Later I will connect these ways to specific types of mixed methods designs. First, the forms of linking procedures are as follows: I can (1) merge the two datasets by bringing them into a single framework, (2) connect the two datasets by starting with quantitative data and then using its results to inform qualitative data collection, (3) connect the two datasets again but start with qualitative data and use its results to inform the subsequent quantitative data collection, and (4) use quantitative and qualitative data to augment a framework or process (e.g., an experimental process or an evaluation framework). These approaches represent ways to conduct the procedures of research.
Second, I can link the two datasets and consider my intent. I can link the two databases with the intent of (1) comparing the results from the two databases, (2) using the qualitative data to help explain the quantitative results in more detail, (3) using the qualitative data to help understand the sample or population and then modifying measures for adaptation to a group, and (4) support a framework or process by bringing in diverse perspectives grounded in the quantitative and qualitative data. In summary, I recommend considering both procedure and intent of integrating the two datasets.
A Mixed Methods Design
With an understanding of how and why I am integrating datasets in my study, I can now choose an appropriate mixed methods design. This choice is a difficult step. In the literature on mixed methods, many classifications, names, diagrams, and procedures exist (Creswell & Plano Clark, 2018, in press). Our thinking is to advance a parsimonious set of designs, a strategy we believe helps the new or international researcher. In my books on mixed methods (Creswell, 2022; Creswell & Plano Clark, 2018), we suggest two categories of designs: core designs found in all mixed methods projects and complex designs based on incorporating cores designs into a larger process or framework.
Core designs are foundational procedures in mixed methods. Complex designs represent a new frontier in designs, and writers use diverse names to refer to them, such as scaffold designs (Fetters, 2019) or advanced designs (Plano Clark & Ivankova, 2016).
The first core design, a convergent design, uses the procedure of gathering both quantitative and qualitative data and merging them into one framework. The intent or purpose is to compare the results of the two databases or to validate one set of data with the other. These two aspects form the integration in a convergent design. Another core design is the explanatory sequential design. The procedure involves first collecting quantitative data, analyzing it, and using the analysis to identify participants and research questions for a qualitative follow-up. The intent or purpose of this design is to use the follow-up qualitative data to help explain confusing, surprising, or important findings from the quantitative data. The final core design is an exploratory sequential design. The flow of research procedures involves three phases: first, collect and analyze qualitative data; second, use the qualitative findings to design or modify existing quantitative instruments, scales, or variables; and third, test the designed or modified quantitative measures and gather scores. This design is intended to adapt quantitative measures to fit a particular sample or population.
After using these core designs for many years, researchers began presenting us with studies involving many team members, lengthy studies, and extensive resources (Nastasi & Hitchcock, 2015). Today mixed methods as a methodology are being linked to other methodologies (e.g., evaluation studies), theoretical frameworks (e.g., feminist studies), and approaches (e.g., participatory action research) (Plano Clark & Ivankova, 2016). An example of a complex design is embedding a qualitative follow-up phase consisting of one or more of the core designs into the experiment. It could involve collecting qualitative data prior to the experiment, during the experiment, or after an experiment. It could apply to an evaluation project with core designs embedded in different phases of the evaluation.
We further learned from US funding sources that mixed methods were too complex to understand because of the multiple phases and forms of data collection and analysis. Consequently, from our earliest discussions about designs (Creswell et al., 2003), we began drawing diagrams of the design procedures. In time, these have become more sophisticated, and they are now found in most published mixed methods studies. These mixed methods diagrams provide a useful summary of the procedures helpful for informing research team members, stakeholders, and graduate committees.
A Joint Display Table
After data collection, the mixed methods researcher needs to analyze the data. The quantitative results are analyzed statistically, whereas the qualitative findings are analyzed for codes and themes. The sequence for accomplishing these two analyses depends on the particular mixed methods design. After independent quantitative and qualitative analysis, the researcher turns to mixed methods data analysis. Writers have only recently clearly addressed the procedures (Fetters, 2019). Mixed methods data analysis involves analyzing the integration of the quantitative and qualitative data. Both databases need to be placed side-by-side to see insights to emerge beyond analyzing the quantitative and qualitative data. Traditionally, this side-by-side analysis occurred by first discussing results from one database and then the results from the second database in the discussion section of a journal article. In the last few years the idea of a joint display emerged as a way to plan, present, and publish findings of the two databases side-by-side. Developing a joint display has become a creative part of mixed methods research with many options. It can be represented in a table (e.g., qualitative themes on the horizontal axis and scores on the vertical axis with quotes or scores in the cells). It can be a visual with photographs and words, a graphic design circle, or maps. Published articles report these variations in types of joint displays. Regardless of the presentation style, the key idea is to link the two databases following procedures in a mixed method design, and draw conclusions from the side-by-side comparison.
Metainferences
These conclusions are called metainferences, which involve analyzing the side-by-side comparison of the two datasets and drawing conclusions or making interpretations. I feel this is easiest to accomplish by inserting another column, a metainferences column, into a joint display. The researcher looks across the rows and down the columns. What does this analysis tell about the integration or linking of the two databases? It might find that individuals in the high scoring category differed in their views of themes or that on one theme, the low, medium, and high scoring individuals held similar or different perspectives. The joint display table provides an opportunity to present the quantitative and qualitative data together and draw conclusions from the integration. These conclusions or insights from integration can be compared with findings from the literature, related to existing theories, or juxtaposed with personal experiences. The insights could also be taken back to a few participants to check the accuracy of the researcher’s conclusions—a form of mixed methods member checking.
Worldviews and Theory
Surrounding the research components are worldview and theoretical considerations. These might be introduced into a mixed methods study before, during, or added on towards the study’s end. Worldviews are beliefs or values that the researcher brings to a study that informs many aspects of the research (Creswell & Creswell, 2017). Theories, on the other hand, typically emerge from the scholarly literature to provide explanations or predictions about the results. Worldviews and theories link into the mixed methods designs.
In mixed methods research, many worldviews have been advanced. A recent summary by Shannon-Baker (2016) conveys a current state thinking. She suggests that mixed methods researchers draw on four worldviews: pragmatism, critical realism, transformative-emancipatory, and dialectic pluralism perspectives. Going into details about these worldviews is beyond the scope of this paper. However, these beliefs about conducting research result from a researcher’s community of scholars and the cultures that shape their lives. Theories from the literature, such as cognitive theories, behavioral theories, life-span theories, and more, represent an explanation (or prediction) for a study that relates to the quantitative phase, the qualitative phase, or the entire mixed methods project.
Applying the Model to a Hypothetical Language Learning Mixed Methods Study
Returning to Figure 1, we can see that phases in mixed methods research build on each other: the problem, the data inventory, the integration intent and procedures, the design, the joint display, the metainferences, and the worldview and theory. We can now apply these phases to a practical project.
I am a language learner myself. Over the last three years, I have participated in Japanese language instruction at our local community center in Ashiya. I have experienced three different Japanese language teachers in each year based on the program’s emphasis on rotating instructors. I have also noticed that my teachers differ considerably in their use of the English language. I could construct a mixed method study titled Do English-Speaking Students Learn Japanese Better when their Japanese Teachers Use Japanese or English in a Community Center Program? This topic lends itself to mixed methods research. I would want to know what students think about this topic (trends from a quantitative survey) and their learnings in Japanese over a year (qualitative views about writing, speaking, and conversation). I think it would be helpful to know the individual experiences of a few students and talk with them about their perspectives. I could draw a diagram of the procedures to share and encourage them to participate in the study. This diagram would show that I am starting with a survey and following up with individual interviews—an explanatory sequential design. I can analyze my data to see if the results change from the survey to the interviews and link this into existing theories. I might find my interpretation from personal experiences is reinforced: When my Japanese teachers have some knowledge of English, they help to promote my learning (e.g., motivation to study, conversational abilities, and so forth). With this study, I have (1) formed a problem best addressed through quantitative and qualitative data; (2) decided to integrate the two databases by one following the other; (3) used an acceptable mixed method design and drawn a diagram of it; (4) analyzed the data with a joint display; and (5) formed metainferences from the integration, bringing in my own personal experiences (or worldview).
As my hypothetical example illustrates, mixed methods research can be applied to language learning and teaching. It will present your research as state-of-the-art methodology, give insight into your problem beyond the quantitative and qualitative results, and allow you to be creative with your research.
References
Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6(1), 97–113. https://doi.org/10.1177/1468794106058877
Creswell, J. W. (2022). A concise introduction to mixed methods research (2nd ed.). SAGE.
Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). SAGE.
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE.
Creswell, J. W., & Plano Clark, V. L. (in press). Revisiting mixed methods research designs twenty years later. In C. N. Poth (Ed.), SAGE Handbook of Mixed Methods Design. SAGE.
Creswell, J. W., Plano Clark, V. L., & Guttmann, M. (2003). Advanced mixed method research designs. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed methods research in social and behavioral research (pp. 209–240). SAGE.
Curran, G. M. (2020). Implementation science made too simple: A teaching tool. Implementation Science, 1(27), 1–3. https://doi.org/10.1186/s43058-020-00001-z
Ebadi, S., & Rahimi, M. (2018). An exploration into the impact of WebQuest-based classroom on EFL learners’ critical thinking and academic writing skills: A mixed-method study. Computer Assisted Language Learning, 31(5-6), 617–651. https://doi.org/10.1080/09588221.2018.1449757
Fetters, M. D. (2019). The mixed methods research workbook: Activities for designing, implementing, and publishing projects. SAGE.
Lynam, T., Damayanti, R., Titaley, C. R., Suharno, N., Bradley, M., & Krentel, A. (2019). Reframing integration for mixed methods research. Journal of Mixed Method Research, 14(3), 336–357. http://dx.doi.org/10.1177/1558689819879352
Moeller, A., Creswell, J., & Saville, N. (2016). Second language assessment and mixed methods. Cambridge University Press.
Moseholm, E., & Fetters, M. D. (2017). Conceptual models to guide integration during analysis in convergent mixed methods studies. Methodological Innovations, 10(2), 1–11. https://doi.org/10.1177/2059799117703118
Nastasi, B. K., & Hitchcock, J. (2015). Mixed methods research and culture-specific interventions: Program design and evaluation. SAGE.
Park, F., Youngjoo, Y., & Jang, J. (2021). Mixed methods research in second language writing: A systematic research synthesis. Journal of Language and Linguistic Studies, 17(Special issue 1), 570–579. http://dx.doi.org/10.17263/jlls.903512
Plano Clark, V. L., & Ivankova, N. V. (2016). Mixed methods research: A guide to the field. SAGE.
Riazi, A. M. (2017). Mixed methods research in language teaching and learning. Equinox Publishing.
Shannon-Baker, P. (2016). Making paradigms meaningful in mixed methods research. Journal of Mixed Methods Research, 10(4), 319–334. http://dx.doi.org/10.1177/1558689815575861
Wesely, P. M. (2010). Language learning motivation in early adolescents: Using mixed methods research to explore contradiction. Journal of Mixed Methods Research, 4(4), 295–312. http://dx.doi.org/10.1177/1558689810375816
