Mixed Methods Research Designs in EFL

Parlindungan Pardede

parlpard2010@gmail.com

Universitas Kristen Indonesia

Abstract

Although mixed methods research designs are relatively new, their use in social sciences studies has been quickly growing.  The integration of quantitative and qualitative data collection and analysis in these designs effectively facilitates triangulation which offers a fuller account of the research problem and, thus, enriches a study’s conclusions. This article overviews the current ideas of mixed methods research designs. The discussion focuses on nature, rationales for use, types, and process of mixed methods research designs. To provide more vivid descriptions, the important design features are illustrated using some studies published in the EFL literature. The discussion ends with conclusions and recommendation.

Keywords: mixed methods, quantitative, qualitative, research designs, ELT

Full text in PDF

INTRODUCTION

Research is generally viewed as an organized and systematic process directed to discover new knowledge and/or to use existing knowledge in a new and creative way to generate new concepts, methodologies, and understandings. In the context of English as a Foreign Language learning and teaching, research is conducted primarily for making EFL education more effective. The results of good research provide new insights for making EFL learning and teaching process more effective (Mackey & Gass, 2011). Since EFL education is a multidisciplinary phenomenon (Riazi & Candlin, 2014) and covers various factors, such as students, teachers, textbooks, learning strategies, learning goals, media, teaching approaches, education policies, etc., EFL researchers need to adopt the most appropriate research methods to their research purpose and their stance of paradigm. Adopting an appropriate methodology is ultimately important for a project (Sahragard, 2004) to get sound results.

The research paradigm is determined by epistemological assumptions held by the researchers. Epistemology, or the study of the nature, scope, and production of knowledge, consists of two strikingly different approaches: positivism and constructivism (or anti-positivism, or post-positivism, or interpretivism). Positivists hold that every phenomenon, like the physical world, functions according to general laws which could be understood by way of objective research (Walker, 2014). Thus, social reality, including EFL learning and teaching, should be studied in ways similar to natural science. Positivists believe that, by searching for patterns and relationships among the relevant variables, what has happened in a social reality can be explained and what will happen can be predicted. Common research method adopted under positivism is called quantitative employed in experimental, correlational and survey research. By contrast, interpretivists (also called post-positivists or constructivists) hold that social phenomena are multi-layered and deserve multiple interpretations. They view that, different from the world of nature, human societies have unique characteristics like rules, norms, symbols, meanings, and values (HLS, 2011). Thus, social phenomena are multi-layered and deserve multiple interpretations. As a result, researchers should examine human behavior in-depth and try to explain the reasons that govern such behavior. The research method based on constructivism is called qualitative research which is also called the naturalistic inquiry, post-positive, constructivist or interpretative approaches (Creswell, 1995). The common methods adopted under qualitative research are ethnographic fieldwork, grounded theory, discourse analysis, etc.

To mediate the “paradigm war” between quantitative and qualitative research methods, over the past 20 years, a new research methodology called the mixed-method research was introduced into the social sciences. Some years later, researches in EFL fields also adopted it. This article overviews current ideas on mixed methods research designs and attempts to show how it is applied in various EFL studies. It begins by discussing the nature of mixed methods research design, including its features, and origins. After that, the discussion focuses on the rationales for use, types, and process of mixed methods research designs. Finally, the article ends with conclusions and a recommendation.

DISCUSSION

What is mixed method research?

Mixed methods approach refers to the research mixing or integrating both qualitative and quantitative approach within a single study to produce a fuller account of the research problem. According to Johnson, et al (2007) mixed methods is “… the type of research in which a researcher or team of researchers combine elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purposes of breadth and depth of understanding and corroboration” (p. 123).  This echoes Creswell and Clark’s (2007) view that mixed method is “a procedure for collecting, analyzing, and “mixing” both quantitative and qualitative methods in a single study or a series of studies to understand a research problem” (p. 5). Leech and Onwuegbuzie (2009) accentuate that mixed methods research involves the collection, analysis, and interpretation of both quantitative and qualitative data in a single research or in a series of research that investigates the same underlying phenomenon.

The combination of qualitative and quantitative research approaches in mixed methods makes this design rich because it can add words, pictures, and narrative (qualitative data0 to numbers (quantitative data). Therefore, mixed methods can address both of the ‘what’ (quantitative and qualitative) questions and ‘how’ or ‘why’ (qualitative) questions. This enables the researchers to understand the different interpretations of a certain phenomenon.

Origins of Mixed Methods

Although the mixed methods literature rarely mentions the natural sciences, the combined use of quantitative and qualitative strategies preceded in the natural sciences (Maxwell, 2016). Galileo, for instance, reported the results of a study of various heavenly bodies using a telescope in both observational description and quantitative measurement. Geology has also long combined quantitative and qualitative methods and data. The integration of qualitative and quantitative methods has been used since at least the mid-19th century in medicine and epidemiology.

Mixed methods research developed with the earliest social research projects, particularly the studies of poverty within families conducted in the 1800s in Europe which incorporated quantitative and qualitative techniques, including demographic analysis, participant surveys and observations, and social mapping techniques uses (Hesse-Biber, 2010). The report of DuBois (1899), for example, had deliberately integrated qualitative and quantitative methods and data. However, Campbell and Fiske’s (1959) study of the validation of psychological traits is considered the first work encouraging a single study to use multiple methods and to collect multiple forms of data (Sieber, 1973). Then, since the beginning of the 21st century, mixed methods research began to gain momentum as a feasible alternative research method. 

Possible Integration Points of Qualitative and Quantitative Components

As has been previously discussed, mixed methods research integrates both qualitative and quantitative approach within a single study. Thus, it is the integration or interface that makes the qualitative and quantitative components joined, and integration can take place at any stage(s) of the research process (Glogowska, 2011). However, the integration of both features is not necessarily in equal portion. A mixed methods research can be qualitative dominant or qualitatively driven, in which the researcher counts on a qualitative view of the research process, while quantitative components are added as supporting elements to fortify the study’s conclusion.

Ponce and Pagán-Maldonado (2015) suggest five possible points in the research process to integrate or combine quantitative and qualitative approaches. First, in the planning phase where the research plan is developed, through which the researcher clarifies what to investigate and how the quantitative and qualitative approaches will be used. Second, in the step of combining or integrating research questions from quantitative and qualitative approaches that will guide the researcher into the complexity of the problem studied. Third, in the use of quantitative measurement instruments with qualitative research techniques to generate quantitative and qualitative data for the research problem. Fourth, in the analysis of the combined quantitative and qualitative data. Fifth, in the use of combined quantitative and qualitative data in the presentation of the study findings.

Rationales for Mixed Methods

Before the mid-1980s, researchers were indiscriminately mixing quantitative and qualitative methods and data without providing supporting reasons for doing so (Greene et al, 1989). This caused scholars began to express concern to the rationales for the practice, and various rationales were then identified. Greene et al. (1989), listed five important reasons researchers should consider in using mixed methods. The first reason is methods triangulation, i.e. the use of more than one method while studying the same research question to “examine the same dimension of a research problem” (Jick, 1979, p. 602). Triangulation enables the researcher to look for a convergence of the collected data to enhance the research findings’ validity. Triangulation eventually strengthens and enriches a study’s conclusions, making them more acceptable to advocates of both qualitative and quantitative methods.

The second reason is complementarity, which allows the researcher to gain a deeper and complete understanding of the research problem and/or to clarify a given research result. The combination of quantitative and qualitative data provides a better understanding of the research problem than either type by itself. For example, before creating a questionnaire for collecting numerical data, interviewing some of the respondents can give a wealth of narrative information, and this narrative information can provide insights for making the questionnaire better. 

The third reason concerns with development. In this context, one form of data can help the researcher to develop the instrument needed for the next data collection. For example, the statistical data collected from a quantitative method can help the researcher to shape an interview or panel discussion questions for the qualitative portion of his study. The fourth rationale for using mixed methods is initiation. Sometimes, the results of a study contradict previous studies’ findings so that new studies need to conduct to clarify the contradiction.

Finally, the rationale for doing mixed methods research is expansion, which is projected to “extend the breadth and range of the inquiry” (Greene et al., 1989, p. 259). The integration of quantitative and qualitative data provides richer and more detailed findings, and such findings facilitate future research undertakings and allow researchers to continuously employ different and mixed methods in their search of new or modified research questions.

Types of Mixed Methods Research Designs

Typologies of mixed methods research designs have been developed depicting mostly from approaches used in the evaluation. Creswell (2012), in the context of education research, classified mixed methods research designs into six primary types: three sequential (explanatory, exploratory, and transformative) and three convergent or parallel (triangulation, nested, and transformative). Each varies in regard to its use of an explicit theoretical lens, implementation approach (sequential or concurrent data collection procedures), priority given to the qualitative and quantitative data (equal or unequal), stage at which the data are analyzed and integrated (separated, transformed, or connected), and notations of procedure. The followings are a brief description of each type.

The first, convergent (or parallel or concurrent) mixed methods design intends to collect both quantitative and qualitative data in one phase. The data are analyzed separately and then compared and/or combined. For example, a researcher collects through a survey and an interview at the same time and then analyzes each separately. The results are then compared (see Figure 1). This method is employed to confirm, cross-validate or verify findings.  It is often employed for overcoming weakness in one method using the strengths of another. It can also be beneficial in expanding quantitative data through open-ended qualitative data collection.

The second type, explanatory sequential mixed methods design (also called a two-phase model), aims at providing relevant information necessitated to understand the research problem more efficiently. It begins with quantitative data collection which proceeds with qualitative data collection intended to help explain or elaborate the previously obtained quantitative results (Figure 2). For example, to investigate students’ view of the use of a new textbook, a researcher surveys them to collect quantitative data (phase I). To more insights related to the survey answers, members of the group are selected for interviews (phase II).

The third type, exploratory sequential mixed methods design is also a two-phase design. This design aims at exploring the research problem. Exploratory research refers to the initial research which lays a groundwork for future research. It is used when very little is known about the research problem. In the exploratory sequential mixed methods design, the qualitative data is collected first to explore the experience of participants with the phenomenon under study (e.g., the culture or values of a group or the views of a group having just implemented a new approach) through an interview or observation (phase I). Having analyzed the qualitative data, the researcher understands the research problem better and can design a quantitative study (phase II) to define or measure the findings of the qualitative phase (phase I) in a sample of the population under study. For example, say that the principle of a school wants to change all computers used by more than 100 teachers in the school. As he does not know the teachers’ needs and interests, the principle starts a study by interviewing 10 teachers randomly selected (phase I). Based on the interview results, he designs a questionnaire (quantitative phase II) to be administered to all teachers.

 The fourth type, the embedded design, aims at collecting quantitative and qualitative data simultaneously or sequentially but priority is given to one approach that guides the project, and the other approach is embedded into the project and plays the supplementary role, i.e. it may be  included to help answer an altogether different question or set of questions. Usually, data analysis includes transforming the data, and integration usually happens during the stage of data analysis.

This design is useful for gaining a broader perspective on the research topic or for studying different groups, or levels, within a single study. For example, a researcher experimented to test the effectiveness of (a) singing and (b) “karaokeing” techniques to students’ pronunciation skill with a control group (without singing and “karaoke”), experimental group one (singing), and experimental group 2 (“karaokeing”). The experiment revealed that singing was significantly effective but “karaokeing was more significantly effective. However, it did not reveal how the techniques worked and how the students viewed them. To get the answers, a qualitative research approach was carried out by interviewing the students. In this instance, the experiment is the primary research method and the qualitative approach is the complementary method because it is employed to compensate for the experiment methodological deficiencies.

The fifth type, the transformative mixed methods design, uses one of the four designs (convergent, explanatory, exploratory, or embedded) but is guided by a theoretical perspective or framework or advocacy lens (e.g. critical theory, education for liberation, etc.) which is usually reflected in the purpose statement, research questions, and implications. The objective of this design is to evaluate that perspective (or framework or lens) at different levels of analysis. That is why this design is useful for giving voice to diverse or alternative perspectives. If it is set as a concurrent transformative design, it involves concurrent data collection of both quantitative and qualitative data. If it is a sequential transformative design, it involves sequential data collection (quantitative data first and qualitative data next, or vice versa).  The results from both methods are combined at the end of the study.

The sixth type, the multiphase mixed methods designs, is used when researchers or a team of researchers examine a problem or topic through a series of phases or separate studies (Figure 6). This design requires different research approaches and different groups or samples. To illustrate, the chief of an English department decides that blended learning will be implemented in all classes. Every class will run in 40-60% face-to-face and 40-60% online sessions. To assess the extent of this decision before implementation, the measure should be understood from different perspectives of what it will entail for faculty, students and parents.  To study a problem like this, it is necessary to use a multilevel design or a study that uses various quantitative and qualitative approaches using different population groups or samples, as part of the research design.

The Steps of Conducting Mixed Methods Research Designs

Designing a mixed methods research includes several steps. Many of these steps are similar to those taken in traditional research methods. They include determining the research purpose, formulating the research questions, determining the type of data to collect, and reporting the study. Yet, designing a mixed-methods study includes some other additional steps, i.e. determining the mixed-method study feasibility, data collection procedures identification, and data analysis and integration procedures identification. These steps roughly occur successively, with one informing and affecting the others.

Cresswell (2012) proposed seven steps to conduct mixed methods research (Figure 7). He cautioned that these steps are not lockstep procedures. They should be taken as a general guide to help to start mixed methods research. Step 1, determining if a mixed-method study is feasible, is taken to ascertain that the researcher or group of researchers have the skills to collect both quantitative and qualitative data, time to collect extensive information, and the knowledge of implementing the different types of designs. This step is also taken to consider whether the target audiences (e.g. graduate committees, publishers, other researchers, and practitioners in educational settings will appreciate the complexity of the mixed methods research.

Step 2, identifying a rationale for mixing methods refers to the researcher’s consideration of the defendable reason for collecting both quantitative and qualitative data. Since the rationale provides a good starting point for carrying out the study and be included in the research plan and report, it should be explicitly stated and/or reported.

Step 3, identifying a data collection strategy concerns with (1) the priority the researcher will give to quantitative and qualitative data; (2) the sequence of his data collection, if he does not plan to collect it concurrently; and (3) the specific forms of quantitative data (e.g., students’ tests score) and qualitative data (e.g., pictures) to collect.

Step 4, identifying quantitative, qualitative, and mixed methods questions, could be conducted before a study or they may emerge during the study. It depends on the type of the design. In a concurrent or convergent mixed-method research design, the questions are presented and specified in detail before data collection. But in a two-phase or sequential design, the questions for the second phase will emerge as the study progresses. Thus, they cannot be specifically identified early in the study. However, it is also possible that the questions for a convergent design are presented and specified in detail before data collection.

 Cresswell (2012) suggested that mixed methods research designs usually present both exploratory questions and analytic-variable questions. Thus the quantitative questions should specify the relationship of the independent and dependent variables. The questions can be expressed in the null form, but they are often written to convey an alternative directional form (e.g., the more frequently students read scientific articles, the higher their academic writing skills). Qualitative questions are usually open-ended and non-directional and strive for describing the phenomenon (e.g., Are scientific articles interesting to the students? Why?).

After identifying quantitative and qualitative questions, the researcher needs to pose a mixed-methods question. It is essentially a question that is to be answered by the mixed methods research design being used. Based on Cresswell’s (2012) suggestion, the following are examples of mixed methods question for the six types of mixed methods research design. Using the quantitative and qualitative questions above, for a concurrent design, the question of the mixed methods could be “Do the two databases (the effect of frequent reading to academic writing skills and students’ interest in reading scientific articles) converge and present consistent findings? For a sequential design, the question of the mixed methods could be “Do the two databases diverge and show contradictory findings?” For an explanatory design, the question might run, “How does the data concerning the students’ interest in reading provide a better understanding of the effect of frequent reading to academic writing skills collected in the first phase? In an exploratory design, the question could run, “Is the instrument developed in the second phase (as a result of first phase exploration) a better instrument than those available to measure the variables?” The question for an embedded design can be: “How do the qualitative data support and heighten understanding for the quantitative results?” The question for a transformative design might run: “How can students’ academic writing skills be better addressed using the results of both quantitative and qualitative data?” The question for a multiphase design can be a combination of these questions introduced in various phases or projects in the line of the investigation.

Collecting quantitative and qualitative data, step 5, employs the rigorous procedures for quantitative data collection and the persuasive procedures for collecting qualitative data. The sequence of the data collection depends on the design types. In the exploratory design, qualitative data is collected first, and based on the deeper understanding obtained from its analysis, quantitative data is collected.  The explanatory design starts by collecting quantitative data, and its analysis results inform the follow-up qualitative phase. The convergent design includes qualitative and quantitative data collection and analysis at parallel times, followed by an incorporated analysis.

Step 6, analyzing data separately, concurrently, or both, also relates to the type of mixed methods design. Quantitative and qualitative data can be analyzed separately, as in the explanatory and exploratory designs. Both data type can also be analyzed in an integrated way as in the convergent design.

Step 7, writing the report, is reporting the study in a scholarly report. Mixed methods research studies are generally reported in two alternatives: one phase or two phases, depending on the research design chosen for the project (Creswell, 2012).

Illustration of Mixed Methods Research Designs in EFL Studies

To illustrate the implementation of mixed methods research designs in ELT field, 10 current studies are analyzed by emphasizing on the conceptual issues, including implementation, priority, integration and data analysis, major elements included in the research reports:  purpose statement, research questions, mixed methods design use rationale, and the data collection and analysis procedures.

These studies were identified and selected by searching various online journals related to EFL four times between December 2017 and March 2018, locating various articles reporting EFL studies employing mixed methods research designs written in English. This search procedure resulted in the identification of 25 studies published between 2003 and 2017. Among the 25 studies, 2 used the convergent or concurrent design, 2 others used the nested or embedded design, and the other 21 employed the sequential explanatory design. Concerning this, the writer decided to include only 2 out of the 21 studies employing the sequential explanatory design. Thus, his section analyzes only 6 research reports; 2 reports illustrate each of the 3 designs.

Chen (2008) used a concurrent triangulation design to investigate to study the factors which influence teachers in integrating the Internet into their instruction. Three hundred eleven EFL teachers from Northern Taiwanese higher education participated in the quantitative part of the study and 22 participated in the qualitative part. The quantitative part was conducted by asking the participants to respond to the questionnaire either through an online survey or paper survey. The qualitative part was done by inviting 22 of the responding teachers after the survey was collected.

This study covered the seven steps of conducting mixed methods research proposed by Cresswell (2012). The author identified the rationale for using a mixed-methods design, specified the study’s purpose and research questions, collected data simultaneously (QUAN and QUAL at the same time), treated the data with equal priority, and integrated the data after analyzing them (during the interpretation phase). Specifically, quantitative data, in the form of scores obtained from the responses to multiple-choice questions and qualitative data, in the form of digitally recorded responses to 10 open-ended questions, were collected to examine the factors influencing teachers in integrating the Internet into their instruction. Having analyzed the quantitative and qualitative data separately, the results were triangulated (i.e., compared and synthesized). The triangulation revealed the factors influencing teachers’ use of the Internet identified in both parts were overlapping with three additional factors specifically found in the qualitative data collection part. Another example of studies using concurrent triangulation designs listed in Table 1 is Salman (2017).

Iyitoglu and Erisen (2017) also employed concurrent mixed methods research design. However, they used a concurrent nested (or embedded) design to study flipped classroom model quality and efficiency in increasing university prep students’ overall academic performance in EFL and the durability of that performance. Forty one EFL students at Foreign Language School, Gebze Technical University Turkey participated in this study. They were randomly assigned as experimental (N= 21) and control group (N=20). Quantitative data was gathered through the administration of EFL pre-test and post-test, while qualitative data was collected through follow-up semi-controlled interviews with 9 experiment group students from different achievement groups intended to gather their perception about their flipped classroom experiences.

Overall, this study included the seven steps to conduct mixed methods research proposed by Cresswell (2012). The authors identified the rationale for the mixed methods study they would conduct and reviewed the related literature to construct the achievement tests and the questions used in the semi-controlled interviews. They also stated the study’s purpose and rationale for using a mixed-methods design, reported six research questions (5 QUAN and 1 qual, which focused on different issues), implemented data collection concurrently (QUAN and qual at the same time), prioritized the quantitative data, and integrated the results during the interpretation. Specifically, quantitative data, in the form of scores obtained from the pre-test and post-test, were collected and analyzed, followed by qualitative data, in the form of the participants’ self-reported opinions about their flipped experience. Qualitative data were included to help the researcher get a better understanding of the issue under the studied setting. The quantitative data were analyzed in six folds using Statistical Packages for Social Sciences (SPSS) 21 for Windows and Item and Test Analysis Program –ITEMAN 4, while the qualitative data was manually analyzed using content analysis procedures. O’Bryan & Hegelheimer (2009) is another example of studies using concurrent triangulation designs listed in Table 1.

As stated earlier, the majority (21) of the 25 studies identified through various online journals related to EFL employed the sequential explanatory design. One of them is Pardede (2015) who used a sequential explanatory design to study the perception of pre-service EFL teachers of Edmodo use as a supplementary learning tool. Fifty four students of the English Education Department of Universitas Kristen Indonesia Jakarta who had been attending classes using Edmodo as a complementary learning tool participated in the study. All the 54 students participated in the survey, and nine of them were randomly selected to participate in the interview.

Overall, this study covered the seven steps of conducting mixed methods research proposed by Cresswell (2012). In the study, the author acknowledged the rationale for the mixed methods study he would conduct, reviewed the related literature to get and adapted the closed-ended survey questionnaire and the questions used in the semi-structured open-ended interview, stated the study’s purpose and three research questions, implemented data collection sequentially (QUAN followed by qual), prioritized the quantitative data, and integrated the data after analyzing them during the interpretation phase (as shown in the discussion). Specifically, quantitative data, in the form of scores obtained from the participants’ responses to a 5-point Likert scale survey questionnaire consisting of four dimensions of perceptions about: (1) readiness to use Edmodo, (2) advantages of using Edmodo, (3) disadvantages of using Edmodo, and (4) experiences in using Edmodo in the courses were collected and analyzed, followed by qualitative data, in the form of verbal responses to semi-structured open-ended interview. The quantitative data were analyzed employing the descriptive statistical operation in terms of percentages and means, while qualitative data were analyzed descriptively. The qualitative data were used to get more insights for clarifying and elaborating the quantitative data. Another example of studies using the sequential explanatory design listed in Table 1 is Sehic (2017).

As stated previously, among the 25 identified studies only three mixed-method research designs—concurrent triangulation, concurrent embedded, and sequential explanatory—were employed. No one employed the other three designs, i.e., exploratory sequential, transformative, and multiphase designs. This can be a drawback because mixed methods is a relatively new alternative to purely quantitative or qualitative methods and designs. The availability of studies employing the whole mixed methods designs in the literature can serve as models for EFL researchers and educators to conduct mixed-method researches.

CONCLUSION AND SUGGESTION

With the increased popularity of mixed methods research designs, researchers in EFL fields are challenged to include it in their projects. These designs include quantitative and qualitative data collection, analysis, and integration in a single or multiphase study to facilitate triangulation which will provide a complete account of the research problem and, thus, enrich a study’s conclusions. The combination of quantitative and qualitative data also provides complementarity, development, initiation, and expansion.

On the other hand, the combination of quantitative and qualitative data admittedly makes the process of mixed methods studies more challenging than the traditional researches. Despite the availability of various mixed-methods textbooks which can guide researchers to conduct studies employing these designs, EFL studies with mixed methods designs are still very limited in the literature. Among the limited number of studies, only three designs have been detected in use, i.e. concurrent triangulation, concurrent embedded, and sequential explanatory. Concerning this condition, it is recommended to EFL senior researchers to conduct and publish more and more mixed methods studies to provide models for EFL novice researchers and educators.

REFERENCESS

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitraitmultimethod matrix. Psychological Bulletin, 56, 81-105.

Chen, Y. (2008). A mixed-method study of EFL teachers’ Internet use in language instruction. Teaching and Teacher Education, 24, pp. 1015 – 1028

Creswell, J.W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. (4th Ed.). Boston, MA: Pearson.

Creswell, J. W. (1995). Research design: Qualitative and quantitative approaches. Thousand Oaks, CA: Sage Publications.

Creswell, J. W., & Clark, V. L. P. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage Publications.

DuBois, W. E. B (1899). The Philadelphia Negro: A social study. Philadelphia, PA: University of Pennsylvania Press

Glogowska, M. (2011) Paradigms, pragmatism and possibilities: Mixed-methods research in speech and language therapy. International Journal of Language & Communication Disorders, 46, 251-260.

Greene, J. C., & Caracelli, V. J. (Eds.). (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Evaluation, 74. San Francisco: Jossey-Bass.

Hesse-Biber, S. N. (2010). Mixed methods research: Merging theory and practice. New York, NY: Guilford Press

HLS. (2011). Anti-positivism. Retrieved January 2016 from http://www.historylearningsite.co.uk/anti_positivism.htm.

Iyitoglu, O. and  Erisen, Y. (2017). Delving into Flipping EFL Classroom:  A Mixed Method Study. European Journal of English Language Teaching, 3(1), pp. 120-152. DOI: 10.5281/zenodo.1045310

Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24(4), 602–611.

Johnson, R.B., Onwuegbuzie, A.J., Turner, L.A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1, pp. 112-133. DOI: 10.1177/1558689806298224

Leech, N., & Onwuegbuzie, A. (2009). A typology of mixed methods research designs. Quality & Quantity: International Journal of Methodology, 43(2), 265-275.

Mackey, A., and Gass, S. M. (2011). Second language research: Methodology and design. New York; London: Routledge.

O’Bryan, A., & Hegelheimer, V. (2009). Using a mixed methods approach to explore strategies, metacognitive awareness, and the effects of task design on listening development. Canadian Journal of Applied Linguistics, 12(1), 9-33

Pardede, P. (2015). Pre-Service EFL Teachers’ Perception of Edmodo Use as a Complementary Learning Tool. A paper presented at UKI English Education Department Collegiate Forum held on Friday, June 12, 2015.

Ponce, O. A., & Pagan-Maldonado. (2015). Mixed methods research in education: Capturing the complexity of the profession. International Journal of Educational Excellence, 1(1), 111-135.

Riazi, A. M., & Candlin, C. N. (2014). Mixed-methods research in language teaching and learning: Opportunities, issues and challenges. Language Teaching, 47(2), 135-173

Sahragard, R. (2004). In search of research methodology for applied linguistics. Pan-Pacific Association of Applied Linguistics, 9(1), 258272.

Salman, I. (2017). A Mixed-methods Study of English Language Learners’ Academic Achievements in a Spanish Language Immersion School. A Dissertations submitted to the Graduate School at the University of Missouri–St. Louis. Retrieved January 2018 from https://irl.umsl.edu/dissertation/658

Sehic, S. (2017). The Effect of English Language Learning on Creative Thinking Skills: A Mixed Methods Case Study. English Language Teaching; 10(3), pp. 82-94. DOI: 10.5539/elt.v10n3p82

Sieber, S. D. (1973). The integration of fieldwork and survey methods. American Journal of Sociology, 73, 1335-1359.

This article was presented in UKI English Education Department Collegiate Forum held on Friday, April 6, 2018. Published in PROCEEDING English Education Department Collegiate Forum (EED CF) 2015-2018. UKI Press, Indonesia, Jakarta, pp. 230-243. ISBN 978 623 7256 25 0

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.