Twenty Years of Research on Mixed Methods

This research is a qualitative study based on systematic analysis of the articles on mixed methods via the bibliometric analysis using the R programming language. Thus , this study analyzed articles using mixed methods in journals in the Web of Science database. 1,623 articles, which was published in 1999 -2018 and included ‘mixed methods research’ in the article title, abstract or keywords, were analyzed as a whole. This analysis was the bibliometric analysis using the R programming language. At the same time, content analysis was used to show the relationships between the subdomains of the studies using mixed methods and the development of time. The study focused more on resource impacts, the most cited countries, keyword plus cloud, co -occurrence network, co-citation, author collaboration. The study results in a discussion about the use of mixed methods on the part of future studies.


Introduction
When the bibliometric analysis used in the study is examined, scientific researches should be in communication in order to be effective in scientific sense. In this century, scientists have been in contact with books and scientific journals in order to be in touch. In this sense, scientists should use citation and references to ensure communication, to determine the importance of their research, or to demonstrate the authenticity of their contributions (Merton, 1957). Citation and references have many functions such as getting the approval of the premises, following the origin of a new idea and distinguishing new findings from the current findings. The bibliometric data developed by quantitative analysis benefit from references and citations of scientific studies. (De Bellis, 2014). Nowadays, researchers use different qualitative and quantitative literature approaches to understand and interpret the results of the published studies using their previous findings. Among these, bibliometric analysis has the pote ntial to make scientific activities more simple, understandable, and reproducible with statistical techniques. Unlike the techniques used in the literature, bibliometric analysis provides the reader with more reliable and non-subjective analysis (Aria & Cuccurullo, 2017). According to Crane (1972) by applying bibliometric analysis to the magnitude of new knowledge, notional progress and data, it is possible to present trends of time, to explore themes, to identify changes in disciplines, to identify productive researchers and organizations, and to give a large picture of the current study. In this context, the bibliometric analysis used in the study is carried out as follows;  search for new research paths,  identify recommendations for further research,  distinguish what needs to be done,  discover important variables related to the subject,  identify the relations between researchers, journals and countries,  develop and acquire the subject dictionary,  understand the structure of the subject,  relate ideas and theory to practices In this study, bibliometric analysis was applied to the mixed methods research (MMR) field by using R programming language. The research subject of the mixed methods has been in high demand for academic research in the last 30 years. This interest leads to an ever-increasing scientific study of mixed methods and requires a comprehensive perspective on the subject. Mixed methods research is a research design with philosophical assumptions and a research method. As a design, the mixed methods involves many stages of the research process and includes philosophical assumptions that guide the management of data collection and analysis operations with a mixture of quantitative approaches. Mixed methods focuses on the analysis, blending of both qualitative and quantitative data collection in a research or a series of studies. The main antecedent is the combined use of qualitative -quantitative data to ensure that the research problem is better understood than any other method used alone. (Creswell & Clark, 2007). Historically, scientists have been involved in the use or approach of quantitative or qualitative methods that have caused many researchers to isolate themselves against methodologies (Molina-Azorin & Cameron, 2010). For example, quantitative research methods emerge from a positivist / postpositivist paradigm and qualitative research methods belong to an interpretive or constructivist paradigm. Thus, each is thought to have a separate epistemology, ontology and axiology (Wiggins, 2011). Mixed methods have a few reasons and justifications. First, all research methods have their own limitations. Most mixed methods try to use both quantitative and qualitative methods to balance each other's strengths and weaknesses or confuse methods to answe r a research question or questions in all ways (Tashakkori & Creswell, 2007). Research problems that are appropriate for mixed methods are problems that a data source may be inadequate. In addition, the results should generally be explained, the discovery findings need to be generalized, the expansion or improvement of a primary experimental design, the comparison or enhancement of more than one case, the participants need to participate in the research, or the needs of a program to be evaluated. Over the y ears, the authors of mixed methods listed several reasons for using mixed methods (Bryman, 2006).
The aim of this study is to provide a broad overview of the main features of MMR publications based on bibliometric analysis by defining the existing data by statistical and algorithmic methods. The information presented in this study will guide the progress of future studies in the field of MMR. On the other hand, it can help researchers to shape their future work on authors, journals, countries, institutions, references and issues working in the field of MMR. In this context, the aim of the study is to present a 20-year historical development of MMR with bibliometric analysis and to present a road map to researchers who will work in this field. For this purpose, the following questions were sought; 1. What is the conceptual map of the MMR? 2. What is the productivity of scientific researchers, countries and academic journals in MMR? 3. What are the collaborations between the authors, journals and countries in the MMR?

Method
This research is a qualitative study based on systematic analysis of the articles on mixed methods via the bibliometric analysis using the R programming language. A systematic analysis is a type of literature review that collects and critically analyzes multiple research studies or papers on a topic or question (Davies, 2004). Purposive sampling method and criterion sampling technique, which is common in qualitative research methods, (Palys, 2008) were used and key words were the criteria of sampling. In the research, because of the discovery nature, "mixed methods research" words were searched by the article title, abstract and keywords via Web of Science (WoS), which was the referee-controlled literature database. Since Web of Science is the most commonly used database for the follow -up and analysis of scientific publications, this database was preferred in the study (Yang et al., 2013). Quotes are used to improve the quality of the results during the search. (Liu, Zhan, Hong, Niu, & Liu, 2012).
From 1999 to 2018, research articles published in journals in all languages were discussed. In total, 1,623 publications related to mixed methods research were identified. Only articles of publication were included in the search. So, WoS was preferred in this study because it contains many details such as publication year, author names, countries and schools of authors, titles, abstracts, sources and journals required for bibliometric study.
Data were collected in a single bibliography database and recorded in BibTeX form in order to ensure that the bibliographic information is displayed in the desired place on the basis of this database in the references within the study. These approaches are described in Cobo et al. (2011) and Börner et al. (2005) follows the general bibliometric methods described in the studies.
The articles accessed were examined through concepts used in keyword plus, author's keyword and document titles. For the analysis of keywords, network analysis and co-citation, collaboration, co-occurrence analysis were used. On the other hand, in order to determine the main development points for the development of the field, author and journal impact was discussed and evaluated in the area of mixed methods.
The R programming language was used in conducting the analyzes and the ggplot2 libraryhttps://cran.r-project.org/ was used in the creation of the visuals.

Findings
The WoS data set used in the study included 1,623 different articles with a total of 4,866 authors. When the data set was examined in a broad sense, 4,866 authors use 4,836 different keywords to group or categorize their articles. In addition, the number of citations per article is 11.42. This indicated that the work is strongly citation and the weight of the work in many these areas. When the distribution of the authors in the data was examined, there were 315 single authors. Other 1,308 articles were written by 4,551 different people. There were 3.29 (sd = 3.00) authors per article in the field of MMR, which meant that the work in this field was generally the product of collaborative work. The quality of the studies carried out in a field was in line with the number of peer-reviewed publications in that field. As shown in Figure 1, the number of studies in the field of MMR has increased since 1999, especially since 2007 (annual growth rate: 38.06%). In 1999, there was only one publication in MMR. However, the number of studies remained limited until 2006 and since 2007, an increasing number of publications have been observed. However, from 2007 to 2018, there was a continuous decrease in the mean total number of citations per article and the mean total number of citations per average year. In 2018, the mean number of citations per year increased again.

Journal analysis
Many researchers observed that scientists who knew the impact factor were driving the publishing strategies. Similarly, according to Archambault & Larivière (2009), journal editors aimed to reinforce impact factors, sometimes using strategies that differed significantly from the widely held beliefs about the basic ethics of science. In this way, Table 1 provides a comprehensive overview of the MMR. As known, MMR studies bring together different disciplines or subjects. In this context, it was seen that 1,623 articles which were the subject of the research were published in 962 different journals. Table 1, which includes information such as h -index, average number of citations and publication year, provides information on the 20 most active journals in the field of MMR. These were 20 journals published nearly one fifth of all MMR publications (19%; n = 305/1623). Key journal in the field was 'Journal of Mixed Methods Research' with 108 publications on the topic, respectively. In Figure 2, Table 1 shows the 20year development of the journals in the first seven. According to this, JMMR published 10 articles in its first year of publication and published the most articles in 2012 and then there was a decrease. In this period, it can be interpreted that it attached importance to different subjects than MMR. After 2016, JMMR displayed an upward trend. The most important detail observed in the figure was that KZSS publishes 12 articles in 2017 and it did not broadcast in this field before or after this. Examining citation per article, most effective journals are as follows: JMMR, QQ, QI, IJSRM, BPH. The 1,623 articles were written by a total of 4,866 different authors. As a result of the analysis of these authors and articles, the following results emerged: 315 authors of single -authored documents, 4,551 authors of multi-authored documents, 345 single-authored documents, 3 authors per document, 3.29 co-authors per documents. Table 2 shows the h-index, the total number of citations and the number of studies of the top 20 authors with the highest contribution to the MMR field. The ranking was sorted by the author's total number of publications and, if the number of publications was equal, by last name. Creswell and Onwuegbuzie were the most prolific writers in 11 publications on MMR, followed by Clark and Fetters in 9 publications. Looking at the top 20 authors, it showed a broad citation and naturally interest in the field. Only in the analysis of the MMR area, Creswell's h-index is highest and Onwuegbuzie, Clark and Molina-Azorin were followed respectively. Onwuegbuzie (TC = 1,470) was the most cited in total, followed by Johnson (TC = 1,062), Creswell (TC = 869), Clark (TC = 523) and Morgan (TC = 519). Onwuegbuzie, Johnson, Morgan and Bryman were located respectively in the number of citations per publication. Participation of researchers from different countries in a study indicated that the study was in strong cooperation. In the field of MMR, three out of four studies were multinational. On the other hand, it showed that an area with many co-authors worked collaboratively and would provide opportunities for future studies. The cooperation pattern (i.e. co-authorship) of the authors publishing on MMR was analyzed. Evaluation of single country publications (

Cited and co-citation analysis
Cited analysis gave the number of citations of published studies in the MMR area listed in WoS. Of the 1,623 articles reviewed in the study, 18,535 times were used as references in other publications, and with 11.82 citations per article. It should be noted that the number of citations was a general assumption that a publication reflected the impact and reputation and hence the quality of a publicatio n (Ugolini et al., 2015). On the other hand, the very citation of a study stated that it measured its visibility rather than its quality. Therefore, this definition can be explained more clearly due to the increasing citation of open access journal publications (Chiu & Ho, 2007). Table 3 shows the 20 most frequently cited documents. The most cited paper was 'Toward a definition of mixed methods research' from Johnson, Onwuegbuzie & Turner. The paper has been cited 1,015 global and 184 local times since its publication in 2007 (until May 21, 2019). Looking at the top 20 most cited studies, JMMR was the most published journal with seven publications, followed by QQ and IJNS with two publications. When the first four publications were examined, it was observed that it was published in JJMR. It was noteworthy that the other journals in this list were published in the field of health in general. The citation of a publication is highly related to the time elapsed since its publication. Obviously, older publications have a greater chance of receiving citations from new publications, but this does not indicate that recent studies are less important. For example, the 2017 study on the list received more citations from older publications.

Issue 1, 2020
Journal of Mixed Methods Studies / JOMES Issue 1, 2020 Journal of Mixed Methods Studies / JOMES Cited references were the articles, books or other materials listed in a reference or as works cited in a publication. When Table 4 examined, 10 books, seven articles and three book chapters were the 20 most cited references. On the other hand, it can be said that Creswell, Clark, Johnson, Onwuegbuzie, Tashakkori and Teddlie had a strong influence on MMR. The most cited reference was "Designing and Conducting Mixed Methods Research" from Creswell and Clark. The second-best cited reference was the publication of the same book that was revised by the same authors in different year. Both references received about 200 citations from 1,623 articles. When the first 20 cited references were examined, it was observed that they were all method -based publications. Co-citation analysis is a widely used method for empirical analysis of scientific research structures. The study shows how strongly the results were affected by selection criteria for citation relationships (Gmür, 2003). Co-citation analysis reveals the relationship between the other publications and the interaction between the two publications in the field. The similarities mentioned in both publications can be supposed (Li & Hale, 2015). 66,338 references were used in the 1,623 publications on MMR. In the study, the Walktrap clustering algorithm was used to analyze and visualize the accompanying excerpts. A reference from a maximum of 40 citations was used in the bibliography of 1,623 MMR publications to be included in the common quote map. The result of the common quote analysis is given in Figure 4 in two different clusters. The ellipse shape observed in different sizes indicated the number of citations, in other words, the larger an ellipse, the more co-citations were specified in the MMR field. There was a betweenness higher centrality (btwc), a stronger relationship, and a higher similarity between the two publications. The ellipse of the same color pointed to a similar issue among these publications.  Table 4).

Country and institution analysis
MMR publications originated from 90 different countries. The cooperation of countries with each other was visualized in Figure 5 in five different clusters. According to this, the USA had a strong cooperation with its own cluster and other cluster members. In the UK, Australia, Canada, which were in th e same cluster as the USA, they were making strong collaborations respectively. Germany, in its blue cluster, had strong cooperation with the USA, the UK, Australia, Canada and Singapore, except for its own cluster. South Africa in the orange cluster was in strong cooperation with the USA, UK, Australia and Canada, except for its own cluster. Green cluster had its own cluster of China and strong cooperation with the USA, UK and Australia. The most frequent collaborations were as follows; USA -UK (f= 26), UK -Australia (f= 22), USA -Australia (f= 20), USA -Canada (f= 14), UK -Canada (f= 13), UK -Ireland (f= 11), USA -China (f=10). Figure 5 shows the top 40 of the most producer countries in the field of MMR. All of the 20 largest industrialized countries in the world were in the top 14 of most productive countries published in MMR. This showed that economic development contributed to scientific and academic investments. As in different scientific fields, the most productive countries were actually more s uccessful in making multinational publications (Zheng et al., 2016). In 1,623 publications, 1,483 different research organizations with institutional information participated. 69 organizations (4.7%) performed at least ten publications in area of MMR. Table 5 shows details on the top 10 of most producer organizations broadcasting in MMR. The third most productive institution was in the United States. The most published institution (n = 42) was the University of Michigan. The top 10 institutions were universities. As observed in the collaboration of countries, it was observed that universities in the USA, Australia, Canada and UK influenced MMR area.  The University of British Columbia Canada 21 10 McGill University Canada 20 10 The University of Melbourne Australia 20 10 University of Pennsylvania USA 20

Keyword analysis
Statistical analysis of keywords plus can be used to identify new trends in science, also show how effective the field is in understanding and advancing the boundaries. On the other hand, keywords plus show interdisciplinary power among all articles co-cited (Garfield & Sher, 1993). The bibliometric analysis technique is used in order to understand the tendency of the field by analyzing the keywords plus in the studies published at different times (Wang, Liu, Jia, & Zhang, 2015). In this sense, keywords plus was researched in 1,623 articles. In keywords plus, 3,229 most commonly used words were found and 50 of them are visualized by the tree map in Figure 6. Health, education, care, impact and qualitative research words were most frequently used after MMR which was the most frequently used according to the table. When word tree map was examined, it can be said that MMP studies were done in almost all areas, especially in the fields of health, education, psychology and management. Each unique word was visualized by co-occurring matches, weighted by co-occurrence, represented by keywords plus a network edge, in nodes within associated network graphics (see Figure 7)."The size of the rectangular boxes represented the occurrence of a word, i.e. the higher the size, the more it appeared in the abstracts of a word, the author's keywords, and the title of the MMR publications. The general distance indicator between terms provided general information about their relationship to each other. Btwc numerical size indicated that the relationship was strong. The association of words was determined by counting the number of occurrences of words in titles, author's keywords and abstracts. The colors are different because they are used to distinguish clusters. The co-occurrence map showed how the words of MMR studies were aggregated, and clearly three clusters: red cluster, blue cluster, and green cluster. The most common keywords collected in the red cluster were: health (btwc= 43.59), care (btwc= 26.39), children (btwc= 24.51), behavior (btwc= 13.92) and experiences (btwc= 10.94). The Issue 1, 2020 Journal of Mixed Methods Studies / JOMES 40 most common keywords in the blue cluster were: impact (btwc= 31.15), MMR (btwc= 26.39), qualitative research (btwc= 18.00), management (btwc= 17.79), perceptions (btwc= 11.94) and science (btwc= 11.68). The most common keywords in the green cluster were: education (btwc= 15.92), student (btwc= 8.74), teachers (btwc= 0.91). The green cluster seemed to entail publications on education. The blue cluster seemed to entail publications on methods and management. Figure 8 visualized how the main element of the three fields in the form of authors, keywords plus and journals was related to each other by the Sankey diagram, which was a specific type of flowchart, where the width of the arrows was shown proportional to the amount of flow. The authors whose names are included in the figure should be considered as the first author in the published studies. According to this, Creswell, J.W. was working in the field of health, care, perception, validation, qualitative research and most of the studies in these areas were published in JMMR. Molina-Azorin, J.F. published studies on performance and impact issues in JMMR, Johnson, R.B. worked on mixed methods, knowledge, validation and health issues and these issues were mainly published in JMMR. In QQ, there was a tendency towards performance, qualitative and mixed research, attitudes, health, management and children. Seven of the eleven studies by Fetters, M.D. were directed at different words. Onwuegbuzie, Creswell and Clark's six of seven works, and Guetterman's six works were directed to different words.

Discussion and Conclusion
In this study, the tendency of the English studies in the field of MMR in WoS between 1999 and 2018 was evaluated. Studies in the field of MMR continued to increase with an annual growth rate of 38.06%. The study included 1,623 publications on MMR covering 4,866 authors, 962 journals, 90 countries, and 1,483 institutions.
The results of the analyzes in the study can be summarized as follows. On the 1,623 articles reviewed in this study, 1,299 (78.8%) authors were included in only one publication, and a small group of producer authors contributed to a significant portion of the publications in MMR. While there were 912 journals in this field, 851 (88.5%) published only one or two publications. The first 20 journals in Table  1 contributed 305 (18.8%) articles to the MMR field. Of 90 countries publishing on MMR, 58.9% produced ten or less publications. 12.2% of the contributing countries published more than 50 publications on the subject. While 61.1% of the institutions were in only one publication, 4.7% of the contributing institutions published at least ten publications on MMR. Universities were the dominant contributors in institutions. Most of the MMR publications (23.7%) have not yet been mentioned by others, and only a small number (3.8%) of the publications were cited 50 times or more. Creswell, Tashakkori and Johnson appeared to be the most influential authors. These three authors were bonded to the United States and published in the field of MMR area. The most cited paper was "Toward a definition of mixed methods research" from Johnson, Onwuegbuzie & Turner (2007), and the paper with the highest average citations per year. The journal JMMR was the key journal publishing on MMR. The USA, UK, Australia, Canada, South Africa can be considered strong countries in the field of MMR. When looking at the cooperation network, other countries were necessarily connected to one of these countries. The most cited countries in these five countries, except for South Africa, were the same, and Netherlands was added to this list. The University of Michigan (USA) was at the forefront of 1,483 institutions. When it came to the most productive institutions, the top 10 were mostly in the USA. In keyword analysis, studies on health, methods, management and education were found to be weighted.

Issue 1, 2020
Journal of Mixed Methods Studies / JOMES