An Examination of Turkish Students' PISA 2015 Collaborative Problem-Solving Competencies

Technological advancements initially made it possible for individuals with different cultures from different parts of the world to consider working together, and eventually collaboration turned into a necessity. Based on this necessity, OECD measured collaborative problem-solving (CPS) competencies of 15-year-old students from the countries participating in PISA assessment in 2015. The objective of this study is to examine the data on Turkey from PISA 2015 CPS survey through Deterministic-Input, Noisy-Or-Gate (DINO) cognitive diagnostic model and to determine the students’ levels of CPS competencies as well as difficulty levels of these competencies. In this context, primarily, attributes of CPS competency domain were examined and the model with the most appropriate number of attributes, classification accuracy and consistency was determined for the analysis of PISA 2015 CPS data. The sample of this descriptive study consists of 435 students. As a result of the analysis, the Q-matrix of the model data fit was observed to have the best goodness-of-fit when it had three attributes whereas model fit and classification consistency decreased as the number of attributes increased. Furthermore, the easiest competency for Turkish students was determined to be ‘taking appropriate action to solve the problem’ while the most difficult was ‘establishing and maintaining shared understanding’. Finally, latent class distribution exposed that 56% of the students did not have any of the competencies, and 35% had all. Results of this examination indicate that all CPS competencies of Turkish students need improvement.


INTRODUCTION
All individuals face several problems throughout their daily lives. It is of great importance that individuals have problem-solving competencies in overcoming these problems. To this end, it would be befitting to determine the strengths of individuals in problem-solving and to examine these aspects since they might provide an insight into the abilities of individuals to use basic thought processes and to face challenges in life (Lesh & Zawojewski, 2007:769). Training individuals to have problem-solving competency is among the objectives of the education provided both by families and in schools as part of the government policies (Ministry of Education [MEB], 2009:6-16). (Organisation for Economic Co-operation and Development [OECD], 2013:13;Soylu & Soylu, 2006:97). Problem has a variety of definitions although none seems to have been agreed-upon. For instance, Turkish Language Association CONTACT: Emive YAVUZ  yavuzemine0@gmail.com  Erciyes University, Department of Educational Science, Kayseri, Turkey ISSN-e: 2148-7456 /© IJATE 2020 (TDK, 2019) defines it as "a question, an issue or a matter that needs to be solved by theories and rules". It was defined as situations that require mathematical operations by Mayer and Hegarty (1996:31) and as issues requiring qualitative or quantitative solutions by Krulik and Rudnick (1985:686). There are also researchers who define the problem as questions requiring higher-order thinking skills such as critical, creative and analytical thinking (Nancarrow, 2004:22;Schoenfeld, 1992:337). Thus, it may be asserted that a situation can be considered as a problem if it leads to an increase in mental activities of an individual (Baki, 2006:20). It is also possible to make an overall definition of problem as the difference between the condition at hand and the condition to be attained (Kneeland, 1999:5). The reason why the problems are described in different ways, as seen here, may be that they vary in their structure. Based on their structures, problems are generally divided into two as routine and non-routine ones (Dede & Yaman, 2006). In the literature, routine problems are expressed as single-solution problems, four-operation problems or well-structured problems. These problems can be rendered as analogous situations where individuals were able to find resolutions before or similar situations or issues in which they can make use of their previous experiences (gained knowledge) (Polya, 1973:171). Since these problems require the use of formulas, rules or methods that individuals have previously learned (Arslan & Altun, 2007:50), they are significant in terms of reinforcing what has already been learned. Non-routine problems, on the other hand, appear to be called ill-structured or multiple-solution problems in the literature. Because non-routine problems cannot be explained clearly and their solutions vary depending on the criteria (Frederiksen, 1984:367), they might have results varying from person to person. For this reason, the focus in problem-solving shifted from the answer of problem to the solution process (Altun, 2000). In addition, since these problems are not well-structured, it is not enough for people to have the necessary information in the fields related to the problem (Dede & Yaman, 2006:126). People are supposed to use their higher-order thinking skills as well (Altun, 2014:340-355;Hartman, 1998:1;Nancarrow, 2004:73). Problem-solving may be defined as a cognitive process to transform a given condition into a target condition when there is no obvious method of solution (Mayer, 1990 as cited in OECD, 2017a:9). As problem solving has been an ongoing exercise for centuries and will continue to be so in the future, it is not easy to determine its framework and measure it from student performances (OECD, 2013:125). Also, in the literature, problem-solving process is approached in various ways. For example, Dewey (1933:107-116) studied the problemsolving process in five sub-processes: encountering a problem, understanding and identifying the problem, determining solutions, applying solutions, and reviewing results. Polya (1973:5-6) covered the problem-solving process as four sub-processes. These processes are understanding the problem, making plans for the solution, implementing the plan, and checking the result. As can be seen, the way Dewey (1933:107) and Polya (1973:5-6) handle problem solving processes are similar, and it is possible to recount more examples from the literature.
Upon recognition of its significance, problem-solving started to be an integral part of various educational programs to make students acquire this skill (National Council of Teachers of Mathematics [NCTM], 1989;MEB, 2009). Thus, the evaluation of the problem-solving competencies of the students by teachers, national exams and international exams has gathered speed. Programme for International Student Assessment (PISA) is one of the international exams in which students' problem-solving competencies are evaluated.

Evaluation of Problem-solving Skill in PISA
PISA is an international exam aiming to measure and evaluate the literacy of 15-year-old students in three basic domains (mathematics, science and reading) and competencies such as problem-solving, which is of common interest to all domains. PISA is concerned with the literacy capacities that students will need in adult life rather than curriculum and school knowledge (OECD, 1999:9). Therefore, PISA measures students' problem-solving skills in each key domain, as well as their performance of interdisciplinary problem-solving capacities (OECD, 2003:175). First implemented in 2000, this exam is repeated every three years and each time the aforementioned domains have a varying weight of examination. For instance, Reading in 2000, Mathematics in 2003 and Science in 2006 were focused on more closely than other fields, and this detailed examination continued in other surveys in the same order. During the period between the first (2000) and the last (2018) survey of PISA, both the changes in the qualifications required for the business world and the development of measurement methods and technology have brought about some changes in the definition and measurement of this competency and problem types used in the process. For example, in PISA 2003, 2006 surveys, problem-solving is defined as "an individual's capacity to use cognitive processes to confront and resolve real, cross-disciplinary situations where the solution path is not immediately obvious and where the literacy domains or curricular areas that might be applicable are not within a single domain of mathematics, science or reading (OECD, 2003:156;2006:3;2009:11)." In these surveys, routine and nonroutine daily life problems were measured by a pen-and-paper test. In 2012, with the development of technology and measurement methods, problem-solving assessment started to be applied computer-based in PISA 2012 and student interaction with the problem became one of the focal points of evaluation. As software development tools got better and network computers became widespread, students were able to interact with an increasing variety of problems (OECD, 2013:127). Thus, static and interactive problems started to be used in PISA 2012 survey. If the information about a problem solving given to the student prior to the procedure is complete, the problem is called static. However, if the information about the problem given to the student is incomplete, and if the student needs to make trial-error or discovery to reach additional information, it is an interactive problem (OECD, 2013:125). Some examples for the interactive problems can be remote control units, personal devices (such as cellphones), electronic appliances and vending machines, sports education, and animal husbandry. An interactive problem situation may also be dynamic. This means that situation may be altered by influences beyond the student's control (OECD, 2013:125). For example, if no button is pressed for 10 seconds during a transaction, a ticket vending machine might reset. The evaluation of such interactive problems delivered on computers in an international survey took place for the first time in PISA 2012. (OECD, 2013:131). In addition, when interactive problem situations are included in this computer-based problem-solving assessment, it becomes possible to deliver more realistic, real-life scenarios (simulations) than pen-and-paper tests would allow. PISA 2012 defines problem-solving competency as "an individual's capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. It includes the willingness to engage with such situations in order to achieve one's potential as a constructive and reflective citizen." The first sentence of this definition is almost the same as the first part of the definition for problem-solving assessment in PISA 2003. Even though the 2003 definition only has a cognitive dimension (OECD, 2003:156), the second part of the 2012 definition highlights the OECD's definition of "competency" by stressing the cross-disciplinary structure (OECD, 2013:122). The main aspect that distinguishes the problem-solving assessment in 2012 from that of 2003 is not the definition of problem-solving competency, but the use of computers and the inclusion of interactive problems (OECD, 2013:122). Therefore, creative problem-solving competency was measured and evaluated in PISA 2012 survey (OECD, 2013:120). Delivering problems via computer allows data collection and scoring of things such as the length, frequency, type, and order of the actions taken by the student while responding to items, or subsequent performance analysis of students in a way that would not be possible in pen-andpaper assessment (OECD, 2003:182). However, since the software required to measure collaboration skills had not been developed yet (OECD, 2013:120), only individual problemsolving competency was examined in PISA 2012 survey.

Evaluation of Collaborative Problem-solving in PISA
Technological advancements initially made it possible for individuals with different cultures from different parts of the world to consider working together, and eventually collaboration turned into a necessity (OECD, 2011:38). Hence, collaborative problem-solving (CPS) emerges as a vital and requisite competency in workforce and education (OECD, 2017a:3; b:3). In addition, this competency becomes even more significant as the workforce with such competency is a criterion for determining the position of countries in the future world economy (Yalçın, 2018:184). In the literature, 21st century skills especially needed by the business world are identified in different ways by different circles. The generally accepted framework was presented by the Partnership for 21st Century Skills (P21). P21 framework lays out 21st century skills under three titles: life and career skills; information, media and technology skills; and finally learning and innovation skills. Learning and innovation skills are examined under four skills: communication; creativity and innovation; critical thinking and problem solving; and collaboration (P21, 2009). Considering the fact that individuals employ some critical thinking, (Soylu & Soylu, 2006:99;Şenşekerci & Bilgin, 2008:17) a little creativity and innovation skills (Bozkurt & Çakır, 2016:71;Öztürk, 2014:158;Schoenfeld, 1992:337;Soylu & Soylu, 2006:99) as well as communication skills while working in collaboration (Hmelo & Silver, 2004:244), collaborative problem-solving competency may be deemed as an umbrella term encompassing learning and innovation skills from 21st century skills. Therefore, it is significant for both individuals and countries to measure and evaluate this competency at an early age and to determine the aspects where individuals are lacking or advantageous in the components of this competency. In the learning sciences, there was a big shift in the 1990s from "cooperative learning" to "collaborative learning" (Hesse, Care, Buder, Sassenberg & Griffin, 2015:38). This was because Dillenbourg, Baker, Blaye and O'Malley (1996:189-211) determined the difference between these two terms. Knowing the difference between these two terms in order to fully comprehend the measured CPS structure is thought to be essential. According to the work of Dillenbourg et al. (1996:205), a cooperative view is called an activity with the division of labor. In other words, in a cooperative study, while students can coordinate at certain parts of their activities, they tend to work in parallel. Many scientists have stated that cooperative learning cannot fully exploit the potential of a group and that people cannot demonstrate all social skills while working together (Hesse et al., 2015:38). This has resulted in a focus on collaborative learning. During collaborative learning, students organize their activities together to address a task or problem. Students' activities are entangled; their contributions are mutual, and a student's actions can be carried out or completed by others (Hesse et al., 2015:39). To measure this collaboration, the CPS was evaluated for the first time in PISA 2015 survey through virtual tasks that require participants to collaborate with virtual participants (agents/representatives) on the computer (Herborn, Stadler, Mustafić, & Greiff, 2018:2).
Collaboration has evident advantages over individual problem-solving. These advantages are: a productive labor division, multiple perspectives of information, sources and experiences of knowledge, and the quality of solutions and creativity promoted by other group members' ideas (OECD, 2017a:3). Nonetheless, collaboration can also bring about potential challenges to group members. These challenges are unequal or inefficient division of workload among group members, unfair contribution of some members, members prioritizing their goals rather than the group's, conflict among the group members that prevent developing creative solutions, and not being able to coordinate tasks effectively (OECD, 2017a:3). In order to avoid the negative challenges of collaboration, students were made to interact with simulated participants (agents/representatives) on the computer in PISA 2015 (Rosen & Tager, 2013:3). To take advantage of collaboration, there is a message section on one side of the screen where problem tasks are presented, so that students can communicate with virtual participants, and when students come to a wrong conclusion about the problem task, virtual participants provide various suggestions to make students continue their collaboration and problem-solving.
PISA 2015 defines collaborative problem-solving as follows: "Collaborative problem-solving competency is the capacity of an individual to effectively engage in a process whereby two or more agents attempt to solve a problem by sharing the understanding and effort required to come to a solution and pooling their knowledge, skills and efforts to reach that solution." As it is seen, unlike other PISA surveys (2000,2003,2006,2009,2012), this definition emphasizes the active participation of the individual in the collaboration process. When the evaluation frameworks of problem-solving processes in PISA surveys are examined, it is possible to observe how related they are to each other. For example, in PISA 2003, the problem-solving process was divided into six sub-processes: understanding the problem, characterizing the problem, representing/modeling the problem, solving, reflecting the solution, and communicating the problem (OECD, 2003:175). On the other hand, sub-processes in the evaluation of the problem-solving process in PISA 2012 were determined by considering the recent studies on complex and dynamic problem-solving, based on the problem-solving and reasoning studies of Polya's research and cognitive psychologists (1973) (OECD, 2013:126). These sub processes are composed of exploring and understanding, representing and formulating, planning and executing, monitoring and reflecting. Consisting of six sub-processes in PISA 2003 and reduced to four sub-processes by PISA 2012, the problem-solving structure was the source for the PISA 2015 CPS structure and also within the context of this study for the formation of the conditions, namely the number of attributes, to examine the performance of Deterministic-Input, Noisy-Or-Gate (DINO) model. The framework for the evaluation of the collaborative problem-solving process in PISA 2015 survey includes definitions and theoretical structures based on research (such as computerassisted collaborative work, individual problem solving, etc.) and practices in various fields where the skills related to CPS are evaluated. The framework also includes information collected from the PISA 2015 CPS assessment, including Assessment and Teaching of 21st-Century Skills (ATC21s), Programme for the International Assessment of Adult Competencies (PIAAC), Partnership for 21st-Century Skills (P21) and PISA 2012 individual problem-solving assessment (OECD, 2017a:5). In PISA 2015 survey, the CPS assessment framework consists of three core competencies: "Establishing and maintaining shared understanding" signifies group members' sharing their perspectives on problem states and determining a common vision. "Taking appropriate action to solve the problem" means determining types of collaborative problem-solving actions needed to solve the problem and executing them to reach the solution. Finally, "establishing and maintaining team organization" involves understanding one's role as an agent as well as other virtual participants', following the rules while activating one's role, observing group organization, and facilitating the changes needed to deal with communication issues and obstacles to optimize the problem and the performance. Studies related to CPS in Turkey was found to concentrate on teaching cooperative problemsolving strategies in the learning environment (e.g., Gök & Sılay, 2008Yazlık & Erdoğan, 2016) or problem-based teaching (e.g., Özgen & Pesen, 2010). With the progress of technology as well as assessment and evaluation methods since the initial studies, this competency has started to be examined in a way to obtain maximum information about the competency of the participant in computer-assisted practices (see OECD, 2017a:4;b:27). There are quite a few studies in Turkey, which focus on computer-assisted collaborative problemsolving competency (e.g., Arıcı, 2019;Uzunosmanoğlu, 2013). In one of these studies, Uzunosmanoğlu (2013) examined the participants' eye movements and the areas where their eyes were concentrated while solving problems in pairs, and as a result, it was found that the points that individuals in pairs with high CPS levels looked at overlapped more. Findings of this study also revealed that these pairs were more capable of predicting a point where the next action would take place, helping each other and forming a mutual understanding. In another study, Arıcı (2019) examined the variables related to the success of Turkish students in PISA 2015 CPS with mediation models in which direct and indirect relationships could be determined. Arıcı (2019) examined the relationship of variables with this competency by using students' CPS scores, and no research was conducted on the sub-competencies. Similarly, no study was encountered in which a detailed examination of the CPS competencies of Turkish students was undertaken. Considering the role of this competency domain in current and future life conditions, a detailed examination of Turkish students' PISA 2015 CPS competencies appears to be necessary. The objective of this study is to examine the data on Turkey from PISA 2015 CPS assessment through DINO cognitive diagnostic model and to determine the students' levels of CPS competencies as well as the difficulty levels of these competencies. In this context, primarily, attributes of CPS competency domain were examined and the model with the most appropriate number of attributes, classification accuracy and consistency was determined for the analysis of PISA 2015 CPS data. In line with this purpose, following are the research questions of this study: 1. With how many Q-matrix attributes (3, 7, 11) does DINO model have the best model fit? 2. How does the number of attributes given in the Q-matrix affect the classification accuracy and consistency of the DINO model? 3. What are the mastery percentages of Turkish students' PISA 2015 CPS competencies? 4. What are the difficulty levels of competencies that make up PISA 2015 CPS domain?
In order to answer these research questions, the method section provides information about how the research was conducted; which forms of the PISA 2015 CPS survey were used; which model was selected by which criteria for data analysis; and also the validity and reliability. The findings obtained from data analysis are summarized in their relevant sections listed in the same order as the research questions. In the discussion, conclusion and suggestions section of the article, firstly, the findings of the study are discussed and then suggestions for the implementation and research are given in the light of the results of the study.

Research Design
This article was designed as descriptive research, because objective of this study is to reveal the classification performance of DINO model when the Q-matrix has different number of attributes on PISA 2015 CPS data and the students' performance on the sub-competencies that constitute PISA 2015 CPS competency.

Population and Sample
PISA 2015 data includes information on 5895 Turkish students detailed in 66 forms (booklets). The population of the study consists of 5895 Turkish students participating in this survey. Not all the students participating in the survey in Turkey receive the same CPS form. For this reason, some forms had to be determined as research sample. An examination of items that make up the forms exposed that the number of CPS skills that some forms represent are higher than some others. Because Form 93 and Form 96 have more items that represent the structure, they were decided to be studied. Because these forms contain the same items, only the order of these items in the forms is different, so all the data in these forms were combined. Also, in order for content validity not to decrease, polytomously scored items were modified as dichotomously scored items without removing them. In order to encode the items in binary, option distributions were examined. The remaining options in the first 50% of the distribution were coded as "0", and the other options as "1". Following missing data extraction, data on 435 people consisting of 77 items were obtained. The sample of the study comprises these 435 people who answered Form 93 and Form 96 completely in PISA 2015 CPS survey.

Data Collection Tool
Six units were developed and used in PISA 2015 CPS evaluation (OECD, 2017a:18;b:66). A cluster was created by combining each two units. Units that take 30 minutes to answer are made up of many scenarios where students work on a problem. Eventually, data gathered from a total of 117 items that make up six units or three clusters are used to analyze and scale collaborative problem-solving performance. All students participating in the CPS assessment completed one or two clusters of CPS in their assessment.
There are no open-ended items in the CPS assessment. All items urged students to pick one of multiple choices in various ways to respond to group members or drag and drop icons into appropriate slots or click an option within the visual display area. Thus, the ability to collaborate was evaluated through student responses in their interactions with one or more virtual participants. Within the framework of PISA 2015 survey, the structure of the CPS competency domain was represented by a cross matrix of individual problem-solving processes and collaboration processes, resulting in a skills matrix of 12 cells in total. This matrix is given in Table 1. As can be seen in Table 1, the CPS structure has been gathered under three basic competency areas (OECD, 2017a:12;b:50). Essential attributes to provide correct responses to the items on PISA 2015 CPS survey were determined and reported according to this table. The Q-matrix used in DINO analysis of this study was created on the basis of this table and technical reports (see OECD, 2017a;. There are no items measuring A3 cell in PISA 2015 CPS survey. Therefore, the Q-matrix was created from 11 attributes instead of 12, originally expected to be one of the study conditions. This situation constitutes a limitation for the study. In Table 2, the Q matrix with three different number of attributes is given for the first 20 items that constitute the study conditions. In matrices, rows indicate items, and columns indicate attributes. Names of attributes are given in the first row to show how attributes of matrices are determined. When the first rows of matrices are studied, it can be observed that attributes in the Q-matrixes were created with the symbols representing the cells in Table 1.

Data Analysis
In the research, the data were arranged in SPSS 22.0 (IBM Corp., 2013) and analyzed by DINO model, one of the cognitive diagnostic models, using the R software CDM package (Robitzsch, Kiefer, George, & Uenlue, 2019).

Cognitive Diagnostic Models (CDM)
CDMs are restricted latent class models (Templin & Henson, 2006:290). In CDMs, students' responses to the items are associated with the attributes (skills) expected of the students through Q-matrices. Rows and columns of a Q matrix are populated with the information about items and attributes respectively. Table 3 demonstrates a Q-matrix with 3 items and 4 attributes. As can be seen from this matrix, students must master attribute 1 and 3 to provide correct responses for item 1, attribute 2 and 3 for item 2, and attribute 1 for item 3. In this study, Q-matrices of PISA 2015 CPS data were created in accordance with PISA reports and presented under the heading "data collection tool" to make their creation process more comprehensible. Since individual contribution of attributes in the analysis of items in PISA reports were not specified, Q-matrixes were formed with attributes of two categories.
In the mathematical formulations of CDMs whose Q-matrix has two categories, the probability of individuals' being in these two categories is shown with ʋ1 and ʋ2. In general, the probability of individual r answering item i from latent class c correct is calculated as shown below (Rupp, Templin & Henson, 2010:113): In equation (1), Xri indicates the score of individual observed for item i. This equation may also be interpreted as the probability of all participants in class c answering item i correctly or the difficulty of item i for the participants in class c. In CDMs, class membership probability is generally formulated as below (Rupp et al., 2010:114): (2) In equation (2), P(·) denotes probability, Xr and xr are responses received from all participants; xir is response of individual r for item i, ʋc is membership probability of class c, πic is the probability of a correct response for item i by an individual from latent class c; and ∑(·) means sum of all latent classes from c=1 to C a. Π(·) denotes that probabilities of all items from i=1 to I will be multiplied. In the formula, the items represented by I can only take values of 0 and 1, and the formula refers to a set of Bernoulli random variables for the items of I. This provides joint probability or likelihood of a given response pattern for each latent class. Since the multiplication part of the formula is the component that connects the unobservable latent variables with observable data, this portion is called measurement component. The addition part is called the structural component of the model because it is used to describe the relationship between latent attribute variables. Cognitive diagnostic models can be classified in different ways under different conditions. One of these classifications (compensatory and non-compensatory models) is the relationship between the attributes required for the correct response to items. In non-compensatory models, individuals must master all the attributes required by an item in order to produce a correct response for that item. On the other hand, in compensatory models, it is not necessary to have all the necessary attributes for the correct response of the item. In a study where preservice teachers' problem-solving process competencies were examined (Yavuz, 2014:55), it was found that they were able to solve problems even if they did not have all process competencies. For this reason, within the scope of this study, PISA 2015 CPS competency was examined via Deterministic-Input, Noisy-Or-Gate (DINO) model, which is a dichotomously scored compensatory model.

Deterministic-Input, Noisy-Or-Gate (DINO)
DINO, one of the compensatory models, increases the probability of a correct response of the item if only one of the attributes required for the correct response is mastered. In other words, although the students do not have all the necessary skills to answer the item correctly, they can still provide the correct answer. In the model, there are two parameters affecting the probability of a correct response for each given attribute: slipping (s) and guessing (g). The equation for the DINO model is as below (Rupp et al., 2010:132): The first component of the equation (3) is latent response variable, ωic. It refers to the responses provided by individuals in class C for item i. Because this variable uses the disjunctive condensation rule that indicates whether the individual has at least one of the measured attributes, it constitutes the deterministic input part of the model. If an attribute a does not measure item i, then qia=0. If an item i can be measured by attribute a, then qia=1, regardless of whether αca=0 or αca=1. αca=1 means that individuals in latent class c possess at least one attribute. Thus, the presence of one attribute can compensate for the absence of others. In the DINO model, the probability of a correct response for item i by an individual in latent class c is calculated as follows: πic = P(Xic=1| ωic)=(1-si) ωic gi 1-ωic (4) As can be seen in equation (4), the probability of a correct response for an item depends on the parameters of latent response variable (ωic), slipping (s), and guessing (g). Here, P is the probability, πic is the probability of a correct response for item i in latent class c by an individual, Xic is the observed response for item i by students in class c, ωic is the latent response variable for item i in latent class c, (1-si) is the probability of not slipping for item i, and g is the probability of guessing for item i. Equations (5 and 6) for the slipping ( ) and guessing (g) parameters of the DINO model are as follows: The guessing parameter shows the probability of a score 1 when the individual has none of the measured attributes whereas slipping parameter stands for the probability of a score 0 when the individual has at least one of the measured attributes.

Evaluation of model fit
Evaluation of the model fit is the evaluation of the degree of fit between the predicted model and the observed data (DiBello, Roussos & Stout, 2007:988) and is a validity indicator. In CDMs, model fit can be determined through two different perspectives. One of these is to compare indexes of the model with certain criteria (absolute fit), and the other is to compare the model with other models (relative fit). Both model fit approaches were used to answer the research problem. To evaluate model fit within the scope of this study, MADcor and MADQ3 were examined as absolute fit index, AIC and BIC values as relative fit index. In addition, as an indicator of model fit, mean RMSEA values were examined to check whether item-model fit was achieved on item basis. In R software, the CDM package produces absolute fit measurements by comparing the observed frequencies of the data with the estimated frequencies. MADcor examined within the scope of the study is calculated as the mean of the absolute difference between the correlations of observed and estimated pairs of items (DiBello, et al., 2007:989); MADQ3 is calculated as the mean of the absolute values of the Q3 statistics (Yen, 1984:125) indicating the correlations related to item errors. The criterion for evaluating MADQ3 (Ravand, 2016:8) and MADcor (DiBello et al., 2007:989) indexes is .05. Statistics smaller than .05 is evaluated as a good model fit.
AIC and BIC relative fit indexes are log-likelihood values, which is a maximum likelihood estimation. The smaller of these indexes belonging to different models, or the model closer to 0 is considered to fit the data better.
As an indicator of item-model fit, "item mean of RMSEA" is a chi-square value. It is calculated by comparing the observed and estimated values of the correct responses to the item by individuals in different latent classes (Kunina-Habenicht, Rupp & Wilhelm, 2009:67). An index smaller than .05 is considered as good fit, smaller than .1 is considered as medium fit, and greater than .1 is considered as poor fit.
After the evaluation of model fit in the study, the classification accuracy and consistency of the models were examined. Classification accuracy and consistency are the analyzed model's indicators of reliability in assigning students to the correct attribute classes (Sinharay & Johnson, 2019). After detecting the model with good model fit, high classification accuracy and consistency, students' level of CPS competencies and difficulty levels of competencies were determined.

RESULTS / FINDINGS 3.1. With how many Q-matrix attributes (3, 7, 11) does DINO model have the best model fit?
The model fit indexes obtained in cases where the Q-matrix of DINO model has different number of attributes are given in Table 4. It can be seen that MADcor values are .047 under the condition of three and seven attributes and .05 with 11 attributes. Considering the evaluation criterion (.05) of this index, the model can be claimed to fit the data better under the condition of three and seven attributes. MADQ3 value seems to decrease as the number of attributes increases (.050, .049 and .049 respectively) and the model fits the data better. As the number of attributes increases in both AIC and BIC values, it is seen that the model-data fit is getting worse. In other words, the condition under which these indexes take the closest value to 0 is when the Q-matrix is expressed with 3 attributes. Examination of mean RMSEA reveals that item-model fit is achieved under the condition of 3 attributes. As a result, the best model-data fit is achieved when the Q-matrix is expressed with 3 attributes.

How does the number of attributes given in the Q-matrix affect the classification accuracy and consistency of the DINO model?
The values regarding the classification accuracy and consistency of the DINO model on the basis of the attributes given in the Q-matrix are listed in Table 5.  Table 5 reveals that the classification accuracy values of the DINO model are between 0.696 and 0.922. The model has the lowest classification accuracy when the Q-matrix is formed with 7 attributes, and the highest classification accuracy when the Q-matrix has three attributes. It can also be seen in Table 5 that the classification consistency of the DINO model is between 0.608 and 0.905. Classification consistency of the DINO model decreases as the number of attributes given in the Q-matrix increases. In conclusion, the classification accuracy and consistency of the DINO model is found to be the best when the Q-matrix is composed of 3 attributes.

3.3.
What are the mastery percentages of Turkish students' PISA 2015 CPS competencies? As a result of the first and second research questions, it can be observed that the DINO model adapts better to the data and the classification performance is better when the Q-matrix has three attributes. Therefore, the answer to the third and fourth research questions are based on the situation where the Q-matrix is given with three attributes. These three attributes indicate PISA 2015 CPS competencies and are named as "establishing and maintaining shared understanding", "taking appropriate action to solve the problem", and "establishing and maintaining team organization". Students are divided into eight (2 3 ) latent classes in the Qmatrix with three attributes. The mastery percentages of Turkish students' PISA 2015 CPS competencies are given in Table 6. Latent class distribution in Table 6 Table 6 that the number of students in these latent classes is almost nonexistent.

What are the difficulty levels of competencies that make up PISA 2015 CPS domain?
The difficulty levels of competencies that make up PISA 2015 CPS domain are given in Table  7.  Table 7 is examined, it is seen that the competency of "taking appropriate action to solve the problem" is easier for students than the other competencies and its difficulty level is 0.43. The difficulty level for the competency of "establishing and maintaining team organization" is 0.38 and 0.37 for "establishing and maintaining shared understanding". In addition, it can be inferred from Table 7 that the most difficult field of competency for students is "establishing and maintaining shared understanding".

DISCUSSION, CONCLUSION and SUGGESTIONS
Today, individuals need to be competent problem solvers to be constructive and reflective citizens (OECD, 2013:121). Moreover, since it will contribute to the success in their lives, individuals should master problem-solving competency in all educational institutions and levels (Dede & Yaman, 2006:126). With the development of technology, more and more individuals started to work together, and this required them to develop not only individual problem-solving skills but also collaborative problem-solving competencies. Collaborative problem-solving competency may be defined as finding solutions for a problem after working together and exchanging ideas. The objective of this study is to examine the data on Turkey from PISA 2015 CPS survey through DINO cognitive diagnostic model and to determine the students' levels of CPS competencies as well as the difficulty levels of the competencies themselves. In this context, primarily, attributes of CPS competency domain were examined and the model with the most appropriate number of attributes, classification accuracy and consistency was determined for the analysis of PISA 2015 CPS data. Since all students participating in the PISA survey did not receive the same forms, the study was carried out with the data from Form 93 and Form 96, considering the content and construct validity. The results of the study were discussed with this limitation in mind. Considering the CPS structure laid out in the PISA 2015 technical reports, it was decided to form the Q-matrix with 3, 7 and 11 attributes. Upon an analysis of DINO model, the Q-matrix with three attributes was found to provide the best model fit. These three attributes making up PISA 2015 CPS competencies are named as "establishing and maintaining shared understanding", "taking appropriate action to solve the problem", and "establishing and maintaining team organization". It was demonstrated that the classification consistency of the DINO model decreased inversely with the number of attributes in the Q-matrix, similar to the findings of Cui, Gierl, and Chang (2012) and Cui, Gierl, and Guo (2015). This reveals the relationship of classification consistency with the number of attributes. In CDMs, if the total number of items remains the same and the number of items that measure an attribute increases, the increase in the complexity of the Q-matrix can cause a decrease in the classification efficiency of the model (Madison & Bradshaw, 2015:509). Of the Q-matrices created within the scope of the present study, Q-matrices with 3 and 11 attributes have a simple structure, that is, a single attribute is required for each item to be responded correctly. Seven-attribute Q-matrices are complex and each item is measured with more than one attribute. Therefore, the attributes in seven-attribute Q-matrixes were measured more than those attributes in the other Q-matrix, resulting in a more complex Q-matrix structure. In the study, it was determined that the classification accuracy of the model was lower in the seven-attribute Q-matrices than in those with eleven attributes. The reason for this is thought to be the complex Q-matrix structure as mentioned in the studies of Madison and Bradshaw (2015:509).
It was discovered that more than half of Turkish students (56%) did not have any of the three competencies that constitute the CPS structure. 35% of students was found to have all three CPS competencies. There were almost no students who had one or two of these competencies. Furthermore, there were no students who only had the third competency. These findings seem to be in line with the performance of Turkey, which was ranked last among OECD countries (OECD, 2017b:33), in PISA 2015 CPS domain. Özkan and Öner (2019) conducted an experimental study with Turkish students on the geometric thinking development of secondary school students in computer-based learning environments. The researchers, who classified students as successful and unsuccessful based on their geometric thinking development, observed all PISA 2015 CPS competencies in the behaviors of group members whereas they uncovered that members of unsuccessful groups displayed less behaviors within the competencies of "establishing and maintaining shared understanding" and "establishing and maintaining team organization". Unlike this study, Li and Liu (2017) discussed and analyzed the CPS structure in a 12-skill matrix. They found that the Taiwanese students had high competency especially in the skills of "describing roles and team organization (B3)" and "communicating with team members about the actions to be performed (C1)", but they had low competency in the skill of "building a shared representation and negotiating the meaning of the problem (B1)". Moreover, the easiest competency for Turkish students was determined to be "taking appropriate action to solve the problem" while the most difficult one was "establishing and maintaining shared understanding". This finding seems to support the findings of Özkan and Öner (2019). Chang et al. (2017), in their study with Taiwanese students of 11th grade, detected that students displayed similar behaviors in "establishing and maintaining a shared understanding" and "establishing and maintaining a team organization" but more so than in "taking appropriate action to solve the problem". In other words, the competency of "taking appropriate action to solve the problem" was challenging for Taiwanese students. Similar to Chang et al. (2017), Kuo et al. (2019) determined that the most difficult competency for Taiwanese students was "taking appropriate action to solve the problem". Researchers also highlighted that the competency of "establishing and maintaining shared understanding" was easier for Taiwanese students than the other two. Although the CPS competencies might be found easy or difficult depending on the countries where students are from, it can be asserted that this competency needs to be improved, considering the fact that students can increase their learning in the environments where they can easily share and discuss their opinions (Kutluca, 2013(Kutluca, :1517).
In the current study, it was discovered that Turkish students had almost no competency in "establishing and maintaining shared understanding", "taking appropriate action to solve the problem", and "establishing and maintaining team organization". However, studies conducted in recent years offer some optimism for improving CPS skills. In their study with Malaysian students, Nordin and Osman (2018) found that the module they used improved all CPS skills. Lin, Yu, Hsiao, Chang, and Chien (2018) conducted a semi-experimental study with Taiwanese students. In their study, the researchers tried to improve the CPS competencies of three groups of students who received STEM education with teacher-guided web-based applications in the first group, with only web-based applications in the second group and only in-class activities in the third group. In the end, they detected that the CPS competencies of the first and second group members increased in general, and the CPS competency of "establishing and maintaining a shared understanding" was more developed than the other two. In this regard, it is possible to think that the CPS competencies of Turkish students can be improved with various applications. For this reason, there is a need for various projects or experimental research on how to develop CPS competencies.
Finally, although there are many CPS studies focusing on it as a 21st century skill and exploring its significance, measurement and evaluation, there seems to be no applications or courses aiming to teach it within the Ministry of Education (MEB) context. Also, thanks to Fatih project, it is possible to provide computer-assisted education more comfortably today. From this point of view, the collaborative problem-solving competency can be incorporated as a course into the curricula of schools by MEB, whether it is computer-assisted or not. Similarly, OECD (2013:121) states that problem-solving competency can be improved through a process of highquality education.

Declaration of Conflicting Interests and Ethics
The authors declare no conflict of interest. This research study complies with research publishing ethics. The scientific and legal responsibility for manuscripts published in IJATE belongs to the author(s).