Effect of PBL implementation on student and teacher perceptions of improvement in 21st century skills

Background Problem-based learning (PBL) provides an encouraging learning environment to develop generic, research, and creative thinking skills, which have been identified as essential for facing 21 st century challenges. However, how PBL is implemented into a curriculum can be critical for achieving these outcomes. This study explores the effects of shifting the implementation of PBL from a PBL-module integrated into traditional subjects to full interdisciplinary PBL-courses in undergraduate health sciences students. Methods This study involved 651 participants, with 364 taking the PBL-module and 287 taking the PBL-courses. Students’ and tutors’ perceptions about the acquisition of the skill set as well as their satisfaction with the learning experience were collected through surveys and compared in both models. Results Results showed that in interdisciplinary PBL-courses, both students and teachers had the perception that both skills improvement and PBL usefulness were better, and overall were more satisfied with the course. A strong correlation was observed between the acquisition of generic and research skills and the perception of PBL usefulness and overall satisfaction with the experience. The study also showed how creative skills are developed during the different PBL phases. Conclusions This study provides new evidence on PBL characteristics that favour the development of these skills. as facilitator. In PBL-courses, learning outcomes are clearly focused on generic and research skills, while in PBL-module, these skills could be blurred within the other learning outcomes from the traditional subjects. Consequently, assessment in PBL-courses is skill-oriented and uses different tools, such as rubrics and observation grids, to foster metacognition and guide the skills development. The results suggest that students in PBL-courses have better assimilated the intended learning goals, and that assessment tools might have contributed to improving their skills development.


Introduction
Social constructivism defines learning as a constructive process in which students construct or reconstruct their knowledge networks, building personal interpretations of the world based on prior ideas, experiences, and social and collaborative interactions (1)(2)(3)(4). The impact of social and cultural factors and peer interactions on cognitive development has been recognised and suggests that learning takes place through active participation in purposeful and collaborative activities (1,5,6).
Educational research states that activating prior knowledge helps students to relate new information to existing knowledge, leading to richer knowledge structures. Thus, learners should be involved actively during this learning process for activation of prior knowledge (2). In fact, social constructivism advocates that individuals learn naturally when they are engaged in solving problems that concern them. Pedagogical approaches include active learning strategies in which students engage in meaningful activities as part of their learning process, which have been defined as "doing things and thinking about what they are doing", and these approaches are more effective than passive learning in the process of knowledge construction (1,4,7).
Problem-based learning (PBL) has been used as a methodology in accordance with this learning paradigm. PBL empowers learners to conduct research, integrate theory and practice, and apply knowledge and skills to develop a viable solution to a defined problem (8). Researchers define PBL as a focused and experiential learning organised around the investigation, explanation, and resolution of meaningful problems, in which students work in small collaborative groups and learn what they need to know to solve a problem (8,9). As the students understand the problem better, they generate hypotheses about possible solutions. During this learning cycle, learners define knowledge deficiencies related to the problem, known as learning issues that students should research during the self-directed learning part of the process. Afterwards, students have to apply their new knowledge and evaluate their hypothesis on what they have learned in collaboration by integrating a wide range of disciplines (8,9).
The PBL methodology promotes most of the skills that have been identified as essential for addressing 21st century challenges. These are focused in cognitive skills (non-routine problem solving, critical thinking, and creativity); intrapersonal skills (metacognitive skills such as selfmanagement, time management, self-development, self-regulation, and adaptability); interpersonal skills (complex communication, and social skills such as collaboration, teamwork, cultural sensibility, and dealing with diversity) and technical skills (research and information fluency skills, as well as entrepreneurial skills). These have been highlighted as 21st century skills that must be built into curricula, taught, and assessed in higher education (10).
As PBL promotes students to share their knowledge, experiences, and perspectives with other group members, it facilitates the integration of multiple perspectives as part of the problem-solving process.
The PBL learning process addresses the effective development of research and problem-solving skills, refines higher-order thinking skills, creative and critical thinking, leadership and team work skills, selfregulated learning habits, and metacognition processes, and increases motivation and engagement for learning (8,9,11). In fact, there appears to be a close connection between the opportunities provided by PBL for developing communication (teamwork and interpersonal), research (problemsolving and self-directing learning), and cognitive (critical thinking and inquiry) skills (12). Although the PBL methodology fosters self-regulation of learning, this learning process needs guidance to reach its goals and allow students to develop this set of skills and knowledge content, which must be provided by a tutor. In PBL, tutors are expected to facilitate or activate student learning and effective group functioning by encouraging participation of all members, monitoring the quality of learning, and intervening when necessary. The facilitator also plays an active role in scaffolding the students' learning by providing a framework that can be used by students to construct knowledge on their own. Thus, the role of the tutor is crucial for PBL to be effective (4,13).
The way in which PBL is implemented in the curricula and the educational settings can be critical for its success in achieving the intended learning outcomes. A broad variety of PBL instructional designs have been described, such as problem-solving activities inside a subject, PBL modules integrating learning objectives from different subjects, PBL courses, and entire curriculums based on PBL.
According to the degree of self-directedness, learning processes, and problem structuredness, different PBL models have also been used (14). Issues related to human factors, such as behaviours of students and tutors, small group interactions, and resources and workload, might also affect students' learning outcomes. Thus, when implementing PBL, it is important to consider which model will produce the desired effects by taking into account the learners' characteristics and instructional needs (15).
In our university, we started the PBL implementation for Bachelor of Biology students using a model of problems with contents and objectives from different subjects, which we called the PBL-module (Carrió et al., 2016). The results of this experience showed us that students needed more feedback, more guidance in metacognition processes by tutors, a better clarification of evaluation criteria, and tools for self-assessment and for promoting creative thinking (Carrió et al., 2018). As a consequence, a new PBL implementation strategy was introduced into the new Bachelors of Human Biology (which substituted the Bachelor of Biology) and Medicine in 2008-2009, which intended to foster the development of generic and research skills and creative thinking. The main changes implemented in the new model to improve the identified dysfunctions were: 1) interdisciplinary 10-week-long PBL courses in the curriculum, 2) each group of students had the same tutor during all the course, 3) the evaluation process included metacognitive activities, such as self-and peer-assessment through rubrics, or written reflections about their learning process. We termed this strategy the PBL-courses.
We hypothesized that interdisciplinary PBL-courses can directly influence the quality of student learning. To address this, we explored whether changing the characteristics of the PBL implementation improved the desired students learning outcomes. Specifically, we examined the perceptions of the students and tutors about the acquisition of generic, research, and creative thinking skills through PBL-courses, as well as assessing the satisfaction of students and tutors with this pedagogical approach.

Research Context
In 2004, our Faculty started a pilot study to explore the feasibility of creating a hybrid model of teaching that included PBL with traditional lecture-based model, which consisted of introducing PBL activities into the Bachelors of Biology curriculum. In the pilot study, 20% of the teaching time was devoted to PBL tutorials, and the remaining time was used for activities such as lectures, lab courses, and seminars. An integrated module, with interdisciplinary problems including learning outcomes from the different subjects of the term, was designed. All faculty members participated in the problem design and supervised PBL activities as tutors. In this model, each tutor participated in only one PBL case, and a new tutor was assigned for the new case for each group (1,17). In the context of this article, we called this pilot study the PBL-module.  (Table 1). Tutors who participated in these courses were also asked to participate in the survey, and 50% (10 out of 20) accepted. Some data were obtained from a previous study based on the pilot experience (PBL module) in the academic years 2005-2006, 2006-2007, and 2007-2008. In this previous study, 330 students and 34 tutors participated.

Data Collection
This study used descriptive-evaluative research based on the combination of quantitative statistical techniques and qualitative content analysis research methods. Empirical data were collected through an anonymous questionnaire and delivered to the students and tutors who participated in the PBL subjects at the end of the courses. The questionnaire included several closed-ended questions and an open section of general comments and opinions. Participants scored each item included in the closedended questions from 0 (not at all) to 10 (very much). Its dimensions were: Skills development: Research skills: Identification of a relevant research question, hypothesis foundation, data collection, and analysis and discussion of findings.
Generic skills: Oral communication, writing ability, teamwork, critical search of information, time management, and autonomy.
Group dynamics: Participants scored their agreement to the following statements: "During the brainstorming session, the ideas of all group members were considered", "All group members contributed to the research", "All relevant aspects were discussed with all the group and a common synthesis was carried out", and "All group members collaborated in writing up the final report". Finally, field notes were collected during the tutorial sessions, to complement the qualitative results and to analyze the development of creative thinking skills during the different PBL phases.

Data Analysis
Data were analysed both qualitatively and quantitatively. The SPSS software was used for statistical analysis. The Pearson correlation was used to analyse the correlation between the quantitative variables, as all variables have a linear relation. To determine whether there were differences between medical and biological students, the independent samples t-test and Mann-Whitney U-test were used, depending on the variable (18). Finally, to analyse the student and tutor open-ended questions, the qualitative content analysis approach was used as a technique within a constructivist paradigm. Codes and categories that emerged during analysis were refined after multiple coding iterations of the content (19)(20)(21).
For the analysis of the creative thinking skills through PBL phases, the process of codification was guided by the Burnett and Keller-Mathers (2017) framework (22). Creative skills described by this framework were identified through student and tutor comments and field notes from PBL tutorials observations.

Comparison between the PBL-Module and PBL-Courses Generic and Research Skills Dimensions
Scores were given by students to each skill included under Generic Skills dimension (Fig. 1). Overall, the scores of PBL-courses were significantly higher than those of PBL-module approach. The same was true for research skills: students from PBL-courses gave higher scores than those from PBLmodule ( Fig. 2). respectively), which were significantly higher than those that participated in the PBL-module (means of 6.63 and 6.57, respectively) ( Table 2). Notably, the tutors' perceptions of the development of generic and research skills through PBL scored higher than students in both PBL-module and PBL-courses ( Table 2). Significant differences were also observed between the perception of tutors who participated in the newer PBL-courses and those who participated in the PBL-module. Tutors felt that generic and research skills were better developed in PBL-courses (means of 8.13 and 8.15, respectively; non-significant differences were found) than in the PBL-module (means of 7.03 and 6.83, respectively; non-significant differences were found).

Learning Experience Dimension
The student and tutor evaluations of satisfaction and usefulness of the PBL approaches are depicted in Table 2. The students gave high scores to the variables of satisfaction and usefulness in the PBLcourses (means of 7.75 and 8.27, respectively). These values are significantly higher than those obtained with the PBL-module (means of 5.41 and 5.77, respectively). As observed for the skills evaluation (see 3.1.1), tutors scored both variables higher than the students (Table 2). Similar to students, tutors scored the satisfaction and usefulness of the methodology as higher in PBL-courses (means of 9.20 and 9.30, respectively) than in PBL-module (means of 7.52 and 7.73, respectively).

Assessment of Group Dynamics Dimension
Group dynamics was assessed to identify whether the group was working cooperatively. Students and tutors gave high scores for group dynamics (mean of 7.78 and 8.12, respectively), which indicated that the learning process was not hindered by dysfunctional group interactions. All four different items of group dynamics that were assessed were rated high (from 7.57 to 8.43) (Fig. 4). No differences in ratings from HB or M students were found.

Development of Creative Thinking Skills
Overall, 86% of the students felt that PBL contributed to helping develop their creative thinking skills, while 14% felt that they did not (Fig. 5). Tutors were less convinced about this aspect: 61% of them agreed that PBL fostered creative thinking in the students, but 32% felt that the courses did not contribute to this skill. To complement these data, we qualitatively analysed students' comments on the questionnaires, as well as the field notes taken while observing the PBL tutorial sessions, to better understand how creative skills were developed throughout the different PBL phases (Table 4).

Qualitative Analysis of Implementation of PBL-Courses
We also analysed the development of generic and research skills using qualitative data obtained from the student comments on the PBL-courses. In this analysis, we identified two main categories: the learning process and the learning outcomes.
Students highlighted the benefits of social interactions in the process of knowledge construction that takes place in PBL cycle. Specifically, they commented that working with peers made them integrate each one's ideas, perspectives, and points of view. They also recognized that having to analyse, cooperatively, the situation and propose ideas to apply and integrate knowledge from different fields to look for a solution of the problem facilitates their learning process. Some examples of comments are: "Through cooperative work, we learn to share and collaborate and also to integrate each other's ideas", or "We learn while solving a problem in a group with knowledge that we have to first look for and then apply".
Students identified generic and research skills as the main learning outcomes. They were aware of their own development in generic skills, such as cooperative work, communication skills, critical search of information, and self-directed learning, as exemplified by comments such as: "We learn to dialogue and communicate better among ourselves" and "It empowers our autodidactic ability, since we have to know what we have to learn". They also gained experience in research skills, formulating hypothesis from an unstructured scenario, analysing problems, looking for solutions, and evaluating their final outcome, as shown by statements such as: "We have learned how to analyse complex mechanisms" and "We have to think about new questions and discuss what we know to solve a problem". Integrating different perspectives, formulating new ideas, relating knowledge from different fields, and doing research makes students develop their creative thinking, as they also state: "We ask questions and then research them, so we analyse knowledge deeply, and we can think and reflect differently than before", and "Working with different peers make us integrate different perspectives and think about different hypothesis and questions".
The students' learning experiences through the PBL methodology was also analysed using qualitative data obtained from the student comments. Five main categories were identified: satisfaction, usefulness, role of tutor, evaluation, and limitations.
Satisfaction: Students were satisfied with the methodology and perceived that PBL promotes long-term retention of knowledge and acquisition of skills: "This methodology is more effective than traditional learning because we consolidate knowledge in a better and deeper way".
Usefulness: Students see PBL as useful for their future professions. Medical students emphasize this aspect more than human biology students. Students start learning how research is performed and consider the skills they learned to be important for their development: "I think this kind of methodology is very positive and essential for our development as future professionals of health sciences".
Role of tutors: Students identified the role of the tutor as relevant. They pointed out that there are different kinds of tutors; most considered tutors with previous experience in PBL, who guides and acts as a facilitator, to be ideal. "It would be nice if all tutors had the same, concrete indications to guide the sessions in a proper way, and if too directive tutors could be avoided".
Evaluation: Students perceived the evaluation in a positive way, although they noted that some criteria are subjective, and that the tutor can influence the qualifications. "I think that in some evaluations, some criteria are subjective and differ according to the tutor".
Limitations: Students identified some limitations of the methodology, such as organisational issues, variability of tutor instructions, and small group issues: "Small groups are suitable for performing PBL, however, sometimes there is tension between us" and "Some tutors make fewer comments than others. PBL is easier for the groups that have a tutor who gives more guidance". According to Chng et al. (2014), the tutor plays a role in facilitating student learning; thus, rather than simply conveying knowledge, the tutors question, make suggestions, and challenged the ideas raised by students (13). They have to make the transition from teacher as knowledge provider, to tutor as a manager and learning facilitator (8). This task demands a great amount of time and preparation;

Discussion
further, the better the tutor knows the students and the group interactions, the better he/she can guide the students' learning processes. In PBL-courses, the same tutor was present for the full 10week course; in contrast, in the PBL-module, the tutors changed for every problem, and each tutor was only present for 3 weeks. Therefore, we partly attribute the improvement of the students' skills development to having long-term tutors, who were in a position to better guide the learning processes. In this study, students recognised the pivotal role of the tutor in their development and considered that an ideal tutor should facilitate and guide the learning processes as well as promote creative thinking. Whenever the tutor had not maintained this role, he/she was identified as a limitation of the PBL methodology.
Thus, our results suggest that both students and tutors perceived that the students developed a high level of generic and research skills in PBL-courses. Additionally, the results showed that a strong correlation existed between the development of generic skills and that of research skills. These results are also aligned with comments from both students and tutors.
Active learning and student-centred methodologies, such as PBL, imply that learners play an active role in planning, monitoring, and evaluating the learning process. Thus, students have to consider different ways to approach a task, set clear goals, select strategies for achieving these goals, anticipate what has to be done, and evaluate the process and the product of the learning cycle (2).
Development of both generic and research skills are intrinsically tied to this process. In this light, the strong correlation between the acquisitions of these two sets of skills is not surprising. In fact, the ability to become a knowledge seeker, to be able to collaborate and communicate, and to regulate and self-direct this learning process are essential skills necessary for defining a problem, analysing the situation, and integrating and applying knowledge to develop solutions for new situations (9,11,12). Additionally, students and tutors think that this methodology can also enhance creative thinking. There is a general consensus that domain-knowledge and skills are major components of creativity, and that creativity occurs when investigating various aspects of a problem (23,24).
Scientific exploration and activities, such as definition of scientific problems, hypothesis formulation, design of a research, and evaluation of evidence, are considered key elements in scientific creativity development (25).
Our hypothesis that cooperative work during the PBL learning process can enhance the development The PBL collaborative learning environment is also favourable to cultivating creativity. A high percentage of students (86%) thought that they developed their creativity through these PBL-courses.
Based on the creative skills described by Burnett and Keller-Mathers (2017), we determined how these skills are developed during the PBL phases. In the first phase, students are encouraged to produce and consider many alternatives to solve the problem, and then they have to combine the different ideas and synthesise them into a working plan. In the second phase, students must share their research results and highlight the essence, while keeping an open mind to new ideas or perspectives as possible solutions to the problem, and then elaborate their own answers. In the third phase, students require originality to present their findings, and they put their new ideas into a bigger framework, based on the problematic situation. They also perform a self-evaluation about their participation in the group, during which they become aware of their emotions, an aspect that has also been linked to creativity. Identifying how these skills are implicit in the PBL learning cycle can help educators to emphasize them and hence better promote creative skills development.
Both students and tutors in the PBL-courses were more satisfied overall than those in the PBL-module, and those in PBL-courses scored the usefulness of this pedagogical approach higher. Notably, however, significant differences between the students of the different Bachelors have been found in PBL-courses. Because satisfaction and usefulness show a strong correlation to each other, the differences in these two items can be attributed to the perception that medical students find learning through PBL more useful for their future professional lives than do human biology students. Indeed, the skills applicability is very clear for the medical profession, and less clear for human biology students (whose career goals are often less clear), even though all develop essential skills for their future professions.
Students who took the PBL-courses stated that they were satisfied with this methodology as it allowed them to develop useful skills, which is also demonstrated by the high correlation between satisfaction and usefulness, and the self-perceived generic and research skills acquisition. Also, students perceived that they retained the knowledge gained over the long term. Thus, this study demonstrated that PBL promoted the development of generic, domain-specific, and self-reflection skills and longterm knowledge retention that should enable individuals to gain and apply new knowledge and skills as needed, as has been previously shown (26,27). These higher cognitive abilities, such as problemsolving, collaboration, or creative thinking, will be required to confront new future and social challenges in our ever-changing world (26,28,29).
The increase of student satisfaction for PBL-courses as compared to that for PBL-module could also be attributed to the recognition of, and adjustment to, the student workload, which was identified as a critical aspect in the PBL-module. In that first model, students noted their dissatisfaction with the fact that the amount of workload was not rewarded enough in the final course mark (16). This problem was solved in PBL-courses, as the final marks were independent from traditional teaching. Students evaluated positively the different evaluative tasks, and the only weakness they noticed was the role that subjectivity played in some cases. Student perceptions of assessment are influenced by previous experiences; this means that any intervention involving assessment can be perceived in various ways by students, and thus can affect them and their learning process (30). In this study, the student perceptions about assessment had to be considered to help clarify student concerns.
Our study has several limitations, the most important of which is related to its own characteristics.
The overarching aim of the study was to explore whether changing the PBL implementation model enhances student learning outcomes and satisfaction with the learning experience. It was performed in a naturalistic academic environment, and no experimental interventions were carried out. In this non-interventional design, many variables changed between the two models that were compared.
Consequently, we cannot identify which factors contributed the most to enhancing the aspects analysed (e.g., the perceived improvement of learning outcome and better satisfaction with the courses). Another limitation is the lack of evidence of the real student development in generic (including creative thinking) and research skills in each model. This was not possible as the assessment methods used in these models were not comparable; therefore, we focused the study on perceptions of students and tutors. However, we consider that our results are of interest as they show significant differences between both models, and describe how these skills were developed, from the students' and tutors' perspectives.

Conclusions
We conclude that the shift from PBL-module (integrated within traditional subjects) to full PBL-courses (with courses devoted to using PBL) improved the student perceptions about their improvement in developing generic and research skills, their opinion about the educational value of this pedagogical approach, and their satisfaction with their learning experience. It is important to outline that tutors also considered PBL activities and learning experiences to be satisfactory and useful. Evaluating the individually scored PBL implementation strategies also showed that PBL-courses were scored higher than the PBL-module. Correlation analyses showed the important relationship between the acquisition of generic and research skills as well as between perception of the usefulness of the PBL activities and satisfaction with the experience. This is especially important as student satisfaction may enhance the student engagement, thereby leading to improved learning. Our study also showed how creative skills are obtained in the different steps of PBL tutorials. We hypothesise that the main educational settings in the PBL-courses that contributed to this improvement were i) the use of assessment tools that were better aligned with desired learning outcomes (which encourage students to reflect about their own learning processes) and ii) maintaining the same tutor for the entire course duration.
Finally, the results of this study suggest that PBL-courses are a suitable pedagogical approach to develop generic and research skills, which have been identified as 21 st century skills that must be taught and assessed in higher education (31).

Declarations
Ethics approval and consent to participate: The Academic Coordination Office board of the School of Health and Life Sciences approved the study protocol. The protocol required that: i) participants were informed of the objective and the methods of the research; ii) participation of students and tutors was voluntary and anonymous; and iii) students' decision had no consequence in any way. Participants were later informed of the study characteristics, gave their oral consent to participate, and accepted to follow study requirements. Written informed consent was collected for classroom observations.

Availability of data and material:
The datasets analyzed during the current study is available from the corresponding author on reasonable request.

Competing interests:
The authors declare that they have no competing interests.

Funding:
The project was funded by CLIK (Center for Learning Innovation and Knowledge) at Pompeu Fabra University and it did not play any role in the study design.
Authors' contributions: GR made substantial contributions to the study design, acquisition, analysis and interpretation of data. She was responsible for surveying the participants through the questionnaires, observing the PBL tutorial sessions, analyzing the data (quantitative and qualitative) and drafted the conclusions. She has also been involved in writing the manuscript and has also given final approval of the version to be published. MC has contributed to the study design, data collection, analysis and interpretation of data. For the qualitative data analysis process, she carried out analysis of the data independently, then discussed and agreed themes jointly with GR. She was the academic coordinator of the PBL-module and PBL-courses that are analyzed in this research. She has also been involved in writing the manuscript, revising it critically and giving the final approval of the version to be published. JEB and has been involved in revising the manuscript critically for important intellectual content and has also given final approval of the version to be published.

Figure 1
Mean scores given by students in the generic skills dimension in the PBL-module (n = 233 students) and PBL-courses (n = 277 students). ** p < 0.01

Figure 2
Mean scores given by students in the research skills dimension in the PBL-module (n = 233 students) and PBL-courses (n = 277 students). ** p < 0.01   Responses of students and tutors in the PBL-courses to the question: "Do you consider that the PBL approach contributed to developing your/the students' creative skills?" (n = 277 students, n = 10 tutors). Data are expressed as percentages.