Guiding Students' Clinical Writing and Critical Thinking: Utilizing Scholarly Teaching to Develop and Implement a Clinical Writing Rubric

Recommended Citation Halvorson-Bourgeois, Bonnie; Riotte, Mary; Lambrecht Smith, Susan; and Maxwell, Lesley (2020) "Guiding Students' Clinical Writing and Critical Thinking: Utilizing Scholarly Teaching to Develop and Implement a Clinical Writing Rubric," Teaching and Learning in Communication Sciences & Disorders: Vol. 4 : Iss. 2 , Article 3. DOI: https://doi.org/10.30707/TLCSD4.2/GWME8876 Available at: https://ir.library.illinoisstate.edu/tlcsd/vol4/iss2/3


Introduction
The future of the Communication Sciences and Disorders field depends upon the excellence of its educational programs. Just as speech-language pathologists (SLPs) embrace the necessity of providing evidence-based clinical practice (EBP), those who teach new clinicians are called to provide evidence-based education (EBE) to maximize student learning (Friberg, 2015;Ginsberg, 2010). Given rising health-care costs, SLPs are increasingly held accountable for client outcomes; similarly, instructors in higher education are accountable for positive student learning outcomes to justify program and salary expenditures. Scholarly teaching involves asking questions about the effectiveness of teaching, developing ways to assess those questions, and then modifying teaching methods based on the results. By disseminating these results to a wider audience, instructors engage in the scholarship of teaching and learning (SoTL) thus contributing to the evidence-base that others draw upon when taking a scholarly teaching approach.
Clinical writing and critical thinking skills are two essential attributes for today's health professionals, necessary for accurate interpretation and clarity in conveying complex client information. As communication and language experts, SLP graduate students must meet the American Speech-Language-Hearing Association (ASHA) competencies for exemplary oral and written communication skills (Standard V-B). When writing, students learn to convey complex information using a clear, concise, and objective writing style. Critical thinking provides the foundation for sound clinical decision making and as such, is considered a core competency for the SLP profession (Finn et al., 2016).
There is increasing interest in how graduate education for SLP students addresses aspects of critical thinking in the general context of graduate student development (Finn, 2011;Hancock & Brundage, 2010). Schneider-Cline (2017) found promising evidence that graduate students benefit from workshops specifically targeted for instruction in self-regulation and critical thinking in writing.
Likewise, a recent review of literature exploring the use of rubrics in higher education highlighted their utility for self-regulation and making assessment criteria more transparent for students (Cockett & Jackson, 2018). Those rubrics geared specifically for teaching writing in the health professions (e.g., Hancock & Brundage, 2010;Staltari, Baft-Neff et al., 2010;Van Gilder & Street-Tobin, 2011;Willis & Piazza, 2019), provide good direction for inclusion of content, clarity of expression, and use of professional language but do not go far enough to explicate how critical thinking is approached in student writing.
As such the current paper presents a clinical writing rubric that addresses critical thinking within the context of clinical writing with a high degree of specificity. One means for conceptualizing critical thinking is through Facione's interrelated Core Critical Thinking Skills: analysis, inference, evaluation, interpretation, evaluation and self-regulation (Facione, 2015). These core critical thinking skills are closely aligned with the healthcare professions (Facione & Facione, 2008). In healthcare education, effective tools are needed for clinical educators to assess student proficiency in critical thinking within clinical writing so that instruction can enhance student learning in these complex and related areas. This paper explains how a group of clinical educators used a scholarly teaching approach to design, refine, and implement a clinical writing rubric with a focus on critical thinking

Rubric Development
SoTL is more than just good teaching. It begins by asking and investigating questions related to student learning. The clinical educators involved in this project have long been interested in the synergy between writing and critical thinking and were motivated to improve the feedback process to more effectively advance student learning in both areas. This group wondered about many initial questions including: (1) What is the role of clinical writing in developing critical thinking and deepening academic content knowledge for SLP graduate students? (2) How can feedback on clinical writing be used to develop self-knowledge, reflection and growth in these areas? (3) Given increasing class sizes, how can excellence and effectiveness be maintained in teaching while increasing efficiency? In response to these questions and a desire to improve student writing outcomes, the clinical educators applied for and were granted institutional funding that supported the creation of a clinical writing rubric.
Review of Relevant Literature. The clinical educators were interested in developing a rubric to enhance clinical writing instruction because rubrics are widely used to increase grading efficiency and consistency of feedback across students (Jönsson & Svingby, 2007;Stevens & Levi, 2013). As shown in Figure 1, clinical educators began the design process by locating and evaluating previously published writing rubrics, initially focusing on those related to SLP education. Articles by Staltari et al., (2010) and Van Gilder and Street-Tobin (2011) were relevant to this project because they described rubrics used to provide feedback on graduate student clinical writing. While their design and implementation processes were of interest, neither rubric emphasized critical thinking aspects of clinical writing. The rubric by Stalteri et al. (2010) grouped 17 criteria under three domains (content, organization, and mechanics), but no criteria referred to critical, evaluative, or analytical thinking. The rubric by Van Gilder and Street-Tobin (2011) included one reference to critical thinking (interpretation) out of 12 characteristics grouped under five domains (content, professional writing, grammar and proofreading, and cohesion). A third rubric developed by Hancock and Brundage (2010) to evaluate SLP professional competencies provided a model that was more in line with Facione's core critical thinking skills. Their rubric included a domain for critical thinking comprised of three criteria: interpreting information, analytical reasoning, and addressing alternative possibilities. Two broad criteria pertaining to written communication were included under a separate domain for communication skills. While Hancock and Brundage's (2010) rubric addressed aspects of both critical thinking and written communication, it did so within a wider context of overall clinical competency rather than looking specifically at critical thinking within clinical writing. Clinical educators also expanded the literature search to include rubrics targeting writing and critical thinking beyond the SLP literature discussed above (e.g. Association of American Colleges and Universities, 2009). These rubrics targeted either writing or critical thinking and often in ways that were more applicable to general college writing than the clinical writing used by health professionals. In light of these findings, the clinical educators identified the need for a rubric to provide feedback on critical thinking within clinical writing assignments that is both nuanced and comprehensive. More recently, Willis and Piazza (2019) published a clinical writing rubric for SLP students that added analysis of data and recommendations, an indication that the field of CSD is moving toward refining how critical thinking is addressed for developing clinical writers.

Literature Review Topics
The clinical educators delved into existent literature focusing on effectiveness, design, implementation, and scoring reliability (i.e. consensus) of rubrics. Educators also drew information from research related to the initial questions in the areas of effective adult learning, and the relationship between writing and critical thinking ( Figure 1). Information from this literature review was integrated into the rubric development process, design, and implementation procedures as explained below.
Regarding effectiveness, the research consulted confirmed that rubrics can promote student learning outcomes for a variety of reasons: they clarify expectations; provide efficient, fair, and transparent feedback (Hancock & Brundage, 2010;Jönsson & Svingby, 2007); they draw attention to the writing process (Andrade, 2001), and they support student self-efficacy and self-regulation (Panadero & Jönsson, 2013). Ambrose et al. (2010), in their synthesis of adult learning research, identify one core principle for effective adult learning as "goal directed practice coupled with targeted feedback" (p. 125). The use of a well-designed formative rubric allows for specific feedback that can be used to create goals for future learning. Students may also use rubrics to selfassess their own knowledge or performance gaps to create goals. Furthermore, the process of writing itself has long been perceived as an avenue for deepening academic knowledge and facilitating critical thinking. Students not only "learn to write" but also "write to learn" (Bean, 2011). A body of research has supported writing's role in shaping writers' abilities to analyze problems and understand topical material (Hendricson, et al., 2006;Langer & Applebee, 1987). Therefore, in practicing clinical writing, students may also deepen their understanding of complex or conflicting information and hone their critical thinking.
A pair of staff instructional designers with graduate degrees in the field of education, specializing in instructional design and assessment, assisted with the design of the rubric. A meeting was held with the designers before the initial stages of development, where they provided resources including Stevens & Levi's (2013) introductory book on developing rubrics. A follow-up meeting was held during the rubric design process, and again at the completion of the rubric prototype. At that time, instructional designers provided informal feedback regarding the rubric's strength and adherence to best practices. In addition, some of the clinical educators were introduced to the book How Learning Works by Ambrose et al. (2010) during a faculty book club lead by the instructional designers. This book became the clinical educators' primary resource for current research on adult learning.

Rubric Design Process.
During the developmental process, the clinical educators and students had input in designing the rubric in order to include multiple stakeholder perspectives, as recommended by Stevens and Levi (2013). All clinical educators in the CSD first year in-house graduate center participated in developing the rubric criteria. While keeping in mind the information gleaned from the literature review, the clinical educators were invested in creating criteria that truly reflected their own perceptions of clinical writing. They were involved throughout the design process, which promoted 'buy in" when it came time to implement the rubric with their students. To begin the design process, the clinical educators met together and were asked to draw upon their experience in reviewing student writing to brainstorm two lists: (1) characteristics of strong clinical writing and (2) characteristics of weaker clinical writing. From these lists they identified themes, and through further discussion, created a list of priorities to include in the rubric. A smaller group of clinical educators then took this information, consulted existent rubrics for guidance, and synthesized initial domains and associated criteria. This process was challenging in several regards including categorizing criteria under the domains; for example: does a criterion addressing passive voice belong under "style" or "form"?; and how does a style criterion for succinctness differ from content-related completeness? Through resolving these questions in both smaller and whole team discussions, the team refined their own understanding of clinical writing characteristics. In addition to the clinical educator team, the graduate students were also key stakeholders and were involved in the rubric design as well. Graduate student volunteers participated by applying a prototype rubric to a sample report and evaluating the rubric, as described in the following section.
Rubric Description. The design process resulted in an analytical rubric consisting of five main domains for evaluation which are scored on a 4-point scale (refer to Clinical Writing Rubric, Appendix A). The domains were ordered systematically with two "big picture" areas content and critical thinking listed first, followed by three foundational writing domains (organization, style, and form). Each domain was further delineated into four specific criteria. The rubric's scoring scale conveys clear expectations for learners while using positive language (Stevens & Levi, 2013). An even-numbered scoring scale was used to dissuade instructors from choosing a neutral middle score (Weems & Onwuegbuzie, 2001). The criteria and associated performance descriptions were intended to be specific enough to provide consistent and meaningful feedback, yet general enough for the rubric to be used for a variety of clinical writing applications. The clinical educators were also interested in research examining how rubrics are perceived by different stakeholders. Reddy & Andrade (2010) advocate that instructors shift their perceptions from using rubrics merely as an efficient grading tool to a means of enhancing student learning. Accordingly, the developed rubric is formative in nature as it communicates students' strengths and areas for further growth but does not assign a total final grade. The rubric was created in an Excel document which automatically calculates an average score for each domain to promote efficient use. Facione's (2015) Core Critical Thinking Skills as they relate specifically to clinical writing are a unique aspect of this rubric. These core skills served as the base with which to explicate the subskills that were expected in graduate student clinical writing. For ease of description, Table 1 provides Facione's skills and the related writing components that were operationalized within the performance descriptions in the rubric. Writing clear and concise performance descriptions for the critical thinking criteria was challenging, and these descriptions went through numerous revisions given student and educator input.

Evaluation
Evaluates quality and validity of information

Inference
Formulates salient conclusions, functional implications, appropriate recommendations

Rubric Evaluation and Implementation
Once clinical educators had a prototype rubric, it was refined using the iterative process illustrated in Figure 2.

Figure 2
Iterative Process for Evaluating and Implementing the Rubric In order to gather student perspectives on the rubric, a cohort of second year graduate student volunteers (n=8) evaluated the prototype rubric during the 2015 summer semester. Information on how the students self-identified in terms of culture, race, and gender etc. was not collected. Each student applied the rubric to one of two randomly assigned diagnostic reports, completed an anonymous online survey, and participated in a student discussion group led by two clinical educators from the CSD department. There was no grade attached to the students' voluntary participation, and the clinical educators who facilitated the discussion were not involved in the students' external clinical placements. The objective of the student feedback group was to provide the clinical educators information regarding the usefulness, efficiency, and completeness of the rubric, as well as to follow-up on the survey results. The survey consisted of nine questions on rubric clarity and usefulness using a 4-point Likert Scale (1 = strongly disagree and 4 = strongly agree). The average score was 3.06 across the nine questions, suggesting that students perceived the rubric as a beneficial feedback tool; however, students also specified a preference for combining the rubric with commentary and edits by the clinical instructors directly on the document, and provision of sample reports. Student feedback highlighted the need for improved clarity regarding the rubric domains for content and critical thinking.
The rubric was also used to provide feedback on the final progress note written by a small cohort of students (n=4) enrolled in a Certificate of Advanced Study practicum during the 2015 summer. These students also voluntarily completed the same anonymous survey described in the previous paragraph. Results from this second administration of the survey indicated an average score of 2.8 across the nine questions (using a 4-point Likert Scale of 1 = strongly disagree and 4 = strongly agree). Survey results from both groups of respondents agreed that the rubric provided clear expectations and provided feedback that supported student learning. In response to both student cohorts' feedback and survey results, the clinical educators clarified wording in the performance descriptions, particularly in the critical thinking domain, and clarified criteria for all domains.
After revisions on the prototype were complete, the clinical educators were re-introduced to the rubric. To support consistency in rubric scores across clinical educators, rater training is critical . During agreement testing, the clinical educators used the handbook and rubric to assess the same report. They submitted their rubrics anonymously to a shared drive and the rubric scores were then entered into a spreadsheet. The clinical educators then met to reach consensus on scoring. Any criterion where scores differed across the clinical educators was discussed until the team members reached consensus. During this process, further revisions were made to the rubric to refine the clarity of the criteria and performance descriptions as well as information in the supplementary handbook. Agreement testing was repeated across three semesters using revised versions of the rubric.
Since the initial student input and agreement meetings in 2015, the rubric has been in continuous use to provide formative feedback on two major clinical writing assignments during first-year graduate students' in-house placement: a diagnostic assignment (first semester) and a final progress note (second semester). The clinical educators introduce the rubric to students prior to their formative clinical writing assignments. The rubric and supplementary handbook are reviewed and applied in class, as well as in weekly clinical team meetings held by their assigned clinical educators. Per student feedback during the development phase, the clinical educators continue to provide written feedback on students' clinical writing assignments in addition to the rubric. The written feedback takes the form of specific edits, use of questions and the provision of general suggestions directly to the assignment. This feedback, along with the completed rubric, is meant to guide the students' revision accuracy and document re-submission. The rubric feedback is provided for the initial submission of the assignment except for a final criterion, Revision Expectations, which evaluates the student's response to the feedback and is scored on the revised draft.
In addition to providing feedback, rubrics are also recommended for students to reflect on and evaluate their work (Ambrose et al., 2010); however, student input from the student feedback group during the developmental phase indicated that students felt unprepared to accurately selfevaluate their own clinical writing during their first semester of inhouse placement. Therefore, the clinical educators adopted a gradual approach to self-evaluation of clinical writing as illustrated in Figure 3. Generating student learning goals based on their writing first semester and then requiring students to evaluate their own writing second semester systematically promotes independence in self-regulation while transitioning towards self-supervision (Anderson, 1988). Students submit their self-evaluation rubric with their assignment prior to receiving feedback.

Rubric Usage to Promote Self-Evaluation of Clinical Writing Skills
As previously noted, research indicates that adult learning is enhanced when Students formulate specific learning goals and have consistent opportunities to practice and receive feedback (Ambrose et al., 2010). At the end of each clinical placement, students meet with their clinical educator or preceptor to reflect on their level of independence on a list of clinical competencies based on the Council of Academic Accreditation (CAA) standards and formulate clinical learning goals for the next semester. The clinical educators have revised the clinical writing competencies used across placements so they map onto the rubric criteria. For example, the competency for critical thinking was expanded to encompass the inferential and explanation criteria from the rubric, and additional competencies were included to capture key aspects of clinical writing style (concise, smooth, and objective language). This allows the rubric feedback to be clearly translated into specific clinical competency-based learning goals in the area of clinical writing. For complex abilities, such as writing and critical thinking, allowing carry-over of learning goals across semesters may provide additional mentored learning experiences necessary to meet these goals.
Research reviewed by Ambrose et al., (2010) indicates that the time spent in deliberate practice towards a specific learning goal predicts learning.
In addition to the iterative process of improving the clarity of scoring criteria to encourage feedback consistency, the clinical educators were also interested in student perceptions of the rubric (Riotte et al., 2016). After implementation of the rubric into the clinical curriculum in the Fall 2015, students were given a voluntary online survey at the end of the Spring 2016 semester to assess their perception of the rubric (n=37, 63% response rate). Results showed that 73% of students agreed or strongly agreed that the rubric's feedback for critical thinking solidified understanding of their strengths and weaknesses in this domain; this result was somewhat lower, but broadly aligned with ratings for the other rubric domains (78% content, 81% organization, 78% style and 81% form). When students were asked to rate the importance of a variety of teaching methods as contributing to their growth in clinical writing, responses were consistent with the earlier initial student feedback group, with most students continuing to identify written feedback and edits on the assignment (97% of students) and sample assignments (92% of students) as important; in comparison, 68% of students rated the rubric as important.
Although not formally measured, the clinical educator buy-in for the rubric has been strong. The clinical educators informally report that using a common set of criteria ensures their feedback addresses each aspect of clinical writing for every student. The rubric allows overall strengths and challenges to be more clearly communicated to students. It also has the potential to provide more nuanced feedback. The clinical faculty use patterns of scores across criteria to show students how aspects of writing interact with each other. For example, a student who lacks cohesive ties or transitions between sentences (an organization criterion) may also be vague in their interpretation of interactions among testing data (a critical thinking criterion). While agreement on specific criteria across supervisors has been an ongoing challenge, overall consistency of feedback has likely improved compared to faculty's previous process where no rubric was used.
Although beyond the scope of this paper, considerable effort went into designing a rubric with adequate consensus across scorers. Despite training and iterative re-design for criteria clarity, statistical reliability measures remained low. Jönsson and Svingby (2007) noted that the rubric application studies they reviewed generally reported low reliability coefficients, and that the openended nature of complex performance assessments are prone to lower reliability.

Future Directions
As noted above, difficulty in achieving consistent scoring across the clinical educators has been one limitation to the rubric's development; in addition, the validity of the rubric has not been evaluated. The educational literature indicates that studying the reliability and validity of rubrics is an ongoing area of need. An additional consideration is conducting reliability analysis with a larger number of raters so that applications such as generalizability theory may be used to consider multiple sources of variability.
While the clinical educators are not using the present rubric to assign a letter or numerical grade (where consistent ratings across clinical faculty would be critical), further honing of the rubric training would be beneficial to ensure the clinical educators have consistent understanding of the criteria. Another challenge regarding scoring consistency is allocating time needed for periodic recalibration across clinical educators, particularly in response to staff turn-over. An additional training strategy that has not yet been implemented are annotated "anchor assignments" or exemplars to provide rating models which may be useful to both faculty and students. While students are currently provided with assignment samples, annotating these samples to align them with rubric criteria may help make the criteria more transparent. Although the clinical educators have responded favorably to the rubric, a more rigorous measure of their perceptions would provide more objective data regarding rubric efficiency and effectiveness.
Sharing the rubric with outside practicum supervisors is a logical next step to provide more consistent feedback on clinical writing across semesters, allowing more opportunities to practice and meet learning goals. Tracking students' self-evaluation skills across semesters by comparing data from their own completed rubric to their clinical educator's rubric feedback would be valuable in assessing student outcomes related to their clinical writing over time.

Conclusion
Strong critical thinking is crucial for today's healthcare professionals to guide decision-making and to clearly and accurately communicate complex information orally and in writing. The process of clinical writing, while a professional competency in itself, can serve as a primary modality for developing deep knowledge and critical thinking. The clinical writing rubric discussed here and the larger context in which it is embedded, provides students with opportunities to practice writing, receive and respond to feedback, and create personal goals to self-regulate their future learning. Most importantly, it provides clear and specific guidelines for focusing students' critical thinking during the writing process. According to survey results, students found the rubric beneficial in understanding their strengths and challenges in critical thinking. The rigorous scholarly teaching approach used to create, implement, evaluate and refine this clinical tool has enhanced educator awareness and understanding of the tenets of clinical writing, resulting in improved procedures for clinical teaching. SoTL extends scholarly teaching by contributing to the evidence base through disseminating results. The team involved in this project hopes to inspire a wider circle of clinical educators to engage in systematic, evidence-based inquiry when designing and implementing clinical learning opportunities.