Writing evaluation in university English preparatory programs: Two universities of Turkey and Saudi Arabia

Scholars have given little attention to testing’s influence on the outcomes of university English preparatory programs (UEPP). Variations on the two main classifications of writing evaluation, the holistic and the analytic were examined. The objectives were to identify the assumptions for inclusion of writing in UEPPs, analyze the skills and abilities tested and finally, to examine the correlation between program assumptions and testing and the potential of different testing methods to impact student motivation. The aims and assumptions of the programs and course materials were analyzed via synchronic and diachronic comparisons of program structures and teaching materials, using two examples from the past and one currently in use. Results revealed that testing instruments designed and used only for grading, failing and promotion of students do not provide constructive student feedback, which is a demotivating factor. Testing and evaluation in general should be primarily constructive and positive. UEPP writing examinations should be evaluated analytically rather than holistically for reasons of fairness and to provide constructive and serious feedback to students. Rubrics should be constructed for the marking of paragraphs and essays to ensure fair and consistent marking in large programs with team teaching. The objective testing of writing skills must be implemented to support instructional goals. Thus, evaluation should be analytical, not holistic. The attention drawn to the linking of student motivation to elements of analytical writing evaluation is the significant contribution of this study. © 2020 JLLS and the Author Published by JLLS.


Introduction
Most university preparatory programs include and divide instruction according the four skills of listening, speaking, reading and writing, with many treating grammar as if it were a fifth skill. The experience of the writing components of two university English preparatory programs in the academic years (AYs) 2010-11 and 2011-2, one in the Kingdom of Saudi Arabia and one in Turkey, are specifically treated in this study. An update for the present AY 2018-9 is also presented for Turkish program (Table 1). While the focus of the study is the evaluation of writing, other relevant factors to be briefly considered are program goals, curricula, instructional materials, methodologies for efficient teaching, and program restraints including the number of hours per week, student-teacher ratios and the qualifications and experience of instructors. Table 1. Two English preparatory programs: 2010-2012-2019 While the differences between the two programs outnumbered the similarities in the earlier period, the similarities between the preparatory program at Qassim University in the Kingdom of Saudi Arabia and that of Düzce University in the Republic of Turkey are important to consider. The most obvious characteristic shared by the two programs has been that they were preparatory programs. That means that they were intended to prepare students in the programs for study in their colleges with English as a medium of instruction. This brings up the first difference. In AY 2010-2011, the Düzce University program was comprised of ten hours per week of combined reading and writing, ten hours of combined speaking and listening and two hours of supervised conversation and study for a total of 22 hours per week. This academic year, the balance is ten for listening and speaking, ten for reading and writing, and four for communicative grammar for a total of 24 hours per week. The Qassim University program, however, was made up of six hours of reading, six hours of writing and four hours of listening for sixteen hours of English language instruction out of a total preparatory program of thirty hours per week, with courses in Mathematics, Computer Studies, Natural Science and Study Skills making up the remainder of hours. With the exception of Study Skills, all materials and tests in those parts of the preparatory program were in English as well. Hussain, Albasher, & Salam (2016), describe and evaluate a recent Qassim University Preparatory Year Program. However, their article does not touch on student evaluation. Peaci/Peachy (2013a) compares the Qassim and Düzce administratively, but again, without dealing with the question of student evaluation. The two programs are compared in Table 2  Before getting into the more specific area of the teaching of writing, it is important to mention that, in the earlier period, Düzce University students were admitted directly to their respective university faculties and schools, and that success in the preparatory program was not required for them to continue in their faculties and schools, only for registration in the optional and limited number of courses (to a maximum of 30%) that used English as a medium of instruction. At many Turkish universities, departments did not or could not see any student interest and faculty support for such courses. (Peaci/Peachy, 2014) At Qassim University, (1) entering students were admitted only to the preparatory program of the University, (2) they had to pass all subjects in the preparatory year in order to continue at the University, and (3) (with the exception of the Faculty of Economics and Business Administration), they competed in academic performance for admission to their respective colleges with students having the best grade-point averages (GPA's) winning their first choices of the faculties and schools. In the later period (AY 2018-2019), Düzce University students had the option of doing the English Preparatory Year Program (DUPYP) if they so wished. Nor were they obliged to succeed in the program to continue their studies in their major program. While it was understood and expected that many students would fail to qualify for continued study at Qassim University, at Düzce University, in contrast, performance had not been a concern, let alone a problem, as no student was held back due to failure in English language studies. When students at Qassim University failed at achieving the 60% average passing mark in English or any other subject, they could repeat that subject, usually in a summer term. Such a facility has never been present at Düzce University.

Teaching of writing
Against this background, the teaching of the skill of writing can now be examined in the two programs during the AY 2011-2012 (Table 3). The first difference between the two programs was the number of hours devoted to each. While Writing was part of a ten-hour block of Reading and Writing at Düzce University, one can surmise that instructors allotted it no more than five hours of class instruction out of the twenty-four hours per week of English instruction, 20.8%. In an informal survey of prep program instructors, this author learned that more like seven hours were actually used for reading work and just three for writing. Thus the actual weekly percentage for writing was 12.5%. Five hours of writing instruction times the twenty-four weeks of actual instruction gives a product of 120 hours of writing instruction over the academic year with a weight of 20% to 25% in quizzes and examinations. At three hours per week, 72 was the product. At Qassim University, instruction in Writing was separate from Reading and got a certain six out of sixteen hours per week, 37.5%, and the same weight in evaluations. Twenty-six weeks at six hours per week give a product of 156 hours over the academic year. That also meant about 30% to more time in classroom instruction in writing at Qassim University than at Düzce University. Positive backwash from standardized marking was possibly limited at Düzce University since other necessary elements alluded to above were absent. These missing pillars included little or no marking of homework, mainly because students could not be persuaded to do it out of class or even in class. The reasons for such student resistance were that (1) there was and still is no perceived need to pass English even at 50% with no faculty or school offering any classes with English as a medium of instruction, (2) there was little or no credit for doing homework, and (3) the standards used in test marking were unevenly appreciated and applied to whatever homework was submitted.
It can be seen that there is a premium on success in English language study at the Saudi university while success at the Turkish university hardly matters. This situation greatly affected writing instruction at both, one positively, the other negatively. While it begs the question of why such contrasting situations exist, that question is not the subject of this study. It need only be noted here and now, that any effort to elevate the value of the program in the minds of the students has a significant and positive effect on performance and results in the skill of writing. We can, however, point to important tools and measures that can be taken for the elevation and improvement of student writing skills.

The student-teacher ratio
The student-teacher ratio must be the first measure to improve if one accepts that the feedback from the instructor to the student is crucial to success in writing. Brown (2012) stresses that positive, constructive, written feedback plays a great role in promoting success in writing. He focuses on the promotion of awareness and motivation of student engagement. In the career of this author, he has had to cope with up to seventy-five students in a class. Thirty to forty has been the norm for most of his career. As an administrator, he continually struggled to limit initial enrollment per section to twentyfive with the hope that attrition would bring the number to lower than twenty per section. At a ratio of 20:1, a student could theoretically get up to two and a half minutes of individual attention during a 50minute class period. There can be no surprise that class size affects the quality of teaching and the learning experience. In both programs in 2012, class size was about twenty to twenty-five. In this academic year at Duzce University, the class size is about the same.

Instructors' teaching loads
The second measure to boost the amount and quality of teacher feedback should also be obvious, but unfortunately, it is honored more in the breach than in the observance. Instructors' teaching loads must be kept at a level that permits the correction and evaluation of writing homework. Heretofore, in the Qassim program, which this author developed and led for ten years, almost all instructors felt that for a successful outcome, Writing needed more individual feedback from instructors to students. Thus, teaching loads for individual instructors were balanced between Writing, Reading and Listening. A significant part of the mark in Writing (20%) was allotted for completion of writing assignments. The main reason behind such a large proportion of credit for writing homework was not for formal assessment, but to encourage students with positive feedback when they were successful and for correction of mistakes and guidance in achieving success through the practice needed for progress in writing. It was not deemed constructive to rely on quizzes, midterms and a final exam, especially when negative results led not to encouragement, i.e., not to motivation, but to demotivation and discouragement. Much of the class time for writing was based on in-class, supervised, guided writing tasks. Basically, students had to complete a first draft and an acceptable final draft. Often that meant that most students had to submit not two drafts in total, but three or more. All drafts had to be corrected by the writing instructor for grammar, spelling, punctuation, form and organization. Therefore all writing instructors had heavy loads of draft correction. Administratively, this suggested either the assignment of not more than two Writing sections and a Listening class for an instructor, or a reduced teaching load. Indeed, it was felt that feedback in writing instruction was crucial for success and that it could not be overemphasized. This author recommends at least two drafts per week per student. A class of twenty students should be producing forty to fifty drafts per week. Up to a third of that total, perhaps, can be reviewed in class time. Thus, it is suggested that an instructor with one Writing section should teach no more than eighteen hours per week, an instructor with two sections should be doing less, no more than sixteen hours a week. It is the responsibility of the university administrations and the authorities over them, to ensure that English language programs are adequately staffed. Although young language instructors may welcome the chance to earn extra money to meet the expenses of establishing new homes and families because their salaries are inadequate to meet such needs, letting them teach twenty to thirty-five hours per week does not allow for proper class preparation, let alone for giving enough attention to the correction and evaluation of essential homework assignments. The solutions to the drawbacks of high teaching loads are lighter loads and higher salaries.

The quality of the instructional staff
The quality of the instructional staff should also be a priority. Nowhere is this more evident than in foreign language teaching, and especially in the preparation and hiring of those instructors with skill in the teaching of writing. As an administrator, the author found that not all language instructors could write well, let alone teach writing well. This, he has seen, is quite as true for language instructors who were teaching their mother tongue as it was for those who were teaching their second or third languages. Those who were trained in writing and the teaching of writing, and especially those who had solid experience in writing instruction were not common.

The choice of textbooks
The choice of textbooks or other instructional material for the teaching of writing is another very important factor. Texts which are culturally appropriate can serve to motivate the students to write about topics with which they are familiar and in which they are interested. Peachy (2013b) Providing good model paragraphs to serve as examples is also vital. Students can internalize good writing habits by copying and emulating them. Peaci/Peachy and Tosuncuoglu (2018) point out value of the Ottoman precedents from the Palace School, i.e. ,the Enderun, for teaching of writing.

Testing of writing
"Requiring students" or "Students are required" are phrases seen in course descriptions. Is it not enough to have an inspiring teacher? Can an instructor alone overcome apathy, disinterest and indifference? It is assumed that through testing and evaluation for institutional and societal purposes that students are required, pushed to perform. While quizzes, tests and examinations do provide vital feedback to both instructors and students, they are also a stick, both to measure achievement and skill as well as a prod to direct those with insufficient internal drive and motivation. "Washback" is a term used by those who theorize testing to describe its wanted and unwanted consequences. (Cheng, Watanabi, & Curtis, 2004) While testing comes at the tail end of a unit, a course or a program, it is often that "tail that wags the dog." In other words, students and instructors, more often than not, perform their instruction and learning with a testing instrument in mind. The prevalence of the practice of "teaching to the test" is a contentious issue in education these days.
If one accepts the contention that testing has a crucial, if not excessive, influence on education and pedagogy, objective testing that concentrates on its own effects and consequences should be the standard, an ideal for instructors, test writers and academic administrators. Standardized, objective, fair testing is widely accepted in the testing of reading and listening, the two "passive" skills. The science of writing multiple-choice questions (MCQ's) for standardized, reliable instruments of evaluation is well developed, even if the quality of its application is uneven from writer to writer. However, it is in the evaluation of progress and achievement in speaking and writing, the two "active," productive skills, that competence and sophistication lags behind in many programs.
Without training and experience, there can be no competence. While one knowledgeable and experienced in preparing MCQ's can write a test for a thousand or a million and a machine can score them in seconds, minutes or hours, a paragraph or an essay cannot be machine read and/or scanned and scored. Perelman (2013) refuted and confuted Shermis and Hammer's (Shermis & Hammer, 2012) supportive analysis of automated essay scoring.

Rubrics
In the Düzce program, the general feeling of many instructors was that, since success in the Writing component of the examinations was spotty, students should not be held back or even penalized for failure in writing. It has been mentioned above that Writing bore relatedly less weight than Reading except in the later weeks of the academic year. It should also be mentioned that Grammar was taught as a separate subject. Although the grammar textbook was ostensibly communicative in nature, it was tested as passive knowledge using multiple-choice questions (MCQs). Thus, it contributed little to the skill of writing since most mistakes on writing quizzes and examinations were those of grammar usage. It was felt by the foreign language departmental administration that rubrics for the Writing examination scripts would contribute to standardized marking. Consequently, such rubrics were designed by the instructional staff. The result was that marking was largely standardized by the end of the academic year.
Flinders University Centre for University Teaching (2010) has a succinct webpage dealing with rubrics: "What is a rubric?" It explains the two types of rubrics: the holistic and the analytical. With regard to performance, the analytical rubrics "break down and disaggregate multiple criteria," while the holistic ones do the opposite. That is, they mix and package depictions of a performance in order to make a broad evaluation. In an earlier article, Clapham (2003) clearly sets out criteria for text construction and evaluation and defines the two types of rubrics and marking scales in a valuable glossary of terms. Lam and Foong (1986), on the basis of a very small sample, compare the two types of marking in a study that concludes that holistic marking is more generous. That suggests that the holistic model is less reliable. In a rather technical article, Galti, Saidu, Yusuf, and Goni (2018) compare the two marking scales. They emphasize in their "Conclusions and Suggestions" that suitable feedback creates opportunities for students to write and encourages them to do so. They imply that content and creativity are more important than other features. Thus, they favor the holistic. This author prefers the analytical if staffing levels permit it since the feedback and washback are specific and can guide students to correct and improve their writing assignments and products.

Holistic marking
Here is an example of the holistic marking criteria the author has had to use in a placement test.
Example of a Holistic Marking Writing Rubric Used Several Years Ago 1. Can use key words in the first sentence of the paragraph. -3 points 2. Can make himself/herself clear through supporting statements (at least five supporting sentences). -10 points There may be some tense or preposition errors. There may be some punctuations errors. 3. Supporting sentences are presented in some form of an order. -6 points 4. Can use mostly complete sentences; there are few run-on sentences -5 points 5. Can use basic vocabulary or words related to the topic. -6 points For a single instructor with a lot of experience, holistic marking can be useful for compositions of students studying the composition of their mother tongues. They will (1) first be fluent in their native language, (2) they will have had years of schooling in their mother tongue, and, (3) presumably, their levels of lexical and syntactical achievement will be high. With them, such a holistic marking scheme will be possible, but maybe not preferable, except in a situation where the instructor many papers to mark in a short time.
Unfortunately, in English Language Learner (ELL) classes, holistic marking problems are many. As one might infer from what the author has just mentioned, consistent, fair holistic marking for a large number of students with many different instructors is not possible. Nor does holistic marking provide concrete feedback to students who, unsurprisingly, want to know exactly what they have done well and where they need improvement.

Analytical marking
Here is a recent example of an analytical rubric that the author helped design and used several years ago: For those who have not used such an analytic rubric, it can be baffling and frustrating at first. Even later, it is obvious that each paper, be it homework or a quiz, a test or an examination, will take time to mark using such a rubric. The positives, the rewards of marking homework and tests with such an analytic rubric are the following: (1) Criteria can be applied fairly and consistently from student to student and from instructor to instructor, and importantly, student and community perception of fairness can nurtured. (2) If the rubric is related to international standards like the American Common Core or the European CEFR, the consistency can be used for international recognition and certification. They can become badges of writing proficiency or pages in a language passport. (3) Students can take marking as a diagnosis of problems. (4) Instructors can collate results to redirect times allocated to learning tasks, eliminate some tasks and add others.
Two examples of writing rubrics used at Düzce University in AY2018-2019 are shown in the Appendices. The first (Appendix A) is largely holistic, while the second (Appendix B) shows the influence of the analytic approach.

Conclusions
In conclusion, the author knows of none who object to the perception that writing is one of the two productive skills having the weakest outcomes in the programs. He stresses the importance of regular homework and good writing models. Feedback is a particularly essential part of instruction and interaction in writing. Students must be made to understand the importance of a step-by-step approach, a structured procedure in the form of guided writing. Grammar cannot be taught in isolation. At the lower levels, accuracy and form should be stressed over creativity and critical thinking. The objective, fair testing of writing skills must be implemented. Thus, he recommends that evaluation should be analytic rather than holistic. A rubric should be constructed for the marking of paragraphs and essays to ensure fair and consistent marking in large programs with team teaching, using the same standards set by the homework and in-class writing assignments. In short, writing is a learned skill, not a list of rules to memorize. Practice makes perfect, and evaluation is both a stick in both its senses of a measure and rod. Primarily, it should also be a carrot.

Ethics Committee Approval
The author confirms that this study does not need ethics committee approval.