The Language Demands in Mathematics: A Case of the Kenya Certificate of Secondary Education Mathematics Examination

This paper reports that the results of a mixed-methods study of the Kenya Certificate of Secondary Education (KCSE) mathematics examination, a high-stakes test taken by students upon the completion of secondary education. All the KCSE examinations are administered in English, which is the second language for almost all Kenyan students. The general research question is: Do examinations, such as the KCSE mathematics assessment, measure knowledge of the content area independent of advanced knowledge of English? The statistical analyses (correlation and regression) revealed that English scores predicted students’ mathematics scores. Linguistic analysis of the English language requirements of the test revealed that the test items require a specialized knowledge of English. In this paper, we present our analysis and discuss the implications for future research and professional practice.


Introduction
The Kenya Certificate of Secondary Education (KCSE) examinations are high-stakes tests taken by students upon the completion of secondary education. These tests determine whether a student will be awarded the certificate of completion of secondary education, entrance to post-secondary education, and admission into competitive courses in Kenya. The KCSE examinations are administered in English, which is the second or perhaps third language for almost all Kenyan students. Research shows that for all the content areas assessed by the KCSE, Kenyan students perform most poorly in mathematics (Aduda, 2003). Previous research indicates that the knowledge of students for whom the language of assessment and instruction is not the first language, often is underestimated.
The focus on this paper is on the relationship between knowledge of English and performance on the KCSE mathematics examination. The specific research questions addressed are as follows: 1. Is performance on the KCSE mathematics examination related to performance on the English language examination? 2. How can we characterize the English language complexity of the mathematics examination?
The findings from this study have the potential to contribute to a broader understanding of the relationship between English language and mathematics learning and assessment in Kenya.

Overview
In Kenya, instruction is delivered in English after standard three (Grade 3). However, for many students, English proficiency that is sufficient to support both mathematics instruction and valid assessment may be problematic throughout their schooling. To examine these relationships, we conducted an analysis of the KCSE 2011 Mathematics and English examination scores of a sample of 49,815 students of the 413,492 students who took the examination in that year. The relationship between the predictor variable (English scores) and the outcome variable (mathematics scores) was modeled for these students. To examine the potential source of this association, a detailed linguistic analysis was performed for one test item and the examination instructions.

Background: Education in Kenya
Kenyan National Policy asserts an urgent need for becoming "a newly industrializing, middle income country providing high quality life for all its citizens by the year 2030" (Republic of Kenya, 2007). The recently launched development blueprint: The Kenya Vision 2030 reflects the national policy. As a result, major reforms in the education system are being implemented. The goal of these new reforms is to produce mathematically and scientifically proficient citizens. One consequence of the implementation is that the results of the KCSE are used in selecting students for entry into higher education. That is, students' grades in mathematics and science determine their access to competitive courses of study. The rationale for this policy is the potential of students to contribute meaningfully to the country's economy (Republic of Kenya, 2007).
Kenya emphasizes education for all citizens. Kenya embraces an 8-4-4 system of education, where students enroll in eight years of primary education (elementary), four years of secondary education (high school), and four years of tertiary education (university). Although every child has a right to education in Kenya, the cost of education depends upon the school that a student attends. Consequently, schooling is not without costs for students. Entry into secondary level is determined by students' performance on the Kenya Certificate of Primary Education (KCPE) examination, while entry into the university is determined by performance on the KCSE examination.
Kenyan students' performance on the KCSE mathematics examination is poor, and this represents a decades long trend. Prior research attributes the poor performance in mathematics to student-related factors, for example, attitude, school-related factors, such as inadequate teaching and learning resources, and teacher-related factors like inappropriate teaching methods (Mbugua, Kibet, Muthaa, & Nkonke, 2012).
Notably absent from past discussions about this finding is the role of English proficiency as a potential source of variability in students' performance on Kenyan high-stakes examinations, such as the KCSE. English is the language of both instruction and assessment from standard four (Grade 4) to the completion of university education in Kenya. English language is taught as a compulsory subject together with Kiswahili and both are the official languages in Kenya. Students' usage of their first language (mother tongue) is a punishable offense in some schools. In others, the use of Kiswahili is forbidden. However, teachers are allowed to mix the two languages-code-switch-between English and Kiswahili while teaching, if they deem it helpful in facilitating students' learning.

High-Stakes Testing in Kenya
In Kenya, high-stakes examinations are given in English, which is not the first language of almost all students. Consequently, questions arise about the linguistic challenges that students encounter, while taking these high-stakes examinations. Solano-Flores and Trumbull (2003) noted that "assessments often confound language skills of examinees with their academic aptitudes" (p. 3). And they posit that, "The fundamental notion of test validity is that low test scores should not occur because of factors that are irrelevant to the construct an instrument intends to measure" (p. 3). Cuevas (1984) recommended that the primary language of instruction, the student's level of reading proficiency in that language, and the skill measured should all be taken in consideration when interpreting the scores of students from a language minority on mathematics achievement. With reference to the KCSE, he following questions arise: 1. Do the test items of the KCSE demand both advanced English linguistic and disciplinary knowledge from the students? 2. What English skills are required by these examinations? 3. What linguistic challenges might hinder the students' understanding, and hence their responses to individual test items and their understanding of the instructions for taking the examination?

Academic Language: The Mathematics Register
Each academic discipline requires students to master the ways of thinking and communicating including the use of the special language or register of the discipline (Wilkinson & Silliman, 2008). These academic registers are complex and distinct from "everyday language". Texts and tests are constructed by using the register of the discipline. O'Halloran (2005) has argued that for mathematics: Natural language, symbolism, models, and visual displays are combined for expressing mathematical ideas: This is referenced as the "mathematical register".
The linguistic complexity of test items may account, at least in part, for the poor performance of Kenyan students on achievement tests, such as the KCSE. Some of the linguistic factors that may affect students' test comprehension include: unfamiliar vocabulary, complex grammatical structures, nominalization, multiple embedded clauses, and passive voice constructions (Abedi & Linquanti, 2014). Additional linguistic challenges inherent in high-stakes mathematics examinations include technical vocabulary, prepositions, conjunctions, and pronouns, ellipsis, with-preposition phrases, abstract nouns, lengthy nouns, complex sentences, if-clauses, background knowledge, "having" and "being" verbs, and symbolic representations (Fang, 2006;Schleppegrell, 2007).
Adequate knowledge of academic English is particularly important for those Kenyan students who have not attained full English proficiency. Moschkovich and Scott (2013) identified the language features of written mathematics that present problems for English learners: background knowledge, syntactic, and lexical. Students may not have the background knowledge necessary to interpret the test item correctly. At the lexical level, there can be problems with unfamiliar words, unfamiliar phrases, and unfamiliar connotations of words with multiple meanings. At the syntactic level, challenges for English learners include complex sentences, multiple subordinate clauses, and dense noun phrases and the use of the passive voice without an agent.
To understand the potential linguistic challenges in the KCSE mathematics tests in Kenya, we analyzed the 2011 mathematics examination. The examination assesses students' knowledge of mathematical concepts and their applications. We note, however, that this kind of assessment has been characterized as unreliable, especially among students whose first language is not English (Llabre & Cuevas, 1983). Examinations that emphasize English learners' knowledge of concepts are more likely to confound their language abilities with academic skills.

Background Information
Upon completion of the KCSE (a four-year secondary school course of study) students take the paper and pencil national examination (http://www.elimu.net/Secondary/Kenya/KCSE_Student/Maths/Intro.htm). This examination is given, administered, and scored by the Kenya National Examination Council (KNEC). Students must have taken at least seven subjects to qualify for the KCSE certification. They must have taken three compulsory subjects-English, Kiswahili, and Mathematics, at least two science subjects, at least one humanities subject, and at least one of the technical or practical subjects. The full examination assesses students' knowledge of concepts and their applications. The mathematics examination consists of two sets of 24 test items. The test items were classified independently into various mathematical strands based on the National Council of Teachers of Mathematics (NCTM) (2000), Principles and Standards for School Mathematics by Professor Uptegrove (Uptegrove, personal communication) and the first author. Items of the same strand were tallied, and Table 1 summarizes the strands of mathematics assessed by the test including both the number of test items and the percentage of the test designated by each.

Data Organization
The KCSE test results were drawn from the KNEC. The authors acknowledge and appreciate the permission by KNEC to use the results for education purposes. The KNEC provided the data as a text file, which was transformed for the subsequent analyses reported in this paper. Python programming language and the R statistics package were used to restructure the data into appropriate usable format for analysis (The R Project for Statistical Computing, 2015). The information in this data set consists of information about: the school code, school random number, school name, and student's gender, aggregate points scored by a student, and letter grades for each subject completed by the students.
The sample analyzed for this study consisted of the examination results from 50,584 students from the Western region of Kenya (12%) of the total 413,492 students who took the KCSE examination in 2011. The Western region includes 699 schools. Of the 50,584 students who had registered for the KCSE examination in the Western region of Kenya, 22,178 (44%) were female while 28,406 (56%) were male. Of the total 50,584 registered students, 49,815 (98.5%) were scored. A missing subject score or examination malpractice resulted in cancelling the examination eligibility for some students. Consequently, the remaining 769 students did not receive their scores, because they did not provide the required documentation during registration or they were absent during the exams.

Results
The results are reported in two sections: First, we provide data on the relationship between scores on the mathematics and English examinations. Second, we offer an analysis of the instructions for taking the test and provide a detailed linguistic analysis of one test item measuring geometry and arithmetic knowledge (Question 7 from Paper 1).

Scores for the English Examination and the Mathematics Examination
The following data analyses show how the students performed in various subjects in the Western region of Kenya (see Table 2). Mathematics is characterized by the highest variability, while English shows the lowest variability. The box plots reveal that all the other subjects appear to have a higher median score than mathematics, which is characterized by a non-normal distribution. This analysis illustrates the differences in variation in the subject scores and indicates outliers for mathematics and chemistry. The spread of the scores in the subjects is fairly symmetrical, except for mathematics, as indicated by the relative positions of the medians within the respective boxes.

The Relationship of Scores: English Examinations and Mathematics Examination
One research question concerns whether the mathematics scores are associated with the English scores, and if so: How strong is the association? If these two variables are related, the question arises about potential factors causing that effect. A finding that both subject scores are related is not a sufficient basis for proving causality. Revealing that a correlation exists is the first step in searching for causality. Conversely, if a correlation does not exist between the two subjects, a causal relationship can be ruled out.
We calculated the Spearman rank order correlation coefficient to examine the relationship among the two variables. The scores for both variables are of ordinal scaling. The rankings of the grades ranged from one to 12, with 12 representing the highest score and one representing the lowest score. The Spearman's r s = 0.6006 indicated a fairly strong positive relationship. This test confirmed a relationship between mathematics scores and English scores. This finding suggests that the use of linear regression to examine the relationship for prediction would be appropriate.

Statistical Significance
We tested a mathematical model in order to determine the relationship between the English scores (independent variable) and the mathematics scores (dependent variable). We determined that the equation of the regression line for predicting grade point average mathematics scores from English scores was: Mathematics score = 0.7557 (English score) -0.6322. This equation provides a reasonable and accurate prediction. The amount of variability accounted for was 36.07%. We recognize that there are other variables in addition to English language that might affect one's mathematics score. These may include, for example, the amount of time students spend practicing and studying different aspects and topics in mathematics, motivation to achieve high grades and interest in other subjects among other factors.

Analysis of the English Language Requirements of the KCSE Mathematics Examinations
A second research question concerns factors that may be responsible for the association between mathematics and English achievement on this examination. We examined the overall instructions and one mathematics test item from 2011 KCSE mathematics examination. Our goal was to identify the linguistic demands that might hinder students' comprehension and hence their ability to solve the item correctly.

Analysis of the Test instructions
The following analysis concerns the written instructions for the examinations. Instructions should be clear and parsimonious. The written language of the instructions should communicate effectively what students have to do to complete the examination. Table 3 reproduces the instructions for taking the examination. Table 3 Instructions for the KCSE Mathematics Examination Paper 1: Instruction to candidates 1. Write your name and index number in the spaces provided above. 2. Sign and write the date of the examination in the spaces provided above. 3. This paper consists of two sections: Section I and Section II. 4. Answer all the questions in Section I and only five questions in Section II. 5. All answers and working must be written on the question paper in the spaces provided below each question. 6. Show all steps in your calculations, giving your answers at each stage in the spaces below each question. 7. Marks may be given for each correct working even if the answer is wrong. 8. Non-programmable silent electronic calculators and KNEC mathematical tables may be used except when stated otherwise. 9. This paper consists of 19 printed pages. 10. Candidates should check the question paper to ascertain that all the pages are printed as indicated and that no questions are missing.
Our analysis addresses Instructions 5, 6, 9, and 10, which may be potentially problematic for students' comprehension: 5. All answers and working must be written on the question paper in the spaces provided below each question.
6. Show all the steps in your calculation, giving your answers at each stage in the spaces below each question.
The instructions should provide a short description of what is meant by "workings". These may include the use of appropriate substitutions, diagrams, graphs, and charts among other examples. "All answers" and "working" may be interpreted to mean that the answers are more important than "workings", that is, the students' written presentation of their calculations-the steps of solving the problem. However, the students' "work" precedes arriving at the answer. Furthermore, students may score points for showing the steps in their calculations. In order to understand the "steps" and "stage", students first must understand Instruction 6.
Moreover, the second part in Instruction 6, "giving your answers at each stage in the spaces below each question", may be interpreted to mean that there will be more than one answer for each question or for each "stage" (of the "working"). Asking students to show all their "calculations" may constrain students' ability to portray accurately what they know, since there are multiple ways solve the problems.
The following modified version of these two instructions reduces the complexity (Professor Alston, personal communication) * : 5. Solve each problem and write down your final answer in the space on the question paper. 6. Write down the steps you used to reach your answer in the space on the question paper. Regarding Instructions 9 and 10: 9. This paper consists of 19 printed pages. 10. Candidates should check the question paper to ascertain that all the pages are printed as indicated and that no questions are missing.
The question arises whether Instructions 9 and 10 are necessary, as they may distract students from the purpose of the examination.
Analysis of a test item on geometry and arithmetic. The following test item selected from the KCSE (Paper 1, Question 7) appears to be designed to measure students' knowledge of arithmetic and geometry: The external length, width and height of an open rectangular container are 41cm, 21 cm, and 15.5 cm, respectively. The thickness of the material making the container is 5 mm. If the container has 8 litres of water, calculate the internal height above the water level.
The test item includes aspects of "mathematical register", such technical vocabulary terms of "length", "rectangle", "width", and"height" and the grammatical feature of dense noun phrases ("an open-rectangular"). The technical vocabulary terms define the particular geometric shape referenced in this item. Students would have to know the precise meaning of these words in order to comprehend the problem. Additionally, the item includes derivations ("respectively", "container", and "thickness"). This item includes grammatical structures that are complex and require students to deploy limited cognitive resources to make sense of the problem prior to solving it. Students would have to have the ability to deconstruct this test item according to the syntactic requirements that may complicate processing and their efforts to make sense of the problem. Some of these structures present render the meaning difficult to predict, including long subject nouns (the long pre-modifier in the first sentence: "the external length, width, and height of an open-rectangular container") and non-finite clauses ("making the container"). Other aspects tax working memory, such as long distance ellipsis (in the last sentence: "of the container") (Scott & Koonce, 2014). Finally, the test item employs the use of the conditional ("if"), and a long adverbial phrase with an embedded dense noun phrase ("the internal height above the water level"); both of these are syntactically complex. This analysis is summarized below in Table 4 (Professor Silliman, personal communication) * . The following modified version of this test item reduces the syntactic complexity and consequently, the burden on students' working memory. This version includes two defining features of the "mathematics register": Technical vocabulary and dense noun phrases: An open-rectangular container has the following dimensions: Length of 41 centimeters; width of 21 centimeters; and height of 15.5 centimeters. The thickness of the walls of the container is five millimeters. A total of 8 litres of water was poured in the container. Find the internal height of the container that remains above the water level. This analysis suggests that advanced knowledge of English syntax is required by student to understand this test item and the instructions for the examination. Questions about the validity of this test item are considered: 1. Does the item measure knowledge of arithmetic and geometry or knowledge of the "mathematics register" and English syntactic complexity?
2. If students can earn points for their "workings" (according to the instructions), then they should understand that the steps of their problem-solving process must be presented in writing on the test booklet page. The modified version of the item encourages that representation of the students' work.

Summary and Discussion of Findings and Implications for Practice
In this paper, we present the analyses of the scores for both mathematics and English. The results indicated that students' English scores predicted their mathematics scores for this sample of 49,815 KCSE test-takers. Our subsequent analyses of the mathematics test focused upon the English language requirements of the test items and instructions. The results show that for one test item measuring knowledge of arithmetic and geometry, specialized knowledge of English was required to understand the meaning of the questions posed: (a) general academic and mathematics-specific vocabulary items; (b) a higher frequency of informational words as the means to achieve more concise expression; (c) complex syntax, which allows embedding of complex ideas into fewer words. These are features of the "mathematics register". Further examination of the language demands of the KCSE may be useful for educators in Kenya and worldwide, as we all seek to answer the questions: 1. What considerations should one make when implementing mathematics examinations in English for students for whom English is not the first language?
2. How should students be prepared for this examination in the English language, both written and oral? 3. How can students best be prepared to demonstrate their knowledge of mathematics on high-stakes tests, such as the KCSE?