Indonesian Language and Culture Learning Service (INCULS) is a department in the faculty of cultural science, Gadjah Mada University. The overall goal of INCULS is to prepare foreign students with Indonesian language and culture through a training course period. Foreign students from developing countries that get scholarship from Indonesian government to pursue master degree in Indonesia have to be trained in INCULS so that they can survive in term of communication in their everyday lives as well as their academic environment. The program was conducted during an eight-month period from September to May 2007. An evaluation was intended to make judgments about the merit, value, or worth of the training program conducted at INCULS. This evaluation study was to find out the reaction of the students participating in the course and to find out if the program was effective and efficient in delivering knowledge to students. It was envisaged that this evaluation study would increase efficiency, effectiveness and good performance in delivery good quality of learning program at INCULS.
Evaluating reaction is the same thing as measuring customer satisfaction (Kirckpatrick, 1998:25). He continued that if training is going to be effective, it is important that trainees react favorably to the training or they will not be motivated to learn. Motivation has been considered a major impact on human behavior (Sprinhall, N.A &Sprinhall R.C, 1987: 463). According to Brophy (1987), motivation to learn is a competence acquired "through general experience but stimulated most directly through modeling, communication of expectations, and direct instruction or socialization by significant others (especially teachers)." The beliefs teachers themselves have about teaching and learning and the nature of the expectations they hold for students also exert a powerful influence (Raffini, 1993:286). As Stipek (1988) notes, "To a very large degree, students expect to learn if their teachers expect them to learn." School wide goals, policies, and procedures also interact with classroom climate and practices to affirm or alter students' increasingly complex learning-related attitudes and beliefs. Classroom climate is important. If students experience the classroom as a caring, supportive place where there is a sense of belonging and everyone is valued and respected, they will tend to participate more fully in the process of learning. Effective learning in the classroom depends on the teacher's ability to maintain the interest that brought students to the course in the first place" (Ericksen, 1978:3). Courses should be instrumentally motivating by being obviously related to preparation for future goals and aspirations (McPartland & Braddock, 2000).
Cognitive theorists view learning as a reorganization of a number of perceptions; this reorganization allows the learner to perceive new relationships, solve new problems, and gain a basic understanding of a subject area (Sprinhall, N.A &Sprinhall R.C, 1987:192). In addition, learning can be defined as the extent to which participants change attitudes, improve knowledge, and/or increase skill as a result of attending the program (Kirkpatrick, 1998:20).
Thus, it can be assumed that positive reactions of students toward a training program tend to help them learn more. Trainees can learn effectively and will be able to apply the knowledge learned efficiently once they have high motivation to learn. It is in the hand of instructors, administrator and school climate that can facilitate the learning process and increase students' motivation.
The first two levels of Kirkpatrick's (1998) model of evaluation, namely reaction and learning was applied in evaluating the training program. A combination of qualitative and quantitative approach was used to study the population in order to obtain valid and reliable data.
Data collection techniques included observation inside and outside class activities, interview, documents and questionnaire. Twelve foreign students from various countries attending the program were the population of this evaluation study.
1.Reaction: How well did the students like the training course? Each student completed the reaction sheet which was divided into content, instruction and overall course satisfaction. The course content was referred to the clarity of the course objective, the agreement between course objective and lesson assignment, and the well planned, organized and used of time period. The instruction was focused on the knowledgeability of instructors to the subject matter, the teaching methods and techniques employed by instructors, and the understanding and facilitating of the instructors toward students needs. Finally, the overall evaluation of the program was meant to evaluate the whole satisfaction of students to the training program. A five-point scale was used to scale the responses, with 5 being the highest satisfaction and 1 the lowest. The interview was conducted to check the reliability.
2.Learning: Is there knowledge gained after the training course? Can students use the language learned? All students before getting the training did not know anything about Indonesian language; this means that they had zero knowledge of Indonesian language. To evaluate learning, the students' test scores were compared to see if there were improvements in vocabulary, reading, grammar, writing, and conversation skills. The observation was made to see if the students used the language learned in public.
The evaluation was conducted as planned successfully and the results were communicated as follows:
The reaction score to the content, instruction, and overall evaluation was:
The results suggested that the students were fairly satisfied with the program. However, according to the interview, some students revealed that the program was little of important for their future study when they do master degree since they were told they could use English. They perceived that the program was very important for everyday life but not academic. Some students complained that the book was not very attractive and there were no clear objectives in each lesson. Answering to the question if a variety of media was used to help teaching learning process, most students said no and some teachers come to teach without preparation.
There was absolutely learning occurred in the program. It was known that students had zero knowledge of Indonesian before they came to study. This means that they did not know anything about Indonesian; however, after studying they can communicate Indonesian effectively, they can read some paper and even do some writing for their class. To be sure, mean scores of two tests which were done in the middle and at the end of the course were taken to comparison. The report of the test was obtained from INCULS and the results revealed:
Vocabulary Writing Reading Grammar Conversation Total Mean
Test 1 77.58 83.83 85 72.33 77.75 79.29
Test 2 80.25 84 76.91 72.58 80.83 79.91
Total Gain 2.67 0.17 -8.09 0.25 3.08 0.62
The scores show that there was tremendous improvement of students from 0% (zero knowledge of Indonesian) to 79.29% in the first test and 79.91 in the second test. There was an overall gain of 0.62% at the end of the course; however, if we look at each subject in the first and second test, reading score decreases to 8.09%. This means students' reading ability is very limited. Conversational score increases 3.08% which is the highest gain among the five subjects and the vocabulary gain score is 2.67% which is the second highest. The observation was made in school campus and public place including market, football pitch...etc, and it amazingly revealed that most of the time students used Indonesian language to communicate among friends. Indonesian students who communicated with the population reported that they understood the communication 90% from various topics in Indonesian. The foreign students further stated that they preferred communicating in Indonesian to reading or writing.
>>>Conclusions and Recommendations
The students conversationally learned a lot from the program. Language training program in INCULS is very important for foreign students since it equips students with the ability to use and understand Indonesian in various contexts and culture. The score of the reaction can always be improved if some aspects are taken into consideration. The content of the course should be improved. The content should be valid, significant, interested, learnable and consistent with social realities, and utility. The course should be separated into two levels. First level is for general language using ability; that is the ability to use Indonesian in everyday life communication. Finally, the language should be focused on each subject that students will be doing in their master program. The objectives of each lesson must be stated clearly. The students will be more motivated to learn if they know real objectives and the benefit of learning. Instructors should plan their lessons well and present them attractively and effectively to the class. High satisfaction and learning quality will be improved if all those mentioned factors are considered. In contrast, if all the negative aspects are not taken into consideration, the learning situation might be worse for the next program.
I would like to express my gratitude to all those who gave me the possibility to complete this evaluation study.
I am deeply indebted to Prof. Kumaidi who gave so freely of his time with valuable comments and support to me during the study in Indonesia. His stimulating suggestions and encouragement helped me in all the time of research for and writing of this evaluation study. His priceless help and advice contribute tremendously to my educational achievements.
I want to thank Dr. Ida Rochani Adi for giving me permission to conduct this study at INCULS.
Students, friends, and staff at INCULS supported me in this evaluation study. I want to thank them for all their help, support, interest and valuable hints.
Brophy, J. (1987). On Motivating Students. Occasional Paper No. 101. East Lansing, Michigan: Institute for Research on Teaching, Michigan State University.
Ericksen, S. C. (1978). "The Lecture." Memo to the Faculty, no. 60. Ann Arbor: Center for Research on Teaching and Learning, University of Michigan.
Kirkpatrick, D. L. (1998). Evaluating Traning Programs: The Four Levels. San Francisco. Berrect-Koehler Publishers, Inc.
McPartland & Braddock. (2000). A Conceptual Framework on Learning Environments and Student Motivation for Language Minority and Other Underserved Populations. Available at (Access: 11 May 2007)
Raffini, J. (1993). Winners Without Losers: Structures And Strategies For Increasing Student Motivation To Learn. Boston: Allyn and Bacon
Stipek, D. (1988). Motivation To Learn: From Theory To Practice. Englewood Cliffs, New Jersey: Prentice Hall.