Item analysis and optimizing multiple-choice questions for a viable question bank in ophthalmology: A cross-sectional study
It was a prospective, cross-sectional study involving MBBS students taking Ophthalmology examination. Forty MCQs were part of the final formative assessment, which included a 3-h Question paper for 100 marks with long essays, short essays, short answer questions, and MCQs. After taking permission from the Dean of the college and consent from study participants, 120 students were assessed in Ophthalmology. The MCQs for the same were chosen randomly and were prevalidated by two departmental colleagues. The 40 MCQ items of 'single response type' having a stem, a key and three distractors were chosen. Each correct response (key) was awarded “half mark,” an incorrect response or un-attempted Item was awarded “Zero mark.” Time allotted for attempting the MCQs was initial 30 minutes. Strict invigilation was carried out to avoid malpractices. The faculties have evaluated the MCQs with the correct keys. After scoring, students were ranked in order of merit using Microsoft Excel 2010. Bottom third (40) of low achievers (L) and top third (40) of high achievers (H) were used for item analysis to serve the twin purpose of having groups large enough to be representative and different enough to be meaningful. Each Item was analyzed for difficulty index (DIF I), discrimination index (DI), and distractor efficiency (DE) using the standard formulae. DIF I is the percentage of students who answered the Item correctly and given by DIF I = [(H + L)/N] × 100. DI is a measure of the effectiveness of an item in discriminating between L from H and is given by DI = 2 × [(H-L)/N], where H is the number of students answering the Item correctly in the high achieving group, L is the number of students answering the Item correctly in the low achieving group, and N is the total number of students in the two groups (including non-responders). Distractors which were chosen by less than 5% of students were considered as a nonfunctional distractor (NFD). DE was determined for each Item based on the number of NFDs in it and ranged from zero to 100%. DE of an item was considered as 100% if there were no NFD and 66.6%, 33.3% and 0% if it contained 1, 2, or 3 NFDs, respectively. The relationship between the DIF I and DI were determined by Pearson correlation coefficient (r) using an online calculator ( https://www.socscistatistics.com/tests/pearson/ ). P value of <0.05 was considered to be statistically significant. Items were categorized according to their DIF I, DI, and DE with simple proportions, mean, standard deviations, and correlation. Data were interpreted, as shown in . The range of values for DIF I is 0%–100%. The range of values for DI is –1.0 to +1.0. The DI becomes negative if more students in the L group answer correctly than those in the H group. 

The score obtained by the study group ranged from 6.5 to 20, with a mean of 13.53 ± 2.62. Mean of DIF I, DI, and DE was 53.22 ± 22.44, 0.26 ± 0.16 and 78.32 ± 32.11, respectively. The distribution of DIF I and DI of the items and their corresponding DE with actions proposed are shown in Tables and , respectively. The scattered diagram  shows the relationship between the DIF I and DI. DI was poor for both easy and difficult items. Pearson correlation between DI and DIF I was small and negative ( r = –0.207) and not significant at P < 0.05 ( P = 0.199). The relationship of NFDs with mean DIF I and DI of items are shown in . Item analysis showed 20 items (50%) to be defective, and the subject expert safely added the remaining 20 items to the question bank with an item card for each Item. Of 20 defective items, 17 items were found to have bad stem or distractors, and these were optimized and added to the question bank. It included five easy items with normal DI, five easy items with poor DI, one easy Item with negative DI, two difficult items with normal DI, one difficult item poor DI, two items with acceptable difficulty but poor DI, 1 item with good DI and DIF I but two defective NFD. One difficult Item and one Item with acceptable difficulty having poor DI did not have any flaws and added to the question bank without alteration. One difficult Item with negative DI was dropped because there was one more question, based on the same data in the same paper with a hint to the correct answer. Feedback was given to the faculty on all difficult items for corrective measures in teaching and framing the Item. All optimized items were appropriately tagged via the item card for future reassessment. 

The Item analysis should be regularly carried out for creating a good question bank updating recent advances so that MCQs can be used in the evaluation of the cognitive skills of medical students effectively. The MCQs will provide feedback to the teachers on their educational actions. Designing MCQ is a complicated and time-consuming process in a multidisciplinary and integrated curriculum. In this study, the mean DIF, DI, and DE were 53.22 ± 22.43, 0.26 ± 0.15, 78.31 ± 32.11, respectively, which were in the acceptable range, and a few other studies reported similar results. In this study, 60% of items were of acceptable difficulty (DIF I = 30-70%), 27.5% items easy, and 12.5% difficult. Similar results were seen in a few of the studies. Nearly 80% of the items were in an acceptable range in three studies. In studies by Kheyami D (53.4%), Shenoy PJ (40%), Gajjar S (48%), and Rajkumar P (46.6%), it was less. In the last two studies, the acceptable range was defined as 30%–60%, unlike our and other (30%–70%) studies. In this study, 28 (70%) items had good DI (>0.2). Similar findings were seen in a few studies but was less compared to studies by Karelia BN (78%), Kaur M (86%), Shenoy PJ (80%), Pande SS (75%), Rao C (85%) and Mozaffer RH (88%). In our study, 15 (37.5%) items had one or more NFD which was similar to a study by Kaur M (38%) but less compared to study by Mozaffer RH (58%), Rajkumar P (43.3%) and Shenoy PJ (65%). In our study, of 120 distractors, 26 (21.6%) were non-functional, which is comparable to previous studies. More NFD was seen by Shenoy PJ (33.3%), but less NFD was observed by Gajjar S (11.3%) and Rao C (5%). Seven items with two or more NFD and four items with 1 NFD with poor DI/DIF I was optimized and added the question bank. Three items with 1 NFD had normal DI, DIF I, and distractors and were added to the question bank without alteration. One of Item with 2 NFD, had good DI and DIF I but distractors were defective and were optimized and added to the question bank. One of the easy Item with multiple writing flaws with a negative DI and 3 NFDs re-establishes the finding by Omer AA that flawed items affected high achievers more than low achievers. The defective Items identified by item analysis should be appropriately analyzed for flaws, and it should be optimized. These should not be blindly dropped; otherwise, some skills may be left out of the assessment. In early formative assessments with fewer topics to be studied and short term memory assessed, majority of students may answer the Item correctly (DIF I >70%) reducing the DI complimenting an efficient teaching-learning process. Decreased or negative DI can also occur with poor teaching-learning interaction with DIF I <30%. In this scenario, both high and low achievers answer the MCQs by guesswork. DI decreases when the Item becomes too easy or too difficult. This was noted in this study where maximum discrimination (0.4) was seen when the DIF I is 40-60%, and the relationship was dome-shaped as found in many studies. In our study, Pearson correlation between DI and DIF I was small and negative ( r = –0.207) and not significant at P < 0.05 ( P = 0.199). A study by Mitra et al. also showed a moderate negative correlation in their study ( r = –325, significant at P < 0.01). Karelia et al. , showed a small positive correlation ( r = 0.11, insignificant at P < 0.05). Two other studies showed a slight positive correlation significant at P < 0.01. In this study, items that were not acceptable due to poor DIF I, DI, or bad distractors were analyzed for writing flaws and optimized keeping in mind the guidelines. In this study, some of the Item writing flaws were unique to Ophthalmology. The presence of two conflicting terms in options which cannot co-exist, like mydriasis and miosis, lid retraction and ptosis, intumescent lens, and shrunken lens suggesting that one of them has to be chosen or excluded even to a low achiever was one of the writing flaws. Other flaws detected were All of the above as the key, the EXCEPT in the stem not capitalized and made bold, options used were not uniform, presence of abbreviations in the stem and options, the part of the word same in stem and options such as phacomorphic and phacoanaphylaxis. Few others being, questions related to controversial information in the books recommended for undergraduates and distractors which are unrelated and implausible. One of the easy Item (DIF I = 92.5%) with negative DI and 3 NFD which low achievers answered better was Bitot spots are seen in a) Vitamin A deficiency b) Dry eye c) Sjogren syndrome d) Lagophthalmos. It was optimized as 'Bitot spot can be seen in all the following EXCEPT a. Cirrhosis of the liver, b. Celiac disease, c. Dietary deficiency of vitamin A, d. Primary Sjogren syndrome' which tests the same cognitive domain in a better way. Defective Items need to be appropriately reconstructed; validated and feedback should be given to the faculties for corrective action. The New Graduate medical regulation 2019 released by the Medical Council of India has made it mandatory to include MCQs in the formative and summative assessment in a competency-based medical curriculum. We hope item analysis will serve as a helpful tool to generate question banks at departmental and university levels which will provide items with acceptable difficulty and discrimination indices. Our study has included only 40 MCQs. Periodical Item analysis with more MCQs is necessary for validation of viable question bank in important subjects like Ophthalmology. 

To conclude, Item analysis is a valuable tool in detecting poor MCQs, and optimizing them is a critical step. It will help us to identify poorly constructed items and give attention to optimize them to improve the quality of the question bank. With nearly half of the items detected to be less than optimum, efforts to optimize the stem and distractor is essential, without which the purpose of the assessment will be defeated. It also points toward the need to have more frequent faculty development programs in creating standard MCQs, effective pre-validation, and to have viable question bank in ophthalmology. Financial support and sponsorship Nil. Conflicts of interest There are no conflicts of interest. 

Nil. 

There are no conflicts of interest. 

