Students’ perception and scores in Paediatrics end-of-clerkship and final professional Objective Structured Clinical Examination (OSCE): A comparative study

Objectives: This study aims to understand students’ perception of the usefulness of end-of-clerkship (EOC) as compared to professional exam Objective Structured Clinical Examinations (OSCE) and to compare student performance in both examinations. Methods: We conducted a cross-sectional study of medical students who were studying in the final year at Shifa College of Medicine, Shifa Tameer-e-Millat University, and appeared in both end-of-clerkship and final professional exams. The study was conducted in October 2019. The data was collected through a self-constructed questionnaire. The scores of all participants were also recorded. The data obtained was analyzed on IBM’s statistical package for the social sciences (SPSS) version 23 (IBM, Armonk, NY). Descriptive statistics were used for qualitative variables. Mean and standard deviation was calculated for quantitative variables. Chi-square test was also applied to assess for significant differences. Results: Out of 115 participants, 57 (49.6%) were males and 58 (50.4%) were females. Most of the participants (75.7%) agreed that both OSCEs were well-structured. Participants found that both the exams give a good cross-section of paediatrics and allowed them to learn something new (p=0.00). 45% of participants perceived that the end-of-clerkship OSCE exam helped in orienting them for the final professional exam by identifying their weaknesses. Conclusion: Although the EOC OSCE served as a good preparatory exercise, it did not alleviate the stress levels. However, the results in both OSCEs were comparable and students were satisfied with the current weightage of marks distribution.


INTRODUCTION
The assessment of a student's clinical competence is of paramount importance, and there are several means of evaluating student performance in medical examinations. The Objective Structured Clinical Examination (OSCE) is an approach to assess students in aspects of clinical competence and is evaluated in a comprehensive, consistent and structured manner, with close attention to the objectivity of the process. 1 This method has become a standard and universal format of assessment with good reliability and validity. 2 The OSCE is superior to the oral clinical examination because it assesses a broad area of competency, resulting in better reliability and validity. 3 The OSCE has demonstrated validity and reliability for assessing medical trainees' clinical skills in multiple disciplines. 3,4 Literature supports the use of OSCEs to evaluate clinical knowledge and competence, professional judgment, problem-solving skills, and interpersonal and communication skills. [5][6][7] The combination of OSCE with standardized board examinations has the potential to become the gold standard for measuring physician competence. 8 However, on the other hand conducting an OSCE needs a dedicated team, time for preparing the exam, logistic support, and financially more costly as compared to other means of assessment.
OSCE was introduced in 1975 and first described as an assessment format in Paediatrics (Child Health) in 1980. 9,10 since its inception, the OSCE has been increasingly used to provide formative and summative assessment in various medical disciplines worldwide, including non-clinical disciplines.
At Shifa College of Medicine, OSCE is being used as an assessment tool since 2009. We use OSCE as part of assessment in both end of clerkship (continuous assessment) and final professional Bachelor of Medicine, Bachelor of Surgery (MBBS) exit exams in all clinical subjects. In Paediatrics, students appear in end-of-clerkship (EOC) OSCE after eight weeks of structured clinical clerkship. End-of-clerkship assessments contribute 40% marks to final professional assessment. The major difference between the end-of-clerkship and final professional exam is the presence of external examiners in the later exam. Apart from this, the format, number of stations, and a variety of cases, timing at each station are the same. A table of specification is followed in both OSCEs to increase its reliability.
Despite OSCEs being a good tool for assessment, studies have not been done as to how students perceive the significance of the End-of-Clerkship (EOC) exam in preparing them for the final professional exam. As a part of the learner-based teaching process, we aim to understand students' perception on usefulness of end-of clerkship as compared to professional OSCEs, particularly concerning satisfaction about the weightage of both exams and to compare their performance in both examinations in an attempt to determine if stress levels are managed differently for both exams. The study will highlight areas of improvement and feedback from students to further improve the assessment process.

METHODS
This is a cross-sectional analytical study. The study included 115 final year students who appeared in the final professional Paediatrics exam held by Shifa College of Medicine, Shifa Tameer-e-Millat University. The study was conducted in October 2019. Students had already taken their End of Clerkship Paediatric exam. The study was approved by the IRB of Shifa International Hospital (IRB#256-746-2019). Informed consent was taken from students. Data was collected on a structured questionnaire.
The questionnaire comprised of questions regarding the structure and attributes of both end of clerkship and final professional OSCE. It also consisted of statements such as fairness of examination, weightage of the end-of-clerkship OSCE, gender, ethnicity, attitude of examiners, adequacy of time. The participants were given a 5-point Likert scale varying from 'Strongly Disagree' to 'Strongly Agree' to assess the degree to which the aforementioned statements had a role in affecting the OSCE scores and structure as a whole.
Participants were also instructed to comment upon stress levels, curriculum, and usefulness of end-of-clerkship OSCE experience in helping and stimulating them for better performance in final professional OSCE. Marks obtained by students in both OSCEs were also compared.
The final professional OSCE exam continued for four days which was also the duration of the study. 117 students were divided into four groups, each group comprising of 29 or 30 students each. On each day, groups were further divided into subgroups of 12 to 13 students each, and OSCE was run in two circuits. Table of specifications is shown in Table-I. There were a total of 12 stations in both OSCEs. There were eight interactive and four static stations. The duration (including transit time) for each station was five minutes. There were three observed stations for 'Long Case' and two observed stations for 'Short Case'. The total score for both OSCEs was 100 points each. A similar format was followed for end-of-clerkship OSCE for each group that attended the Pediatric rotation. The table of specifications was the same.

Students' Perception about OSCEs
At the end of the OSCE, a questionnaire was distributed to all the students. These questionnaires were filled out in the presence of a preceptor to ensure that all queries be addressed and responses are recorded appropriately. The data obtained was analyzed on IBM's statistical package for the social sciences (SPSS) version 23 (IBM, Armonk, NY). Descriptive statistics  were used to analyze and describe the data. Frequencies and percentages were calculated for qualitative variables like gender. Mean and standard deviation (SD) were calculated for quantitative variables like scores.

RESULTS
One hundred fifteen (98.3%) students completed the questionnaire and were included as participants of the study. There were 57(49.6%) male participants and 58(50.4%) female participants. The students had a mean score of 66.03±10.18 in end-of-clerkship OSCE and 64.73±8.00 in final professional OSCE. The minimum score obtained was 31.60 in end-of-clerkship OSCE and 43. 19 in final professional OSCE. The maximum score obtained was 84.16 in end-of-clerkship OSCE and 82.80 in final professional OSCE.
Most of the students (65.0% to 75.7%) commented positively about the structure of both end-of-clerkship and final professional OSCE. Participants found that both the exams gave a good cross-section of paediatrics and posed an opportunity to learn something new. A similar set of statements was used in the questionnaire for final professional OSCE. These results have been further explained in Table-II and III. We also compared the differences in percentage when similar statements were brought into consideration. Chi-square was used to test if these differences were significant. P-value less than 0.05 was considered significant. This is shown in Table-IV.

DISCUSSION
OSCE is considered an excellent assessment tool for clinical skills. 11 In our set up it's part of our continuous assessment and final professional exit exam and accepted well by students. Students agreed that both OSCEs provided a good cross-section of Pediatrics cases (p=0.00) and provided real-life scenarios. Students agreed that both OSCEs were well structured (p=0.03). Gender and ethnicity did not affect the results (P=0.00). This is in contrast to other studies done internationally where students were neutral that OSCE served the purpose of motivation to learn new concepts. 12 Regarding the reliability of scoring, students agreed that the results were reflective of their performance (p=0.00).
When a comparison of both OSCEs was done the study identified that 45% of participants perceived that end-of-clerkship Pediatrics OSCE helped them prepare better for their professional  exams. 13 It served a good purpose in identifying their areas of weaknesses (p=0.00). This has been advised in other studies on the significance of OSCE in clinical exams that orientation to this exam format is important for students to perform better. 14 However, the majority (53.0%) of the participants agreed that more weightage should be given to the final professional exam. This is attributed to the fact that 61.7% of the participants took the final professional exam more seriously than the end-of-clerkship exam. Furthermore, EOC assessment serves as a good orientation exercise to help them be more prepared in Final professional OSCE. 15 Participants (76.5%) found the professional exam to be more intimidating and stressful (p=0.00) 16 . Prior experience in terms of EOC OSCE did not seem to alleviate this stress. The same trends have been seen in other studies that anxiety and stress levels run high in Professional exam. 17 In our study, the presence of external examiners can also be a factor contributing to their stress levels. 18 Students believed that examiners were stricter in the final professional exam (46.1%) as compared to the EOC exam (14.8%). 19 Comparing the scores of both OSCEs the minimum marks obtained in FP OSCE were higher than EOC highlighting improved performance (31.6 versus 43.9).This study highlighted the fact that EOC OSCE is perceived by students as a good tool for orienting them to final professional OSCE. 20 It also helps them identify areas of improvement and work on their clinical skills as observed by improved minimum scores in Final professional OSCE. Students (59%) agreed that marks in internal assessment also served as a stimulus to perform better in professional exams. It focuses on customized learning for each individual. Students contended with the 40 % weightage of EOC exam results contributing to the final exam scores (67.8%).

Limitation of the study:
This study was performed in one discipline only and the weightage of continuous assessment can be different from other institutions. These examinations should be implemented in other institutes and not only medical schools so that an organized framework is brought to the learner's mind when such situations arrive in the practical and professional lives of the learners.

CONCLUSIONS
Although the EOC OSCE served as a good preparatory exercise, it did not alleviate the stress levels. However, the results in both OSCEs were comparable and students were satisfied with the current weightage of marks distribution.