Portfolio based assessment can improve quality of exit Post graduate examinations in Orthopedics?

Print

Point of View
Portfolio based assessment can
improve quality of 
exit Post graduate
examinations in Orthopedics?
The suggested changes are required in the
whole system & all disciplines

Prof. Faisal Nazeer Hussain*

Pakistan in a short span of over seven decades has set in place a very robust post graduate training system. It has made us self sufficient and we have contributed a lot of specialist work force abroad as well. We can all feel proud of our achievements. At present post graduate training programs of FCPS and MS have been designed to follow specific curricular requirements to ensure acquisition of knowledge and skills suitable for following a successful career as Orthopedic Surgeon. Both systems tend to converge on similar outcomes though apparently both lay down different road maps.

FCPS training is more elaborate in describing the structured steps of training to be followed by the learner and the teacher. An attempt at formative assessment has been formalized in the program through use of e-log book. A dissertation or a thesis has to be submitted in partial fulfilment of requirements to appear in the exit final examination at the end of a five year training period. The objective being to give some practice in medical writing. In the recent Covid environment unlike the past the clinical examination are being curtailed in detail when real patients were avoided to be replaced by computer assisted clinical scenarios. Though unusual but the change improved the overall passing percentage in results markedly. World over the post graduate medical examinations are known to result in high failure rate.1-4


Prof. Faisal Nazeer Hussain

This points to variation in the quality of examination that can occur when a change is introduced abruptly. Only repeated use of such examination material can provide definitive proof of sensitivity of the assessment such as item difficulty, validity and discrimination index. Although post graduate training systems have been designed to give a comprehensive picture of the trainee at the time of exit from the program but such major variations can be avoided if the formative part of the assessment is formally included in the final assessment as per a pre-determined weightage.5,6 In our case during training achievements are not considered at the time of final exit examination. The system evaluates them and does not bring the results forwards for peer review at the time of certification examination. We must ensure that the final results of exit examinations fully represent all aspects of the training not just one day performance.

A similar situation can be interesting to learn from. American College of Orthopedic Surgeons introduced recertification for every board certified surgeon after every 10 years.5-7 The process included fulfilment of a certain amount of Continued Medical Education, some recommended readings and attending some peer meetings. The final recertification examination is modelled around the primary certification examination based on multiple choice questions. Critics of the “assessment exercise” like Joseph Bernstien say that the examination is too general and disjoint from practice based issues, costly, dependent upon old material (probably he means the same material that he faced in his primary certification examination).6,7 He points out that now he is a sports surgeon and does not really need to cram material about degenerative disorders. He contends that though the Board (ABOS) has taken this step to start a self re-assessment effort to increase the acceptability of the diploma which serves as more or less a surrogate of a regulatory registration in a way. David F. Martin and Peter M. Murray defend the ABOS reiterating the use of Profile based assessment where the practicing surgeon chooses the future content of the Maintenance of Certification (MOC) based on his professional interests/practice.6 They say that MOC can be acquired through a Computer-Based Recertification Examination or a practice based Oral Examination. The American Board of Surgery (ABS) is once per year for Qualifying Examination (QE) and five times per year for Certifying Examination (CE).7 These eliminate the general orthopedic questions, add nine subspecialty options basing the questions exclusively on that subspecialty. A contactless oral examination has been developed. The recent introduction of ABOS Web-based Longitudinal Assessment which is a knowledge-assessment option recently, submission of Case Lists, complete CME and Self-Assessment Examination activities makes a portfolio which undergoes peer review.5 The decision to maintain certification will be culmination of both formative and summative assessments. The decision for retention of the certificate will always be a summative decision for MOC participation by the peers after analysis of all the summative and formative assessments. Similarly portfolio of the post graduate trainee candidate should be made part of the final assessment of the candidate. This will eliminate large scale variations in the examination standards.

UK introduced Modernizing Medical Careers in 2007 asking every aspiring candidate to undergo two years of foundation training (FY1&2), four years of specialty training(ST3-6) in a structured manner and two years of Subspecialty training(ST7&8). These eight years of planned structured training produces subspecialty trained surgeons.5 The process gives the candidate a completion certificate of surgical training(CCST). In 2009 the working hours for trainees were reduced to 56 from 80 per week raising eye brows regarding concerns regarding adequacy of training imparted in the decreased work hours.6,9 Like CPSP e-logbook trainee progress in UK, Australia & US is continually monitored by maintaining an operative logbook experience, research experience, and formal competency assessments records of various procedures.5 The US residency programs employ an annual standardized national examination based on MCQs and without patient contact in exit examinations. Maninder SK et al have failed to find much difference in clinical performance between those who are Board Certified or non-board certified at the end of their training.9 James H Henderson and his colleagues have concluded very fairly that the predictors of passing the ABOS part-I & part-II examination in an earlier attempt lies in the scores obtained and maintained during the training period evaluations.5 This points again to the importance of the formative period of training which is hardly given any weightage in the final assessment in most exit examinations. It becomes all the more important in our system where we get to train so many at the same time in our over-crowded wards that the mentor cannot fulfill his supervisory duties to his fullest desire and requirements of the system.

There are still systems in the world where even lesser forms of exit examinations is being patronized. The European Board in Orthopedic Surgery leans heavily on end of training summative assessment. Candidates from European Union (EU) countries or from non-EU countries who have completed or neared completion of their final orthopaedic training can sit the exam if they can provide reference from their program director or clinical supervisors regarding their clinical orthopaedic knowledge and their level of training.7 It comprises of two parts with part-I being MCQ based and part-II comprising of clinical component. None of the parts uses real patients during examination. Joint Surgical Colleges Fellowship Examinations’ (JSCFE) in Ireland is a similar qualification which is not backed by a structured training program is not register able as is FRCS O&T.2 West African College of Surgeons (WACS) suffers from a very low pass rate in its post graduate examinations-another summative assessment.4 This is yet another weak counter argument to the use of formative assessment as part of final decision to impart the honor of a professional fellowship as a qualification.

Our post graduate training system is educating our future specialists and probably the numbers of trainees is among the largest groups in the world in some ways. The end product quality must be ensured at all costs. We must see the system progressively and go a step further ahead in order to stay abreast of what the world is doing. Our local scene is waiting for a long overdue review. A pre-part II peer review of the training quality, publications, and trainings completed and workshops attended can help a set of examiners suggest if the training quality is sufficient enough for the candidate to appear for exit examinations. It can be before final examination paper based review, e-communication or interviews by a panel of experts from senior faculty so that we can decrease the load of candidates appearing in the final examination of post graduate examinations. It will not only help monitor the quality of training being given by the parent units and will decrease the pressure on the examining bodies to base imparting of resultant qualification upon an exit examination where the examiners and the candidates are burdened by possibilities of unforced errors of judgement leading to a variable passing rate. The spirit will be to make the trainer share the burden of maintaining quality of training so that having completed the training also ensures maintenance of a certain quality. The question remains are we ready to upgrade to the next step and move on to a portfolio based final assessment for exit post graduate examinations where errors of assessments within the final stages will not affect the overall results of a training completed in five years. Quality should prevail especially of training tenure.

At present our sole dependence upon an exit examination is indirectly absolving our supervisors from maintaining a reliable quality of training. A relevant example of a portfolio based assessment can be like the Yearly Appraisal system adopted by General Medical Council. Like many medical licensing authorities GMC also desires that every medical practitioner maintains a minimum standard of staying abreast of the latest techniques. To ensure this, standards had long been laid down requiring a minimum level of learning a physician (CME hours) must maintain in order for GMC to retain its name on the register. Continuous Medical Education activities were developed in most countries based on the study hours earned. In 2011 the system of continuous formative assessment by trained appraisers, responsible officers was introduced by GMC. 10 Those keen to become appraisers were trained and inducted into the system. Purpose was to monitor the physicians self guided activities at regular intervals by trained appraisers who would submit a report to GMC which will make a decision every five years to retain the physicians name on the general or specialist register when the time for renewal of license comes. Those who review the application have the opportunity to review the results of formative assessments submitted each year by the responsible persons. GMC has made ROs (Responsible Officers) in each scenario of the medical practice e.g Chief of service or Hospital Director where in house medical practice is being supervised who sees all the five years record and enters the practitioner into the GMC register if he finds the candidate fit to maintain the registration or may defer the renewal asking for improving or repeating certain activities. Similarly the trainees also get monitored as is the training program (Post graduate training institutes), the supervisors and the GMC run as a parallel arrangement.11 Conference of Post Graduate Deans of UK-COPMeD in 2016 outlined the ways an RO is to evaluate the performance of a post graduate doctor in training. 12 They say” Any significant or recurrent breaches of, or failures to meet, expectations of professionalism or conduct as outlined by GMC Good Medical Practice guidelines should be considered a cause for concern about a doctor”. 11 In this way the supervisor in Pakistan can be assisted and supervised both by the appraiser or the Training Mentor who will ensure that the training standards are maintained at the highest level possible. The yearly reports of the appraisers along with e-log book or thesis/dissertation duly endorsed by the supervisor can be reviewed by a panel of examiners (surrogate of a an RO) before the final day of exit examination given formative assessments a weighted scoring so that the candidate is sent to the final examination with an advantage if he deserves it. Mishap failures will be prevented and the final product will have a more predictable quality. We can learn from all these experiences of the medical fraternity globally to improve our system. Pakistan Orthopedics Association, Universities, Higher Education Commission and Pakistan Medical Commission can sit together to set rules for maintaining a continuous system of monitoring which should supervise the training period and provide the trainee & the trainer with a regular structured feedbacks (maybe yearly-like mentoring). The weightage of formative and summative assessments can be allocated by examining bodies so that dependence upon an exit examination can be reduced and the accuracy of the examination be improved. No improvement is possible until we are merciless in reviewing our faults and generous in appreciating our achievements.

REFERENCES

  1. Saeed Farooq: Failure rate in Postgraduate Medical Examinations - Sign of a widespread Disease?; Pak J. Med Sc.:2005 May;55(5):214-7. PMID: 15960290
  2. Joint Surgical Colleges Fellowship Examinations’ (JSCFE): https://www.jscfe.co.uk/Content/content.aspx
  3. B Caeser, L David:: Levelling the playing Field: Matt Freudmann: Series editor: Ann R Coll Surg Engl (Suppl) 2008: 90:344–345
  4. Jonathan L Ajah: Low pass-rate in postgraduate surgical examination in Nigeria and its contribution to the low surgeon workforce in the country; a review article: SICOT-J 2018, 4, 36 ,The Author, published by EDP Sciences, 2018 https://doi.org/10.1051/sicotj/2018008
  5. Joseph Bernstein: Not the Last Word: The Time to Fix ABOS Recertification Has Arrived. Clin Orthop Relat Res. 2018 Oct; 476(10): 1928–1930.doi: 10.1097/CORR.0000000000000455
  6. Shakir S, Aun HM, Asghar A: A Brief Comparison of Orthopedic Training in English Speaking Countries: Ann R Coll Surg Engl. 2009 Apr; 91(3): 226–231. doi: 10.1308/003588409X35930
  7. Daniel Ryan: Helpful experience with the European Board of Orthopaedics and Traumatology; November 29, 2018 9 min read: examhttps://www.healio.com/news/orthopedics/20181127/helpful-experience-with-the-european-board-of-orthopaedics-and-traumatology-exam.
  8. Mininder S. Kocher, Laura Dichtel, James R. Kasser, Mark C. Gebhardt, Jeffrey N. Katz: Orthopedic Board Certification and Physician Performance: An Analysis of Medical Malpractice, Hospital Disciplinary Action, and State Medical Board Disciplinary Action Rates: Am J Orthop. 2008;37(2):73-75.
  9. James H. Herndon, Bassan J. Allan, George Dyer, Andrew Jawa, David Zurakowski: Predictors of success in ABOS: JBJS Volume 467, Number 9, September 2009. p 2436-2445
  10. https://www.copmed.org.uk/images/docs/revalidation/Guidance_on_supporting_information_for_revalidation_17th_November_2020.pdf
  11. https://www.gmc-uk.org/-/media/gmc-site-images/registration-and-licensing/responsible-officer-protocol_pdf-56096180.pdf?la=en&hash=720839113F754EFFD3D2F7C076A720D2728BA802
  12. The Gold Guide 8th edition incorporating the ‘Purple Guide’ for foundation training version: 2019 Version: GG8 – 31 March 2020. https://www.copmed.org.uk/images/docs/gold_guide_8th_edition/Gold_Guide_8th_Edition_March_2020.pdf

* The author is Professor of Orthopaedics Surgery at Avicenna Medical College, Lahore. Pakistan.

© Professional Medical Publications. All rights reserved.