ACEM Examination Bulletin – April 2018

The College held an additional sitting of the Fellowship Clinical Examination (OSCE) in July 2017, for the second year running, in an attempt to accommodate the increasing number of trainees approaching completion of the training program and needing to complete that examination. Unfortunately an excess of applications was received for the October/November sitting of the OSCE and a number of candidates were forced to delay their sitting until 2018.

In recognition of the challenges caused by such a delay, the Examinations Subcommittee reviewed and subsequently determined to redesign the structure of the OSCE from a three-day 15 station OSCE to a two-day 12 station OSCE, to be implemented from OSCE 2018.1 in May 2018. The new structure has the capacity to offer 50% more places to candidates, if required, within a six-day OSCE schedule.

The new structure also provided an opportunity to revisit the format of the OSCE and to make modifications to the examination that were reported in the Examination Bulletin of December 2017.

As reported previously the new model OSCE will comprise 12 stations and each candidate will sit the 12 stations over two consecutive days. All stations will be of 11 minutes duration which includes four minutes’ reading time.

To pass the OSCE, candidates will need to attain the passing score for the examination and achieve a score that is at least ‘Just at Standard’ in a minimum of eight of the twelve stations. A ‘Just at Standard’ candidate is one who has demonstrated a minimum or sufficient level of competence to ensure safe practice and meet the core needs of the patient.

A new station type has been introduced, the Standardised Case-Based Discussion (SCBD) station and the former double team-based simulation station will be presented as a single station with a modified format.

Within the new model OSCE, up to three stations will be SCBD stations and at least one of the twelve stations will be a single team-based simulation station. Details of the new model OSCE and other documents to support trainees preparing to sit the OSCE can be found in the Resources for Fellowship Examination on the eLearning website. These now include:

• Information about the Fellowship Examination (OSCE)
• Preparing for the Fellowship Examination (OSCE)
• Standardised Case Based Discussions (SCBD) - General Instructions and example
• OSCE Single Simulation Stations (New)
• ACEM Glossary of Terms (New)
• OSCE Domain Criteria (New)

Fellows of the College have recently developed a series of videos of mock OSCE stations for the Life in the Fast Lane website. Members of the Examinations Subcommittee have reviewed the videos and consider them to show good models of stations, done well by candidates. Each OSCE includes a realistic scenario, the domains being assessed, an example of a framework to address the scenario and hints throughout.

Trainees may find these to be a very helpful resource to familiarise themselves with a model of the OSCE candidate-examiner interaction and a good candidate performance. Of note, these videos may not necessarily represent or reflect the exact format or content of the ACEM OSCE stations and should be considered as examples only. The videos may be viewed via the link to Life in the Fast Lane.

The Fellowship Clinical Examination (OSCE)

The following enhancements have been implemented during 2017 and 2018 to date:

• A minimum of two examiners, who mark independently, are now appointed to each OSCE station to ensure candidates have two data points for each OSCE encounter.
• The examiner briefing prior to the examination now includes a section on awareness of potential unconscious bias, to ensure all trainees continue to receive assessment that is fair and objective, irrespective of their personal attributes.
• Enhanced rehearsal, trialling and workshopping of stations occurs on each examination day to standardise the assessment of individual examiners.
• An examiner calibration session is held after the first round of candidates for each station for the same purpose and to ensure the results given are consistent for all candidates.
• Written feedback is provided to examiners on their marking patterns in comparison to their co-marker and the examiner cohort for the station they examined.
• Comprehensive examination reports are produced that include a summary of the results, statistical details and information about the stations including the domains assessed and the associated Curriculum Framework criteria, expectations of candidates’ performance and the percentage of candidates who reached the station cut score. Examiners’ overall comments about the standard of candidate performance in each station are also included.
• During the examination, live streaming of stations is used to enhance the training provided to new examiners.
• The College has committed to publishing two OSCE stations, each year from 2018.
• Examiners are now required to provide feedback on the mark sheet for each candidate whose performance at a station was rated ‘Just at’, ‘Below’ or ‘Well Below’ standard and to record any observed serious errors, which if made in actual practice, would have the potential to cause significant harm to the patient.
• Written feedback from examiners as outlined above is provided to all candidates who have been unsuccessful in the examination.
• All candidates whether successful or unsuccessful, receive feedback where an observed serious error was recorded, to inform their subsequent practice and in consideration of patient safety.

Fellowship Written Examinations

During 2017 the Fellowship SAQ paper was consolidated to contain a maximum of 360 marks. New sample SAQs were added to the resources for trainees on the website.

A ‘Marking Centre’ approach was introduced for examiners marking the SAQ papers and this has continued in 2018. For the Fellowship Written SAQ examination 2018.1, Marking Centres were held in two locations on 9 March 2018, Melbourne and Sydney which all examiners attended, completing their marking on the day. To enhance the reliability of the marking and fairness to candidates all candidate SAQ papers are now double marked by two examiners who mark independently, following an initial calibration stage.

It is planned that in the future the Fellowship SAQ will be conducted as an online examination as per the SCQ paper, and work on the development of the online SAQ examination platform is underway. The College hopes to offer some candidates the opportunity to volunteer to undertake their SAQ examination online for the examination of 2018.2.

In the first instance the online examination will be offered in Melbourne and Brisbane only, as a trial, however candidates from other regions, wishing to take the 2018.2 examination online, are welcome to apply to sit the examination in either of those cities. Further information about the platform development will be provided in the next few months, including the opportunity to undertake a practice online examination before volunteering for the examination in August.

On completion of the platform, all candidates will be required to sit the FE Written SAQ online. At this stage it is envisaged this will occur from the SAQ 2020.1 examination. For candidates who feel unable to sit the SAQ online, an application in line with the College’s Exceptional Circumstances and Special Consideration Policy, may enable alternative provisions to be approved.

Work on a new question bank for the Fellowship SCQ questions has commenced and a project to review and code all questions to enhance their quality and enable blueprinting to the ACEM Curriculum Framework and the Study Guide for the Fellowship Examinations, is underway. MCQ Writing and Review Workshops are currently being offered in all regions in a further move to improve both the quantity and the quality of the items.

The standard setting process used for both the Fellowship and the Primary Written examinations was refined in 2017. A document summarising the standard setting processes for all examinations will be available from the Resources section of eLearning in due course.

The Primary Examinations

A new model for the scoring of the Primary Clinical Viva examination was considered by the Examinations Subcommittee and an associated score sheet was trialled at the September 2017 examination. Following approval by the Council of Education the new marking system was introduced to the Viva of 2018.1.

The revised scoring system requires examiners to mark each of the five components of a Viva separately, with the candidates receiving an aggregated score rather than a global score. The five components are pharmacology, pathology, physiology, anatomy and the clinical building blocks.

Prior to the Primary and Fellowship Written examinations of 2018.1, trainees were informed of contingency plans regarding online SCQ examinations, in the light of reported systems failures at another Specialist Medical College. These plans include the following measures:

• The ACEM IT team carries out stringent testing prior to each examination to try to ensure every eventuality is covered.
• ACEM technical support is available for the entirety of the examination day.
• ACEM has three servers in different locations at the user end of the examination site which help to balance the load and provide backup in the unlikely case of a server failing.
• At the back end, two servers support each other to store the examination answers. In the unlikely case of a server at the back end failing, the other server will copy all data and take over data storage within 30 seconds.
• From the FE Written of 2018.1 an Assessment Team or Education Unit staff member will be in attendance at each of the examination centres, in addition to FACEM and Clifton’s invigilators, to oversee processes and advise in case of an incident occurring.
• A paper copy of the examination will be available for use in the unlikely case of an issue occurring with the online system such as a power failure at an examination centre.

It is proposed that the same contingency arrangements will be in place for the online Fellowship SAQ examination that is currently in development.

Changes have been made to the way trainees are informed of their examination results. Trainees now receive their examination results individually through the member portal rather than these being posted publicly by their College ID number as previously. In addition, all trainees are now issued with a randomised examinations ID number that is applied to all examinations to further enhance candidate privacy.

Measures to support trainees on receiving their OSCE result have been implemented and in their feedback letter trainees are encouraged to discuss their examination results with their DEMT or another FACEM/mentor.

Trainees are strongly encouraged to reflect on their individual performance, identify those areas needing further development and to discuss their results with their DEMT, and/or a mentor who is supporting them in their training.

From April 2018, DEMTs will also be informed of their trainees’ results so that they may offer support or assistance where needed. Where possible, examinations results are issued midweek, where more support is thought to be generally available.

As previously advised the examinations calendar will change in 2019. In effect, the Primary Viva and the Fellowship OSCE will change places with the Viva taking place in early May and early November and the OSCE taking place in mid-March and mid-September. The Primary Written Examinations will remain in February and August, but the Fellowship Written Examinations will move from late February to mid-May and from August to mid-November.

These changes will allow Provisional Trainees three months to prepare for their Viva if they choose to sit the subsequent examination rather than the current six weeks. Advanced Trainees will have four months to prepare for their OSCE if they are successful in the Fellowship Written Examination, rather than the current three months.

These changes have been made with full consideration of the requirements of the FACEM Training Program.

Examinations candidates are advised that changes to the following policies were approved by COE in March 2018.

Exceptional Circumstances and Special Consideration Policy

Reconsideration, Review and Appeals Policy

A plan for the implementation of required training in three Assessing Cultural Competence modules by COE entities including all members of COE committees and the Examinations Subcommittee, the Court of Examiners, trainees and DEMTs is under development. A timeline for module completion will be published after its approval by COE.

All College members are encouraged to undertake the Assessing Cultural Competence modules that are currently available on the ACEM eLearning site at their earliest convenience, particularly if they will be involved in examination activities in 2018.

As previously reported, trainees can expect to be assessed on these areas as per the other domains of the ACEM Curriculum Framework. Trainees, DEMTs, supervisors and examiners should reference the Fellowship Examination Recommended References and Study Guide for the Domain of Medical Expertise, which can be found on the ACEM eLearning site.

In response to a call to FACEMs for Expressions of Interest to join the Court of Examiners in 2018, a further twelve new examiners were appointed in March 2018:

We welcome these new examiners to the Court and appreciate their interest in contributing to the work of the College in this way.

The new examiners will undertake orientation and training activities, commencing at the May 2018 OSCE. The College will continue to monitor, recruit and train new examiners, in order to be able to meet candidate numbers and assist with sharing the time and workload commitment that is involved in being an examiner.

The College sincerely thanks the following Fellows, on their retirement from the Court of Examiners, for their longstanding commitment as examiners: Dr Simon Brown and Dr Michael Ragg.