ACEM introduced Emergency Medicine Workplace-based Assessment (WBAs- (including CbD, DOPS, Mini-CEX and Shift Reports) in 2015. WBAs assist assessors in providing structured feedback to trainees and provide detailed information to DEMTs and Trainee Progression Review Panels about the development of trainees’ clinical skills, clinical reasoning and decision making. Since their introduction, WBAs have become an integral part of the FACEM Training Program with over 86,000 being completed since 2015. The College is committed to continuing to utilise WBAs as a valuable assessment tool.
As part of our quality improvement cycle, all aspects of the training program undergo regular evaluation. In early 2019, an evaluation of the WBA forms was initiated to determine whether any improvements could or should be made. The Council of Education set up a WBA Review Working Group to evaluate whether the forms were meeting the needs of Trainees, Fellows and Assessment panels.
The Working Group undertook a survey to gather feedback from members including FACEMs, Trainees, DEMTs and panel members. Overall, members were generally happy with the forms and process. However, there were several areas identified for improvement and some suggestions made. The feedback indicated that some assessors and trainees felt that the rating scale was unclear and at times difficult to apply. There was support for the complexity calculator and most agreed that the components for each tool were all relevant and necessary.
After considering the feedback, as well as current best practice and literature available, the Working Group revised the forms.
Following several rounds of consultation regarding the rating scale and the forms, the forms have been simplified, the rating scale revised, and some data collection fields removed. The forms were approved for implementation by the Council of Education in November 2019.
-
What is the purpose of these changes?
When discussing what improvements could be made to the assessment forms, the guiding objective of the WBA Working Group was to improve learning outcomes for trainees.
To this end, the new forms have been simplified, are designed to be more user-friendly and as a result should take less time to complete but still provide meaningful information.
An additional benefit of the new look forms should also be greater consistency across assessors and an increased buy-in from FACEMs who disliked some of the features and the rating scales of the current forms.
-
What are the major changes?
- Assessment is now made on a 5-point scale rather than a 9-point scale.
- Presentation categories have been revised for CbD and Mini-CEX with sub-categories removed.
- Succinct prompts are now provided for assessors for each assessment component.
- Inclusion on all forms to specify if patient is adult or paediatric.
- Complexity rating has been removed from DOPS.
- Unique/different rating criteria introduced for CbD.
- Assessment is stressed to be an evaluation of a singular instance of practice, not a predictive tool for future performance.
-
Important things to remember
All ratings on the new forms are anchored at the top end as being at the level of a New FACEM, TPR panels will review WBAs with this in mind.This means that a trainee who has just begun Early Phase Advanced Training should generally be expected to receive ratings towards the lower end of the scale, depending on previous experience and the procedure or skill being assessed.
Conversely, a trainee at the end of Late Phase Advanced Training should be performing towards the top end of the scale in most assessments, with the aim of having all WBAs at the level of New FACEM by the end of late phase and while completing the Maintenance Pathway.
Local WBA Coordinators will be receiving an induction on the use of the new forms and will be the principal point of support for troubleshooting any challenges that may arise.
Before the release of the new forms, a range of resources will be made available including introduction and how-to-use videos, assessor calibration exercises and updated training modules that should ideally be completed by all assessors and trainees.
New Forms
See the revisions that are specific to each WBA instrument below and view the updated forms through the links (please note these forms are for display purposes only, layout and the look of the forms will change when the release goes live):
-
Case-based Discussions (CbD)
View CbD Form
In a CbD, the trainee presents three sets of case notes from within the previous 4 weeks. The trainee needs to have had a major role in the management of the patient. The assessor selects one case to be discussed from the three cases presented and engages the trainee in discussion of the selected case. The assessor rates and provides structured feedback on the trainee’s clinical reasoning in the selected case. The discussion of the case can be up to 20 minutes with about 10 minutes for feedback.
Updates:
- On the new forms there are two different sections – one to assess the management of the case, and one to assess the trainee’s ability to reflect on the case and evaluate the issues raised. The global rating scale for the CbD focuses on the level of insight demonstrated by the trainee.
- Presentation categories have been revised and sub-categories removed.
- The number of mandatory comment boxes have been reduced.
-
Mini-Clinical Evaluation Exercises (Mini-CEX)
View Mini-CEX Form
A Mini-CEX involves a trainee being directly observed by an assessor whilst performing a focused clinical task during a specific patient encounter. A Mini-CEX must include observation of at least one component of the patient encounter i.e. History Taking, Physical Examination, Clinical Synthesis or Patient Consultation. The assessor rates and provides structured feedback on the trainee’s performance in this specific instance. The observation should take approximately 20 minutes with the provision of feedback to take about 10 minutes.
Updates:
- Presentation categories have been revised and sub-categories removed.
-
Direct Observation of Procedural Skills (DOPS)
View DOPS Form
A DOPS involves a trainee being observed by an assessor whilst performing a specific clinical procedure in the ED. The assessor rates and provides structured feedback on the trainee’s performance of this procedure. Generally, a DOPS can take between 10 and 30 minutes, from the observation of the technical part of the procedure to the provision of feedback.
Updates:
- As case complexity is not applicable to DOPS, this has been removed from the assessment. Comments regarding the case can still be entered for reference.
-
Shift Report
View Shift Report Form
A Shift Report involves the observation of the trainee at multiple time points during a clinical shift. The trainee may or may not be in-charge of the floor, though a minimum of two shift-in-charge assessments are required by late phase advanced trainees. The assessor rates and provides structured feedback on the trainee’s performance during the specific shift. The provision of final feedback, after the shift has concluded, is expected to take about 10 minutes.
Updates:
- As case complexity is not applicable to shift report, this has been removed from the assessment. Comments regarding the shift can still be entered for reference.
New Rating Scales
Context
The most noticeable difference to the rating scale is the reduction from 9-points to 5-points, however the first change to take note of is the context of the scale.
On the current forms, the context is: Based on this encounter, if a similar case occurred, the involvement of a senior clinician with this trainee should be…
As can be seen in the new scale above, the context of assessment is not about a similar future case, it’s only concerned with trainee performance ‘on this observed occasion’.
It was agreed by the Working Group, and supported by survey results, that it is important to focus on the performance on a single observed occasion as some Assessors reported that they found it difficult to speculate on future performance and that WBAs shouldn’t be used to ‘sign-off’ on particular clinical capabilities.
5 Point Scale
After comprehensive consultation it was clear that a 5-point scale was the preferred option among trainees and Assessors. It was acknowledged that the current scale created confusion for some assessors and that some of the criteria were not relevant to the specific WBA instrument.
The new scale has been introduced to simplify and streamline the WBA assessment process and although they are different types of assessments, the new scale also aligns with In-Training Assessments (ITAs). It is intended that a reduction in options will make the trainee’s performance easier to indicate on the scale, and along with high-quality comments, WBAs will enhance their role as a meaningful device for trainees to evaluate and improve their practice.
Contact
For more information please see your Local WBA Coordinator or contact the ACEM WBA team at email: [email protected].
Phone: +61 3 9320 0444