- Frame of Reference training (FOR): Increase rating accuracy by focusing on the different levels of performance (Salas et al. - Reportedly, it is seemingly straightforward to train a single group of raters and achieve a relatively high level of inter-rater agreement and accuracy when compared to a standard set rated by an expert (Salas et al. - The rate of between-group rater agreement and accuracy is known to drop significantly without addressing such errors.. - Salas et al. - (2001) outlined guidelines for training raters in the use of behavioural markers. - There is limited recent evidence on effective rater training in the medical domain and so these guidelines were followed to develop our rater training for ANTS. - These guidelines were previously followed by Flin and Glavin’s research group (Fletcher et al. - 2003) to evaluate inter-rater agreement for ANTS.. - Baker et al (2001) employed an eight-hour rater training programme for a behavioural marker system. - They achieved an adequate level of inter-rater agreement and accuracy in this time frame. - Workpackage Report 7 from the University of Aberdeen investigating ANTS reports a four-hour training programme (Fletcher et al. - With this minimal amount of training, inter-rater agreement of rwg. - During the initial evaluation of ANTS by the Scottish team, feedback from experts and calibration were deliberately excluded from rater training to isolate the impact of these on inter-rater agreement. - In this way, the training provided to their participants was intentionally limited to try and isolate the reliability of the tool itself. - With our research we hoped to move this forward a phase by including feedback from experts to test calibration, and thus improve inter-rater agreement.. - Rater Training Day. - Were the college to have a widespread roll-out of ANTS as an assessment tool, a short course would be necessary if many assessors were to be trained.. - Using ANTS for Workplace Assessment 195 Pre-reading was sent out prior to the workshop including the ANTS Handbook (<www.abdn.ac.uk/ANTS>) and ‘Recommendations for the use of behavioural markers’ (Klampfer et al. - An outline of the rater training is as follows:. - The attendees found the behavioural observation training to be an enjoyable experience. - Five videos were shown in the afternoon for the purpose of practice assessment and calibration. - Expert ratings and a discussion followed for the purpose of calibration.. - Many of the learning points that we gained from this project arose during these sessions and will be discussed later.. - Participants rated performance in the five test videos, scoring in each case for 15 identified skill elements. - ICC allows different sources of variability to be included and is essentially another way of calculating inter-rater agreement. - This reflects the many sources of variability that are introduced when investigating inter-rater agreement.. - Intraclass correlations were calculated for each element of ANTS (see Figure 12.1). - As you will see later in this chapter, we learnt a number of lessons while trying to achieve inter-rater reliability with such a large group.. - Comparison of scores with those of the ‘expert raters’ also showed unsatisfactory results.. - All participants were involved in the supervision of ANZCA trainees with a range of experience in terms of supervision from 1–30 years. - Of the 26 participants, eight had one year or less of supervising experience.. - Only seven participants already had a system for assessment in the workplace.. - The most common aims the participants identified for the workshop were to. - ‘provide better or constructive feedback’, and ‘to have a systematic method for assessment’ (22 out of 26). - No one felt that the elements of the current in-training assessment process were useful for assessment.. - Figure 12.1 intraclass correlations calculated for each component of AnTS. - Using ANTS for Workplace Assessment 197. - The level for most of the training was scored at ‘just right’.. - Each participant felt that ANTS was useful as a formative assessment tool.. - Most also felt that it would highlight the importance of non-technical skills.. - The question as to whether ANTS was suitable as a summative assessment tool divided the participants. - Thirteen raters thought it was suitable as a summative assessment tool and 13 thought more work should be done. - During which year of training summative assessment of non-technical skills should be performed also divided opinion. - Comments from ANZCA fellows regarding the use of ANTS as a summative assessment tool:. - Anaesthesia trainees should be exposed to non-technical skills training early to encourage good behaviours.. - It seems unusual that the Scottish team investigating ANTS achieved an inter-rater reliability of r=0.5–0.7 with minimal training yet, in comparison, our correlation is so poor. - What was different? We were hoping to see the introduction of ANTS as a summative assessment tool and unfortunately, this does not look to be a great start.. - At the time we thought we were seeing potential problems with our rater training, but what we actually saw were some warnings for the use of workplace-based assessment in general.. - We believe these two problems were mainly responsible for the lack of inter- rater agreement. - This was obvious from some of the ‘lively’ discussions during calibration. - However, this still does not account for the large difference between the Scottish raters and the Australian ones.. - When we designed our study, we kept in mind the fact that ANTS could be used, in the future, on a large scale and by a variety of anaesthetists.. - The participants in our rater training day were all specialist anaesthetists of varying experience, but without specific training in education or simulation. - All the anaesthetists in the Scottish study had some involvement in education and training activities (Fletcher et al. - The large difference in inter-rater agreement could be an effect of education. - Using ANTS for Workplace Assessment 199 introduced. - We learnt many lessons from our rater training day, and realized how many sources of bias can be introduced.. - Our trainee raters agreed in the post-workshop questionnaire.. - We failed to demonstrate acceptable inter-rater reliability with the level of training we offered. - If the data are compressed to an average score from each rater then the inter-rater agreement substantially increases. - Changing the latter would detract from the validity of the real world experience.. - However, the voluntary nature of the participation introduces bias. - Resistance to implementation could be predicted to occur with many of the stakeholders. - ANTS does not appear to be a feasible tool to use for summative assessment in its current state. - Significant training and practice are likely to be needed, leading to low interest in potential assessors. - The validity of this modification would also need to be examined.. - General lessons learnt about implementation of a workplace-based assessment tool.. - Video has been shown to have similar validity and reliability to real-time observation (Hays et al.. - 2002) but is unlikely to be useful in our setting for trainee assessment due to the increased cost.. - The dynamics of the group impacted more on results than we would have expected, with vigorous discussion becoming unhealthy at times. - We appeared unable to dislodge preconceptions about some behaviours, despite others in the group making it clear these beliefs were held by only a very small minority. - Observations of those with dissenting opinions in the group tallied closely with their standing as an outlier in the rating process.. - The overriding strength of ANTS is its content validity, with effective coverage of the domains of non-technical practice of anaesthesia. - This coverage is also its downfall, giving it a complexity that limits its feasibility as a summative assessment tool. - The poor inter-rater reliability that we demonstrated is likely to be a feature of any workplace-based assessment tool for anaesthesia as the subtleties of medical practice make maintaining high validity and reliability together difficult. - (2002) WP7 Report: Evaluation of the Prototype Anaesthetist’s Non-Technical Skills (ANTS) Behavioural Marker System: University of Aberdeen Workpackage Report for SCPMDE. - Fletcher, G., Flin, R., McGeorge, P., Glavin, R., Maran., N and Patey., R. - (2003) Anaesthetists’ Non-Technical Skills (ANTS): Evaluation of a behavioural marker system. - (2004) Rating non-technical skills: Developing a behavioural marker system for use in anaesthesia. - Spike, N., Alexander, H., Elliott, S., Hazlett, C., Kilminster S., Prideaux, D., and Roberts, T. - (2000) In-training assessment – its potential in enhancing clinical teaching. - (1994) Rater training for performance appraisal: A quantitative review. - Working in groups is widespread in medicine, especially in the operating room.. - Human factors such as breakdown in the quality of teamwork have been identified as a main source of failures in medical treatment (Arbous et al. - 2001, Cooper et al. - 2002, Gaba 2000, Helmreich and Davies 1996, Lingard et al. - 2004, Reason 2005, Sexton et al. - There is growing evidence that the ability of medical teams to deal with the required complex work processes strongly depends on adaptive team coordination (e.g., Manser et al. - 2008, Risser et al. - 1999, Rosen et al. - 2008, Salas et al. - 2007b, Schaafstal et al. - 2001, Zala- Mezö et al. - Coordination has been defined as the ‘structured patterning of within-group activities by which groups strive to achieve their goal’ (Arrow et al. - The induction of anaesthesia is particularly demanding compared to the other tasks involved in the anaesthetic process (see Phipps et al
Xem thử không khả dụng, vui lòng xem tại trang nguồn hoặc xem
Tóm tắt