- Our study views feasibility as a key assessment characteristic and it is formally considered within the study’s design. - Following each case, the supervising consultant completes a PBA or OSATS before giving feedback to the trainee. - We observe this process and record the time taken to complete the assessment form and the duration of feedback. - The follow-up questionnaires which we distribute to trainees and assessors approximately one month after completing assessments, in the relevant specialty, include questions to address feasibility issues, e.g., time added to operating list, available room for feedback, ease of use of tools. - We have not directly addressed cost effectiveness in this study, although the data could inform future research.. - Are the Assessment Tools Acceptable to Stakeholders?. - User acceptability is the extent to which an assessment tool or method is accepted by the stakeholders involved in the assessment. - It is a crucial factor in the design and successful implementation of assessment programmes.. - The acceptability of the assessment tools to trainees and assessors is being evaluated during the study’s implementation, as the future direction of competency- based assessment in the UK will be influenced by the opinions of surgical trainees and trainers. - After each feedback session and in the follow-up questionnaires we address the acceptability of the assessments and the assessment tools.. - However, we consider acceptability from the overall perspective of the people directly involved in the assessments. - We ask supervising consultants and trainees to rate their overall satisfaction with the assessment tools immediately after feedback. - Thematic Analysis of the Problem-solving Approach to the Study implementation. - We have used a number of themes below to illustrate the problem-solving approaches we have adopted during the study implementation. - We have identified lessons learnt and made some practical suggestions which may be of use to those undertaking similar work.. - Matching the Research Team to the Study Design. - We have assembled a research team which has the skills required to evaluate assessment in the operating room. - All three independent assessors are practising surgeons with expertise in many of the surgical specialities, having received training in assessment and feedback through the ‘Training the Trainers’ course at the Royal College of Surgeons, and NOTSS training facilitated by the NOTSS research team, in Edinburgh.. - familiarity and confidence with working in the operating room environment;. - We are carrying out research in real time in the workplace. - One of the challenges we faced in implementing the study was ensuring that we used a timely and appropriate method to inform the surgical teams involved. - We used the following methods of communication, often in combination, to disseminate the purpose and design of the research study:. - Our main aim in the advance communication was to familiarize staff with the study before we moved the research into their specialty. - This approach recognized that research and assessment in the operating room could be viewed by staff as. - threatening or unnecessary, unless we clearly explained the aims of the study within the context of work-place based assessment and the surgical curricula.. - The information packs include an overview of the study and examples of the assessment tools with guidance on their use for assessment and feedback. - As the study has rolled out, we have recognized the need to support email communication with meetings, written information and face-to-face discussion. - We have been able to use formal meetings at times but this has often been constrained by practical considerations, for example the size, organizational structure and availability of the target audience. - It has not been feasible to organize formal presentations within work time for scrub team and anaesthetic staff, which we have managed by arranging smaller ad hoc meetings. - Overall we have found that trainee and assessor engagement has been best achieved using face-to-face discussions in the field with supporting written information, having provided a background to the study by email communication and presentations where possible.. - Consider your resources and the feasibility of your approach to communicate and disseminate the study.. - Be flexible and revise your approach to overcome time constraints and structural barriers in the workplace.. - Further to communicating and disseminating the study methodology, there was the need for us to familiarize and train staff involved in assessing and/or giving feedback. - The breadth of the personnel involved in the study and the feasibility of providing training within work time were also significant barriers to the study implementation.. - We have made great efforts to train all assessors to ensure confidence in and credibility of the assessor ratings. - We have provided all assessors with information. - packs, including the relevant assessment tools, guidance on their use and access to web-based training which we have usually supported with one to one familiarization before they have undertaken any assessments.. - We acknowledge that we have been unable to achieve an entirely level playing field for training assessors in the use of workplace assessment tools, which reflects the reality of the surgical workplace. - However, the study design includes several ‘quality measures’ of training. - This will be incorporated in the data analysis. - Furthermore, our data analysis using generalizability theory will enable an examination of the variance in ratings for different assessor designations, for example a comparison of NOTSS ratings between independent assessors, anaesthetists and scrub nurses. - Despite training all assessors in the appropriate use of the tools, we have observed some inconsistencies in the way the tools are used:. - Supervisor training to this level is beyond the scope of our study, but is what will be required of trainers in the future if training is to become more effective.. - Providing training in the workplace requires flexibility and tenacity to ensure full coverage.. - Consider consistency in the training approach you adopt, taking into account differences in staff engagement and attitude.. - These factors are often not confirmed until the day of the operation and the case cannot go ahead as part of the study if any one of them is missing:. - Furthermore, the ethical requirement of the study is that patients receive a Patient Information Sheet 24 hours before they are approached for consent to give them time to consider. - We provided an open discussion of the training system, acknowledging the role of supervised operating in training.. - Consider the complexity of consenting and recruiting patient participants for observational studies in the surgical environment and the resources required.. - Communicate at all levels to make full potential of the team’s resources and to maximize the recruitment of suitable cases.. - Ethical approval for the study dictates written consent from patients as participants but not from trainees. - However, from a training perspective and during the implementation of the study, we have also considered surgical trainees to be study participants. - Our approach has been to seek verbal consent from surgical trainees before their involvement in the study. - We provide all trainees with an invitation letter before introducing the study into a new surgical specialty and give them an opportunity to discuss the research with the study coordinator, thereby ensuring participation without duress. - We have experienced initial hesitancy from some trainees regarding their involvement in the study, often centred on misunderstanding the purpose of the study and concerns that research data could affect their training, for example in the event of a critical incident occurring. - One of the clear messages we have conveyed is that the study is designed to assess the assessment tools themselves, in particular their validity and reliability across different surgeons, cases and specialties, and it is not assessing an individual’s level of surgical skill or competence. - (See Theme 6, ‘Research versus Training Agenda’, for a fuller discussion.) We have continued in our attempts to collect data to suit the statistical model employed for the study, the optimal data being different combination of trainees and assessors across the cases. - Some trainees operate more frequently, some senior surgical trainees perform complex procedures which are not covered by the study and some consultant lists have more index procedures. - However, moving trainees across lists for assessments in the operating room proved unworkable. - For the same reasons, if there were late changes to the trainee covering a particular list, we decided to exclude the case to focus the study on authentic workplace assessments.. - Engage trainees in the process of assessment by communicating effectively the research purpose and the role of their involvement.. - The study protocol includes the validation of assessment tools which are in current use in the workplace for surgical training. - (See the earlier background of this chapter for an overview of PBA and OSATS tools within surgical training programmes.) There are opportunities for educational research to form collaborations or conflicts with the training agenda, and this is illustrated by discussing our role as independent assessors in the study.. - Examples of collaboration Providing opportunities for training in workplace- based assessment has moved beyond the research agenda to provide trainees and trainers with valuable, timely training on the tools which are integral to the new surgical curricula.. - The study has provided ring-fenced opportunities for training and assessment, which has resulted in the increasing engagement of trainees in the study. - Participation in the study has also encouraged a number of trainers and trainees to use the PBA tool for the first time. - We have been able to show trainers and trainees that suitable cases for workplace training can be identified opportunistically, and that the process of assessment and feedback is feasible, adding little time to an operating list.. - Highlighting the role of formative assessment in driving learning has encouraged appropriate use of the PBA within the curriculum. - For example, use of parts of the PBA for trainees not ready to complete the whole operation under supervision.. - Our ‘field testing’ of the assessment tools has generated suggestions for tool modification which have been forwarded to the relevant bodies for consideration.. - therefore becomes an important assessment item which was not part of the original template.. - Examples of dichotomy All assessors score independently without conferring or discussion until the assessment tools for the case are completed. - However, it is inevitable in the clinical setting for some discussion to take place with the presence of trained independent assessors who are facilitating cases for assessment purposes. - Assessment in the operating theatre is a relatively new training method, and the research team are seen to represent a body of expertise in assessment and feedback.. - We have found that our role as a complete observer in the independent assessment of skills and behaviours in the operating room is both unrealistic and unworkable. - for example, what constitutes a good surgical technique and why? Judgements on skills and behaviours are rated independently between assessors, although discussion surrounding the subjectivity of skills and behaviours is seen as a necessary outcome of the study implementation.. - Another area of conflict we have recognized relates to the quantity and quality of the research data. - The supervising consultant is often not present in the operating room until the case commences, which omits the pre-operative preparation section of the PBA and equivalent sections on the OSATS. - We have found it challenging, particularly when new trainees and/or trainers become involved in the study, to provide sufficiently general information to support the assessments and the use of the assessment tools, without introducing specific prompts which would affect the assessment ratings and overall data quality.. - We have made compromises to uphold the research agenda, as from a training standpoint, independent assessor(s) would prompt trainers on the appropriate use of the assessment tools, such as to address the inconsistencies in tool use highlighted above in Theme 3.. - Some discussions surrounding the assessment of skills and behaviours are a necessary condition for implementing workplace assessments.. - Developments in Study Design and Methodology during Implementation Developments to the study design and methodology have arisen to meet the requirements of the study aim and to promote future research directions. - It is only when the study design and methodology are subjected to field testing during their implementation that the full requirements of the study protocol can be realized.. - These developments have been driven by the foresight of the research team and with guidance from the Studys Steering Committee. - External review of the original study protocol advised us to consider more than two assessment tools (originally PBA and OSATS). - The development of rating non-technical skills in the operating room, supported by the literature evidence for the relationship of these skills to surgical skills and safety, stimulated the addition of NOTSS to the study protocol.. - We originally recruited patients from one specialty at a time because of the logistical difficulties in working in multiple specialties simultaneously. - We needed to collect a larger dataset of assessments in this specialty compared to the others to allow a comparison of the tools. - However, recruiting cases in obstetrics and gynaecology has been very successful because it lends itself to providing large numbers of suitable cases (see Theme 8, ‘The Significance of Context’, for a full discussion). - The addition of this specialty has included a non-elective index procedure, urgent caesarean, which provides an opportunity for video assessment and use of the NOTSS tool in urgent surgical cases.. - However, the number of cases required to obtain sufficient data to test each of the assessment tools would be very large, certainly unachievable within this study. - We recognize that differences exist between the assessment tools which make their simultaneous use in assessment problematic.. - intentions throughout, also advised in the PBA validation document for training assessors. - In specialties where we realized that recruitment would not improve, we decided to move on rather than risk a training effect by remaining in the speciality. - Preliminary fieldwork can require changes to be made to the study design and methodology, which may involve ethics amendments.. - The individual surgical specialties have offered different advantages and disadvantages to the study implementation. - The obstetrics and gynaecology specialty has leant itself well to the study methodology. - Within orthopaedics, operating lists were often amended at short notice, giving insufficient time to inform patients about the study. - In obstetrics and gynaecology there is an established culture for objective assessments of surgical competence and a requirement of consultants to provide assessment and feedback to trainees, which facilitated our introduction of the study. - The RCOG has phased in the use of
Xem thử không khả dụng, vui lòng xem tại trang nguồn hoặc xem
Tóm tắt