The purpose of this study is to test the use of a tool (NOTSSdk)to assess and provide feedback to trainee surgeons' non-technical skills. It focuses on two aspects: reliability of ratings of trainee surgeons using NOTSSdk; and the content and quality of feedback given to trainees based on observations using NOTSSdk.

The specific research questions are:

• How many situations (=operations) are necessary to assess using NOTSSdk, to get reliable ratings of trainee surgeons' NTS?
• What is the impact of the NOTSSdk feedback tool on the content of feedback conversations regarding NTS?
• How do trainees perceive the usefulness of the feedback session?
• How do raters perceive the usefulness of the ratings and the feedback session?

Materials and methods

We aim for a sample of 15 trainees in one hospital.

Demographic data will be collected regarding: gender, age, postgraduate experience (years/months), surgical experience (years/months), specialist training level for both assessors and trainees.

Procedure The head of the surgical department will be informed about the project and will be asked permission to recruit trainees and assessors.

Participants will be informed about the project orally and in writing prior to commencement. Surgeon assessors and trainees will be given the NOTSSdk handbook to familiarize themselves with the system before the study.

The assessors will receive a short guidance on how to provide assessments and give feedback using NOTSSdk immediately before the first operation. After the first assessment and feedback session, the assessor will be offered a further short structured guidance on how to fill in the assessment form and how to provide structured feedback. It will also be allowed for the assessor to solve ambiguities, ask questions and to get further explanation of the categories and elements of NOTSSdk.

Eligible operations: laparoscopic cholecystectomy, hernia repair, laparoscopic appendectomy, small bowel obstruction, hemicolectomy - depending on the trainee's competence.

Reliability study Each trainee will be assessed by one surgeon assessor using NOTSSdk during an operation. Each trainee will be assessed on two occasions. Assessors will vary according to availability. This will produce app. 30 assessments in total.

Feedback study Data will be collected in connection with data gathering in the reliability study that is, data will be provided by the same assessors and the same trainees on the same occasions. After the operation the surgeon will provide feedback to the trainee based on his/her observations. This feedback session will be observed and audio recorded by the primary investigator. After each feedback session the trainee and the surgeon will be individually surveyed exploring their perception of the feedback session and its usefulness. After the last feedback session the assessors will be surveyed about their perceptions of NOTSSdk feasibility.

Data analysis

Reliability study Generalizability theory will be applied to analyze how many operations are necessary to observe for reliable assessment. First the different variance components (trainees, assessors, cases etc.) will be estimated (three facet, balanced design with trainees nested within assessors and nested within operations). These will be used to generate Generalizability coefficients to estimate overall reliability of the use of NOTSSdk depending on number of assessors and operations observed.

Feedback study

We will use qualitative content analysis of emergent themes to explore the contents of the feedback sessions with special focus on:

• Frequency of NTS topics discussed over time
• Frequency of TS topics discussed over time Descriptive statistics will be used to present trainees' perceptions of the usefulness of the feedback and to present assessors' perceptions of the usefulness and feasibility of the ratings and the feedback.

Inclusion Criteria:

• trainees in surgical specialist training

Exclusion Criteria:

• trainees not in surgical specialist training

Trainee surgeons in one surgical department