Abstract
Introduction
Valid and reliable assessments of operative performance are essential to competency-based surgical education. The Otologic Technical Instruction and Competency (OTIC) Rubric was created to assess technical performance of essential otologic procedures; it scores global Procedure and Component performance and classifies errors. To study construct validity, Procedure and Component scores were examined in relation to experience, level of training, and operative components.
Methods
We conducted a longitudinal validation study of intraoperative OTIC assessments collected from October 2022 through June 2026 at an academic otolaryngology residency program. Twenty-two residents (PGY-1 to PGY-5) were evaluated during postauricular approach with canal incisions (PAA group 1), postauricular approach without canal incisions (PAA group 2), tympanotomy, mastoidectomy, stapedotomy, facial recess approach, and tympanoplasty. Cumulative procedure-specific attempts, interval between attempts, and postgraduate year (PGY) were analyzed by multivariable mixed-effects ordinal logistic regression. Spearman correlations and procedural-step models evaluated relationships between Component Skills and Procedure scores; unstable multivariable step models were summarized using Spearman correlations. Trainees’ response to Rubric implementation was obtained by anonymous survey.
Results
A greater number of procedure-specific attempts were independently associated with higher Procedure scores for PAA group 1 (OR 1.58, 95% CI 1.14–2.18; p=.007), PAA group 2 (OR 1.20, 95% CI 1.04–1.39; p=.011), tympanotomy (OR 1.59, 95% CI 1.14–2.21; p=.007), and tympanoplasty (OR 2.48, 95% CI 1.25–4.94; p=.011). Facial recess demonstrated a similar positive association that was not significant (OR 1.75, 95% CI 0.95–3.19; p=.069). PGY experience was independently associated with competency only for mastoidectomy (OR 2.70 per year of training, 95% CI 1.20–6.04; p=.017). Time between consecutive attempts was not independently associated with competency for any procedure.
Procedural-step models identified independent predictors of competency: PAA Group 1-canal incision (OR 3.33, 95% CI 1.59–6.99; p=.002) and deep dissection (OR 8.65, 95% CI 2.95–25.39; p .001); tympanotomy-lateral mallear ligament disarticulation (OR 74.15, 95% CI 2.75–2002.15; p=.010); and mastoidectomy-tegmen (OR 10.89, 95% CI 3.55–33.40; p .001) and antrum dissection (OR 44.74, 95% CI 10.91–183.53; p .001). Significant component–Procedure correlations were observed for PAA Group 2 (ρ=.750–.786), stapedotomy (ρ=.764–.807), facial recess (ρ=.441–.711; p≤.006), and tympanoplasty (ρ=.672–.906; all p .001), with graft placement showing the strongest association (ρ=.906).
Participants (n=13) evaluated rubric-based appraisals. All agreed or strongly agreed that appraisals help identify deficiencies, set practice goals, enhance awareness of errors/near-misses, and add value to residency training. Additionally, 92.3% agreed or strongly agreed that appraisals enhance intra-operative learning and assist in planning and executing procedural steps. All agreed or strongly agreed that the rubric is easy to understand and scoring anchors are clear; 92.3% agreed or strongly agreed that it can be effectively applied across all training levels.
Conclusion
The OTIC Rubric is a practical tool for competency-based education in otologic surgery. This study supports its construct validity by demonstrating expected relationships with increasing procedural experience, progression through residency, and performance of key operative components. Trainees’ responses were an overwhelming endorsement of its operational clarity, appraisal value, coverage of core constructs, and practical utility. These findings support further development of the OTIC Rubric as a formative and summative assessment.