observer variability
Recently Published Documents

Objective: To apply the St Thomas’ Hospital (STH) classification of round window type, in a Pakistani pediatric population undergoing cochlear implantation, and rate the inter observer variability of applying this classification. Study Design: Cross sectional study. Place and Duration of Study: Combined Military Hospital Rawalpindi, from Apr 2019 to Dec 2020. Methodology: Patients were examined per-operatively by a panel of four surgeons after "optimal" posterior tympanotomy for round window variations, as per STH classification of approachability of RWM. The observations of the four surgeons were recorded and interobserver variation was assessed and analyzed. Results: A total of 100 patients were operated, 45 females and 55 males. Mean age was 3.8 years. There was minimal inter observer variability with regards to round window type and extent of "optimal" posterior tympanotomy. Three patients had type I, 76 had type IIA, 15 had type IIB and 6 patients had type III. Round window insertion/membranous cochleostomy was possible in 70 patients, whereas the rest require extended round window approach or bony cochleostomy. Conclusion: The STH classification is a useful predictor of route of CI electrode insertion and most patients can undergo RW insertion with confidence based on minimal variation between surgeons when applying the STH classification as well as when deciding the extent of surgical exposure.

Inter- and intra-rater variability negatively affects the reliability of various histopathology grading scales used as prognostic aids in human and veterinary medicine. The Kenney–Doig categorization (grading) scale, which is used to associate equine endometrial histologic lesions with prognostic estimation of a broodmare’s reproductive potential, has not been evaluated for inter- or intra-rater variability, to our knowledge. To assess whether the Kenney–Doig system produces reliable results among observers, 8 pathologists, all with American College of Veterinary Pathologists certification, were recruited to blindly categorize the same set of 63 digital equine endometrial biopsy slides as well as to re-evaluate anonymously 21 of 63 of these slides at a later time. Cohen kappa values for pairwise comparison of final Kenney–Doig categories were −0.05 to 0.46 (unweighted) and 0.08–0.64 (weighted), with an average Light kappa of 0.19 (unweighted) and 0.36 (weighted) across all 8 pathologists, 0.14 (unweighted) and 0.33 (weighted) for pathologists at different institutions, and 0.22 (unweighted) and 0.46 (weighted) for pathologists at the same institution. Intra-class correlations measuring intra-rater agreement were 0.12–0.77 with an average of 0.55 for all 8 pathologists. We found that only slight-to-moderate inter-rater agreement and poor-to-good intra-rater agreement was produced by 8 pathologists using the Kenney–Doig scale, suggesting that the system is subject to significant observer variability and care should be taken when communicating Kenney–Doig categories to submitting clinicians with emphasis on the quality of endometrial lesions present instead of the category and associated expected foaling rate.

Abstract Objective The aim of this study was to investigate the reproducibility of anterior–posterior diameter (APdmax) and three-dimensional lumen volume (3DLV) measurements of abdominal aortic aneurysms (AAA) in a classical murine AAA model. We also compared the magnitude of change in the aortic size detected with each method of assessment. Methods Periadventitial application of porcine pancreatic elastase (PPE AAA) or sham surgery was performed in two cohorts of mice. Cohort 1 was used to assess for observer variability with the APdmax and 3DLV measurements. Cohort 2 highlighted the relationship between APdmax and 3DLV and changes in AAA detected. Results There was no significant observer variability detected with APdmax measurement. Similarly, no significant intraobserver variability was evident with 3DLV; however, a small but significant interobserver difference was present. APdmax and 3DLV measurements of PPE AAA significantly correlated. However, changes in the AAA morphology were detected earlier with 3DLV. Conclusion APdmax and 3DLV are both reliable methods for measuring an AAA. Both these methods correlate with each other. However, changes in AAA morphology were detected earlier with 3DLV, which is important to detect subtle but important changes to aortic geometry in a laboratory setting. 3DLV measurement of AAA is a simple, reproducible, and comprehensive method for assessing changes in disease morphology.

Abstract Background Mitral annular plane systolic excursion (MAPSE) and left ventricular (LV) early diastolic velocity (e’) are key metrics of systolic and diastolic function, but not often measured by cardiovascular magnetic resonance (CMR). Its derivation is possible with manual, precise annotation of the mitral valve (MV) insertion points along the cardiac cycle in both two and four-chamber long-axis cines, but this process is highly time-consuming, laborious, and prone to errors. A fully automated, consistent, fast, and accurate method for MV plane tracking is lacking. In this study, we propose MVnet, a deep learning approach for MV point localization and tracking capable of deriving such clinical metrics comparable to human expert-level performance, and validated it in a multi-vendor, multi-center clinical population. Methods The proposed pipeline first performs a coarse MV point annotation in a given cine accurately enough to apply an automated linear transformation task, which standardizes the size, cropping, resolution, and heart orientation, and second, tracks the MV points with high accuracy. The model was trained and evaluated on 38,854 cine images from 703 patients with diverse cardiovascular conditions, scanned on equipment from 3 main vendors, 16 centers, and 7 countries, and manually annotated by 10 observers. Agreement was assessed by the intra-class correlation coefficient (ICC) for both clinical metrics and by the distance error in the MV plane displacement. For inter-observer variability analysis, an additional pair of observers performed manual annotations in a randomly chosen set of 50 patients. Results MVnet achieved a fast segmentation (<1 s/cine) with excellent ICCs of 0.94 (MAPSE) and 0.93 (LV e’) and a MV plane tracking error of −0.10 ± 0.97 mm. In a similar manner, the inter-observer variability analysis yielded ICCs of 0.95 and 0.89 and a tracking error of −0.15 ± 1.18 mm, respectively. Conclusion A dual-stage deep learning approach for automated annotation of MV points for systolic and diastolic evaluation in CMR long-axis cine images was developed. The method is able to carefully track these points with high accuracy and in a timely manner. This will improve the feasibility of CMR methods which rely on valve tracking and increase their utility in a clinical setting.

The Kenney–Doig scale is a histopathology categorization (grading) system often used as the standard for assessing endometrial disease and communicating prognostic fertility information for equine breeding prospects. We investigated how Kenney–Doig categories compared within the same institution and across different institutions to determine if observer variability may contribute to category frequencies. We conducted a retrospective analysis of all equine endometrial submission records between 1998 and 2018 at the Western College of Veterinary Medicine (WCVM) and Prairie Diagnostic Services (PDS). Of 726 biopsies, we found the following category distribution: 46 of 726 (6.3%) I, 307 of 726 (42.3%) IIA, 326 of 726 (44.9%) IIB, and 47 of 726 (6.5%) III. We also conducted a review of the literature and included 6 studies reporting Kenney–Doig category distributions. Chi-square analysis showed significant differences between the category distribution found at WCVM and PDS and the category distribution reported in the 6 studies. To account for differences in mare populations, individual category distributions were generated for 5 pathologists at the WCVM and PDS. The Fisher exact test among these 5 Kenney–Doig categories revealed significant differences in category tendencies, suggesting that observer variation affects the use of the scale. Our results suggest that there is a need for prospective inter-rater and intra-rater agreement studies of the repeatability of the Kenney–Doig scale.

Abstract Introduction This study aimed to investigate the uncertainty in organ delineation of low-dose computed tomography (CT) images using a high-strength iterative reconstruction (IR) during radiotherapy planning for the treatment of prostate cancer. Methods Two CT datasets were prepared with different dose levels by adjusting the reconstruction slice thickness. Two observers independently delineated the prostate, seminal vesicles, bladder and rectum on both images without referring to other modality images. The delineated organ volumes were compared between both images. Observer delineation variability was assessed using Dice similarity coefficient (DSC) and mean distance to agreement. Results No significant differences regarding the delineated organ volumes were observed between the low- and standard-dose images for all organs. Regarding inter-observer variability, the DSC was relatively high for both images, whereas mean distance to agreement was not significantly different between images (p > 0·05 for all). Intra-observer variability for each observer showed high DSC (>0·8 and >0·9 for seminal vesicles and other organs, respectively) but no significant differences in the mean distance to agreement (p > 0·05 for all). Conclusions Our results indicate that low-dose CT images with high-strength IR would be available for organ delineation in the radiotherapy treatment planning for prostate cancer.