Diagnostic Accuracy of Digital Screening Mammography with and without Computer-aided Detection

Introduction

Computer-aided detection (CAD) for mammography is intended to assist radiologists in identifying subtle cancers that might otherwise be missed. CAD marks potential areas of concern on the mammogram and the radiologist determines whether the area warrants further evaluation. Although CAD for mammography was approved by the Food and Drug Administration (FDA) in 1998 (1), by 2001, less than 5% of screening mammograms were interpreted with CAD in the United States. However, in 2002, the Centers for Medicare and Medicaid Services (CMS) increased reimbursement for CAD, and by 2008, 74% of all screening mammograms in the Medicare population were interpreted with CAD (2,3).

Measuring the true impact of CAD on the accuracy of mammographic interpretation has proved challenging. Findings on potential benefits and harms are inconsistent and contradictory (4–19). Study designs include reader studies of enriched case sets; (4–7) prospective “sequential reading” clinical studies in which a radiologist records a mammogram interpretation without CAD assistance, then immediately reviews and records an interpretation with CAD assistance; (8–12) and retrospective observational studies using historical controls (13–16). One large European trial used a randomized clinical trial design to compare mammographic interpretations by a single reader with CAD compared to double readings without CAD (17).

Comparisons of mammography interpretations with versus without CAD in U.S. community practice have not supported improved performance with CAD (18,19). However, these studies were limited by relatively small numbers and a focus on older women. Another limitation was that CAD technology was studied relatively early in its adoption, so examinations were interpreted during the early part of radiologists’ learning curves and included examinations with outdated film screen mammography. Our study addresses these limitations by using a large database of more than 495,000 full-field digital screening mammograms interpreted with CAD, accounting for radiologists’ early learning curves, and adjusting for patient and radiologist variables. We also assessed performance within a subset of radiologists who interpreted with and without CAD during the study period.

Methods

Results

Digital screening mammography and CAD use increased from 2000 to 2012. In 2003, only 5% of all screening mammograms in the BCSC were digital with CAD; by 2012, 83% of all screening mammograms were acquired digitally and interpreted with CAD assistance (Figure 1).

Among 323,973 women aged 40 to 89, 625,625 digital screening mammography exams were performed (495,818 interpreted with CAD and 129,807 without CAD) between 2003 and 2009 by 271 radiologists. Breast cancer was diagnosed in 3,159 women within 12 months of the screening mammogram and prior to the next screening mammogram. Women undergoing screening mammography with and without CAD assistance were similar in age, menopausal status, family history of breast cancer, time since last mammogram, and breast density. Women undergoing screening mammography with CAD were more likely to be non-Hispanic white than women whose mammograms were interpreted without CAD (Table 1).

Discussion

We found no evidence that CAD applied to digital mammography in U.S. community practice improves screening mammography performance on any performance measure or in any subgroup of women. In fact, mammography sensitivity was decreased in the subset of radiologists who interpreted mammograms with and without CAD. This study builds on prior studies (18–19) by demonstrating that radiologists’ early learning curve and patient characteristics do not account for the lack of benefit from CAD.

Whether CAD provides added value to women undergoing screening mammography is a topic of strong debate (27–36). The lack of consensus may be partly explained by wide variation in CAD use and inherent biases in the methods used to study the impact of CAD on screening mammography. Early studies supporting the efficacy of CAD were laboratory based and measured the ability of CAD programs to mark cancers on selected mammograms. The reported “high sensitivities” of CAD from these studies did not translate to higher cancer detection in clinical practice. In clinical practice, the vast majority of positive marks by CAD must be reviewed and discounted by a radiologist to avoid unacceptably high rates of false-positive exams and unnecessary biopsies, and to practice within acceptable performance parameters recommended by the American College of Radiology (24). The most optimistic view of CAD is that it improves mammography sensitivity by 20% (8,28,30,32). If this were true, cancer detection rates of 4–5 per 1000 without CAD would increase to 5–6 per 1000 with CAD. In other words, for every 1000 women whose screening mammograms were interpreted with CAD, one cancer would be identified that was missed by the unassisted radiologist interpretation. To achieve that single true-positive CAD marking in 1000 women, CAD would render between 2000 and 4000 false-positive marks. Thus, under this scenario, a radiologist would need to recommend diagnostic evaluation for the single CAD mark of the otherwise missed cancer, while discounting thousands of false-positive CAD marks.

Consistent with reports of a prior BCSC cohort study (18) and SEER-Medicare data (2) which primarily evaluated film-screen mammography, we found higher rates of DCIS lesions detected with CAD on digital mammography, but no differences in sensitivity for cancer (whether for DCIS or invasive) and no differences in rates of invasive cancers detected. A meta-analysis in 2008 of 10 studies of CAD applied to screening mammography concluded that CAD significantly increased recall rate with no significant improvement in cancer detection rates compared to readings without CAD (37). The largest recent reader study of digital mammography obtained during the Digital Mammography Imaging Screening Trial (DMIST) found no impact of CAD on radiologist interpretations of mammograms (5). In that report, the authors concluded that radiologists overall were not influenced by CAD markings and CAD had no impact, either beneficial or detrimental, on mammography interpretations.

Our study had sufficiently large numbers to compare interpretations of mammograms read by radiologists who practiced at some sites with CAD and at other sites without CAD. We are concerned that, in these comparisons, sensitivity was lower in CAD-assisted mammograms. Prior reports have confirmed that not all cancers are marked by CAD, and that cancers are overlooked more often if CAD fails to mark a visible lesion. In a large reader study, Taplin et al. (7) reported that visible, non-calcified lesions that went unmarked by CAD were significantly less likely to be assessed as abnormal by radiologists. However, our finding of lower sensitivity with CAD was in a subgroup analysis and should be interpreted with caution.

Given the observational methods of our study, we could not compare mammography performance among women who had their mammograms interpreted both with and without CAD. It is possible that CAD was used preferentially in women whose mammograms were more challenging. However, given the large sample size we were able to control for multiple key factors known to influence mammography performance including patient age, breast density, menopausal status and time since last mammogram. We also were not able to control for radiologist characteristics, such as experience, and thus compared performance with and without CAD in the same radiologists, to address across-radiologist variability.

Our study found no beneficial impact of CAD on mammography interpretation. However, CAD may offer advantages beyond interpretation, such as improved workflow or reduced search time for faint calcifications. Future research on potential applications of CAD may emphasize the contribution of CAD to guide decision-making about management of a radiologist-detected lesion, with the worthy goals of reducing unnecessary biopsy of a mammography lesion with specific benign features or supporting biopsy of a lesion with specific malignant features. Finally, CAD might improve mammography performance when appropriate training is provided on how to use it to enhance performance. Nevertheless, given that the evidence of the current application of CAD in community practice does not show an improvement in diagnostic accuracy, we question the policy of continuing to charge for a technology that provides no established benefits to women.

Gross et al. reported that the costs of breast cancer screening exceed $1 billion annually in the Medicare fee-for-service population (38). Consistent with our findings, they found wide variation in CAD use and very limited effectiveness, and encouraged attention to more appropriate and evidence-based application of new technologies in breast cancer screening programs. Despite its overall lack of improvement on interpretive performance, CAD has become routine practice in mammography interpretations in the United States. Seventeen years have passed since the FDA approved the use of CAD in screening mammography and 14 years have passed since Congress mandated Medicare coverage of CAD. Ten years ago, the Institute of Medicine stated that more information on CAD applied to mammography was needed before making conclusions about its effect on interpretation (39). The U.S. FDA estimates that 38.8 million mammograms are performed each year in the United States. In the BCSC database, 80% of mammograms are performed for screening and by 2012, 83% of screening mammograms in the BCSC were digital examinations interpreted with CAD. Current CMS reimbursement for CAD is roughly $7 per exam and many private insurers pay over $20 per exam for CAD, translating to over $400 million per year in current U.S. healthcare expenditures, with no added value and in some cases decreased performance.

In the era of Choosing Wisely® and clear commitments to support technology that brings added value to the patient experience, while aggressively reducing waste and containing costs, (40) CAD is a technology that does not appear to warrant added compensation beyond coverage of the mammographic examination. The results of our comprehensive study lend no support for continued reimbursement for CAD as a method to increase mammography performance or improve patient outcomes.

Supplementary Material