The detection of yarn hairiness is essential in the production of chemical fiber yarn packages, and it is difficult to detect them because of their small features, which are easy to be missed and confused with another non-defective feature broken ends. To detect defects accurately and efficiently in the appearance of yarn packages, a CenterNet defect detection algorithm (CenterNet-CBAM) combining with attention mechanism is proposed. Two types of confusing target images, “yarn hairiness” and “broken ends”, are collected, and an object detection model based on CenterNet-CBAM is constructed, and the Recall of CenterNet-CBAM in the two categories of “yarn hairiness” and “broken ends” is 90.20% and 85.42%, Precision is 93.88% and 93.48%, AP is 90.91% and 90.93%, and MAP is 90.92% for the two categories, respectively, which were better than CenterNet and YOLOv4, which verified the effectiveness of the experimental method.