Defect detection is a basic and essential task in automatic parts production, especially for automotive engine precision parts. In this paper, we propose a new idea to construct a deep convolutional network combining related knowledge of feature processing and the representation ability of deep learning. Our algorithm consists of a pixel-wise segmentation Deep Neural Network (DNN) and a feature refining network. The fully convolutional DNN is presented to learn basic features of parts defects. After that, several typical traditional methods which are used to refine the segmentation results are transformed into convolutional manners and integrated. We assemble these methods as a shallow network with fixed weights and empirical thresholds. These thresholds are then released to enhance its adaptation ability and realize end-to-end training. Testing results on different datasets show that the proposed method has good portability and outperforms the state-of-the-art algorithms.