Search Results (569)

Recent developments in affordable depth imaging hardware and the use of 2D Convolutional Neural Networks (CNN) in object detection and segmentation have accelerated the adoption of machine vision in a range of applications, with mainstream models often out-performing previous application-specific architectures. The need for the release of training and test datasets with any work reporting model development is emphasized to enable the re-evaluation of published work. An additional reporting need is the documentation of the performance of the re-training of a given model, quantifying the impact of stochastic processes in training. Three mango orchard applications were considered: the (i) fruit count, (ii) fruit size and (iii) branch avoidance in automated harvesting. All training and test datasets used in this work are available publicly. The mAP ‘coefficient of variation’ (Standard Deviation, SD, divided by mean of predictions using models of repeated trainings × 100) was approximately 0.2% for the fruit detection model and 1 and 2% for the fruit and branch segmentation models, respectively. A YOLOv8m model achieved a mAP50 of 99.3%, outperforming the previous benchmark, the purpose-designed ‘MangoYOLO’, for the application of the real-time detection of mango fruit on images of tree canopies using an edge computing device as a viable use case. YOLOv8 and v9 models outperformed the benchmark MaskR-CNN model in terms of their accuracy and inference time, achieving up to a 98.8% mAP50 on fruit predictions and 66.2% on branches in a leafy canopy. For fruit sizing, the accuracy of YOLOv8m-seg was like that achieved using Mask R-CNN, but the inference time was much shorter, again an enabler for the field adoption of this technology. A branch avoidance algorithm was proposed, where the implementation of this algorithm in real-time on an edge computing device was enabled by the short inference time of a YOLOv8-seg model for branches and fruit. This capability contributes to the development of automated fruit harvesting. Full article

Ovine pulmonary adenocarcinoma (OPA) is a contagious lung tumour caused by the Jaagsiekte Sheep Retrovirus (JSRV). Histopathological diagnosis is the gold standard for OPA diagnosis. However, interpretation of traditional pathology images is complex and operator dependent. The mask regional convolutional neural network (Mask R-CNN) has emerged as a valuable tool in pathological diagnosis. This study utilized 54 typical OPA whole slide images (WSI) to extract 7167 typical lesion images containing OPA to construct a Common Objects in Context (COCO) dataset for OPA pathological images. The dataset was categorized into training and test sets (8:2 ratio) for model training and validation. Mean average specificity (mASp) and average sensitivity (ASe) were used to evaluate model performance. Six WSI-level pathological images (three OPA and three non-OPA images), not included in the dataset, were used for anti-peeking model validation. A random selection of 500 images, not included in the dataset establishment, was used to compare the performance of the model with assessment by pathologists. Accuracy, sensitivity, specificity, and concordance rate were evaluated. The model achieved a mASp of 0.573 and an ASe of 0.745, demonstrating effective lesion detection and alignment with expert annotation. In Anti-Peeking verification, the model showed good performance in locating OPA lesions and distinguished OPA from non-OPA pathological images. In the random 500-image diagnosis, the model achieved 92.8% accuracy, 100% sensitivity, and 88% specificity. The agreement rates between junior and senior pathologists were 100% and 96.5%, respectively. In conclusion, the Mask R-CNN-based OPA diagnostic model developed for OPA facilitates rapid and accurate diagnosis in practical applications. Full article

The health monitoring of vegetable and fruit plants, especially during the critical seedling growth stage, is essential to protect them from various environmental stresses and prevent yield loss. Different environmental stresses may cause similar symptoms, making visual inspection alone unreliable and potentially leading to an incorrect diagnosis and delayed corrective actions. This study aimed to address these challenges by proposing a segmented regression model and a Mask R-CNN model for detecting the initiation time and symptoms of nutrient stress in cucumber seedlings within a controlled environment. Nutrient stress was induced by applying two different treatments: an indicative nutrient deficiency with an electrical conductivity (EC) of 0 dSm⁻¹, and excess nutrients with a high-concentration nutrient solution and an EC of 6 dSm⁻¹. Images of the seedlings were collected using an automatic image acquisition system two weeks after germination. The early initiation of nutrient stress was detected using a segmented regression analysis, which analyzed morphological and textural features extracted from the images. For the Mask R-CNN model, 800 seedling images were annotated based on the segmented regression analysis results. Nutrient-stressed seedlings were identified from the initiation day to 4.2 days after treatment application. The Mask R-CNN model, implemented using ResNet-101 for feature extraction, leveraged transfer learning to train the network with a smaller dataset, thereby reducing the processing time. This study identifies the top projected canopy area (TPCA), energy, entropy, and homogeneity as prospective indicators of nutritional deficits in cucumber seedlings. The results from the Mask R-CNN model are promising, with the best-fit image achieving an F1 score of 93.4%, a precision of 93%, and a recall of 94%. These findings demonstrate the effectiveness of the integrated statistical and machine learning (ML) methods for the early and accurate diagnosis of nutrient stress. The use of segmented regression for initial detection, followed by the Mask R-CNN for precise identification, emphasizes the potential of this approach to enhance agricultural practices. By facilitating the early detection and accurate diagnosis of nutrient stress, this approach allows for quicker and more precise treatments, which improve crop health and productivity. Future research could expand this methodology to other crop types and field conditions to enhance image processing techniques, and researchers may also integrate real-time monitoring systems. Full article

The severity of periodontitis can be analyzed by calculating the loss of alveolar crest (ALC) level and the level of bone loss between the tooth’s bone and the cemento-enamel junction (CEJ). However, dentists need to manually mark symptoms on periapical radiographs (PAs) to assess bone loss, a process that is both time-consuming and prone to errors. This study proposes the following new method that contributes to the evaluation of disease and reduces errors. Firstly, innovative periodontitis image enhancement methods are employed to improve PA image quality. Subsequently, single teeth can be accurately extracted from PA images by object detection with a maximum accuracy of 97.01%. An instance segmentation developed in this study accurately extracts regions of interest, enabling the generation of masks for tooth bone and tooth crown with accuracies of 93.48% and 96.95%. Finally, a novel detection algorithm is proposed to automatically mark the CEJ and ALC of symptomatic teeth, facilitating faster accurate assessment of bone loss severity by dentists. The PA image database used in this study, with the IRB number 02002030B0 provided by Chang Gung Medical Center, Taiwan, significantly reduces the time required for dental diagnosis and enhances healthcare quality through the techniques developed in this research. Full article

Objects in cluttered environments may have similar sizes and shapes, which remains a huge challenge for robot grasping manipulation. The existing segmentation methods, such as Mask R-CNN and Yolo-v8, tend to lose the shape details of objects when dealing with messy scenes, and this loss of detail limits the grasp performance of robots in complex environments. This paper proposes a high-performance grasp detection algorithm with a multi-target semantic segmentation model, which can effectively improve a robot’s grasp success rate in cluttered environments. The algorithm consists of two cascades: Semantic Segmentation and Grasp Detection modules (SS-GD), in which the backbone network of the semantic segmentation module is developed by using the state-of-the-art Swin Transformer structure. It can extract the detailed features of objects in cluttered environments and enable a robot to understand the position and shape of the candidate object. To construct the grasp schema SS-GD focused on important vision features, a grasp detection module is designed based on the Squeeze-and-Excitation (SE) attention mechanism, to predict the corresponding grasp configuration accurately. The grasp detection experiments were conducted on an actual UR5 robot platform to verify the robustness and generalization of the proposed SS-GD method in cluttered environments. A best grasp success rate of 91.7% was achieved for cluttered multi-target workspaces. Full article

The accurate distribution of joints on the tunnel face is crucial for assessing the stability and safety of surrounding rock during tunnel construction. This paper introduces the Mask R-CNN image segmentation algorithm, a state-of-the-art deep learning model, to achieve efficient and accurate identification and extraction of joints on tunnel face images. First, digital images of tunnel faces were captured and stitched, resulting in 286 complete images suitable for analysis. Then, the joints on the tunnel face were extracted using traditional image processing algorithms, the commonly used U-net image segmentation model, and the Mask R-CNN image segmentation model introduced in this paper to address the lack of recognition accuracy. Finally, the extraction results obtained by the three methods were compared. The comparison results show that the joint extraction method based on the Mask R-CNN image segmentation deep learning model introduced in this paper achieved the best joint extraction effect with a Dice similarity coefficient of 87.48%, outperforming traditional methods and the U-net model, which scored 60.59% and 75.36%, respectively, realizing accurate and efficient acquisition of tunnel face rock joints. These findings suggest that the Mask R-CNN model can be effectively implemented in real-time monitoring systems for tunnel construction projects. Full article

Using computer vision technology to estimate pig live weight is an important method to realize pig welfare. But there are two key issues that affect pigs’ weight estimation: one is the uneven illumination, which leads to unclear contour extraction of pigs, and the other is the bending of the pig body, which leads to incorrect pig body information. For the first one, Mask R-CNN was used to extract the contour of the pig, and the obtained mask image was converted into a binary image from which we were able to obtain a more accurate contour image. For the second one, the body length, hip width and the distance from the camera to the pig back were corrected by XGBoost and actual measured information. Then we analyzed the rationality of the extracted features. Three feature combination strategies were used to predict pig weight. In total, 1505 back images of 39 pigs obtained using Azure kinect DK were used in the numerical experiments. The highest prediction accuracy is XGBoost, with an MAE of 0.389, RMSE of 0.576, MAPE of 0.318% and $R^2$ of 0.995. We also recommend using the Mask R-CNN + RFR method because it has fairly high precision in each strategy. The experimental results show that our proposed method has excellent performance in live weight estimation of pigs. Full article

This study addresses the challenges inherent in the maintenance and inspection of wind turbines through the application of deep learning methodologies for fault detection on Wind Turbine Blades (WTBs). Specifically, this research focuses on defect detection on the blades of small-scale WTBs due to the unavailability of commercial wind turbines. This research compared popular object localization architectures, YOLO and Mask R-CNN, to identify the most effective model to detect common WTB defects, including cracks, holes, and erosion. YOLOv9 C emerged as the most effective model, with the highest scores of mAP50 and mAP50-95 of 0.849 and 0.539, respectively. Modifications to Mask R-CNN, specifically integrating a ResNet18-FPN network, reduced computational complexity by 32 layers and achieved a mAP50 of 0.8415. The findings highlight the potential of deep learning and computer vision in improving WTB fault analysis and inspection. Full article

The ability to recognize the body sizes of sheep is significantly influenced by posture, especially without artificial fixation, leading to more noticeable changes. This study presents a recognition model using the Mask R-CNN convolutional neural network to identify the sides and backs of sheep. The proposed approach includes an algorithm for extracting key frames through mask calculation and specific algorithms for head-down, head-up, and jumping postures of Ujumqin sheep. The study reported an accuracy of 94.70% in posture classification. We measured the body size parameters of Ujumqin sheep of different sexes and in different walking states, including observations of head-down and head-up. The errors for the head-down position of rams, in terms of body slanting length, withers height, hip height, and chest depth, were recorded as 0.08 ± 0.06, 0.09 ± 0.07, 0.07 ± 0.05, and 0.12 ± 0.09, respectively. For rams in the head-up position, the corresponding errors were 0.06 ± 0.05, 0.06 ± 0.05, 0.07 ± 0.05, and 0.13 ± 0.07, respectively. The errors for the head-down position of ewes, in terms of body slanting length, withers height, hip height, and chest depth, were recorded as 0.06 ± 0.05, 0.09 ± 0.08, 0.07 ± 0.06, and 0.13 ± 0.10, respectively. For ewes in the head-up position, the corresponding errors were 0.06 ± 0.05, 0.08 ± 0.06, 0.06 ± 0.04, and 0.16 ± 0.12, respectively. The study observed that sheep walking through a passage exhibited a more curved knee posture compared to normal measurements, often with a lowered head. This research presents a cost-effective data collection scheme for studying multiple postures in animal husbandry. Full article

The lack of spatial pose information and the low positioning accuracy of the picking target are the key factors affecting the picking function of citrus-picking robots. In this paper, a new method for automatic citrus fruit harvest is proposed, which uses semantic segmentation and rotating target detection to estimate the pose of a single culture. First, Faster R-CNN is used for grab detection to identify candidate grab frames. At the same time, the semantic segmentation network extracts the contour information of the citrus fruit to be harvested. Then, the capture frame with the highest confidence is selected for each target fruit using the semantic segmentation results, and the rough angle is estimated. The network uses image-processing technology and a camera-imaging model to further segment the mask image of the fruit and its epiphyllous branches and realize the fitting of contour, fruit centroid, and fruit minimum outer rectangular frame and three-dimensional boundary frame. The positional relationship of the citrus fruit to its epiphytic branches was used to estimate the three-dimensional pose of the citrus fruit. The effectiveness of the method was verified through citrus-planting experiments, and then field picking experiments were carried out in the natural environment of orchards. The results showed that the success rate of citrus fruit recognition and positioning was 93.6%, the average attitude estimation angle error was 7.9°, and the success rate of picking was 85.1%. The average picking time is 5.6 s, indicating that the robot can effectively perform intelligent picking operations. Full article

The study of the intricate flight patterns and behaviors of swarm insects, such as drosophilas, has long been a subject of interest in both the biological and computational realms. Tracking drosophilas is an essential and indispensable method for researching drosophilas’ behaviors. Still, it remains a challenging task due to the highly dynamic nature of these drosophilas and their partial occlusion in multi-target environments. To address these challenges, particularly in environments where multiple targets (drosophilas) interact and overlap, we have developed a long-term Trajectory 3D Position and Orientation Tracking Method (Long 3D-POT) that combines deep learning with particle filtering. Our approach employs a detection model based on an improved Mask-RCNN to accurately detect the position and state of drosophilas from frames, even when they are partially occluded. Following detection, improved particle filtering is used to predict and update the motion of the drosophilas. To further enhance accuracy, we have introduced a prediction module based on the self-attention backbone that predicts the drosophila’s next state and updates the particles’ weights accordingly. Compared with previous methods by Ameni, Cheng, and Wang, our method has demonstrated a higher degree of accuracy and robustness in tracking the long-term trajectories of drosophilas, even those that are partially occluded. Specifically, Ameni employs the Interacting Multiple Model (IMM) combined with the Global Nearest Neighbor (GNN) assignment algorithm, primarily designed for tracking larger, more predictable targets like aircraft, which tends to perform poorly with small, fast-moving objects like drosophilas. The method by Cheng then integrates particle filtering with LSTM networks to predict particle weights, enhancing trajectory prediction under kinetic uncertainties. Wang’s approach builds on Cheng’s by incorporating an estimation of the orientation of drosophilas in order to refine tracking further. Compared with those methods, our method performs with higher accuracy on detection, which increases by more than 10% on the F1 Score, and tracks more long-term trajectories, showing stability. Full article

This study represents a significant advancement in structural health monitoring by integrating infrared thermography (IRT) with cutting-edge deep learning techniques, specifically through the use of the Mask R-CNN neural network. This approach targets the precise detection and segmentation of hidden defects within the interfacial layers of Fiber-Reinforced Polymer (FRP)-reinforced concrete structures. Employing a dual RGB and thermal camera setup, we captured and meticulously aligned image data, which were then annotated for semantic segmentation to train the deep learning model. The fusion of the RGB and thermal imaging significantly enhanced the model’s capabilities, achieving an average accuracy of 96.28% across a 5-fold cross-validation. The model demonstrated robust performance, consistently identifying true negatives with an average specificity of 96.78% and maintaining high precision at 96.42% in accurately delineating damaged areas. It also showed a high recall rate of 96.91%, effectively recognizing almost all actual cases of damage, which is crucial for the maintenance of structural integrity. The balanced precision and recall culminated in an average F1-score of 96.78%, highlighting the model’s effectiveness in comprehensive damage assessment. Overall, this synergistic approach of combining IRT and deep learning provides a powerful tool for the automated inspection and preservation of critical infrastructure components. Full article

Quickly and accurately assessing the damage level of buildings is a challenging task for post-disaster emergency response. Most of the existing research mainly adopts semantic segmentation and object detection methods, which have yielded good results. However, for high-resolution Unmanned Aerial Vehicle (UAV) imagery, these methods may result in the problem of various damage categories within a building and fail to accurately extract building edges, thus hindering post-disaster rescue and fine-grained assessment. To address this issue, we proposed an improved instance segmentation model that enhances classification accuracy by incorporating a Mixed Local Channel Attention (MLCA) mechanism in the backbone and improving small object segmentation accuracy by refining the Neck part. The method was tested on the Yangbi earthquake UVA images. The experimental results indicated that the modified model outperformed the original model by 1.07% and 1.11% in the two mean Average Precision (mAP) evaluation metrics,${mAP}_{bbox}^{50}$ and ${mAP}_{seg}^{50}$, respectively. Importantly, the classification accuracy of the intact category was improved by 2.73% and 2.73%, respectively, while the collapse category saw an improvement of 2.58% and 2.14%. In addition, the proposed method was also compared with state-of-the-art instance segmentation models, e.g., Mask-R-CNN and YOLO V9-Seg. The results demonstrated that the proposed model exhibits advantages in both accuracy and efficiency. Specifically, the efficiency of the proposed model is three times faster than other models with similar accuracy. The proposed method can provide a valuable solution for fine-grained building damage evaluation. Full article

Concrete crack identification methods based on machine learning can greatly improve extraction efficiency and precision. However, in many cases, model training requires a large amount of sample data, and insufficient data makes it difficult to effectively obtain model parameters. This study introduces a deep learning framework that integrates filters, principal component analysis, and attention mechanisms suitable for small sample sizes. Firstly, the histogram equalization method is used for the raw images, which can effectively enhance image contrast. Then, to acquire effective images of the crack, different methods are employed for crack detection, which are subsequently handled by principal component analysis (PCA) for optimal feature choice. Att-Unet and Att-Mask R-cnn segmentation models are used to design the detection for concrete cracks. To raise the learning ability of the segmentation models, an attention mechanism is applied to each feature layer of the decoder, and the loss function is evaluated using a combination of the Focal function and Cross Entropy. To verify the effectiveness of the proposed method, Deep Crack datasets and 76 sets of concrete crack data were collected for testing. Experimental results have shown that the method proposed can significantly reduce the model’s demand for data volume and improve training speed, which provides a new direction for small-sample crack extraction. Full article

The accurate instance segmentation of individual crop plants is crucial for achieving a high-throughput phenotypic analysis of seedlings and smart field management in agriculture. Current crop monitoring techniques employing remote sensing predominantly focus on population analysis, thereby lacking precise estimations for individual plants. This study concentrates on maize, a critical staple crop, and leverages multispectral remote sensing data sourced from unmanned aerial vehicles (UAVs). A large-scale SAM image segmentation model is employed to efficiently annotate maize plant instances, thereby constructing a dataset for maize seedling instance segmentation. The study evaluates the experimental accuracy of six instance segmentation algorithms: Mask R-CNN, Cascade Mask R-CNN, PointRend, YOLOv5, Mask Scoring R-CNN, and YOLOv8, employing various combinations of multispectral bands for a comparative analysis. The experimental findings indicate that the YOLOv8 model exhibits exceptional segmentation accuracy, notably in the NRG band, with bbox_mAP50 and segm_mAP50 accuracies reaching 95.2% and 94%, respectively, surpassing other models. Furthermore, YOLOv8 demonstrates robust performance in generalization experiments, indicating its adaptability across diverse environments and conditions. Additionally, this study simulates and analyzes the impact of different resolutions on the model’s segmentation accuracy. The findings reveal that the YOLOv8 model sustains high segmentation accuracy even at reduced resolutions (1.333 cm/px), meeting the phenotypic analysis and field management criteria. Full article