Search Results (48)

With the continuous development and construction of new power systems, using drones to inspect the condition of transmission line insulators has become an inevitable trend. To facilitate the deployment of drone hardware equipment, this paper proposes IDD-YOLO (Insulator Defect Detection-YOLO), a lightweight insulator defect detection model. Initially, the backbone network of IDD-YOLO employs GhostNet for feature extraction. However, due to the limited feature extraction capability of GhostNet, we designed a lightweight attention mechanism called LCSA (Lightweight Channel-Spatial Attention), which is combined with GhostNet to capture features more comprehensively. Secondly, the neck network of IDD-YOLO utilizes PANet for feature transformation and introduces GSConv and C3Ghost convolution modules to reduce redundant parameters and lighten the network. The head network employs the YOLO detection head, incorporating the EIOU loss function and Mish activation function to optimize the speed and accuracy of insulator defect detection. Finally, the model is optimized using TensorRT and deployed on the NVIDIA Jetson TX2 NX mobile platform to test the actual inference speed of the model. The experimental results demonstrate that the model exhibits outstanding performance on both the proprietary ID-2024 insulator defect dataset and the public SFID insulator dataset. After optimization with TensorRT, the actual inference speed of the IDD-YOLO model reached 20.83 frames per second (FPS), meeting the demands for accurate and real-time inspection of insulator defects by drones. Full article

During aerial combat, when an aircraft is facing an infrared air-to-air missile strike, infrared baiting technology is an important means of penetration, and the strategy of effective delivery of infrared bait is critical. To address this issue, this study proposes an improved deep deterministic policy gradient (DDPG) algorithm-based intelligent bait-dropping control method. Firstly, by modeling the relative motion between aircraft, bait, and incoming missiles, the Markov decision process of aircraft-bait-missile infrared effect was constructed with visual distance and line of sight angle as states. Then, the DDPG algorithm was improved by means of pre-training and classification sampling. Significantly, the infrared bait-dropping decision network was trained through interaction with the environment and iterative learning, which led to the development of the bait-dropping strategy. Finally, the corresponding environment was transferred to the Nvidia Jetson TX2 embedded platform for comparative testing. The simulation results showed that the convergence speed of this method was 46.3% faster than the traditional DDPG algorithm. More importantly, it was able to generate an effective bait-throwing strategy, enabling the aircraft to successfully evade the attack of the incoming missile. The strategy instruction generation time is only about 2.5 ms, giving it the ability to make online decisions. Full article

To effectively combat the re-infestation of boll weevils (Anthonomus grandis L.) in cotton fields, it is necessary to address the detection of volunteer cotton (VC) plants (Gossypium hirsutum L.) in rotation crops such as corn (Zea mays L.) and sorghum (Sorghum bicolor L.). The current practice involves manual field scouting at the field edges, which often leads to the oversight of VC plants growing in the middle of fields alongside corn and sorghum. As these VC plants reach the pinhead squaring stage (5–6 leaves), they can become hosts for boll weevil pests. Consequently, it becomes crucial to detect, locate, and accurately spot-spray these plants with appropriate chemicals. This paper focuses on the application of YOLOv5m to detect and locate VC plants during the tasseling (VT) growth stage of cornfields. Our results demonstrate that VC plants can be detected with a mean average precision (mAP) of 79% at an Intersection over Union (IoU) of 50% and a classification accuracy of 78% on images sized 1207 × 923 pixels. The average detection inference speed is 47 frames per second (FPS) on the NVIDIA Tesla P100 GPU-16 GB and 0.4 FPS on the NVIDIA Jetson TX2 GPU, which underscores the relevance and impact of detection speed on the feasibility of real-time applications. Additionally, we show the application of a customized unmanned aircraft system (UAS) for spot-spray applications through simulation based on the developed computer vision (CV) algorithm. This UAS-based approach enables the near-real-time detection and mitigation of VC plants in corn fields, with near-real-time defined as approximately 0.02 s per frame on the NVIDIA Tesla P100 GPU and 2.5 s per frame on the NVIDIA Jetson TX2 GPU, thereby offering an efficient management solution for controlling boll weevil pests. Full article

With the continuous evolution of autonomous driving and unmanned driving systems, traditional limitations such as a limited field-of-view, poor ranging accuracy, and real-time display are becoming inadequate to satisfy the requirements of binocular stereo-perception systems. Firstly, we designed a binocular stereo-imaging-perception system with a wide-field-of-view and infrared- and visible light-dual-band fusion. Secondly we proposed a binocular stereo-perception optical imaging system with a wide field-of-view of 120.3°, which solves the small field-of-view of current binocular stereo-perception systems. Thirdly, For image aberration caused by the wide-field-of-view system design, we propose an ellipsoidal-image-aberration algorithm with a low consumption of memory resources and no loss of field-of-view. This algorithm simultaneously solves visible light and infrared images with an aberration rate of 45% and 47%, respectively. Fourthly, a multi-scale infrared- and visible light-image-fusion algorithm is used, which improves the situational-awareness capabilities of a binocular stereo-sensing system in a scene and enhances image details to improve ranging accuracy. Furthermore, this paper is based on the Taylor model-calibration binocular stereo-sensing system of internal and external parameters for limit correction; the implemented algorithms are integrated into an NVIDIA Jetson TX2 + FPGA hardware framework, enabling near-distance ranging experiments. The fusion-ranging accuracy within 20 m achieved an error of 0.02 m, outperforming both visible light- and infrared-ranging methods. It generates the fusion-ranging-image output with a minimal delay of only 22.31 ms at a frame rate of 50 Hz. Full article

There is an exciting synergy between deep learning and robotics, combining the perception skills a deep learning system can achieve with the wide variety of physical responses a robot can perform. This article describes an embedded system integrated into a mobile robot capable of identifying and following a specific person reliably based on a convolutional neural network pipeline. In addition, the design incorporates an optical tracking system for supporting the inferences of the neural networks, allowing the determination of the position of a person using an RGB depth camera. The system runs on an NVIDIA Jetson TX2 board, an embedded System-on-Module capable of performing computationally demanding tasks onboard and handling the complexity needed to run a solid tracking and following algorithm. A robotic mobile base with the Jetson TX2 board attached receives velocity orders to move the system toward the target. The proposed approach has been validated on a mobile robotic platform that successfully follows a determined person, relying on the robustness of the combination of deep learning with optical tracking for working efficiently in a real environment. Full article

A launch vehicle needs to adapt to a complex flight environment during flight, and traditional guidance and control algorithms can hardly deal with multi-factor uncertainties due to the high dependency on control models. To solve this problem, this paper designs a new intelligent flight control method for a rocket based on the deep reinforcement learning algorithm driven by knowledge and data. In this process, the Markov decision process of the rocket landing section is established by designing a reinforcement function with consideration of the combination effect on the return of the terminal constraint of the launch vehicle and the cumulative return of the flight process of the rocket. Meanwhile, to improve the training speed of the landing process of the launch vehicle and to enhance the generalization ability of the model, the strategic neural network model is obtained and trained via the form of a long short-term memory (LSTM) network combined with a full connection layer as a landing guidance strategy network. The proximal policy optimization (PPO) is the training algorithm of reinforcement learning network parameters combined with behavioral cloning (BC) as the reinforcement learning pre-training imitation learning algorithm. Notably, the rocket-borne environment is transplanted to the Nvidia Jetson TX2 embedded platform for the comparative testing and verification of this intelligent model, which is then used to generate real-time control commands for guiding the actual flying and landing process of the rocket. Further, comparisons of the results obtained from convex landing optimization and the proposed method in this work are performed to prove the effectiveness of this proposed method. The simulation results show that the intelligent control method in this work can meet the landing accuracy requirements of the launch vehicle with a fast convergence speed of 84 steps, and the decision time is only 2.5 ms. Additionally, it has the ability of online autonomous decision making as deployed on the embedded platform. Full article

Over the past several years, significant progress has been made in object tracking, but challenges persist in tracking objects in high-resolution images captured from drones. Such images usually contain very tiny objects, and the movement of the drone causes rapid changes in the scene. In addition, the computing power of mission computers on drones is often insufficient to achieve real-time processing of deep learning-based object tracking. This paper presents a real-time on-drone pedestrian tracker that takes as the input 4K aerial images. The proposed tracker effectively hides the long latency required for deep learning-based detection (e.g., YOLO) by exploiting both the CPU and GPU equipped in the mission computer. We also propose techniques to minimize detection loss in drone-captured images, including a tracker-assisted confidence boosting and an ensemble for identity association. In our experiments, using real-world inputs captured by drones at a height of 50 m, the proposed method with an NVIDIA Jetson TX2 proves its efficacy by achieving real-time detection and tracking in 4K video streams. Full article

In order to address the issue of environmental perception in autonomous driving on structured roads, we propose MultiNet-GS, a convolutional neural network model based on an encoder–decoder architecture that tackles multiple tasks simultaneously. We use the main structure of the latest object detection model, the YOLOv8 model, as the encoder structure of our model. We introduce a new dynamic sparse attention mechanism, BiFormer, in the feature extraction part of the model to achieve more flexible computing resource allocation, which can significantly improve the computational efficiency and occupy a small computational overhead. We introduce a lightweight convolution, GSConv, in the feature fusion part of the network, which is used to build the neck part into a new slim-neck structure so as to reduce the computational complexity and inference time of the detector. We also add an additional detector for tiny objects to the conventional three-head detector structure. Finally, we introduce a lane detection method based on guide lines in the lane detection part, which can aggregate the lane feature information into multiple key points, obtain the lane heat map response through conditional convolution, and then describe the lane line through the adaptive decoder, which effectively makes up for the shortcomings of the traditional lane detection method. Our comparative experiments on the BDD100K dataset on the embedded platform NVIDIA Jetson TX2 show that compared with SOTA(YOLOPv2), the [email protected] of the model in traffic object detection reaches 82.1%, which is increased by 2.7%. The accuracy of the model in drivable area detection reaches 93.2%, which is increased by 0.5%. The accuracy of the model in lane detection reaches 85.7%, which is increased by 4.3%. The Params and FLOPs of the model reach 47.5 M and 117.5, which are reduced by 6.6 M and 8.3, respectively. The model achieves 72 FPS, which is increased by 5. Our MultiNet-GS model has the highest detection accuracy among the current mainstream models while maintaining a good detection speed and has certain superiority. Full article

With the accelerated growth of the UAV industry, researchers are paying close attention to the flight safety of UAVs. When a UAV loses its GPS signal or encounters unusual conditions, it must perform an emergency landing. Therefore, real-time recognition of emergency landing zones on the ground is an important research topic. This paper employs a semantic segmentation approach for recognizing emergency landing zones. First, we created a dataset of UAV aerial images, denoted as UAV-City. A total of 600 UAV aerial images were densely annotated with 12 semantic categories. Given the complex backgrounds, diverse categories, and small UAV aerial image targets, we propose the STDC-CT real-time semantic segmentation network for UAV recognition of emergency landing zones. The STDC-CT network is composed of three branches: detail guidance, small object attention extractor, and multi-scale contextual information. The fusion of detailed and contextual information branches is guided by small object attention. We conducted extensive experiments on the UAV-City, Cityscapes, and UAVid datasets to demonstrate that the STDC-CT method is superior for attaining a balance between segmentation accuracy and inference speed. Our method improves the segmentation accuracy of small objects and achieves 76.5% mIoU on the Cityscapes test set at 122.6 FPS, 68.4% mIoU on the UAVid test set, and 67.3% mIoU on the UAV-City dataset at 196.8 FPS on an NVIDIA RTX 2080Ti GPU. Finally, we deployed the STDC-CT model on Jetson TX2 for testing in a real-world environment, attaining real-time semantic segmentation with an average inference speed of 58.32 ms per image. Full article

This paper proposes the design of a 360° map establishment and real-time simultaneous localization and mapping (SLAM) algorithm based on equirectangular projection. All equirectangular projection images with an aspect ratio of 2:1 are supported for input image types of the proposed system, allowing an unlimited number and arrangement of cameras. Firstly, the proposed system uses dual back-to-back fisheye cameras to capture 360° images, followed by the adoption of the perspective transformation with any yaw degree given to shrink the feature extraction area in order to reduce the computational time, as well as retain the 360° field of view. Secondly, the oriented fast and rotated brief (ORB) feature points extracted from perspective images with a GPU acceleration are used for tracking, mapping, and camera pose estimation in the system. The 360° binary map supports the functions of saving, loading, and online updating to enhance the flexibility, convenience, and stability of the 360° system. The proposed system is also implemented on an nVidia Jetson TX2 embedded platform with 1% accumulated RMS error of 250 m. The average performance of the proposed system achieves 20 frames per second (FPS) in the case with a single-fisheye camera of resolution 1024 × 768, and the system performs panoramic stitching and blending under 1416 × 708 resolution from a dual-fisheye camera at the same time. Full article

Autonomous vehicles require efficient self-localisation mechanisms and cameras are the most common sensors due to their low cost and rich input. However, the computational intensity of visual localisation varies depending on the environment and requires real-time processing and energy-efficient decision-making. FPGAs provide a solution for prototyping and estimating such energy savings. We propose a distributed solution for implementing a large bio-inspired visual localisation model. The workflow includes (1) an image processing IP that provides pixel information for each visual landmark detected in each captured image, (2) an implementation of N-LOC, a bio-inspired neural architecture, on an FPGA board and (3) a distributed version of N-LOC with evaluation on a single FPGA and a design for use on a multi-FPGA platform. Comparisons with a pure software solution demonstrate that our hardware-based IP implementation yields up to $9\times$ lower latency and $7\times$ higher throughput (frames/second) while maintaining energy efficiency. Our system has a power footprint as low as 2.741 W for the whole system, which is up to 5.5–6× less than what Nvidia Jetson TX2 consumes on average. Our proposed solution offers a promising approach for implementing energy-efficient visual localisation models on FPGA platforms. Full article

Nowadays, Brain–Computer Interfaces (BCIs) still captivate large interest because of multiple advantages offered in numerous domains, explicitly assisting people with motor disabilities in communicating with the surrounding environment. However, challenges of portability, instantaneous processing time, and accurate data processing remain for numerous BCI system setups. This work implements an embedded multi-tasks classifier based on motor imagery using the EEGNet network integrated into the NVIDIA Jetson TX2 card. Therefore, two strategies are developed to select the most discriminant channels. The former uses the accuracy based-classifier criterion, while the latter evaluates electrode mutual information to form discriminant channel subsets. Next, the EEGNet network is implemented to classify discriminant channel signals. Additionally, a cyclic learning algorithm is implemented at the software level to accelerate the model learning convergence and fully profit from the NJT2 hardware resources. Finally, motor imagery Electroencephalogram (EEG) signals provided by HaLT’s public benchmark were used, in addition to the k-fold cross-validation method. Average accuracies of 83.7% and 81.3% were achieved by classifying EEG signals per subject and motor imagery task, respectively. Each task was processed with an average latency of 48.7 ms. This framework offers an alternative for online EEG-BCI systems’ requirements, dealing with short processing times and reliable classification accuracy. Full article

The latest advancements in satellite technology have allowed us to obtain video imagery from satellites. Nanosatellites are becoming widely used for earth-observing missions as they require a low budget and short development time. Thus, there is a real interest in using nanosatellites with a video payload camera, especially for disaster monitoring and fleet tracking. However, as video data requires much storage and high communication costs, it is challenging to use nanosatellites for such missions. This paper proposes an effective onboard deep-learning-based video scene analysis method to reduce the high communication cost. The proposed method will train a CNN+LSTM-based model to identify mission-related sceneries such as flood-disaster-related scenery from satellite videos on the ground and then load the model onboard the nanosatellite to perform the scene analysis before sending the video data to the ground. We experimented with the proposed method using Nvidia Jetson TX2 as OBC and achieved an 89% test accuracy. Additionally, by implementing our approach, we can minimize the nanosatellite video data download cost by 30% which allows us to send the important mission video payload data to the ground using S-band communication. Therefore, we believe that our new approach can be effectively applied to obtain large video data from a nanosatellite. Full article

This paper presents a benchmark analysis of NVIDIA Jetson platforms when operating deep learning-based 3D object detection frameworks. Three-dimensional (3D) object detection could be highly beneficial for the autonomous navigation of robotic platforms, such as autonomous vehicles, robots, and drones. Since the function provides one-shot inference that extracts 3D positions with depth information and the heading direction of neighboring objects, robots can generate a reliable path to navigate without collision. To enable the smooth functioning of 3D object detection, several approaches have been developed to build detectors using deep learning for fast and accurate inference. In this paper, we investigate 3D object detectors and analyze their performance on the NVIDIA Jetson series that contain an onboard graphical processing unit (GPU) for deep learning computation. Since robotic platforms often require real-time control to avoid dynamic obstacles, onboard processing with a built-in computer is an emerging trend. The Jetson series satisfies such requirements with a compact board size and suitable computational performance for autonomous navigation. However, a proper benchmark that analyzes the Jetson for a computationally expensive task, such as point cloud processing, has not yet been extensively studied. In order to examine the Jetson series for such expensive tasks, we tested the performance of all commercially available boards (i.e., Nano, TX2, NX, and AGX) with state-of-the-art 3D object detectors. We also evaluated the effect of the TensorRT library to optimize a deep learning model for faster inference and lower resource utilization on the Jetson platforms. We present benchmark results in terms of three metrics, including detection accuracy, frame per second (FPS), and resource usage with power consumption. From the experiments, we observe that all Jetson boards, on average, consume over 80% of GPU resources. Moreover, TensorRT could remarkably increase inference speed (i.e., four times faster) and reduce the central processing unit (CPU) and memory consumption in half. By analyzing such metrics in detail, we establish research foundations on edge device-based 3D object detection for the efficient operation of various robotic applications. Full article

The rapid spread of coronavirus disease 2019 (COVID-19) has posed enormous challenges to the global public health system. To deal with the COVID-19 pandemic crisis, the more accurate and convenient diagnosis of patients needs to be developed. This paper proposes a deep-learning-based COVID-19 detection method and evaluates its performance on embedded edge-computing devices. By adding an attention module and mixed loss into the original VGG19 model, the method can effectively reduce the parameters of the model and increase the classification accuracy. The improved model was first trained and tested on the PC X86 GPU platform using a large dataset (COVIDx CT-2A) and a medium dataset (integrated CT scan); the weight parameters of the model were reduced by around six times compared to the original model, but it still approximately achieved 98.80%and 97.84% accuracy, outperforming most existing methods. The trained model was subsequently transferred to embedded NVIDIA Jetson devices (TX2, Nano), where it achieved 97% accuracy at a 0.6−1 FPS inference speed using the NVIDIA TensorRT engine. The experimental results demonstrate that the proposed method is practicable and convenient; it can be used on a low-cost medical edge-computing terminal. The source code is available on GitHub for researchers. Full article