Sign Language Sentence Recognition Using Hybrid Graph Embedding and Adaptive Convolutional Networks

Abstract

Sign language plays a crucial role in bridging communication barriers within the Deaf community. Recognizing sign language sentences remains a significant challenge due to their complex structure, variations in signing styles, and temporal dynamics. This study introduces an innovative sign language sentence recognition (SLSR) approach using Hybrid Graph Embedding and Adaptive Convolutional Networks (HGE-ACN) specifically developed for single-handed wearable glove devices. The system relies on sensor data from a glove with six-axis inertial sensors and five-finger curvature sensors. The proposed HGE-ACN framework integrates graph-based embeddings to capture dynamic spatial–temporal relationships in motion and curvature data. At the same time, the Adaptive Convolutional Networks extract robust glove-based features to handle variations in signing speed, transitions between gestures, and individual signer styles. The lightweight design enables real-time processing and enhances recognition accuracy, making it suitable for practical use. Extensive experiments demonstrate that HGE-ACN achieves superior accuracy and computational efficiency compared to existing glove-based recognition methods. The system maintains robustness under various conditions, including inconsistent signing speeds and environmental noise. This work has promising applications in real-time assistive tools, educational technologies, and human–computer interaction systems, facilitating more inclusive and accessible communication platforms for the deaf and hard-of-hearing communities. Future work will explore multi-lingual sign language recognition and real-world deployment across diverse environments.

1. Background and Significance

Despite technological advances, deaf and hard-of-hearing people face difficulties with language [1]. Deaf people worldwide face important communication gaps because traditional communication tools struggle to understand sign language’s multidimensionality [2]. Computer-sensor-based machine learning and assistive technology are used in sign language recognition (SLR). Unlike spoken language translation, decoding complex visual-spatial linguistic structures in sign language recognition requires advanced computing methods [3]. Automatic recognition is computationally intensive and complex due to signing styles, environmental elements, and contextual fluctuations [4]. Recent advances in deep learning and neural network topologies have enabled more accurate and resilient sign language recognition systems. SLR could improve accessibility for deaf people in healthcare, education, social interaction, and the workplace [5].

1.1. Advanced Feature Extraction and Representation

Build a sophisticated computer system that can interpret sign language using multidimensional, intricate features [6]. The primary objective is to create a hybrid method to understand sign language words with all their semantic nuances, temporal dynamics, and geographical links extracted. By combining graph-based embeddings with Adaptive Convolutional Networks, the study hopes to solve the problem of standard recognition restrictions and handle variations in signing velocity, individual styles, and environmental circumstances [7].

1.2. Performance Enhancement and Robustness

Improve sign language sentence recognition accuracy and processing efficiency with novel algorithms [8]. This study seeks to create a model that performs well across varied datasets, minimizes recognition errors, and handles complicated linguistic scenarios [9]. This goal involves building adaptive algorithms to simplify signing settings to create a more reliable and versatile recognition system for real-world communication [10].

1.3. Accessibility and Technological Innovation

Assist with the study and development of accessible communication tools that may improve social interactions for individuals that are deaf or hard of hearing [11]. The study aims to develop more precise and flexible sign language recognition systems to create technological solutions that go over translation, promoting true communication accessibility. This goal highlights the importance of new technology as well as the more significant social effects of assisting deaf people in their educational, occupational, and social integration, as well as making communication more inclusive [12].

1.4. Significance

This study seeks to create reliable and flexible systems for recognizing sign language to close the gap in assistive communication technology [13]. This method achieves linguistic equity by acknowledging sign language as a multi-faceted communication system. Advanced sign language recognition systems have far-reaching social and technological consequences, and this study offers a thorough theoretical framework that highlights both. For the deaf community, it seeks to build truly inclusive communication platforms.

The progression learning convolution neural model demonstrated significant improvements in the credit of sign language gestures using data from wearable glove devices, showcasing its potential for sentence-level translation [14]. The development of a sensor data fusion framework combined with an optimized Elman neural model effectively enhanced the robustness and accuracy of sign language recognition systems [15].

The study provides an innovative approach to sign language recognition using HGE-ACN, which stands for Hybrid Graph Embedding and Adaptive Convolutional Networks. The process includes adaptable convolutional frameworks, multimodal feature extraction, graph-based embedding methods, and extensive validation on several standard sign language datasets. This approach aims to improve sign language recognition and capture complicated spatial–temporal interactions. The key objective points of the study are as follows:

To develop an AI-based framework for glove-based sign language sentence recognition using HGE-ACN to process data from wearable glove devices, focusing on six-axis inertial sensors and five-finger curvature sensors and eliminating sensor-based-based inputs.

To optimize sentence-level recognition using graph-based embeddings and adaptive convolution to solve difficulties including signing speed, signer styles, and hand gesture transitions.

To build a scalable, real-time recognition system for educational tools, accessibility technologies, and human–computer interaction with multilingual and real-world deployment capabilities to promote inclusive communication systems.

An overview of the research follows. Section 2 describes a comprehensive literature and research methodology review. Section 3 describes the research plan, methodologies, and processing. Section 4 describes the results of the analysis. The conclusion and future work are in Section 5.

2. Literature Survey

Venugopalan and Reghunadhan [16] presented that as COVID-19 disrupts clinical services nationwide, deaf patients struggle to communicate for care. The study demonstrates the way automatic sign language identification helps Indian deaf individuals and healthcare workers interact. A new dataset of dynamic hand gestures for ISL words averages 83.36% accuracy. The model achieved 97% accuracy on an alternative ISL dataset and 99.34 ± 0.66% on a benchmarking hand gesture dataset. Yet, SLR accuracy issues remain when handling various gesture data, detecting and translating continuous gesture sequences, and recognizing sign languages with hands and body motions.

Tang et al. [17] suggested MSeqGraph, a graph-based multimodal sequential embedding network that uses implicit temporal cues and multimodal complementarity for sign language translation (SLT). The MSeqGraph model incorporates a GEU that combines channel-wise and temporal-wise, as well as multimodal, relational embedding. The GEU incorporates cross-modal edges, temporal neighborhood edges, and parallel convolutions as part of the GCN computation. Dense multimodal cues are used using a hierarchical GEU stacker. Using the USTC-CSL as well as BOSTON-104 datasets, experiments show that the suggested strategy works and that representation and performance are improved using multimodal cues.

Muthusamy and Murugan [18] provided that deaf and hard-of-hearing people use sign language (SL), although it might be difficult with strangers. Space–temporal multi-cue (STMC) networks from deep learning improve SL prediction. These networks take longer to preprocess and annotate crucial locations. This study presents a spatiotemporal hybrid cue network (STHCN) using a Dynamic Dense Spatiotemporal Graph Convolutional Neural Network (DDSTGCNN) and feature extractor network to address ISL recognition and translation tasks for sequence learning and inference, and BLSTM encoders, CTCs, and self-attention-based LSTM decoders use extracted features. The STHCN model detects ISL 93.65% accurately.

Kan et al.’s study [19], given that the deaf community relies heavily on sign language translation (SLT), employs sequence-to-sequence learning. On the other hand, sign languages use a variety of visual and manual cues to communicate. The hierarchical spatiotemporal graph neural network (HST-GNN) is a new architecture for deep learning proposed in this paper to learn such graph patterns. The network characterizes local and global graph attributes using graph convolutions and graph self-attentions with the neighborhood context. Proving the efficacy of the suggested HSTGNN, experiments were conducted on benchmark datasets such as PHOENIX-2014-T and CSL.

Xu et al. [20] debuted an isolated word sign language recognition model built on a hybrid SKResNet-TCN network. The model reduces memory and processing requirements using grouped convolution, causal convolution, and inflation convolution. Subsequently, it outperforms conventional 3D convolutional networks in terms of accuracy, using fewer parameters and requiring easier procedures. The following area of study will be continuous sign language, but the study acknowledges that there may be discrepancies between the translated results and the actual order of speech.

Gupta et al. [21] introduced Convolutional Neural Networks (CNN) and 1D convolutional layers, and Long Short-Term Memory (LSTM) networks were utilized in the development of a novel approach to the recognition of sign language. This technique offers a precise classification of sign language motions, allowing for the identification of temporal patterns and the management of long-term interdependence. Expanding recognition skills to incorporate other sign languages and enabling user customization are two future topics that have been pursued to promote accessibility and inclusivity for people with hearing problems.

Noor et al. [22] showed that Saudi Arabia only employs hearing-impaired sign language interpreters. This study’s hybrid model recognizes Arabic Sign Language using deep learning, Convolutional Neural Networks (CNN), and LSTM classifiers. Sign language spatial and hand movement sequence data are collected by the model. A total of 500 dynamic gesture word films and 4000 Arabic Sign Language (ArSL) photos from a 20-word dataset showed the model’s performance. The model’s accuracy of 82.70% and accuracy of 94.40% implied that it could help Saudi Arabia’s hearing-impaired communication.

Huang and Ye [23] suggested a new boundary-adaptive learning-based encoder-decoder approach to Chinese sign language recognition (SLR). With the help of the boundary-adaptive encoder (BAE), this technique encodes the hierarchical structure inside sign language signals. A location-based window attention model makes the decoding step more efficient. Additionally, the approach incorporates both discrete and continuous SLR in Chinese through subword units in sign language. While the approach has shown competitive results on several major benchmarks, it is not yet lightweight or optimized for real-time performance. The study’s primary limitation lies in its focus on enhancing efficiency.

Kumar et al. [24] solved sign language machine translation problems with 3D motion capture and graph matching. Three-dimensional sign matching issues include identifying the same signs in multiple motion frames and extracting signs from non-sign hand movements. Two-phase graphs that match early estimates help solve these problems. Intra-graph matching searches databases for motion-intensive frames. Early-estimation model inter-graphs match motion-derived queries and datasets. With this method, graph-matching estimates sign faster with less frames. The model is evaluated on 350 Indian sign language terms with four variations for each sign by five signers at different hand speeds and motions.

Rajalakshmi et al. [25] indicated that the public struggles to learn sign language for speech and hearing-impaired purposes. Sign language could aid interaction. Current approaches are expensive because they require worn sensors. A unique sensor-based-based hybrid deep neural net approach identifies Indian and Russian sign motions. The framework detects non-manual and manual co-articulations. Attention-based Bi-LSTM extracts temporal and sequential features, while a 3D deep neural net with atrous convolutions extracts spatial features. Modified autoencoders and hybrid attention modules select abstract and discriminative features. Models outperform present ones.

3. Proposed Hybrid Graph Embedding and Adaptive Convolution Networks

The Hybrid Graph Embedding and Adaptive Convolutional Networks (HGE-ACN) approach is a new framework for efficient sign language sentence recognition (SLSR) using data from wearable glove devices. The method uses multimodal sensor data from a single-handed glove with six-axis inertial and five-finger curvature sensors, capturing spatial–temporal dynamics critical for recognizing complex sentences. The HGE-ACN integrates graph-based embeddings to model dynamic relationships between sequential gestures and employs an adaptive convolutional framework for robust feature extraction. This hybrid HGE-ACN framework addresses variations in signing speed and seamless transitions between gestures, enhancing recognition accuracy and computational efficiency.

In Figure 1, the wearable glove HGE-ACN system recognizes sign language sentences. It uses six-axis IMU sensors to track hand movement and orientation in three dimensions and finger curvature sensors to quantify finger bending and positioning. The system extracts Inertial Features, Motion Features, and Temporal Sequences from raw sensor data. The primary processing block has two main sections: HGE (Hybrid Graph Embedding), which captures geographical graphs, temporal graphs, and feature embedding, and ACN, which dynamically alters processing settings based on input characteristics. The transition model examines sentence sign transitions.

The final output step incorporates sentence analysis, pattern recognition, and language modeling. Figure 1 depicts that sensor data flows through the system: initial sensor data go to feature extraction, extracted features go to core processing, and both contribute to final recognition. HGE-ACN framework handles sign language recognition difficulties such as multimodal integration, adaptive processing, contextual understanding, and efficiency. The key contributions are key contributions of graph embeddings for spatial–temporal relationship modeling, Adaptive Convolutional Networks for signing variants, style processing for user-independent recognition, and explicit modeling of phrase sign transitions. The system’s design covers signing speed, style, and intricate transitions between signs without predefined templates. Real-time sign language recognition, learning tools, assistive technology, and HCI systems are practical applications. The HGE-ACN framework improves wearable-based sign language recognition by eliminating camera-based systems, recognizing complete sentences more accurately, handling natural signing variations, and operating efficiently enough for real-time applications. This design meets the paper’s goal of enhancing sign language recognition accuracy and processing economy. The hybrid technique uses graph-based representations and adaptive convolutional processing to handle sign language communication’s complexities.

3.1. Mathematical Foundation

HGE-ACN recognizes sign language words using graph-based temporal modeling and Adaptive Convolutional Networks. The approach relies on the dataset, a matrix of frames, frame height, width, and channel count (e.g., RGB). N keypoints with geographic coordinates and confidence scores indicate the landmarks of each frame’s hand and stance. Matrix representations of each frame’s number of frames, height, breadth, and channels form the HGE-ACN’s mathematical supports.

To improve the mathematical foundation, the values of $x, y, \mathrm{a}\mathrm{n}\mathrm{d} c$ should be constrained within bounded intervals instead of unrestricted in $\mathbb{R}$. The Equation (1) is as follows:

$$\left.\begin{array}{c}V=\left\{{\mathrm{f}}_1,{\mathrm{f}}_2,\dots \dots {\mathrm{f}}_{\mathrm{n}}\right\}, {\mathrm{f}}_{\mathrm{t}}ϵD\subseteq {\mathbb{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{C}}\\ D=[a_H,a_H]\times [a_W,a_W]\times [a_C,a_C]\end{array}\right\}$$

(1)

In Equation (1), $\mathrm{D}$ represents a bounded domain, and ${\mathrm{f}}_{\mathrm{t}}$ represents the frame at time $\mathrm{t}.$ $\mathrm{H} \mathrm{a}\mathrm{n}\mathrm{d} \mathrm{W}$ represent the frame’s height and width, and C represents the number of channels. This guarantees that the model operates within a well-defined mathematical space.

Each frame ${\mathrm{f}}_{\mathrm{t}}$ contains $\mathrm{N}$ keypoints corresponding to hand and pose landmarks:

$${\mathrm{k}}_{\mathrm{t}}=\left\{{\mathrm{k}}_1^{\mathrm{t}},{\mathrm{k}}_2^{\mathrm{t}}\dots ..{\mathrm{k}}_{\mathrm{n}}^{\mathrm{t}}\right\},{\mathrm{k}}_{\mathrm{i}}^{\mathrm{t}}=({\mathrm{x}}_{\mathrm{i}}^{\mathrm{t}},{\mathrm{y}}_{\mathrm{i}}^{\mathrm{t}},{\mathrm{c}}_{\mathrm{i}}^{\mathrm{t}})$$

(2)

In Equation (2), ${\mathrm{x}}_{\mathrm{i}}^{\mathrm{t}} \mathrm{a}\mathrm{n}\mathrm{d} {\mathrm{y}}_{\mathrm{i}}^{\mathrm{t}}$ represent the spatial coordinates of the ith keypoint, and ${\mathrm{c}}_{\mathrm{i}}^{\mathrm{t}}$ represents the confidence score of the keypoint.

The HGE-ACN model optimizes learning using hyperparameters for robust feature extraction, efficient spatial–temporal modeling, and computational efficiency. Key hyperparameters are detailed in

Table 1. Hyperparameter configuration of HGE-ACN.

Hyperparameter	Notation	Value(s)	Description
Graph Construction Threshold	τ	0.5–0.7	Defines connectivity in the spatial–temporal graph.
Graph Embedding Dimensionality	d	64, 128	Determines feature embedding size in the latent space.
Adaptive Convolution Kernel Size	k	3 × 3, 5 × 5	Controls receptive field in feature extraction.
Normalization Factor in Edge Weight Calculation	σ	0.1–1.0	Regulates sensitivity of graph edge weights.
Attention Weight Factor	α	0.1–0.9	Adjusts attention-based feature importance.
Dropout Rate	p	0.3–0.5	Prevents overfitting in graph and convolution layers.
Learning Rate	η	0.001–0.005	Determines optimization step size for convergence.
Batch Size	B	32, 64	Number of samples processed per training step.
Number of GCN Layers	L	2, 3	Defines depth of graph convolutional layers.
Number of Attention Heads	h	4, 8	Controls multi-head self-attention operations.
Weight Decay (L2 Regularization)	λ	1 × 10−4, 1 × 10−5	Reduces overfitting by penalizing large weights.

, providing notations and values.

3.2. Feature Extraction

HGE-ACN sign language phrase recognition relies on the feature extraction and keypoint detection process Figure 2a below. It involves collecting and analyzing wearable glove motion and location data. The wearable glove has two main sensor types: six-axis IMU (Inertial Measurement Unit) sensors for three-dimensional motion tracking and finger curvature sensors for finger bending angles and hand shape configurations. The HGE-ACN technique relies on feature extraction through three parallel channels: Inertial Features, Motion Features, and Temporal Sequences. Motion Features measure velocity and acceleration patterns, identify movement segments, and assess dynamic signing elements, whereas Inertial Features extract acceleration and gyroscopic information. Temporal Sequences explore time-based patterns in signing, recognizing movements and places and capturing gesture durations.

In Figure 2a, feature extraction is a crucial step in processing sensor data for accurate recognition and classification. The process begins with sensor data acquisition, where real-time signals from multiple sensors are collected. Preprocessing enhances signal quality, followed by segmentation, where continuous data streams are divided into meaningful gesture segments. Features are extracted, including spatial, temporal, and motion-based metrics, such as velocity, acceleration, angular displacement, and curvature metrics. A spatial–temporal model is developed to capture intricate motion dependencies, improving the representation of gestures. Fusion and optimization techniques are employed to refine the extracted features, integrate multimodal features, and reduce redundancy through dimensionality reduction methods like PCA or LDA. The final step, output feature vectors, represents the optimized dataset, ready for classification models. This structured approach eliminates redundancy and enhances clarity in feature extraction.

$${\mathrm{L}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}=\left\{{\mathrm{L}}_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{v}\mathrm{e}},{\mathrm{L}}_{\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e}}\right\},$$

(3)

In Equation (3), the following can be noted: ${\mathrm{L}}_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{v}\mathrm{e}}$: Hand glove landmarks record delicate finger and hand movements, and ${\mathrm{L}}_{\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e}}$: position landmarks describe the body’s posture and movements. Dimensions: $\left\{{\mathrm{L}}_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{v}\mathrm{e}},{\mathrm{L}}_{\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{e}}\right\}\in {\mathbb{R}}^{N\times 3 }$, where N is the number of keypoints with 3D spatial coordinates. The landmark set captures every key component of human communication for downstream processing. The Landmark Feature Vector representation for each landmark i in a frame is a high-dimensional feature vector:

$${\mathrm{X}}_{\mathrm{i}}=[{\mathrm{p}}_{\mathrm{x}},{\mathrm{p}}_{\mathrm{y}},{\mathrm{p}}_{\mathrm{z}},{\mathrm{v}}_{\mathrm{x}},{\mathrm{v}}_{\mathrm{y}},{\mathrm{v}}_{\mathrm{z}},\mathrm{c}]$$

(4)

In Equation (4), ${\mathrm{p}}_{\mathrm{x}}$, ${\mathrm{p}}_{\mathrm{y}}, {\mathrm{a}\mathrm{n}\mathrm{d} \mathrm{p}}_{\mathrm{z}}$ represent 3D spatial coordinates for the precise location of the landmark. ${\mathrm{v}}_{\mathrm{x}},{\mathrm{v}}_{\mathrm{y}}, \mathrm{a}\mathrm{n}\mathrm{d} {\mathrm{v}}_{\mathrm{z}}$ represent velocity components for encoding the motion dynamics of the landmark over time. c represents the confidence score, indicating the reliability of the detected landmark as determined by MediaPipe. Position components: ${\mathrm{p}}_{\mathrm{x}}$, ${\mathrm{p}}_{\mathrm{y}},{\mathrm{a}\mathrm{n}\mathrm{d} \mathrm{p}}_{\mathrm{z}}$ in meters (${\mathbb{R}}^3)$. Velocity components: ${\mathrm{v}}_{\mathrm{x}},{\mathrm{v}}_{\mathrm{y}},\mathrm{a}\mathrm{n}\mathrm{d} {\mathrm{v}}_{\mathrm{z}}$ in meters per second (${\mathbb{R}}^3)$. Dimension: ${X_i\in \mathbb{R}}^7$ for each keypoint.

Velocity components measure landmark position changes throughout time. Use finite differences as calculated in Equation (5) as follows:

$$\left.\begin{array}{c}{\mathrm{v}}_{\mathrm{x}}={\displaystyle \frac{{\mathrm{x}}_{\mathrm{t}+1}-{\mathrm{x}}_{\mathrm{t}}}{{∆}_{\mathrm{t}}}},\\ {\mathrm{v}}_{\mathrm{y}}={\displaystyle \frac{{\mathrm{y}}_{\mathrm{t}+1}-{\mathrm{y}}_{\mathrm{t}}}{{∆}_{\mathrm{t}}}},\\ {\mathrm{v}}_{\mathrm{z}}={\displaystyle \frac{{\mathrm{z}}_{\mathrm{t}+1}-{\mathrm{z}}_{\mathrm{t}}}{{∆}_{\mathrm{t}}}}\end{array}\right\}$$

(5)

In Equation (5), Dimension: ${\mathrm{v}}_{\mathrm{x}},{\mathrm{v}}_{\mathrm{y}},{\mathrm{a}\mathrm{n}\mathrm{d} \mathrm{v}}_{\mathrm{z}}\in \mathbb{R}$ measured in meters per second (m/s). Gesture identification requires time information because movement sequences have semantic significance. To gather information from users, the system makes use of wearable gadgets. The wearable gadgets have sensors built in so they can record your hand motions and gestures. A variety of sensors, including magnetometers, accelerometers, and gyroscopes, are built into the wearable gadgets to record hand motions and orientation from various angles. Figure 2b shows the configuration used during this operation, which uses the WULALA data glove hardware space.

Figure 2b shows the WULALA data glove specifications, not the processing environment. Figure 2c shows the captured sign language gesture using a sample glove sensor.

In Figure 2c, the accuracy of sign language identification is enhanced through the fusion of multiple sensor data streams, including motion, curvature, and inertial sensor readings. This fusion process reduces inconsistencies in gesture recognition by integrating complementary information from different modalities. Before fusion, individual sensor data may contain noise or missing values, leading to potential misclassification. After fusion, the combined dataset provides a more comprehensive motion trajectory, improving recognition reliability. The benefits of this fusion process are detailed in Figure 2c, which demonstrates the way multiple sensor inputs contribute to a more robust sign language recognition framework. To ensure consistency across sensor data, normalization is applied before fusion. This process involves scaling values to a common range, typically between 0 and 1, to standardize formats and improve comparability. Data fusion is then applied to integrate previously separate datasets into a single, more complete representation. Overall, data redundancy is minimized, improving the efficiency of the analysis during the fusion process. By combining motion data from multiple sensors, the system compensates for the limitations of individual sensors, which may capture only partial data despite their effectiveness. This method enhances the comprehension of sign language gestures by constructing a richer representation of movement and hand positions. Additionally, fusion techniques mitigate noise, disturbances, and other sensor errors, improving accuracy and robustness in sign language recognition.

3.3. Spatiotemporal Graph Convolutional Networks (ST-GCN)

The Graph Construction Module is essential for tasks such as sign language identification because it rapidly extracts spatial–temporal patterns and arranges them into a structured Spatial–Temporal Graph (STG). This graph efficiently captures intra-frame and inter-frame interactions. The graph definition is defined as follows:

$$\mathrm{G}=(\mathrm{V},\mathrm{E})$$

(6)

In Equation (6), $\mathrm{V}$ represents a set of vertices where each vertex corresponds to a detected keypoint. E represents a set of edges that capture spatial connections (keypoint relationships within a frame) and temporal connections (keypoint relationships across frames). Dimension: $G\in {\mathbb{R}}^{N\times N}$ where $N$ is the number of keypoints.

Edge weights are calculated using a Gaussian similarity function to quantify node relationships, ensuring stronger connections are represented with higher weights.

$${\mathrm{w}}_{\mathrm{i}\mathrm{j}}=\exp \left(-{\displaystyle \frac{{‖{\mathrm{x}}_{\mathrm{i}}-{\mathrm{x}}_{\mathrm{j}}‖}^2}{{2\sigma }^2}}\right)$$

(7)

In Equation (7), ${\mathrm{w}}_{\mathrm{i}\mathrm{j}}$ represented as the edge weight between nodes $\mathrm{i}$ and $\mathrm{j}$. ${\mathrm{x}}_{\mathrm{I}} \mathrm{a}\mathrm{n}\mathrm{d} {\mathrm{x}}_{\mathrm{j}}$ represented as the nodes of the feature vector i and j. The scale parameter controlling the sensitivity of weight computation is $\sigma$. Dimension: $w_{ij}$∈$\mathbb{R}$.

In Figure 3, the Graph Construction Module organizes collected keypoints into a structured Spatial–Temporal Graph (STG) that efficiently captures intra-frame dependencies and inter-frame interactions. This structure is essential for extracting spatial–temporal patterns for sign language recognition. A graph has vertices representing keypoints and edges representing spatial and temporal connections. A Gaussian similarity function quantifies node relationships to calculate edge weights.

The module starts with each sensor data frame’s hand and body landmarks. It then uses anatomical or geographic relationships to create spatial and temporal links between keypoints in the same frame and consecutive frames to record motion dynamics. The Gaussian similarity formula assigns larger weights to closer or more similar keypoints in feature space for each edge. The final graph blends spatial and temporal connections to depict keypoint data spatially and temporally.

3.4. Hybrid Graph Embedding Module

The sign language recognition system uses wearable gloves to create a spatial–temporal network from keypoint detection outputs. These inputs comprise six-axis IMU sensor hand-position keypoints, finger curvature sensor finger-joint keypoints, velocity vector motion characteristics, acceleration data, temporal sequence data, and confidence scores for each keypoint. The inputs are arranged into S, T, and Combined Feature Vectors. Each input frame has 21 keypoints (one wrist + five fingers × four joints), six motion parameters (three linear + three angular), confidence ratings, and a timestamp. The data structure is a float–timestamp matrix with nodes representing keypoints, edges representing spatial and temporal correlations, and edge weights calculated. These input data create the sign language recognition feature representation using spatial graph convolution, temporal attention mechanism, and feature fusion. Confidence scores affect feature reliability weighting. The system generates sign language recognition features using spatial graph convolution, temporal attention, and feature fusion from these input data.

In Figure 4, to understand the temporal dynamics of movements, the Hybrid Graph Embedding Approach uses a combination of methods, including 3D keypoint extraction from sensor data frames, building a Spatial–Temporal Graph, learning representations of spatial features, and computing attention weights. The feature representation step is responsible for fusing the output of various modules and generating the final feature vector via weighted summing of multi-scale features. This combined graph-based modeling and Adaptive Convolutional Network strategy aims to improve the accuracy and resilience of sign language recognition by capturing the complex spatial–temporal interactions inherent in sign language communication. It gives a thorough overview of the proposed methodology in Figure 4, showing the flow of information and mathematical formulas underpinning each essential component.

3.4.1. Spatial Graph Convolution

In this process, local spatial information is extracted by applying convolution across the graph structure. The formula is as follows:

$${\mathrm{h}}_{\mathrm{i}}^{(\mathrm{l}+1)}=\sigma \left({\sum }_{\mathrm{j}\in \mathrm{N}\left(\mathrm{i}\right)}{\displaystyle \frac{1}{{\mathrm{c}}_{\mathrm{i}\mathrm{j}}}}{\mathrm{W}}^{\left(\mathrm{l}\right)}{\mathrm{h}}_{\mathrm{j}}^{\left(\mathrm{l}\right)}\right)$$

(8a)

In Equation (8a), ${\mathrm{h}}_{\mathrm{i}}^{\left(\mathrm{l}\right)}$ is denoted as the feature representation of node i at layer l. ${\mathrm{W}}^{\left(\mathrm{l}\right)}$ is denoted as a learnable weight matrix. $\mathrm{N}\left(\mathrm{i}\right)$ is denoted as a set of neighbors for node i. ${\mathrm{c}}_{\mathrm{i}\mathrm{j}}$ is represented as the normalization factor for edge ij. $\sigma$ is an activation function (e.g., ReLU). This operation groups information from neighboring nodes to inform the representation of each node.

Dimension: ${\mathrm{h}}_{\mathrm{i}}^{\left(\mathrm{l}\right)}\in {\mathbb{R}}^{\mathrm{d}}$, where $\mathrm{d}$ is the feature dimension.

In Equation (8a), the term neighbors refers to the set of adjacent nodes used in the spatial graph convolution operation. The connectivity pattern of the graph representation of keypoints in the sign language recognition model determines the number of neighbors.

The eight-connected neighborhood structure improves spatial representation, contextual awareness, and recognition accuracy by capturing diagonal relationships in hand motion. It enhances feature aggregation, leading to robust model performance and a well-defined graph convolution framework, ensuring a more detailed understanding of hand motion.

For a node located at a position $\left(\mathrm{i},\mathrm{j}\right),$ the neighboring nodes are defined as follows:

$$\mathrm{N}(\mathrm{i},\mathrm{j})=\{\left(\mathrm{i}-1,\mathrm{j}-1\right),\left(\mathrm{i}-1,\mathrm{j}\right),\left(\mathrm{i}-1,\mathrm{j}+1\right),\left(\mathrm{i},\mathrm{j}-1\right),\left(\mathrm{i},\mathrm{j}+1\right),\left(\mathrm{i}+1,\mathrm{j}-1\right),\left(\mathrm{i}+1,\mathrm{j}\right),\left(\mathrm{i}+1,\mathrm{j}+1\right)\}$$

(8b)

In Equation (8b), $\left(\mathrm{i}-1,\mathrm{j}-1\right)$,$\left(\mathrm{i}-1,\mathrm{j}+1\right)$,$\left(\mathrm{i}+1,\mathrm{j}-1\right),$ and $\left(\mathrm{i}+1,\mathrm{j}+1\right)$ are diagonal neighbors. $\left(\mathrm{i}-1,\mathrm{j}\right),\left(\mathrm{i},\mathrm{j}-1\right)$,$\left(\mathrm{i}+1,\mathrm{j}\right),$ and $\left(\mathrm{i},\mathrm{j}+1\right)$ are direct horizontal and vertical neighbors.

3.4.2. Temporal Attention Mechanism

The method simulates feature temporal connections throughout time. The computation of attention weights (${\alpha }_{\mathrm{i}\mathrm{j}}$) in Equation (9) is as follows:

$${\alpha }_{\mathrm{i}\mathrm{j}}=\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\right({\mathrm{a}}^{\mathrm{\top }}[\mathrm{W}{\mathrm{h}}_{\mathrm{i}}\mid \mid \mathrm{W}{\mathrm{h}}_{\mathrm{j}}]\left)\right)$$

(9)

In Equation (9), $[\mathrm{W}{\mathrm{h}}_{\mathrm{i}}\mid \mid \mathrm{W}{\mathrm{h}}_{\mathrm{j}}]$ is denoted as a concatenation of features from nodes i and j, transformed by weight matrix W. $\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}$ is denoted as an Activation function adding non-linearity, and $\mathrm{a}$ is denoted as a learnable attention vector. This process allocates a greater position to temporally relevant connections, dynamically weighting node interactions. Dimension: ${\alpha }_{\mathrm{i}\mathrm{j}}$∈$\mathbb{R}$, where attention weights are scalars.

3.5. Adaptive Convolutional Network

The Adaptive Convolutional Network is essential for handling feature representation fluctuations across scales and temporal contexts. Parts of Dynamic Kernel Generation and multi-scale sensor data fusion are discussed below:

3.5.1. Dynamic Kernel Generation

In this stage, the weights of convolutional kernels are adjusted dynamically according to the input features at a given time step.

$${\mathrm{K}}_{\mathrm{t}}={\mathrm{f}}_{\mathrm{a}\mathrm{d}\mathrm{a}\mathrm{p}\mathrm{t}}\left({\mathrm{F}}_{\mathrm{t}}\right)$$

(10)

In Equation (10), ${\mathrm{K}}_{\mathrm{t}}$ represents adaptive convolution kernel weights for time step t. ${\mathrm{f}}_{\mathrm{a}\mathrm{d}\mathrm{a}\mathrm{p}\mathrm{t}}$ represents a function (typically implemented as a neural network) that dynamically generates kernel weights, and ${\mathrm{F}}_{\mathrm{t}}$ represents input features extracted at time t. Dimension: ${\mathrm{K}}_{\mathrm{t}}\in {\mathbb{R}}^{k\times k}$, where $k$ is the kernel size. Encoding spatial or temporal information requires extracting input properties from data at a certain time, such as frames in a sensor data series. A convolution kernel is often generated by a dynamic function with the help of a small neural network trained to convert feature inputs into kernel weights. After that, the network processes the input features using the kernel, which enables it to dynamically specialize its convolution operation according to the data’s content. Because the kernel changes depending on the input characteristics, the network can better deal with variances in the input data, leading to content-aware processing. Furthermore, dynamic kernels improve generalizability by decreasing sensitivity to noise and environmental factors.

3.5.2. Multi-Scale Sensor Data Fusion

The method creates a cohesive picture by merging features from different spatial or temporal scales.

$${\mathrm{F}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}={\sum }_{\mathrm{i}=1}^{\mathrm{M}}{\mathrm{w}}_{\mathrm{i}}\cdot {\mathrm{F}}_{\mathrm{i}}$$

(11)

In Equation (11), ${\mathrm{F}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}$ represents the fused feature representation capturing information from multiple scales. ${\mathrm{F}}_{\mathrm{i}}$ represents the features extracted at scale i. Each scale represents features computed with different kernel sizes or temporal windows, capturing fine-grained and coarse-grained information.${\mathrm{w}}_{\mathrm{i}}$ represents learnable sensor data fusion weights for each scale, determining the contribution of ${\mathrm{F}}_{\mathrm{i}}$ to the final fused representation, and M represents the number of scales used. Dimension: ${\mathrm{F}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}\in {\mathbb{R}}^d$, where $d$ is the feature dimension.

Feature extraction, learning weights, and summing are used for local and global patterns. Small-scale features capture local details, while large-scale features capture patterns. Learnable weights favor task-specific informative scales. Sensor data fusion ensures multi-scale awareness and task-specific weighting, improving accuracy and efficiency. Summarization integrates information across scales.

Sign language sentence recognition using Hybrid Graph Embedding and Adaptive Convolutional Networks is robust using Algorithm 1. Initiates an empty graph and keypoints, extracts keypoints from sensor data frames, builds vertices with spatial and temporal edges, and assigns edge weights using Gaussian similarity. Spatial–Temporal Graph Convolution Networks aggregate graph node characteristics, whereas adaptive convolution generates frame-wise kernels. Finally, graph and convolution module embeddings are concatenated and processed through dense layers and a softmax classifier to predict words, accurately recognizing complicated spatial–temporal sign gestures.

Algorithm 1: HGE-ACN Method

Input: Sensor data frames {F1, F2, …, Fn}, Keypoint Detector D Output: Predicted Sentence S Step 1: Begin    G ← ∅# Initialize Graph, K ← ∅# Initialize Keypoints Step 2: for Fi in {F1, F2, …, Fn} do    K[i] ← D(Fi) # Extract keypoints      V ← K[i]; E ← SpatialEdges(V) ∪ TemporalEdges(V) Step 3: for vi, vj ∈ V do      ${\mathrm{w}}_{\mathrm{i}\mathrm{j}}=\exp \left(-{\displaystyle \frac{{‖{\mathrm{x}}_{\mathrm{i}}-{\mathrm{x}}_{\mathrm{j}}‖}^2}{{2\sigma }^2}}\right)$ (Equation (7)) # Gaussian edge weight Step 4: for vi ∈ Vdo      ${\mathrm{h}}_{\mathrm{i}}^{(\mathrm{l}+1)}=\sigma \left(\sum _{\mathrm{j}\in \mathrm{N}\left(\mathrm{i}\right)}{\displaystyle \frac{1}{{\mathrm{c}}_{\mathrm{i}\mathrm{j}}}}{\mathrm{W}}^{\left(\mathrm{l}\right)}{\mathrm{h}}_{\mathrm{j}}^{\left(\mathrm{l}\right)}\right)$ (Equation (8))       ${\alpha }_{\mathrm{i}\mathrm{j}}=\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\right({\mathrm{a}}^{\mathrm{\top }}[\mathrm{W}{\mathrm{h}}_{\mathrm{i}}\mid \mid \mathrm{W}{\mathrm{h}}_{\mathrm{j}}]\left)\right)$ (Equation (9)) # ST-GCN Step 5: for t ∈ {1, 2, …, n}do       ${\mathrm{K}}_{\mathrm{t}}={\mathrm{f}}_{\mathrm{a}\mathrm{d}\mathrm{a}\mathrm{p}\mathrm{t}}\left({\mathrm{F}}_{\mathrm{t}}\right)$ (Equation (10)) # Adaptive Kernel Convolution       ${\mathrm{F}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}=\sum _{\mathrm{i}=1}^{\mathrm{M}}{\mathrm{w}}_{\mathrm{i}}\cdot {\mathrm{F}}_{\mathrm{i}}$ (Equation (11)) Z ← Concatenate (G_emb, Ft_fused) Y ← Dense(Z); S ← Softmax(Y) Output: S End

3.5.3. Training Procedure for the Model

The HGE-ACN framework is a training method that aims to efficiently learn spatial–temporal patterns from wearable sensor data. The training process involves dataset preparation, preprocessing, graph construction, model initialization, feature extraction, graph convolution, adaptive convolution, loss function and optimization, batch processing, backpropagation, early stopping, and model evaluation. The input data are from a wearable glove with six-axis IMU sensors and five-finger curvature sensors. The raw sensor signals are normalized, synchronized, and transformed into structured feature vectors. A spatial–temporal graph is built, with vertices representing keypoints and edges representing spatial and temporal relationships. The Graph Convolutional Network (GCN) and Adaptive Convolutional Networks (ACNs) are initialized with random weights and edge weights computed using the Gaussian similarity function. The model’s classification output is obtained using Softmax, cross-entropy loss is used for optimization, and the Adam optimizer is used with a learning rate of 10⁻³. The training process includes batch processing, backpropagation, early stopping, and model evaluation. The HGE-ACN model achieves high recognition accuracy, robustness, and real-time performance.

3.6. Classification

The HGE-ACN framework integrates processed characteristics and predicts output at several levels, as shown in Figure 5 below. Features from multiple spatial and temporal dimensions are fused and classified to capture fine-grained and coarse-grained patterns. The feature vector encodes spatial–temporal dynamics of the full input sensor data sequence. Classification modules with dense layers, softmax layers, and three levels, the Word Level (individual sign recognition), Phrase Level (sign sequences), and Sentence Level (complete sentence translation), process the fused output feature vector.

Figure 5 shows that the system delivers three-level hierarchical recognition outputs for varied use cases. Individual signs are examined by frame or short sequence to predict the most likely word for a motion. This level is ideal for language-learning and vocabulary-building apps. The phrase level recognizes small sequences of related indicators in temporal order to produce meaningful phrases. It works for brief commands and AAC. The technology generates fully structured sentences with word order changes and auxiliary phrases to match spoken and written language. Real-time communication, accessibility, and sign language translation improve at this level. The three-level output structure makes the system flexible for education, accessibility, and real-world human–computer interaction for simple and complicated sign language recognition tasks.

4. Results and Discussion

The proposed framework was tested on datasets featuring both sensor-based inputs and traditional glove sensor data. Initial results indicate improved recognition accuracy and reduced inference time compared to baseline methods. Further analysis highlights the system’s ability to adapt to varied signing speeds and environmental conditions, making it a viable solution for real-world applications.

Performance Comparison Study:

HGE-ACN outperforms the Dynamic Dense Spatiotemporal Graph Convolutional Neural Network (DDSTGCNN) [18], Graph-Based Multimodal Sequential Embedding Network (MSeqGraph) [17], and Boundary-Adaptive Encoder (BAE) [23] in sign language sentence recognition. Compared to previous approaches, HGE-ACN has higher recognition accuracy and reduced mistake rates. Real-time sign language recognition systems can rely on its adaptive convolutional technique and sensor data fusion to handle signing speed and style changes. HGE-ACN can create inclusive deaf communication systems, as shown by this comparison.

4.1. Accuracy

A method’s accuracy is the ratio of correctly predicted sign language sentences to total predictions. Overall, it validates the model.

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{l}\mathrm{y} \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{g}\mathrm{n}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{d} \mathrm{S}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{s} \left(\mathrm{C}\right)}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l} \mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{S}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{s} \left(\mathrm{N}\right)}}\times 100$$

(12)

In Equation (12), C stands for the successfully predicted sentences, and N denotes the total number of sentences in the test dataset.

Cost criterion: The cost criterion in a model is defined using the cross-entropy loss function, which quantifies the classification error between the predicted sign language sentence and the actual ground truth.

Accuracy is a performance metric, while the cost function is used for model optimization. The model’s accuracy is evaluated using Equation (12), which assesses the proportion of correctly predicted sentences. The training of the model minimizes a cost function based on the categorical cross-entropy loss, penalizing incorrect predictions more heavily when the model is highly confident about an incorrect class. Cross-entropy is suitable for classification tasks like sign language sentence recognition as it improves sentence prediction accuracy, directly impacting performance in Equation (12). Softmax normalization ensures a probability distribution over classes, stabilizing training and preventing divergence. Gradient-based optimization (Adam or SGD) ensures convergence towards a model that maximizes classification performance. The cost criterion used during training is the categorical cross-entropy loss function, while accuracy is a post-training evaluation metric. This distinction should be explicitly mentioned when presenting the model to avoid ambiguity.

Figure 6 indicates that the hybrid HGE-ACN model, which uses adaptive convolution and graph-based embeddings, recognizes sign language better than DDSTGCNN, MSeqGraph, and BAE. It outperforms DDSTGCNN, MSeqGraph, and BAE with 94.8% accuracy under normal settings. HGE-ACN remains accurate despite noise or compounding obstructions. Excellent performance is due to its spatiotemporal dynamics recording and sign language motion resilience.

4.2. The Inference Time

Inference time is the model’s mean duration to analyze a single input (sensor data) and generate recognition outcomes. It is essential for assessing the appropriateness of real-time applications.

$$\mathrm{I}\mathrm{n}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e} \mathrm{T}\mathrm{i}\mathrm{m}\mathrm{e}={\displaystyle \frac{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l} \mathrm{P}\mathrm{r}\mathrm{o}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{T}\mathrm{i}\mathrm{m}\mathrm{e} \mathrm{f}\mathrm{o}\mathrm{r} \mathrm{A}\mathrm{l}\mathrm{l} \mathrm{s}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{o}\mathrm{r} \mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a} \left(\mathrm{T}\right)}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l} \mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{s}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{o}\mathrm{r} \mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\left(\mathrm{V}\right)}}(\mathrm{m}\mathrm{i}\mathrm{l}\mathrm{l}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{d}\mathrm{s} \mathrm{p}\mathrm{e}\mathrm{r} \mathrm{s}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{o}\mathrm{r} \mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a})$$

(13)

In Equation (13), inference time, the average processing time for sensor data, is a key parameter for model computational efficiency. Divide the total processing time for all sensor data (T) by the total number of sensor data (V) in milliseconds per sensor data. Less inference time means faster processing and scalability in real-time applications.

Figure 7 of the inference time series compares the execution timings of four models, HGE-ACN, DDSTGCNN, MSeqGraph, and BAE, under five scenarios. With a time of 45 ms, HGE-ACN is the quickest, whereas BAE takes 85 ms. With a minimum inference time of 48 ms, the variable signing speed remains the fastest, while other models exhibit a significant rise in time. All models experience an increase in signature variability, except HGE-ACN at 50 ms and BAE at 95 ms. As a result of environmental noise, inference times increase; for example, HGE-ACN takes 52 ms and BAE 98 ms. When tested under extreme conditions, HGE-ACN shows its best performance at 55 ms.

In

Table 2. Computational complexity of HGE-ACN model.

Component	Operation	Computational Complexity
Graph Convolution	Node feature aggregation and transformation	$O\left(N{F}^{2}L\right)$
Adaptive Convolution	Feature extraction over spatial dimensions	$O\left(HWC{k}^2\right)$
Total Complexity	Combined graph and convolutional computations	$O(N{F}^{2}L+HWC{k}^2{L}_C)$

, the computational complexity of the HGE-ACN is analyzed based on graph convolution and adaptive convolution operations. Graph convolution has a complexity of $\mathrm{O}\left(\mathrm{N}{\mathrm{F}}^2\mathrm{L}\right)$, where $\mathrm{N}$ is the number of keypoints, $\mathrm{F}$ is the feature dimension, and $\mathrm{L}$ is the number of graph convolution layers. Adaptive convolution operates over spatial dimensions with complexity $\mathrm{O}\left(\mathrm{H}\mathrm{W}\mathrm{C}{\mathrm{k}}^2\right)$, where $\mathrm{H} \mathrm{a}\mathrm{n}\mathrm{d} \mathrm{W}$ are frame dimensions, $\mathrm{C}$ is the number of channels, and $\mathrm{k}$ is the kernel size. The combined complexity ensures scalable and efficient inference.

4.3. Error Rate

The percentage of sentences that the model erroneously identified is called the error rate. Through the quantification of inaccurate predictions, it emphasizes the model’s dependability.

$$\mathrm{E}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r} \mathrm{R}\mathrm{a}\mathrm{t}\mathrm{e}={\displaystyle \frac{\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{I}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{l}\mathrm{y} \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{g}\mathrm{n}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{d} \mathrm{S}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{s} \left(\mathrm{E}\right)}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l} \mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{S}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{s} \left(\mathrm{N}\right)}}\times 100$$

(14)

In Equation (14), E is the number of errors, and N is the total number of test sentences.

In Figure 8, different conditions are used to compare four models: HGE-ACN, DDSTGCNN, MSeqGraph, and BAE. With an error rate of 5.2% under typical circumstances, HGE-ACN outperforms BAE, which has the highest at 11.3%. When the difficulty level rises, so does the rate of inaccuracy for every model. While BAE’s error rate is the greatest at 15.8% under a variable signing speed, HGE-ACN performs the best with a rate of 6.5%. With error rates of 7.2% for HGE-ACN and 16.2% for BAE, respectively, signal variability and environmental noise affect performance.

4.4. Levenshtein Distance

The minimal number of single-character modifications required to turn one sentence into another is measured by the Equation d (i,j). When comparing anticipated and real sentences, it is useful to utilize the function d(i,j) as the difference between them. The purpose of the function is to compare the anticipated and real sentences for similarity. When comparing expected and actual sentence similarity, the following equation is helpful.

$$\mathrm{d}\left(\mathrm{i},\mathrm{j}\right)=\left\{\begin{array}{c}\mathrm{i} \mathrm{i}\mathrm{f} \mathrm{j}=0,\\ \mathrm{j} \mathrm{i}\mathrm{f} \mathrm{j}=0,\\ \mathrm{m}\mathrm{i}\mathrm{n}\left\{\begin{array}{c}\mathrm{d}\left(\mathrm{i}-1,\mathrm{j}\right)+1,\\ \mathrm{d}\left(\mathrm{i},\mathrm{j}-1\right)+1, \mathrm{otherwise}\\ \mathrm{d}\left(\mathrm{i}-1,\mathrm{j}-1\right)+\mathrm{cost}(\mathrm{i},\mathrm{j})\end{array}\right.\end{array}\right.$$

(15)

Equation (15) calculates the Levenshtein Distance, the least edit operations needed to change a string. It involves d (i, j), indicating the Levenshtein Distance between string A’s first i and j characters and cost (i, j), reflecting the cost of changing A[i] into B[j]. The distance is the length of the opposite string, if empty, and the minimum of insertion, deletion, and substitution.

Figure 9 shows the Levenshtein Distance, also known as edit distance, used in error correction, DNA sequence alignment, and natural language processing. Differences between strings are determined by the fewest operations needed to transform them. Figure 9 indicates that the Levenshtein Distance compares HGE-ACN, DDSTGCNN, MSeqGraph, and BAE under different conditions. All models increase error rates when conditions are challenging, although HGE-ACN frequently maintains accuracy. This shows that HGE-ACN can endure noisy, dynamic environments.

4.5. Model Experimentation

To ensure robust generalization, the HGE-ACN model was tested on a structured sign language recognition dataset with many samples, various signers, and real-world variations. Refinements for dataset details and ablation studies are as follows:

The dataset includes 120 h of recorded sign language gestures from 50 different signers, covering 250,000 gesture sequences. The data were collected in indoor and outdoor settings under three lighting conditions and varying noise levels. The sensor configuration used a wearable glove-based system with six-axis IMU sensors and five-finger curvature sensors for precise motion and hand positioning. Preprocessing methods included min–max normalization, low-pass filtering, and temporal alignment techniques to improve signal quality before feature extraction. This dataset configuration ensures the model effectively generalizes across different user styles, environmental settings, and signing speeds.

Ablation Study: Validating Module Contributions

To validate the contribution of each HGE-ACN module, ablation studies were conducted by removing or modifying individual components and analyzing the impact on accuracy and inference time in

Table 3. Ablation study of HGE-ACN components.

Experiment	Description	Accuracy (%)	Inference Time (ms)
Full HGE-ACN Model	Baseline model with all modules enabled.	94.8%	45 ms
Without Graph Embedding (HGE Removed)	Only adaptive convolution is used for feature extraction.	88.3%	39 ms
Without Adaptive Convolution (ACN Removed)	Only graph embeddings are used without adaptive convolution.	85.7%	42 ms
Without Multi-Scale Sensor Fusion	Raw sensor inputs are used without fusion techniques.	81.5%	38 ms
Without Attention Mechanism	Temporal attention is disabled in the graph network.	86.1%	44 ms

.

4.6. Model Comparison

The HGE-ACN model was compared to advanced sign language recognition methods, ensuring fair evaluation based on accuracy, inference time, complexity, and standardized parameter settings for consistency.

4.6.1. Comparison with Recent Methods

To validate the effectiveness of HGE-ACN, it was compared against four recent deep-learning-based models used for sign language recognition, as shown in

Table 4. Comparison with recent methods.

Model	Methodology	Accuracy (%)	Inference Time (ms)	Number of Parameters (Million)
HGE-ACN (Proposed)	Hybrid Graph Embedding + Adaptive Convolution	94.8%	45 ms	12.5 M
DDSTGCNN [1]	Dynamic Dense Spatiotemporal Graph CNN	89.2%	50 ms	14.3 M
MSeqGraph [2]	Multimodal Sequential Graph Embedding	91.1%	56 ms	15.2 M
BAE [3]	Boundary-Adaptive Encoder for Sign Language	87.4%	62 ms	11.8 M
SKResNet-TCN [4]	Hybrid ResNet + Temporal Convolution	92.3%	48 ms	13.1 M

:

4.6.2. Parameter Settings for Benchmark Models

All models were trained with identical dataset configurations and preprocessing techniques for a fair comparison. The hyperparameter settings used for each model are summarized in the

Table 5. Parameter settings for benchmark models.

Model	Learning Rate	Batch Size	Number of Layers	Dropout Rate	Optimizer
HGE-ACN (Proposed)	0.001	64	3 GCN + 2 ACN	0.4	AdamW
DDSTGCNN	0.0015	64	4 GCN	0.5	Adam
MSeqGraph	0.002	32	2 GCN + 1 LSTM	0.3	RMSprop
BAE	0.001	32	2 LSTM	0.3	Adam
SKResNet-TCN	0.0008	64	3 ResNet + 2 TCN	0.4	Adam

below:

The HGE-ACN model was tested in real-world scenarios to evaluate its performance. The model showed high accuracy (above 87%) even under challenging conditions, demonstrating robust generalization. Inference time increased slightly in real-world scenarios but remained efficient for real-time applications. Error rates were highest in low-light and high-noise conditions, indicating sensitivity to external environmental factors. The model performed consistently across signers with different hand sizes, proving adaptability to user variability. However, the model has limitations, including sensitivity to low-light conditions, computational complexity in large-scale deployment, and a limited dataset for extreme conditions. Future improvements include integrating adaptive filtering and feature enhancement techniques, optimizing model architecture with quantization and pruning techniques, and expanding the dataset with additional real-world variations.

5. Conclusions and Future Work

The Hybrid Graph Embedding and Adaptive Convolutional Networks (HGE-ACN) are a game-changer in sign language sentence recognition (SLSR). This innovative framework combines graph-based embeddings with Adaptive Convolutional Networks to address the challenges of decoding complex sign language communication. Among the significant achievements are the following: constructing a robust SLSR system using a wearable glove with six-axis inertial and five-finger curvature sensors; demonstrating that hybrid modeling techniques can improve recognition accuracy; and lastly, developing a versatile and adaptive recognition framework capable of surpassing the shortcomings of existing deep learning models. There are significant implications for accessibility, communication technology, and inclusive design, as stated in the paper. The proposed method has the potential to improve communication tools for the deaf population by improving machine understanding of sign language, which in turn can change platforms for social and professional, as well as educational, interactions. The next stages should center on further studies and finding practical uses for the HGE-ACN technique. Some of these areas of focus include investigating interactions between humans and computers in the context of adaptive communication, gesture-based interface design, rehabilitation efforts, and therapy technologies; incorporating advanced machine learning techniques; optimizing deployment for real-world use; building context-aware sign language interpretation systems; and expanding the current framework to support recognition across different sign languages. Building on HGE-ACN’s work in these areas, further research can improve sign language recognition algorithms to make them more accessible and valuable for a wider range of users.