Keywords

1 Introduction

Physics-aware machine learning (PhML) is an emerging field within ML that strives to integrate physics-based models with data-driven DL to reap the benefits of both approaches. Physics-based models represent the high level of domain knowledge gained from a study of the electromagnetic backscatter from surfaces and objects of the years. It can also capture phenomenological factors integral to the sensing scenario as well as known sensor properties. However, physics-based models are less adept at capturing the nuances of environment-specific, sensor-specific, or subject-specific properties, which lie at the heart of Data-Driven Dynamic Adaptive Systems (DDDAS). Here, deep learning can provide tremendous insight through data-driven learning.

Fig. 1.
figure 1

The physics-aware ML tradeoff.

Full size image

DL relies on the availability of massive amounts of data to learn a model for the underlying relationships reflected in the data. Unfortunately, in sensing problems, it is not common to have a lot of data. The limitations in training sample support ultimately also limit the accuracy and efficacy of DL in sensing. Moreover, no model is perfect - while more complex models could surely be developed to improve accuracy, the dynamic nature of the sensing environment ensures that there will always be some part of the signal that is unknown. This is where leveraging data-driven DL can provide a powerful tool when used in tandem with physics-based models. The resulting hybrid approach, PhML (Fig. 1) combines the strengths of DL and physics-based modeling to optimize trade-offs between prior versus new knowledge, models vs. data, uncertainty, complexity, and computation time, for greater accuracy and robustness. In this way, PhML also represents an integral concept to DDDAS, which uses applications modeling, mathematical algorithms and measurement systems to work with dynamic systems. ATR is a dynamic system challenged by variations of the sensor, target and environment in both space and time. This paper develops DDDAS-based ATR [1,2,3] methods empowered by PhML to improve the ability of deep models to learn from data via guidance of physics-based models.

Much of current literature involving physics-aware machine learning has focused on the solution of ordinary differential equations (ODEs) [4, 5], data-driven discovery of physical laws [6, 7], uncertainty quantification [8] and data generation - both to synthesize data for validation on simulated data in cases where acquiring real data is not feasible and for physics-guided initialization to pre-train deep models. The question of whether GAN-generated samples conform to physical constraints has recently been raised in the context of turbulent flow simulation, where both deterministic constraints (conservation laws) [9] and statistical constraints (energy spectrum of turbulent flows) [10] have been proposed for incorporation into the loss function. These constraints were shown to yield improvements in performance relative to that attained by standard GANs because the synthetic samples more faithfully emulated certain physical properties of the system, while also significantly reducing (by up to %80) the training time.

This paper addresses the design of physics-aware synthetic data generation techniques for training deep models for ATR using radar micro-Doppler signatures. In particular, the limitations of current generative adversarial network (GAN)-based methods, physics-based methods for data synthesis, new ways domain knowledge may be integrated for new GAN architectures and domain adaptation of signatures from different, but related sources of RF data, are presented. The implications for DL-based ATR in DDDAS due to fully-adaptive transmissions are discussed.

2 Physics-Based Models for μD Signature Synthesis

There are two main approaches for simulating micro-Doppler (μD) signatures (μDS) [11]: kinematic modeling and motion capture (MOCAP)-based animation. Both methods are based on the idea of representing the overall target backscatter as the sum of the backscatter from a finite number of point targets. MOCAP data have been effective in capturing the time-varying positions of moving humans and animals [12,13,14,15], while fixed body approximations to the mechanics of rotating parts are utilized to model vehicles, helicopters, aircraft or drones [16, 17]. Biomechanical models of human gait, such as the Boulic walking model [18], can also be used to animate the motion of point-targets. Thus, the radar return can be modeled as the sum of returns from point targets distributed throughout the target [19]. For K points,

$${\widehat{s}}_{h}\left[n\right]={\sum }_{i=1}^{K}\left(A/{R}_{n,i}^{4}\right)exp\left[-j\frac{4\pi {f}_{t}}{c}{R}_{n,i}\right],$$
(1)

where ft is the transmit frequency, c is the speed of light, and Rn,i is the time-varying range between the radar and ith point target at discrete time n. The parameter A is modeled by the radar range equation, and is a function of the radar system parameters, atmospheric and system loss factors, and radar cross-section. The simulated μDS are computed as the square modulus the short-time Fourier Transform.

Because MOCAP-based point tracking relies on actual measurements from a radar, the size of the dataset is still limited by the human effort, time and cost of data collections. To overcome this limitation, diversification [20] can be applied to generate physically meaningful physics-based variants of target behavior within a motion class. In the case of human recognition, skeleton-level diversification can be accomplished by scaling in time and size, while applying perturbations on the trajectory of each to generate statistically independent samples (Fig. 2). Data augmentation on the point-target model is important because augmentation methods used in image processing, such as rotation and flipping, result in physically impossible variations of the μDS, which degrades model training. In this way, a small amount of MOCAP data can be leveraged to generate a large number of synthetic μDS for model training.

To give an example of the benefit of this approach, this diversification technique was applied to 55 MOCAP samples, generating a total of 32,000 synthetic samples. The synthetic samples were then used to initialize a 15-layer residual DNN (DivNet-15), which was then fine-tuned with approximately 40 samples/class. Note that the depth of 15 layers was much deeper than the maximum depth of 7-layers for a convolutional neural network (CNN) trained with real data only. The 95% accuracy attained by DivNet-15 surpassed alternative approaches, including convolutional autoencoders, genetic algorithm optimized frequency warped cepstral coefficients [21], transfer learning from VGGnet, a CNN or autoencoder (AE) trained on real data, and support vector machine classifier using 50 features extracted from μD.

Fig. 2.
figure 2

Diversification of MOCAP-based synthetic micro-Doppler [20].

Full size image

Thus, physics-aware initialization with knowledge transfer from model-based simulations is a powerful technique for overcoming the problem of limited training data and can also improve generalization performance by exploiting the simulation of scenarios for which real data acquisition may impractical.

3 Adversarial Learning-Based μD Signature Synthesis

While model-based training data synthesis has been quite effective in replication of target signatures, it does not account for other sources of noise and interference, such as sensor artifacts and ground clutter. Because interference sources may be device specific or environment-specific, data-driven methods for data synthesis such as adversarial learning are well-suited to account for such factors. Adversarial learning can be exploited in several different ways to learn and transfer knowledge in offline model training, as illustrated in Fig. 3; for example,

  • To improve realism of synthetic data generated from physics-based models;

  • To adapt data from a different source to resemble data from the target domain; and

  • To directly synthesize both target and clutter components of measured RF data.

The main benefit of using adversarial learning to improve the realism of synthetic images generated from physics-based models is that it preserves the micro-Doppler signature properties that are bound by the physical constraints of the human body and kinematics while using the adversarial neural network to learn features in the data unrelated to the target model, e.g. sensor artifacts and clutter. The goal for improving realism [22] is to generate training images that better capture the characteristics of each class, and thus improve the resulting test accuracy. However, as the goal of the refiner is merely to improve its similarity to real data, a one-to-one correspondence is maitained between synthetic and refined samples. In other words, however much data we have at the outset, as generated by physics-based models, is the same amount of data that we have after the refinement process - no additional data is synthesized.

Fig. 3.
figure 3

Ways of exploiting adversarial learning for training data generation.

Full size image

Alternatively, the data from a source domain may be adapted to resemble real data acquired in the target domain [23]; then, the adapted data is used for network initialization. In this approach, the source domain can be real data acquired using a different RF sensor with different transmit parameters (frequency, bandwidth, pulse repetition interval), while the target domain is that which is to be classified. For example, consider the case where the target domain is RF data acquired with a 77 GHz frequency modulated continuous wave (FMCW) automotive radar, but there is insufficient data to adequately train a DNN for classification. Perhaps data from some other sensor, however, is available: this could be data from a publicly released dataset, or data from a different RF sensor. Suppose we have ample real data from two other RF sensors - a 10 GHz ultra-wide band impulse radar and a 24 GHz FMCW radar. Although the data from these three RF sensors will be similar for the same activity, there are sufficient differences in the μD that direct transfer learning suffers from great performance degradation. While the classification accuracy of 77 GHz data with training data from the same sensor can be as high as 91%, the accuracy attained when trained on 24 GHz and 10 GHz data is just 27% and 20% [24], respectively. This represents over 65% poorer accuracy. On the other hand, when adversarial domain adaptation is applied to first transform the 10 GHz and 24 GHz data to resemble that of the target 77 GHz data, classification accuracies that surpass that of training with just real target data can be achieved [25].

Effective data synthesis requires being able to represent both target as well as clutter components. While model-based methods effectively capture target kinematics, they do not capture environmental factors. Conversely, while GANs can capture sensor artifacts and clutter, they face challenges in accurately representing target kinematics. The efficacy of different GANs to accurately synthesize radar μD signatures varies. Auxiliary conditional GANs (ACGANs) generate crisper μD signatures than conditional variational autoencoders (CVAEs) [26]. For radar μD applications, however, fidelity cannot be evaluated by considering image quality alone. A critical challenge is the possibility of generating misleading synthetic signatures. The μD signature characteristics are constrained not only by the physics of electromagnetic scattering, but also by human kinematics. The skeleton physically constrains the possible variations of the spectrogram corresponding to a given class. But, GANs have no knowledge of these constraints. It is thus possible for GANs to generate synthetic samples that may appear visually similar but are in fact incompatible with possible human/animal motion.

Examples of erroneous data generation are given in Fig. 4(a) for an Auxiliary-Conditional GAN (ACGAN), which visually show some of the physically impossible or out-of-class samples synthesized by GANs. In fact, classification accuracy greatly increases when such fallacious synthetic samples are identified and discarded from the training data. For example, when an ACGAN was used to synthesize 40,000 samples, 9000 samples were identified as kinematic outliers and discarded from the training dataset, but in doing so the classification accuracy increased by 10% [26].

Fig. 4.
figure 4

(a) Errors observed ACGAN-synthesized μDS; (b) MBGAN

Full size image

4 Physics-Aware GAN Design

Physics-aware GAN design aims at preventing the generation of kinematically flawed data that is inconsistent with possible target behavior by integrating physics-based models and domain knowledge into data-driven DL. This can be done through modifying the GAN architecture, the way the GAN is trained, and by developing more meaningful loss functions. Current GAN architectures typically use distance measures rooted in statistics, such as cross-entropy or Earth mover’s distance, to derive a loss metric reflective of the degree of discrepancy between true and predicted samples. Statistical distance metrics, however, do not reflect discrepancies in the underlying kinematics of actual target motion versus predicted motion. However, target kinematics are physically constrained. One way of informing a GAN of such constraints is through the design of a physics-based loss term that is added to the standard statistical loss term:

$$Loss=\left(D\left(x\right)-D\left(G\left(z\right)\right)\right)+\gamma \left({\Vert {\nabla }_{\widehat{x}}D\left(\widehat{x}\right)\Vert }^{2}\right)+{Loss}_{phyiscs},$$
(2)

where the first term is the critic loss, the second term is the gradient penalty, \(x\) is the real data instance, \(z\) is noise, \(\gamma \) is a hyperparameter, and \(D\left( \cdot \right)\) and \(G\left( \cdot \right)\) represent the discriminator and generator functions, respectively. In this way, a penalty will now be incurred if the resulting signature exhibits deviant physical properties. In the case of micro-Doppler, one property of special significance is the envelopes of the signature, as they form a physical upper bound of the radial velocity incurred during motion. Thus, physics-based loss metrics reflective of consistency in envelope between synthetic and real samples are proposed for μDS classification.

Envelope information is provided to the GAN in two different ways: 1) through the addition of one or two auxiliary branches in the discriminator, which take as input the upper and lower envelope of the μDS, as shown in Fig. 4(b) of the resulting Multi-Branch GAN (MBGAN); and 2) physics-based loss metrics such as the Dynamic Time Warping (DTW) distance. In time series analysis, DTW is an algorithm for measuring the similarity between two temporal sequences that may vary in time or speed [27]. Utilizing a convolutional autoencoder trained on GAN-synthesized data for a 15-class activity recognition problem, we found that use of physics-aware loss in the MBGAN architecture resulting a 9% improvement in classification accuracy over the use of real data alone, and 6% improvement over that of a Wasserstein GAN with gradient penalty. For ambulatory data use of just a single auxiliary branch taking the upper envelope worked best. But in classification of 100 words of sign language [28], the dual branch MBGAN worked best. This illustrates the degree to which understanding the kinematics of the problem can affect results: for ambulation, the dominant μD are positive and reflected in the upper envelope. But in signing, both positive and negative frequencies hold great significance. For both datasets (ambulatory and signing), we found that direct synthesis of μDS outperformed domain adaptation from other RF data sources.

5 Conclusion

This paper provided a broad overview of physics-aware machine learning, including use of synthetic datasets to train deep models, with focus on aspects relevant to classification of RF μDS. The design of physics-aware GANs that surpass alternatives to data synthesis, including physics-based models, standard GANs, and domain adaptation was presented. This work has significant implications for cognitive radar design, where the adaptive transmissions result in physics-based mismatch between training and test datasets. Extension of physics-aware techniques to domain adaptation can help mitigate potential degradation in accuracy incurred due to RF frequency mismatch.