Laser Range Scanners for Enabling Zero-overhead WiFi-based Indoor Localization System

This module aims to define rules (referred to as spatial rules) that characterize every location in the environment based on its distance to the reference APs (i.e., APs where locations are known). These rules represent the expected RSS relations between every possible pair of reference APs at each location. To enable the effective calculation of spatial rules, LiPhi++ discretizes the area by creating a grid that is virtually superimposed on the layout of the area of interest, and the center of each grid cell is referred to as a reference point. The density of the grid cells is a configurable parameter that could be selected to balance between computational overhead and accuracy (Section 5.2.1). As a result, several reference points (i.e., grid cell points) are defined such that they are uniformly distributed over the area of interest. Given an arbitrarily defined point in the area, the Euclidean distances between that point and every detectable reference AP are calculated. Then, for each point, the spatial pairwise rules are derived based on the distance between the point in question and every possible pair of reference APs. In particular, assume there are two reference APs \(A\) and \(B\) with distances \(d_A\) and \(d_B\) from the point, respectively. A spatial rule can be defined as \(d_B \lt d_A\) if \(B\) is closer to that point than \(A\) . It is expected that this rule should apply to the signal strengths detected by the mobile device at that point. That is, the RSS level from \(B\) should be expectedly higher than the RSS level from \(A\) forming an RSS rule \(x_B \gt x_A\) (Figure 6). For \(m\) APs present in the area of interest, a total of \(\binom{m}{2}\) pairwise spatial rules are defined for each point. Note that considering relative relations between different reference APs rather than the absolute RSS values lead to device-invariant location estimates.

The goal of this module is to estimate the user’s most probable location, which is then used to annotate the corresponding WiFi scan. This can be done by locating the user’s mobile device based on the measured RSS signals received from the reference APs during the normal movement of the users of the building (i.e., crowdsourcing). Subsequently, the location of the user’s device is estimated using the pre-calculated spatial rules of each point. During the resolution process, short-term variations in the wireless channel (e.g., arising from link blockages between the device and the reference APs) may lead to erroneous matches. To combat this, this module employs a scoring approach, outlier removal, and smoothing techniques, described as follows.

The process of finding a rough user location is actually resolved to a number of best-matching reference points, as selected by their matching score. Therefore, the rough user location is estimated as the center of mass of reference points with the top matching scores. More specifically, all reference points are initialized with a score of zero (i.e., \(s = 0\) ). Matches between the given RSS rules and the rules for a given reference point increase the score for that reference point by one for each successful match. Naturally, this implies that a number of reference points (if not all) will have nonzero matching scores. Therefore, the points with the top \(\gamma\) scores are selected, and their weighted average is reported as the user’s location with an assigned weight equaling their corresponding score. Empirically, \(\gamma\) is one of the system parameters and we found that \(\gamma\) of 10% is sufficient to achieve good performance enabling better matching and location refinement (as described in Section 4.3).

To further boost the robustness against the occasional environmental dynamics, LiPhi++ leverages multiple consecutive location estimates in its operation, which constitutes a window of \(v\) locations. In particular, given a window of \(v\) locations (i.e., consecutive estimates in time), LiPhi++ removes the anomalous location estimates that may deviate from the majority of scans in that window. As a result, the predicted consecutive locations are ensured to be correlated. To do that, it calculates the distance between each estimated location and the remaining \(v-1\) reported locations, a value we term inter-location distance and define as [15, 41]where \(d(r_i,r_j)\) is the Euclidean distance between locations \(r_i\) and \(r_j\) . Then, we calculate the average \(d_{\textrm {avg}}\) of inter-location distance of all locations. This module rejects the location estimates that do not satisfy \(d_i\le d_{avg}\) [44]. Finally, a robust estimate of the user location \(r^*\) is calculated as the center of mass (i.e., weighted average) of the remaining \(u\) locations after removing outliers aswhere \(r_i\) is the \(i{\textrm {th}}\) reported location after avoiding outliers and \(s_{i}\) is its corresponding confidence score. In calculating this average, the weight used for a given location is its corresponding normalized matching score.

The number of APs detected at a fixed location varies with time as discussed in Section 2.3. LiPhi++ handles this variation of APs’ density problem by randomly switching off some of the APs (i.e., setting their RSS to \(-100\) ) in the training samples and generating new synthetic samples reflecting this behavior. This has the effect of making the training process noisy, forcing the model to learn and generalize from incomplete yet plausible input vectors. This ensures that the model can generalize well in real-world conditions that are, in practice, noisy. Specifically, LiPhi++ generates a binary mask that can be multiplied by the RSS input vector to selectively drop certain APs in a scan (see Figure 9). This trick helps the model generalize and take account of such practical phenomena and also provides the required data for efficient training of the model as discussed in Section 5.2.6.

Due to the noise in the wireless channels and the diversity of the WiFi chips in different devices, the magnitude of the RSS may be shifted with some variance as discussed in Section 2.3 and justified in References [36, 38, 69]. This module generates new synthetic samples to emulate this behavior. Specifically, LiPhi++ adds white Gaussian noise to the different entries of the RSS vectors as shown in Figure 10. The generated data are then combined with the original real data at hand to train the localization model. Signal-shifting yields improved performance relative to the base case. This is due to the variation added to the underlying data distribution for every AP that further enhances the generalization ability of the considered deep model.

The structure of the deep neural network, considered by the LiPhi++ system, is shown in Figure 11. Specifically, the network consists of multiple cascaded LSTM layers typically two followed by three fully connected layers (i.e., dense layers). In addition, considering multiple layers enhances the network’s ability to learn more complexity. In particular, each layer consists of a number of LSTM blocks, each of which has a built-in memory to memorize a sequence of data over time. This memory consists of three non-linear gate units that control the action that should be done on the content stored in memory. A LSTM block operates upon feeding the input sequence then the gates within the block compute a new state and output from the supplied input. The memory of the block is believed to characterize the sequential nature of the input data, which in this case is the historical RSS input. Therefore, it is a good model to capture the evolution of the cell information over time per user location.

Is a sequence of \(u\) timesteps, where each timestep consists of a \(u\) -dimensional RSS input vector ( \(n\) ) as demonstrated in Figure 11. This input is fed to the recurrent neural network during the offline and online phases. Constructing such an input structure involves segmenting the RSS stream into sequences of predefined fixed-length \(u\) , such that the beginning of each RSS sequence overlaps with the end of the preceding RSS sequence. Specifically, each RSS sequence is constituted by \(u-1\) preceding scans in addition to the current one (i.e., successive RSS sequences are shifted by one timestep).

Each entry in the sequence consists of the RSS observation from all the covering \(n\) WiFi access points at an arbitrary time instant. None-heard access points in a given scan are set to \(-\) 100 dBm. Therefore, the input to the deep localization model has a fixed size even with the presence of fluctuating access points (Section 4.6.4 handles this problem). To speed up the model convergence, an additional re-scaling is applied to the RSS sequences’, such that the data spans the range of \([0,1]\) .

Consists of a number of neurons corresponding to the number of considered grid cells (represented by the center of the grid). This network is trained to operate as a multinomial (multi-class) classifier by leveraging a softmax activation function in the output layer. This leads to a probability distribution over the reference fingerprint locations given an input scan.

More formally, each sequence of WiFi scans \(c_i = (c_{i1}, c_{i2}, ..c_{in})\) is fed to the network. The corresponding discrete outputs (i.e., logits) is \(a_i = (a_{i1},a_{i2},\ldots ,a_{im})\) capture the score for each reference points from the possible \(m\) reference points to be the estimated point. The softmax function converts the logit score \(a_{ij}\) (for sequence \(i\) to be from reference point \(j\) ) into a probability as

During the offline phase, the output probability label vector \(P(a_i) = [p(a_{i1}), p(a_{i2})\ldots p(a_{im})]\) is formalized using one-hot-encoding to obtain the ground-truth vector \(g_i\) . This encoding has a probability of one for the correct reference point and zeros for others.

The optimal model is captured through a repeated application of the Backpropagation Through Time algorithm [20]. The model is trained using the Adaptive Moment Estimation (Adam optimizer [25]) to minimize the average cross-entropy between the estimated output probability distribution \(P(a_i)\) and the one-hot-encoded vector \(g_i\) . The loss function is defined as follows:where \(P(a_i)\) is obtained using the softmax function, \(g_i\) is the one-hot encoded vector of the \(i{\rm th}\) input sequence, \(M_s\) is the number of input sequences available for training, and \(D(P(a_i),g_i)\) is the cross-entropy function defined as

The deep RNN architecture is adopted due to its ability to capture the underlying relationship between the signals received from the detectable access points as well as the temporal correlation (i.e., historical changes) between successive scans, leading to better positioning performance. However, it builds undesirable co-dependence between the model’s inputs (i.e., different APs), which may lead to deterioration in the LiPhi++ performance with the high APs’ fluctuation. To increase the model robustness against over-fitting and variation of APs’ density and also accelerate the training process, LiPhi++ employs two regularization techniques and batch normalization: First, we use dropout regularization [56], which has been proved effective in training deep networks [26, 43, 44]. It is used during training to prevent the neurons of a network from developing co-dependencies among each other. In this technique, neurons are stochastically removed from the network architecture during the training phase at different epochs. As a result, these neurons do not contribute to the forward nor the back-propagation passes. This technique can be viewed as sampling different architectures during training, but all these architectures share weights. Second, we leveraged early stopping so that training would stop once the performance improvements are no longer achieved. This is practically achieved by training the network until the performance starts to degrade and subsequently adopting the model that achieved the best performance during the training process. Therefore, the number of epochs is automatically selected to avoid under- and overfitting [10].

Finally, batch normalization is adopted to increase the stability of the neural network and reduce the required time for convergence. It also allows each layer of a network to learn by itself a little bit more independently of other layers. This is done by normalizing the output of a previous activation layer by subtracting the batch mean and dividing by the batch standard deviation.

The goal of this phase is to locate the user in real time using the received WiFi signals from the detectable access points in the area of interest. This can be done by processing the scanned WiFi information and extracting the corresponding RSS sequence as described previously. Thereafter, this sequence is then fed to the trained localization model to get a location estimate as one of the defined reference points (i.e., grid cell centers). The point \(a^*\) with the maximum probability given the RSS input sequence ( \(c\) ) can be selected as the estimated location. That is, we want to find

The challenge here is that the built model can predict the user locations and only at a few discrete locations. As such, the estimated locations, even with a very accurate localization model, will be spaced out leading to a bad user experience. Therefore, this phase aims to track the user in the continuous spatial space (i.e., anywhere even in locations different from the reference points). To do so, LiPhi++ reports the center of mass of all reference points, i.e., by applying a spatial weighted average over the reference points, where the weights of each point are chosen as their corresponding likelihood as output from the classifier network [24]. More formally,where \(a_{ix}\) and \(a_{iy}\) are the spatial coordinates of reference point \(a_i\) , and \(P_i\) is its corresponding softmax likelihood.

This part of the module is optional and can operate only when the localization model becomes outdated due to long-term variations, e.g., changes in the building structure, furniture placement, the available APs, and so on. LiPhi++ does this in two steps (1) updating the fingerprint database and (2) generating a new model. In such cases, we can temporarily add transportable LRSs to the area of interest for collecting a new fingerprint database by the systems’ users and the labels are transparently assigned in the same way as discussed previously. This database is then used to train the deep model. To minimize training time and reduce the data needed as compared to the model creation time, transfer learning is leveraged. Specifically, LiPhi++ adopts fine-tuning to update the learned RSS-to-location mapping by learning a new mapping when the RSS level changes. It is worth mentioning that the hyperparameters do not change in the model update process. Additionally, the time taken for re-training the model depends on the amount of the newly collected data and the convergence time of the model. The fresh data are usually small (as low as one sample per location [29]) if the environment does not significantly change, and the model is frequently updated. The convergence time of the model is, in general, fast (around 13 epochs), since it is initialized with a well-trained prior state (from the previous training runs), as confirmed in References [1, 29].

Figure 14 shows the effect of changing the virtual cell spacing (i.e., reference point density) on the overall accuracy of the system and the corresponding time required by the WiFi-based Trace Estimator module to provide a location estimate. The run-time is calculated using a Lenovo Thinkpad X1 laptop running a 2.2-GHz Intel i7-8750H processor with 64 GB RAM. The figure shows that, as expected, a smaller spacing between reference points yields high localization accuracy with a negligible delay in the calculation time. Nevertheless, this is performed only during the offline phase and LiPhi++ does not affect the real-time performance of the system. Additionally, the figure shows that a reference point spacing of up to 0.5 m is enough to maintain the high accuracy of LiPhi++. It is worth mentioning that at a grid spacing of 1.5 m, a remarkable relative drop in the system accuracy is observed as virtual reference points have been defined over some non-accessible locations.

Figure 15 shows the effect of changing the APs density on the system accuracy. For this, we uniformly and incrementally removed APs from the eight total reference APs present in the area of interest. The figure shows the accuracy of the WiFi trace estimator module degrades as the number of available APs is reduced. However, even with as low as five reference APs, LiPhi++ maintains a steady localization error of around 1 m. This high accuracy with a relatively low number of APs can be explained as a result of two processes: First, the location resetting using the accurate LRS trace estimator. Second, the used data augmentation techniques help the model to maintain its localization accuracy even with low AP densities. This highlights the robustness of LiPhi++.

In this section, we study the influence of training the localization model of LiPhi++ using WiFi scans labeled by LRS as compared to labeling using the WiFi-based Trace Estimator module only. Figure 16 shows boxplots of the localization error of the system in both cases. The figure depicts that, as expected, leveraging LRS gives a drastic improvement in median error (273.8%), compared to the case of relying only on the coarse-grained WiFi-based labels. This can be attributed to the LRSs’ refinement of the training data, which significantly enhances the learning of the localization model and justifies the impact of using the place-and-play LRSs on the LiPhi++ system.

Deep learning is designed to provide a hierarchical learning ability that can be achieved through cascading different layers. Therefore the number of layers of the deep network is one of the effective hyperparameters to boost the system performance. Figure 17 shows the effect of changing the number of layers on LiPhi++ accuracy. Empirically, the figure shows that increasing the number of layers increases the accuracy. This can be justified as the deeper models have more parameters and better learning ability. The figure also shows that, beyond an optimal value of five layers, the model tends to overfit the training data, leading to an accuracy drop.

Figure 18 shows the effect of varying the number of the WiFi scans utilized in estimating a location by the WiFi-based estimator module (Section 4.2). The figure shows that the more scans are fed to the module, the better localization accuracy until it reaches an optimal value at \(v= 5\) beyond which degradation occurs. This can be justified by two opposing factors. (1) Increasing \(v\) results in more information for location smoothing and outlier avoidance. (2) However, as \(v\) increases, more time is spent to collect these samples, which may lead to locating the user in a preceding location (i.e., latency in response). A balance is achieved at a window size of five scans, which leads to the best performance.

Initially, 13 K and 208 K samples are automatically collected and labeled at the Office and the Floor testbeds, respectively. Then, the data augmentation module increases the amount of training data multiple times, enabling efficient utilization of deep learning models. Figure 19 shows the effect of leveraging the augmented data on the localization performance. The figure shows that data augmentation improves the LiPhi++ performance compared to augmentation-free training by 66.2%. The figure also confirms that the more training samples generated by the augmentation technique, the better the performance to cope with real-world deployments by implicitly simulating the inherent variation of the noisy wireless channel. Beyond using three times multiple of the original data, LiPhi++ performance tends to saturate then deteriorate as the noisy data becomes dominant relative to the non-synthetic/original samples.

Figure 20 shows the performance of LiPhi++ when varying the number of timesteps of the sequence that is fed to the RNN as an input. The figure depicts that as the input sequence gets longer, the positioning accuracy improves. This is due to the fact that the localization model has more information (multiple scans) over time, which helps the model to avoid spurious samples generated due to temporal signal variations. The model saturates at an optimal value of five timesteps (i.e., 5 seconds), which yields the best performance. Note that since LiPhi++ works with overlapping sequences, it provides an estimate for every one (the scanning rate), enabling real-time tracking. Longer sequences exceeding this length may lead to a drop in the system performance as the sequence will cover multiple user locations.

In this section, we evaluate how the system would perform in two different testbeds: the Floor testbed and the Office testbed (Figures 12 and 13). The former is larger (629 m \(^2\) ) with many rooms; therefore, it requires a wider coverage by LRSs while the latter is smaller (240 m \(^2\) ) an open area without inner walls. In the Floor testbed, six LRSs are leveraged to cover the whole area of 12 rooms incrementally, and the Office testbed has four LRSs, covering the entire area of interest. Figure 21 shows that LiPhi++ obtains better performance in the Floor testbed as compared to the Office testbed. This can be justified due to two reasons: (1) The number of considered APs (input vector) in the Floor testbed is more (136 and 52 on the Floor and the Office testbeds, respectively), which favors its performance as the model learns more information, and (2) WiFi signatures are more location discriminative in the Floor testbed due to the presence of walls and the richer multi-path environment.

Figure 22 shows the localization performance of LiPhi++ when training the localization model from scratch with the whole dataset compared to fine-tuning the already trained model with a few new samples collected later after some changes in the environment (typically 1,000 samples). The figure shows that LiPhi++ behaves equally in terms of localization accuracy for the two cases. However, fine-tuning provides tremendous savings of training time in the online phase as it takes as low as only 13 epochs for convergence. However, training the model from scratch requires 1,225 epochs to converge. This is because fine-tuning starts with good initial values of the parameters as compared to starting from a random set of parameters in the other case.

Figures 23 and 24 show the CDF of distance error for the three techniques with temporal variations. Figure 23 shows the accuracy of the three systems when tested with a fresh fingerprint. Specifically, LiPhi++ and WiDeep [1] are matched in performance even though WiDeep [1] is trained with manual fingerprinted data while LiPhi++ is trained with automatically constructed fingerprints. This can be attributed due to the accurate labeling obtained from the LRS-based trace estimator module and the well-designed deep localization model that considers the evolution of the data among consecutive WiFi scans. HybridLoc [52] obtains the lowest performance, as BLE-based tracking provides coarse-grained labeling, which is not enough for annotating WiFi scans. Additionally, it uses a probabilistic method that does not benefit from the deep learning methods advantages.

Figure 24 shows how all systems would perform when tested four months later. The figure illustrates that our LiPhi++ system achieves improvements in localization performance by 284.7% and 418%, under this condition, as compared to HybridLoc and WiDeep; respectively. This can be explained by the combination of the data augmentation methods (AP dropping and signal-shifting) in the training data and the adoption of different regularization techniques, which gives LiPhi++ greater flexibility and generalization ability than the other systems. Additionally, LiPhi++ and HybridLoc [52] obtained better performance as compared to WiDeep [1] as they have provisions to update the fingerprint database and the localization model. Despite the high overhead spent to collect data for WiDeep [1] as a traditional fingerprinting technique, its accuracy cannot be maintained without re-doing the arduous calibration process.

LiPhi++ outperforms the earlier work in Reference [48], when tested with fresh and time-variant test data, by 18.5% and 37.3%. This can be justified due to the ability of the new localization model (multimodal deep recurrent neural network to learn/estimate the user location from a sequence of input scans rather than a single scan as in Reference [48]. As a result, the localization model learns the underlying relationship between the RSSs received from APs and the temporal correlation (i.e., signal evolution) between successive scans, leading to better localization performance.

In summary—as shown in Table 3—LiPhi++ is robust to variation over time surpassing the other techniques. This highlights the promise of LiPhi++ in enabling an accurate localization model with zero calibration overhead.

We used a Lenovo Thinkpad X1 laptop running a 2.2-GHz Intel i7-8750H processor with 64 GB RAM for evaluating the end-to-end running time of the different techniques. Figure 25 shows the results. The figure shows that as LiPhi++, Liphi (the earlier work in Reference [48]), and WiDeep [1] are all deep neural network-based systems, they need to pass the data through all the layers of the network. This takes more time than the traditional probabilistic technique proposed in HybridLoc [52]. LiPhi++ needs less location-inference time compared to Liphi and WiDeep, as LiPhi++ has a fewer number of layers and neurons and, by extension, a smaller number of calculations. Nevertheless, since the sampling rate is set to 1 ms, all techniques’ running time allows them to provide real-time location tracking.

In this section, we evaluate the robustness of all systems to device heterogeneity. Initially, all systems are trained and tested with data collected by the same set of devices (i.e., Samsung Note 8, HTC One X9, Motorola Moto G5), which is shown in Figure 26. The figure also shows the performance of all systems when tested with data collected by Google Pixel XL (i.e., not included in the training set), which has completely different form factors and WiFi chips. HybridLoc [52] provides acceptable adaptability to the device heterogeneity as it utilizes probabilistic techniques, which are known to perform well in the presence of uncertainty. WiDeep [1] leverages denoising autoencoders and models device heterogeneity effect as an additive noise leading to remarkable robustness. The figure confirms that LiPhi++ provides superior robustness to the device heterogeneity problem (approximately the same accuracy when testing with the different testing device as when testing with the same training devices). This is due to the combination of data augmentation (i.e., Signal-shifting and spatial discretizer) in the training data and the adoption of a recurrent neural network. The RNN implicitly learns the location from relative RSS values in the input sequence rather than the absolute RSS amplitudes in a single scan, which gives LiPhi++ greater flexibility than the other systems.