Latency Compensation in Ultrasound Tactile Presentation by Linear Prediction of Hand Posture

Abstract

Interaction systems using ultrasound haptics technology present tactile stimuli in fixed coordinates in space; hence, system delays cause not only temporal differences in the stimuli but also spatial shifts in presentation points when targets are moving. In particular, if the transducer array is placed surrounding the hand workspace, a shift in ultrasound focus could result in providing strong tactile stimuli to unintended parts of the hand, such as the opposite side of the fingers. In this study, we examine the feasibility of mitigating the delay effect by predicting the surface shape of the hand. The verification system fits the hand surface shape acquired by a depth camera with a hand model represented by low-dimensional posture parameters, and then performs Kalman prediction on the parameter transitions. The results of the user study show that for finger contacts under constant velocity motion conditions, the prediction method can mitigate the decrease in perceived intensity due to ultrasonic focus shift and increase in perceived intensity at undesirable areas.

Appendix

The hand-tracking algorithm implemented in our system is briefly described below. First, downsampling is performed on the point cloud data obtained from the depth cameras. Then, the workspace was divided into voxels, and the point cloud was downsampled to the group of centroids of the points within the voxels. The posture parameters ${\boldsymbol{x}}$ and shape parameters ${\boldsymbol{\beta }}$ are optimized to fit the downsampled point cloud. Accordingly, the following cost function is minimized:

$$\begin{aligned} E({\boldsymbol{x}}, {\boldsymbol{\beta }}) = \lambda _{point}E_{point}({\boldsymbol{x}}, {\boldsymbol{\beta }}) +\lambda _{rot}E_{rot}({\boldsymbol{x}}) + \lambda _{shape}E_{shape}({\boldsymbol{\beta }}) . \end{aligned}$$

(15)

$E_{point}({\boldsymbol{x}}, {\boldsymbol{\beta }})$ is the sum of the distances between points and their nearest vertices in the hand model. However, if the inner product of the normals of a point and its closest vertex is less than a given threshold, then the pair of distances is not included in the sum.

$E_{rot}({\boldsymbol{x}})$ represent the constraints on the direction and magnitude of the rotation vector. When the rotation parameter at joint i is represented as ${\boldsymbol{\theta }}_i \in \mathbb {R}^3$,

$$\begin{aligned} E_{rot}({\boldsymbol{x}}) = \sum _{i = 1}^{16} \Vert A_i {\boldsymbol{\theta }}_i \Vert _2^2. \end{aligned}$$

(16)

$A_i \in \mathbb {R}^{3\times 3}$ is a matrix that constrains the direction of the rotation vector, which is different at each joint.

$E_{shape}({\boldsymbol{\beta }})$ controls individual differences in hand shape. Because in MANO, shape parameters ${\boldsymbol{\beta }}$ are normalized quantities based on principal component analysis,

$$\begin{aligned} E_{shape}({\boldsymbol{\beta }}) = \Vert {\boldsymbol{\beta }} \Vert ^2_2. \end{aligned}$$

(17)

In our system, the above cost function is minimized using the Levenberg–Marquardt method to estimate the mesh model along the point cloud.