Score rectification for online assessments in robot-assisted arm rehabilitation

2.1 Disturbance identification

Neurological factors such as progress in learning, motivation and fatigue influence the outcome of robotic scores and bias the estimation of neuromuscular abilities (these and further essential factors listed in Figure 1). Additionally, the robotic system itself influences the estimated score by limiting the active range of motion of the patient due to the robot topology (as observed by [8, 9]). Although recent developments such as ANYexo methods as shown for instance in scores of movement quality [10], harmony [11], and other devices [12] could increase the allowable upper-limb range of motion significantly by including shoulder degrees of freedom, other factors such as perceived robot inertia, maximum speed, comfort of the patient, alignment and systematic measurement errors influence the resulting scores, too. Also the type of assessment task can result in different scoring methods as shown for instance in scores of movement quality [8, 13]. Hereby, type of assessment task is a combination of different task characteristics (see Section 2.1.1) and its manifestations (manipulability effects, i.e. proximal vs distal movements, work against gravity, and path length [14]). Measured scores also depend on type and level of robotic assistance. Type of assistance is defined as the combination of none, one or multiple controller characteristics such as arm weight compensation (as, for instance presented by Just et al. [15]), guidance and tunnel controlller [16], active pushing controller (as, for instance, presented by Baur et al. [5]), or audiovisual feedback [17].

Figure 1:

Identified disturbance vector d. Each entry alters a given robotic score s ^T and makes this score incomparable to previous scores for therapists.

The complexity of our disturbance model was reduced by neglecting influences that stayed constant, as it is sufficient for our rectified score to be relatively comparable (i.e. no ground truth). Disturbances from the robot could not be neglected if data needs to be compared between different systems and clinical environments, however these scenarios were not considered in this work. Further, psychological influences (i.e. motivation and learning effects) on the patient were assumed to stay constant for simplicity. The disturbances considered in this work are: Arm Weight Assistance d _A,AW, Work against Gravity d _T,WG and Muscle Fatigue d _P,FA.

2.1.1 Task characteristics

In a typical rehabilitation exercise, type of task, mapped joints, scoring method, audio, visual and haptic feedback are designed in an integrated manner. Although the tasks of different types share fundamental characteristics (based on [18], e.g. task types can be distinguished in spatial, spatio-temporal or temporal type), these shared characteristics are usually not exploited for the development of context-independent scoring methods. If scoring based on these characteristics is achieved, partial scores could be compared between different task types. As an example, games like holding penalty (C), navigating a bird (I), collecting stars in given order (E) and others (tasks A–D, G–I, see Figure 2) share the characteristic that the user has to reach a specific target area without time constraints, while navigating a bird and similar types (tasks C, G–I, see Figure 2) additionally share the characteristic that the user has to be at specific positions at given times. Further, we can distinguish between tasks where the target can be reached with an arbitrary path (tasks A–C, see Figure 2), and tasks where a clear reference path is given (tasks D–G, see Figure 2). We introduce a process called Score Decomposition (see Section 2.3) and show how these characteristics can be exploited by the robotic accuracy score, exemplified for four 2D tasks (task types A, C, D and G, see Figure 2).

Figure 2:

Typical visual displays of different exercises or games. The associated tasks can be grouped as follows: classical reaching task (A), reaching task with obstacles (B), defend goal task with moving target (C, Armeo Power Game) are all spatial tasks without reference path. Classical follow path task (D), collecting the stars in given order (E) [19], follow the path (F) [19] are all spatial tasks with reference path, but without timing. Classical follow trajectory task (G), inverted pendulum (H) [20]), navigating a bird (I) [21] are all spatio-temporal tasks either with (G, H) or without (I) reference path.

2.1.2 Robotic platform

Exercises and tasks are typically designed robot platform-specific to match the available degrees of freedom and range of motion of the device. Similar to the exercises, this integrated manner often cannot be altered, although the underlying game mechanics would be similar for different devices and different joints in e.g. clinical space or cartesian world space. To achieve independence, joints actively involved in training can be mapped into a normalized training space q = {q ₁, q ₂, …} (joint pose in training space is further denoted as cursor position), where each axis is scaled such that joint minima in clinical or cartesian space map to −1 and joint maxima map to 1. Circular goal areas in clinical or cartesian space thus map to ellipsoids with principal axis v ∈ R m , where m denotes the number of dimensions mapped to the normalized training space (e.g. m = 3 for a standard end-effector training in world space). We implemented this mapping on ANYexo [22], an upper-limb rehabilitation robot designed as a research platform (see Figure 3). The exoskeleton featured six actuated joints of type ANYdrive 2.0 (ANYbotics AG, Switzerland) that were aligned with the human joints of the shoulder and arm. All actuators had a maximum torque of 70 Nm and the joint speed was limited to 6 rad/s. The user and the robot physically interacted with each other at two cushioned locations: one at the upper arm and one at the forearm close to the wrist including a handle. For the scope of this work, we mapped Plane of Elevation (POE) and Angle of Elevation (AOE) of the clinical coordinate system to the training space’s X and Y axis respectively. Kinematic data q was recorded with a sampling frequency of 800 Hz. Tasks on average were executed at a frequency of about 0.3 Hz.

Figure 3:

A healthy participant is training with ANYexo, an upper-limb robotic platform featuring six actuated joints. Plane of Elevation (POE) and Angle of Elevation (AOE) of the clinical coordinate system were mapped to training space X and Y axis.

2.2 Score rectification procedure

Recorded kinematic data q of the robot are dependent on neuromuscular abilities of the patient n _A , the disturbances d , and task type characteristics c _T (e.g. of a spatial or a temporal task) (see Figure 4 data recording). For any given list of ( n _A , d , c _T ), the resulting kinematic data are further processed by various metrics into scores s ^T evaluated at the end of every task. Besides the fact that the scores are influenced by the disturbances, the scores are usually designed task-specific such that they are compliant with the underlying set of characteristics. Even a score on accuracy has been specifically designed for each task [8] (e.g. spatial accuracy based on the distance to goal in a reach goal task [10] versus spatio-temporal accuracy in path following task [23]). To decouple the scores from specific task types, we have introduced a score orthogonalization method where scores can be defined for each characteristic individually via a scoring function g( q , c _T ), and then merged together according to the set of enabled task characteristics c T e with function h c T , i e , s 1 T , s 2 T , … (see Figure 4 score decomposition). Finally, the rectified score s ̂ T presented to the therapist is calculated with function f(s ^T , d) that minimizes the influence of disturbances d according to a model calibrated by experimental data (see Figure 4 disturbance rejection).

Figure 4:

Full balancing model with score orthogonalization functions g and h to decouple the robotic scores from task characteristics, and balancing function f that decouples the scores from the disturbances of the current task T.

2.3 Score decomposition

The score decomposition procedure consists in (1) producing of a list of orthogonal scores (g), and (2) combining them to obtain task type-independent score s ^T  (h). Starting from a classification of tasks in spatial and temporal components, spatial tasks can further be dissected into longitudinal (i.e. towards a goal position) and radial (i.e. toward a reference path) components. Considering accuracy, following a reference trajectory would lead to a combination of both spatial components and the temporal one while following a reference path would lead to a combination of the spatial components only. The resulting possibility to compare different tasks is particularly important when the number of task types with different characteristics increases, e.g. when a large number of activities of daily living are to be trained. In the scope of this work, we applied the decomposition procedure on the accuracy score. Other scores could follow a similar procedure. Example given, smoothness could also be dissected spatially and temporally, but is only assessed for tasks where path or trajectory are available. Other scores such as aiming angle [13] might already be one-dimensional, i.e. no orthogonalization procedure would be necessary.

2.3.2 Radial accuracy

In order to formulate the radial accuracy, the reference path p is defined as a series of N−1 path segments with its corresponding N discretization points p ₁ … p _N (see Figure 5 Radial). Provided a certain cursor position c(t) at time t, one can define its projection on the path c ̄ ( t ) , its radial distance ϑ c ( t ) = | c ( t ) − c ̄ ( t ) | , and the current closest discretization point p _i . Further, ϑ _r,i is defined as the distance from c ̄ ( t ) to the intersection of the nominal target region with the line segment between c ̄ ( t ) and c(t). This distance is scaled with the score baseline s _B and the score is normalized according to Eq. (1). The corresponding partial radial accuracy for discretization point p _i averages all cursor observations O _i (i.e. all update steps where the cursor projected on segment i) within the corresponding path segment, and is updated each observation as follows:

(2) s R , i n + 1 = O i n s R , i n + f n ( ϑ c ( t ) , 2 ϑ r , i , 0 , ϑ r , i ) O i n + 1 ,

where s R , i 0 , O i 0 = 0 and O i n + 1 = O i n + 1 . A reasonable boundary for the worst possible score s _min was found to be double the distance to the target area. The baseline score of 0 is achieved as soon as the cursor reaches the surface of the ellipsoid. The radial accuracy score s R T weights all partial accuracy scores with the path length ξ _p,i of the corresponding segment:

(3) s R T = 1 ξ p ∑ i = 0 N − 1 ξ p , i s R , i .

Figure 5:

Decomposition of the accuracy score into longitudinal, radial and temporal component enables comparability of these scores between tasks of different types. Orange: target, green: reference, yellow: segment of interest.

2.4 Disturbance rejection

Disturbance rejection can be divided into three parts: work against gravity, fatigue and assistance. To reject work against gravity (i.e. compensating for the direction of movement), the model can be simplified if we assume a set of distinct directions v _i . The function f for the current task can be approximated by a linear combination of the set of functions f _i , i.e. disturbance rejection for direction i. The functions are weighted according to their angular distance towards the current direction v ̄ :

The work against gravity can then be rejected by correcting the score to the mean value s ̄ i achieved for direction i. Further, we defined the rejection method as a subsequent process of assistance f _A,i , fatigue f _F,i and work against gravity rejection:

3.1 Model tuning

To be able to validate our model and generalize the results, we split the data of the six participants into equal-sized training and test set. Since a model that requires participant-specific calibration tasks prior to training would contradict our initial goal of shifting from discrete to online assessments, we decided on comparability between different movement directions. Therefore we split the tasks into predefined training set (T ₁, T ₃, T ₅, T ₇) and test set (T ₂, T ₄, T ₆, T ₈). Since work against gravity has a significant influence on the disturbance model and is highest along the vertical axis, it was decided not to randomize the train-test split but rather assign tasks to the training set along the global horizontal and vertical axis. The parameters and models as described in the previous chapter were then tuned on the training set using regression, and applied to the test set (see Table 2). The total amount of data points for training was n = 768 (six participants with eight sessions each, containing four exercises with four task directions each).

4.1 Fatigue rejection

The fatigue and regeneration parameters λ _i and μ _i were tuned for the directions along the horizontal and vertical axis (see Table 3) on data from all participants. After applying the accumulated fatigue model to the unrectified combined score s ^T (t) for direction 2 of the test set, the long-term difference in mean score between sessions S ₁ and S ₈ dropped from Δs ^T (S ₁, S ₈) = 0.11 to Δ s ̂ T ( S 1 , S 8 ) = 0.02 (see Figure 7). The difference in mean score between exercises E 12 = E 1 + E 2 2 and E 34 = E 3 + E 4 2 for session S ₈ dropped significantly from Δs ^T (E ₁₂, E ₃₄) = 0.12 to Δ s ̂ T (₁₂, E ₃₄) = −0.06. The difference between scores for exercises with high assistance level (l _A > 0.5) changed insignificantly after applying the fatigue model, as the accumulated fatigue C(t) depleted during sessions with high assistance. C(t) was highest for session S ₈ and S ₁.

Figure 7:

Fatigue Rectification for T ₂. For each session S ₁ − S ₈ and exercise E ₁ − E ₄, data from all participant were aggregated. The difference in the mean score between S ₁ and S ₈ (Δs ^T (S ₁, S ₈)) as well as the difference between E ₁₂ and E ₃₄ for S ₈ (Δs ^T (E ₁₂, E ₃₄) were significantly lower after the tuned fatigue model (C(t)) was applied on the unrectified score. The accumulated fatigue C(t) is inversely proportional to s ^T .

4.2 Full disturbance rejection pipeline

The full rejection pipeline is exemplified on participant P6 (see Figure 8). Without any rejection, scores were dependent on the movement direction. P6 achieved a score of s ^T = −0.29 without assistance in upwards direction (T = 1, see Figure 8A). The unrectified score between assistance levels ( s 1 , l A = 0 T = − 0.29 , s 1 , l A = 1 T = 0.64 ) was eminent. For l _A = 1, similar scores were observed independent of the direction of movement. Applying the fatigue model seemed to increase scores of all assistance levels for upwards directions. Applying the assistance model seemed to decrease scores of high assistance level. Finally, applying the work against gravity model seemed to increase all scores in all directions. With the exception of direction T ₈, scores in the range between s ^T = 0.65 and s ^T = 0.74 were observed for all directions and assistance levels after applying the full disturbance rejection pipeline.

Figure 8:

Full disturbance rejection pipeline demonstrated on participant P6: the accuracy score s ^T ∈ { − 1, 1} was averaged over all data for a given direction from sessions with equal assistance level l _A . Process steps are: A − B: fatigue rejection, B − C: assistance rejection, C − D: work against gravity rejection. Green arrows indicate regions with most significant changes in the score.

The rectification model was applied to all six participants (see Figure 9). The difference in unrectified scores for participants P2 and P4 in all directions and all assistance levels was small, while a significant increase in the unrectified score with increasing assistance level was observed for participants P5 and P6. For participants P1 and P3, increasing the assistance level had negative effects on the score (e.g. Δs ^T = −0.11 for P3 in direction T = 2 with Δl _A = 1). Over all participants, an increase in the average accuracy score in all directions could be observed after applying the rectification model. The difference in standard deviation of all data points (n = 32, 8 directions × 4 assistance levels) per participant between unrectified and rectified scores seemed to correlate with the assistance-dependent differences in unrectified scores (see Figure 10). For participant P5, the standard deviation could be reduced significantly from σ _s = 0.15 to σ _s = 0.05 after applying rectification. For participants where the assistance level had negative effects on the score (i.e. P1 and P3), the standard deviation of the scores increased slightly from σ _s = 0.11 to σ _s = 0.14 for P1.

Figure 9:

Comparison between unrectified normalized scores s ^T ∈ {−1, 1} (red) and rectified scores (blue) for different levels of assistance (l _A ) and participants P1–P6. Rectification effects were strongest for participants with significant responses to the provided assistance (P5, P6). Opposite effects were observed for participants where applied assistance led to a decrease in scores.

Figure 10:

Standard deviation between all unrectified (red) and rectified scores (blue) over all sessions and directions for each participant. Rectification led to a significant decrease in scoring variability for participants with high responses to the provided assistance (P5, P6).

5.1 Score decomposition could deepen insights into the user’s neuromotor abilities during training

Decomposition and (re-)combination of scores was applied on the accuracy score for robot-assisted rehabilitation therapy. This process could result in deeper insights into the performance of patients within the current and past tasks, and their respective contexts. While decomposed scores allowed to extract information about the patient’s neuromotor abilities independent of the context that the movement was performed in, the combination of these orthogonal components could help the therapist to observe how the patient changes behavior upon contextual variation. For example, it could be determined by looking at decomposed and combined scores to what functional components patient’s optimized their movement for when provided only with the combined scores as feedback.

5.2 Score decomposition allows combination of scores with different units

Further, the problem of combining scores with different units is not new, and various approaches have been proposed in the past to solve this issue (e.g. to compare spatial and temporal errors [26]). However, these approaches were specifically designed for a predefined set of robotic scores dependent on the given task. In our approach we sought for functional orthogonal components that could be normalized into a generalized training space, i.e. to make them independent of the type of task. Our approach has the additional advantage that therapists or automated high-level algorithms can set individualized, patient-specific baselines for each of the components. Increasing the set of tasks does not alter the decomposed scoring functions and thus comparability of orthogonalized scores within new types of tasks is guaranteed. The generalizability of the procedure to other robotic scores such as movement smoothness, aiming angle, reaction time, or joint speed, seems straightforward but needs to be proven in the future.

5.3 Fatigue rejection model could predict optimal exercise switching in therapy

We were able to show accurate modeling of fatigue throughout our experiment sessions for each healthy participant (see resulting rectification scores in Figure 9). The model rejected disturbances in scoring over a whole training (T(S ₁, S ₈) was reduced from 0.11 to 0.001, see Figure 7) and over each session (Δ ^T (E ₁, E ₄) was reduced from 0.35 to 0.04). We are thus confident that our fatigue model could generalize well to other healthy participants. It could be investigated if less data (e.g. only considering direction T = 0) could also be sufficient to tune this model. Generalization to patients has to be proven in future work, however, as additional influences would have to be considered, e.g. compensatory movements upon high muscle fatigue. Such compensatory movements were also observed during our experiment with healthy participants, especially for exercises 3 and 4. Muscle fatigue is an important factor in therapy planning and execution, as it deteriorates proprioception and motor control during training [20]. Usually, therapists estimate fatigue qualitatively and counteract the issue by e.g. switching to a different exercise targeting different muscle groups. With our approach, therapy systems could switch automatically from one exercise to another in an attempt of keeping the patient at an optimal challenge point.

5.4 Similar balancing effects as in motor learning for healthy participants were observed

Results showed that fatigue rejection increased scores of all assistance levels for upwards directions, whereby scores with lower assistance levels were increased stronger than scores with higher assistance. In contrast, assistance rejection targeted scores with high assistance values and lowered them for participants with significant responses to the provided assistance, i.e. it balanced the scores after fatigue rejection. Interestingly, scoring errors of P1 and P3 were amplified, since assistance rejection was inverted. Similar to effects observed for motor learning [27], over-applying assistance to healthy participants hampered scoring. These findings may stimulate further research of this effect, as important conclusions could be drawn towards the generalization potential of models trained on healthy participants when applying them to patients.

5.5 Comparable online assessments are technically feasible and potentially applicable to multiple areas

The combination of score decomposition and disturbance rejection worked well for healthy participants where assistance had a reasonable influence on the score. Overall, we could see clear evidence that comparable online assessments are technically feasible with an appropriate decomposition and disturbance rejection method, as the overall standard deviation of the assessed score in changing context could be reduced by a factor of 3.2 in the best case, and 0.8 in the worst case (see Figure 10). Being able to compare robotic scores online has advantages in a multitude of areas: In robot-assisted motor learning for healthy trainees (e.g. strength training combined with an accuracy task as given in our experiments), but also in neuro-rehabilitation for patients, as introduced. For the latter, further steps would be needed to transfer the method into the rehabilitation setting, by validating the method on patients and analyzing the correlation to established clinical scores such as Fugl–Meyer. Further, our approach could enable the direct comparison of the performance of developed controllers between different research groups, robotic devices, and task settings. The existent comparability issue is especially distinctive in neuro-rehabilitation, considering the high amount of different controllers that were developed and tested on small patient groups with questionable generalizability. For this, the feasibility of transferring the decomposition method to various other robotic scores could also be explored.

Finally, it has to be investigated if disturbances besides fatigue, assistance, and work against gravity could also have a significant influence on the scoring, which was not covered in this work: path length, motivation, and different assistance methods such as tunnel or guidance controllers.