A multimodal framework for the evaluation of patients’ weaknesses, supporting the design of customised AAL solutions

The recovery of motion abilities after a physical or neurological trauma is a long and winding road that is usually supported by medical staff. In particular, occupational therapists (OTs) play a fundamental role in assessing the performances of patients in daily life tasks and suggesting better practices and aids. The goal of OTs is to promote the patients’ abilities, fostering their independence with the minimum needed supervision. To this purpose, the possibility of remotely operating while being able to monitor the subject with a sufficient level of detail is highly desirable. The development of advanced tools for patient monitoring and supervision with limited intrusiveness is a scientific challenge that spans different technological areas: sensing, localisation, tracking, measurement, business intelligence. The final goal is to provide medical doctors with a set of meaningful metrics related to the patient’s activity. Among them, occupation of living spaces, motion patterns, posture, as well as fine-grained motion-related parameters (e.g., grasping objects, performing elementary tasks of variable difficulty), can be of great help for the medical staff to assess the degree of independence of the user and to recommend suitable assisting devices, more appropriate organisation of living spaces, effective rehabilitation procedures. In this work, we describe a sophisticated setup that has been developed in strict cooperation with medical experts and researchers in both engineering and medicine, to create an augmented reality physical environment supporting occupational therapists and rehabilitation staff in evaluating their patients’ performance and therapy activities. The proposed system is fully automatic and the users detection and tracking is completely markerless and vision-based. It comprises a set of realistic highly-infrastructured living environments, including sensing and actuation devices of different types, which are thoroughly described in Tables 1 and 2 The collected data are analysed by an automated expert system to extract the requested information, which is then presented inside the user interface. The underlying technological complexity is hidden to both patients and the medical staff, to guarantee minimum intrusiveness for the former and maximum operational easiness for the latter. During and after the patients staying inside the living spaces, the medical personnel is enabled to access a large variety of data through a unified interface and is provided with a pre-filled version of the standard patient card, a document that is otherwise filled in by hand. This allows both reducing the time requested for a medical diagnosis and producing a more objective and measurable assessment. A fully operating setup of our system is currently deployed within the living lab AUSILIA (Pisoni et al. 2016).