-
Bridging the Communication Gap: Artificial Agents Learning Sign Language through Imitation
Authors:
Federico Tavella,
Aphrodite Galata,
Angelo Cangelosi
Abstract:
Artificial agents, particularly humanoid robots, interact with their environment, objects, and people using cameras, actuators, and physical presence. Their communication methods are often pre-programmed, limiting their actions and interactions. Our research explores acquiring non-verbal communication skills through learning from demonstrations, with potential applications in sign language compreh…
▽ More
Artificial agents, particularly humanoid robots, interact with their environment, objects, and people using cameras, actuators, and physical presence. Their communication methods are often pre-programmed, limiting their actions and interactions. Our research explores acquiring non-verbal communication skills through learning from demonstrations, with potential applications in sign language comprehension and expression. In particular, we focus on imitation learning for artificial agents, exemplified by teaching a simulated humanoid American Sign Language. We use computer vision and deep learning to extract information from videos, and reinforcement learning to enable the agent to replicate observed actions. Compared to other methods, our approach eliminates the need for additional hardware to acquire information. We demonstrate how the combination of these different techniques offers a viable way to learn sign language. Our methodology successfully teaches 5 different signs involving the upper body (i.e., arms and hands). This research paves the way for advanced communication skills in artificial agents.
△ Less
Submitted 14 June, 2024;
originally announced June 2024.
-
Signs of Language: Embodied Sign Language Fingerspelling Acquisition from Demonstrations for Human-Robot Interaction
Authors:
Federico Tavella,
Aphrodite Galata,
Angelo Cangelosi
Abstract:
Learning fine-grained movements is a challenging topic in robotics, particularly in the context of robotic hands. One specific instance of this challenge is the acquisition of fingerspelling sign language in robots. In this paper, we propose an approach for learning dexterous motor imitation from video examples without additional information. To achieve this, we first build a URDF model of a robot…
▽ More
Learning fine-grained movements is a challenging topic in robotics, particularly in the context of robotic hands. One specific instance of this challenge is the acquisition of fingerspelling sign language in robots. In this paper, we propose an approach for learning dexterous motor imitation from video examples without additional information. To achieve this, we first build a URDF model of a robotic hand with a single actuator for each joint. We then leverage pre-trained deep vision models to extract the 3D pose of the hand from RGB videos. Next, using state-of-the-art reinforcement learning algorithms for motion imitation (namely, proximal policy optimization and soft actor-critic), we train a policy to reproduce the movement extracted from the demonstrations. We identify the optimal set of hyperparameters for imitation based on a reference motion. Finally, we demonstrate the generalizability of our approach by testing it on six different tasks, corresponding to fingerspelled letters. Our results show that our approach is able to successfully imitate these fine-grained movements without additional information, highlighting its potential for real-world applications in robotics.
△ Less
Submitted 5 June, 2023; v1 submitted 12 September, 2022;
originally announced September 2022.
-
WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language
Authors:
Federico Tavella,
Viktor Schlegel,
Marta Romeo,
Aphrodite Galata,
Angelo Cangelosi
Abstract:
Signed Language Processing (SLP) concerns the automated processing of signed languages, the main means of communication of Deaf and hearing impaired individuals. SLP features many different tasks, ranging from sign recognition to translation and production of signed speech, but has been overlooked by the NLP community thus far. In this paper, we bring to attention the task of modelling the phonolo…
▽ More
Signed Language Processing (SLP) concerns the automated processing of signed languages, the main means of communication of Deaf and hearing impaired individuals. SLP features many different tasks, ranging from sign recognition to translation and production of signed speech, but has been overlooked by the NLP community thus far. In this paper, we bring to attention the task of modelling the phonology of sign languages. We leverage existing resources to construct a large-scale dataset of American Sign Language signs annotated with six different phonological properties. We then conduct an extensive empirical study to investigate whether data-driven end-to-end and feature-based approaches can be optimised to automatically recognise these properties. We find that, despite the inherent challenges of the task, graph-based neural networks that operate over skeleton features extracted from raw videos are able to succeed at the task to a varying degree. Most importantly, we show that this performance pertains even on signs unobserved during training.
△ Less
Submitted 11 March, 2022;
originally announced March 2022.
-
Phonology Recognition in American Sign Language
Authors:
Federico Tavella,
Aphrodite Galata,
Angelo Cangelosi
Abstract:
Inspired by recent developments in natural language processing, we propose a novel approach to sign language processing based on phonological properties validated by American Sign Language users. By taking advantage of datasets composed of phonological data and people speaking sign language, we use a pretrained deep model based on mesh reconstruction to extract the 3D coordinates of the signers ke…
▽ More
Inspired by recent developments in natural language processing, we propose a novel approach to sign language processing based on phonological properties validated by American Sign Language users. By taking advantage of datasets composed of phonological data and people speaking sign language, we use a pretrained deep model based on mesh reconstruction to extract the 3D coordinates of the signers keypoints. Then, we train standard statistical and deep machine learning models in order to assign phonological classes to each temporal sequence of coordinates.
Our paper introduces the idea of exploiting the phonological properties manually assigned by sign language users to classify videos of people performing signs by regressing a 3D mesh. We establish a new baseline for this problem based on the statistical distribution of 725 different signs. Our best-performing models achieve a micro-averaged F1-score of 58% for the major location class and 70% for the sign type using statistical and deep learning algorithms, compared to their corresponding baselines of 35% and 39%.
△ Less
Submitted 1 October, 2021;
originally announced October 2021.
-
A novel numerical tool to study electron energy distribution functions of spatially-anisotropic and non-homogeneous ECR plasmas
Authors:
B. Mishra,
A. Pidatella,
S. Biri,
A. Galatà,
E. Naselli,
R. Rácz,
G. Torrisi,
D. Mascali
Abstract:
A numerical tool for analysing spatially anisotropic electron populations in electron cyclotron resonance (ECR) plasmas has been developed, using a trial-and-error electron energy distribution function (EEDF) fitting method. The method has been tested on space-resolved warm electrons in the energy range $2-20\,\mathrm{keV}$, obtained from self-consistent simulations modelling only electron dynamic…
▽ More
A numerical tool for analysing spatially anisotropic electron populations in electron cyclotron resonance (ECR) plasmas has been developed, using a trial-and-error electron energy distribution function (EEDF) fitting method. The method has been tested on space-resolved warm electrons in the energy range $2-20\,\mathrm{keV}$, obtained from self-consistent simulations modelling only electron dynamics in ECR devices, but lacked real-world validation. For experimentally benchmarking the method, we attempted to numerically reproduce the experimental X-ray emission spectrum measured from an argon plasma. Results of this analysis have provided crucial information about density and temperature of warm electrons, and competing distributions of warm and hot electron components. This information can be fed back to simulation models to generate more realistic data. Subsequent application of the numerical tool as described to the improved simulation data can result in continuous EEDFs that reflect the nature of charge distributions in anisotropic ECR plasmas. These functions can be also applied to electron dependent reactions, in order to reproduce experimental results, like those concerning space-dependent K$α$ emissions.
△ Less
Submitted 3 July, 2021;
originally announced July 2021.
-
In-plasma study of opacity relevant for compact binary ejecta
Authors:
Angelo Pidatella,
Sergio Cristallo,
Alessio Galatà,
Marco La Cognata,
Maria Mazzaglia,
Albino Perego,
Roberta Spartà,
Aurora Tumino,
Diego Vescovi,
David Mascali
Abstract:
In the context of the INFN project PANDORA_Gr3 (Plasma for Astrophysics, Nuclear Decays Observation and Radiation for Archaeometry) and of multi-messenger astronomy, we propose a feasibility study for in-laboratory plasma's opacity investigation, in an environment resembling thermodynamic conditions typical of the ejecta of compact binary mergers containing at least a neutron star. We aim to advan…
▽ More
In the context of the INFN project PANDORA_Gr3 (Plasma for Astrophysics, Nuclear Decays Observation and Radiation for Archaeometry) and of multi-messenger astronomy, we propose a feasibility study for in-laboratory plasma's opacity investigation, in an environment resembling thermodynamic conditions typical of the ejecta of compact binary mergers containing at least a neutron star. We aim to advance knowledge on the physics of kilonovae, the electromagnetic transients following a merger, which are relevant for the study of the origin of heavy nuclei in the Universe produced via r-process nucleosynthesis. In this paper, we present preliminary results of numerical simulations for some physics cases considered in the light of a possible experimental setup for future in-laboratory opacity spectroscopic measurements.
△ Less
Submitted 14 January, 2021;
originally announced January 2021.
-
Self-consistent simulations of ECR-based charge breeders: evidence and impact of the plasmoid-halo structure
Authors:
A. Galatà,
C. S. Gallo,
D. Mascali,
G. Torrisi,
M. Caldara
Abstract:
This paper discusses the capture of an ion beam in a magnetized plasma of an Electron Cyclotron Resonance Ion Source based Charge Breeder, as modelled by numerical simulations. As a relevant step forward with respect to previous works, here the capture is modeled by considering a plasma structure determined in a self-consisent way. The plasmoid-halo structure of the ECR plasma - that is consisting…
▽ More
This paper discusses the capture of an ion beam in a magnetized plasma of an Electron Cyclotron Resonance Ion Source based Charge Breeder, as modelled by numerical simulations. As a relevant step forward with respect to previous works, here the capture is modeled by considering a plasma structure determined in a self-consisent way. The plasmoid-halo structure of the ECR plasma - that is consisting of a dense core (the plasmoid) surrounded by a rarefied halo - is further confirmed by the self-consistent calculations, having also some fine structures affected by the electromagnetic field distribution and by the magnetostatic field profile. The capture of Rb1+ ions has been investigated in details, vs. various plasma parameters, and then compared to experimental results.
△ Less
Submitted 4 December, 2019;
originally announced December 2019.
-
ChoiceNet: CNN learning through choice of multiple feature map representations
Authors:
Farshid Rayhan,
Aphrodite Galata,
Timothy F. Cootes
Abstract:
We introduce a new architecture called ChoiceNet where each layer of the network is highly connected with skip connections and channelwise concatenations. This enables the network to alleviate the problem of vanishing gradients, reduces the number of parameters without sacrificing performance, and encourages feature reuse. We evaluate our proposed architecture on three benchmark datasets for objec…
▽ More
We introduce a new architecture called ChoiceNet where each layer of the network is highly connected with skip connections and channelwise concatenations. This enables the network to alleviate the problem of vanishing gradients, reduces the number of parameters without sacrificing performance, and encourages feature reuse. We evaluate our proposed architecture on three benchmark datasets for object recognition tasks (ImageNet, CIFAR- 10, CIFAR-100, SVHN) and on a semantic segmentation dataset (CamVid).
△ Less
Submitted 24 August, 2019; v1 submitted 20 April, 2019;
originally announced April 2019.
-
Experimental nuclear astrophysics in Italy
Authors:
C. Broggini,
O. Straniero,
M. G. F. Taiuti,
G. de Angelis,
G. Benzoni,
G. E. Bruno,
S. Bufalino,
G. Cardella,
N. Colonna,
M. Contalbrigo,
G. Cosentino,
S. Cristallo,
C. Curceanu,
E. De Filippo,
R. Depalo,
A. Di Leva,
A. Feliciello,
S. Gammino,
A. Galatà,
M. La Cognata,
R. Lea,
S. Leoni,
I. Lombardo,
V. Manzari,
D. Mascali
, et al. (14 additional authors not shown)
Abstract:
Nuclear astrophysics, the union of nuclear physics and astronomy, went through an impressive expansion during the last twenty years. This could be achieved thanks to milestone improvements in astronomical observations, cross section measurements, powerful computer simulations and much refined stellar models. Italian groups are giving quite important contributions to every domain of nuclear astroph…
▽ More
Nuclear astrophysics, the union of nuclear physics and astronomy, went through an impressive expansion during the last twenty years. This could be achieved thanks to milestone improvements in astronomical observations, cross section measurements, powerful computer simulations and much refined stellar models. Italian groups are giving quite important contributions to every domain of nuclear astrophysics, sometimes being the leaders of worldwide unique experiments. In this paper we will discuss the astrophysical scenarios where nuclear astrophysics plays a key role and we will provide detailed descriptions of the present and future of the experiments on nuclear astrophysics which belong to the scientific programme of INFN (the National Institute for Nuclear Physics in Italy).
△ Less
Submitted 9 April, 2019; v1 submitted 14 February, 2019;
originally announced February 2019.
-
PANDORA a New Facility for Interdisciplinary In-Plasma Physics
Authors:
D. Mascali,
A. Musumarra,
F. Leone,
F. P. Romano,
A. Galatà,
S. Gammino,
C. Massimi
Abstract:
PANDORA, Plasmas for Astrophysics, Nuclear Decays Observation and Radiation for Archaeometry, is planned as a new facility based on a state-of-the-art plasma trap confining energetic plasma for performing interdisciplinary research in the fields of Nuclear Astrophysics, Astrophysics, Plasma Physics and Applications in Material Science and Archaeometry: the plasmas become the environment for measur…
▽ More
PANDORA, Plasmas for Astrophysics, Nuclear Decays Observation and Radiation for Archaeometry, is planned as a new facility based on a state-of-the-art plasma trap confining energetic plasma for performing interdisciplinary research in the fields of Nuclear Astrophysics, Astrophysics, Plasma Physics and Applications in Material Science and Archaeometry: the plasmas become the environment for measuring nuclear decays rates in stellar-like condition (such as 7Be decay and beta-decay involved in s-process nucleosynthesis), especially as a function of the ionization state of the plasma ions. These studies are of paramount importance for addressing several astrophysical issues in both stellar and primordial nucleosynthesis environment (e.g. determination of solar neutrino flux and 7Li Cosmological Problem), moreover the confined energetic plasma will be a unique light source for high performance stellar spectroscopy measurements in the visible, UV and X-ray domains, offering advancements in observational astronomy. As to magnetic fields, the experimental validation of theoretical first and second order Landé factors will drive the layout of next generation polarimetric units for the high resolution spectrograph of the future giant telescopes. In PANDORA new plasma heating methods will be explored, that will push forward the ion beam output, in terms of extracted intensity and charge states. More, advanced and optimized injection methods of ions in an ECR plasma will be experimented, with the aim at optimizing its capture efficiency. This will be applied to the ECR-based Charge Breeding technique, that will improve the performances of the SPES ISOL-facility currently installed at Laboratori Nazionali di Legnaro-INFN. Finally, PANDORA will be suitable for energy conversion, making the plasma as a source of electromagnetic radiation, for applications in Material Science and Archaeometry.
△ Less
Submitted 1 March, 2017;
originally announced March 2017.
