This article presents a framework for semi-automated building damage assessment due to earthquake... more This article presents a framework for semi-automated building damage assessment due to earthquakes from remote sensing data and other supplementary datasets, while also leveraging recent advances in machine-learning algorithms. The framework integrates high-resolution building inventory data with earthquake ground shaking intensity maps and surface-level changes detected by comparing pre-and post-event InSAR (interferometric synthetic aperture radar) images. We demonstrate the use of ensemble models in a machine-learning approach to classify the damage state of buildings in the area affected by an earthquake. Both multi-class and binary damage classification are attempted for four recent earthquakes and we compare the predicted damage labels with ground truth damage grade labels reported in field surveys. For three out of the four earthquakes studied, the model is able to identify over fifty percent or nearly half of the damaged buildings successfully when using binary classification. Multi-class damage grade classification using InSAR data has rarely been attempted previously, and the case studies presented in this report represent one of the first such attempts using InSAR data. 1 Introduction In the immediate aftermath of an earthquake, timely and reliable assessment of the impact in terms of the damage sustained by built assets and the associated repair and replacement costs assumes a crucial role in the strategic organization and prioritization of the response and recovery efforts. Selection of a particular approach for damage and loss evaluation typically involves a trade-off between the level of details collected and the timeliness of evaluation. Within the public sector, estimation of damage and loss after a natural disaster is typically performed through field missions using paper forms, or more recently, data capture tools on tablet and mobile devices (Esri, 2021; United Nations Development Programme, 2021). This approach requires the mobilization of technical experts to the affected areas. While field-surveys still remain the most accurate solution, they can be highly resource intensive and time-consuming for disasters affecting large areas. Lack of available and experienced personnel can also be an issue especially for large scale events. Furthermore, for some disasters it might not even be possible to access some of the affected regions due to disruption of the transportation network.
This paper introduces the general editorial policy of i-ETC, a new online Open Access Journal. Th... more This paper introduces the general editorial policy of i-ETC, a new online Open Access Journal. The publication of this academic journal is based on the software Open Journal System and it is maintained by the Department of Electronics, Telecommunications and Computer Engineering (DEETC), at the High Institute of Engineering of Lisbon (ISEL) in Lisbon.
This paper investigates the application of a Convolutional Neural Network (CNN), AlexNet, on the ... more This paper investigates the application of a Convolutional Neural Network (CNN), AlexNet, on the authentication of paintings by different artists, including Portuguese painter Amadeo de Souza Cardoso, Chinese painter Daqian Zhang and Dutch painter Vincent van Gogh. The research is motivated by the studies on the identification of the works by Amadeo based on the painter’s brushstroke implementing Machine Learning algorithms combined with material analysis. The employment of CNN intends to improve the performance of the brushstroke analysis and increase the accuracy while authenticating an artist’ works. The results show that the implementation of AlexNet produces higher accuracies than its counterparts applying previous brushstroke analysis. Notably, when Amadeo drawings are included in the testing based on Amadeo paintings, the accuracies obtained with the original algorithm drop substantially, whilst the counterparts attained with AlexNet improved considerably. However, when other testing sets are introduced, especially the Chinese paintings, the accuracies show a great increase with the original algorithm but a significant decrease with AlexNet. It implies that AlexNet surpasses the traditional computation through learning by examples; it can potentially solve the problem of limited number of artworks by a specific artist for training.
Open research Areas in Information Retrieval, 2013
Currently, the Web is a powerful channel to store and share personal information, particularly th... more Currently, the Web is a powerful channel to store and share personal information, particularly through the several social networks such as Facebook, Twitter or Flickr. These applications are used as privileged tools for communication and information availability. With the proliferation of new devices and with new types of user interfaces that can be used by social networking applications, suitable user interaction methods are becoming even more relevant. In this paper we propose a natural, gesture based, user interface, to interact with a Flickr client application on a Smart TV. The interface is based on a depth sensor and on a image processing method for gestures identification. It can be used to search and browse pictures using only gestures.
This research presents the results of the implementation of deep learning neural networks in the ... more This research presents the results of the implementation of deep learning neural networks in the identification of pure pigments of heritage artwork, namely paintings. Our paper applies an innovative three-branch deep learning model to maximise the correct identification of pure pigments. The model proposed combines the feature maps obtained from hyperspectral images through multiple convolutional neural networks, and numerical, hyperspectral metric data with respect to a set of reference reflectances. The results obtained exhibit an accurate representation of the pure predicted pigments which are confirmed through the use of analytical techniques. The model presented outperformed the compared counterparts and is deemed to be an important direction, not only in terms of utilisation of hyperspectral data and concrete pigment data in heritage analysis, but also in the application of deep learning in other fields. CCS CONCEPTS • Computing methodologies → Neural networks; Artificial intelligence; • Applied computing → Arts and humanities.
Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
This paper compares the test results generated by applying the method for the authentication of p... more This paper compares the test results generated by applying the method for the authentication of paintings by Portuguese artist Amadeo de Souza Cardoso in the interest of exploring the generalisation properties of the algorithm on other artists or genres. This sets the base for the method to be improved and developed accordingly in future applications for a broader audience in a wider setting. The obtained results show that the classifier obtained from the algorithm using paintings appears not to be directly applicable to drawings of the same artist. When the classifier is retrained for a different genre like Chinese paintings or artists like van Gogh, the algorithm appears to perform as well as the classifier on Amadeo paintings, i.e. the algorithm is sufficient for the classification of a specific type of artist or genre.
21st European Signal Processing Conference (EUSIPCO 2013), 2013
In this work we propose a system for region segmentation of hyperspectral painting images based o... more In this work we propose a system for region segmentation of hyperspectral painting images based on pigments identification method and clustering. Our system combines the molecular information (obtained by hyperspectral camera) with the elemental analysis. The data acquired on the paintings are compared with a database of reference materials that belonged to the artist. The use of reference materials turns out to be crucial in overcoming some of the limitations in the interpretation of the reflectance spectra. In addition, our system is able to detect areas of the painting that were done using materials not usual for our artist; this feature may be of great help in forgery detection.
Recentemente, a retroacção de relevância tem sido utilizada para melhorar o desempenho dos sistem... more Recentemente, a retroacção de relevância tem sido utilizada para melhorar o desempenho dos sistemas de pesquisa em base de dados de imagens. Este artigo apresenta um método de Retroacção de Relevância baseado no classificador de Mínimos Quadrados Regularizado e numa técnica de selecção de imagens que permite aumentar a capacidade de aprendizagem do método. São apresentados resultados de testes experimentais.
2019 IEEE 6th Portuguese Meeting on Bioengineering (ENBENG), 2019
This paper describes the study conducted to design, implement and evaluate the LeucoDiff mobile a... more This paper describes the study conducted to design, implement and evaluate the LeucoDiff mobile application, in terms of a tool to help hematology students and professionals learning or improving their competences in calculating the leukocyte formula. LeucoDiff also includes gaming features to motivate the students in the learning process. The application was evaluated with 25 hematology students using a database of 3494 images of blood samples, taken in an optical microscopy, properly evaluated by a set of experts in the field (clinical pathologists).
This paper compares the test results generated by applying the method for the authentication of p... more This paper compares the test results generated by applying the method for the authentication of paintings by Portuguese artist Amadeo de Souza Cardoso in the interest of exploring the generalisation properties of the algorithm on other artists or genres. This sets the base for the method to be improved and developed accordingly in future applications for a broader audience in a wider setting. The obtained results show that the classifier obtained from the algorithm using paintings appears not to be directly applicable to drawings of the same artist. When the classifier is retrained for a different genre like Chinese paintings or artists like van Gogh, the algorithm appears to perform as well as the classifier on Amadeo paintings, i.e. the algorithm is sufficient for the classification of a specific type of artist or genre.
We describe our experiments for the shot boundary detection, high-level feature extraction and se... more We describe our experiments for the shot boundary detection, high-level feature extraction and search. In the shot boundary detection task, we employ our proven method based on the calculation of distances between colour histograms of frames over a range of timescales. For the search task, we tested a different system. This year, content based search is complemented with a new relevance feedback method. Results of one interactive run are presented to evaluate the performance of the new system. In the high-level feature detection task we tested two new methods: naïve model and non-parametric density estimation. We evaluated these models with all keywords.
The value of a landscape is often associated with its visual aesthetic value, but a landscape goe... more The value of a landscape is often associated with its visual aesthetic value, but a landscape goes beyond that. It has other important values, such as historical or social value. The guides and tourist itineraries are useful to help visiting monuments but as far as reading and experiencing a landscape, these are still insufficient. This work consists of developing a mobile application with gaming and social features, to support the experience of visit and reading places with valuable landscapes. The paper describes the design principles used and presents the main issues of the study conducted during the development process of the application. It also presents the results obtained by the tests carried out to evaluate the user experience when visiting and appreciating the value of a landscape with the application. The experiments were carried out in the Cultural Landscape of Sintra, considered world heritage since 1995 [6].
Research methods and tools to deal with the different aspects of producing, describing, processin... more Research methods and tools to deal with the different aspects of producing, describing, processing and presenting multimedia information Project Memoria Content based image and video search Retrieval system using multimedia content and context features Interfaces for multiple purposes (leisure, learning) and users Annotation of the multimedia materials
2021 14th International Conference on Human System Interaction (HSI)
Natural hazard risk is an increasingly important topic in education. This initial study developed... more Natural hazard risk is an increasingly important topic in education. This initial study developed a survey instrument to identify knowledge gaps in Moroccan high school children related to tsunami risk. Morocco is located in a seismically active region of North Africa and has a history of tsunami impacts. Gamification of risk communication and particularly decisions related to natural hazards is thought to be an effective way to communicate the issues in an effective way which motivates learners. This study formed the initial stage of requirements elicitation for the development of a serious game (SG) designed to communicate tsunami risk to school children. The survey comprised 17 Likert items, and was distributed online to Moroccan high school students who were asked the extent to which they agreed or disagreed with the statements. The analysis of the frequency of survey responses showed a lack of recognition related to the natural warning signs of tsunami, and significant uncertainty about degree of tsunami risk in Morocco. The study has informed the development of a set of initial user requirements for an SG which are presented in this paper.
The research conducted a thorough review of evidence and effectiveness of digital serious games (... more The research conducted a thorough review of evidence and effectiveness of digital serious games (SGs) for communication of action-oriented key messages (related to tsunami evacuation) to children. Significant scientific research investigates tsunami risk which is not always effectively communicated to children who are expected to make appropriate decisions during a tsunami event. SGs can motivate children and enhance risk communication. Computing technologies offer a delivery platform to access digital SGs which require little educator training and motivate learners. The research reviewed evidence of effective SGs that address tsunami risk and are deployable by educators. Based on this review, it is evident that very few suitable classroom deployable, digital SGs that address tsunami evacuation decision-making are available. Results showed only 2 suitable, available games. Similarly, while general evidence of the effectiveness of digital SGs was found, the evaluation of and evidence...
Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts - CHI PLAY Companion '16, 2016
This paper proposes and discusses a horse racing game which explores the user behavior when playi... more This paper proposes and discusses a horse racing game which explores the user behavior when playing, to change the gameplay. The game includes a module to capture and analyze, in real time, the body movement and the audio produced by the player. The paper describes the rational and design principles for the game and presents the main issues of the study conducted during the development of the game.
EAI Endorsed Transactions on Creative Technologies
The landscape, in general, is full of meaning and has been studied over time by experts in differ... more The landscape, in general, is full of meaning and has been studied over time by experts in different fields. In 1992, the landscape was considered a World Heritage Site with the designation of "Cultural Landscape". Despite this relevance, in many regions, there are places with valuable landscapes that are unknown to most of the population. This paper describes a mobile tool for finding and sharing places with valuable landscapes, which includes gamification components to motivate users to search for new places and share them. It is also presented an evaluation of the tool, in terms of user experience and including the assessment of gamification features. The tests were carried out in the Cultural Landscape of Sintra, considered world heritage since 1995
This article presents a framework for semi-automated building damage assessment due to earthquake... more This article presents a framework for semi-automated building damage assessment due to earthquakes from remote sensing data and other supplementary datasets, while also leveraging recent advances in machine-learning algorithms. The framework integrates high-resolution building inventory data with earthquake ground shaking intensity maps and surface-level changes detected by comparing pre-and post-event InSAR (interferometric synthetic aperture radar) images. We demonstrate the use of ensemble models in a machine-learning approach to classify the damage state of buildings in the area affected by an earthquake. Both multi-class and binary damage classification are attempted for four recent earthquakes and we compare the predicted damage labels with ground truth damage grade labels reported in field surveys. For three out of the four earthquakes studied, the model is able to identify over fifty percent or nearly half of the damaged buildings successfully when using binary classification. Multi-class damage grade classification using InSAR data has rarely been attempted previously, and the case studies presented in this report represent one of the first such attempts using InSAR data. 1 Introduction In the immediate aftermath of an earthquake, timely and reliable assessment of the impact in terms of the damage sustained by built assets and the associated repair and replacement costs assumes a crucial role in the strategic organization and prioritization of the response and recovery efforts. Selection of a particular approach for damage and loss evaluation typically involves a trade-off between the level of details collected and the timeliness of evaluation. Within the public sector, estimation of damage and loss after a natural disaster is typically performed through field missions using paper forms, or more recently, data capture tools on tablet and mobile devices (Esri, 2021; United Nations Development Programme, 2021). This approach requires the mobilization of technical experts to the affected areas. While field-surveys still remain the most accurate solution, they can be highly resource intensive and time-consuming for disasters affecting large areas. Lack of available and experienced personnel can also be an issue especially for large scale events. Furthermore, for some disasters it might not even be possible to access some of the affected regions due to disruption of the transportation network.
This paper introduces the general editorial policy of i-ETC, a new online Open Access Journal. Th... more This paper introduces the general editorial policy of i-ETC, a new online Open Access Journal. The publication of this academic journal is based on the software Open Journal System and it is maintained by the Department of Electronics, Telecommunications and Computer Engineering (DEETC), at the High Institute of Engineering of Lisbon (ISEL) in Lisbon.
This paper investigates the application of a Convolutional Neural Network (CNN), AlexNet, on the ... more This paper investigates the application of a Convolutional Neural Network (CNN), AlexNet, on the authentication of paintings by different artists, including Portuguese painter Amadeo de Souza Cardoso, Chinese painter Daqian Zhang and Dutch painter Vincent van Gogh. The research is motivated by the studies on the identification of the works by Amadeo based on the painter’s brushstroke implementing Machine Learning algorithms combined with material analysis. The employment of CNN intends to improve the performance of the brushstroke analysis and increase the accuracy while authenticating an artist’ works. The results show that the implementation of AlexNet produces higher accuracies than its counterparts applying previous brushstroke analysis. Notably, when Amadeo drawings are included in the testing based on Amadeo paintings, the accuracies obtained with the original algorithm drop substantially, whilst the counterparts attained with AlexNet improved considerably. However, when other testing sets are introduced, especially the Chinese paintings, the accuracies show a great increase with the original algorithm but a significant decrease with AlexNet. It implies that AlexNet surpasses the traditional computation through learning by examples; it can potentially solve the problem of limited number of artworks by a specific artist for training.
Open research Areas in Information Retrieval, 2013
Currently, the Web is a powerful channel to store and share personal information, particularly th... more Currently, the Web is a powerful channel to store and share personal information, particularly through the several social networks such as Facebook, Twitter or Flickr. These applications are used as privileged tools for communication and information availability. With the proliferation of new devices and with new types of user interfaces that can be used by social networking applications, suitable user interaction methods are becoming even more relevant. In this paper we propose a natural, gesture based, user interface, to interact with a Flickr client application on a Smart TV. The interface is based on a depth sensor and on a image processing method for gestures identification. It can be used to search and browse pictures using only gestures.
This research presents the results of the implementation of deep learning neural networks in the ... more This research presents the results of the implementation of deep learning neural networks in the identification of pure pigments of heritage artwork, namely paintings. Our paper applies an innovative three-branch deep learning model to maximise the correct identification of pure pigments. The model proposed combines the feature maps obtained from hyperspectral images through multiple convolutional neural networks, and numerical, hyperspectral metric data with respect to a set of reference reflectances. The results obtained exhibit an accurate representation of the pure predicted pigments which are confirmed through the use of analytical techniques. The model presented outperformed the compared counterparts and is deemed to be an important direction, not only in terms of utilisation of hyperspectral data and concrete pigment data in heritage analysis, but also in the application of deep learning in other fields. CCS CONCEPTS • Computing methodologies → Neural networks; Artificial intelligence; • Applied computing → Arts and humanities.
Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
This paper compares the test results generated by applying the method for the authentication of p... more This paper compares the test results generated by applying the method for the authentication of paintings by Portuguese artist Amadeo de Souza Cardoso in the interest of exploring the generalisation properties of the algorithm on other artists or genres. This sets the base for the method to be improved and developed accordingly in future applications for a broader audience in a wider setting. The obtained results show that the classifier obtained from the algorithm using paintings appears not to be directly applicable to drawings of the same artist. When the classifier is retrained for a different genre like Chinese paintings or artists like van Gogh, the algorithm appears to perform as well as the classifier on Amadeo paintings, i.e. the algorithm is sufficient for the classification of a specific type of artist or genre.
21st European Signal Processing Conference (EUSIPCO 2013), 2013
In this work we propose a system for region segmentation of hyperspectral painting images based o... more In this work we propose a system for region segmentation of hyperspectral painting images based on pigments identification method and clustering. Our system combines the molecular information (obtained by hyperspectral camera) with the elemental analysis. The data acquired on the paintings are compared with a database of reference materials that belonged to the artist. The use of reference materials turns out to be crucial in overcoming some of the limitations in the interpretation of the reflectance spectra. In addition, our system is able to detect areas of the painting that were done using materials not usual for our artist; this feature may be of great help in forgery detection.
Recentemente, a retroacção de relevância tem sido utilizada para melhorar o desempenho dos sistem... more Recentemente, a retroacção de relevância tem sido utilizada para melhorar o desempenho dos sistemas de pesquisa em base de dados de imagens. Este artigo apresenta um método de Retroacção de Relevância baseado no classificador de Mínimos Quadrados Regularizado e numa técnica de selecção de imagens que permite aumentar a capacidade de aprendizagem do método. São apresentados resultados de testes experimentais.
2019 IEEE 6th Portuguese Meeting on Bioengineering (ENBENG), 2019
This paper describes the study conducted to design, implement and evaluate the LeucoDiff mobile a... more This paper describes the study conducted to design, implement and evaluate the LeucoDiff mobile application, in terms of a tool to help hematology students and professionals learning or improving their competences in calculating the leukocyte formula. LeucoDiff also includes gaming features to motivate the students in the learning process. The application was evaluated with 25 hematology students using a database of 3494 images of blood samples, taken in an optical microscopy, properly evaluated by a set of experts in the field (clinical pathologists).
This paper compares the test results generated by applying the method for the authentication of p... more This paper compares the test results generated by applying the method for the authentication of paintings by Portuguese artist Amadeo de Souza Cardoso in the interest of exploring the generalisation properties of the algorithm on other artists or genres. This sets the base for the method to be improved and developed accordingly in future applications for a broader audience in a wider setting. The obtained results show that the classifier obtained from the algorithm using paintings appears not to be directly applicable to drawings of the same artist. When the classifier is retrained for a different genre like Chinese paintings or artists like van Gogh, the algorithm appears to perform as well as the classifier on Amadeo paintings, i.e. the algorithm is sufficient for the classification of a specific type of artist or genre.
We describe our experiments for the shot boundary detection, high-level feature extraction and se... more We describe our experiments for the shot boundary detection, high-level feature extraction and search. In the shot boundary detection task, we employ our proven method based on the calculation of distances between colour histograms of frames over a range of timescales. For the search task, we tested a different system. This year, content based search is complemented with a new relevance feedback method. Results of one interactive run are presented to evaluate the performance of the new system. In the high-level feature detection task we tested two new methods: naïve model and non-parametric density estimation. We evaluated these models with all keywords.
The value of a landscape is often associated with its visual aesthetic value, but a landscape goe... more The value of a landscape is often associated with its visual aesthetic value, but a landscape goes beyond that. It has other important values, such as historical or social value. The guides and tourist itineraries are useful to help visiting monuments but as far as reading and experiencing a landscape, these are still insufficient. This work consists of developing a mobile application with gaming and social features, to support the experience of visit and reading places with valuable landscapes. The paper describes the design principles used and presents the main issues of the study conducted during the development process of the application. It also presents the results obtained by the tests carried out to evaluate the user experience when visiting and appreciating the value of a landscape with the application. The experiments were carried out in the Cultural Landscape of Sintra, considered world heritage since 1995 [6].
Research methods and tools to deal with the different aspects of producing, describing, processin... more Research methods and tools to deal with the different aspects of producing, describing, processing and presenting multimedia information Project Memoria Content based image and video search Retrieval system using multimedia content and context features Interfaces for multiple purposes (leisure, learning) and users Annotation of the multimedia materials
2021 14th International Conference on Human System Interaction (HSI)
Natural hazard risk is an increasingly important topic in education. This initial study developed... more Natural hazard risk is an increasingly important topic in education. This initial study developed a survey instrument to identify knowledge gaps in Moroccan high school children related to tsunami risk. Morocco is located in a seismically active region of North Africa and has a history of tsunami impacts. Gamification of risk communication and particularly decisions related to natural hazards is thought to be an effective way to communicate the issues in an effective way which motivates learners. This study formed the initial stage of requirements elicitation for the development of a serious game (SG) designed to communicate tsunami risk to school children. The survey comprised 17 Likert items, and was distributed online to Moroccan high school students who were asked the extent to which they agreed or disagreed with the statements. The analysis of the frequency of survey responses showed a lack of recognition related to the natural warning signs of tsunami, and significant uncertainty about degree of tsunami risk in Morocco. The study has informed the development of a set of initial user requirements for an SG which are presented in this paper.
The research conducted a thorough review of evidence and effectiveness of digital serious games (... more The research conducted a thorough review of evidence and effectiveness of digital serious games (SGs) for communication of action-oriented key messages (related to tsunami evacuation) to children. Significant scientific research investigates tsunami risk which is not always effectively communicated to children who are expected to make appropriate decisions during a tsunami event. SGs can motivate children and enhance risk communication. Computing technologies offer a delivery platform to access digital SGs which require little educator training and motivate learners. The research reviewed evidence of effective SGs that address tsunami risk and are deployable by educators. Based on this review, it is evident that very few suitable classroom deployable, digital SGs that address tsunami evacuation decision-making are available. Results showed only 2 suitable, available games. Similarly, while general evidence of the effectiveness of digital SGs was found, the evaluation of and evidence...
Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts - CHI PLAY Companion '16, 2016
This paper proposes and discusses a horse racing game which explores the user behavior when playi... more This paper proposes and discusses a horse racing game which explores the user behavior when playing, to change the gameplay. The game includes a module to capture and analyze, in real time, the body movement and the audio produced by the player. The paper describes the rational and design principles for the game and presents the main issues of the study conducted during the development of the game.
EAI Endorsed Transactions on Creative Technologies
The landscape, in general, is full of meaning and has been studied over time by experts in differ... more The landscape, in general, is full of meaning and has been studied over time by experts in different fields. In 1992, the landscape was considered a World Heritage Site with the designation of "Cultural Landscape". Despite this relevance, in many regions, there are places with valuable landscapes that are unknown to most of the population. This paper describes a mobile tool for finding and sharing places with valuable landscapes, which includes gamification components to motivate users to search for new places and share them. It is also presented an evaluation of the tool, in terms of user experience and including the assessment of gamification features. The tests were carried out in the Cultural Landscape of Sintra, considered world heritage since 1995
Uploads
Papers by Rui Jesus