-
The long-term and disparate impact of job loss on individual mobility behaviour
Authors:
Simone Centellegher,
Marco De Nadai,
Marco Tonin,
Bruno Lepri,
Lorenzo Lucchini
Abstract:
In today's interconnected world of widespread mobility, ubiquitous social interaction, and rapid information dissemination, the demand for individuals to swiftly adapt their behaviors has increased dramatically. Timely decision-making faces new challenges due to the necessity of using finely temporal-resolved anonymised individual data to keep up with fast-paced behavioural changes. To tackle this…
▽ More
In today's interconnected world of widespread mobility, ubiquitous social interaction, and rapid information dissemination, the demand for individuals to swiftly adapt their behaviors has increased dramatically. Timely decision-making faces new challenges due to the necessity of using finely temporal-resolved anonymised individual data to keep up with fast-paced behavioural changes. To tackle this issue, we propose a general framework that leverages privacy-enhanced GPS data from mobile devices alongside census information to infer the employment status of individuals over time. By analysing the mobility patterns of employed and unemployed individuals, we unveil significant differences in behaviours between the two groups, showing a contraction in visited locations and a general decline in the exploratory behaviour of unemployed individuals. Remarkably, these differences intensify over time since job loss, particularly affecting individuals from more vulnerable demographic groups. These findings highlight the importance of early monitoring of unemployed individuals who may face enduring levels of distress. Overall, our findings shed light on the dynamics of employment-related behaviour, emphasizing the importance of implementing timely interventions to support the unemployed and vulnerable populations.
△ Less
Submitted 15 March, 2024;
originally announced March 2024.
-
Towards Graph Foundation Models for Personalization
Authors:
Andreas Damianou,
Francesco Fabbri,
Paul Gigioli,
Marco De Nadai,
Alice Wang,
Enrico Palumbo,
Mounia Lalmas
Abstract:
In the realm of personalization, integrating diverse information sources such as consumption signals and content-based representations is becoming increasingly critical to build state-of-the-art solutions. In this regard, two of the biggest trends in research around this subject are Graph Neural Networks (GNNs) and Foundation Models (FMs). While GNNs emerged as a popular solution in industry for p…
▽ More
In the realm of personalization, integrating diverse information sources such as consumption signals and content-based representations is becoming increasingly critical to build state-of-the-art solutions. In this regard, two of the biggest trends in research around this subject are Graph Neural Networks (GNNs) and Foundation Models (FMs). While GNNs emerged as a popular solution in industry for powering personalization at scale, FMs have only recently caught attention for their promising performance in personalization tasks like ranking and retrieval. In this paper, we present a graph-based foundation modeling approach tailored to personalization. Central to this approach is a Heterogeneous GNN (HGNN) designed to capture multi-hop content and consumption relationships across a range of recommendable item types. To ensure the generality required from a Foundation Model, we employ a Large Language Model (LLM) text-based featurization of nodes that accommodates all item types, and construct the graph using co-interaction signals, which inherently transcend content specificity. To facilitate practical generalization, we further couple the HGNN with an adaptation mechanism based on a two-tower (2T) architecture, which also operates agnostically to content type. This multi-stage approach ensures high scalability; while the HGNN produces general purpose embeddings, the 2T component models in a continuous space the sheer size of user-item interaction data. Our comprehensive approach has been rigorously tested and proven effective in delivering recommendations across a diverse array of products within a real-world, industrial audio streaming platform.
△ Less
Submitted 12 March, 2024;
originally announced March 2024.
-
Personalized Audiobook Recommendations at Spotify Through Graph Neural Networks
Authors:
Marco De Nadai,
Francesco Fabbri,
Paul Gigioli,
Alice Wang,
Ang Li,
Fabrizio Silvestri,
Laura Kim,
Shawn Lin,
Vladan Radosavljevic,
Sandeep Ghael,
David Nyhan,
Hugues Bouchard,
Mounia Lalmas-Roelleke,
Andreas Damianou
Abstract:
In the ever-evolving digital audio landscape, Spotify, well-known for its music and talk content, has recently introduced audiobooks to its vast user base. While promising, this move presents significant challenges for personalized recommendations. Unlike music and podcasts, audiobooks, initially available for a fee, cannot be easily skimmed before purchase, posing higher stakes for the relevance…
▽ More
In the ever-evolving digital audio landscape, Spotify, well-known for its music and talk content, has recently introduced audiobooks to its vast user base. While promising, this move presents significant challenges for personalized recommendations. Unlike music and podcasts, audiobooks, initially available for a fee, cannot be easily skimmed before purchase, posing higher stakes for the relevance of recommendations. Furthermore, introducing a new content type into an existing platform confronts extreme data sparsity, as most users are unfamiliar with this new content type. Lastly, recommending content to millions of users requires the model to react fast and be scalable. To address these challenges, we leverage podcast and music user preferences and introduce 2T-HGNN, a scalable recommendation system comprising Heterogeneous Graph Neural Networks (HGNNs) and a Two Tower (2T) model. This novel approach uncovers nuanced item relationships while ensuring low latency and complexity. We decouple users from the HGNN graph and propose an innovative multi-link neighbor sampler. These choices, together with the 2T component, significantly reduce the complexity of the HGNN model. Empirical evaluations involving millions of users show significant improvement in the quality of personalized recommendations, resulting in a +46% increase in new audiobooks start rate and a +23% boost in streaming rates. Intriguingly, our model's impact extends beyond audiobooks, benefiting established products like podcasts.
△ Less
Submitted 8 March, 2024;
originally announced March 2024.
-
Smooth image-to-image translations with latent space interpolations
Authors:
Yahui Liu,
Enver Sangineto,
Yajing Chen,
Linchao Bao,
Haoxian Zhang,
Nicu Sebe,
Bruno Lepri,
Marco De Nadai
Abstract:
Multi-domain image-to-image (I2I) translations can transform a source image according to the style of a target domain. One important, desired characteristic of these transformations, is their graduality, which corresponds to a smooth change between the source and the target image when their respective latent-space representations are linearly interpolated. However, state-of-the-art methods usually…
▽ More
Multi-domain image-to-image (I2I) translations can transform a source image according to the style of a target domain. One important, desired characteristic of these transformations, is their graduality, which corresponds to a smooth change between the source and the target image when their respective latent-space representations are linearly interpolated. However, state-of-the-art methods usually perform poorly when evaluated using inter-domain interpolations, often producing abrupt changes in the appearance or non-realistic intermediate images. In this paper, we argue that one of the main reasons behind this problem is the lack of sufficient inter-domain training data and we propose two different regularization methods to alleviate this issue: a new shrinkage loss, which compacts the latent space, and a Mixup data-augmentation strategy, which flattens the style representations between domains. We also propose a new metric to quantitatively evaluate the degree of the interpolation smoothness, an aspect which is not sufficiently covered by the existing I2I translation metrics. Using both our proposed metric and standard evaluation protocols, we show that our regularization techniques can improve the state-of-the-art multi-domain I2I translations by a large margin. Our code will be made publicly available upon the acceptance of this article.
△ Less
Submitted 14 March, 2023; v1 submitted 3 October, 2022;
originally announced October 2022.
-
Spatial Entropy as an Inductive Bias for Vision Transformers
Authors:
Elia Peruzzo,
Enver Sangineto,
Yahui Liu,
Marco De Nadai,
Wei Bi,
Bruno Lepri,
Nicu Sebe
Abstract:
Recent work on Vision Transformers (VTs) showed that introducing a local inductive bias in the VT architecture helps reducing the number of samples necessary for training. However, the architecture modifications lead to a loss of generality of the Transformer backbone, partially contradicting the push towards the development of uniform architectures, shared, e.g., by both the Computer Vision and t…
▽ More
Recent work on Vision Transformers (VTs) showed that introducing a local inductive bias in the VT architecture helps reducing the number of samples necessary for training. However, the architecture modifications lead to a loss of generality of the Transformer backbone, partially contradicting the push towards the development of uniform architectures, shared, e.g., by both the Computer Vision and the Natural Language Processing areas. In this work, we propose a different and complementary direction, in which a local bias is introduced using an auxiliary self-supervised task, performed jointly with standard supervised training. Specifically, we exploit the observation that the attention maps of VTs, when trained with self-supervision, can contain a semantic segmentation structure which does not spontaneously emerge when training is supervised. Thus, we explicitly encourage the emergence of this spatial clustering as a form of training regularization. In more detail, we exploit the assumption that, in a given image, objects usually correspond to few connected regions, and we propose a spatial formulation of the information entropy to quantify this object-based inductive bias. By minimizing the proposed spatial entropy, we include an additional self-supervised signal during training. Using extensive experiments, we show that the proposed regularization leads to equivalent or better results than other VT proposals which include a local bias by changing the basic Transformer architecture, and it can drastically boost the VT final accuracy when using small-medium training sets. The code is available at https://github.com/helia95/SAR.
△ Less
Submitted 14 March, 2023; v1 submitted 9 June, 2022;
originally announced June 2022.
-
ISF-GAN: An Implicit Style Function for High-Resolution Image-to-Image Translation
Authors:
Yahui Liu,
Yajing Chen,
Linchao Bao,
Nicu Sebe,
Bruno Lepri,
Marco De Nadai
Abstract:
Recently, there has been an increasing interest in image editing methods that employ pre-trained unconditional image generators (e.g., StyleGAN). However, applying these methods to translate images to multiple visual domains remains challenging. Existing works do not often preserve the domain-invariant part of the image (e.g., the identity in human face translations), they do not usually handle mu…
▽ More
Recently, there has been an increasing interest in image editing methods that employ pre-trained unconditional image generators (e.g., StyleGAN). However, applying these methods to translate images to multiple visual domains remains challenging. Existing works do not often preserve the domain-invariant part of the image (e.g., the identity in human face translations), they do not usually handle multiple domains, or do not allow for multi-modal translations. This work proposes an implicit style function (ISF) to straightforwardly achieve multi-modal and multi-domain image-to-image translation from pre-trained unconditional generators. The ISF manipulates the semantics of an input latent code to make the image generated from it lying in the desired visual domain. Our results in human face and animal manipulations show significantly improved results over the baselines. Our model enables cost-effective multi-modal unsupervised image-to-image translations at high resolution using pre-trained unconditional GANs. The code and data are available at: \url{https://github.com/yhlleo/stylegan-mmuit}.
△ Less
Submitted 23 February, 2022; v1 submitted 26 September, 2021;
originally announced September 2021.
-
Click to Move: Controlling Video Generation with Sparse Motion
Authors:
Pierfrancesco Ardino,
Marco De Nadai,
Bruno Lepri,
Elisa Ricci,
Stéphane Lathuilière
Abstract:
This paper introduces Click to Move (C2M), a novel framework for video generation where the user can control the motion of the synthesized video through mouse clicks specifying simple object trajectories of the key objects in the scene. Our model receives as input an initial frame, its corresponding segmentation map and the sparse motion vectors encoding the input provided by the user. It outputs…
▽ More
This paper introduces Click to Move (C2M), a novel framework for video generation where the user can control the motion of the synthesized video through mouse clicks specifying simple object trajectories of the key objects in the scene. Our model receives as input an initial frame, its corresponding segmentation map and the sparse motion vectors encoding the input provided by the user. It outputs a plausible video sequence starting from the given frame and with a motion that is consistent with user input. Notably, our proposed deep architecture incorporates a Graph Convolution Network (GCN) modelling the movements of all the objects in the scene in a holistic manner and effectively combining the sparse user motion information and image features. Experimental results show that C2M outperforms existing methods on two publicly available datasets, thus demonstrating the effectiveness of our GCN framework at modelling object interactions. The source code is publicly available at https://github.com/PierfrancescoArdino/C2M.
△ Less
Submitted 19 August, 2021;
originally announced August 2021.
-
Living in a pandemic: adaptation of individual mobility and social activity in the US
Authors:
Lorenzo Lucchini,
Simone Centellegher,
Luca Pappalardo,
Riccardo Gallotti,
Filippo Privitera,
Bruno Lepri,
Marco De Nadai
Abstract:
The non-pharmaceutical interventions (NPIs), aimed at reducing the diffusion of the COVID-19 pandemic, has dramatically influenced our behaviour in everyday life. In this work, we study how individuals adapted their daily movements and person-to-person contact patterns over time in response to the COVID-19 pandemic and the NPIs. We leverage longitudinal GPS mobility data of hundreds of thousands o…
▽ More
The non-pharmaceutical interventions (NPIs), aimed at reducing the diffusion of the COVID-19 pandemic, has dramatically influenced our behaviour in everyday life. In this work, we study how individuals adapted their daily movements and person-to-person contact patterns over time in response to the COVID-19 pandemic and the NPIs. We leverage longitudinal GPS mobility data of hundreds of thousands of anonymous individuals in four US states and empirically show the dramatic disruption in people's life. We find that local interventions did not just impact the number of visits to different venues but also how people experience them. Individuals spend less time in venues, preferring simpler and more predictable routines and reducing person-to-person contact activities. Moreover, we show that the stringency of interventions alone does explain the number and duration of visits to venues: individual patterns of visits seem to be influenced by the local severity of the pandemic and a risk adaptation factor, which increases the people's mobility regardless of the stringency of interventions.
△ Less
Submitted 17 August, 2021; v1 submitted 26 July, 2021;
originally announced July 2021.
-
Smoothing the Disentangled Latent Style Space for Unsupervised Image-to-Image Translation
Authors:
Yahui Liu,
Enver Sangineto,
Yajing Chen,
Linchao Bao,
Haoxian Zhang,
Nicu Sebe,
Bruno Lepri,
Wei Wang,
Marco De Nadai
Abstract:
Image-to-Image (I2I) multi-domain translation models are usually evaluated also using the quality of their semantic interpolation results. However, state-of-the-art models frequently show abrupt changes in the image appearance during interpolation, and usually perform poorly in interpolations across domains. In this paper, we propose a new training protocol based on three specific losses which hel…
▽ More
Image-to-Image (I2I) multi-domain translation models are usually evaluated also using the quality of their semantic interpolation results. However, state-of-the-art models frequently show abrupt changes in the image appearance during interpolation, and usually perform poorly in interpolations across domains. In this paper, we propose a new training protocol based on three specific losses which help a translation network to learn a smooth and disentangled latent style space in which: 1) Both intra- and inter-domain interpolations correspond to gradual changes in the generated images and 2) The content of the source image is better preserved during the translation. Moreover, we propose a novel evaluation metric to properly measure the smoothness of latent style space of I2I translation models. The proposed method can be plugged into existing translation approaches, and our extensive experiments on different datasets show that it can significantly boost the quality of the generated images and the graduality of the interpolations.
△ Less
Submitted 16 June, 2021;
originally announced June 2021.
-
Efficient Training of Visual Transformers with Small Datasets
Authors:
Yahui Liu,
Enver Sangineto,
Wei Bi,
Nicu Sebe,
Bruno Lepri,
Marco De Nadai
Abstract:
Visual Transformers (VTs) are emerging as an architectural paradigm alternative to Convolutional networks (CNNs). Differently from CNNs, VTs can capture global relations between image elements and they potentially have a larger representation capacity. However, the lack of the typical convolutional inductive bias makes these models more data-hungry than common CNNs. In fact, some local properties…
▽ More
Visual Transformers (VTs) are emerging as an architectural paradigm alternative to Convolutional networks (CNNs). Differently from CNNs, VTs can capture global relations between image elements and they potentially have a larger representation capacity. However, the lack of the typical convolutional inductive bias makes these models more data-hungry than common CNNs. In fact, some local properties of the visual domain which are embedded in the CNN architectural design, in VTs should be learned from samples. In this paper, we empirically analyse different VTs, comparing their robustness in a small training-set regime, and we show that, despite having a comparable accuracy when trained on ImageNet, their performance on smaller datasets can be largely different. Moreover, we propose a self-supervised task which can extract additional information from images with only a negligible computational overhead. This task encourages the VTs to learn spatial relations within an image and makes the VT training much more robust when training data are scarce. Our task is used jointly with the standard (supervised) training and it does not depend on specific architectural choices, thus it can be easily plugged in the existing VTs. Using an extensive evaluation with different VTs and datasets, we show that our method can improve (sometimes dramatically) the final accuracy of the VTs. Our code is available at: https://github.com/yhlleo/VTs-Drloc.
△ Less
Submitted 14 November, 2021; v1 submitted 7 June, 2021;
originally announced June 2021.
-
Semantic-Guided Inpainting Network for Complex Urban Scenes Manipulation
Authors:
Pierfrancesco Ardino,
Yahui Liu,
Elisa Ricci,
Bruno Lepri,
Marco De Nadai
Abstract:
Manipulating images of complex scenes to reconstruct, insert and/or remove specific object instances is a challenging task. Complex scenes contain multiple semantics and objects, which are frequently cluttered or ambiguous, thus hampering the performance of inpainting models. Conventional techniques often rely on structural information such as object contours in multi-stage approaches that generat…
▽ More
Manipulating images of complex scenes to reconstruct, insert and/or remove specific object instances is a challenging task. Complex scenes contain multiple semantics and objects, which are frequently cluttered or ambiguous, thus hampering the performance of inpainting models. Conventional techniques often rely on structural information such as object contours in multi-stage approaches that generate unreliable results and boundaries. In this work, we propose a novel deep learning model to alter a complex urban scene by removing a user-specified portion of the image and coherently inserting a new object (e.g. car or pedestrian) in that scene. Inspired by recent works on image inpainting, our proposed method leverages the semantic segmentation to model the content and structure of the image, and learn the best shape and location of the object to insert. To generate reliable results, we design a new decoder block that combines the semantic segmentation and generation task to guide better the generation of new objects and scenes, which have to be semantically consistent with the image. Our experiments, conducted on two large-scale datasets of urban scenes (Cityscapes and Indian Driving), show that our proposed approach successfully address the problem of semantically-guided inpainting of complex urban scene.
△ Less
Submitted 19 October, 2020;
originally announced October 2020.
-
Retrieval Guided Unsupervised Multi-domain Image-to-Image Translation
Authors:
Raul Gomez,
Yahui Liu,
Marco De Nadai,
Dimosthenis Karatzas,
Bruno Lepri,
Nicu Sebe
Abstract:
Image to image translation aims to learn a mapping that transforms an image from one visual domain to another. Recent works assume that images descriptors can be disentangled into a domain-invariant content representation and a domain-specific style representation. Thus, translation models seek to preserve the content of source images while changing the style to a target visual domain. However, sy…
▽ More
Image to image translation aims to learn a mapping that transforms an image from one visual domain to another. Recent works assume that images descriptors can be disentangled into a domain-invariant content representation and a domain-specific style representation. Thus, translation models seek to preserve the content of source images while changing the style to a target visual domain. However, synthesizing new images is extremely challenging especially in multi-domain translations, as the network has to compose content and style to generate reliable and diverse images in multiple domains. In this paper we propose the use of an image retrieval system to assist the image-to-image translation task. First, we train an image-to-image translation model to map images to multiple domains. Then, we train an image retrieval model using real and generated images to find images similar to a query one in content but in a different domain. Finally, we exploit the image retrieval system to fine-tune the image-to-image translation model and generate higher quality images. Our experiments show the effectiveness of the proposed solution and highlight the contribution of the retrieval network, which can benefit from additional unlabeled data and help image-to-image translation models in the presence of scarce data.
△ Less
Submitted 11 August, 2020;
originally announced August 2020.
-
Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach
Authors:
Yahui Liu,
Marco De Nadai,
Deng Cai,
Huayang Li,
Xavier Alameda-Pineda,
Nicu Sebe,
Bruno Lepri
Abstract:
Manipulating visual attributes of images through human-written text is a very challenging task. On the one hand, models have to learn the manipulation without the ground truth of the desired output. On the other hand, models have to deal with the inherent ambiguity of natural language. Previous research usually requires either the user to describe all the characteristics of the desired image or to…
▽ More
Manipulating visual attributes of images through human-written text is a very challenging task. On the one hand, models have to learn the manipulation without the ground truth of the desired output. On the other hand, models have to deal with the inherent ambiguity of natural language. Previous research usually requires either the user to describe all the characteristics of the desired image or to use richly-annotated image captioning datasets. In this work, we propose a novel unsupervised approach, based on image-to-image translation, that alters the attributes of a given image through a command-like sentence such as "change the hair color to black". Contrarily to state-of-the-art approaches, our model does not require a human-annotated dataset nor a textual description of all the attributes of the desired image, but only those that have to be modified. Our proposed model disentangles the image content from the visual attributes, and it learns to modify the latter using the textual description, before generating a new image from the content and the modified attribute representation. Because text might be inherently ambiguous (blond hair may refer to different shadows of blond, e.g. golden, icy, sandy), our method generates multiple stochastic versions of the same translation. Experiments show that the proposed model achieves promising performances on two large-scale public datasets: CelebA and CUB. We believe our approach will pave the way to new avenues of research combining textual and speech commands with visual attributes.
△ Less
Submitted 10 August, 2020;
originally announced August 2020.
-
Socio-economic, built environment, and mobility conditions associated with crime: A study of multiple cities
Authors:
Marco De Nadai,
Yanyan Xu,
Emmanuel Letouzé,
Marta C. González,
Bruno Lepri
Abstract:
Nowadays, 23% of the world population lives in multi-million cities. In these metropolises, criminal activity is much higher and violent than in either small cities or rural areas. Thus, understanding what factors influence urban crime in big cities is a pressing need. Mainstream studies analyse crime records through historical panel data or analysis of historical patterns combined with ecological…
▽ More
Nowadays, 23% of the world population lives in multi-million cities. In these metropolises, criminal activity is much higher and violent than in either small cities or rural areas. Thus, understanding what factors influence urban crime in big cities is a pressing need. Mainstream studies analyse crime records through historical panel data or analysis of historical patterns combined with ecological factor and exploratory mapping. More recently, machine learning methods have provided informed crime prediction over time. However, previous studies have focused on a single city at a time, considering only a limited number of factors (such as socio-economical characteristics) and often at large spatial units. Hence, our understanding of the factors influencing crime across cultures and cities is very limited. Here we propose a Bayesian model to explore how crime is related not only to socio-economic factors but also to the built environmental (e.g. land use) and mobility characteristics of neighbourhoods. To that end, we integrate multiple open data sources with mobile phone traces and compare how the different factors correlate with crime in diverse cities, namely Boston, Bogotá, Los Angeles and Chicago. We find that the combined use of socio-economic conditions, mobility information and physical characteristics of the neighbourhood effectively explain the emergence of crime, and improve the performance of the traditional approaches. However, we show that the socio-ecological factors of neighbourhoods relate to crime very differently from one city to another. Thus there is clearly no "one fits all" model.
△ Less
Submitted 13 April, 2020;
originally announced April 2020.
-
Mobile phone data and COVID-19: Missing an opportunity?
Authors:
Nuria Oliver,
Emmanuel Letouzé,
Harald Sterly,
Sébastien Delataille,
Marco De Nadai,
Bruno Lepri,
Renaud Lambiotte,
Richard Benjamins,
Ciro Cattuto,
Vittoria Colizza,
Nicolas de Cordes,
Samuel P. Fraiberger,
Till Koebe,
Sune Lehmann,
Juan Murillo,
Alex Pentland,
Phuong N Pham,
Frédéric Pivetta,
Albert Ali Salah,
Jari Saramäki,
Samuel V. Scarpino,
Michele Tizzoni,
Stefaan Verhulst,
Patrick Vinck
Abstract:
This paper describes how mobile phone data can guide government and public health authorities in determining the best course of action to control the COVID-19 pandemic and in assessing the effectiveness of control measures such as physical distancing. It identifies key gaps and reasons why this kind of data is only scarcely used, although their value in similar epidemics has proven in a number of…
▽ More
This paper describes how mobile phone data can guide government and public health authorities in determining the best course of action to control the COVID-19 pandemic and in assessing the effectiveness of control measures such as physical distancing. It identifies key gaps and reasons why this kind of data is only scarcely used, although their value in similar epidemics has proven in a number of use cases. It presents ways to overcome these gaps and key recommendations for urgent action, most notably the establishment of mixed expert groups on national and regional level, and the inclusion and support of governments and public authorities early on. It is authored by a group of experienced data scientists, epidemiologists, demographers and representatives of mobile network operators who jointly put their work at the service of the global effort to combat the COVID-19 pandemic.
△ Less
Submitted 27 March, 2020;
originally announced March 2020.
-
GMM-UNIT: Unsupervised Multi-Domain and Multi-Modal Image-to-Image Translation via Attribute Gaussian Mixture Modeling
Authors:
Yahui Liu,
Marco De Nadai,
Jian Yao,
Nicu Sebe,
Bruno Lepri,
Xavier Alameda-Pineda
Abstract:
Unsupervised image-to-image translation (UNIT) aims at learning a mapping between several visual domains by using unpaired training images. Recent studies have shown remarkable success for multiple domains but they suffer from two main limitations: they are either built from several two-domain mappings that are required to be learned independently, or they generate low-diversity results, a problem…
▽ More
Unsupervised image-to-image translation (UNIT) aims at learning a mapping between several visual domains by using unpaired training images. Recent studies have shown remarkable success for multiple domains but they suffer from two main limitations: they are either built from several two-domain mappings that are required to be learned independently, or they generate low-diversity results, a problem known as mode collapse. To overcome these limitations, we propose a method named GMM-UNIT, which is based on a content-attribute disentangled representation where the attribute space is fitted with a GMM. Each GMM component represents a domain, and this simple assumption has two prominent advantages. First, it can be easily extended to most multi-domain and multi-modal image-to-image translation tasks. Second, the continuous domain encoding allows for interpolation between domains and for extrapolation to unseen domains and translations. Additionally, we show how GMM-UNIT can be constrained down to different methods in the literature, meaning that GMM-UNIT is a unifying framework for unsupervised image-to-image translation.
△ Less
Submitted 21 March, 2020; v1 submitted 15 March, 2020;
originally announced March 2020.
-
Understanding individual behaviour: from virtual to physical patterns
Authors:
Marco De Nadai,
Bruno Lepri,
Nuria Oliver
Abstract:
As "Big Data" has become pervasive, an increasing amount of research has connected the dots between human behaviour in the offline and online worlds. Consequently, researchers have exploited these new findings to create models that better predict different aspects of human life and recommend future behaviour. To date, however, we do not yet fully understand the similarities and differences of huma…
▽ More
As "Big Data" has become pervasive, an increasing amount of research has connected the dots between human behaviour in the offline and online worlds. Consequently, researchers have exploited these new findings to create models that better predict different aspects of human life and recommend future behaviour. To date, however, we do not yet fully understand the similarities and differences of human behaviour in these virtual and physical worlds. Here, we analyse and discuss the mobility and application usage of 400,000 individuals over eight months. We find an astonishing similarity between people's mobility in the physical space and how they move from app to app in smartphones. Our data shows that individuals use and visit a finite number of apps and places, but they keep exploring over time. In particular, two distinct profiles of individuals emerge: those that keep changing places and services, and those that are stable over time, named as "explorers" and "keepers". We see these findings as crucial to enrich a discussion for the potentials and the challenges of building human-centric AI systems, which might leverage recent results in Computational Social Science.
△ Less
Submitted 13 February, 2020;
originally announced February 2020.
-
Gesture-to-Gesture Translation in the Wild via Category-Independent Conditional Maps
Authors:
Yahui Liu,
Marco De Nadai,
Gloria Zen,
Nicu Sebe,
Bruno Lepri
Abstract:
Recent works have shown Generative Adversarial Networks (GANs) to be particularly effective in image-to-image translations. However, in tasks such as body pose and hand gesture translation, existing methods usually require precise annotations, e.g. key-points or skeletons, which are time-consuming to draw. In this work, we propose a novel GAN architecture that decouples the required annotations in…
▽ More
Recent works have shown Generative Adversarial Networks (GANs) to be particularly effective in image-to-image translations. However, in tasks such as body pose and hand gesture translation, existing methods usually require precise annotations, e.g. key-points or skeletons, which are time-consuming to draw. In this work, we propose a novel GAN architecture that decouples the required annotations into a category label - that specifies the gesture type - and a simple-to-draw category-independent conditional map - that expresses the location, rotation and size of the hand gesture. Our architecture synthesizes the target gesture while preserving the background context, thus effectively dealing with gesture translation in the wild. To this aim, we use an attention module and a rolling guidance approach, which loops the generated images back into the network and produces higher quality images compared to competing works. Thus, our GAN learns to generate new images from simple annotations without requiring key-points or skeleton labels. Results on two public datasets show that our method outperforms state of the art approaches both quantitatively and qualitatively. To the best of our knowledge, no work so far has addressed the gesture-to-gesture translation in the wild by requiring user-friendly annotations.
△ Less
Submitted 31 July, 2019; v1 submitted 12 July, 2019;
originally announced July 2019.
-
Apps, Places and People: strategies, limitations and trade-offs in the physical and digital worlds
Authors:
Marco De Nadai,
Angelo Cardoso,
Antonio Lima,
Bruno Lepri,
Nuria Oliver
Abstract:
Cognition has been found to constrain several aspects of human behaviour, such as the number of friends and the number of favourite places a person keeps stable over time. This limitation has been empirically defined in the physical and social spaces. But do people exhibit similar constraints in the digital space? We address this question through the analysis of pseudonymised mobility and mobile a…
▽ More
Cognition has been found to constrain several aspects of human behaviour, such as the number of friends and the number of favourite places a person keeps stable over time. This limitation has been empirically defined in the physical and social spaces. But do people exhibit similar constraints in the digital space? We address this question through the analysis of pseudonymised mobility and mobile application (app) usage data of 400,000 individuals in a European country for six months. Despite the enormous heterogeneity of apps usage, we find that individuals exhibit a conserved capacity that limits the number of applications they regularly use. Moreover, we find that this capacity steadily decreases with age, as does the capacity in the physical space but with more complex dynamics. Even though people might have the same capacity, applications get added and removed over time. In this respect, we identify two profiles of individuals: app keepers and explorers, which differ in their stable (keepers) vs exploratory (explorers) behaviour regarding their use of mobile applications. Finally, we show that the capacity of applications predicts mobility capacity and vice-versa. By contrast, the behaviour of keepers and explorers may considerably vary across the two domains. Our empirical findings provide an intriguing picture linking human behaviour in the physical and digital worlds which bridges research studies from Computer Science, Social Physics and Computational Social Sciences.
△ Less
Submitted 19 April, 2019;
originally announced April 2019.
-
The economic value of neighborhoods: Predicting real estate prices from the urban environment
Authors:
Marco De Nadai,
Bruno Lepri
Abstract:
Housing costs have a significant impact on individuals, families, businesses, and governments. Recently, online companies such as Zillow have developed proprietary systems that provide automated estimates of housing prices without the immediate need of professional appraisers. Yet, our understanding of what drives the value of houses is very limited. In this paper, we use multiple sources of data…
▽ More
Housing costs have a significant impact on individuals, families, businesses, and governments. Recently, online companies such as Zillow have developed proprietary systems that provide automated estimates of housing prices without the immediate need of professional appraisers. Yet, our understanding of what drives the value of houses is very limited. In this paper, we use multiple sources of data to entangle the economic contribution of the neighborhood's characteristics such as walkability and security perception. We also develop and release a framework able to now-cast housing prices from Open data, without the need for historical transactions. Experiments involving 70,000 houses in 8 Italian cities highlight that the neighborhood's vitality and walkability seem to drive more than 20% of the housing value. Moreover, the use of this information improves the nowcast by 60%. Hence, the use of property's surroundings' characteristics can be an invaluable resource to appraise the economic and social value of houses after neighborhood changes and, potentially, anticipate gentrification.
△ Less
Submitted 7 August, 2018;
originally announced August 2018.
-
Is social capital associated with synchronization in human communication? An analysis of Italian call records and measures of civic engagement
Authors:
Marco Mamei,
Francesca Pancotto,
Marco De Nadai,
Bruno Lepri,
Michele Vescovi,
Franco Zambonelli,
Alex Pentland
Abstract:
Social capital has been studied in economics, sociology and political science as one of the key elements that promote the development of modern societies. It can be defined as the source of capital that facilitates cooperation through shared social norms. In this work, we investigate whether and to what extent synchronization aspects of mobile communication patterns are associated with social capi…
▽ More
Social capital has been studied in economics, sociology and political science as one of the key elements that promote the development of modern societies. It can be defined as the source of capital that facilitates cooperation through shared social norms. In this work, we investigate whether and to what extent synchronization aspects of mobile communication patterns are associated with social capital metrics. Interestingly, our results show that our synchronization-based approach well correlates with existing social capital metrics (i.e., Referendum turnout, Blood donations, and Association density), being also able to characterize the different role played by high synchronization within a close proximity-based community and high synchronization among different communities. Hence, the proposed approach can provide timely, effective analysis at a limited cost over a large territory.
△ Less
Submitted 6 July, 2018; v1 submitted 18 June, 2018;
originally announced June 2018.
-
Are Safer Looking Neighborhoods More Lively? A Multimodal Investigation into Urban Life
Authors:
Marco De Nadai,
Radu L. Vieriu,
Gloria Zen,
Stefan Dragicevic,
Nikhil Naik,
Michele Caraviello,
Cesar A. Hidalgo,
Nicu Sebe,
Bruno Lepri
Abstract:
Policy makers, urban planners, architects, sociologists, and economists are interested in creating urban areas that are both lively and safe. But are the safety and liveliness of neighborhoods independent characteristics? Or are they just two sides of the same coin? In a world where people avoid unsafe looking places, neighborhoods that look unsafe will be less lively, and will fail to harness the…
▽ More
Policy makers, urban planners, architects, sociologists, and economists are interested in creating urban areas that are both lively and safe. But are the safety and liveliness of neighborhoods independent characteristics? Or are they just two sides of the same coin? In a world where people avoid unsafe looking places, neighborhoods that look unsafe will be less lively, and will fail to harness the natural surveillance of human activity. But in a world where the preference for safe looking neighborhoods is small, the connection between the perception of safety and liveliness will be either weak or nonexistent. In this paper we explore the connection between the levels of activity and the perception of safety of neighborhoods in two major Italian cities by combining mobile phone data (as a proxy for activity or liveliness) with scores of perceived safety estimated using a Convolutional Neural Network trained on a dataset of Google Street View images scored using a crowdsourced visual perception survey. We find that: (i) safer looking neighborhoods are more active than what is expected from their population density, employee density, and distance to the city centre; and (ii) that the correlation between appearance of safety and activity is positive, strong, and significant, for females and people over 50, but negative for people under 30, suggesting that the behavioral impact of perception depends on the demographic of the population. Finally, we use occlusion techniques to identify the urban features that contribute to the appearance of safety, finding that greenery and street facing windows contribute to a positive appearance of safety (in agreement with Oscar Newman's defensible space theory). These results suggest that urban appearance modulates levels of human activity and, consequently, a neighborhood's rate of natural surveillance.
△ Less
Submitted 1 August, 2016;
originally announced August 2016.
-
The Death and Life of Great Italian Cities: A Mobile Phone Data Perspective
Authors:
Marco De Nadai,
Jacopo Staiano,
Roberto Larcher,
Nicu Sebe,
Daniele Quercia,
Bruno Lepri
Abstract:
The Death and Life of Great American Cities was written in 1961 and is now one of the most influential book in city planning. In it, Jane Jacobs proposed four conditions that promote life in a city. However, these conditions have not been empirically tested until recently. This is mainly because it is hard to collect data about "city life". The city of Seoul recently collected pedestrian activity…
▽ More
The Death and Life of Great American Cities was written in 1961 and is now one of the most influential book in city planning. In it, Jane Jacobs proposed four conditions that promote life in a city. However, these conditions have not been empirically tested until recently. This is mainly because it is hard to collect data about "city life". The city of Seoul recently collected pedestrian activity through surveys at an unprecedented scale, with an effort spanning more than a decade, allowing researchers to conduct the first study successfully testing Jacobs's conditions. In this paper, we identify a valuable alternative to the lengthy and costly collection of activity survey data: mobile phone data. We extract human activity from such data, collect land use and socio-demographic information from the Italian Census and Open Street Map, and test the four conditions in six Italian cities. Although these cities are very different from the places for which Jacobs's conditions were spelled out (i.e., great American cities) and from the places in which they were recently tested (i.e., the Asian city of Seoul), we find those conditions to be indeed associated with urban life in Italy as well. Our methodology promises to have a great impact on urban studies, not least because, if replicated, it will make it possible to test Jacobs's theories at scale.
△ Less
Submitted 13 March, 2016;
originally announced March 2016.
