-
Plots Unlock Time-Series Understanding in Multimodal Models
Authors:
Mayank Daswani,
Mathias M. J. Bellaiche,
Marc Wilson,
Desislav Ivanov,
Mikhail Papkov,
Eva Schnider,
Jing Tang,
Kay Lamerigts,
Gabriela Botea,
Michael A. Sanchez,
Yojan Patel,
Shruthi Prabhakara,
Shravya Shetty,
Umesh Telang
Abstract:
While multimodal foundation models can now natively work with data beyond text, they remain underutilized in analyzing the considerable amounts of multi-dimensional time-series data in fields like healthcare, finance, and social sciences, representing a missed opportunity for richer, data-driven insights. This paper proposes a simple but effective method that leverages the existing vision encoders…
▽ More
While multimodal foundation models can now natively work with data beyond text, they remain underutilized in analyzing the considerable amounts of multi-dimensional time-series data in fields like healthcare, finance, and social sciences, representing a missed opportunity for richer, data-driven insights. This paper proposes a simple but effective method that leverages the existing vision encoders of these models to "see" time-series data via plots, avoiding the need for additional, potentially costly, model training. Our empirical evaluations show that this approach outperforms providing the raw time-series data as text, with the additional benefit that visual time-series representations demonstrate up to a 90% reduction in model API costs. We validate our hypothesis through synthetic data tasks of increasing complexity, progressing from simple functional form identification on clean data, to extracting trends from noisy scatter plots. To demonstrate generalizability from synthetic tasks with clear reasoning steps to more complex, real-world scenarios, we apply our approach to consumer health tasks - specifically fall detection, activity recognition, and readiness assessment - which involve heterogeneous, noisy data and multi-step reasoning. The overall success in plot performance over text performance (up to an 120% performance increase on zero-shot synthetic tasks, and up to 150% performance increase on real-world tasks), across both GPT and Gemini model families, highlights our approach's potential for making the best use of the native capabilities of foundation models.
△ Less
Submitted 3 October, 2024;
originally announced October 2024.
-
Predicting Cardiovascular Disease Risk using Photoplethysmography and Deep Learning
Authors:
Wei-Hung Weng,
Sebastien Baur,
Mayank Daswani,
Christina Chen,
Lauren Harrell,
Sujay Kakarmath,
Mariam Jabara,
Babak Behsaz,
Cory Y. McLean,
Yossi Matias,
Greg S. Corrado,
Shravya Shetty,
Shruthi Prabhakara,
Yun Liu,
Goodarz Danaei,
Diego Ardila
Abstract:
Cardiovascular diseases (CVDs) are responsible for a large proportion of premature deaths in low- and middle-income countries. Early CVD detection and intervention is critical in these populations, yet many existing CVD risk scores require a physical examination or lab measurements, which can be challenging in such health systems due to limited accessibility. Here we investigated the potential to…
▽ More
Cardiovascular diseases (CVDs) are responsible for a large proportion of premature deaths in low- and middle-income countries. Early CVD detection and intervention is critical in these populations, yet many existing CVD risk scores require a physical examination or lab measurements, which can be challenging in such health systems due to limited accessibility. Here we investigated the potential to use photoplethysmography (PPG), a sensing technology available on most smartphones that can potentially enable large-scale screening at low cost, for CVD risk prediction. We developed a deep learning PPG-based CVD risk score (DLS) to predict the probability of having major adverse cardiovascular events (MACE: non-fatal myocardial infarction, stroke, and cardiovascular death) within ten years, given only age, sex, smoking status and PPG as predictors. We compared the DLS with the office-based refit-WHO score, which adopts the shared predictors from WHO and Globorisk scores (age, sex, smoking status, height, weight and systolic blood pressure) but refitted on the UK Biobank (UKB) cohort. In UKB cohort, DLS's C-statistic (71.1%, 95% CI 69.9-72.4) was non-inferior to office-based refit-WHO score (70.9%, 95% CI 69.7-72.2; non-inferiority margin of 2.5%, p<0.01). The calibration of the DLS was satisfactory, with a 1.8% mean absolute calibration error. Adding DLS features to the office-based score increased the C-statistic by 1.0% (95% CI 0.6-1.4). DLS predicts ten-year MACE risk comparable with the office-based refit-WHO score. It provides a proof-of-concept and suggests the potential of a PPG-based approach strategies for community-based primary prevention in resource-limited regions.
△ Less
Submitted 9 May, 2023;
originally announced May 2023.
-
DEWPython: A Python Implementation of the Deep Earth Water Model and Application to Ocean Worlds
Authors:
Andrew Chan,
Mohit Melwani Daswani,
Steven Vance
Abstract:
There are two main methods of calculating the thermodynamic properties of water and solutes: mass action (including the Helgeson-Kirkham-Flowers (HKF) equations of state and model) and Gibbs free energy minimization (e.g. Leal et al., 2016). However, in certain regions of pressure and temperature the HKF model inaccurately predicts the speciation and concentration of solutes (e.g. Miron et al., 20…
▽ More
There are two main methods of calculating the thermodynamic properties of water and solutes: mass action (including the Helgeson-Kirkham-Flowers (HKF) equations of state and model) and Gibbs free energy minimization (e.g. Leal et al., 2016). However, in certain regions of pressure and temperature the HKF model inaccurately predicts the speciation and concentration of solutes (e.g. Miron et al., 2019). The Deep Earth Water (DEW) model uses a series of HKF-type equations to calculate the properties of water and solute concentrations at high temperatures (373 - 1473 K) and pressures (0.1 - 6 GPa) (e.g. Huang and Sverjensky, 2019; Pan et al., 2013; Sverjensky et al., 2014). The DEW model is synthesized in an Excel spreadsheet and calculates Gibbs energies of formation, equilibrium constants, and standard volume changes for reactions. Here we present an object-oriented Python implementation of the DEW model, called DEWPython. Our model expands on DEW by increasing model efficiency, streamlining the input process, and incorporating SUPCRT in-line. Additionally, our model builds in minerals and aqueous species from the thermodynamic database slop16.dat (Boyer, 2019) which would normally be calculated separately. We also present a set of reactions relevant to icy ocean world interiors calculated with the DEWPython. The favorability of these reactions indicates likely formation of certain organic species under extreme pressures relevant to ocean worlds.
△ Less
Submitted 28 May, 2021;
originally announced May 2021.
-
White Paper Towards a Fuller Understanding of Icy Satellite Seafloors, Interiors, and Habitability
Authors:
Paul K. Byrne,
Andrew J. Dombard,
Catherine M. Elder,
Steven A. Hauck, II,
Mohit Melwani Daswani,
Paul V. Regensburger,
Steven D. Vance
Abstract:
Icy satellites represent compelling astrobiological targets, but their rocky interiors must be better characterized. Fundamental research programs and thematic workshops promoting ocean world interdisciplinarity are key. Future missions to icy satellites should explicitly include objectives to characterize interfaces between rock and water or high-pressure ices.
Icy satellites represent compelling astrobiological targets, but their rocky interiors must be better characterized. Fundamental research programs and thematic workshops promoting ocean world interdisciplinarity are key. Future missions to icy satellites should explicitly include objectives to characterize interfaces between rock and water or high-pressure ices.
△ Less
Submitted 19 July, 2020;
originally announced July 2020.
-
Geochemistry constrains global hydrology on Early Mars
Authors:
Edwin S. Kite,
Mohit Melwani Daswani
Abstract:
Ancient hydrology is recorded by sedimentary rocks on Mars. The most voluminous sedimentary rocks that formed during Mars' Hesperian period are sulfate-rich rocks, explored by the $Opportunity$ rover from 2004-2012 and soon to be investigated by the $Curiosity$ rover at Gale crater. A leading hypothesis for the origin of these sulfates is that the cations were derived from evaporation of deep-sour…
▽ More
Ancient hydrology is recorded by sedimentary rocks on Mars. The most voluminous sedimentary rocks that formed during Mars' Hesperian period are sulfate-rich rocks, explored by the $Opportunity$ rover from 2004-2012 and soon to be investigated by the $Curiosity$ rover at Gale crater. A leading hypothesis for the origin of these sulfates is that the cations were derived from evaporation of deep-sourced groundwater, as part of a global circulation of groundwater. Global groundwater circulation would imply sustained warm Earthlike conditions on Early Mars. Global circulation of groundwater including infiltration of water initially in equilibrium with Mars' CO$_2$ atmosphere implies subsurface formation of carbonate. We find that the CO$_2$ sequestration implied by the global groundwater hypothesis for the origin of sulfate-rich rocks on Mars is 30-5000 bars if the $Opportunity$ data are representative of Hesperian sulfate-rich rocks, which is so large that (even accounting for volcanic outgassing) it would bury the atmosphere. This disfavors the hypothesis that the cations for Mars' Hesperian sulfates were derived from upwelling of deep sourced groundwater. If, instead, Hesperian sulfate-rich rocks are approximated as pure Mg-sulfate (no Fe), then the CO$_2$ sequestration is 0.3-400 bars. The low end of this range is consistent with the hypothesis that the cations for Mars' Hesperian sulfates were derived from upwelling of deep sourced groundwater. In both cases, carbon sequestration by global groundwater circulation actively works to terminate surface habitability, rather than being a passive marker of warm Earthlike conditions. $Curiosity$ will soon be in a position to discriminate between these two hypotheses. Our work links Mars sulfate cation composition, carbon isotopes, and climate change.
△ Less
Submitted 18 July, 2019;
originally announced July 2019.
-
Paleohydrology on Mars constrained by mass balance and mineralogy of pre-Amazonian sodium chloride lakes
Authors:
Mohit Melwani Daswani,
Edwin S. Kite
Abstract:
Chloride-bearing deposits on Mars record high-elevation lakes during the waning stages of Mars' wet era (mid-Noachian to late Hesperian). The water source pathways, seasonality, salinity, depth, lifetime, and paleoclimatic drivers of these widespread lakes are all unknown. Here we combine reaction-transport modeling, orbital spectroscopy, and new volume estimates from high-resolution digital terra…
▽ More
Chloride-bearing deposits on Mars record high-elevation lakes during the waning stages of Mars' wet era (mid-Noachian to late Hesperian). The water source pathways, seasonality, salinity, depth, lifetime, and paleoclimatic drivers of these widespread lakes are all unknown. Here we combine reaction-transport modeling, orbital spectroscopy, and new volume estimates from high-resolution digital terrain models, in order to constrain the hydrologic boundary conditions for forming the chlorides. Considering a T = 0 degrees C system, we find: (1) individual lakes were >100 m deep and lasted decades or longer; (2) if volcanic degassing was the source of chlorine, then the water-to-rock ratio or the total water volume were probably low, consistent with brief excursions above the melting point and/or arid climate; (3) if the chlorine source was igneous chlorapatite, then Cl-leaching events would require a (cumulative) time of >10 yr at the melting point; (4) Cl masses, divided by catchment area, give column densities 0.1 - 50 kg Cl/m^2, and these column densities bracket the expected chlorapatite-Cl content for a seasonally-warm active layer. Deep groundwater was not required. Taken together, our results are consistent with Mars having a usually cold, horizontally segregated hydrosphere by the time chlorides formed.
△ Less
Submitted 27 September, 2017;
originally announced September 2017.
-
Self-Modification of Policy and Utility Function in Rational Agents
Authors:
Tom Everitt,
Daniel Filan,
Mayank Daswani,
Marcus Hutter
Abstract:
Any agent that is part of the environment it interacts with and has versatile actuators (such as arms and fingers), will in principle have the ability to self-modify -- for example by changing its own source code. As we continue to create more and more intelligent agents, chances increase that they will learn about this ability. The question is: will they want to use it? For example, highly intell…
▽ More
Any agent that is part of the environment it interacts with and has versatile actuators (such as arms and fingers), will in principle have the ability to self-modify -- for example by changing its own source code. As we continue to create more and more intelligent agents, chances increase that they will learn about this ability. The question is: will they want to use it? For example, highly intelligent systems may find ways to change their goals to something more easily achievable, thereby `escaping' the control of their designers. In an important paper, Omohundro (2008) argued that goal preservation is a fundamental drive of any intelligent system, since a goal is more likely to be achieved if future versions of the agent strive towards the same goal. In this paper, we formalise this argument in general reinforcement learning, and explore situations where it fails. Our conclusion is that the self-modification possibility is harmless if and only if the value function of the agent anticipates the consequences of self-modifications and use the current utility function when evaluating the future.
△ Less
Submitted 10 May, 2016;
originally announced May 2016.
-
A Definition of Happiness for Reinforcement Learning Agents
Authors:
Mayank Daswani,
Jan Leike
Abstract:
What is happiness for reinforcement learning agents? We seek a formal definition satisfying a list of desiderata. Our proposed definition of happiness is the temporal difference error, i.e. the difference between the value of the obtained reward and observation and the agent's expectation of this value. This definition satisfies most of our desiderata and is compatible with empirical research on h…
▽ More
What is happiness for reinforcement learning agents? We seek a formal definition satisfying a list of desiderata. Our proposed definition of happiness is the temporal difference error, i.e. the difference between the value of the obtained reward and observation and the agent's expectation of this value. This definition satisfies most of our desiderata and is compatible with empirical research on humans. We state several implications and discuss examples.
△ Less
Submitted 17 May, 2015;
originally announced May 2015.
