-
Implicit Bias in Noisy-SGD: With Applications to Differentially Private Training
Authors:
Tom Sander,
Maxime Sylvestre,
Alain Durmus
Abstract:
Training Deep Neural Networks (DNNs) with small batches using Stochastic Gradient Descent (SGD) yields superior test performance compared to larger batches. The specific noise structure inherent to SGD is known to be responsible for this implicit bias. DP-SGD, used to ensure differential privacy (DP) in DNNs' training, adds Gaussian noise to the clipped gradients. Surprisingly, large-batch trainin…
▽ More
Training Deep Neural Networks (DNNs) with small batches using Stochastic Gradient Descent (SGD) yields superior test performance compared to larger batches. The specific noise structure inherent to SGD is known to be responsible for this implicit bias. DP-SGD, used to ensure differential privacy (DP) in DNNs' training, adds Gaussian noise to the clipped gradients. Surprisingly, large-batch training still results in a significant decrease in performance, which poses an important challenge because strong DP guarantees necessitate the use of massive batches. We first show that the phenomenon extends to Noisy-SGD (DP-SGD without clipping), suggesting that the stochasticity (and not the clipping) is the cause of this implicit bias, even with additional isotropic Gaussian noise. We theoretically analyse the solutions obtained with continuous versions of Noisy-SGD for the Linear Least Square and Diagonal Linear Network settings, and reveal that the implicit bias is indeed amplified by the additional noise. Thus, the performance issues of large-batch DP-SGD training are rooted in the same underlying principles as SGD, offering hope for potential improvements in large batch training strategies.
△ Less
Submitted 13 February, 2024;
originally announced February 2024.
-
Convergence Rates of the Regularized Optimal Transport : Disentangling Suboptimality and Entropy
Authors:
Hugo Malamut,
Maxime Sylvestre
Abstract:
We study the convergence of the transport plans $γ_ε$ towards $γ_0$ as well as the cost of the entropy-regularized optimal transport $(c,γ_ε)$ towards $(c,γ_0)$ as the regularization parameter $ε$ vanishes in the setting of finite entropy marginals. We show that under the assumption of infinitesimally twisted cost and compactly supported marginals the distance $W_2(γ_ε,γ_0)$ is asymptotically grea…
▽ More
We study the convergence of the transport plans $γ_ε$ towards $γ_0$ as well as the cost of the entropy-regularized optimal transport $(c,γ_ε)$ towards $(c,γ_0)$ as the regularization parameter $ε$ vanishes in the setting of finite entropy marginals. We show that under the assumption of infinitesimally twisted cost and compactly supported marginals the distance $W_2(γ_ε,γ_0)$ is asymptotically greater than $C\sqrtε$ and the suboptimality $(c,γ_ε)-(c,γ_0)$ is of order $ε$. In the quadratic cost case the compactness assumption is relaxed into a moment of order $2+δ$ assumption. Moreover, in the case of a Lipschitz transport map for the non-regularized problem, the distance $W_2(γ_ε,γ_0)$ converges to $0$ at rate $\sqrtε$. Finally, if in addition the marginals have finite Fisher information, we prove $(c,γ_ε)-(c,γ_0) \sim dε/2$ and we provide a companion expansion of $H(γ_ε)$. These results are achieved by disentangling the role of the cost and the entropy in the regularized problem.
△ Less
Submitted 12 June, 2023;
originally announced June 2023.
-
Monotone comparative statics for submodular functions, with an application to aggregated deferred acceptance
Authors:
Alfred Galichon,
Yu-Wei Hsieh,
Maxime Sylvestre
Abstract:
We propose monotone comparative statics results for maximizers of submodular functions, as opposed to maximizers of supermodular functions as in the classical theory put forth by Veinott, Topkis, Milgrom, and Shannon among others. We introduce matrons, a natural structure that is dual to sublattices that generalizes existing structures such as matroids and polymatroids in combinatorial optimizatio…
▽ More
We propose monotone comparative statics results for maximizers of submodular functions, as opposed to maximizers of supermodular functions as in the classical theory put forth by Veinott, Topkis, Milgrom, and Shannon among others. We introduce matrons, a natural structure that is dual to sublattices that generalizes existing structures such as matroids and polymatroids in combinatorial optimization and M-sets in discrete convex analysis. Our monotone comparative statics result is based on a natural order on matrons, which is dual in some sense to Veinott's strong set order on sublattices. As an application, we propose a deferred acceptance algorithm that operates in the case of divisible goods, and we study its convergence properties.
△ Less
Submitted 24 April, 2023;
originally announced April 2023.
-
Ethane in Titan's Stratosphere from Cassini CIRS Far- and Mid-Infrared Spectra
Authors:
Nicholas A Lombardo,
Conor A Nixon,
Melody Sylvestre,
Donald E Jennings,
Nicholas Teanby,
Patrick G J Irwin,
F Michael Flasar
Abstract:
The Cassini Composite Infrared Spectrometer (CIRS) observed thermal emission in the far- and mid-infrared (from 10 cm$^{-1}$ to 1500 cm$^{-1}$), enabling spatiotemporal studies of ethane on Titan across the span of the Cassini mission from 2004 through 2017. Many previous measurements of ethane on Titan have relied on modeling the molecule's mid-infrared $ν_{12}$ band, centered on 822 cm$^{-1}$. O…
▽ More
The Cassini Composite Infrared Spectrometer (CIRS) observed thermal emission in the far- and mid-infrared (from 10 cm$^{-1}$ to 1500 cm$^{-1}$), enabling spatiotemporal studies of ethane on Titan across the span of the Cassini mission from 2004 through 2017. Many previous measurements of ethane on Titan have relied on modeling the molecule's mid-infrared $ν_{12}$ band, centered on 822 cm$^{-1}$. Other bands of ethane at shorter and longer wavelengths were seen, but have not been modeled to measure ethane abundance. Spectral line lists of the far-infrared $ν_{4}$ torsional band at 289 cm$^{-1}$ and the mid-infrared $ν_{8}$ band centered ay 1468 cm$^{-1}$ have recently been studied in the laboratory. We model CIRS observations of each of these bands (along with the $ν_{12}$ band) separately and compare retrieved mixing ratios from each spectral region. Nadir observations of of the $ν_{4}$ band probe the low stratosphere below 100 km. Our equatorial measurements at 289 cm$^{-1}$ show an abundance of (1.0$\pm$0.4) $\times$10$^{-5}$ at 88 km, from 2007 to 2017. This mixing ratio is consistent with measurements at higher altitudes, in contrast to the depletion that many photochemical models predict. Measurements from the $ν_{12}$ and $ν_{8}$ bands are comparable to each other, with the $ν_{12}$ band probing an altitude range that extends deeper in the atmosphere. We suggest future studies of planetary atmospheres may observe the $ν_{8}$ band, enabling shorter wavelength studies of ethane. There may also be an advantage to observing both the ethane $ν_{8}$ band and nearby methane $ν_{4}$ band in the same spectral window.
△ Less
Submitted 5 August, 2019;
originally announced August 2019.
-
Seasonal evolution of temperatures in Titan's lower stratosphere
Authors:
M. Sylvestre,
N. A. Teanby,
J. Vatant d'Ollone,
S. Vinatier,
B. Bézard,
S. Lebonnois,
P. G. J. Irwin
Abstract:
The Cassini mission offered us the opportunity to monitor the seasonal evolution of Titan's atmosphere from 2004 to 2017, i.e. half a Titan year. The lower part of the stratosphere (pressures greater than 10 mbar) is a region of particular interest as there are few available temperature measurements, and because its thermal response to the seasonal and meridional insolation variations undergone by…
▽ More
The Cassini mission offered us the opportunity to monitor the seasonal evolution of Titan's atmosphere from 2004 to 2017, i.e. half a Titan year. The lower part of the stratosphere (pressures greater than 10 mbar) is a region of particular interest as there are few available temperature measurements, and because its thermal response to the seasonal and meridional insolation variations undergone by Titan remains poorly known. In this study, we measure temperatures in Titan's lower stratosphere between 6 mbar and 25 mbar using Cassini/CIRS spectra covering the whole duration of the mission (from 2004 to 2017) and the whole latitude range. We can thus characterize the meridional distribution of temperatures in Titan's lower stratosphere, and how it evolves from northern winter (2004) to summer solstice (2017). Our measurements show that Titan's lower stratosphere undergoes significant seasonal changes, especially at the South pole, where temperature decreases by 19 K at 15 mbar in 4 years.
△ Less
Submitted 5 February, 2019;
originally announced February 2019.
-
Global climate modeling of Saturn's atmosphere. Part II: multi-annual high-resolution dynamical simulations
Authors:
Aymeric Spiga,
Sandrine Guerlet,
Ehouarn Millour,
Mikel Indurain,
Yann Meurdesoif,
Simon Cabanes,
Thomas Dubos,
Jérémy Leconte,
Alexandre Boissinot,
Sébastien Lebonnois,
Mélody Sylvestre,
Thierry Fouchet
Abstract:
The Cassini mission unveiled the intense and diverse activity in Saturn's atmosphere: banded jets, waves, vortices, equatorial oscillations. To set the path towards a better understanding of those phenomena, we performed high-resolution multi-annual numerical simulations of Saturn's atmospheric dynamics. We built a new Global Climate Model [GCM] for Saturn, named the Saturn DYNAMICO GCM, by combin…
▽ More
The Cassini mission unveiled the intense and diverse activity in Saturn's atmosphere: banded jets, waves, vortices, equatorial oscillations. To set the path towards a better understanding of those phenomena, we performed high-resolution multi-annual numerical simulations of Saturn's atmospheric dynamics. We built a new Global Climate Model [GCM] for Saturn, named the Saturn DYNAMICO GCM, by combining a radiative-seasonal model tailored for Saturn to a hydrodynamical solver based on an icosahedral grid suitable for massively-parallel architectures. The impact of numerical dissipation, and the conservation of angular momentum, are examined in the model before a reference simulation employing the Saturn DYNAMICO GCM with a $1/2^{\circ}$ latitude-longitude resolution is considered for analysis. Mid-latitude banded jets showing similarity with observations are reproduced by our model. Those jets are accelerated and maintained by eddy momentum transfers to the mean flow, with the magnitude of momentum fluxes compliant with the observed values. The eddy activity is not regularly distributed with time, but appears as bursts; both barotropic and baroclinic instabilities could play a role in the eddy activity. The steady-state latitude of occurrence of jets is controlled by poleward migration during the spin-up of our model. At the equator, a weakly-superrotating tropospheric jet and vertically-stacked alternating stratospheric jets are obtained in our GCM simulations. The model produces Yanai (Rossby-gravity), Rossby and Kelvin waves at the equator, as well as extratropical Rossby waves, and large-scale vortices in polar regions. Challenges remain to reproduce Saturn's powerful superrotating jet and hexagon-shaped circumpolar jet in the troposphere, and downward-propagating equatorial oscillation in the stratosphere.
△ Less
Submitted 1 August, 2019; v1 submitted 3 November, 2018;
originally announced November 2018.
-
Seasonal evolution of $\mathrm{C_2N_2}$, $\mathrm{C_3H_4}$, and $\mathrm{C_4H_2}$ abundances in Titan's lower stratosphere
Authors:
M. Sylvestre,
N. A. Teanby,
S. Vinatier,
S. Lebonnois,
P. G. J. Irwin
Abstract:
We study the seasonal evolution of Titan's lower stratosphere (around 15~mbar) in order to better understand the atmospheric dynamics and chemistry in this part of the atmosphere. We analysed Cassini/CIRS far-IR observations from 2006 to 2016 in order to measure the seasonal variations of three photochemical by-products: $\mathrm{C_4H_2}$, $\mathrm{C_3H_4}$, and $\mathrm{C_2N_2}$. We show that the…
▽ More
We study the seasonal evolution of Titan's lower stratosphere (around 15~mbar) in order to better understand the atmospheric dynamics and chemistry in this part of the atmosphere. We analysed Cassini/CIRS far-IR observations from 2006 to 2016 in order to measure the seasonal variations of three photochemical by-products: $\mathrm{C_4H_2}$, $\mathrm{C_3H_4}$, and $\mathrm{C_2N_2}$. We show that the abundances of these three gases have evolved significantly at northern and southern high latitudes since 2006. We measure a sudden and steep increase of the volume mixing ratios of $\mathrm{C_4H_2}$, $\mathrm{C_3H_4}$, and $\mathrm{C_2N_2}$ at the south pole from 2012 to 2013, whereas the abundances of these gases remained approximately constant at the north pole over the same period. At northern mid-latitudes, $\mathrm{C_2N_2}$ and $\mathrm{C_4H_2}$ abundances decrease after 2012 while $\mathrm{C_3H_4}$ abundances stay constant. The comparison of these volume mixing ratio variations with the predictions of photochemical and dynamical models provides constraints on the seasonal evolution of atmospheric circulation and chemical processes at play.
△ Less
Submitted 28 September, 2017;
originally announced September 2017.
-
Search for cool giant exoplanets around young and nearby stars - VLT/NaCo near-infrared phase-coronagraphic and differential imaging
Authors:
A. -L. Maire,
A. Boccaletti,
J. Rameau,
G. Chauvin,
A. -M. Lagrange,
M. Bonnefoy,
S. Desidera,
M. Sylvestre,
P. Baudoz,
R. Galicher,
D. Mouillet
Abstract:
[Abridged] Context. Spectral differential imaging (SDI) is part of the observing strategy of current and future high-contrast imaging instruments. It aims to reduce the stellar speckles that prevent the detection of cool planets by using in/out methane-band images. It attenuates the signature of off-axis companions to the star, such as angular differential imaging (ADI). However, this attenuation…
▽ More
[Abridged] Context. Spectral differential imaging (SDI) is part of the observing strategy of current and future high-contrast imaging instruments. It aims to reduce the stellar speckles that prevent the detection of cool planets by using in/out methane-band images. It attenuates the signature of off-axis companions to the star, such as angular differential imaging (ADI). However, this attenuation depends on the spectral properties of the low-mass companions we are searching for. The implications of this particularity on estimating the detection limits have been poorly explored so far. Aims. We perform an imaging survey to search for cool (Teff<1000-1300 K) giant planets at separations as close as 5-10 AU. We also aim to assess the sensitivity limits in SDI data taking the photometric bias into account. This will lead to a better view of the SDI performance. Methods. We observed a selected sample of 16 stars (age < 200 Myr, d < 25 pc) with the phase-mask coronagraph, SDI, and ADI modes of VLT/NaCo. Results. We do not detect any companions. As for the sensitivity limits, we argue that the SDI residual noise cannot be converted into mass limits because it represents a differential flux, unlike the case of single-band images. This results in degeneracies for the mass limits, which may be removed with the use of single-band constraints. We instead employ a method of directly determining the mass limits. The survey is sensitive to cool giant planets beyond 10 AU for 65% and 30 AU for 100% of the sample. Conclusions. For close-in separations, the optimal regime for SDI corresponds to SDI flux ratios >2. According to the BT-Settl model, this translates into Teff<800 K. The methods described here can be applied to the data interpretation of SPHERE. We expect better performance with the dual-band imager IRDIS, thanks to more suitable filter characteristics and better image quality.
△ Less
Submitted 5 May, 2014; v1 submitted 14 April, 2014;
originally announced April 2014.
-
Search for cool extrasolar giant planets combining coronagraphy, spectral and angular differential imaging
Authors:
A. -L. Maire,
A. Boccaletti,
J. Rameau,
G. Chauvin,
A. -M. Lagrange,
M. Bonnefoy,
S. Desidera,
M. Sylvestre,
P. Baudoz,
R. Galicher,
D. Mouillet
Abstract:
Spectral differential imaging (SDI) is part of the observing strategy of current and on-going high-contrast imaging instruments on ground-based telescopes. Although it improves the star light rejection, SDI attenuates the signature of off-axis companions to the star, just like angular differential imaging (ADI). However, the attenuation due to SDI has the peculiarity of being dependent on the spec…
▽ More
Spectral differential imaging (SDI) is part of the observing strategy of current and on-going high-contrast imaging instruments on ground-based telescopes. Although it improves the star light rejection, SDI attenuates the signature of off-axis companions to the star, just like angular differential imaging (ADI). However, the attenuation due to SDI has the peculiarity of being dependent on the spectral properties of the companions. To date, no study has investigated these effects. Our team is addressing this problem based on data from a direct imaging survey of 16 stars combining the phase-mask coronagraph, the SDI and the ADI modes of VLT/NaCo. The objective of the survey is to search for cool (Teff<1000-1300 K) giant planets at separations of 5-10 AU orbiting young, nearby stars (<200 Myr, <25 pc). The data analysis did not yield any detections. As for the estimation of the sensitivity limits of SDI-processed images, we show that it requires a different analysis than that used in ADI-based surveys. Based on a method using the flux predictions of evolutionary models and avoiding the estimation of contrast, we determine directly the mass sensitivity limits of the survey for the ADI processing alone and with the combination of SDI and ADI. We show that SDI does not systematically improve the sensitivity due to the spectral properties and self-subtraction of point sources.
△ Less
Submitted 6 April, 2014;
originally announced April 2014.
