Showing 1–11 of 11 results for author: Pastore, M

Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers

Authors: Federico Bassetti, Marco Gherardi, Alessandro Ingrosso, Mauro Pastore, Pietro Rotondo

Abstract: Deep linear networks have been extensively studied, as they provide simplified models of deep learning. However, little is known in the case of finite-width architectures with multiple outputs and convolutional layers. In this manuscript, we provide rigorous results for the statistics of functions implemented by the aforementioned class of networks, thus moving closer to a complete characterizatio… ▽ More Deep linear networks have been extensively studied, as they provide simplified models of deep learning. However, little is known in the case of finite-width architectures with multiple outputs and convolutional layers. In this manuscript, we provide rigorous results for the statistics of functions implemented by the aforementioned class of networks, thus moving closer to a complete characterization of feature learning in the Bayesian setting. Our results include: (i) an exact and elementary non-asymptotic integral representation for the joint prior distribution over the outputs, given in terms of a mixture of Gaussians; (ii) an analytical formula for the posterior distribution in the case of squared error loss function (Gaussian likelihood); (iii) a quantitative description of the feature learning infinite-width regime, using large deviation theory. From a physical perspective, deep architectures with multiple outputs or convolutional layers represent different manifestations of kernel shape renormalization, and our work provides a dictionary that translates this physics intuition and terminology into rigorous Bayesian statistics. △ Less

Submitted 5 June, 2024; originally announced June 2024.

MSC Class: 62E20; 62E15; 82B44

Restoring balance: principled under/oversampling of data for optimal classification

Authors: Emanuele Loffredo, Mauro Pastore, Simona Cocco, Rémi Monasson

Abstract: Class imbalance in real-world data poses a common bottleneck for machine learning tasks, since achieving good generalization on under-represented examples is often challenging. Mitigation strategies, such as under or oversampling the data depending on their abundances, are routinely proposed and tested empirically, but how they should adapt to the data statistics remains poorly understood. In this… ▽ More Class imbalance in real-world data poses a common bottleneck for machine learning tasks, since achieving good generalization on under-represented examples is often challenging. Mitigation strategies, such as under or oversampling the data depending on their abundances, are routinely proposed and tested empirically, but how they should adapt to the data statistics remains poorly understood. In this work, we determine exact analytical expressions of the generalization curves in the high-dimensional regime for linear classifiers (Support Vector Machines). We also provide a sharp prediction of the effects of under/oversampling strategies depending on class imbalance, first and second moments of the data, and the metrics of performance considered. We show that mixed strategies involving under and oversampling of data lead to performance improvement. Through numerical experiments, we show the relevance of our theoretical predictions on real datasets, on deeper architectures and with sampling strategies based on unsupervised probabilistic models. △ Less

Submitted 15 May, 2024; originally announced May 2024.

Comments: 9 pages + appendix, 3 figures

Random features and polynomial rules

Authors: Fabián Aguirre-López, Silvio Franz, Mauro Pastore

Abstract: Random features models play a distinguished role in the theory of deep learning, describing the behavior of neural networks close to their infinite-width limit. In this work, we present a thorough analysis of the generalization performance of random features models for generic supervised learning problems with Gaussian data. Our approach, built with tools from the statistical mechanics of disorder… ▽ More Random features models play a distinguished role in the theory of deep learning, describing the behavior of neural networks close to their infinite-width limit. In this work, we present a thorough analysis of the generalization performance of random features models for generic supervised learning problems with Gaussian data. Our approach, built with tools from the statistical mechanics of disordered systems, maps the random features model to an equivalent polynomial model, and allows us to plot average generalization curves as functions of the two main control parameters of the problem: the number of random features $N$ and the size $P$ of the training set, both assumed to scale as powers in the input dimension $D$. Our results extend the case of proportional scaling between $N$, $P$ and $D$. They are in accordance with rigorous bounds known for certain particular learning tasks and are in quantitative agreement with numerical experiments performed over many order of magnitudes of $N$ and $P$. We find good agreement also far from the asymptotic limits where $D\to \infty$ and at least one between $P/D^K$, $N/D^L$ remains finite. △ Less

Submitted 15 February, 2024; originally announced February 2024.

Comments: 11 pages + appendix, 4 figures. Comments are welcome

A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit

Authors: R. Pacelli, S. Ariosto, M. Pastore, F. Ginelli, M. Gherardi, P. Rotondo

Abstract: Despite the practical success of deep neural networks, a comprehensive theoretical framework that can predict practically relevant scores, such as the test accuracy, from knowledge of the training data is currently lacking. Huge simplifications arise in the infinite-width limit, where the number of units $N_\ell$ in each hidden layer ($\ell=1,\dots, L$, being $L$ the depth of the network) far exce… ▽ More Despite the practical success of deep neural networks, a comprehensive theoretical framework that can predict practically relevant scores, such as the test accuracy, from knowledge of the training data is currently lacking. Huge simplifications arise in the infinite-width limit, where the number of units $N_\ell$ in each hidden layer ($\ell=1,\dots, L$, being $L$ the depth of the network) far exceeds the number $P$ of training examples. This idealisation, however, blatantly departs from the reality of deep learning practice. Here, we use the toolset of statistical mechanics to overcome these limitations and derive an approximate partition function for fully-connected deep neural architectures, which encodes information about the trained models. The computation holds in the ''thermodynamic limit'' where both $N_\ell$ and $P$ are large and their ratio $α_\ell = P/N_\ell$ is finite. This advance allows us to obtain (i) a closed formula for the generalisation error associated to a regression task in a one-hidden layer network with finite $α_1$; (ii) an approximate expression of the partition function for deep architectures (via an ''effective action'' that depends on a finite number of ''order parameters''); (iii) a link between deep neural networks in the proportional asymptotic limit and Student's $t$ processes. △ Less

Submitted 9 December, 2023; v1 submitted 11 September, 2022; originally announced September 2022.

Comments: Supplemental material included

Journal ref: Nat Mach Intell 5, 1497-1507 (2023)

Satisfiability transition in asymmetric neural networks

Authors: Fabián Aguirre-López, Mauro Pastore, Silvio Franz

Abstract: Asymmetry in the synaptic interactions between neurons plays a crucial role in determining the memory storage and retrieval properties of recurrent neural networks. In this work, we analyze the problem of storing random memories in a network of neurons connected by a synaptic matrix with a definite degree of asymmetry. We study the corresponding satisfiability and clustering transitions in the spa… ▽ More Asymmetry in the synaptic interactions between neurons plays a crucial role in determining the memory storage and retrieval properties of recurrent neural networks. In this work, we analyze the problem of storing random memories in a network of neurons connected by a synaptic matrix with a definite degree of asymmetry. We study the corresponding satisfiability and clustering transitions in the space of solutions of the constraint satisfaction problem associated with finding synaptic matrices given the memories. We find, besides the usual SAT/UNSAT transition at a critical number of memories to store in the network, an additional transition for very asymmetric matrices, where the competing constraints (definite asymmetry vs. memories storage) induce enough frustration in the problem to make it impossible to solve. This finding is particularly striking in the case of a single memory to store, where no quenched disorder is present in the system. △ Less

Submitted 10 October, 2022; v1 submitted 7 April, 2022; originally announced April 2022.

Comments: 24 pages + appendix, 7 figures

Journal ref: J. Phys. A: Math. Theor. 55 305001 (2022)

Critical properties of the SAT/UNSAT transitions in the classification problem of structured data

Authors: Mauro Pastore

Abstract: The classification problem of structured data can be solved with different strategies: a supervised learning approach, starting from a labeled training set, and an unsupervised learning one, where only the structure of the patterns in the dataset is used to find a classification compatible with it. The two strategies can be interpreted as extreme cases of a semi-supervised approach to learn multi-… ▽ More The classification problem of structured data can be solved with different strategies: a supervised learning approach, starting from a labeled training set, and an unsupervised learning one, where only the structure of the patterns in the dataset is used to find a classification compatible with it. The two strategies can be interpreted as extreme cases of a semi-supervised approach to learn multi-view data, relevant for applications. In this paper I study the critical properties of the two storage problems associated with these tasks, in the case of the linear binary classification of doublets of points sharing the same label, within replica theory. While the first approach presents a SAT/UNSAT transition in a (marginally) stable replica-symmetric phase, in the second one the satisfiability line lies in a full replica-symmetry-broken phase. A similar behavior in the problem of learning with a margin is also pointed out. △ Less

Submitted 8 November, 2021; v1 submitted 17 September, 2021; originally announced September 2021.

Comments: 20 pages + appendices, 8 figures

Journal ref: J. Stat. Mech. (2021) 113301

Self-induced glassy phase in multimodal cavity quantum electrodynamics

Authors: Vittorio Erba, Mauro Pastore, Pietro Rotondo

Abstract: We provide strong evidence that the effective spin-spin interaction in a multimodal confocal optical cavity gives rise to a self-induced glassy phase, which emerges exclusively from the peculiar euclidean correlations and is not related to the presence of disorder as in standard spin glasses. As recently shown, this spin-spin effective interaction is both non-local and non-translational invariant,… ▽ More We provide strong evidence that the effective spin-spin interaction in a multimodal confocal optical cavity gives rise to a self-induced glassy phase, which emerges exclusively from the peculiar euclidean correlations and is not related to the presence of disorder as in standard spin glasses. As recently shown, this spin-spin effective interaction is both non-local and non-translational invariant, and randomness in the atoms positions produces a spin glass phase. Here we consider the simplest feasible disorder-free setting where atoms form a one-dimensional regular chain and we study the thermodynamics of the resulting effective Ising model. We present extensive results showing that the system has a low-temperature glassy phase. Notably, for rational values of the only free adimensional parameter $α=p/q$ of the interaction, the number of metastable states at low temperature grows exponentially with $q$ and the problem of finding the ground state rapidly becomes computationally intractable, suggesting that the system develops high energy barriers and ergodicity breaking occurs. △ Less

Submitted 11 January, 2021; originally announced January 2021.

Journal ref: Phys. Rev. Lett. 126, 183601 (2021)

Statistical learning theory of structured data

Authors: Mauro Pastore, Pietro Rotondo, Vittorio Erba, Marco Gherardi

Abstract: The traditional approach of statistical physics to supervised learning routinely assumes unrealistic generative models for the data: usually inputs are independent random variables, uncorrelated with their labels. Only recently, statistical physicists started to explore more complex forms of data, such as equally-labelled points lying on (possibly low dimensional) object manifolds. Here we provide… ▽ More The traditional approach of statistical physics to supervised learning routinely assumes unrealistic generative models for the data: usually inputs are independent random variables, uncorrelated with their labels. Only recently, statistical physicists started to explore more complex forms of data, such as equally-labelled points lying on (possibly low dimensional) object manifolds. Here we provide a bridge between this recently-established research area and the framework of statistical learning theory, a branch of mathematics devoted to inference in machine learning. The overarching motivation is the inadequacy of the classic rigorous results in explaining the remarkable generalization properties of deep learning. We propose a way to integrate physical models of data into statistical learning theory, and address, with both combinatorial and statistical mechanics methods, the computation of the Vapnik-Chervonenkis entropy, which counts the number of different binary classifications compatible with the loss class. As a proof of concept, we focus on kernel machines and on two simple realizations of data structure introduced in recent physics literature: $k$-dimensional simplexes with prescribed geometric relations and spherical manifolds (equivalent to margin classification). Entropy, contrary to what happens for unstructured data, is nonmonotonic in the sample size, in contrast with the rigorous bounds. Moreover, data structure induces a novel transition beyond the storage capacity, which we advocate as a proxy of the nonmonotonicity, and ultimately a cue of low generalization error. The identification of a synaptic volume vanishing at the transition allows a quantification of the impact of data structure within replica theory, applicable in cases where combinatorial methods are not available, as we demonstrate for margin learning. △ Less

Submitted 20 May, 2020; originally announced May 2020.

Comments: 19 pages, 3 figures

Journal ref: Phys. Rev. E 102, 032119 (2020)

Beyond the storage capacity: data driven satisfiability transition

Authors: Pietro Rotondo, Mauro Pastore, Marco Gherardi

Abstract: Data structure has a dramatic impact on the properties of neural networks, yet its significance in the established theoretical frameworks is poorly understood. Here we compute the Vapnik-Chervonenkis entropy of a kernel machine operating on data grouped into equally labelled subsets. At variance with the unstructured scenario, entropy is non-monotonic in the size of the training set, and displays… ▽ More Data structure has a dramatic impact on the properties of neural networks, yet its significance in the established theoretical frameworks is poorly understood. Here we compute the Vapnik-Chervonenkis entropy of a kernel machine operating on data grouped into equally labelled subsets. At variance with the unstructured scenario, entropy is non-monotonic in the size of the training set, and displays an additional critical point besides the storage capacity. Remarkably, the same behavior occurs in margin classifiers even with randomly labelled data, as is elucidated by identifying the synaptic volume encoding the transition. These findings reveal aspects of expressivity lying beyond the condensed description provided by the storage capacity, and they indicate the path towards more realistic bounds for the generalization error of neural networks. △ Less

Submitted 20 May, 2020; originally announced May 2020.

Comments: 5 pages, 2 figures

Journal ref: Phys. Rev. Lett. 125, 120601 (2020)

Large deviations of the free energy in the p-spin glass spherical model

Authors: Mauro Pastore, Andrea Di Gioacchino, Pietro Rotondo

Abstract: We investigate the behavior of the rare fluctuations of the free energy in the p-spin spherical model, evaluating the corresponding rate function via the Gärtner-Ellis theorem. This approach requires the knowledge of the analytic continuation of the disorder-averaged replicated partition function to arbitrary real number of replicas. In zero external magnetic field, we show via a one-step replica… ▽ More We investigate the behavior of the rare fluctuations of the free energy in the p-spin spherical model, evaluating the corresponding rate function via the Gärtner-Ellis theorem. This approach requires the knowledge of the analytic continuation of the disorder-averaged replicated partition function to arbitrary real number of replicas. In zero external magnetic field, we show via a one-step replica symmetry breaking (1RSB) calculation that the rate function is infinite for fluctuations of the free energy above its typical value, corresponding to an anomalous, super-extensive suppression of rare fluctuations. We extend this calculation to non-zero magnetic field, showing that in this case this very large deviation disappears and we try to motivate this finding in light of a geometrical interpretation of the scaled cumulant generating function. △ Less

Submitted 25 October, 2019; v1 submitted 13 September, 2019; originally announced September 2019.

Comments: 17 pages, 4 figures, better graphics

Journal ref: Phys. Rev. Research 1, 033116 (2019)

Remarks on replica diagonal collective field condensations in SYK

Authors: Sergio Caracciolo, Matteo A. Cardella, Mauro Pastore

Abstract: In the Sachdev-Ye-Kitaev model with generic order $q \ge 4$ random couplings, we compute the critical temperature relating the Majorana fermions high temperature perturbative vacuum to the vacuum where the replica diagonal collective field $G(τ, τ')$ condenses. We study, by a finite temperature diagrammatic analysis, the effective action of an auxiliary Hubbard-Stratonovich bilocal field related t… ▽ More In the Sachdev-Ye-Kitaev model with generic order $q \ge 4$ random couplings, we compute the critical temperature relating the Majorana fermions high temperature perturbative vacuum to the vacuum where the replica diagonal collective field $G(τ, τ')$ condenses. We study, by a finite temperature diagrammatic analysis, the effective action of an auxiliary Hubbard-Stratonovich bilocal field related to $G(τ, τ')$ in the large $N$ limit. Subtelties that arise in switching from the operatorial to the functional integral representation of the SYK thermal partition function are also discussed. △ Less

Submitted 22 November, 2018; v1 submitted 26 July, 2018; originally announced July 2018.

Comments: The perturbative expansion has not kept into account the necessary subtractions