Showing 1–50 of 108 results for author: Yun, C

Position Coupling: Leveraging Task Structure for Improved Length Generalization of Transformers

Authors: Hanseul Cho, Jaeyoung Cha, Pranjal Awasthi, Srinadh Bhojanapalli, Anupam Gupta, Chulhee Yun

Abstract: Even for simple arithmetic tasks like integer addition, it is challenging for Transformers to generalize to longer sequences than those encountered during training. To tackle this problem, we propose position coupling, a simple yet effective method that directly embeds the structure of the tasks into the positional encoding of a (decoder-only) Transformer. Taking a departure from the vanilla absol… ▽ More Even for simple arithmetic tasks like integer addition, it is challenging for Transformers to generalize to longer sequences than those encountered during training. To tackle this problem, we propose position coupling, a simple yet effective method that directly embeds the structure of the tasks into the positional encoding of a (decoder-only) Transformer. Taking a departure from the vanilla absolute position mechanism assigning unique position IDs to each of the tokens, we assign the same position IDs to two or more "relevant" tokens; for integer addition tasks, we regard digits of the same significance as in the same position. On the empirical side, we show that with the proposed position coupling, a small (1-layer) Transformer trained on 1 to 30-digit additions can generalize up to 200-digit additions (6.67x of the trained length). On the theoretical side, we prove that a 1-layer Transformer with coupled positions can solve the addition task involving exponentially many digits, whereas any 1-layer Transformer without positional information cannot entirely solve it. We also demonstrate that position coupling can be applied to other algorithmic tasks such as addition with multiple summands, Nx2 multiplication, copy/reverse, and a two-dimensional task. △ Less

Submitted 31 May, 2024; originally announced May 2024.

Comments: 73 pages, 20 figures, 90 tables

Does SGD really happen in tiny subspaces?

Authors: Minhak Song, Kwangjun Ahn, Chulhee Yun

Abstract: Understanding the training dynamics of deep neural networks is challenging due to their high-dimensional nature and intricate loss landscapes. Recent studies have revealed that, along the training trajectory, the gradient approximately aligns with a low-rank top eigenspace of the training loss Hessian, referred to as the dominant subspace. Given this alignment, this paper explores whether neural n… ▽ More Understanding the training dynamics of deep neural networks is challenging due to their high-dimensional nature and intricate loss landscapes. Recent studies have revealed that, along the training trajectory, the gradient approximately aligns with a low-rank top eigenspace of the training loss Hessian, referred to as the dominant subspace. Given this alignment, this paper explores whether neural networks can be trained within the dominant subspace, which, if feasible, could lead to more efficient training methods. Our primary observation is that when the SGD update is projected onto the dominant subspace, the training loss does not decrease further. This suggests that the observed alignment between the gradient and the dominant subspace is spurious. Surprisingly, projecting out the dominant subspace proves to be just as effective as the original update, despite removing the majority of the original update component. Similar observations are made for the large learning rate regime (also known as Edge of Stability) and Sharpness-Aware Minimization. We discuss the main causes and implications of this spurious alignment, shedding light on the intricate dynamics of neural network training. △ Less

Submitted 24 May, 2024; originally announced May 2024.

Comments: 22 pages

Optimal Switching Networks for Paired-Egress Bell State Analyzer Pools

Authors: Marii Koyama, Claire Yun, Amin Taherkhani, Naphan Benchasattabuse, Bernard Ousmane Sane, Michal Hajdušek, Shota Nagayama, Rodney Van Meter

Abstract: To scale quantum computers to useful levels, we must build networks of quantum computational nodes that can share entanglement for use in distributed forms of quantum algorithms. In one proposed architecture, node-to-node entanglement is created when nodes emit photons entangled with stationary memories, with the photons routed through a switched interconnect to a shared pool of Bell state analyze… ▽ More To scale quantum computers to useful levels, we must build networks of quantum computational nodes that can share entanglement for use in distributed forms of quantum algorithms. In one proposed architecture, node-to-node entanglement is created when nodes emit photons entangled with stationary memories, with the photons routed through a switched interconnect to a shared pool of Bell state analyzers (BSAs). Designs that optimize switching circuits will reduce loss and crosstalk, raising entanglement rates and fidelity. We present optimal designs for switched interconnects constrained to planar layouts, appropriate for silicon waveguides and Mach-Zehnder interferometer (MZI) $2 \times 2$ switch points. The architectures for the optimal designs are scalable and algorithmically structured to pair any arbitrary inputs in a rearrangeable, non-blocking way. For pairing $N$ inputs, $N(N - 2)/4$ switches are required, which is less than half of number of switches required for full permutation switching networks. An efficient routing algorithm is also presented for each architecture. These designs can also be employed in reverse for entanglement generation using a shared pool of entangled paired photon sources. △ Less

Submitted 16 May, 2024; originally announced May 2024.

Comments: 11 pages, 8 figures, 1 table

Nonvolatile optical control of interlayer stacking order in 1T-TaS2

Authors: Junde Liu, Pei Liu, Liu Yang, Sung-Hoon Lee, Mojun Pan, Famin Chen, Jierui Huang, Bei Jiang, Mingzhe Hu, Yuchong Zhang, Zhaoyang Xie, Gang Wang, Mengxue Guan, Wei Jiang, Huaixin Yang, Jianqi Li, Chenxia Yun, Zhiwei Wang, Sheng Meng, Yugui Yao, Tian Qian, Xun Shi

Abstract: Nonvolatile optical manipulation of material properties on demand is a highly sought-after feature in the advancement of future optoelectronic applications. While the discovery of such metastable transition in various materials holds good promise for achieving this goal, their practical implementation is still in the nascent stage. Here, we unravel the nature of the ultrafast laser-induced hidden… ▽ More Nonvolatile optical manipulation of material properties on demand is a highly sought-after feature in the advancement of future optoelectronic applications. While the discovery of such metastable transition in various materials holds good promise for achieving this goal, their practical implementation is still in the nascent stage. Here, we unravel the nature of the ultrafast laser-induced hidden state in 1T-TaS2 by systematically characterizing the electronic structure evolution throughout the reversible transition cycle. We identify it as a mixed-stacking state involving two similarly low-energy interlayer orders, which is manifested as the charge density wave phase disruption. Furthermore, our comparative experiments utilizing the single-pulse writing, pulse-train erasing and pulse-pair control explicitly reveal the distinct mechanism of the bidirectional transformations -- the ultrafast formation of the hidden state is initiated by a coherent phonon which triggers a competition of interlayer stacking orders, while its recovery to the initial state is governed by the progressive domain coarsening. Our work highlights the deterministic role of the competing interlayer orders in the nonvolatile phase transition in the layered material 1T-TaS2, and promises the coherent control of the phase transition and switching speed. More importantly, these results establish all-optical engineering of stacking orders in low-dimensional materials as a viable strategy for achieving desirable nonvolatile electronic devices. △ Less

Submitted 5 May, 2024; originally announced May 2024.

On the topology of the moduli of tropical unramified $p$-covers

Authors: Yassine El Maazouz, Paul Alexander Helminck, Felix Röhrle, Pedro Souza, Claudia He Yun

Abstract: We study the topology of the moduli space of tropical unramified $\mathbb{Z}/p$-covers of tropical curves of genus $g \geq 2$ where $p$ is a prime number. We use recent techniques by Chan--Galatius--Payne to identify a contractible subcomplex of the moduli space. We then use this contractibility result to show that this moduli space is simply connected for all $g$ and $p$. In the case of genus… ▽ More We study the topology of the moduli space of tropical unramified $\mathbb{Z}/p$-covers of tropical curves of genus $g \geq 2$ where $p$ is a prime number. We use recent techniques by Chan--Galatius--Payne to identify a contractible subcomplex of the moduli space. We then use this contractibility result to show that this moduli space is simply connected for all $g$ and $p$. In the case of genus $2$ we fully determine the homotopy type of this moduli space for all primes $p$. This work is motivated by prospective applications to the top-weight cohomology of the space of prime cyclic étale covers of smooth algebraic curves. △ Less

Submitted 12 March, 2024; v1 submitted 11 March, 2024; originally announced March 2024.

Comments: 38 pages, 11 figures, 5 tables

MSC Class: 14T20; 05E14; 14H10

Fundamental Benefit of Alternating Updates in Minimax Optimization

Authors: Jaewook Lee, Hanseul Cho, Chulhee Yun

Abstract: The Gradient Descent-Ascent (GDA) algorithm, designed to solve minimax optimization problems, takes the descent and ascent steps either simultaneously (Sim-GDA) or alternately (Alt-GDA). While Alt-GDA is commonly observed to converge faster, the performance gap between the two is not yet well understood theoretically, especially in terms of global convergence rates. To address this theory-practice… ▽ More The Gradient Descent-Ascent (GDA) algorithm, designed to solve minimax optimization problems, takes the descent and ascent steps either simultaneously (Sim-GDA) or alternately (Alt-GDA). While Alt-GDA is commonly observed to converge faster, the performance gap between the two is not yet well understood theoretically, especially in terms of global convergence rates. To address this theory-practice gap, we present fine-grained convergence analyses of both algorithms for strongly-convex-strongly-concave and Lipschitz-gradient objectives. Our new iteration complexity upper bound of Alt-GDA is strictly smaller than the lower bound of Sim-GDA; i.e., Alt-GDA is provably faster. Moreover, we propose Alternating-Extrapolation GDA (Alex-GDA), a general algorithmic framework that subsumes Sim-GDA and Alt-GDA, for which the main idea is to alternately take gradients from extrapolations of the iterates. We show that Alex-GDA satisfies a smaller iteration complexity bound, identical to that of the Extra-gradient method, while requiring less gradient computations. We also prove that Alex-GDA enjoys linear convergence for bilinear problems, for which both Sim-GDA and Alt-GDA fail to converge at all. △ Less

Submitted 15 July, 2024; v1 submitted 16 February, 2024; originally announced February 2024.

Comments: Accepted to ICML 2024 (Spotlight). 76 pages, 2 figures. Additional experiments (quadratic game, GAN) and proofs

Coronary CTA and Quantitative Cardiac CT Perfusion (CCTP) in Coronary Artery Disease

Authors: Hao Wu, Yingnan Song, Ammar Hoori, Ananya Subramaniam, Juhwan Lee, Justin Kim, Tao Hu, Sadeer Al-Kindi, Wei-Ming Huang, Chun-Ho Yun, Chung-Lieh Hung, Sanjay Rajagopalan, David L. Wilson

Abstract: We assessed the benefit of combining stress cardiac CT perfusion (CCTP) myocardial blood flow (MBF) with coronary CT angiography (CCTA) using our innovative CCTP software. By combining CCTA and CCTP, one can uniquely identify a flow limiting stenosis (obstructive-lesion + low-MBF) versus MVD (no-obstructive-lesion + low-MBF. We retrospectively evaluated 104 patients with suspected CAD, including 1… ▽ More We assessed the benefit of combining stress cardiac CT perfusion (CCTP) myocardial blood flow (MBF) with coronary CT angiography (CCTA) using our innovative CCTP software. By combining CCTA and CCTP, one can uniquely identify a flow limiting stenosis (obstructive-lesion + low-MBF) versus MVD (no-obstructive-lesion + low-MBF. We retrospectively evaluated 104 patients with suspected CAD, including 18 with diabetes, who underwent CCTA+CCTP. Whole heart and territorial MBF was assessed using our automated pipeline for CCTP analysis that included beam hardening correction; temporal scan registration; automated segmentation; fast, accurate, robust MBF estimation; and visualization. Stenosis severity was scored using the CCTA coronary-artery-disease-reporting-and-data-system (CAD-RADS), with obstructive stenosis deemed as CAD-RADS>=3. We established a threshold MBF (MBF=199-mL/min-100g) for normal perfusion. In patients with CAD-RADS>=3, 28/37(76%) patients showed ischemia in the corresponding territory. Two patients with obstructive disease had normal perfusion, suggesting collaterals and/or a hemodynamically insignificant stenosis. Among diabetics, 10 of 18 (56%) demonstrated diffuse ischemia consistent with MVD. Among non-diabetics, only 6% had MVD. Sex-specific prevalence of MVD was 21%/24% (M/F). On a per-vessel basis (n=256), MBF showed a significant difference between territories with and without obstructive stenosis (165 +/- 61 mL/min-100g vs. 274 +/- 62 mL/min-100g, p <0.05). A significant and negative rank correlation (rho=-0.53, p<0.05) between territory MBF and CAD-RADS was seen. CCTA in conjunction with a new automated quantitative CCTP approach can augment the interpretation of CAD, enabling the distinction of ischemia due to obstructive lesions and MVD. △ Less

Submitted 30 January, 2024; originally announced January 2024.

Pericoronary adipose tissue feature analysis in CT calcium score images with comparison to coronary CTA

Authors: Yingnan Song, Hao Wu, Juhwan Lee, Justin Kim, Ammar Hoori, Tao Hu, Vladislav Zimin, Mohamed Makhlouf, Sadeer Al-Kindi, Sanjay Rajagopalan, Chun-Ho Yun, Chung-Lieh Hung, David L. Wilson

Abstract: We investigated the feasibility and advantages of using non-contrast CT calcium score (CTCS) images to assess pericoronary adipose tissue (PCAT) and its association with major adverse cardiovascular events (MACE). PCAT features from coronary CTA (CCTA) have been shown to be associated with cardiovascular risk but are potentially confounded by iodine. If PCAT in CTCS images can be similarly analyze… ▽ More We investigated the feasibility and advantages of using non-contrast CT calcium score (CTCS) images to assess pericoronary adipose tissue (PCAT) and its association with major adverse cardiovascular events (MACE). PCAT features from coronary CTA (CCTA) have been shown to be associated with cardiovascular risk but are potentially confounded by iodine. If PCAT in CTCS images can be similarly analyzed, it would avoid this issue and enable its inclusion in formal risk assessment from readily available, low-cost CTCS images. To identify coronaries in CTCS images that have subtle visual evidence of vessels, we registered CTCS with paired CCTA images having coronary labels. We developed a novel axial-disk method giving regions for analyzing PCAT features in three main coronary arteries. We analyzed novel hand-crafted and radiomic features using univariate and multivariate logistic regression prediction of MACE and compared results against those from CCTA. Registration accuracy was sufficient to enable the identification of PCAT regions in CTCS images. Motion or beam hardening artifacts were often present in high-contrast CCTA but not CTCS. Mean HU and volume were increased in both CTCS and CCTA for MACE group. There were significant positive correlations between some CTCS and CCTA features, suggesting that similar characteristics were obtained. Using hand-crafted/radiomics from CTCS and CCTA, AUCs were 0.82/0.79 and 0.83/0.77 respectively, while Agatston gave AUC=0.73. Preliminarily, PCAT features can be assessed from three main coronary arteries in non-contrast CTCS images with performance characteristics that are at the very least comparable to CCTA. △ Less

Submitted 27 January, 2024; originally announced January 2024.

Comments: 24 pages,10 figures

Evidence for an Excitonic Insulator State in Ta$_2$Pd$_3$Te$_5$

Authors: Jierui Huang, Bei Jiang, Jingyu Yao, Dayu Yan, Xincheng Lei, Jiacheng Gao, Zhaopeng Guo, Feng Jin, Yupeng Li, Zhenyu Yuan, Congcong Chai, Haohao Sheng, Mojun Pan, Famin Chen, Junde Liu, Shunye Gao, Gexing Qu, Bo Liu, Zhicheng Jiang, Zhengtai Liu, Xiaoyan Ma, Shiming Zhou, Yaobo Huang, Chenxia Yun, Qingming Zhang , et al. (8 additional authors not shown)

Abstract: The excitonic insulator (EI) is an exotic ground state of narrow-gap semiconductors and semimetals arising from spontaneous condensation of electron-hole pairs bound by attractive Coulomb interaction. Despite research on EIs dating back to half a century ago, their existence in real materials remains a subject of ongoing debate. In this study, through systematic experimental and theoretical invest… ▽ More The excitonic insulator (EI) is an exotic ground state of narrow-gap semiconductors and semimetals arising from spontaneous condensation of electron-hole pairs bound by attractive Coulomb interaction. Despite research on EIs dating back to half a century ago, their existence in real materials remains a subject of ongoing debate. In this study, through systematic experimental and theoretical investigations, we provide evidence for the existence of an EI ground state in a van der Waals compound Ta$_2$Pd$_3$Te$_5$. Density-functional-theory calculations suggest that it is a semimetal with a small band overlap, whereas various experiments exhibit an insulating ground state with a clear band gap. Upon incorporating electron-hole Coulomb interaction into our calculations, we obtain an EI phase where the electronic symmetry breaking opens a many-body gap. Angle-resolved photoemission spectroscopy measurements exhibit that the band gap is closed with a significant change in the dispersions as the number of thermally excited charge carriers becomes sufficiently large in both equilibrium and nonequilibrium states. Structural measurements reveal a slight breaking of crystal symmetry with exceptionally small lattice distortion in the insulating state, which cannot account for the significant gap opening. Therefore, we attribute the insulating ground state with a gap opening in Ta$_2$Pd$_3$Te$_5$ to exciton condensation, where the coupling to the symmetry-breaking electronic state induces a subtle change in the crystal structure. △ Less

Submitted 14 March, 2024; v1 submitted 22 December, 2023; originally announced December 2023.

Comments: 10 pages, 5 figures

Journal ref: Phys. Rev. X 14, 011046, 2024

Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapults

Authors: Prin Phunyaphibarn, Junghyun Lee, Bohan Wang, Huishuai Zhang, Chulhee Yun

Abstract: Although gradient descent with Polyak's momentum is widely used in modern machine and deep learning, a concrete understanding of its effects on the training trajectory remains elusive. In this work, we empirically show that for linear diagonal networks and nonlinear neural networks, momentum gradient descent with a large learning rate displays large catapults, driving the iterates towards much fla… ▽ More Although gradient descent with Polyak's momentum is widely used in modern machine and deep learning, a concrete understanding of its effects on the training trajectory remains elusive. In this work, we empirically show that for linear diagonal networks and nonlinear neural networks, momentum gradient descent with a large learning rate displays large catapults, driving the iterates towards much flatter minima than those found by gradient descent. We hypothesize that the large catapult is caused by momentum "prolonging" the self-stabilization effect (Damian et al., 2023). We provide theoretical and empirical support for our hypothesis in a simple toy example and empirical evidence supporting our hypothesis for linear diagonal networks. △ Less

Submitted 29 May, 2024; v1 submitted 25 November, 2023; originally announced November 2023.

Comments: v3: major updates; 25 pages, 17 figures; the first two authors contributed equally. The preliminary version was accepted to the NeurIPS 2023 M3L Workshop (oral) under the title "Large Catapults in Momentum Gradient Descent with Warmup: An Empirical Study."

Fair Streaming Principal Component Analysis: Statistical and Algorithmic Viewpoint

Authors: Junghyun Lee, Hanseul Cho, Se-Young Yun, Chulhee Yun

Abstract: Fair Principal Component Analysis (PCA) is a problem setting where we aim to perform PCA while making the resulting representation fair in that the projected distributions, conditional on the sensitive attributes, match one another. However, existing approaches to fair PCA have two main problems: theoretically, there has been no statistical foundation of fair PCA in terms of learnability; practica… ▽ More Fair Principal Component Analysis (PCA) is a problem setting where we aim to perform PCA while making the resulting representation fair in that the projected distributions, conditional on the sensitive attributes, match one another. However, existing approaches to fair PCA have two main problems: theoretically, there has been no statistical foundation of fair PCA in terms of learnability; practically, limited memory prevents us from using existing approaches, as they explicitly rely on full access to the entire data. On the theoretical side, we rigorously formulate fair PCA using a new notion called \emph{probably approximately fair and optimal} (PAFO) learnability. On the practical side, motivated by recent advances in streaming algorithms for addressing memory limitation, we propose a new setting called \emph{fair streaming PCA} along with a memory-efficient algorithm, fair noisy power method (FNPM). We then provide its {\it statistical} guarantee in terms of PAFO-learnability, which is the first of its kind in fair PCA literature. Lastly, we verify the efficacy and memory efficiency of our algorithm on real-world datasets. △ Less

Submitted 28 October, 2023; originally announced October 2023.

Comments: 42 pages, 5 figures, 4 tables. Accepted to the 37th Conference on Neural Information Processing Systems (NeurIPS 2023)

Time-resolved ARPES with probe energy of 6.0/7.2 eV and switchable resolution configuration

Authors: Mojun Pan, Junde Liu, Famin Chen, Ji Wang, ChenXia Yun, Tian Qian

Abstract: We present a detailed exposition of the design for time- and angle-resolved photoemission spectroscopy using a UV probe laser source that combines the nonlinear effects of \b{eta}-BaB2O4 and KBe2BO3F2 optical crystals. The photon energy of the probe laser can be switched between 6.0 and 7.2 eV, with the flexibility to operate each photon energy setting under two distinct resolution configurations.… ▽ More We present a detailed exposition of the design for time- and angle-resolved photoemission spectroscopy using a UV probe laser source that combines the nonlinear effects of \b{eta}-BaB2O4 and KBe2BO3F2 optical crystals. The photon energy of the probe laser can be switched between 6.0 and 7.2 eV, with the flexibility to operate each photon energy setting under two distinct resolution configurations. Under the fully optimized energy resolution configuration, we achieve an energy resolution of 8.5 meV at 6.0 eV and 10 meV at 7.2 eV. Alternatively, switching to the other configuration enhances temporal resolution, yielding a temporal resolution of 72 fs for 6.0 eV and 185 fs for 7.2 eV. We validated the performance and reliability of our system by applying it to measuring two typical materials: the topological insulator MnBi2Te4 and excitonic insulator candidate Ta2NiSe5. △ Less

Submitted 12 October, 2023; originally announced October 2023.

Linear attention is (maybe) all you need (to understand transformer optimization)

Authors: Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jadbabaie, Suvrit Sra

Abstract: Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics. We make progress towards understanding the subtleties of training Transformers by carefully studying a simple yet canonical linearized shallow Transformer model. Specifically, we train linear Transformers to solve regression tasks, inspired by J.~von Oswald et al.~(ICML 2023), and… ▽ More Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics. We make progress towards understanding the subtleties of training Transformers by carefully studying a simple yet canonical linearized shallow Transformer model. Specifically, we train linear Transformers to solve regression tasks, inspired by J.~von Oswald et al.~(ICML 2023), and K.~Ahn et al.~(NeurIPS 2023). Most importantly, we observe that our proposed linearized models can reproduce several prominent aspects of Transformer training dynamics. Consequently, the results obtained in this paper suggest that a simple linearized Transformer model could actually be a valuable, realistic abstraction for understanding Transformer optimization. △ Less

Submitted 13 March, 2024; v1 submitted 2 October, 2023; originally announced October 2023.

Comments: Published at ICLR 2024

Error Localization for Sequential Effect Systems (Extended Version)

Authors: Colin S. Gordon, Chaewon Yun

Abstract: We describe a new concrete approach to giving predictable error locations for sequential (flow-sensitive) effect systems. Prior implementations of sequential effect systems rely on either computing a bottom-up effect and comparing it to a declaration (e.g., method annotation) or leaning on constraint-based type inference. These approaches do not necessarily report program locations that precisely… ▽ More We describe a new concrete approach to giving predictable error locations for sequential (flow-sensitive) effect systems. Prior implementations of sequential effect systems rely on either computing a bottom-up effect and comparing it to a declaration (e.g., method annotation) or leaning on constraint-based type inference. These approaches do not necessarily report program locations that precisely indicate where a program may "go wrong" at runtime. Instead of relying on constraint solving, we draw on the notion of a residual from literature on ordered algebraic structures. Applying these to effect quantales (a large class of sequential effect systems) yields an implementation approach which accepts exactly the same program as an original effect quantale, but for effect-incorrect programs is guaranteed to fail type-checking with predictable error locations tied to evaluation order. We have implemented this idea in a generic effect system implementation framework for Java, and report on experiences applying effect systems from the literature and novel effect systems to Java programs. We find that the reported error locations with our technique are significantly closer to the program points that lead to failed effect checks. △ Less

Submitted 28 July, 2023; originally announced July 2023.

Comments: Extended report of upcoming Static Analysis Symposium 2023 paper

PGL orbits in tree varieties

Authors: Izzet Coskun, Demir Eken, Chris Yun

Abstract: In this paper, we introduce tree varieties as a natural generalization of products of partial flag varieties. We study orbits of the PGL action on tree varieties. We characterize tree varieties with finitely many PGL orbits, generalizing a celebrated theorem of Magyar, Weyman and Zelevinsky. We give criteria that guarantee that a tree variety has a dense PGL orbit and provide many examples of tree… ▽ More In this paper, we introduce tree varieties as a natural generalization of products of partial flag varieties. We study orbits of the PGL action on tree varieties. We characterize tree varieties with finitely many PGL orbits, generalizing a celebrated theorem of Magyar, Weyman and Zelevinsky. We give criteria that guarantee that a tree variety has a dense PGL orbit and provide many examples of tree varieties that do not have dense PGL orbits. We show that a triple of two-step flag varieties $F(k_1, k_2; n)^3$ has a dense PGL orbit if and only if $k_1 + k_2 \not= n$. △ Less

Submitted 18 July, 2023; originally announced July 2023.

Comments: 25 pages

MSC Class: Primary: 14L30; 14M15; 14M17. Secondary: 14L35; 51N30

Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory

Authors: Minhak Song, Chulhee Yun

Abstract: Cohen et al. (2021) empirically study the evolution of the largest eigenvalue of the loss Hessian, also known as sharpness, along the gradient descent (GD) trajectory and observe the Edge of Stability (EoS) phenomenon. The sharpness increases at the early phase of training (referred to as progressive sharpening), and eventually saturates close to the threshold of $2 / \text{(step size)}$. In this… ▽ More Cohen et al. (2021) empirically study the evolution of the largest eigenvalue of the loss Hessian, also known as sharpness, along the gradient descent (GD) trajectory and observe the Edge of Stability (EoS) phenomenon. The sharpness increases at the early phase of training (referred to as progressive sharpening), and eventually saturates close to the threshold of $2 / \text{(step size)}$. In this paper, we start by demonstrating through empirical studies that when the EoS phenomenon occurs, different GD trajectories (after a proper reparameterization) align on a specific bifurcation diagram independent of initialization. We then rigorously prove this trajectory alignment phenomenon for a two-layer fully-connected linear network and a single-neuron nonlinear network trained with a single data point. Our trajectory alignment analysis establishes both progressive sharpening and EoS phenomena, encompassing and extending recent findings in the literature. △ Less

Submitted 26 October, 2023; v1 submitted 9 July, 2023; originally announced July 2023.

Comments: NeurIPS 2023 camera-ready; 51 pages

A Serre spectral sequence for the moduli space of tropical curves

Authors: Christin Bibby, Melody Chan, Nir Gadish, Claudia He Yun

Abstract: We construct, for all $g\geq 2$ and $n\geq 0$, a spectral sequence of rational $S_n$-representations which computes the $S_n$-equivariant reduced rational cohomology of the tropical moduli spaces of curves $Δ_{g,n}$ in terms of compactly supported cohomology groups of configuration spaces of $n$ points on graphs of genus $g$. Using the canonical $S_n$-equivariant isomorphisms… ▽ More We construct, for all $g\geq 2$ and $n\geq 0$, a spectral sequence of rational $S_n$-representations which computes the $S_n$-equivariant reduced rational cohomology of the tropical moduli spaces of curves $Δ_{g,n}$ in terms of compactly supported cohomology groups of configuration spaces of $n$ points on graphs of genus $g$. Using the canonical $S_n$-equivariant isomorphisms $\widetilde{H}^{i-1}(Δ_{g,n};\mathbb{Q}) \cong W_0 H^i_c(\mathcal{M}_{g,n};\mathbb{Q})$, we calculate the weight $0$, compactly supported rational cohomology of the moduli spaces $\mathcal{M}_{g,n}$ in the range $g=3$ and $n\leq 9$, with partial computations available for $n\leq 13$. △ Less

Submitted 15 April, 2024; v1 submitted 4 July, 2023; originally announced July 2023.

Comments: 24 pages plus appendix

MSC Class: 14H10; 14Q05; 14T20; 55N30; 55R80; 55T10

Efficient current-induced spin torques and field-free magnetization switching in a room-temperature van der Waals magnet

Authors: Chao Yun, Haoran Guo, Zhongchong Lin, Licong Peng, Zhongyu Liang, Miao Meng, Biao Zhang, Zijing Zhao, Leran Wang, Yifei Ma, Yajing Liu, Weiwei Li, Shuai Ning, Yanglong Hou, Jinbo Yang, Zhaochu Luo

Abstract: The discovery of magnetism in van der Waals (vdW) materials has established unique building blocks for the research of emergent spintronic phenomena. In particular, owing to their intrinsically clean surface without dangling bonds, the vdW magnets hold the potential to construct a superior interface that allows for efficient electrical manipulation of magnetism. Despite several attempts in this di… ▽ More The discovery of magnetism in van der Waals (vdW) materials has established unique building blocks for the research of emergent spintronic phenomena. In particular, owing to their intrinsically clean surface without dangling bonds, the vdW magnets hold the potential to construct a superior interface that allows for efficient electrical manipulation of magnetism. Despite several attempts in this direction, it usually requires a cryogenic condition and the assistance of external magnetic fields, which is detrimental to the real application. Here, we fabricate heterostructures based on Fe3GaTe2 flakes that possess room-temperature ferromagnetism with excellent perpendicular magnetic anisotropy. The current-driven non-reciprocal modulation of coercive fields reveals a high spin-torque efficiency in the Fe3GaTe2/Pt heterostructures, which further leads to a full magnetization switching by current. Moreover, we demonstrate the field-free magnetization switching resulting from out-of-plane polarized spin currents by asymmetric geometry design. Our work could expedite the development of efficient vdW spintronic logic, memory and neuromorphic computing devices. △ Less

Submitted 3 July, 2023; originally announced July 2023.

Cardiac CT perfusion imaging of pericoronary adipose tissue (PCAT) highlights potential confounds in coronary CTA

Authors: Hao Wu, Yingnan Song, Ammar Hoori, Ananya Subramaniam, Juhwan Lee, Justin Kim, Tao Hu, Sadeer Al-Kindi, Wei-Ming Huang, Chun-Ho Yun, Chung-Lieh Hung, Sanjay Rajagopalan, David L. Wilson

Abstract: Features of pericoronary adipose tissue (PCAT) assessed from coronary computed tomography angiography (CCTA) are associated with inflammation and cardiovascular risk. As PCAT is vascularly connected with coronary vasculature, the presence of iodine is a potential confounding factor on PCAT HU and textures that has not been adequately investigated. Use dynamic cardiac CT perfusion (CCTP) to inform… ▽ More Features of pericoronary adipose tissue (PCAT) assessed from coronary computed tomography angiography (CCTA) are associated with inflammation and cardiovascular risk. As PCAT is vascularly connected with coronary vasculature, the presence of iodine is a potential confounding factor on PCAT HU and textures that has not been adequately investigated. Use dynamic cardiac CT perfusion (CCTP) to inform contrast determinants of PCAT assessment. From CCTP, we analyzed HU dynamics of territory-specific PCAT, myocardium, and other adipose depots in patients with coronary artery disease. HU, blood flow, and radiomics were assessed over time. Changes from peak aorta time, Pa, chosen to model the time of CCTA, were obtained. HU in PCAT increased more than in other adipose depots. The estimated blood flow in PCAT was ~23% of that in the contiguous myocardium. Comparing PCAT distal and proximal to a significant stenosis, we found less enhancement and longer time-to-peak distally. Two-second offsets [before, after] Pa resulted in [ 4-HU, 3-HU] differences in PCAT. Due to changes in HU, the apparent PCAT volume reduced ~15% from the first scan (P1) to Pa using a conventional fat window. Comparing radiomic features over time, 78% of features changed >10% relative to P1. CCTP elucidates blood flow in PCAT and enables analysis of PCAT features over time. PCAT assessments (HU, apparent volume, and radiomics) are sensitive to acquisition timing and the presence of obstructive stenosis, which may confound the interpretation of PCAT in CCTA images. Data normalization may be in order. △ Less

Submitted 27 June, 2023; originally announced June 2023.

Comments: 13 pages, 8 figures

Positive del Pezzo Geometry

Authors: Nick Early, Alheydis Geiger, Marta Panizzut, Bernd Sturmfels, Claudia He Yun

Abstract: Real, complex, and tropical algebraic geometry join forces in a new branch of mathematical physics called positive geometry. We develop the positive geometry of del Pezzo surfaces and their moduli spaces, viewed as very affine varieties. Their connected components are derived from polyhedral spaces with Weyl group symmetries. We study their canonical forms and scattering amplitudes, and we solve t… ▽ More Real, complex, and tropical algebraic geometry join forces in a new branch of mathematical physics called positive geometry. We develop the positive geometry of del Pezzo surfaces and their moduli spaces, viewed as very affine varieties. Their connected components are derived from polyhedral spaces with Weyl group symmetries. We study their canonical forms and scattering amplitudes, and we solve the likelihood equations. △ Less

Submitted 23 June, 2023; originally announced June 2023.

Comments: 34 pages, 4 figures

PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning

Authors: Hojoon Lee, Hanseul Cho, Hyunseung Kim, Daehoon Gwak, Joonkee Kim, Jaegul Choo, Se-Young Yun, Chulhee Yun

Abstract: In Reinforcement Learning (RL), enhancing sample efficiency is crucial, particularly in scenarios when data acquisition is costly and risky. In principle, off-policy RL algorithms can improve sample efficiency by allowing multiple updates per environment interaction. However, these multiple updates often lead the model to overfit to earlier interactions, which is referred to as the loss of plastic… ▽ More In Reinforcement Learning (RL), enhancing sample efficiency is crucial, particularly in scenarios when data acquisition is costly and risky. In principle, off-policy RL algorithms can improve sample efficiency by allowing multiple updates per environment interaction. However, these multiple updates often lead the model to overfit to earlier interactions, which is referred to as the loss of plasticity. Our study investigates the underlying causes of this phenomenon by dividing plasticity into two aspects. Input plasticity, which denotes the model's adaptability to changing input data, and label plasticity, which denotes the model's adaptability to evolving input-output relationships. Synthetic experiments on the CIFAR-10 dataset reveal that finding smoother minima of loss landscape enhances input plasticity, whereas refined gradient propagation improves label plasticity. Leveraging these findings, we introduce the PLASTIC algorithm, which harmoniously combines techniques to address both concerns. With minimal architectural modifications, PLASTIC achieves competitive performance on benchmarks including Atari-100k and Deepmind Control Suite. This result emphasizes the importance of preserving the model's plasticity to elevate the sample efficiency in RL. The code is available at https://github.com/dojeon-ai/plastic. △ Less

Submitted 8 December, 2023; v1 submitted 19 June, 2023; originally announced June 2023.

Comments: 26 pages, 6 figures, accepted to NeurIPS 2023

Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima

Authors: Dongkuk Si, Chulhee Yun

Abstract: Sharpness-Aware Minimization (SAM) is an optimizer that takes a descent step based on the gradient at a perturbation $y_t = x_t + ρ\frac{\nabla f(x_t)}{\lVert \nabla f(x_t) \rVert}$ of the current point $x_t$. Existing studies prove convergence of SAM for smooth functions, but they do so by assuming decaying perturbation size $ρ$ and/or no gradient normalization in $y_t$, which is detached from pr… ▽ More Sharpness-Aware Minimization (SAM) is an optimizer that takes a descent step based on the gradient at a perturbation $y_t = x_t + ρ\frac{\nabla f(x_t)}{\lVert \nabla f(x_t) \rVert}$ of the current point $x_t$. Existing studies prove convergence of SAM for smooth functions, but they do so by assuming decaying perturbation size $ρ$ and/or no gradient normalization in $y_t$, which is detached from practice. To address this gap, we study deterministic/stochastic versions of SAM with practical configurations (i.e., constant $ρ$ and gradient normalization in $y_t$) and explore their convergence properties on smooth functions with (non)convexity assumptions. Perhaps surprisingly, in many scenarios, we find out that SAM has limited capability to converge to global minima or stationary points. For smooth strongly convex functions, we show that while deterministic SAM enjoys tight global convergence rates of $\tilde Θ(\frac{1}{T^2})$, the convergence bound of stochastic SAM suffers an inevitable additive term $O(ρ^2)$, indicating convergence only up to neighborhoods of optima. In fact, such $O(ρ^2)$ factors arise for stochastic SAM in all the settings we consider, and also for deterministic SAM in nonconvex cases; importantly, we prove by examples that such terms are unavoidable. Our results highlight vastly different characteristics of SAM with vs. without decaying perturbation size or gradient normalization, and suggest that the intuitions gained from one version may not apply to the other. △ Less

Submitted 27 October, 2023; v1 submitted 16 June, 2023; originally announced June 2023.

Comments: 39 pages. v3 NeurIPS 2023 camera ready version

Provable Benefit of Mixup for Finding Optimal Decision Boundaries

Authors: Junsoo Oh, Chulhee Yun

Abstract: We investigate how pair-wise data augmentation techniques like Mixup affect the sample complexity of finding optimal decision boundaries in a binary linear classification problem. For a family of data distributions with a separability constant $κ$, we analyze how well the optimal classifier in terms of training loss aligns with the optimal one in test accuracy (i.e., Bayes optimal classifier). For… ▽ More We investigate how pair-wise data augmentation techniques like Mixup affect the sample complexity of finding optimal decision boundaries in a binary linear classification problem. For a family of data distributions with a separability constant $κ$, we analyze how well the optimal classifier in terms of training loss aligns with the optimal one in test accuracy (i.e., Bayes optimal classifier). For vanilla training without augmentation, we uncover an interesting phenomenon named the curse of separability. As we increase $κ$ to make the data distribution more separable, the sample complexity of vanilla training increases exponentially in $κ$; perhaps surprisingly, the task of finding optimal decision boundaries becomes harder for more separable distributions. For Mixup training, we show that Mixup mitigates this problem by significantly reducing the sample complexity. To this end, we develop new concentration results applicable to $n^2$ pair-wise augmented data points constructed from $n$ independent data, by carefully dealing with dependencies between overlapping pairs. Lastly, we study other masking-based Mixup-style techniques and show that they can distort the training loss and make its minimizer converge to a suboptimal classifier in terms of test accuracy. △ Less

Submitted 5 June, 2023; v1 submitted 31 May, 2023; originally announced June 2023.

Comments: ICML 2023 camera-ready version; 48 pages

Electrically programmable magnetic coupling in an Ising network exploiting solid-state ionic gating

Authors: Chao Yun, Zhongyu Liang, Aleš Hrabec, Zhentao Liu, Mantao Huang, Leran Wang, Yifei Xiao, Yikun Fang, Wei Li, Wenyun Yang, Yanglong Hou, Jinbo Yang, Laura J. Heyderman, Pietro Gambardella, Zhaochu Luo

Abstract: Two-dimensional arrays of magnetically coupled nanomagnets provide a mesoscopic platform for exploring collective phenomena as well as realizing a broad range of spintronic devices. In particular, the magnetic coupling plays a critical role in determining the nature of the cooperative behaviour and providing new functionalities in nanomagnet-based devices. Here, we create coupled Ising-like nanoma… ▽ More Two-dimensional arrays of magnetically coupled nanomagnets provide a mesoscopic platform for exploring collective phenomena as well as realizing a broad range of spintronic devices. In particular, the magnetic coupling plays a critical role in determining the nature of the cooperative behaviour and providing new functionalities in nanomagnet-based devices. Here, we create coupled Ising-like nanomagnets in which the coupling between adjacent nanomagnetic regions can be reversibly converted between parallel and antiparallel through solid-state ionic gating. This is achieved with the voltage-control of magnetic anisotropies in a nanosized region where the symmetric exchange interaction favours parallel alignment and the antisymmetric exchange interaction, namely the Dzyaloshinskii-Moriya interaction, favours antiparallel alignment. Applying this concept to a two-dimensional lattice, we demonstrate a voltage-controlled phase transition in artificial spin ices. Furthermore, we achieve an addressable control of the individual couplings and realize an electrically programmable Ising network, which opens up new avenues to design nanomagnet-based logic devices and neuromorphic computers △ Less

Submitted 5 May, 2023; originally announced May 2023.

Journal ref: Nat Commun 14, 6367 (2023)

Some thoughts and experiments on Bergman's compact amalgamation problem

Authors: Michael Joswig, Mario Kummer, Andreas Thom, Claudia He Yun

Abstract: We study the question whether copies of $S^1$ in $\mathrm{SU}(3)$ can be amalgamated in a compact group. This is the simplest instance of a fundamental open problem in the theory of compact groups raised by George Bergman in 1987. Considerable computational experiments suggest that the answer is positive in this case. We obtain a positive answer for a relaxed problem using theoretical consideratio… ▽ More We study the question whether copies of $S^1$ in $\mathrm{SU}(3)$ can be amalgamated in a compact group. This is the simplest instance of a fundamental open problem in the theory of compact groups raised by George Bergman in 1987. Considerable computational experiments suggest that the answer is positive in this case. We obtain a positive answer for a relaxed problem using theoretical considerations. △ Less

Submitted 13 July, 2023; v1 submitted 17 April, 2023; originally announced April 2023.

Comments: 15 pages, 2 figures, 3 tables; update contains minor changes that address referee comments

MSC Class: 22C05; 18B99; 90-05; 90C90

Tighter Lower Bounds for Shuffling SGD: Random Permutations and Beyond

Authors: Jaeyoung Cha, Jaewook Lee, Chulhee Yun

Abstract: We study convergence lower bounds of without-replacement stochastic gradient descent (SGD) for solving smooth (strongly-)convex finite-sum minimization problems. Unlike most existing results focusing on final iterate lower bounds in terms of the number of components $n$ and the number of epochs $K$, we seek bounds for arbitrary weighted average iterates that are tight in all factors including the… ▽ More We study convergence lower bounds of without-replacement stochastic gradient descent (SGD) for solving smooth (strongly-)convex finite-sum minimization problems. Unlike most existing results focusing on final iterate lower bounds in terms of the number of components $n$ and the number of epochs $K$, we seek bounds for arbitrary weighted average iterates that are tight in all factors including the condition number $κ$. For SGD with Random Reshuffling, we present lower bounds that have tighter $κ$ dependencies than existing bounds. Our results are the first to perfectly close the gap between lower and upper bounds for weighted average iterates in both strongly-convex and convex cases. We also prove weighted average iterate lower bounds for arbitrary permutation-based SGD, which apply to all variants that carefully choose the best permutation. Our bounds improve the existing bounds in factors of $n$ and $κ$ and thereby match the upper bounds shown for a recently proposed algorithm called GraB. △ Less

Submitted 9 June, 2023; v1 submitted 13 March, 2023; originally announced March 2023.

Comments: 58 pages

On the Training Instability of Shuffling SGD with Batch Normalization

Authors: David X. Wu, Chulhee Yun, Suvrit Sra

Abstract: We uncover how SGD interacts with batch normalization and can exhibit undesirable training dynamics such as divergence. More precisely, we study how Single Shuffle (SS) and Random Reshuffle (RR) -- two widely used variants of SGD -- interact surprisingly differently in the presence of batch normalization: RR leads to much more stable evolution of training loss than SS. As a concrete example, for r… ▽ More We uncover how SGD interacts with batch normalization and can exhibit undesirable training dynamics such as divergence. More precisely, we study how Single Shuffle (SS) and Random Reshuffle (RR) -- two widely used variants of SGD -- interact surprisingly differently in the presence of batch normalization: RR leads to much more stable evolution of training loss than SS. As a concrete example, for regression using a linear network with batch normalization, we prove that SS and RR converge to distinct global optima that are "distorted" away from gradient descent. Thereafter, for classification we characterize conditions under which training divergence for SS and RR can, and cannot occur. We present explicit constructions to show how SS leads to distorted optima in regression and divergence for classification, whereas RR avoids both distortion and divergence. We validate our results by confirming them empirically in realistic settings, and conclude that the separation between SS and RR used with batch normalization is relevant in practice. △ Less

Submitted 14 August, 2023; v1 submitted 23 February, 2023; originally announced February 2023.

Comments: ICML 2023 camera-ready version, added references; 75 pages

Oscillating cosmic evolution and constraints on big bang nucleosynthesis in the extended Starobinsky model

Authors: Jubin Park, Chae-min Yun, Myung-Ki Cheoun, Dukjae Jang

Abstract: We investigate the cosmic evolutions in the extended Starobinsky model (eSM) obtained by adding one $R^{ab}R_{ab}$ term to the Starobinsky model. We discuss the possibility of various cosmic evolutions with a special focus on the radiation-dominated era (RDE). Using simple assumptions, a second-order non-linear differential equation describing the various cosmic evolutions in the eSM is introduced… ▽ More We investigate the cosmic evolutions in the extended Starobinsky model (eSM) obtained by adding one $R^{ab}R_{ab}$ term to the Starobinsky model. We discuss the possibility of various cosmic evolutions with a special focus on the radiation-dominated era (RDE). Using simple assumptions, a second-order non-linear differential equation describing the various cosmic evolutions in the eSM is introduced. By solving this non-linear equation numerically, we show that the various cosmic evolutions, such as the standard cosmic evolution ($a \propto t^{1/2}$) and a unique oscillating cosmic evolution, are feasible due to the effects of higher-order terms introduced beyond Einstein's gravity. Furthermore, we consider big bang nucleosynthesis (BBN), which is the most important observational result in the RDE, to constrain the free parameters of the eSM. The primordial abundances of the light elements, such as $^{4}$He, D, $^{3}$He, $^{7}$Li, and $^{6}$Li by the cosmic evolutions are compared with the most recent observational data. It turns out that most non-standard cosmic evolutions can not easily satisfy these BBN constraints, but a free parameter of the viable models with the oscillating cosmic evolution is shown to have an upper limit by the constraints. In particular, we find that the free parameter is most sensitive to deuterium and $^4$He abundances, which are being precisely measured among other elements. Therefore, more accurate measurements in the near future may enable us to distinguish the eSM from the standard model as well as other models. △ Less

Submitted 1 May, 2023; v1 submitted 22 December, 2022; originally announced December 2022.

Journal ref: Journal of Cosmology and Astroparticle Physics, Volume 2023, May 2023

SGDA with shuffling: faster convergence for nonconvex-PŁ minimax optimization

Authors: Hanseul Cho, Chulhee Yun

Abstract: Stochastic gradient descent-ascent (SGDA) is one of the main workhorses for solving finite-sum minimax optimization problems. Most practical implementations of SGDA randomly reshuffle components and sequentially use them (i.e., without-replacement sampling); however, there are few theoretical results on this approach for minimax algorithms, especially outside the easier-to-analyze (strongly-)monot… ▽ More Stochastic gradient descent-ascent (SGDA) is one of the main workhorses for solving finite-sum minimax optimization problems. Most practical implementations of SGDA randomly reshuffle components and sequentially use them (i.e., without-replacement sampling); however, there are few theoretical results on this approach for minimax algorithms, especially outside the easier-to-analyze (strongly-)monotone setups. To narrow this gap, we study the convergence bounds of SGDA with random reshuffling (SGDA-RR) for smooth nonconvex-nonconcave objectives with Polyak-Łojasiewicz (PŁ) geometry. We analyze both simultaneous and alternating SGDA-RR for nonconvex-PŁ and primal-PŁ-PŁ objectives, and obtain convergence rates faster than with-replacement SGDA. Our rates extend to mini-batch SGDA-RR, recovering known rates for full-batch gradient descent-ascent (GDA). Lastly, we present a comprehensive lower bound for GDA with an arbitrary step-size ratio, which matches the full-batch upper bound for the primal-PŁ-PŁ case. △ Less

Submitted 20 February, 2023; v1 submitted 12 October, 2022; originally announced October 2022.

Comments: ICLR 2023 camera-ready version; 46 pages

Discrete Morse theory for symmetric Delta-complexes

Authors: Claudia He Yun

Abstract: We generalize Forman's discrete Morse theory to the context of symmetric $Δ$-complexes. As an application, we prove that the coloop subcomplex of the link of the origin $LA^{\mathrm{trop},\mathrm{P}}_g$ in the moduli space of principally polarized tropical abelian varieties of dimension $g$ with respect to the perfect cone decomposition is contractible. We generalize Forman's discrete Morse theory to the context of symmetric $Δ$-complexes. As an application, we prove that the coloop subcomplex of the link of the origin $LA^{\mathrm{trop},\mathrm{P}}_g$ in the moduli space of principally polarized tropical abelian varieties of dimension $g$ with respect to the perfect cone decomposition is contractible. △ Less

Submitted 2 September, 2022; originally announced September 2022.

Comments: 16 pages, 5 figures

MSC Class: 57Q70; 14T15

Simulated Lorentz Force Detuning Compensation With A Double Lever Tuner On A Dressed ILC/1.3 GHZ Cavity At Room Temperature

Authors: C. Contreras-Martinez, Y. Pischalnikov, J. C. Yun

Abstract: Pulsed SRF linacs with high accelerating gradients experience large frequency shifts caused by Lorentz force detuning (LFD). A piezoelectric actuator with a resonance control algorithm can maintain the cavity frequency at the nominal level, thus reducing the RF power. This study uses a double lever tuner with a piezoelectric actuator for compensation and another piezoelectric actuator to simulate… ▽ More Pulsed SRF linacs with high accelerating gradients experience large frequency shifts caused by Lorentz force detuning (LFD). A piezoelectric actuator with a resonance control algorithm can maintain the cavity frequency at the nominal level, thus reducing the RF power. This study uses a double lever tuner with a piezoelectric actuator for compensation and another piezoelectric actuator to simulate the effects of the Lorentz force pulse. A double lever tuner has an advantage by increasing the stiffness of the cavity-tuner system, thus reducing the impact of LFD. The tests are conducted at room temperature and with a dressed 1.3 GHz 9-cell cavity. △ Less

Submitted 12 August, 2022; originally announced August 2022.

Report number: FERMILAB-CONF-22-581-TD

Accelerated Lifetime Test of The SRF Dressed Cavity/Tuner System for LCLS II HE Project

Authors: Y. Pischalnikov, T. Arkan, C. Contreras-Martinez, B. Hartsell, J. Kaluzny, R. Pilipenko, J. C. Yun, W. Lahmadi

Abstract: The off-frequency detune method is being considered for application in the LCLS-II-HE superconducting linac to produce multi-energy electron beams for supporting multiple undulator lines simultaneously [1]. Design of the tuner has been changed to deliver roughly 3 times larger frequency tuning range. Working requirements for off-frequency operation (OFO) state that cavities be tuned at least twice… ▽ More The off-frequency detune method is being considered for application in the LCLS-II-HE superconducting linac to produce multi-energy electron beams for supporting multiple undulator lines simultaneously [1]. Design of the tuner has been changed to deliver roughly 3 times larger frequency tuning range. Working requirements for off-frequency operation (OFO) state that cavities be tuned at least twice a month. This specification requires the in-crease of the tuner longevity by 20 times compared with LCLS-II demands. Accelerated longevity tests of the LCLS-II HE dressed cavity with tuner were conducted at FNAL's HTS. Detail analysis of wearing and impacts on performances of the tuner's piezo and stepper motor actuators will be presented. Additionally, results of longevity testing of the dressed cavity bellow, when cooled down to 2 K and compressed by 2.6 mm for roughly 2000 cycles, will be presented. △ Less

Submitted 8 August, 2022; originally announced August 2022.

Report number: FERMILAB-CONF-22-555-TD

An exact solution of the higher-order gravity in standard radiation-dominated era

Authors: Chae-min Yun, Jubin Park, Myung-Ki Cheoun, Dukjae Jang

Abstract: We report that the standard evolution of radiation-dominated era (RDE) universe $a \propto t^{1/2}$ is a sufficient condition for solving a sixth order gravitational field equation derived from the Lagrangian containing $B R^{ab}R_{ab} + C R {R^{;c}}_{c}$ as well as a polynomial $f(R)$ for a spatially flat radiation FLRW universe. By virtue of the similarity between $R^{ab}R_{ab}$ and $R^2$ models… ▽ More We report that the standard evolution of radiation-dominated era (RDE) universe $a \propto t^{1/2}$ is a sufficient condition for solving a sixth order gravitational field equation derived from the Lagrangian containing $B R^{ab}R_{ab} + C R {R^{;c}}_{c}$ as well as a polynomial $f(R)$ for a spatially flat radiation FLRW universe. By virtue of the similarity between $R^{ab}R_{ab}$ and $R^2$ models up to the background order and of the vanishing property of ${R^{;c}}_{c}$ for $ H = 1/(2t)$, the analytical solution can be obtained from a special case to general one. This proves that the standard cosmic evolution is valid even within modified gravitational theory involving higher-order terms. An application of this background solution to the tensor-type perturbation reduces the complicated equation to the standard second order equation of gravitational wave. We discuss the possible ways to discriminate the modified gravity model on the observations such as the gravitational wave from the disturbed universe and primordial abundances. △ Less

Submitted 2 January, 2023; v1 submitted 20 July, 2022; originally announced July 2022.

Equivariant Hodge polynomials of heavy/light moduli spaces

Authors: Siddarth Kannan, Stefano Serpente, Claudia He Yun

Abstract: Let $\bar{\mathcal{M}}_{g, m|n}$ denote Hassett's moduli space of weighted pointed stable curves of genus $g$ for the heavy/light weight data $\left(1^{(m)}, 1/n^{(n)}\right)$, and let $\mathcal{M}_{g, m|n} \subset \bar{\mathcal{M}}_{g, m|n}$ be the locus parameterizing smooth, not necessarily distinctly marked curves. We give a change-of-variables formula which computes the generating function fo… ▽ More Let $\bar{\mathcal{M}}_{g, m|n}$ denote Hassett's moduli space of weighted pointed stable curves of genus $g$ for the heavy/light weight data $\left(1^{(m)}, 1/n^{(n)}\right)$, and let $\mathcal{M}_{g, m|n} \subset \bar{\mathcal{M}}_{g, m|n}$ be the locus parameterizing smooth, not necessarily distinctly marked curves. We give a change-of-variables formula which computes the generating function for $(S_m\times S_n)$-equivariant Hodge-Deligne polynomials of these spaces in terms of the generating functions for $S_{n}$-equivariant Hodge-Deligne polynomials of $\bar{\mathcal{M}}_{g,n}$ and $\mathcal{M}_{g,n}$. △ Less

Submitted 22 April, 2024; v1 submitted 6 July, 2022; originally announced July 2022.

Comments: 21 pages, 3 tables. Edits based on referee suggestions

MSC Class: 14H10

Multi-task Unscented Kalman Inversion for joint inversion of receiver function and surface wave dispersion

Authors: Wang Longlong, Liu Youshan, Chen Yun, Du nanqiao

Abstract: Based on the recently developed theory of Unscented Kalman Inversion in computational mathematics, we proposed a Bayesian joint inversion framework, i.e., Multi-task Unscented Kalman Inversion (MTUKI), and apply it to the joint inversion of receiver function (RF) and surface wave dispersion (SWD). This method can share information between different observations in a derivative-free way and provide… ▽ More Based on the recently developed theory of Unscented Kalman Inversion in computational mathematics, we proposed a Bayesian joint inversion framework, i.e., Multi-task Unscented Kalman Inversion (MTUKI), and apply it to the joint inversion of receiver function (RF) and surface wave dispersion (SWD). This method can share information between different observations in a derivative-free way and provide an efficient Gaussian approximation to the posterior distribution of model parameters (thickness and S-wave velocity in each layer of media). The theory and experiments show that our proposed framework demonstrates superior performance in terms of robustness, accuracy, and high efficiency. △ Less

Submitted 15 January, 2023; v1 submitted 6 April, 2022; originally announced April 2022.

Comments: This version is repeated with another version (arXiv:2202.09544)

Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyond

Authors: Chulhee Yun, Shashank Rajput, Suvrit Sra

Abstract: In distributed learning, local SGD (also known as federated averaging) and its simple baseline minibatch SGD are widely studied optimization methods. Most existing analyses of these methods assume independent and unbiased gradient estimates obtained via with-replacement sampling. In contrast, we study shuffling-based variants: minibatch and local Random Reshuffling, which draw stochastic gradients… ▽ More In distributed learning, local SGD (also known as federated averaging) and its simple baseline minibatch SGD are widely studied optimization methods. Most existing analyses of these methods assume independent and unbiased gradient estimates obtained via with-replacement sampling. In contrast, we study shuffling-based variants: minibatch and local Random Reshuffling, which draw stochastic gradients without replacement and are thus closer to practice. For smooth functions satisfying the Polyak-Łojasiewicz condition, we obtain convergence bounds (in the large epoch regime) which show that these shuffling-based variants converge faster than their with-replacement counterparts. Moreover, we prove matching lower bounds showing that our convergence analysis is tight. Finally, we propose an algorithmic modification called synchronized shuffling that leads to convergence rates faster than our lower bounds in near-homogeneous settings. △ Less

Submitted 23 March, 2022; v1 submitted 19 October, 2021; originally announced October 2021.

Comments: ICLR 2022 camera-ready (selected for an oral presentation); 76 pages, 3 figures

The role of viral infectivity in oncolytic virotherapy outcomes: A mathematical study

Authors: Pantea Pooladvand, Chae-Ok Yun, A-Rum Yoon, Peter S. Kim, Federico Frascoli

Abstract: A model capturing the dynamics between virus and tumour cells in the context of oncolytic virotherapy is presented and analysed. The ability of the virus to be internalised by uninfected cells is described by an infectivity parameter, which is inferred from available experimental data. The parameter is also able to describe the effects of changes in the tumour environment that affect viral uptake… ▽ More A model capturing the dynamics between virus and tumour cells in the context of oncolytic virotherapy is presented and analysed. The ability of the virus to be internalised by uninfected cells is described by an infectivity parameter, which is inferred from available experimental data. The parameter is also able to describe the effects of changes in the tumour environment that affect viral uptake from tumour cells. Results show that when a virus is inoculated inside a growing tumour, strategies for enhancing infectivity do not lead to a complete eradication of the tumour. Within typical times of experiments and treatments, we observe the onset of oscillations, which always prevent a full destruction of the tumour mass. These findings are in good agreement with available laboratory results. Further analysis shows why a fully successful therapy cannot exist for the proposed model and that care must be taken when designing and engineering viral vectors with enhanced features. In particular, bifurcation analysis reveals that creating longer lasting virus particles or using strategies for reducing infected cell lifespan can cause unexpected and unwanted surges in the overall tumour load over time. Our findings suggest that virotherapy alone seems unlikely to be effective in clinical settings unless adjuvant strategies are included. △ Less

Submitted 7 October, 2021; originally announced October 2021.

Comments: 29 pages, 13 figures, 1 table

MSC Class: 92-10

Journal ref: Mathematical Biosciences, 334: 108520 (2021)

Homology representations of compactified configurations on graphs applied to $\mathcal{M}_{2,n}$

Authors: Christin Bibby, Melody Chan, Nir Gadish, Claudia He Yun

Abstract: We obtain new calculations of the top weight rational cohomology of the moduli spaces $\mathcal{M}_{2,n}$, equivalently the rational homology of the tropical moduli spaces $Δ_{2,n}$, as a representation of $S_n$. These calculations are achieved fully for all $n\leq 10$, and partially -- for specific irreducible representations of $S_n$ -- for $n\le 22$. We also present conjectures, verified up to… ▽ More We obtain new calculations of the top weight rational cohomology of the moduli spaces $\mathcal{M}_{2,n}$, equivalently the rational homology of the tropical moduli spaces $Δ_{2,n}$, as a representation of $S_n$. These calculations are achieved fully for all $n\leq 10$, and partially -- for specific irreducible representations of $S_n$ -- for $n\le 22$. We also present conjectures, verified up to $n=22$, for the multiplicities of the irreducible representations $\mathrm{std}_n$ and $\mathrm{std}_n\otimes \mathrm{sgn}_n$. We achieve our calculations via a comparison with the homology of compactified configuration spaces of graphs. These homology groups are equipped with commuting actions of a symmetric group and the outer automorphism group of a free group. In this paper, we construct an efficient free resolution for these homology representations, from which we extract calculations on irreducible representations one at a time, simplifying the calculation of these homology representations. △ Less

Submitted 25 April, 2023; v1 submitted 7 September, 2021; originally announced September 2021.

Comments: 18 pages, minor edits

MSC Class: 05C10 (primary); 14H10; 14Q05; 14T20; 55R80; 55P65

Observation of the Orbital Rashba-Edelstein Magnetoresistance

Authors: Shilei Ding, Zhongyu Liang, Dongwook Go, Chao Yun, Mingzhu Xue, Zhou Liu, Sven Becker, Wenyun Yang, Honglin Du, Changsheng Wang, Yingchang Yang, Gerhard Jakob, Mathias Kläui, Yuriy Mokrousov, Jinbo Yang

Abstract: We report the observation of magnetoresistance (MR) originating from the orbital angular momentum transport (OAM) in a Permalloy (Py) / oxidized Cu (Cu*) heterostructure: the orbital Rashba-Edelstein magnetoresistance. The angular dependence of the MR depends on the relative angle between the induced OAM and the magnetization in a similar fashion as the spin Hall magnetoresistance (SMR). Despite t… ▽ More We report the observation of magnetoresistance (MR) originating from the orbital angular momentum transport (OAM) in a Permalloy (Py) / oxidized Cu (Cu*) heterostructure: the orbital Rashba-Edelstein magnetoresistance. The angular dependence of the MR depends on the relative angle between the induced OAM and the magnetization in a similar fashion as the spin Hall magnetoresistance (SMR). Despite the absence of elements with large spin-orbit coupling, we find a sizable MR ratio, which is in contrast to the conventional SMR which requires heavy elements. By varying the thickness of the Cu* layer, we confirm that the interface is responsible for the MR, suggesting that the orbital Rashba-Edelstein effect is responsible for the generation of the OAM. Through Py thickness-dependence studies, we find that the effective values for the spin diffusion and spin dephasing lengths of Py are significantly larger than the values measured in Py / Pt bilayers, approximately by the factor of 2 and 4, respectively. This implies that another mechanism beyond the conventional spin-based scenario is responsible for the MR observed in Py / Cu* structures originated in a sizeable transport of OAM. Our findings not only unambiguously demonstrate the current-induced torque without using any heavy element via the OAM channel but also provide an important clue towards the microscopic understanding of the role that OAM transport can play for magnetization dynamics. △ Less

Submitted 11 May, 2021; v1 submitted 10 May, 2021; originally announced May 2021.

Comments: 11 pages,3 figures

Can Single-Shuffle SGD be Better than Reshuffling SGD and GD?

Authors: Chulhee Yun, Suvrit Sra, Ali Jadbabaie

Abstract: We propose matrix norm inequalities that extend the Recht-Ré (2012) conjecture on a noncommutative AM-GM inequality by supplementing it with another inequality that accounts for single-shuffle, which is a widely used without-replacement sampling scheme that shuffles only once in the beginning and is overlooked in the Recht-Ré conjecture. Instead of general positive semidefinite matrices, we restri… ▽ More We propose matrix norm inequalities that extend the Recht-Ré (2012) conjecture on a noncommutative AM-GM inequality by supplementing it with another inequality that accounts for single-shuffle, which is a widely used without-replacement sampling scheme that shuffles only once in the beginning and is overlooked in the Recht-Ré conjecture. Instead of general positive semidefinite matrices, we restrict our attention to positive definite matrices with small enough condition numbers, which are more relevant to matrices that arise in the analysis of SGD. For such matrices, we conjecture that the means of matrix products corresponding to with- and without-replacement variants of SGD satisfy a series of spectral norm inequalities that can be summarized as: "single-shuffle SGD converges faster than random-reshuffle SGD, which is in turn faster than with-replacement SGD." We present theorems that support our conjecture by proving several special cases. △ Less

Submitted 11 March, 2021; originally announced March 2021.

Comments: 26 pages, 2 figures

Provable Memorization via Deep Neural Networks using Sub-linear Parameters

Authors: Sejun Park, Jaeho Lee, Chulhee Yun, Jinwoo Shin

Abstract: It is known that $O(N)$ parameters are sufficient for neural networks to memorize arbitrary $N$ input-label pairs. By exploiting depth, we show that $O(N^{2/3})$ parameters suffice to memorize $N$ pairs, under a mild condition on the separation of input points. In particular, deeper networks (even with width $3$) are shown to memorize more pairs than shallow networks, which also agrees with the re… ▽ More It is known that $O(N)$ parameters are sufficient for neural networks to memorize arbitrary $N$ input-label pairs. By exploiting depth, we show that $O(N^{2/3})$ parameters suffice to memorize $N$ pairs, under a mild condition on the separation of input points. In particular, deeper networks (even with width $3$) are shown to memorize more pairs than shallow networks, which also agrees with the recent line of works on the benefits of depth for function approximation. We also provide empirical results that support our theoretical findings. △ Less

Submitted 2 November, 2021; v1 submitted 26 October, 2020; originally announced October 2020.

Topology of tropical moduli spaces of weighted stable curves in higher genus

Authors: Siddarth Kannan, Shiyue Li, Stefano Serpente, Claudia He Yun

Abstract: Given integers $g \geq 0$, $n \geq 1$, and a vector $w \in (\mathbb{Q} \cap (0, 1])^n$ such that ${2g - 2 + \sum w_i > 0}$, we study the topology of the moduli space $Δ_{g, w}$ of $w$-stable tropical curves of genus $g$ with volume 1. The space $Δ_{g, w}$ is the dual complex of the divisor of singular curves in Hassett's moduli space of $w$-stable genus $g$ curves $\overline{\mathcal{M}}_{g, w}$.… ▽ More Given integers $g \geq 0$, $n \geq 1$, and a vector $w \in (\mathbb{Q} \cap (0, 1])^n$ such that ${2g - 2 + \sum w_i > 0}$, we study the topology of the moduli space $Δ_{g, w}$ of $w$-stable tropical curves of genus $g$ with volume 1. The space $Δ_{g, w}$ is the dual complex of the divisor of singular curves in Hassett's moduli space of $w$-stable genus $g$ curves $\overline{\mathcal{M}}_{g, w}$. When $g \geq 1$, we show that $Δ_{g, w}$ is simply connected for all values of $w$. We also give a formula for the Euler characteristic of $Δ_{g, w}$ in terms of the combinatorics of $w$. △ Less

Submitted 15 March, 2022; v1 submitted 22 October, 2020; originally announced October 2020.

Comments: 14 pages; 1 figure; final version accepted at Advances in Geometry

MSC Class: 14T05

DML-GANR: Deep Metric Learning With Generative Adversarial Network Regularization for High Spatial Resolution Remote Sensing Image Retrieval

Authors: Yun Cao, Yuebin Wang, Junhuan Peng, Liqiang Zhang, Linlin Xu, Kai Yan, Lihua Li

Abstract: With a small number of labeled samples for training, it can save considerable manpower and material resources, especially when the amount of high spatial resolution remote sensing images (HSR-RSIs) increases considerably. However, many deep models face the problem of overfitting when using a small number of labeled samples. This might degrade HSRRSI retrieval accuracy. Aiming at obtaining more acc… ▽ More With a small number of labeled samples for training, it can save considerable manpower and material resources, especially when the amount of high spatial resolution remote sensing images (HSR-RSIs) increases considerably. However, many deep models face the problem of overfitting when using a small number of labeled samples. This might degrade HSRRSI retrieval accuracy. Aiming at obtaining more accurate HSR-RSI retrieval performance with small training samples, we develop a deep metric learning approach with generative adversarial network regularization (DML-GANR) for HSR-RSI retrieval. The DML-GANR starts from a high-level feature extraction (HFE) to extract high-level features, which includes convolutional layers and fully connected (FC) layers. Each of the FC layers is constructed by deep metric learning (DML) to maximize the interclass variations and minimize the intraclass variations. The generative adversarial network (GAN) is adopted to mitigate the overfitting problem and validate the qualities of extracted high-level features. DML-GANR is optimized through a customized approach, and the optimal parameters are obtained. The experimental results on the three data sets demonstrate the superior performance of DML-GANR over state-of-the-art techniques in HSR-RSI retrieval. △ Less

Submitted 6 October, 2020; originally announced October 2020.

Comments: 17 pages

SLCRF: Subspace Learning with Conditional Random Field for Hyperspectral Image Classification

Authors: Yun Cao, Jie Mei, Yuebin Wang, Liqiang Zhang, Junhuan Peng, Bing Zhang, Lihua Li, Yibo Zheng

Abstract: Subspace learning (SL) plays an important role in hyperspectral image (HSI) classification, since it can provide an effective solution to reduce the redundant information in the image pixels of HSIs. Previous works about SL aim to improve the accuracy of HSI recognition. Using a large number of labeled samples, related methods can train the parameters of the proposed solutions to obtain better rep… ▽ More Subspace learning (SL) plays an important role in hyperspectral image (HSI) classification, since it can provide an effective solution to reduce the redundant information in the image pixels of HSIs. Previous works about SL aim to improve the accuracy of HSI recognition. Using a large number of labeled samples, related methods can train the parameters of the proposed solutions to obtain better representations of HSI pixels. However, the data instances may not be sufficient enough to learn a precise model for HSI classification in real applications. Moreover, it is well-known that it takes much time, labor and human expertise to label HSI images. To avoid the aforementioned problems, a novel SL method that includes the probability assumption called subspace learning with conditional random field (SLCRF) is developed. In SLCRF, first, the 3D convolutional autoencoder (3DCAE) is introduced to remove the redundant information in HSI pixels. In addition, the relationships are also constructed using the spectral-spatial information among the adjacent pixels. Then, the conditional random field (CRF) framework can be constructed and further embedded into the HSI SL procedure with the semi-supervised approach. Through the linearized alternating direction method termed LADMAP, the objective function of SLCRF is optimized using a defined iterative algorithm. The proposed method is comprehensively evaluated using the challenging public HSI datasets. We can achieve stateof-the-art performance using these HSI sets. △ Less

Submitted 6 October, 2020; originally announced October 2020.

Comments: 13 pages, 6 figures

A Unifying View on Implicit Bias in Training Linear Neural Networks

Authors: Chulhee Yun, Shankar Krishnan, Hossein Mobahi

Abstract: We study the implicit bias of gradient flow (i.e., gradient descent with infinitesimal step size) on linear neural network training. We propose a tensor formulation of neural networks that includes fully-connected, diagonal, and convolutional networks as special cases, and investigate the linear version of the formulation called linear tensor networks. With this formulation, we can characterize th… ▽ More We study the implicit bias of gradient flow (i.e., gradient descent with infinitesimal step size) on linear neural network training. We propose a tensor formulation of neural networks that includes fully-connected, diagonal, and convolutional networks as special cases, and investigate the linear version of the formulation called linear tensor networks. With this formulation, we can characterize the convergence direction of the network parameters as singular vectors of a tensor defined by the network. For $L$-layer linear tensor networks that are orthogonally decomposable, we show that gradient flow on separable classification finds a stationary point of the $\ell_{2/L}$ max-margin problem in a "transformed" input space defined by the network. For underdetermined regression, we prove that gradient flow finds a global minimum which minimizes a norm-like function that interpolates between weighted $\ell_1$ and $\ell_2$ norms in the transformed input space. Our theorems subsume existing results in the literature while removing standard convergence assumptions. We also provide experiments that corroborate our analysis. △ Less

Submitted 10 September, 2021; v1 submitted 6 October, 2020; originally announced October 2020.

Comments: 38 pages, 7 figures. Revision after ICLR 2021 camera-ready version. Figure 2 newly added, theorem statements revised, including correction of Theorem 2

The $S_n$-equivariant rational homology of the tropical moduli spaces $Δ_{2,n}$

Authors: Claudia He Yun

Abstract: We compute the $S_n$-equivariant rational homology of the tropical moduli spaces $Δ_{2,n}$ for $n\leq 8$ using a cellular chain complex for symmetric $Δ$-complexes in Sage. We compute the $S_n$-equivariant rational homology of the tropical moduli spaces $Δ_{2,n}$ for $n\leq 8$ using a cellular chain complex for symmetric $Δ$-complexes in Sage. △ Less

Submitted 10 August, 2020; originally announced August 2020.

Comments: 17 pages, 2 figures, 6 tables

MSC Class: 14T10 (Primary); 14Q05 (Secondary)

Existence and convergence theorems for monotone generalized alpa-nonexpansive mappings in uniformly convex partially ordered hyperbolic metric spaces and its application

Authors: Chang Il Rim, Jong Gyong Kim, Chol-Hui Yun

Abstract: In this paper, we generalize the existence result in [14] and prove convergence theorems of the iterative scheme in [12, 16] for monotone generalized alpa-nonexpansive mappings in uniformly convex partially ordered hyperbolic metric spaces. And we also give a numerical example to show that this scheme converges faster than the scheme in [14] and apply the result to the integral equation. In this paper, we generalize the existence result in [14] and prove convergence theorems of the iterative scheme in [12, 16] for monotone generalized alpa-nonexpansive mappings in uniformly convex partially ordered hyperbolic metric spaces. And we also give a numerical example to show that this scheme converges faster than the scheme in [14] and apply the result to the integral equation. △ Less

Submitted 25 June, 2020; originally announced June 2020.

Minimum Width for Universal Approximation

Authors: Sejun Park, Chulhee Yun, Jaeho Lee, Jinwoo Shin

Abstract: The universal approximation property of width-bounded networks has been studied as a dual of classical universal approximation results on depth-bounded networks. However, the critical width enabling the universal approximation has not been exactly characterized in terms of the input dimension $d_x$ and the output dimension $d_y$. In this work, we provide the first definitive result in this directi… ▽ More The universal approximation property of width-bounded networks has been studied as a dual of classical universal approximation results on depth-bounded networks. However, the critical width enabling the universal approximation has not been exactly characterized in terms of the input dimension $d_x$ and the output dimension $d_y$. In this work, we provide the first definitive result in this direction for networks using the ReLU activation functions: The minimum width required for the universal approximation of the $L^p$ functions is exactly $\max\{d_x+1,d_y\}$. We also prove that the same conclusion does not hold for the uniform approximation with ReLU, but does hold with an additional threshold activation function. Our proof technique can be also used to derive a tighter upper bound on the minimum width required for the universal approximation using networks with general activation functions. △ Less

Submitted 15 June, 2020; originally announced June 2020.

SGD with shuffling: optimal rates without component convexity and large epoch requirements

Authors: Kwangjun Ahn, Chulhee Yun, Suvrit Sra

Abstract: We study without-replacement SGD for solving finite-sum optimization problems. Specifically, depending on how the indices of the finite-sum are shuffled, we consider the RandomShuffle (shuffle at the beginning of each epoch) and SingleShuffle (shuffle only once) algorithms. First, we establish minimax optimal convergence rates of these algorithms up to poly-log factors. Notably, our analysis is ge… ▽ More We study without-replacement SGD for solving finite-sum optimization problems. Specifically, depending on how the indices of the finite-sum are shuffled, we consider the RandomShuffle (shuffle at the beginning of each epoch) and SingleShuffle (shuffle only once) algorithms. First, we establish minimax optimal convergence rates of these algorithms up to poly-log factors. Notably, our analysis is general enough to cover gradient dominated nonconvex costs, and does not rely on the convexity of individual component functions unlike existing optimal convergence results. Secondly, assuming convexity of the individual components, we further sharpen the tight convergence results for RandomShuffle by removing the drawbacks common to all prior arts: large number of epochs required for the results to hold, and extra poly-log factor gaps to the lower bound. △ Less

Submitted 21 June, 2020; v1 submitted 12 June, 2020; originally announced June 2020.

Comments: 53 pages; supersedes the preprint arXiv:2004.08657; v2 corrects an erroneous claim about SingleShuffle and newly adds Theorem 24 and Appendix F for SingleShuffle

$O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers

Authors: Chulhee Yun, Yin-Wen Chang, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, Sanjiv Kumar

Abstract: Recently, Transformer networks have redefined the state of the art in many NLP tasks. However, these models suffer from quadratic computational cost in the input sequence length $n$ to compute pairwise attention in each layer. This has prompted recent research into sparse Transformers that sparsify the connections in the attention layers. While empirically promising for long sequences, fundamental… ▽ More Recently, Transformer networks have redefined the state of the art in many NLP tasks. However, these models suffer from quadratic computational cost in the input sequence length $n$ to compute pairwise attention in each layer. This has prompted recent research into sparse Transformers that sparsify the connections in the attention layers. While empirically promising for long sequences, fundamental questions remain unanswered: Can sparse Transformers approximate any arbitrary sequence-to-sequence function, similar to their dense counterparts? How does the sparsity pattern and the sparsity level affect their performance? In this paper, we address these questions and provide a unifying framework that captures existing sparse attention models. We propose sufficient conditions under which we prove that a sparse attention model can universally approximate any sequence-to-sequence function. Surprisingly, our results show that sparse Transformers with only $O(n)$ connections per attention layer can approximate the same function class as the dense model with $n^2$ connections. Lastly, we present experiments comparing different patterns/levels of sparsity on standard NLP tasks. △ Less

Submitted 19 December, 2020; v1 submitted 8 June, 2020; originally announced June 2020.

Comments: 31 pages, NeurIPS 2020 Camera-ready