Accelerating Hyperbolic t-SNE
Authors: 
Martin Skrodzki, 
Hunter van Geffen, Nicolas F. Chaves-de-Plaza, Thomas Höllt, Elmar Eisemann, Klaus HildebrandtAuthors Info & Claims
Martin Skrodzki, Hunter van Geffen, Nicolas F. Chaves-de-Plaza, Thomas Höllt, Elmar Eisemann, Klaus HildebrandtAuthors Info & ClaimsPages 4403 - 4415
Abstract
The need to understand the structure of hierarchical or high-dimensional data is present in a variety of fields. Hyperbolic spaces have proven to be an important tool for embedding computations and analysis tasks as their non-linear nature lends itself well to tree or graph data. Subsequently, they have also been used in the visualization of high-dimensional data, where they exhibit increased embedding performance. However, none of the existing dimensionality reduction methods for embedding into hyperbolic spaces scale well with the size of the input data. That is because the embeddings are computed via iterative optimization schemes and the computation cost of every iteration is quadratic in the size of the input. Furthermore, due to the non-linear nature of hyperbolic spaces, euclidean acceleration structures cannot directly be translated to the hyperbolic setting. This article introduces the first acceleration structure for hyperbolic embeddings, building upon a polar quadtree. We compare our approach with existing methods and demonstrate that it computes embeddings of similar quality in significantly less time. Implementation and scripts for the experiments can be found at <uri>https://graphics.tudelft.nl/accelerating-hyperbolic-tsne</uri>.
References
[1]
T. S. Andrews, V. Y. Kiselev, D. McCarthy, and M. Hemberg, “Tutorial: Guidelines for the computational analysis of single-cell rna sequencing data,” Nature Protoc., vol. 16, no. 1, pp. 1–9, 2021.
[2]
E. Becht et al., “Dimensionality reduction for visualizing single-cell data using UMAP,” Nature Biotechnol., vol. 37, no. 1, pp. 38–44, 2019.
[3]
M. Belkin and P. Niyogi, “Laplacian eigenmaps and spectral techniques for embedding and clustering,” in Proc. 14th Int. Conf. Neural Inf. Process. Syst., The MIT Press, Boston, 2001, pp. 585–591.
[4]
A. C. Belkina, C. O. Ciccolella, R. Anno, R. Halpert, J. Spidlen, and J. E. Snyder-Cappione, “Automated optimized parameters for t-distributed stochastic neighbor embedding improve visualization and analysis of large datasets,” Nature Commun., vol. 10, no. 1, 2019, Art. no.
[5]
T. Bläsius, T. Friedrich, A. Krohmer, and S. Laue, “Efficient embedding of scale-free graphs in the hyperbolic plane,” IEEE/ACM Trans. Netw., vol. 26, no. 2, pp. 920–933, Apr. 2018.
[6]
M. Boguná, F. Papadopoulos, and D. Krioukov, “Sustaining the internet with hyperbolic mapping,” Nature Commun., vol. 1, no. 1, 2010, Art. no.
[7]
J. N. Böhm, P. Berens, and D. Kobak, “Attraction-repulsion spectrum in neighbor embeddings,” J. Mach. Learn. Res., vol. 23, no. 95, pp. 1–32, 2022.
[8]
J. W. Cannon et al., “Hyperbolic geometry,” Flavors of Geometry, vol. 31, pp. 59–115, 1997.
[9]
X. Cao, “Poincaré fréchet mean,” Pattern Recognit., vol. 137, 2023, Art. no.
[10]
M. Cho and J. Lee, “Riemannian approach to batch normalization,” in Proc. Adv. Neural Inf. Process. Syst., 2017, pp. 5225–5235.
[11]
J. H. Conway, H. Burgiel, and C. Goodman-Strauss, The Symmetries of Things. Natick, MA, USA: A. K. Peters, 2008.
[12]
A. Cvetkovski and M. Crovella, “Multidimensional scaling in the poincaré disk,” Appl. Math. Inf. Sci., vol. 10, no. 1, pp. 125–133, 2016.
[13]
D. Eppstein, “Limitations on realistic hyperbolic graph drawing,” in Proc. Int. Symp. Graph Drawing Netw. Vis., Springer, 2021, pp. 343–357.
[14]
M. Espadoto, R. M. Martins, A. Kerren, N. S. Hirata, and A. C. Telea, “Toward a quantitative survey of dimension reduction techniques,” IEEE Trans. Vis. Comput. Graphics, vol. 27, no. 3, pp. 2153–2173, Mar. 2021.
[15]
C. Fellbaum, WordNet: An Electronic Lexical Database. Cambridge, MA, USA: MIT Press, 1998.
[16]
T. Friedrich and A. Krohmer, “Cliques in hyperbolic random graphs,” in Proc. IEEE Conf. Comput. Commun., 2015, pp. 1544–1552.
[17]
O. Ganea, G. Bécigneul, and T. Hofmann, “Hyperbolic neural networks,” in Proc. 32nd Int. Conf. Neural Inf. Process. Syst., 2018, pp. 5350–5360.
[18]
C. Gulcehre et al., “Hyperbolic attention networks,” 2018, arXiv:1805.09786.
[19]
Y. Guo, H. Guo, and S. X. Yu, “CO-SNE: Dimensionality reduction and visualization for hyperbolic data,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 21–30.
[20]
S. Ingram, T. Munzner, and M. Olano, “Glimmer: Multilevel MDS on the GPU,” IEEE Trans. Vis. Comput. Graphics, vol. 15, no. 2, pp. 249–261, Mar./Apr. 2009.
[21]
H. Isozaki and Y. Kurylev, “Fourier transforms on the hyperbolic space,” in Introduction to Spectral Theory and Inverse Problem on Asymptotically Hyperbolic Manifolds, vol. 32, Tokyo, Japan: Math. Soc. Japan, 2014, pp. 11–46.
[22]
R. A. Jacobs, “Increased rates of convergence through learning rate adaptation,” Neural Netw., vol. 1, no. 4, pp. 295–307, 1988.
[23]
P. Joia, D. Coimbra, J. A. Cuminato, F. V. Paulovich, and L. G. Nonato, “Local affine multidimensional projection,” IEEE Trans. Vis. Comput. Graphics, vol. 17, no. 12, pp. 2563–2571, Dec. 2011.
[24]
M. Keller-Ressel and S. Nargang, “Hydra: A method for strain-minimizing hyperbolic embedding of network-and distance-based data,” J. Complex Netw., vol. 8, no. 1, 2020, Art. no.
[25]
A. Klimovskaia, D. Lopez-Paz, L. Bottou, and M. Nickel, “Poincaré maps for analyzing complex hierarchies in single-cell data,” Nature Commun., vol. 11, no. 1, pp. 1–9, 2020.
[26]
D. Kobak and P. Berens, “The art of using t-SNE for single-cell transcriptomics,” Nature Commun., vol. 10, no. 1, 2019, Art. no.
[27]
D. Kobak and G. C. Linderman, “Initialization is critical for preserving global data structure in both t-SNE and UMAP,” Nature Biotechnol., vol. 39, no. 2, pp. 156–157, 2021.
[28]
T. Kohonen, “Self-organized formation of topologically correct feature maps,” Biol. Cybern., vol. 43, no. 1, pp. 59–69, 1982.
[29]
E. Kopczyński and D. Celińska-Kopczyńska, “Navigating higher dimensional spaces using hyperbolic geometry,” 2021,.
[30]
D. Krioukov, F. Papadopoulos, M. Kitsak, A. Vahdat, and M. Boguná, “Hyperbolic geometry of complex networks,” Phys. Rev. E, vol. 82, no. 3, 2010, Art. no.
[31]
J. Krumsiek, C. Marr, T. Schroeder, and F. J. Theis, “Hierarchical differentiation of myeloid progenitors is encoded in the transcription factor network,” PLoS One, vol. 6, no. 8, 2011, Art. no.
[32]
J. Lamping, R. Rao, and P. Pirolli, “A focus context technique based on hyperbolic geometry for visualizing large hierarchies,” in Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 1995, pp. 401–408.
[33]
Y.-W. E. Lin, R. R. Coifman, G. Mishne, and R. Talmon, “Hyperbolic diffusion embedding and distance for hierarchical representation learning,” in Proc. 40th Int. Conf. Mach. Learn., 2023, Art. no.
[34]
G. C. Linderman, M. Rachh, J. G. Hoskins, S. Steinerberger, and Y. Kluger, “Fast interpolation-based t-SNE for improved visualization of single-cell RNA-seq data,” Nature Methods, vol. 16, no. 3, pp. 243–245, 2019.
[35]
M. V. Looz, H. Meyerhenke, and R. Prutkin, “Generating random hyperbolic graphs in subquadratic time,” in Proc. Int. Symp. Algorithms Comput., Springer, 2015, pp. 467–478.
[36]
A. Lou, I. Katsman, Q. Jiang, S. Belongie, S.-N. Lim, and C. De Sa, “Differentiating through the Fréchet mean,” in Proc. 37th Int. Conf. Mach. Learn., PMLR, 2020, pp. 6393–6403.
[37]
J. Miller, S. Kobourov, and V. Huroyan, “Browser-based hyperbolic visualization of graphs,” in Proc. IEEE 15th Pacific Visual. Symp., 2022, pp. 71–80.
[38]
A. Narechania, A. Karduni, R. Wesslen, and E. Wall, “VITALITY: Promoting serendipitous discovery of academic literature with transformers & visual analytics,” IEEE Trans. Vis. Comput. Graphics, vol. 28, no. 1, pp. 486–496, Jan. 2022.
[39]
M. Nickel and D. Kiela, “Poincaré embeddings for learning hierarchical representations,” in Proc. 31st Int. Conf. Neural Inf. Process. Syst., 2017, pp. 6341–6350.
[40]
J. Ontrup and H. Ritter, “Large-scale data exploration with the hierarchically growing hyperbolic SOM,” Neural Netw., vol. 19, no. 6/7, pp. 751–761, 2006.
[41]
N. Pezzotti, T. Höllt, B. Lelieveldt, E. Eisemann, and A. Vilanova, “Hierarchical stochastic neighbor embedding,” Comput. Graph. Forum, vol. 35, no. 3, pp. 21–30, 2016.
[42]
N. Pezzotti et al., “GPGPU linear complexity t-SNE optimization,” IEEE Trans. Vis. Comput. Graphics, vol. 26, no. 1, pp. 1172–1181, Jan. 2020.
[43]
P. Pirolli, S. K. Card, and M. M. Van Der Wege, “Visual information foraging in a focus context visualization,” in Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 2001, pp. 506–513.
[44]
M. Plass et al., “Cell type atlas and lineage tree of a whole complex animal by single-cell transcriptomics,” Science, vol. 360, no. 6391, 2018, Art. no.
[45]
S. T. Roweis and L. K. Saul, “Nonlinear dimensionality reduction by locally linear embedding,” Science, vol. 290, no. 5500, pp. 2323–2326, 2000.
[46]
M. v. d. Ruit, M. Billeter, and E. Eisemann, “An efficient dual-hierarchy t-SNE minimization,” IEEE Trans. Vis. Comput. Graphics, vol. 28, no. 1, pp. 614–622, Jan. 2022.
[47]
F. Sala, C. De Sa, A. Gu, and C. Ré, “Representation tradeoffs for hyperbolic embeddings,” in Proc. 35th Int. Conf. Mach. Learn., PMLR, 2018, pp. 4460–4469.
[48]
R. Sarkar, “Low distortion delaunay embedding of trees in hyperbolic plane,” in Proc. Int. Symp. Graph Drawing, Springer, 2011, pp. 355–366.
[49]
J. R. Senning, “Computing and estimating the rate of convergence,” Tech. Rep., Gordon College, Wenham, 2019. [Online]. Available: https://www.math-cs.gordon.edu/local/courses/mat342/handouts/rate.pdf
[50]
L. Van Der Maaten, “Accelerating t-SNE using tree-based algorithms,” J. Mach. Learn. Res., vol. 15, no. 1, pp. 3221–3245, 2014.
[51]
L. Van der Maaten and G. Hinton, “Visualizing data using t-SNE,” J. Mach. Learn. Res., vol. 9, no. 11, pp. 2579–2605, 2008.
[52]
K. Verbeek and S. Suri, “Metric embedding, hyperbolic space, and social networks,” Comput. Geometry, vol. 59, pp. 1–12, 2014.
[53]
J. A. Walter, “H-MDS: A new approach for interactive visualization with multidimensional scaling in the hyperbolic space,” Inf. Syst., vol. 29, no. 4, pp. 273–292, 2004.
[54]
J. Wang, W. Zhang, H. Yang, C.-C. M. Yeh, and L. Wang, “Visual analytics for RNN-based deep reinforcement learning,” IEEE Trans. Vis. Comput. Graphics, vol. 28, no. 12, pp. 4141–4155, Dec. 2022.
[55]
J. Wu, D. Liu, Z. Guo, Q. Xu, and Y. Wu, “TacticFlow: Visual analytics of ever-changing tactics in racket sports,” IEEE Trans. Vis. Comput. Graphics, vol. 28, no. 1, pp. 835–845, Jan. 2022.
[56]
Z. Wu and H. Wu, “Accounting for cell type hierarchy in evaluating single cell RNA-seq clustering,” Genome Biol., vol. 21, no. 1, pp. 1–14, 2020.
[57]
J. Xia, Y. Zhang, J. Song, Y. Chen, Y. Wang, and S. Liu, “Revisiting dimensionality reduction techniques for visual cluster analysis: An empirical study,” IEEE Trans. Vis. Comput. Graphics, vol. 28, no. 1, pp. 529–539, Jan. 2022.
[58]
H. Xiao, X. Liu, Y. Song, G. Y. Wong, and S. See, “Complex hyperbolic knowledge graph embeddings with fast fourier transform,” in Proc. Conf. Empir. Methods Natural Lang. Process., Abu Dhabi, 2022, pp. 5228–5239.
[59]
C. Zhang et al., “PromotionLens: Inspecting promotion strategies of online e-commerce via visual analytics,” IEEE Trans. Vis. Comput. Graphics, vol. 29, no. 1, pp. 767–777, Jan. 2023.
[60]
Y. Zhou and T. O. Sharpee, “Hyperbolic geometry of gene expression,” IScience, vol. 24, no. 3, 2021, Art. no.
[61]
Y. Zhou and T. O. Sharpee, “Using global t-SNE to preserve inter-cluster data structure,” Neural Comput., vol. 34, no. 8, pp. 1637–1651, 2022.
[62]
Y. Zhou, B. H. Smith, and T. O. Sharpee, “Hyperbolic geometry of the olfactory space,” Sci. Adv., vol. 4, no. 8, 2018, Art. no.
Recommendations
Accelerating t-SNE using tree-based algorithms
The paper investigates the acceleration of t-SNE--an embedding technique that is commonly used for the visualization of high-dimensional data in scatter plots--using two tree-based algorithms. In particular, the paper develops variants of the Barnes-Hut ...
Conditional t-SNE: more informative t-SNE embeddings
AbstractDimensionality reduction and manifold learning methods such as t-distributed stochastic neighbor embedding (t-SNE) are frequently used to map high-dimensional data into a two-dimensional space to visualize and explore that data. Going beyond the ...
High-Order Multiderivative Time Integrators for Hyperbolic Conservation Laws
Multiderivative time integrators have a long history of development for ordinary differential equations, and yet to date, only a small subset of these methods have been explored as a tool for solving partial differential equations (PDEs). This large ...
Comments
Information & Contributors
Information
Published In
1077-2626 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Educational Activities Department
United States
Publication History
Published: 12 February 2024
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 16 Oct 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in