HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
Authors:
Eric Nguyen,
Michael Poli,
Marjan Faizi,
Armin Thomas,
Callum Birch-Sykes,
Michael Wornow,
Aman Patel,
Clayton Rabideau,
Stefano Massaroli,
Yoshua Bengio,
Stefano Ermon,
Stephen A. Baccus,
Chris Ré
Abstract:
Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous…
▽ More
Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data. On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points. Code at https://github.com/HazyResearch/hyena-dna.
△ Less
Submitted 14 November, 2023; v1 submitted 27 June, 2023;
originally announced June 2023.
Aiding Medical Diagnosis Through the Application of Graph Neural Networks to Functional MRI Scans
Authors:
Katharina Zühlsdorff,
Clayton M. Rabideau
Abstract:
Graph Neural Networks (GNNs) have been shown to be a powerful tool for generating predictions from biological data. Their application to neuroimaging data such as functional magnetic resonance imaging (fMRI) scans has been limited. However, applying GNNs to fMRI scans may substantially improve predictive accuracy and could be used to inform clinical diagnosis in the future. In this paper, we prese…
▽ More
Graph Neural Networks (GNNs) have been shown to be a powerful tool for generating predictions from biological data. Their application to neuroimaging data such as functional magnetic resonance imaging (fMRI) scans has been limited. However, applying GNNs to fMRI scans may substantially improve predictive accuracy and could be used to inform clinical diagnosis in the future. In this paper, we present a novel approach to representing resting-state fMRI data as a graph containing nodes and edges without omitting any of the voxels and thus reducing information loss. We compare multiple GNN architectures and show that they can successfully predict the disease and sex of a person. We hope to provide a basis for future work to exploit the power of GNNs when applied to brain imaging data.
△ Less
Submitted 1 December, 2021;
originally announced December 2021.
Continuous-Depth Neural Models for Dynamic Graph Prediction
Authors:
Michael Poli,
Stefano Massaroli,
Clayton M. Rabideau,
Junyoung Park,
Atsushi Yamashita,
Hajime Asama,
Jinkyoo Park
Abstract:
We introduce the framework of continuous-depth graph neural networks (GNNs). Neural graph differential equations (Neural GDEs) are formalized as the counterpart to GNNs where the input-output relationship is determined by a continuum of GNN layers, blending discrete topological structures and differential equations. The proposed framework is shown to be compatible with static GNN models and is ext…
▽ More
We introduce the framework of continuous-depth graph neural networks (GNNs). Neural graph differential equations (Neural GDEs) are formalized as the counterpart to GNNs where the input-output relationship is determined by a continuum of GNN layers, blending discrete topological structures and differential equations. The proposed framework is shown to be compatible with static GNN models and is extended to dynamic and stochastic settings through hybrid dynamical system theory. Here, Neural GDEs improve performance by exploiting the underlying dynamics geometry, further introducing the ability to accommodate irregularly sampled data. Results prove the effectiveness of the proposed models across applications, such as traffic forecasting or prediction in genetic regulatory networks.
△ Less
Submitted 22 June, 2021;
originally announced June 2021.
