Oct 23, 2019 · We show that both BERT extensions are quick to fine-tune and converge after as little as 1 epoch of training on a small, domain-specific data set.

Several dimensionality reduction algorithms such as RBM, autoencoders, subspace multinomial models (SMM) are used to obtain a low dimensional representation of ...

In this paper, we present an extractive approach to document summarization, the Siamese Hierarchical Transformer Encoders system, that is based on the use of ...

Oct 11, 2022 · In several long document downstream classification tasks, our best HAT model outperforms equally-sized Longformer models while using 10-20% less ...

This repository has a pytorch implementation of hierarchical transformers for long document classification, introduced in our paper.

Instead of modifying multi-head self-attention mechanism to efficiently model long sequences, hierarchical Transformers build on top of vanilla transformer ...

This paper suggests to take advantage of pre-trained sentence transformers to start from semantically meaningful embeddings of the individual sentences.

Jan 31, 2021 · This paper extends BERT for doing long document classification in nlp. They propose BERT variations RoBERT and ToBERT as hierarchical ...

HAT use a hierarchical attention scheme, which is a combination of segment-wise and cross-segment attention operations. You can think segments as paragraphs or ...

We introduce an innovative approach for multi-modal long document classification based on the Hierarchical Prompt and Multi-modal Transformer (HPMT).