-
Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
Authors:
Jamba Team,
Barak Lenz,
Alan Arazi,
Amir Bergman,
Avshalom Manevich,
Barak Peleg,
Ben Aviram,
Chen Almagor,
Clara Fridman,
Dan Padnos,
Daniel Gissin,
Daniel Jannai,
Dor Muhlgay,
Dor Zimberg,
Edden M Gerber,
Elad Dolev,
Eran Krakovsky,
Erez Safahi,
Erez Schwartz,
Gal Cohen,
Gal Shachaf,
Haim Rozenblum,
Hofit Bata,
Ido Blass,
Inbal Magar
, et al. (36 additional authors not shown)
Abstract:
We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, wi…
▽ More
We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a variety of conversational and instruction-following capabilties, and have an effective context length of 256K tokens, the largest amongst open-weight models. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. When evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models achieve excellent results while providing high throughput and outperforming other open-weight models on long-context benchmarks. The model weights for both sizes are publicly available under the Jamba Open Model License and we release ExpertsInt8 as open source.
△ Less
Submitted 22 August, 2024;
originally announced August 2024.
-
NTIRE 2024 Challenge on Night Photography Rendering
Authors:
Egor Ershov,
Artyom Panshin,
Oleg Karasev,
Sergey Korchagin,
Shepelev Lev,
Alexandr Startsev,
Daniil Vladimirov,
Ekaterina Zaychenkova,
Nikola Banić,
Dmitrii Iarchuk,
Maria Efimova,
Radu Timofte,
Arseniy Terekhin,
Shuwei Yue,
Yuyang Liu,
Minchen Wei,
Lu Xu,
Chao Zhang,
Yasi Wang,
Furkan Kınlı,
Doğa Yılmaz,
Barış Özcan,
Furkan Kıraç,
Shuai Liu,
Jingyuan Xiao
, et al. (25 additional authors not shown)
Abstract:
This paper presents a review of the NTIRE 2024 challenge on night photography rendering. The goal of the challenge was to find solutions that process raw camera images taken in nighttime conditions, and thereby produce a photo-quality output images in the standard RGB (sRGB) space. Unlike the previous year's competition, the challenge images were collected with a mobile phone and the speed of algo…
▽ More
This paper presents a review of the NTIRE 2024 challenge on night photography rendering. The goal of the challenge was to find solutions that process raw camera images taken in nighttime conditions, and thereby produce a photo-quality output images in the standard RGB (sRGB) space. Unlike the previous year's competition, the challenge images were collected with a mobile phone and the speed of algorithms was also measured alongside the quality of their output. To evaluate the results, a sufficient number of viewers were asked to assess the visual quality of the proposed solutions, considering the subjective nature of the task. There were 2 nominations: quality and efficiency. Top 5 solutions in terms of output quality were sorted by evaluation time (see Fig. 1). The top ranking participants' solutions effectively represent the state-of-the-art in nighttime photography rendering. More results can be found at https://nightimaging.org.
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
-
Exemplar Guided Active Learning
Authors:
Jason Hartford,
Kevin Leyton-Brown,
Hadas Raviv,
Dan Padnos,
Shahar Lev,
Barak Lenz
Abstract:
We consider the problem of wisely using a limited budget to label a small subset of a large unlabeled dataset. We are motivated by the NLP problem of word sense disambiguation. For any word, we have a set of candidate labels from a knowledge base, but the label set is not necessarily representative of what occurs in the data: there may exist labels in the knowledge base that very rarely occur in t…
▽ More
We consider the problem of wisely using a limited budget to label a small subset of a large unlabeled dataset. We are motivated by the NLP problem of word sense disambiguation. For any word, we have a set of candidate labels from a knowledge base, but the label set is not necessarily representative of what occurs in the data: there may exist labels in the knowledge base that very rarely occur in the corpus because the sense is rare in modern English; and conversely there may exist true labels that do not exist in our knowledge base. Our aim is to obtain a classifier that performs as well as possible on examples of each "common class" that occurs with frequency above a given threshold in the unlabeled set while annotating as few examples as possible from "rare classes" whose labels occur with less than this frequency. The challenge is that we are not informed which labels are common and which are rare, and the true label distribution may exhibit extreme skew. We describe an active learning approach that (1) explicitly searches for rare classes by leveraging the contextual embedding spaces provided by modern language models, and (2) incorporates a stopping rule that ignores classes once we prove that they occur below our target threshold with high probability. We prove that our algorithm only costs logarithmically more than a hypothetical approach that knows all true label frequencies and show experimentally that incorporating automated search can significantly reduce the number of samples needed to reach target accuracy levels.
△ Less
Submitted 2 November, 2020;
originally announced November 2020.
-
Analyzing Organizational Routines in Online Knowledge Collaborations: A Case for Sequence Analysis in CSCW
Authors:
Brian C. Keegan,
Shakked Lev,
Ofer Arazy
Abstract:
Research into socio-technical systems like Wikipedia has overlooked important structural patterns in the coordination of distributed work. This paper argues for a conceptual reorientation towards sequences as a fundamental unit of analysis for understanding work routines in online knowledge collaboration. We outline a research agenda for researchers in computer-supported cooperative work (CSCW) to…
▽ More
Research into socio-technical systems like Wikipedia has overlooked important structural patterns in the coordination of distributed work. This paper argues for a conceptual reorientation towards sequences as a fundamental unit of analysis for understanding work routines in online knowledge collaboration. We outline a research agenda for researchers in computer-supported cooperative work (CSCW) to understand the relationships, patterns, antecedents, and consequences of sequential behavior using methods already developed in fields like bio-informatics. Using a data set of 37,515 revisions from 16,616 unique editors to 96 Wikipedia articles as a case study, we analyze the prevalence and significance of different sequences of editing patterns. We illustrate the mixed method potential of sequence approaches by interpreting the frequent patterns as general classes of behavioral motifs. We conclude by discussing the methodological opportunities for using sequence analysis for expanding existing approaches to analyzing and theorizing about co-production routines in online knowledge collaboration.
△ Less
Submitted 28 August, 2015; v1 submitted 19 August, 2015;
originally announced August 2015.