-
Falcon2-11B Technical Report
Authors:
Quentin Malartic,
Nilabhra Roy Chowdhury,
Ruxandra Cojocaru,
Mugariya Farooq,
Giulia Campesan,
Yasser Abdelaziz Dahou Djilali,
Sanath Narayan,
Ankit Singh,
Maksim Velikanov,
Basma El Amel Boussaha,
Mohammed Al-Yafeai,
Hamza Alobeidli,
Leen Al Qadi,
Mohamed El Amine Seddik,
Kirill Fedyanin,
Reda Alami,
Hakim Hacid
Abstract:
We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model. We report our findings during the training of the Falcon2-11B which follows a multi-stage approach where the early stages are distinguished by their context length and a final stage where we use a curated, high-quality dataset. Additio…
▽ More
We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model. We report our findings during the training of the Falcon2-11B which follows a multi-stage approach where the early stages are distinguished by their context length and a final stage where we use a curated, high-quality dataset. Additionally, we report the effect of doubling the batch size mid-training and how training loss spikes are affected by the learning rate. The downstream performance of the foundation model is evaluated on established benchmarks, including multilingual and code datasets. The foundation model shows strong generalization across all the tasks which makes it suitable for downstream finetuning use cases. For the vision language model, we report the performance on several benchmarks and show that our model achieves a higher average score compared to open-source models of similar size. The model weights and code of both Falcon2-11B and Falcon2-11B-vlm are made available under a permissive license.
△ Less
Submitted 20 July, 2024;
originally announced July 2024.
-
The Falcon Series of Open Language Models
Authors:
Ebtesam Almazrouei,
Hamza Alobeidli,
Abdulaziz Alshamsi,
Alessandro Cappelli,
Ruxandra Cojocaru,
Mérouane Debbah,
Étienne Goffinet,
Daniel Hesslow,
Julien Launay,
Quentin Malartic,
Daniele Mazzotta,
Badreddine Noune,
Baptiste Pannier,
Guilherme Penedo
Abstract:
We introduce the Falcon series: 7B, 40B, and 180B parameters causal decoder-only models trained on a diverse high-quality corpora predominantly assembled from web data. The largest model, Falcon-180B, has been trained on over 3.5 trillion tokens of text--the largest openly documented pretraining run. Falcon-180B significantly outperforms models such as PaLM or Chinchilla, and improves upon concurr…
▽ More
We introduce the Falcon series: 7B, 40B, and 180B parameters causal decoder-only models trained on a diverse high-quality corpora predominantly assembled from web data. The largest model, Falcon-180B, has been trained on over 3.5 trillion tokens of text--the largest openly documented pretraining run. Falcon-180B significantly outperforms models such as PaLM or Chinchilla, and improves upon concurrently developed models such as LLaMA 2 or Inflection-1. It nears the performance of PaLM-2-Large at a reduced pretraining and inference cost, making it, to our knowledge, one of the three best language models in the world along with GPT-4 and PaLM-2-Large. We report detailed evaluations, as well as a deep dive into the methods and custom tooling employed to pretrain Falcon. Notably, we report on our custom distributed training codebase, allowing us to efficiently pretrain these models on up to 4,096 A100s on cloud AWS infrastructure with limited interconnect. We release a 600B tokens extract of our web dataset, as well as the Falcon-7/40/180B models under a permissive license to foster open-science and accelerate the development of an open ecosystem of large language models.
△ Less
Submitted 29 November, 2023; v1 submitted 28 November, 2023;
originally announced November 2023.
-
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
Authors:
Guilherme Penedo,
Quentin Malartic,
Daniel Hesslow,
Ruxandra Cojocaru,
Alessandro Cappelli,
Hamza Alobeidli,
Baptiste Pannier,
Ebtesam Almazrouei,
Julien Launay
Abstract:
Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclea…
▽ More
Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models from the state-of-the-art trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.
△ Less
Submitted 1 June, 2023;
originally announced June 2023.
-
X-ray optics and beam characterisation using random modulation: Experiments
Authors:
Sebastien Berujon,
Ruxandra Cojocaru,
Pierre Piault,
Rafael Celestre,
Thomas Roth,
Raymond Barrett,
Eric Ziegler
Abstract:
In a previous paper, we reviewed theoretically some of the available processing schemes for X-ray wavefront sensing based on random modulation. We here show experimental applications of the technique for characterising both refractive and reflective optical components. These fast and accurate X-ray at-wavelength metrology methods can assist the manufacture of X-ray optics that transport X-ray beam…
▽ More
In a previous paper, we reviewed theoretically some of the available processing schemes for X-ray wavefront sensing based on random modulation. We here show experimental applications of the technique for characterising both refractive and reflective optical components. These fast and accurate X-ray at-wavelength metrology methods can assist the manufacture of X-ray optics that transport X-ray beams with minimum amount of wavefront distortion. We also recall how such methods can facilitate online optimization of active optics.
△ Less
Submitted 2 December, 2019;
originally announced December 2019.
-
X-ray optics and beam characterization using random modulation: Theory
Authors:
Sebastien Berujon,
Ruxandra Cojocaru,
Pierre Piault,
Rafael Celestre,
Thomas Roth,
Raymond Barrett,
Eric Ziegler
Abstract:
X-ray near-field speckle-based phase-sensing approaches provide efficient means to characterise optical elements. Here, we present a theoretical review of several of these speckle methods in the frame of optical characterisation and provide a generalization of the concept. As we also demonstrate experimentally in another paper, the methods theoretically developed here can be applied with different…
▽ More
X-ray near-field speckle-based phase-sensing approaches provide efficient means to characterise optical elements. Here, we present a theoretical review of several of these speckle methods in the frame of optical characterisation and provide a generalization of the concept. As we also demonstrate experimentally in another paper, the methods theoretically developed here can be applied with different beams and optics and within a variety of situations where at-wavelength metrology is desired. By understanding the differences between the various processing methods, it is possible to find and implement the best suited approach for each metrology scenario.
△ Less
Submitted 3 March, 2020; v1 submitted 25 February, 2019;
originally announced February 2019.
-
Axions and the luminosity function of white dwarfs. The thin and thick disks, and the halo
Authors:
Jordi Isern,
Enrique Garcia-Berro,
Santiago Torres,
Roxana Cojocaru,
Silvia Catalan
Abstract:
The evolution of white dwarfs is a simple gravothermal process of cooling. Since the shape of their luminosity function is sensitive to the characteristic cooling time, it is possible to use its slope to test the existence of additional sources or sinks of energy, such as those predicted by alternative physical theories. The aim of this paper is to study if the changes in the slope of the white dw…
▽ More
The evolution of white dwarfs is a simple gravothermal process of cooling. Since the shape of their luminosity function is sensitive to the characteristic cooling time, it is possible to use its slope to test the existence of additional sources or sinks of energy, such as those predicted by alternative physical theories. The aim of this paper is to study if the changes in the slope of the white dwarf luminosity function around bolometric magnitudes ranging from 8 to 10 and previously attributed to axion emission are, effectively, a consequence of the existence of axions and not an artifact introduced by the star formatio rate. We compute theoretical luminosity functions of the thin and thick disk, and of the stellar halo including axion emission and we compare them with the existing observed luminosity functions. Since these stellar populations have different star formation histories, the slope change should be present in all of them at the same place if it is due to axions or any other intrinsic cooling mechanism. The signature of an unexpected cooling seems to be present in the luminosity functions of the thin and thick disks, as well as in the halo luminosity function. This additional cooling is compatible with axion emission, thus supporting the idea that DFSZ axions, with a mass in the range of 4 to 10 meV, could exist. If this were the case, these axions could be detected by the future solar axioscope IAXO.
△ Less
Submitted 30 April, 2018;
originally announced May 2018.
-
The population of single and binary white dwarfs of the Galactic bulge
Authors:
S. Torres,
E. García-Berro,
R. Cojocaru,
A. Calamida
Abstract:
Recent Hubble Space Telescope observations have unveiled the white dwarf cooling sequence of the Galactic bulge. Although the degenerate sequence can be well fitted employing the most up-to-date theoretical cooling sequences, observations show a systematic excess of red objects that cannot be explained by the theoretical models of single carbon-oxygen white dwarfs of the appropriate masses. Here w…
▽ More
Recent Hubble Space Telescope observations have unveiled the white dwarf cooling sequence of the Galactic bulge. Although the degenerate sequence can be well fitted employing the most up-to-date theoretical cooling sequences, observations show a systematic excess of red objects that cannot be explained by the theoretical models of single carbon-oxygen white dwarfs of the appropriate masses. Here we present a population synthesis study of the white dwarf cooling sequence of the Galactic bulge that takes into account the populations of both single white dwarfs and binary systems containing at least one white dwarf. These calculations incorporate state-of-the-art cooling sequences for white dwarfs with hydrogen-rich and hydrogen-deficient atmospheres, for both white dwarfs with carbon-oxygen and helium cores, and also take into account detailed prescriptions of the evolutionary history of binary systems. Our Monte Carlo simulator also incorporates all the known observational biases. This allows us to model with a high degree of realism the white dwarf population of the Galactic bulge. We find that the observed excess of red stars can be partially attributed to white dwarf plus main sequence binaries, and to cataclysmic variables or dwarf novae. Our best fit is obtained with a higher binary fraction and an initial mass function slope steeper than standard values, as well as with the inclusion of differential reddening and blending. Our results also show that the possible contribution of double degenerate systems or young and thick-disk bulge stars is negligible.
△ Less
Submitted 1 February, 2018;
originally announced February 2018.
-
The population of white dwarf-main sequence binaries in the SDSS DR 12
Authors:
R. Cojocaru,
A. Rebassa-Mansergas,
S. Torres,
E. Garcia-Berro
Abstract:
We present a Monte Carlo population synthesis study of white dwarf-main sequence (WD+MS) binaries in the Galactic disk aimed at reproducing the ensemble properties of the entire population observed by the Sloan Digital Sky Survey (SDSS) Data Release 12. Our simulations take into account all known observational biases and use the most up-to-date stellar evolutionary models. This allows us to perfor…
▽ More
We present a Monte Carlo population synthesis study of white dwarf-main sequence (WD+MS) binaries in the Galactic disk aimed at reproducing the ensemble properties of the entire population observed by the Sloan Digital Sky Survey (SDSS) Data Release 12. Our simulations take into account all known observational biases and use the most up-to-date stellar evolutionary models. This allows us to perform a sound comparison between the simulations and the observational data. We find that the properties of the simulated and observed parameter distributions agree best when assuming low values of the common envelope efficiency (0.2-0.3), a result that is in agreement with previous findings obtained by observational and population synthesis studies of close SDSS WD+MS binaries. We also show that all synthetic populations that result from adopting an initial mass ratio distribution with a positive slope are excluded by observations. Finally, we confirm that the properties of the simulated WD+MS binary populations are nearly independent of the age adopted for the thin disk, on the contribution of WD+MS binaries from the thick disk (0-17 per cent of the total population) and on the assumed fraction of the internal energy that is used to eject the envelope during the common envelope phase (0.1-0.5).
△ Less
Submitted 16 May, 2017;
originally announced May 2017.
-
The age-metallicity relation in the solar neighbourhood from a pilot sample of white dwarf-main sequence binaries
Authors:
A. Rebassa-Mansergas,
B. Anguiano,
E. García-Berro,
K. C. Freeman,
R. Cojocaru,
C. J. Manser,
A. F. Pala,
B. T. Gänsicke,
X. -W. Liu
Abstract:
The age-metallicity relation (AMR) is a fundamental observational constraint for understanding how the Galactic disc formed and evolved chemically in time. However, there is not yet an agreement on the observational properties of the AMR for the solar neighbourhood, primarily due to the difficulty in obtaining accurate stellar ages for individual field stars. We have started an observational campa…
▽ More
The age-metallicity relation (AMR) is a fundamental observational constraint for understanding how the Galactic disc formed and evolved chemically in time. However, there is not yet an agreement on the observational properties of the AMR for the solar neighbourhood, primarily due to the difficulty in obtaining accurate stellar ages for individual field stars. We have started an observational campaign for providing the much needed observational input by using wide white dwarf-main sequence (WDMS) binaries. White dwarfs are natural clocks and can be used to derive accurate ages. Metallicities can be obtained from the main sequence companions. Since the progenitors of white dwarfs and the main sequence stars were born at the same time, WDMS binaries provide a unique opportunity to observationally constrain in a robust way the properties of the AMR. In this work we present the AMR derived from analysing a pilot sample of 23 WDMS binaries and provide clear observational evidence for the lack of correlation between age and metallicity at young and intermediate ages (0-7 Gyrs).
△ Less
Submitted 10 August, 2016;
originally announced August 2016.
-
Revisiting the luminosity function of single halo white dwarfs
Authors:
R. Cojocaru,
S. Torres,
L. G. Althaus,
J. Isern,
E. García-Berro
Abstract:
White dwarfs are the fossils left by the evolution of low-and intermediate-mass stars, and have very long evolutionary timescales. This allows us to use them to explore the properties of old populations, like the Galactic halo. We present a population synthesis study of the luminosity function of halo white dwarfs, aimed at investigating which information can be derived from the currently availabl…
▽ More
White dwarfs are the fossils left by the evolution of low-and intermediate-mass stars, and have very long evolutionary timescales. This allows us to use them to explore the properties of old populations, like the Galactic halo. We present a population synthesis study of the luminosity function of halo white dwarfs, aimed at investigating which information can be derived from the currently available observed data. We employ an up-to-date population synthesis code based on Monte Carlo techniques, that incorporates the most recent and reliable cooling sequences for metal poor progenitors as well as an accurate modeling of the observational biases. We find that because the observed sample of halo white dwarfs is restricted to the brightest stars only the hot branch of the white dwarf luminosity function can be used for such purposes, and that its shape function is almost insensitive to the most relevant inputs, like the adopted cooling sequences, the initial mass function, the density profile of the stellar spheroid, or the adopted fraction of unresolved binaries. Moreover, since the cut-off of the observed luminosity has not been yet determined only lower limits to the age of the halo population can be placed. We conclude that the current observed sample of the halo white dwarf population is still too small to obtain definite conclusions about the properties of the stellar halo, and the recently computed white dwarf cooling sequences which incorporate residual hydrogen burning should be assessed using metal-poor globular clusters.
△ Less
Submitted 22 July, 2015;
originally announced July 2015.
-
DA white dwarfs from the LSS-GAC survey DR1: the preliminary luminosity and mass functions and formation rate
Authors:
A. Rebassa-Mansergas,
X. -W. Liu,
R. Cojocaru,
H. -B. Yuan,
S. Torres,
E. Garcia-Berro,
M. -X. Xiang,
Y. Huang,
D. Koester,
Y. Hou,
G. Li,
Y. Zhang
Abstract:
Modern large-scale surveys have allowed the identification of large numbers of white dwarfs. However, these surveys are subject to complicated target selection algorithms, which make it almost impossible to quantify to what extent the observational biases affect the observed populations. The LAMOST (Large Sky Area Multi-Object Fiber Spectroscopic Telescope) Spectroscopic Survey of the Galactic ant…
▽ More
Modern large-scale surveys have allowed the identification of large numbers of white dwarfs. However, these surveys are subject to complicated target selection algorithms, which make it almost impossible to quantify to what extent the observational biases affect the observed populations. The LAMOST (Large Sky Area Multi-Object Fiber Spectroscopic Telescope) Spectroscopic Survey of the Galactic anti-center (LSS-GAC) follows a well-defined set of criteria for selecting targets for observations. This advantage over previous surveys has been fully exploited here to identify a small yet well-characterised magnitude-limited sample of hydrogen-rich (DA) white dwarfs. We derive preliminary LSS-GAC DA white dwarf luminosity and mass functions. The space density and average formation rate of DA white dwarfs we derive are 0.83+/-0.16 x 10^{-3} pc^{-3} and 5.42 +/- 0.08 x 10^{-13} pc^{-3} yr^{-1}, respectively. Additionally, using an existing Monte Carlo population synthesis code we simulate the population of single DA white dwarfs in the Galactic anti-center, under various assumptions. The synthetic populations are passed through the LSS-GAC selection criteria, taking into account all possible observational biases. This allows us to perform a meaningful comparison of the observed and simulated distributions. We find that the LSS-GAC set of criteria is highly efficient in selecting white dwarfs for spectroscopic observations (80-85 per cent) and that, overall, our simulations reproduce well the observed luminosity function. However, they fail at reproducing an excess of massive white dwarfs present in the observed mass function. A plausible explanation for this is that a sizable fraction of massive white dwarfs in the Galaxy are the product of white dwarf-white dwarf mergers.
△ Less
Submitted 18 March, 2015;
originally announced March 2015.
-
The effects of metallicity on the Galactic disk population of white dwarfs
Authors:
Ruxandra Cojocaru,
Santiago Torres,
Jordi Isern,
Enrique García-Berro
Abstract:
It has been known for a long time that stellar metallicity plays a significant role in the determination of the ages of the different Galactic stellar populations, when main sequence evolutionary tracks are employed. Here we analyze the role that metallicity plays on the white dwarf luminosity function of the Galactic disk, which is often used to determine its age. We employ a Monte Carlo populati…
▽ More
It has been known for a long time that stellar metallicity plays a significant role in the determination of the ages of the different Galactic stellar populations, when main sequence evolutionary tracks are employed. Here we analyze the role that metallicity plays on the white dwarf luminosity function of the Galactic disk, which is often used to determine its age. We employ a Monte Carlo population synthesis code that accounts for the properties of the population of Galactic disk white dwarfs. Our code incorporates the most up-to-date evolutionary cooling sequences for white dwarfs with hydrogen-rich and hydrogen-deficient atmospheres for both carbon-oxygen and oxygen-neon cores. We use two different models to assess the evolution of the metallicity, one in which the adopted metallicity is constant with time, but with a moderate dispersion, and a second one in which the metallicity increases with time. We found that our theoretical results are in a very satisfactory agreement with the observational luminosity functions obtained from the Sloan Digital Sky Survey (SDSS) and from the SuperCOSMOS Sky Survey (SSS), independently of the adopted age-metallicity law. In particular, we found that the age-metallicity law has no noticeable impact in shaping the bright branch of the white dwarf luminosity function, and that the position of its cut-off is almost insensitive to the adopoted age-metallicity relationship. Because the shape of the bright branch of the white dwarf luminosity function is insensitive to the age-metallicity law, it can be safely employed to test the theoretical evolutionary sequences, while due to the limited sensitivity of the position of the drop-off to the distribution of metallicities, its location provides a robust indicator of the age of the Galactic disk.
△ Less
Submitted 29 April, 2014;
originally announced April 2014.
