Showing 1–13 of 13 results for author: Heiner, S

Discovering Language Model Behaviors with Model-Written Evaluations

Authors: Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion , et al. (38 additional authors not shown)

Abstract: As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from inst… ▽ More As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer ("sycophancy") and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors. △ Less

Submitted 19 December, 2022; originally announced December 2022.

Comments: for associated data visualizations, see https://www.evals.anthropic.com/model-written/ for full datasets, see https://github.com/anthropics/evals

Measuring Progress on Scalable Oversight for Large Language Models

Authors: Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse , et al. (21 additional authors not shown)

Abstract: Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think abou… ▽ More Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think about this problem, with a focus on ways it can be studied empirically. We first present an experimental design centered on tasks for which human specialists succeed but unaided humans and current general AI systems fail. We then present a proof-of-concept experiment meant to demonstrate a key feature of this experimental design and show its viability with two question-answering tasks: MMLU and time-limited QuALITY. On these tasks, we find that human participants who interact with an unreliable large-language-model dialog assistant through chat -- a trivial baseline strategy for scalable oversight -- substantially outperform both the model alone and their own unaided performance. These results are an encouraging sign that scalable oversight will be tractable to study with present models and bolster recent findings that large language models can productively assist humans with difficult tasks. △ Less

Submitted 11 November, 2022; v1 submitted 4 November, 2022; originally announced November 2022.

Comments: v2 fixes a few typos from v1

Molecular cloud formation as seen in synthetic Hi and molecular gas observations

Authors: Jonathan S. Heiner, Enrique Vázquez-Semadeni, Javier Ballesteros-Paredes

Abstract: We present synthetic Hi and CO observations of a simulation of decaying turbulence in the thermally bistable neutral medium. We first present the simulation, with clouds initially consisting of clustered clumps. Self-gravity causes these clump clusters to form more homogeneous dense clouds. We apply a simple radiative transfer algorithm, and defining every cell with <Av> > 1 as molecular. We then… ▽ More We present synthetic Hi and CO observations of a simulation of decaying turbulence in the thermally bistable neutral medium. We first present the simulation, with clouds initially consisting of clustered clumps. Self-gravity causes these clump clusters to form more homogeneous dense clouds. We apply a simple radiative transfer algorithm, and defining every cell with <Av> > 1 as molecular. We then produce maps of Hi, CO-free molecular gas, and CO, and investigate the following aspects: i) The spatial distribution of the warm, cold, and molecular gas, finding the well-known layered structure, with molecular gas surrounded by cold Hi, surrounded by warm Hi. ii) The velocity of the various components, with atomic gas generally flowing towards the molecular gas, and that this motion is reflected in the frequently observed bimodal shape of the Hi profiles. This conclusion is tentative, because we do not include feedback. iii) The production of Hi self-absorption (HISA) profiles, and the correlation of HISA with molecular gas. We test the suggestion of using the second derivative of the brightness temperature Hi profile to trace HISA and molecular gas, finding limitations. On a scale of ~parsecs, some agreement is obtained between this technique and actual HISA, as well as a correlation between HISA and N(mol). It quickly deteriorates towards sub-parsec scales. iv) The N-PDFs of the actual Hi gas and those recovered from the Hi line profiles, with the latter having a cutoff at column densities where the gas becomes optically thick, thus missing the contribution from the HISA-producing gas. We find that the power-law tail typical of gravitational contraction is only observed in the molecular gas, and that, before the power-law tail develops in the total gas density PDF, no CO is yet present, reinforcing the notion that gravitational contraction is needed to produce this component. (abridged) △ Less

Submitted 25 March, 2014; originally announced March 2014.

Comments: Submitted to MNRAS - comments are welcome. 22 pages, 21 figures

Applying a one-dimensional PDR model to the Taurus molecular cloud and its atomic envelope

Authors: J. S. Heiner, E. Vázquez-Semadeni

Abstract: In this contribution, we test our previously published one-dimensional PDR model for deriving total hydrogen volume densities from HI column density measurements in extragalactic regions by applying it to the Taurus molecular cloud, where its predictions can be compared to available data. Also, we make the first direct detailed comparison of our model to CO(1-0) and far-infrared emission. Using an… ▽ More In this contribution, we test our previously published one-dimensional PDR model for deriving total hydrogen volume densities from HI column density measurements in extragalactic regions by applying it to the Taurus molecular cloud, where its predictions can be compared to available data. Also, we make the first direct detailed comparison of our model to CO(1-0) and far-infrared emission. Using an incident UV flux G0 of 4.25 (χ = 5) throughout the main body of the cloud, we derive total hydrogen volume densities of \approx 430 cm-3, consistent with the extensive literature available on Taurus. The distribution of the volume densities shows a log-normal shape with a hint of a power-law shape on the high density end. We convert our volume densities to H2 column densities assuming a cloud depth of 5 parsec and compare these column densities to observed CO emission. We find a slope equivalent to a CO conversion factor relation that is on the low end of reported values for this factor in the literature (0.9 x 1020 cm-2 (K km s-1)-1), although this value is directly proportional to our assumed value of G0 as well as the cloud depth. We seem to under-predict the total hydrogen gas as compared to 100 μm dust emission, which we speculate may be caused by a higher actual G0 incident on the Taurus cloud than is generally assumed. △ Less

Submitted 17 December, 2012; originally announced December 2012.

Comments: 14 pages, 14 figures. Accepted for publication in MNRAS

Hydrogen volume densities in nearby galaxies I - an automated approach

Authors: J. S. Heiner, J. R. Sánchez-Gallego, L. Rousseau-Nepton, J. H. Knapen

Abstract: Using a simple model of photodissociated atomic hydrogen on a galactic scale, it is possible to derive total hydrogen volume densities. These densities, obtained through a combination of atomic hydrogen, far-ultraviolet and metallicity data, provide an independent probe of the combined atomic and molecular hydrogen gas in galactic disks. We present a new, flexible and fully automated procedure usi… ▽ More Using a simple model of photodissociated atomic hydrogen on a galactic scale, it is possible to derive total hydrogen volume densities. These densities, obtained through a combination of atomic hydrogen, far-ultraviolet and metallicity data, provide an independent probe of the combined atomic and molecular hydrogen gas in galactic disks. We present a new, flexible and fully automated procedure using this simple model. This automated method will allow us to take full advantage of a host of available data on galaxies in order to calculate total hydrogen volume densities of giant molecular clouds surrounding sites of recent star formation. So far this was only possible on a galaxy-by-galaxy basis using by-eye analysis of candidate photodissociation regions. We test the automated method by adopting different models for the dust-to-gas ratio and comparing the resulting densities for M74, including a new metallicity map of M74 produced by integral field spectroscopy. We test the procedure against previously published M83 volume densities based on the same method and find no significant differences. The range of total hydrogen volume densities obtained for M74 is approximately 5-700 cm-3 . Different dust-to-gas ratio models do not result in measurably different densities. The cloud densities presented here add M74 to the list of galaxies analyzed using the assumption of photodissociated atomic hydrogen occurring near sites of recent star formation and further solidify the method. For the first time, full metallicity maps were included in the analysis as opposed to metallicity gradients. The results will need to be compared to other tracers of the interstellar medium and photodissociation regions, such as CO and CII, in order to test our basic assumptions, specifically, our assumption that the HI we detect originates in photodissociation regions. △ Less

Submitted 12 November, 2012; v1 submitted 23 October, 2012; originally announced October 2012.

Comments: Added funding acknowledgement and corrected a few critical typos

Quantified H i Morphology VII: The Morphology of Extended Disks in UV and H i

Authors: B. W. Holwerda, N. Pirzkal, J. S. Heiner

Abstract: Extended UltraViolet (xuv) disks have been found in a substantial fraction of late-type --S0, spiral and irregular-- galaxies. Similarly, most late-type spirals have an extended gas disk, observable in the 21cm radio line (HI). The morphology of galaxies can be quantified well using a series of scale-invariant parameters; Concentration- Asymmetry-Smoothness (CAS), Gini, M20, and GM parameters. In… ▽ More Extended UltraViolet (xuv) disks have been found in a substantial fraction of late-type --S0, spiral and irregular-- galaxies. Similarly, most late-type spirals have an extended gas disk, observable in the 21cm radio line (HI). The morphology of galaxies can be quantified well using a series of scale-invariant parameters; Concentration- Asymmetry-Smoothness (CAS), Gini, M20, and GM parameters. In this paper, we compare the quantified morphology and effective radius (R50) of the Westerbork observations of neutral Hydrogen in Irregular and Spiral galaxies Project (WHISP) HI maps to those of far-and near-ultraviolet images obtained with galex, to explore how close the morphology and scales of HI and UV in these disks correlate. We find that xuv disks do not stand out by their effective radii in UV or HI. However, the concentration index in FUV appears to select some xuv disks. And known xuv disks can be identified via a criterion using Asymmetry and M20; 80% of xuv disks are included but with 55% contamination. This translates into 61 candidate xuv disk out of our 266 galaxies, (23%) consistent with previous findings. We consider three scenarios; tidal features from major mergers, the typical extended Hi disk is a photo- dissociation product of the xuv regions and both Hi and UV features originate in cold flows fueling the main galaxy. We define extended HI and UV disks based on their concentration (CHI > 5 and CFUV > 4 respectively), but note that these two subsamples never overlap in the WHISP sample. This appears to discount a simple photo-dissociation origin of the outer HI disk. Previously, we identified the morphology space occupied by ongoing major mergers. Known xuv disks rarely reside in the merger dominated part of HI morphology space but those that do are Type 1. This suggests cold flows as the origin for the xuv complexes and their surrounding HI structures. △ Less

Submitted 20 July, 2012; originally announced July 2012.

Comments: 18 pages, 13 figures, 2 tables

Atomic Hydrogen produced in M33 Photodissociation Regions

Authors: J. S. Heiner, R. J. Allen, P. C. van der Kruit

Abstract: We derive total (atomic + molecular) hydrogen densities in giant molecular clouds (GMCs) in the nearby spiral galaxy M33 using a method that views the atomic hydrogen near regions of recent star formation as the product of photodissociation. Far-UV photons emanating from a nearby OB association produce a layer of atomic hydrogen on the surfaces of nearby GMCs. Our approach provides an estimate of… ▽ More We derive total (atomic + molecular) hydrogen densities in giant molecular clouds (GMCs) in the nearby spiral galaxy M33 using a method that views the atomic hydrogen near regions of recent star formation as the product of photodissociation. Far-UV photons emanating from a nearby OB association produce a layer of atomic hydrogen on the surfaces of nearby GMCs. Our approach provides an estimate of the total hydrogen density in these GMCs from observations of the excess far-UV emission that reaches the GMC from the OB association, and the excess 21-cm radio HI emission produced after these far-UV photons convert H2 into HI on the GMC surface. The method provides an alternative approach to the use of CO emission as a tracer of H2 in GMCs, and is especially sensitive to a range of density well below the critical density for CO(1-0) emission. We describe our "PDR method" in more detail and apply it using GALEX far-UV and VLA 21-cm radio data to obtain volume densities in a selection of GMCs in the nearby spiral galaxy M33. We have also examined the sensitivity of the method to the linear resolution of the observations used; the results obtained at 20 pc are similar to those for the larger set of data at 80 pc resolution. The cloud densities we derive range from 1 to 500 cm-3, with no clear dependence on galactocentric radius; these results are generally similar to those obtained earlier in M81, M83, and M101 using the same method. △ Less

Submitted 23 February, 2011; originally announced February 2011.

Comments: Accepted for publication in MNRAS. 25 figures, 16 tables, including online-only materials

The JCMT Nearby Galaxies Legacy Survey VI: The distribution of gas and star formation in M81

Authors: J. R. Sánchez-Gallego, J. H. Knapen, J. S. Heiner, C. D. Wilson, B. E. Warren, R. J. Allen, M. Azimlu, P. Barmby, G. J. Bendo, S. Comerón, F. P. Israel, S. Serjeant, R. P. J. Tilanus, C. Vlahakis, P. van der Werf

Abstract: We present a first complete 12 CO J=3-2 map of M81, observed as part of the Nearby Galaxies Legacy Survey being carried out at the James Clerk Maxwell Telescope. We detect 9 regions of significant CO emission located at different positions within the spiral arms, and confirm that the global CO emission in the galaxy is low. We combine these data with a new H-alpha map obtained using the Isaac Newt… ▽ More We present a first complete 12 CO J=3-2 map of M81, observed as part of the Nearby Galaxies Legacy Survey being carried out at the James Clerk Maxwell Telescope. We detect 9 regions of significant CO emission located at different positions within the spiral arms, and confirm that the global CO emission in the galaxy is low. We combine these data with a new H-alpha map obtained using the Isaac Newton Telescope and archival HI, 24 microns and FUV images to uncover a correlation between the molecular gas and star forming regions in M81. For the nine regions detected in CO J=3-2, we combine our CO J=3-2 data with existing CO J=1-0 data to calculate line ratios. We find that the ratio J=(3-2)/(1-0) is in agreement with the range of typical values found in the literature (0.2-0.8). Making reasonable assumptions, this allows us to constrain the hydrogen density to the range (10^3-10^4) cm^{-3}. We also estimated the amount of hydrogen produced in photo-dissociation regions near the locations where CO J=3-2 was detected. △ Less

Submitted 19 December, 2010; v1 submitted 23 November, 2010; originally announced November 2010.

Comments: Accepted for publication in Astronomy & Astrophysics. 11 pages, 4 figures

A new method for obtaining the star formation law in galaxies

Authors: J. S. Heiner, R. J. Allen, P. C. van der Kruit

Abstract: We present a new observational method to evaluate the star formation law as formulated by Schmidt: the power-law expression assumed to relate the rate of star formation in a volume of space to the local total gas volume density. Volume densities in the clouds surrounding an OB association are determined with a simple model which considers atomic hydrogen as a photodissociation product on cloud sur… ▽ More We present a new observational method to evaluate the star formation law as formulated by Schmidt: the power-law expression assumed to relate the rate of star formation in a volume of space to the local total gas volume density. Volume densities in the clouds surrounding an OB association are determined with a simple model which considers atomic hydrogen as a photodissociation product on cloud surfaces. The photodissociating flux incident on the cloud is computed from the far-UV luminosity of the OB association and the geometry. We have applied this "PDR Method" to a sample of star-forming regions in M33 using VLA 21-cm data for the HI and GALEX imagery in the far-UV. It provides an estimate of the total volume density of hydrogen (atomic + molecular) in the gas clouds surrounding the young star cluster. A logarithmic graph of the cluster UV luminosity versus the surrounding gas density is a direct measure of the star formation law. However, this plot is severely affected by observational selection, rendering large areas of the diagram inaccessible to the data. An ordinary least-squares regression fit therefore gives a strongly biased result. Its slope primarily reflects the boundary defined when the 21-cm line becomes optically thick, no longer reliably measuring the HI column density. We use a maximum-likelihood statistical approach which can deal with truncated and skewed data, taking into account the large uncertainties in the derived total gas densities. The exponent we obtain for the Schmidt law in M33 is 1.4 \pm 0.2. △ Less

Submitted 25 June, 2010; originally announced June 2010.

Comments: Accepted for publication in ApJ

A photodissociation model for the morphology of the HI near OB associations in M33

Authors: J. S. Heiner, R. J. Allen, P. C. van der Kruit

Abstract: We present an approach for analysing the morphology and physical properties of Hi features near giant OB asso- ciations in M33, in the context of a model whereby the Hi excess arises from photodissociation of the molecular gas in remnants of the parent Giant Molecular Clouds (GMCs). Examples are presented here in the environs of NGC604 and CPSDPZ204, two prominent Hii regions in M33. These are t… ▽ More We present an approach for analysing the morphology and physical properties of Hi features near giant OB asso- ciations in M33, in the context of a model whereby the Hi excess arises from photodissociation of the molecular gas in remnants of the parent Giant Molecular Clouds (GMCs). Examples are presented here in the environs of NGC604 and CPSDPZ204, two prominent Hii regions in M33. These are the first results of a detailed analysis of the environs of a large number of OB associations in that galaxy. We present evidence for "diffusion" of the far-UV radiation from the OB association through a clumpy remnant GMC, and show further that enhanced CO(1-0) emission appears preferentially associated with GMCs of higher volume density. △ Less

Submitted 23 May, 2009; originally announced May 2009.

Comments: Accepted to ApJ

Report number: STScI E-print #1806

Journal ref: Astrophys.J.700:545-550,2009

The volume densities of giant molecular clouds in M83

Authors: Jonathan S. Heiner, Ronald J. Allen, O. Ivy Wong, Pieter C. van der Kruit

Abstract: Using observed GALEX far-ultraviolet (FUV) fluxes and VLA images of the 21-cm HI column densities, along with estimates of the local dust abundances, we measure the volume densities of a sample of actively star-forming giant molecular clouds (GMCs) in the nearby spiral galaxy M83 on a typical resolution scale of 170 pc. Our approach is based on an equilibrium model for the cycle of molecular hyd… ▽ More Using observed GALEX far-ultraviolet (FUV) fluxes and VLA images of the 21-cm HI column densities, along with estimates of the local dust abundances, we measure the volume densities of a sample of actively star-forming giant molecular clouds (GMCs) in the nearby spiral galaxy M83 on a typical resolution scale of 170 pc. Our approach is based on an equilibrium model for the cycle of molecular hydrogen formation on dust grains and photodissociation under the influence of the FUV radiation on the cloud surfaces of GMCs. We find a range of total volume densities on the surface of GMCs in M83, namely 0.1 - 400 cm-3 inside R25, 0.5 - 50 cm-3 outside R25 . Our data include a number of GMCs in the HI ring surrounding this galaxy. Finally, we discuss the effects of observational selection, which may bias our results. △ Less

Submitted 4 August, 2008; originally announced August 2008.

Comments: 9 pages, 11 figures

The volume densities of GMCs in M81

Authors: Jonathan S. Heiner, Ronald J. Allen, Bjorn H. C. Emonts, Pieter C. van der Kruit

Abstract: HI features near young star clusters in M81 are identified as the photodissociated surfaces of Giant Molecular Clouds (GMCs) from which the young stars have recently formed. The HI column densities of these features show a weak trend, from undetectable values inside R = 3.7 kpc and increasing rapidly to values around 3 x 10^21 cm^-2 near R ~ 7.5 kpc. This trend is similar to that of the radially… ▽ More HI features near young star clusters in M81 are identified as the photodissociated surfaces of Giant Molecular Clouds (GMCs) from which the young stars have recently formed. The HI column densities of these features show a weak trend, from undetectable values inside R = 3.7 kpc and increasing rapidly to values around 3 x 10^21 cm^-2 near R ~ 7.5 kpc. This trend is similar to that of the radially-averaged HI distribution in this galaxy, and implies a constant area covering factor of ~ 0.21 for GMCs throughout M81. The incident UV fluxes G0 of our sample of candidate PDRs decrease radially. A simple equilibrium model of the photodissociation-reformation process connects the observed values of the incident UV flux, the HI column density, and the relative dust content, permitting an independent estimate to be made of the total gas density in the GMC. Within the GMC this gas will be predominantly molecular hydrogen. Volume densities of 1 < n < 200 cm^-3 are derived, with a geometric mean of 17 cm^-3. These values are similar to the densities of GMCs in the Galaxy, but somewhat lower than those found earlier for M101 with similar methods. Low values of molecular density in the GMCs of M81 will result in low levels of collisional excitation of the CO(1-0) transition, and are consistent with the very low surface brightness of CO(1-0) emission observed in the disk of M81. △ Less

Submitted 5 October, 2007; originally announced October 2007.

Comments: 17 pages, 11 figures, 3 tables; Accepted by ApJ

Report number: STScI E-print #1780

Journal ref: Astrophys.J.673:798H,2008

NGC922 - A new drop-through ring galaxy

Authors: O. I. Wong, G. R. Meurer, K. Bekki, D. J. Hanish, R. C. Kennicutt, J. Bland-Hawthorn, E. V. Ryan-Weber, B. Koribalski, V. A. Kilborn, M. E. Putman, J. S. Heiner, R. L. Webster, R. J. Allen, M. A. Dopita, M. T. Doyle, M. J. Drinkwater, H. C. Ferguson, K. C. Freeman, T. M. Heckman, C. Hoopes, P. M. Knezek, M. J. Meyer, M. S. Oey, M. Seibert, R. C. Smith , et al. (4 additional authors not shown)

Abstract: We have found the peculiar galaxy NGC922 to be a new drop-through ring galaxy using multi-wavelength (UV-radio) imaging and spectroscopic observations. Its `C'-shaped morphology and tidal plume indicate a recent strong interaction with its companion which was identified with these observations. Using numerical simulations we demonstrate that the main properties of the system can be generated by… ▽ More We have found the peculiar galaxy NGC922 to be a new drop-through ring galaxy using multi-wavelength (UV-radio) imaging and spectroscopic observations. Its `C'-shaped morphology and tidal plume indicate a recent strong interaction with its companion which was identified with these observations. Using numerical simulations we demonstrate that the main properties of the system can be generated by a high-speed off-axis drop-through collision of a small galaxy with a larger disk system, thus making NGC922 one of the nearest known collisional ring galaxies. While these systems are rare in the local Universe, recent deep HST images suggest they were more common in the early Universe. △ Less

Submitted 24 May, 2006; originally announced May 2006.

Comments: 5 pages, accepted for publication in MNRAS Letters

Journal ref: Mon.Not.Roy.Astron.Soc.370:1607-1611,2006