Accelerating the availability of COVID-19 vaccines is critical to preventing further waves and mi... more Accelerating the availability of COVID-19 vaccines is critical to preventing further waves and mitigating the impact on society. However, preparations for large-scale manufacturing, such as building production facilities, are typically delayed until a vaccine is proven safe and effective. This makes sense from a commercial perspective, but incurs great costs in terms of lives lost and damage to the economy. Several policy options are available to reduce this delay, all of which involve incentives or subsidies to invest in production facilities. We review existing approaches, then propose a novel alternative using “option-based guarantees” in which the government commits to paying a proportion of the manufacturer’s preparation costs should the product turn out not to be viable. Counterintuitively, this “payment for failure” is appropriate because in the case of success, a company makes a profit from the product itself, and does not need additional money from the government. While oth...
Human challenge trials (HCTs) have been proposed as a means to accelerate SARS-CoV-2 vaccine deve... more Human challenge trials (HCTs) have been proposed as a means to accelerate SARS-CoV-2 vaccine development. We identify and discuss 3 potential use cases of HCTs in the current pandemic: evaluating efficacy, converging on correlates of protection, and improving understanding of pathogenesis and the human immune response. We outline the limitations of HCTs and find that HCTs are likely to be most useful for vaccine candidates currently in preclinical stages of development. We conclude that, while currently limited in their application, there are scenarios in which HCTs would be extremely beneficial. Therefore, the option of conducting HCTs to accelerate SARS-CoV-2 vaccine development should be preserved. As HCTs require many months of preparation, we recommend an immediate effort to (1) establish guidelines for HCTs for COVID-19; (2) take the first steps toward HCTs, including preparing challenge virus and making preliminary logistical arrangements; and (3) commit to periodically re-ev...
This paper considers how health education organizations in the World Health Organization's Va... more This paper considers how health education organizations in the World Health Organization's Vaccine Safety Network (VSN) use Twitter to communicate about vaccines with the public, and whether they answer questions and engage in conversations. Almost no research in public health, to our knowledge, has explored conversational structure on social media among posts sent by different accounts. Starting with 1,017,176 tweets by relevant users, we constructed two corpuses of multi-tweet conversations. The first was 1,814 conversations that included VSN members directly, while the second was 2,283 conversations mentioning vaccines or vaccine denialism. The tweets and user metadata was then analyzed using an adaptation of Rhetorical Structure Theory. In the studied data, VSN members tweeted 12,677 times within conversations, compared to their 37,587 lone tweets. Their conversations were shorter than those in the comparison corpus (P < 0.0001), and they were involved in fewer multilogue...
Existing compartmental mathematical modelling methods for epidemics, such as SEIR models, cannot ... more Existing compartmental mathematical modelling methods for epidemics, such as SEIR models, cannot accurately represent effects of contact tracing. This makes them inappropriate for evaluating testing and contact tracing strategies to contain an outbreak. An alternative used in practice is the application of agent- or individual-based models (ABM). However ABMs are complex, less well-understood and much more computationally expensive. This paper presents a new method for accurately including the effects of Testing, contact-Tracing and Isolation (TTI) strategies in standard compartmental models. We derive our method using a careful probabilistic argument to show how contact tracing at the individual level is reflected in aggregate on the population level. We show that the resultant SEIR-TTI model accurately approximates the behaviour of a mechanistic agent-based model at far less computational cost. The computationally efficiency is such that it be easily and cheaply used for explorato...
Recently, human challenge trials (HCTs) have been proposed as a means to accelerate the developme... more Recently, human challenge trials (HCTs) have been proposed as a means to accelerate the development of an effective SARS-CoV-2 vaccine. In this paper, we discuss the potential role for such studies in the current COVID-19 pandemic. First, we present three scenarios in which HCTs could be useful: evaluating efficacy, converging on correlates of protection, and improving understanding of pathogenesis and the human immune response. We go on to outline the practical limitations of HCTs in these scenarios. We conclude that, while currently limited in their application, there are scenarios in which HCTs would be vastly beneficial and, thus, the option of using HCTs to accelerate COVID-19 vaccine development should be preserved. To this end, we recommend an immediate, coordinated effort by all stakeholders to (1) establish ethical and practical guidelines for the use of HCTs for COVID-19; (2) take the first steps toward an HCT, including preparing challenge virus under GMP and making preli...
A wide range of research has promised new tools for forecasting infectious disease dynamics, but ... more A wide range of research has promised new tools for forecasting infectious disease dynamics, but little of that research is currently being applied in practice, because tools do not address key public health needs, do not produce probabilistic forecasts, have not been evaluated on external data, or do not provide sufficient forecast skill to be useful. We developed an open collaborative forecasting challenge to assess probabilistic forecasts for seasonal epidemics of dengue, a major global public health problem. Sixteen teams used a variety of methods and data to generate forecasts for 3 epidemiological targets (peak incidence, the week of the peak, and total incidence) over 8 dengue seasons in Iquitos, Peru and San Juan, Puerto Rico. Forecast skill was highly variable across teams and targets. While numerous forecasts showed high skill for midseason situational awareness, early season skill was low, and skill was generally lowest for high incidence seasons, those for which forecast...
Biosurveillance “systematically collects and analyzes data for the purpose of detecting cases of ... more Biosurveillance “systematically collects and analyzes data for the purpose of detecting cases of disease, [and] outbreaks of disease.” (Wagner, Moore and Aryel, 2006) This typically involves using a set of known sources of epidemiological data, instead of opportunistically using the data sources which become available over time. This work attempts to partially remedy that limitation by using an easily adapted generative Bayesian econometric model to allow incorporation of novel data sources. This is done by building a generative model of the information sources, then using Bayesian Markov-chain Monte-Carlo to find the relationships between data and actual caseloads to use in an epidemiological model 1. While the application presented is limited to three data sources for a single disease (influenza), the methodology is potentially widely applicable, and enables rapid incorporation of a variety of sources and source types.
An important challenge for safety in machine learning and artificial intelligence systems is a se... more An important challenge for safety in machine learning and artificial intelligence systems is a set of related failures involving specification gaming, reward hacking, fragility to distributional shifts, and Goodhart’s or Campbell’s law. This paper presents additional failure modes for interactions within multi-agent systems that are closely related. These multi-agent failure modes are more complex, more problematic, and less well understood than the single-agent case, and are also already occurring, largely unnoticed. After motivating the discussion with examples from poker-playing artificial intelligence (AI), the paper explains why these failure modes are in some senses unavoidable. Following this, the paper categorizes failure modes, provides definitions, and cites examples for each of the modes: accidental steering, coordination failures, adversarial misalignment, input spoofing and filtering, and goal co-option or direct hacking. The paper then discusses how extant literature o...
In response to recommendations to redefine statistical significance to p ≤ .005, we propose that ... more In response to recommendations to redefine statistical significance to p ≤ .005, we propose that researchers should transparently report and justify all choices they make when designing a study, including the alpha level.
Accelerating the availability of COVID-19 vaccines is critical to preventing further waves and mi... more Accelerating the availability of COVID-19 vaccines is critical to preventing further waves and mitigating the impact on society. However, preparations for large-scale manufacturing, such as building production facilities, are typically delayed until a vaccine is proven safe and effective. This makes sense from a commercial perspective, but incurs great costs in terms of lives lost and damage to the economy. Several policy options are available to reduce this delay, all of which involve incentives or subsidies to invest in production facilities. We review existing approaches, then propose a novel alternative using “option-based guarantees” in which the government commits to paying a proportion of the manufacturer’s preparation costs should the product turn out not to be viable. Counterintuitively, this “payment for failure” is appropriate because in the case of success, a company makes a profit from the product itself, and does not need additional money from the government. While oth...
Human challenge trials (HCTs) have been proposed as a means to accelerate SARS-CoV-2 vaccine deve... more Human challenge trials (HCTs) have been proposed as a means to accelerate SARS-CoV-2 vaccine development. We identify and discuss 3 potential use cases of HCTs in the current pandemic: evaluating efficacy, converging on correlates of protection, and improving understanding of pathogenesis and the human immune response. We outline the limitations of HCTs and find that HCTs are likely to be most useful for vaccine candidates currently in preclinical stages of development. We conclude that, while currently limited in their application, there are scenarios in which HCTs would be extremely beneficial. Therefore, the option of conducting HCTs to accelerate SARS-CoV-2 vaccine development should be preserved. As HCTs require many months of preparation, we recommend an immediate effort to (1) establish guidelines for HCTs for COVID-19; (2) take the first steps toward HCTs, including preparing challenge virus and making preliminary logistical arrangements; and (3) commit to periodically re-ev...
This paper considers how health education organizations in the World Health Organization's Va... more This paper considers how health education organizations in the World Health Organization's Vaccine Safety Network (VSN) use Twitter to communicate about vaccines with the public, and whether they answer questions and engage in conversations. Almost no research in public health, to our knowledge, has explored conversational structure on social media among posts sent by different accounts. Starting with 1,017,176 tweets by relevant users, we constructed two corpuses of multi-tweet conversations. The first was 1,814 conversations that included VSN members directly, while the second was 2,283 conversations mentioning vaccines or vaccine denialism. The tweets and user metadata was then analyzed using an adaptation of Rhetorical Structure Theory. In the studied data, VSN members tweeted 12,677 times within conversations, compared to their 37,587 lone tweets. Their conversations were shorter than those in the comparison corpus (P < 0.0001), and they were involved in fewer multilogue...
Existing compartmental mathematical modelling methods for epidemics, such as SEIR models, cannot ... more Existing compartmental mathematical modelling methods for epidemics, such as SEIR models, cannot accurately represent effects of contact tracing. This makes them inappropriate for evaluating testing and contact tracing strategies to contain an outbreak. An alternative used in practice is the application of agent- or individual-based models (ABM). However ABMs are complex, less well-understood and much more computationally expensive. This paper presents a new method for accurately including the effects of Testing, contact-Tracing and Isolation (TTI) strategies in standard compartmental models. We derive our method using a careful probabilistic argument to show how contact tracing at the individual level is reflected in aggregate on the population level. We show that the resultant SEIR-TTI model accurately approximates the behaviour of a mechanistic agent-based model at far less computational cost. The computationally efficiency is such that it be easily and cheaply used for explorato...
Recently, human challenge trials (HCTs) have been proposed as a means to accelerate the developme... more Recently, human challenge trials (HCTs) have been proposed as a means to accelerate the development of an effective SARS-CoV-2 vaccine. In this paper, we discuss the potential role for such studies in the current COVID-19 pandemic. First, we present three scenarios in which HCTs could be useful: evaluating efficacy, converging on correlates of protection, and improving understanding of pathogenesis and the human immune response. We go on to outline the practical limitations of HCTs in these scenarios. We conclude that, while currently limited in their application, there are scenarios in which HCTs would be vastly beneficial and, thus, the option of using HCTs to accelerate COVID-19 vaccine development should be preserved. To this end, we recommend an immediate, coordinated effort by all stakeholders to (1) establish ethical and practical guidelines for the use of HCTs for COVID-19; (2) take the first steps toward an HCT, including preparing challenge virus under GMP and making preli...
A wide range of research has promised new tools for forecasting infectious disease dynamics, but ... more A wide range of research has promised new tools for forecasting infectious disease dynamics, but little of that research is currently being applied in practice, because tools do not address key public health needs, do not produce probabilistic forecasts, have not been evaluated on external data, or do not provide sufficient forecast skill to be useful. We developed an open collaborative forecasting challenge to assess probabilistic forecasts for seasonal epidemics of dengue, a major global public health problem. Sixteen teams used a variety of methods and data to generate forecasts for 3 epidemiological targets (peak incidence, the week of the peak, and total incidence) over 8 dengue seasons in Iquitos, Peru and San Juan, Puerto Rico. Forecast skill was highly variable across teams and targets. While numerous forecasts showed high skill for midseason situational awareness, early season skill was low, and skill was generally lowest for high incidence seasons, those for which forecast...
Biosurveillance “systematically collects and analyzes data for the purpose of detecting cases of ... more Biosurveillance “systematically collects and analyzes data for the purpose of detecting cases of disease, [and] outbreaks of disease.” (Wagner, Moore and Aryel, 2006) This typically involves using a set of known sources of epidemiological data, instead of opportunistically using the data sources which become available over time. This work attempts to partially remedy that limitation by using an easily adapted generative Bayesian econometric model to allow incorporation of novel data sources. This is done by building a generative model of the information sources, then using Bayesian Markov-chain Monte-Carlo to find the relationships between data and actual caseloads to use in an epidemiological model 1. While the application presented is limited to three data sources for a single disease (influenza), the methodology is potentially widely applicable, and enables rapid incorporation of a variety of sources and source types.
An important challenge for safety in machine learning and artificial intelligence systems is a se... more An important challenge for safety in machine learning and artificial intelligence systems is a set of related failures involving specification gaming, reward hacking, fragility to distributional shifts, and Goodhart’s or Campbell’s law. This paper presents additional failure modes for interactions within multi-agent systems that are closely related. These multi-agent failure modes are more complex, more problematic, and less well understood than the single-agent case, and are also already occurring, largely unnoticed. After motivating the discussion with examples from poker-playing artificial intelligence (AI), the paper explains why these failure modes are in some senses unavoidable. Following this, the paper categorizes failure modes, provides definitions, and cites examples for each of the modes: accidental steering, coordination failures, adversarial misalignment, input spoofing and filtering, and goal co-option or direct hacking. The paper then discusses how extant literature o...
In response to recommendations to redefine statistical significance to p ≤ .005, we propose that ... more In response to recommendations to redefine statistical significance to p ≤ .005, we propose that researchers should transparently report and justify all choices they make when designing a study, including the alpha level.
There are potentially promising mitigation activities for epidemic and pandemic scenarios that ar... more There are potentially promising mitigation activities for epidemic and pandemic scenarios that are not currently the subject of significant research effort. Large epidemics and pandemics pose risks that are important to mitigate, even if the likelihood of the events is low and uncertain. While some efforts are the subject of extensive funding and consideration, other approaches are neglected. Here, we consider such neglected interventions which could significantly reduce the impact of such an epidemic or large-scale pandemic. These are identified via a narrative literature review of extant literature reviews and overviews of mitigations in epidemic and pandemic situations, followed by consideration of the economic value of information of further study of heretofore neglected interventions and approaches. Based on that analysis, we considered several classes of mitigations, and conducted more exploratory reviews of each. Those discussed include mitigations for (i) reducing transmission, such as personal protective equipment and encouraging improved hygiene, (ii) reducing exposure by changing norms and targeted changes for high-risk or critical professions and activities, (iii) reducing impact for those infected, and (iv) increasing large scale resilience using disaster and infrastructure continuity planning. Some proposed mitigations are found to be of low marginal value. Other mitigations are likely to be valuable, but the concepts or applications are underdeveloped. In those cases, further research, resources, or preparation are valuable for mitigating both routine and extreme disease outbreak events. Still more areas of research are identified as having uncertain value based on specific but resolvable uncertainties. In both of the latter cases, there is no guarantee that mitigations identified as worthy of further consideration will be valuable, but the argument for further research is clear.
Uploads
Research by David Manheim
Papers by David Manheim
consideration, other approaches are neglected. Here, we consider such neglected interventions which could significantly reduce the impact of such an epidemic or large-scale pandemic. These are identified via a narrative literature review of extant literature reviews and overviews of mitigations in epidemic and pandemic situations, followed by consideration of the economic value of information of further study of
heretofore neglected interventions and approaches. Based on that analysis, we considered several classes of mitigations, and conducted more exploratory reviews of each. Those discussed include mitigations for (i) reducing transmission, such as personal protective equipment and encouraging improved hygiene, (ii) reducing exposure by changing norms and targeted changes for high-risk or critical professions and activities, (iii) reducing impact for those infected, and (iv) increasing large scale resilience using disaster and infrastructure continuity planning. Some proposed mitigations are found to be of low marginal value. Other mitigations are likely to be valuable, but the concepts or
applications are underdeveloped. In those cases, further research, resources, or preparation are valuable for mitigating both routine and extreme disease outbreak events. Still more areas of research are identified as having uncertain value based on specific but resolvable uncertainties. In both of the latter cases, there is no guarantee that mitigations identified as worthy of further consideration will be valuable, but the argument for further research is clear.