Citizen Science, Accessibility, Art & Science, Critical Thinking, Policy and Engagement: Thoughts and Lessons Learned From The REINFORCE Experience

Eur. Phys. J.
Plus (2024) 139:868

https://doi.org/10.1140/epjp/s13360-024-05313-w
Review
Citizen science, accessibility, art & science, critical thinking, policy

and engagement: thoughts and lessons learned from the REINFORCE experience
Stylianos Angelidakis8 , Theodore Avgitas12 , Emmanouil Chaniotakis4 , Johanna Casado13 , Paschal Coyle15 ,
Gwenhaël de Wasseige5 , Francesco Di Renzo12 , Claudia Magdalena Fabian6 , Dimitrios Fassouliotis8 ,
Francesco Fidecaro2,3 , Beatriz Garcia7 , Gary Hemming1,a , Christine Kourkoumelis8,b , Rémy Le Breton17 ,
Jacques Marteau12 , Francesco Mureddu9 , Vincenzo Napolano1 , Francesco Osimanti10 , Enzo Oukacha16 ,
Maria Panagopoulou4 , James Pearson11 , Massimiliano Razzano2,3,c , Sofoklis Sotiriou4 , Stephen Serjeant11 ,
Francesca Spagnuolo1 , Elisabeth Unterfrauner6 , Stylianos Vourakis14
1 European Gravitational Observatory (EGO), Cascina, Pisa 56021, Italy
2 Università di Pisa, Pisa 56127, Italy
3 INFN, Sezione di Pisa, Pisa 56127, Italy
4 Ellinogermanki Agogi, 15351 Attica, Pallini, Greece
5 CP3, Institut de Mathématique et Physique, UCLouvain, Chemin du Cyclotron, 1348 Louvain-la-Neuve, Belgium
6 ZSI–Zentrum für Soziale Innovation, Linke Wienzeile 246, 1150 Wien, Austria
7 Instituto de Tecnologías en Detección y Astropartículas (CNEA, CONICET, UNSAM), Universidad Tecnológica Nacional, Facultad Regional Mendoza,
Mendoza, Argentina
8 Department of Physics, National and Kapodistrian University of Athens, 157 84 Athens, Greece
9 The Lisbon Council for Economic Competitiveness and Social Renewal, IPC-Résidence Palace, 155 rue de la loi, 1040 Brussels, Belgium
10 Trust-IT Srl, Via Francesco Redi 10, Apt. 11–12, 4th floor, Pisa 56124, Italy
11 School of Physical Sciences, The Open University, Walton Hall, Milton Keynes MK7 6AA, UK
12 Univ Lyon, Univ Claude Bernard Lyon 1, CNRS/IN2P3, IP2I Lyon, UMR 5822, 69622 Villeurbanne, France
13 Instituto de Tecnologías en Detección y Astropartículas (CNEA, CONICET, UNSAM), Universidad de Mendoza, Mendoza, Argentina
14 Institute for Accelerating Systems and Applications, University Campus, 10024 Athens, Greece
15 Centre de Physique des Particules de Marseille (CPPM), Marseille, France
16 Laboratoire Astroparticule et Cosmologie, 75013 Paris, France
17 Université Paris-Saclay, CEA, IRFU, 91191 Gif-sur-Yvette, France
Received: 23 January 2024 / Accepted: 24 May 2024

© The Author(s) 2024
Abstract REINFORCE (Research Infrastructures FOR Citizens in Europe) is a Research & Innovation Project, supported by the
European Union’s Horizon 2020 SwafS, ‘Science with and for Society’ work programme (GA872859). The project, which ran from
December, 2019, to November, 2022, engaged the public in a variety of innovative ways. Four citizen-science demonstrator projects
were developed on the world-leading Zooniverse platform, each focussing on a different area of frontier physics: gravitational waves;
neutrino astronomy; particle physics; and muography. A range of art and science events were launched and undertaken. A data-
sonification tool—sonoUno—was developed in order to improve the accessibility of the data used in the four demonstrator projects.
A course on critical thinking and a history of the Second Scientific Revolution was provided on YouTube and in podcast form,
while a senior-citizen-science course was designed, co-developed and implemented. These initiatives were supported by a detailed
engagement plan, a dedicated communications and dissemination strategy, and a constantly evolving assessment and evaluation
approach. The experiences garnered during the project, in conjunction with consultations with project participants, volunteers and
stakeholders, were built into the form of a policy roadmap explaining how to integrate citizen science into research infrastructures
in Europe. The roadmap identifies a series of policy objectives and related policy gaps, associated challenges and lays out a series
of recommendations. This article describes the results of the REINFORCE project and draws together the experiences of each of
the involved twelve partner organisations.
This article is dedicated to the memory of Stavros Katsanevas. Stavros was a brilliant scientist, professor and thinker. He was also the driving force behind
the REINFORCE project and was the Project Coordinator from its inception until the 27th of November, 2022, when he passed away; just three days before
the official conclusion of the project. Stavros was a friend to all in the collaboration and his open and inclusive nature can be felt, diffused throughout the
project as a whole.
a e-mail: gary.hemming@ego-gw.it (corresponding author)

b e-mail: christine.kourkoumelis@cern.ch (corresponding author)
c e-mail: massimiliano.razzano@unipi.it (corresponding author)
0123456789().: V,-vol 123

868 Page 2 of 38 Eur. Phys. J. Plus (2024) 139:868
1 Introduction
Large research infrastructures working in the field of frontier physics continue to open new observational windows on to the universe
and to explore the structure of matter in ever greater detail. As these discoveries become more and more sophisticated, so the expertise
required in order to understand them also increases. As a consequence, the public are often only able to access and interpret the
results of research through the prism of outreach activities, and are thus not able to contribute directly to the development of new
scientific knowledge. This distance contributes to a gap between science on one hand and society on the other; a gap that can be
problematic in the context of public-funded scientific research.
REINFORCE has aimed to co-design, along with citizen scientists themselves, an approach that breaks down the barriers to
access that contribute to the widening of this gap and to build communities of practice around four demonstrator projects on the
Zooniverse platform. Volunteers in these communities have contributed directly to the production of scientific knowledge and have
worked closely with different research teams to classify and understand data subjects in both quantitative and qualitative ways.
The REINFORCE project pursued a set of key aims: engage people to contribute to online frontier science; create a community
of citizens that actively participates in scientific endeavours; introduce responsible research and innovation into the frontier citizen
science landscape; assess the impact of frontier citizen science in science and society; create a policy roadmap for the implementation
of citizen-science projects in large research infrastructures (LRI); and explore the potential of frontier citizen science for inclusion
and diversity.
To reach these goals, a series of initiatives were developed and implemented, ranging from Zooniverse demonstrator projects,
to art and science initiatives, via courses on critical thinking and science for senior citizens, to the development of dedicated data-
sonification tools and courses. All of these initiatives were encompassed within clearly-defined engagement and dissemination
strategies and a thorough and wide-ranging evaluative approach.
The REINFORCE Collaboration is formed of twelve different partners — Centre national de la recherche scientifique (CNRS),
CONICET, Ellinogermaniki Agogi, European Gravitational Observatory (EGO), Institute of Accelerating Systems and Applications
(IASA), Open University, Oxford University, The Lisbon Council for Economic Competitiveness and Social Renewal, Trust-IT
Services, Université catholique de Louvain and Università di Pisa — from seven different countries across two continents —
Austria, Belgium, France, Greece, Italy, the United Kingdom and Argentina. The European Gravitational Observatory took on the
role of Project Coordinator.
The project was funded within the EU Science with and for Society (SwafS) work programme of the Research & Innovation
Project thematic area. Dedicated to the integration of society in science and innovation issues, policies and activities, it promotes the
integration of the interests and values of citizens in these areas with the aim of increasing the quality, relevance, social acceptability
and sustainability of research. REINFORCE also focussed on supporting two specific United Nations Sustainable Development
Goals (4. Quality Education and 5. Gender Equality).1
By developing the work of the project through participatory design, involving all members of the project, from partners to
volunteers, to stakeholders in other fields, the aim was to build a committed, diverse community of practice. As the project evolved,
the experiences garnered across its life cycle, fed into the development of a policy roadmap, designed to signpost issues and provide
recommendations for best practice in terms of the integration of citizen science in LRI across Europe.
This article looks first at the demonstrator projects on Zooniverse themselves. For each of them, a description of the project is
provided, as well as an examination of the successes; what worked well and why. Interaction with volunteers and the demonstrator
project community are also covered, as well as the impacts of the engagement strategy and activity implemented for each of them.
The report on each of the projects concludes with a look at the lessons learned, at the areas that might be approached differently
and the outcomes and future prospects for the long-term exploitation of the project and its resources. Following the demonstrator
projects, the article explores the effectiveness of approaches to increase accessibility to scientific research within the project and the
ways in which the effectiveness of these were understood and how they ultimately fed into the development of the policy roadmap,
which stands as a keystone resource at the end of the project.
2 Demonstrator projects on Zooniverse
At the core of REINFORCE were four demonstrator projects developed on the Zooniverse online platform for people-powered
research.2 On Zooniverse, volunteers contribute to research projects and participate in the study and classification of data. The ethos
behind the platform is that anyone can be a researcher; that working together can accelerate research and enable research that
otherwise would not be possible, as well as potentially leading to new discoveries.
1 https://sdgs.un.org/goals.
2 https://zooniverse.org/.
123
Eur. Phys. J. Plus (2024) 139:868 Page 3 of 38 868
Each of the four REINFORCE demonstrator projects centred on a different area of frontier physics. GWitchHunters3 introduces
volunteers to the data produced by the Virgo gravitational-wave detector to characterise its noise sources and improve its sensitivity.4
New Particle Search at CERN5 allows volunteers to work through different stages classifying displaced vertices; identifying the
signatures of known particles; searching for Higgs boson decays; and looking for long-lived particle decays, in data provided by the
ATLAS detector at the Large Hadron Collider.6 In Deep Sea Explorers,7 volunteers classify bioluminescence signals that constitute
noise in the KM3NeT neutrino detectors8 and identify bioacoustic signals recorded by hydrophones located at the detector sites,
in order to provide information useful for the study of the surrounding marine environment. In Cosmic Muon Images,9 volunteers
identify particle tracks as they passed through the different stages of a muon detector.
Memoranda of understanding between the REINFORCE collaboration and each of the individual collaborations providing data
for the demonstrator projects — Virgo, KM3Net and ATLAS — were agreed in the early stages of the project, which were all
initially developed only in English. Following the launch of each of them, however, it was agreed to also implement alternative
language versions, in order to render them more accessible and to better facilitate their dissemination across target groups and to
contribute to the engagement targets of the project. At time of writing, GWitchHunters is available in both English and Italian, with
the Spanish version due to go online shortly, while New Particle Search at CERN is available in English, Greek and Spanish.
2.1 GWitchHunters
The GWitchHunters project [1] focuses on supporting research in the field of gravitational waves through the contribution of citizen
scientists. Citizens are asked to contribute to the study and characterisation of the data recorded by the Advanced Virgo interferometer
[2]. Virgo is the main gravitational-wave detector in Europe and, together with Advanced LIGO [3] and KAGRA [4], forms part of
an international network of experiments for gravitational-wave detection. These detectors are extremely sensitive machines, and, as
such, need to be shielded as much as possible from potential sources of noise, which can produce spurious signals in the detector data
that can be much larger than those expected from waves produced by astrophysical sources. It is therefore of paramount importance
to properly characterise noise artefacts and to mitigate and possibly remove them. In particular, rapid transient noise events, known as
glitches, are particularly detrimental for the detectors, since they can affect the duty cycle and can mask or mimic real astrophysical
signals.
The characterisation of glitches is not trivial, however, as they appear arbitrarily in time and can exhibit very different types of
behaviour. When plotted on a time-frequency map representing the time evolution of their spectral energy content (spectrogram),
glitches can show diverse morphologies, and it is possible to group them into families.
This approach is based on the premise that, at the origin of the various glitch families, there are also distinct noise sources;
identifying all the times when some of these sources manifest is the first step in further pursuing their identification for the
subsequent implementation of mitigation strategies. Given the high rate of these glitches, and their large number over entire data-
taking periods, studying them one by one can be very time-consuming. Machine learning (ML) is therefore of great help in their
rapid identification, as it has the potential to automatically classify glitches on the basis of their individual spectrogram. As has been
shown in different works, e.g. Zevin et al. [5], Razzano and Cuoco [6], convolutional neural networks are very effective at the task
of glitch classification and can form the basis for an automatic classification pipeline. Additionally, considering the complexity of
their morphology, Artificial Intelligence algorithms are proven to provide precious support in these classification tasks by noticing
details and similarities over very large datasets that the human eye would struggle to identify.
GWitchHunters volunteers provide classification information that can be used to train automatic classification pipelines. The
project has aimed to expand the contribution of citizen scientists to tasks beyond a simple classification, in order to provide more
complex and diverse data and to allow them to dig deeper in the investigation of the causes of the noise in the detector. In particular,
volunteers choose from a variety of workflows with different degrees of difficulty. The first workflow comprises a simple classification
of the shapes that appear in the spectrograms of the strain channel, the main channel in which gravitational-wave signals are detected,
choosing from a limited number of glitch families (Fig. 1). In the second workflow, volunteers identify the glitches drawing one or
more rectangles around the spectrogram regions exhibiting some excess of energy, and then perform their classification as in the
previous workflow. In the third and more complex workflow, volunteers are asked to compare the shapes of the glitches appearing in
strain channel with those of the so-called auxiliary channels, i.e. data coming from sensors that constantly monitor the detector parts
and their physical environment (Fig. 2). The project has been officially launched on the Zooniverse platform the 16th of November,
2021, and also provides a set of mobile workflows, i.e. tasks that can easily be performed on mobile devices.
3 https://www.zooniverse.org/projects/reinforce/gwitchhunters.
4 https://virgo-gw.eu/.
5 https://www.zooniverse.org/projects/reinforce/new-particle-search-at-cern/.
6 https://atlas.cern/Discover/Detector.
7 https://www.zooniverse.org/projects/reinforce/deep-sea-explorers.
8 https://www.km3net.org/.
9 https://www.zooniverse.org/projects/reinforce/cosmic-muon-images.
123
Fig. 1 Identifying a glitch family in GWitchHunters
Fig. 2 Looking for correlations between Virgo auxiliary channels and a glitch in the strain channel. The glitch morphology is superimposed on to each of
the auxiliary channel spectrograms
During the lifetime of REINFORCE, the GWitchHunters project obtained considerable success, both in terms of classifications
and feedback from volunteers. To date, more than 4,600 individual volunteers have participated in the project, undertaking more
than 700,000 classifications of 41,000 data subjects. These numbers are in broad alignment with the expectations for the project
123
that were built on the basis of another Zooniverse project available on Zooniverse: Gravity Spy.10 Volunteers have proven to be very
enthusiastic; often going beyond the more basic tasks. In particular, they were also asked to look for glitches that were not in the
initial set of available glitch families. This aspect is very important in terms of discovering unknown noise sources, an aspect that
is difficult to solve using ML only. Another aspect that has been particularly successful, has been the availability of translations
in languages other than English. In order to test this aspect, a complete translation of the project into Italian was prepared, and
a dedicated press event was held, in order to properly present the project to the volunteers in Italy. The results have been very
interesting, showing that this approach can be very useful in promoting access to the project beyond only people who speak English
to at least some degree.
The GWitchHunters classification results, using data from the O3 data-taking run, have been input to a ML algorithm built on
a convolutional neural network (CNN) [1]. The performance of the ML has been compared to that of citizens, and the results have
shown that this approach, based on citizen science, is promising, not just in terms of preparing datasets for the training of ML
algorithms, but also in terms of contributing actively to the research in general. New datasets are in preparation, also taking into
consideration the upcoming observational campaign — O4 — and it is expected that the contribution of volunteers will become more
and more important. There are also other plans to further optimise the way in which the data subjects are presented to volunteers,
as well to improve the levels adaptation for mobile devices. It is also hoped that introduction of the Spanish-language version of the
project will help to broaden the participant base further.
2.1.1 Sonification of gravitational-wave data
The spectrogram images are a valuable tool for representing gravitational-wave detector data, particularly for the general public,
without signal analysis expertise. To make this kind of data representation even more inclusive, the GWitchHunters project has
developed a dedicated sonification algorithm to convert the images into sounds. At the base of this algorithm there is the association
of frequencies with musical notes, and the signal energy with the notes intensities, as we shall describe with the following example.
Let’s consider the first detection event of a gravitational-wave signal from the coalescence of a pair of black holes, GW150914 [7].
Its spectrogram is shown in the left-hand side part of Fig. 3. The vertical axis of the spectrograms, representing the frequency, has
been divided into intervals determined by the frequencies of the notes of the C-major scale of occidental music, as shown in the
image on the right-hand side part of the figure. This choice, although arbitrary, aligns with the white piano keys and helps convey
the meaning of different frequencies.
The energy in each band corresponds to the intensity of the note, and its evolution can be represented as a sequence of notes,
resulting in the conversion of the spectrogram into a musical score and, in turn, into a melody. No need for musical knowledge is
required to enjoy the latter and to let people able to distinguish different spectrogram morphologies as well as different melodies.
Additional data transformations can be implemented. For example, the gravitational-wave signal from the coalescence of a binary
black hole system is characterised by a duration of a few tenths of seconds and frequencies up to a few hundred Hz. These times are
too fast to appreciate the details of the signal evolution but, once converted to a musical score, the execution can be slowed down at
will.
Similarly, the frequencies of this signal are a bit too low to be felt as enjoyable by the human ear. However, they can easily be
shifted to a higher pitch by transposing the musical score by one octave (that is, doubling the frequencies) to match the human ear
sensitivity range better.
The multisensorial representation of gravitational-wave data, in the form of images and sounds, fosters inclusiveness and increases
the reach of the project by making the data enjoyable to a more vast public. Most importantly, it facilitates the engagement of
individuals with visual impairments in both gravitational-wave research and the exploration of the cosmos.
2.2 New Particle Search at CERN
The New Particle Search at CERN11 demonstrator project engages citizens in the state-of-the-art particle research performed at the
Large Hadron Collider (LHC) of CERN, in the quest for the understanding of the ultimate structure of matter. The demonstrator is
based on data collected by the ATLAS experiment12 which are produced by high-energy proton-proton collisions at the LHC.
Volunteers perform a visual inspection of data samples consisting of events, namely the registered products of proton-proton
collisions. In this way, they contribute to the search for yet undiscovered hypothetical particles predicted by theories Beyond the
Standard Model (BSM) [8]. Their search could lead to a discovery, which would be a direct proof of new physics and would highlight
a path for future research.
To enable the volunteers in their work, a three-stage architecture was adopted in the demonstrator project. The first two stages are
based on selected samples of simulated data and are used to train volunteers, but also to allow for a quantitative assessment of their
performance and a comparison with specially developed automated algorithms. The third stage of the demonstrator is a discovery
10 https://www.zooniverse.org/projects/zooniverse/gravity-spy.
11 https://www.zooniverse.org/projects/reinforce/new-particle-search-at-cern/.
12 https://atlas.cern/.
123
Fig. 3 Alternative spectrogram representations of the GW150914 gravitational-wave signal as observed by the LIGO Hanford detector. a Usual time–fre-
quency representation of the energy of the signal, as in the original article [7]. b Alternative representation where the energy corresponding to the different
frequency bands is represented by the series of curve. The higher the curve, the greater the intensity of the corresponding musical note
Fig. 4 The first stage of New Particle Search at CERN on Zooniverse
stage, employing datasets of real events from the ATLAS Open Data Set Release [9]. The third stage provides two research paths:
(a) study of Higgs boson decays to two photons, one of which could be converted to an electron-positron pair by interaction with
detector material; and (b) search for yet undiscovered neutral long-lived particles, predicted by certain theories of the BSM. The
volunteer research involves the identification of specific signatures, which are produced from the decay of these new long-lived
particles. These decay products could originate from displaced vertices (DVs), namely vertices — formed by two or more tracks —
that are displaced with respect to the main collision point of the two protons.
The demonstrator project provides visual analysis tools that allow the citizens not only to classify static images — in order
to recognise the DVs — but also to interact with the event displays, select specific tracks and calculate kinematical quantities
characteristic of the sought-after particles.
In Stage 1, which is hosted entirely on Zooniverse, volunteers are trained to recognise DVs in a high-purity sample of simulated
data, corresponding to the various scenarios of new particles with displaced vertices. The volunteers only inspect stationary images
of the traces that charged particles leave in the inner part of the ATLAS detector. They look for tracks that intersect at a point other
than the main interaction point and need to inspect both views of the inner detector in order to be able to properly recognise track
intersection. In the two different projections of the inner detector, which are depicted in Fig. 4, the tracks are given different colours,
so that a user may identify the same track in both views. The volunteers are then asked to spot the DVs in both views, and the answer
is internally assessed by Zooniverse, based on the truth information which is also provided to the platform.
The quality of the volunteers’ work is assessed and compared against a specially developed automated algorithm, which detects
the presence of displaced vertices in the events. The algorithm extrapolates the tracks and looks for their intersection, using only
123
Fig. 5 The second stage of new particle search at CERN on HYPATIA
detector information that is also available to the volunteers. The overall goal was to compare the DV-identification efficiency of
the volunteers to that achieved by the algorithm. The efficiency is defined as the fraction of long-lived particle decays that were
successfully identified by the volunteers. The results, from 180,000 classifications provided by the users, were analysed in multiple
ways. The most interesting results were obtained by considering the user consensus as an identification criterion. The identification
efficiency of the user consensus is found to be about 93% on average, depending on the projection of the ATLAS detector provided in
the image (89% in the transverse detector view, 96% in the longitudinal detector view). This efficiency is very close to the respective
identification efficiency obtained by the dedicated automated algorithm, which was 94%.
For Stages 2 and 3, the users are directed from Zooniverse to the HYPATIA [10] event display. Instead of simply examining static
images and locating DV they interact with the event display in order to perform in-depth analysis of the events. In the context of the
demonstrator project, significant additions were made to the HYPATIA platform, in order to incorporate the functionality necessary
to enable the volunteers to perform their analysis in each stage of the project. In addition, a new event format was developed to
include additional information, such as the display of DV.
In Stage 2 volunteers are asked to identify certain particle types that are useful for the next discovery stage. They look for
signatures of muons, electrons, photons and converted photons. The HYPATIA particle information table, shown in Fig. 5, displays
momentum, charge and direction information for each track or cluster of the event. The volunteer can identify tracks and clusters,
using the button that corresponds to the type of track or cluster to which they have determined it belongs (electron, muon, photon
or converted photon). When the volunteer clicks on the Next button to display another event, the selections made are stored in a
back-end database for later processing. Since the sample consists of simulated events, the particle generation identification is already
known and thus can be compared to the user classification, in order to determine the validity of the choice made. The volunteers
made a total of 37,000 classifications in this way, with 80.4% of them being correct.
The volunteer output was also compared to an ML algorithm, which was developed based on Boosted Decision Tree classifiers,
and which uses exactly the same information that is available to the volunteers, in order that this comparison be fair and impartial.
The comparison shows that the identification ability of the volunteer cannot be better than a dedicated ML algorithm, but, in the
cases of the electron and photon, the user identification efficiency is not very far from it.
In the Higgs boson stage, the volunteer can indicate that a cluster belongs to either a photon or a converted photon that could
originate from Higgs boson decays. HYPATIA automatically calculates the invariant mass when the volunteer selects a pair of
photons. In addition, the volunteer can rate the event (from one to five stars) based on the instructions given on the platform. In this
study, users selected 94% of all photon-pair masses in the 106–160 GeV mass range. Distinguishing the Higgs boson signal, which
lies on top of the background, requires a much larger dataset than that available to the volunteers and a sophisticated statistical
analysis of the data. The volunteers rated 1156 events as being worthy of five stars, with most events receiving low star-ratings as
they did not contain unusual Higgs candidate decays. Of those 1156 five-star-rated events, only a few contain extra leptons, but
the invariant mass of the photons (or converted photons) is outside the mass range of the Higgs boson mass, therefore those events
cannot be attributed to complex Higgs boson production and decay mechanisms.
123
In the Long-Lived-Particle (LLP) Hunting part of the demonstrator project, volunteers search for long-lived particles predicted
by some BSM models, through the identification of secondary vertices of particle decays. Following the vertex identification, they
were asked to look for muons originating from the displaced vertex and to examine certain kinematic variables, which could be
characteristic of a candidate long-lived particle. Volunteers were also asked to rate each event from one to five stars, according to
how similar they believed it to be to one of the sought-after decays. A total of 81,894 classifications were performed on a sample
consisting of 2440 events. Events which satisfy all given requirements have a much higher probability of originating from a new
long-lived particle. Since these particles are as yet undiscovered, the relevant candidates are extremely rare. The volunteers only
viewed 27 of them (volunteers viewed each one multiple times) and correctly marked with five stars 23 of those events. Two of the
five-star events were good candidates for new BSM long-lived particles (hypothetical super symmetric particles). The events were
scrutinised by our team and further information (which was not available to the volunteers) was inspected. It turns out that, most
probably, both events are due to the expected background. A number of volunteers did isolate 24 events with more than one muon
in the DV and posted their findings in the project Talk forum, where they were discussed by the wider community. After further
investigation by the research team, it was revealed that these events were either due to the interactions of a known particle with the
detector or faults in the reconstruction of particle tracks.
2.2.1 Interaction with users, engagement and impacts
From the launch of New Particle Search at CERN as an official Zooniverse project on the 19th of October, 2021, until the end of the
REINFORCE project, on the 30th of November, 2022, the project research team received and replied to 2,852 Talk messages (seven
messages per day on average) posted by volunteers on the project forums available on Zooniverse.13 Volunteers communicated
technical questions regarding the stages of the project (especially during its first few months), questions on physics related (or not)
to the tasks provided to them, their results and observations, as well as interesting suggestions and features they would like to see
added. The members of the scientific research group were delighted to respond to volunteer posts on a daily basis, but also motivated
discussion between volunteers, while assigning the role of expert to the most talented among them. Furthermore, in collaboration
with the Ellinogermaniki Agogi (EA) team, various activities during the course of the project were prepared, which served to ensure
the motivation of volunteers remained high. These included:
• Awarding hard-copy certificates to interested volunteers who had completed over 200 classifications with over 50% efficiency,
acknowledging their citizen-science contribution;
• Meeting with volunteers in person at the REINFORCE Summer School, 2022, in Marathon, Greece.14 During this summer school,
volunteers were presented with the ongoing research and the latest achievements in the field of high-energy physics, and had the
opportunity to work together with our scientific team;
• Four challenges (Winter Challenge 2022,15 Easter Challenge 2022,16 Challenge for Greek teachers,17 EPS competition18 ) during
which the performance of each volunteer was closely monitored. At the end of each challenge, the most successful volunteers
were awarded prizes, with the results announced on the Zooniverse forums;
• Online meetings with volunteers, in which their questions and ideas were discussed, as well as virtual visits to LHC experiments
(ATLAS and ALICE). These activities maintained the engagement of citizens in New Particle Search at CERN and volunteers
ended up contributing an impressive 179,887 classifications (441 classifications per day on average) to Stage 1 of the project alone.
According to the data gathered by the Zooniverse platform, volunteers spent about five minutes per classification, on average.
In conjunction with the work undertaken by the REINFORCE engagement team, the New Particle Search at CERN team also
developed a set of feedback metrics and mechanisms, which supported volunteers during their early encounters with the projects.
During the final months of the REINFORCE project, all of the New Particle Search at CERN materials were translated into Greek,
while the REINFORCE partner CONICET also provided a Spanish translation. All three language versions are now available on
the project’s Zooniverse page. Furthermore, data on the particle identification from the second stage was provided to the CONICET
team in order to develop a way to represent the different particle types with distinct sounds. Sound files from a small number of
events were generated as part of this effort and tested during different dissemination events. It was determined that the second stage
of the demonstrator project is the only part of it that can be sonified, as the other two are too visually complex and don’t lend
themselves to auditory representation.
In addition to motivating volunteers, the scientific community was also motivated towards citizen science through presentations
of the project at six international scientific conferences, with contributions to the respective conference proceedings.
13 https://www.zooniverse.org/projects/reinforce/new-particle-search-at-cern/talk.
14 https://reinforce.ea.gr/international-training-course/.
15 https://reinforce.ea.gr/winter-challenge/.
16 https://reinforce.ea.gr/easter-challenge/.
17 https://reinforce.ea.gr/therinos-diagonismos/.
18 https://reinforce.ea.gr/eps-citizen-science-competition/.
123
2.2.2 Outcomes and the future
The final goal of the discovery stages is to have thousands of citizen scientists rate each event with 1–5 stars, according to how
similar they believe them to be to one of the new long-lived particle decays. In this way, the volunteers are directly involved in
potential discoveries, without requiring any prior knowledge of high-level physics or computing skills. Furthermore, they contribute
to a frontier research science topic, performed at the largest particle physics facility in the world.
Finally, a large number of dedicated volunteers left very informative and useful comments in the Talk forum during the classi-
fications of the two discovery stages. Overall, New Particle Search at CERN should be considered a success, as it managed to get
a large number of volunteers involved in complicated analysis of high-energy-physics signatures collected by the state-of-the art
ATLAS detector.
The REINFORCE project ended at the end of 2022, but the demonstrator project remains open and available on the Zooniverse
platform and volunteers continue to classify events. When a new ATLAS Open Data release is available, together with other resources,
new, fresh data will be added.
2.3 Deep Sea Explorers
The Deep Sea Explorers19 demonstrator project takes place in the context of the KM3NeT experiment,20 a neutrino telescope
currently being deployed in the Mediterranean Sea. While the main focus of the KM3NeT Collaboration is on the search for
neutrinos, Deep Sea Explorers volunteers were asked to study events caused by sea mammals and benthic fauna.
KM3NeT is equipped with light sensors to detect Cherenkov light produced by a neutrino interaction in the sea, as well as
hydrophones that are used to calibrate the instrument as it moves with the sea currents. In addition to being able to detect elementary
particles crossing the detector, KM3NeT is also sensitive to its environment. Volunteers were therefore asked to classify the various
signatures of light and acoustic noise produced in this environment. The aim of these classifications was to help to better understand,
on one hand, KM3NeT data and the response of the instrument, and, on the other, the presence of life in the deep sea in which the
detector is deployed.
In the bioluminescence workflow, volunteers were asked to classify data subjects and determine the number of peaks visible within
them. In the bioacoustics workflow, they were asked to study different visual and audio representations of data subjects recorded
by the hydrophones around the detectors and to determine whether they signalled the presence of sperm whales, short-finned pilot
whales, were just pure noise or were likely something else.
As the KM3NeT Collaboration is an experiment that is still in the deployment phase, it was decided to split the acquisition of the
classifications performed by citizen scientists into two separate phases. In addition to giving the scientist team more time to acquire
additional raw data to be classified by the participants, this also created a motivating feature for the citizens. The first phase of the
demonstrator ended in February, 2022, after the launch of the project earlier that year. The first classifications were analysed and
the results, presented in the next section, were presented to the Deep Sea Explorers participants before the start of Phase 2.
A much larger set of events, a factor 10 and 20 for the bioluminescence and bioacoustics workflows, respectively, was created
for Phase 2 and uploaded onto the Zooniverse platform a month later. Both workflows, despite not being fully completed, made it
possible to acquire sufficient data to carry out a comparative study.
2.3.1 Successes: what worked and why
Deep Sea Explorers proved to be a real success. It has been possible to demonstrate that the help of citizen scientists improves the
output of a classifier when labelled datasets are available, i.e. in the case of the bioacoustics workflow. In the case of both workflows,
sufficient trained classifiers were obtained to now be used on the data collected in KM3NeT over its entire lifetime.
After the first phase of the project, it became possible to clearly identify some events as belonging to one of the four categories
made available to volunteers on Zooniverse. As an example, the two events shown in Fig. 6 have been classified as being one peak
events with a 90% and 96% certainty (i.e., this is how often they were classified in this category by the participants). On the other
hand, some events have not been classified unanimously by the participants, such as that displayed in Fig. 7.
The goal of the project was to obtain a set of very well classified data subjects. Despite some events already fulfilling the
requirements to reach this objective, more events were needed for Phase 2. In this second phase, the aim was to provide new subjects
so that the number of events that are classified with a high probability in only one category could be increased. Phase 2 therefore,
had many more events than Phase 1, in order to significantly increase the data available for the final statistics.
At the end of Phase 2, the events of the bioacoustics workflow were used to train a convolutional neural network (CNN), previously
developed using data from hydrophones located at the surface. While the events used in the demonstrator, which were originally
classified with less than 70% accuracy by the already existing neural network, were used to train the new CNN and led to a precision
in the classification of sperm whale and short finned pilot whale of 90% and 100%, respectively.
19 https://www.zooniverse.org/projects/reinforce/deep-sea-explorers.
20 https://km3net.org/.
123
Fig. 6 Bioluminescence events

from Phase 1 that were identified
as being one peak events by the
citizen scientists
We concluded that the data collected from the volunteer classifications provided through the demonstrator project can help us to
enhance the accuracy of classification of KM3NeT events in comparison to the original neural network.
2.3.2 Areas that might be approached differently
The involvement of citizen scientists in active research projects taking place within the KM3NeT collaboration was a real success
with concrete scientific results as proof. However, the personpower required for such a project was misevaluated by the Deep Sea
Explorer team. As the volunteers were discovering new research fields, moving between neutrino physics and astronomy to marine
biology, detailed explanations and continuous support from the science team was needed. The Talk forum, which forms part of every
Zooniverse project, allows for an easy and direct interaction between the science team and the citizen scientists, but, as with any
forum, an active and constant engagement in the discussion takes time.
123
Fig. 7 Example of an event that was classified as a short-finned pilot whale (28%), sperm whale (35%), or pure noise (24%)
2.3.3 Lessons learned
Deep Sea Explorers results demonstrated to the science team, as well as to a considerable fraction of the particle physics community,
the substantial gains that citizen science can bring to event classification. As other projects are added to Zooniverse, an important
characteristic of them might be to keep in mind the time and investment required to properly nurture, grow and support a project of
this kind. The Deep Sea Explorers science team will be developing more projects of this nature and will aim to maintain a support
team that is at least twice the size of the team used for this project.
2.3.4 Interaction with the demonstrator project community
Besides successfully achieving the classification goal, the Deep Sea Explorer volunteer community caused a considered reflection
within the research team on how scientific wording can be perceived. The original name of the demonstrator project was due to
be Deep Sea Hunters, as researchers in the KM3NeT Collaboration often called themselves Neutrino Hunters. During beta testing,
however, the Zooniverse community pointed out that the word hunter could have a negative connotation when associated to marine
fauna and could discourage people to take part in the project. The demonstrator was duly renamed as a result of this advice and
became the Deep Sea Explorers project that ultimately went to official launch.
2.3.5 The future: next steps; short-, mid- and long-term exploitation of the project and its resources
This project has led some members to ask for the help of citizen scientists in relation also to other projects, such as, for example, in
the classification of sub-threshold events recorded by the IceCube Neutrino Observatory, another neutrino telescope located at the
South Pole. While no bioluminescence activity is expected in the ice, the detector itself creates patterns in the data that are not well
modelled and for which the help of citizen scientists in their classification would be useful. This would constitute a new project on
the Zooniverse platform that will be created using the know-how acquired during the development of Deep Sea Explorers.
123
The science team also created fruitful ground for interdisciplinary projects and collaboration with bio-informaticians and biol-
ogists. This collaboration will continue after the end of REINFORCE and throughout the lifetime of KM3NeT. Several grant
applications to hire PhD students and postdocs in this framework have been submitted. The help of citizen scientists to disentangle
neutrinos from the signal sent by living species at the bottom of the sea will be one of the methodologies used in these projects.
2.4 Cosmic Muon Images
The Cosmic Muon Images21 demonstrator project is based on muon tomography [11], an imaging technique to study the internal
density distribution of massive objects. Muon tomography uses muons produced in cosmic ray atmospheric showers as a probe for
inferring the density anisotropies of geological, archaeological and civil engineering objects like volcanoes, pyramids and tumuli.
The principle behind this technique is similar to that behind a medical X-ray; a particle detector is placed downstream of an
object with respect to the incoming particle flux and takes measurements for an extended period of time. The angular profile of the
reconstructed muon tracks is then compared to the theoretical expectation, taking into account the external geometric characteristics
of the object as well as some basic assumptions on its composition. Combining all this information provides an image of the internal
density distribution for the object under study.
The method was first used by Alvarez to search for voids inside the pyramid of Chephren at the Giza plateau, back in the 1960s
[12]. It was proposed for volcanology in the mid-90 s [13] and reached its modern form in the mid-2000s [14]. Since then, the domain
of study has rapidly extended into new fields, such as: the prediction of natural hazards, which includes monitoring for landslides,
lahars and other similar mass displacements; the absorption/retention and distribution of rain water in underground repositories
and aquifers; the investigation of pyramids for unknown voids, as well as other sites of archaeological interest, such as tumuli; the
monitoring of industrial facilities like blast furnaces or nuclear waste disposal containers; and, last but not least, homeland security
applications focused on the detection of nuclear contraband within transportation containers [15].
Since muon tomography brings together particle physics, cosmic ray physics and a diverse conglomerate of scientific fields it
serves as a concrete example of how particle physics can leave the constraints of the laboratory, become mobile and address issues
that affect our everyday lives in more immediate and palpable ways.
The Cosmic Muon Images project focuses on the analysis of data from an archaeological muon tomography expedition that took
place during the summer of 2018 in Greece at the ancient tumulus of Apollonia in Khalkidhiki, which hosts an ancient Macedonian
tomb [16]. In the context of REINFORCE, all data acquired were shaped into a Structured Query Language (SQL) database, which
acted as a repository for plot creation and which were then uploaded to the Zooniverse platform, in order to allow citizen scientists
to inspect and analyse [17].
The goal of the analysis was to identify track-like topologies that had been missed by the algorithm used by the research team.
The extreme horizontality and the closeness of the detector to the studied object made the signal-to-noise ratio for this experiment
a factor that could potentially be ameliorated through the input of the volunteers.
The track-selection algorithm generally used for volcanology data is based on a series of empirical selection criteria that mostly
favour low-multiplicity events, which triggered a low number of detector channels, i.e., fewer than 100 out of the 2,304 possible
combinations. This provides a strict sample of muon tracks for the tomography and rejects events with complicated topologies as
background, even though a track (or more) might be hidden among these busy topologies.
These events were separated into two samples and were then provided to the volunteers in the form of detector-representation
plots, with the possible particle strikes depicted on the detector surfaces. The task at hand was the identification of tracks crossing
the detector and extra particle strikes on the three detector planes.
The first sample comprised simpler events, for which our algorithm could provide a candidate track, while the second contained
events for which no reconstructed tracks were retrieved. The first sample was classified by the volunteers within the Introductory
workflow, which also served as a stepping stone to becoming familiar with the event categorisation procedure. The second, more
complicated sample, was incorporated in the FreesStyle workflow [18].
Both workflows ask volunteers to draw a pair of lines, representing a track, passing through the detector channels that showed
a charge and mark up extra particle strikes, by pairing detector channels that did not participate in the formation of the line. The
difference being that for the Introductory workflow, a pair of lines is already present (representing a particle track) and the volunteers
need to decide if these are valid, if they need to be redrawn properly or if they are just an artifact of our algorithm and the event
should be categorised as background. The logic for the FreeStyle workflow is the same, with the topologies being more complicated
and the volunteers having to point out if it is clearly a background event or if there are tracks that might be identified. For both
workflows there is an extra step after the basic task of identifying the tracks, which is the highlighting of extra particle strikes.
2.4.1 Successes: what has worked and why
Towards the end of the project the volunteers had analysed 9,099 events for the Introductory workflow, having performed 48,577
classifications. Of these, 5,765 events were identified as signal and 3,344 events as pure background. This was an interesting finding,
21 https://www.zooniverse.org/projects/reinforce/cosmic-muon-images.
123
as the algorithm used by the research team would not be expected to reach such a high percentage (30%) of false positives. This is
the first useful result provided by the inspection of the events, and it needs to be cross-checked with appropriate simulations to see
if and why it holds true.
Trying out different ML algorithms from the scikit-learn22 library, the Gradient Boosted Trees algorithm was identified as optimal
for signal-to-noise discrimination. The unblinding of the data showed that this method could potentially provide a better muon sample
for muon tomography imaging, with 10% less noise than the initial sample. The impact of this selection on the muon tomography
result remains to be studied [19].
2.4.2 Improvements; what has not worked and why
The FreeStyle workflow did not yield a similar result. The first cause for this was the complexity of the events combined with the
small number of classified data that did not provide the ML algorithm with enough clear-cut information on which to perform. The
overall participation of the volunteers for this workflow was comparable to the Introductory workflow, with 46,866 classifications
of a dataset of 8,529 subjects, so the failure to train likely relates to the task itself and its origin is more objective than subjective
[19].
The second issue originated from the research team side and the design of the workflow. The fact that there were many degrees of
freedom through which the volunteers could navigate the classifications led to diverse realisations as to what might be background
or not. Some completionists would try to draw all possible tracks they could see, no matter the validity of the physical interpretation,
while others would follow a lazier approach and reject events as background that would otherwise prove to have a track after careful
examination. The complicated topology of a subject would only make things worse, leading to a weak consensus. The heuristic
approach that the research team aspired to was eventually not suitable in the framework of the demonstrator project.
2.4.3 Lessons learned
A key takeaway from the Cosmic Muon Images project is that citizen science projects can provide help even for tasks that at a first
glance might seem simple or mundane. That being said, workflows need to be designed with care and attention and an extended period
for tests and preliminary analyses of the classifications, in order to catch early on any potential misunderstandings and confusion
that might arise when non-experts try to grasp and implement concepts that experts perceive as given and trivial. In retrospect, a
gradual involvement of audiences with decreasing levels of scientific literacy would be beneficial. Maybe going from university
students to other audiences that can function under guidance, such as amateur astronomers, geologists etc., would be beneficial for
the formation of the project, before it reaches its final form and becomes available to the general public.
This proposed strategy springs from the fact that a workflow needs to remain unchanged throughout its lifetime, so that all data
are analysed by the volunteers in a uniform way. The Zooniverse staff, being experienced with this aspect, have similar procedures
in terms of beta testing by volunteers, but the focus is more the optimisation of the tools rather than the scientific end product and
its usefulness. This gradual opening to audiences of different scientific expertise should help with the latter aspect.
Concerning the FreeStyle workflow dataset, it is possible in retrospect to say that it would have been more successful if an
unsupervised ML algorithm had initially been run over it, in order to have caught some basic patterns. It would then have been
possible to identify which of them made physical sense, in terms of tracks and extra particles. In this manner, it would have been
easier to guide and better constrain volunteer classifications. With this method we could have better limited the data sample to
only those events that could have a physical interpretation, rather than leaving this task to the volunteers. This strategy could have
improved the consensus and would have decreased the number of classifications needed per subject.
The first step to remedy these drawbacks would be to re-design the FreeStyle workflow, providing better guidance and a more
constrained path for the classification procedure. It would also be useful to remove events with extreme multiplicities, as, following
discussion with the volunteers, it is clear that these can be a factor of confusion.
2.4.4 Interaction with the demonstrator project community
The interaction with the volunteers has proven to be very useful. Since the start of the project, it has been possible to identify
two kinds of participant: those interested in the actual categorisation process; and those wanting to discuss the instructions, guides
and the other demonstrator materials. Most commonly, people have asked if a specific categorisation that they have done is right.
Answering these questions led gradually to the establishment of a database of examples with proper categorisations. The answers
from the research team clarified why something would work and how certain choices would have made more sense than others,
taking into account the detector operation and the processes involved in the production of the events.
Some volunteers were more inclined to address appearance and form issues relating to the demonstrator project. The research
team was provided with a full list of spelling and grammatical mistakes that had been missed during the writing and correcting of
the project materials, while others offered ideas on how the event depiction of the workflows could have a better, more intuitive
22 https://scikit-learn.org/.
123
representation. This feedback was incorporated into the field guide section of the project, as implementing changes directly on the
workflows after their launch is discouraged.
Interaction with the volunteers also showed that many of them choose to go directly to the workflows and start performing
classifications right away. They would come back to the research team for help only when they realised that their intuition alone,
without going through our explanations, led to inconsistencies. As the project progressed and more answers for issues accumulated,
a gradual decrease of this kind of question was observed.
At the start of the project, the research team aspired to create a small team of volunteers willing to delve deeper into muon
tomography, but it quickly became clear that even the most dedicated individuals only wanted to help through the Zooniverse
platform and nothing more. Another aspect that we found, was that uploading data in small bunches of 5,000 subjects per upload
created some frustration among the volunteers. Most of them wanted the entire dataset to have been uploaded in one go, so that
they would be able to have a clear time horizon for the completion of the project, rather than watch the progress bar reach the 100%
mark, signifying that the dataset had been completely classified, only for it to go down again. It is difficult to say to what extent this
affected the churn of the project, but it is clear that the practice was not received well and should not be repeated. Overall, feedback
proved very useful and the communication with the volunteers serves now as a guide, with which to shape future developments of
the project.
2.4.5 The future
The demonstrator has reached almost 170,000 classifications over a total of 13,626 subjects for the Introductory workflow and 12,867
subjects for the FreeStyle workflow, adding, respectively, 4,257 and 4,338 classified subjects to the numbers mentioned previously.
It is now time to automate the work of volunteers for the Introductory workflow by means of a suitable neural network architecture.
In the case that the current number of subjects is not enough for this task, the research team will need to explain the new course of
its work and enrich the workflow with more data.
The FreeStyle workflow is, at present, suspended and the data analysis should focus more on the different tendencies of the
volunteers towards the classification procedure. It will be necessary to identify those that are pointed in the right direction and to
steer the citizen scientists towards them. The research team will continue to adopt a heuristic approach, but this time there will be
a focus on narrowing down the outcomes to a degree that would make them useful, as is the case with the Introductory workflow.
This should eventually result in a new workflow, with different datasets, tutorials etc.
Finally, an entirely new workflow is being considered. The idea being that this new workflow would provide volunteers with
tomography plots for analysis and would focus on pattern recognition for the fast deciphering of objects and formations. Based on
the experience gained so far, it is clear to the research team that this would be a much more complex task than the tasks provided so
far. It represents an opportunity for the muon tomography community to provide data, offer perspectives and expertise, but also to
involve many expert participants to support and evolve the project. The muon tomography community has limited experience with
citizen science and a discussion on how to integrate the approach should prove beneficial for the domain as a whole.
The demonstrator project is now also an organic component of the outreach activities carried out by the research team and
functions as a useful tool in terms of introducing audiences to the experimental prerequisites of muon tomography, while, at the
same time, benefiting data analysis techniques through the categorisation of more data subjects.
3 Sonification activities
The use of sonification in astronomy has existed for years. Researchers have long highlighted the need for multi-modal approaches
in teaching and learning environments, presenting principles for inclusive material, based on user-centred and universal design.
Recently [20], following an international sound workshop, held in August, 2021, created a repository of existing software, which
collects the results of 98 projects that have been developed since 1962, many of them now discontinued, with a lack of documentation
or without evidence of applications in science. Almost 80% of these sonification projects were carried out between 2011 and 2021.
In most cases, the sonification mapping of the dataset was defined by its creator and shared as a final product or even with some
musicalisation, not clearly devoted to the study of the data, the identification of features or even to research.
Taking into account the increasing examples of sonification in astrophysics, sonoUno is one of the sonification software that
translates data from two or more column tables into sound. It is an open-source application, based on a modular design, which
allows users to open different datasets, and explore them through visual and auditory display, the last permitting them to adjust
visual and sound settings to enhance their perception. The project has been user centred from the beginning, and has been designed
following different steps during which the user has always guided improvements, from the analysis of accessibility using the ISO
9241-171:2008 standard [21, 22], to focus group sessions, with people with and without visual disabilities testing the different
sonoUno versions [23].
The software emerged with an early first release, at the early stages of the REINFORCE project, and, because of this, it was
possible to add multiple new functionalities and developments, including the sonification of diverse sets of data from the demonstrator
projects.
123
3.1 The team working on sonification
The sonification team required specialists and, to this end, a group with people acting on specific modules of the software, working
in a cooperative way, was created. This allowed to improve both the desktop version of the software and the web interface, using
new resources. The team is multidisciplinary, including astronomers, engineers, software designers, educators, disability specialists,
neurologists, and sociologists, both disabled and not.
3.2 User-centred design
From the beginning, sonoUno was designed with the focus on the final user, first with a theoretical framework based on user-testing
analysis of other platforms [22] and the ISO 9241-171:2008 standard; then with a focus group using the sonoUno prototype [23].
One of the conclusions of the focus group sessions was that sonoUno allows people with different sensory styles to explore and
analyse scientific data presented in tabular format, through a synchronised visual and sound representation.
Following the first release, each update of the software was tested with different users. They tested the software with their own
data sets and sent feedback back to the development team by email or through the completion of a form. In general, the users are
contacted once a year, and even when not all participants answer, feedback from at least five people is obtained. This methodology
of user testing gives visibility to sonoUno even, in some cases driving new use cases, such as: Sensing the Dynamic Universe,23
Taller de sonorización dictado por el IES José de Frugoni Pérez en España24,25 and The Sounds of BEARS.26
3.3 sonoUno development
Two versions of the software have been developed: a browser-based version, accessible via the web, and a desktop version.
3.3.1 Desktop-based version
The first approach to sonification was developed for a desktop computer; it used the graphic user interface design obtained from the
previous software and user-centred analysis. Then, with the first prototype based on MIDI sonification, a focus group session was
conducted to test the software’s usability and efficiency. From that analysis, it becomes apparent that user-centred software allows
visually impaired people to explore the data, not only as a user, but also as part of the development and improvement community
[23]. Some bugs were also detected and fixed in future updates; one of the notable modifications was the change of the sound library
to one that allowed for more resolution in the sound.
The current version of sonoUno now contains sonification and visualisation of one-dimensional data sets; math functions; sound-
setting configuration and plot-setting functionalities; a bash script that uses data provided by the user and stores the sonification and
plot files; and three additional scripts to sonify specific data sets related to the demonstrator projects.
3.3.2 Web-based version
The web-based version of the software was developed in response to user recommendations and in order to ensure wide and global
access to the software
The browser version was developed using HTML, CSS and JavaScript and uses the ARIA protocol to ensure communication
with screen readers. For the sound synthesis itself, the tone.js27 library was selected. The result was a versatile application for use
in the sonification of any data set using a tabular format and that can be used via a web browser on a computer, tablet, or mobile
device.
3.4 Sonifying the demonstrator-project data
Different approaches were adopted in order to sonify the datasets of the different demonstrator projects. These are detailed below.
• For GWitchHunters and Deep Sea Explorers data, image sonification was used. The openCV28 library for image manipulation
was used in conjunction with the sonoUno sound library. This consisted of converting a spectrogram image (Fig. 8,29 Figs. 9,
10) to grayscale and translating to sound the normalised total grey value of each column of data. For the sound parameters, the
23 https://lweb.cfa.harvard.edu/sdu/index.html.
24 https://astronomiayeducacion.org/taller-de-introduccion-a-la-sonificacion/.
25 https://astronomiayeducacion.org/taller-2-de-sonificacion-descubriendo-el-universo/.
26 https://stephenserjeant.github.io/sounds-of-bears/.
27 https://tonejs.github.io/.
28 https://opencv.org/.
29 https://www.youtube.com/watch?v=pkiGdZu5gEo.
123
Fig. 8 A Blip type glitch, with

neither grid nor axis edges. The
blue vertical bar indicates the
sonification position
Fig. 9 Example of a Deep Sea Explorers bioacoustics workflow data subject. A click (millisecond-range soundwave) from a sperm whale
brightness value corresponded to the highest tone, and the darkest value corresponded to the lowest tone (silence). The other
parameters, such as instrument and frequency limits of sound, are fixed at the beginning of the code, but it is intended to add a
way to indicate them via command line in the future;
• For New Particle Search at CERN, a different approach was necessary in order to be able to classify different particles using only
sound. Upon detecting activity in any of the detector layers (calorimeter or inner detector) a scanning process was initiated within
a narrow cone covering the active region. This scanning commenced at the centre of the detector and extended radially outward in
three-dimensional space. Exploiting the distinctive interactions of different particles in each detector layer, various sound patterns
and audio volumes were utilised to provide comprehensive information about the activity within the specific detector region.
123
Fig. 10 A data subject from the

Deep Sea Explorers Bioacoustics
workflow, sound-tracked with the
code for sonoUno image sound
enhancement
Each particle is visually represented in two views (an example is shown in (Fig. 11): an electron with a track in the inner detector
that points to a cluster in the calorimeter; a converted photon with two very close tracks in the inner detector that point to a
cluster in the calorimeter; a muon with a long track that traverses all of the detector layers and which could, although this is not
necessarily the case, be pointing to a cluster; a photon with a cluster in the calorimeter, but with no track in the inner detector; or
an unknown particle with only a track in the inner detector or another representation that is not covered by any of those described
above.
The audio representation of the above cases progressed simultaneously in both views and was based on the existence (a continuous
sound of two seconds) or absence (a silence of two seconds) of the track in the inner detector, the existence (a characteristic sound
of one second) or absence (a silence of one second) of the cluster in the calorimeter (the sound volume representing the energy),
and the existence of a track beyond the inner detector (in this case, the initial continuous sound length changed from two seconds
to four seconds)30 ;
• In the case of Cosmic Muon Images, the representation with sound was based on the possibility of correlating the deposit of
energy through the three layers of the detector (Fig. 1231 ). The sonification was based on the one-dimensional plot, with the top
and bottom representations containing 32 channels and the middle one only 16 channels. In this case, 16 piano notes are used,
one for each channel in the middle plot, and for the other plots, one note for two close channels. In the sonification process, the
note heard corresponds to a deposit of energy in the plot; when more than one channel presents a deposit of energy, a combination
of notes is provided.
3.5 Tactile models
Within REINFORCE, the team working on the sonification aspects of the project, also produced a toolbox of tactile models; physical
representations of data subjects used in each of the demonstrator projects. These proved useful resources in workshops held during
the project.
3.6 Challenges: difficulties and issues faced; how they were overcome
One of the most significant challenges of the work of the sonification team in REINFORCE was understanding how best sonify
the individual data subjects, keeping in mind the different data formats and ways in which the visual representations varied. This
required extensive collaboration with the individual demonstrator project research teams and numerous iterations in order to be able
to arrive at a point that was satisfactory for all involved.
It proved necessary to develop the software several times during this, including introducing specific piano notes to describe the
particle tracks, and tick marks, to indicate the beginning and end of a sonification.
3.7 What worked particularly well
Working with the different demonstrator project research teams proved to be a huge benefit to the development of sonoUno overall.
The exchange with the different teams and different ways of investigating and exploring the data, made it possible to produce a
versatile sonification library applicable to multiple scenarios. In this sense, the work done in REINFORCE particularly emphasises
the importance of trans-disciplinary collaborations.
3.7.1 Course on sonification
Previous studies have shown that auditory performance improves when combined with visual stimuli and vice versa [24]. Taking
this into consideration, the sonification team used two sensory pathways, sound and vision, with the intention of performing a test
30 https://www.youtube.com/watch?v=XMaYIJkJIHg.
31 https://www.youtube.com/watch?v=EYhcdyO2w2I.
123
Fig. 11 A comparison of an event displayed in HYPATIA with its representation in sonoUno
of multi-sensory training and with the aim of achieving the familiarisation of participants with this type of technique in signal
detection. In order to schedule training that made it possible to presenting visual and auditory signals, allowing for the identification
of patterns and the saving of this user interaction, the PsychoPy software was used [25]. This software was designed for the creation
of experiments in behavioural sciences (such as psychology, neuroscience, and linguistics, among others) and allows precise spatial
control and synchronisation with different stimuli.
Over two sessions a complete introduction to multi-modal perception (visual, auditory and tactile) as well as specific training in
sonification, was performed. The course was designed so that the complexity of the actions required on the part of the participants
grew over the two sessions. Each session was made up of three blocks, one for each type of data: (1) three types of glitches; (2)
two LHC events; and (3) four muography events (two with the presence of a muon and two without). The first part of the training
session was devoted to understanding the data and knowing how to classify it, presenting examples of each type. In the second part
of the activity, different events were presented for each type of, randomly selected, data.
123
Fig. 12 Views of the same event representing the existence of a muon: 3D view (left), 2D view(right), note the channels mentioned in the text
Following the activity, participants were asked to complete a survey about the web version of sonoUno and the training course.
Responses in direct relation to the web interface were overwhelmingly positive, with 77% of participants feeling that the software
could be used to improve their own work, research, professional and educational activities, while responses in relation to the
complexity of the sonifications of the demonstrator project data proved what were expecting; i.e., that the GWitchHunters glitch
sonifications and those relating to the Cosmic Muon Images project were relatively easy to interpret, while those relating to the New
Particle Search at CERN demonstrator project were less straightforward, owing to their complexity and depth.
3.8 Plans for the future
In the framework of the REINFORCE project, progress was made towards the achievement of two objectives. The software was
made more versatile in terms of the handling of sound effects, going beyond a single time series; while work also progressed in the
provision of sound as a service. This novel idea of embedding the service in a web browser was investigated using a Web Server
Gateway Interface (WSGI) application that acts as a sonoUno server using WebAssembly32 and would maximise code reuse between
the actual sonoUno server, used by researchers for computationally intensive or batch jobs, local browsers for the computers of
citizen scientists and users of Zooniverse. A prototype, using Flask,33 looks very promising, and is being explored.
4 Citizen-engagement strategy
In REINFORCE, citizen science acts as the vehicle that aims to bridge the gap between large research infrastructures in physics and
society, providing the framework and tools for effective and sustainable interaction between them. On one hand, citizens are trained
in frontier science, they are in constant connection with researchers through dedicated communities of practice, they provide their
feedback, they voice their concerns, and they actively contribute to the exploration of the boundaries of knowledge. On the other
hand, researchers can receive help and support to refine their instruments and advance their research. This interplay leads to several
questions, such as: how do we design a successful citizen science project that balances social inclusion and scientific efficiency
[26]? What motivates citizens overall in an online citizen science project and how can we sustain this motivation over time [27, 28]?
Who are the potential citizen scientists [29] and are they actually able to develop new knowledge at the frontiers of physics [30].
32 https://webassembly.org/.
33 https://flask.palletsprojects.com/en/3.0.x/.
123
To address these questions, a citizen-engagement strategy was designed and implemented throughout the duration of the project
implementation [31], with prime focus on balancing inclusion and scientific efficiency. Large-scale events and inclusive design
are very important in terms of raising awareness of large numbers of citizens, offering everyone the opportunity to contribute to
a citizen-science project [32]. According to Spiers et al. [29], a typical Zooniverse project has a classification curve that displays
a peak of activity after launch, when then rapidly declines, a fact indicating that the majority of users contribute for a short time
before moving to another project that captures their attention. Meanwhile, the advanced requirements in terms of content knowledge
for the REINFORCE demonstrator projects and the large number of images for classification, combined with the retirement limits
set, as well as the need for careful classification of every image in order to achieve a scientifically sound result, posed a series of
requirements for the recruitment, continuous engagement and training of participants, which was estimated to be better facilitated
by forming a core group of citizens working on the demonstrator projects.
The strategy formulated was based on: an in-depth review of literature of existing citizen-science activities and public-engagement
frameworks, addressing citizens’ documented needs, motivations, characteristics, and interest in participating in citizen-science
projects in general; a dedicated citizen survey to support the collection of data, together with a series of vision-building workshops
and focus groups with small groups of citizens to identify key motivations of specific target groups (teachers, elderly citizens); a
task analysis of the four REINFORCE demonstrator projects to provide a careful consideration of the kinds of projects offered, the
scientific goals to be achieved, how to coordinate contributions, and arrange tasks within each demonstrator project.
To this end, the engagement team aimed to recruit volunteer citizens to participate in the project demonstrators, and also sought
to minimise the effects of losing momentum following the initial peak in participation, and to even increase and maintain a higher
participation over a longer period of time. More specifically, the strategy described the progressive approach for engaging citizens
in the demonstrator projects, while, at the same time, describing the tasks and the expected contributions from citizens and the
roles of the scientists in the process. A five-step approach is laid out, which involves enrichment of the demonstrator projects with
dedicated educational resources and training materials and encompasses activities aiming to inform citizens, to involve them, to
facilitate collaboration, to receive consultation and to empower them to become ambassadors of the projects in their communities.
Figure 13 details the engagement activities, citizens’ expected contributions and the researcher’s role in each step of the engagement
strategy. The engagement strategy was translated into a series of engagement activities that were implemented throughout the duration
of the project, both before the launch of the demonstrators and throughout their piloting period.
5 Participatory-engagement activities
The REINFORCE engagement activities focused both on broad-reaching activities that increased the project’s visibility and supported
the continuous influx of new interested participants; and on dedicated activities that engaged citizens to enhance their contribution
to the building of a core community of practice around a project that provides meaningful scientific output. It was estimated that
reaching out to at least 100,000 citizens would be an achievable and realistic performance indicator in terms of achieving the project
science and inclusion objectives. The ambitious goal of reaching out to this number of citizens was hindered by the emergence of the
Covid-19 pandemic and the consequent long periods of quarantine in Europe. This situation posed a great challenge, since traditional
forms of impactful engagement activities (such as public visits to large research infrastructures) could not be implemented for most
of the lifetime of the project. To this end, the project engagement activities were shifted to online and hybrid formats (for example
moving from face-to-face visits to virtual visits), while new types of activities were tested for the first time with great success.
Different audiences require different engagement methodologies. To this end, the REINFORCE consortium worked with varying
target groups, including: teachers, students, multi-modally-impaired citizens and senior citizens. Two particular cases are expanded
upon in detail here: using art as a vehicle to engage students in complex topics that go beyond the standard curriculum (Sect. 6.1);
and science for senior citizens (Sect. 7). Throughout the duration of REINFORCE, a total of 96 events, online and face-to-face, were
organised, all following the project’s engagement framework as described in Fig. 13. 47% of the events addressed the general public
and specific citizen groups; 25% were focused on reaching out to academics and researchers; 17% to teachers through dedicated
workshops; and 11% to students through dedicated implementation activities. Aggregating the citizens reached through events, as
well as through the online awareness activities for the project (such as Zooniverse newsletters), more than 500,000 citizens were
informed, while 30,000 citizens participated in events aiming to inform them and provide training. 21,400 citizens collaborated with
the project team and contributed classifications. The project team received consultation from 1,400 citizens, while it is estimated
that more than 1,000 citizens were empowered to become project ambassadors.
5.1 Impact of engagement activities and community building
To assess the impact of different types of events in terms of citizen participation and contribution, citizen engagement was system-
atically monitored throughout the project implementation using the Zooniverse portal and Google analytics. Quantitative measures
were applied to track the quantity of citizen contributions and type of engagement over time. This made it possible to build an
understanding of the dynamics of engagement in each of the REINFORCE demonstrator projects. The overall results are presented
in Table 1.
123
Fig. 13 Details of the REINFORCE citizen-engagement strategy
123
Fig. 14 Engagement activities in REINFORCE. Top left: Informing citizens through face-to-face visits at the Virgo detector; Top middle/right: Involving
citizens through hands-on activities; Bottom left: Inviting citizens to collaborate in dedicated challenges; Bottom middle: Receiving consultancy from citizens
through dedicated meetings; Bottom right: Empowering citizens to become REINFORCE ambassadors through the intensive International Training Course
Table 1 Metrics and results for Metric Value

citizen collaboration with the
REINFORCE demonstrator Total number of citizens collaborating with REINFORCE 21,400
projects
Total number of classifications 1,125,431
Total live-time of REINFORCE demonstrator projects (months) 41.6
Total person-time dedicated by citizens (months) 82.9
Number of discussions in Zooniverse community 5206
According to analysis, citizen scientists devoted double the effort (calculated based on the time allocated by the user logs) invested
by the research team. This is evident from the time dedicated by volunteers compared to the total live-time of the demonstrator
projects. This is a unique outcome of the work performed in the framework of the project, as it demonstrates that the contribution
of citizen scientists could really be quite significant. Figure 15 displays citizen classifications over time during the piloting period
of the project, which began on the 19th of October, 2021, and ended on the 25th of October, 2022.
By organising similar events for the different demonstrator projects, the project team had the chance to acquire useful data in terms
of the user-friendliness of the approaches and the interfaces used and the complexity of the tasks assigned to citizens. It was observed
that the period with the most engagement activities corresponds to a higher classification rate, as well as a considerably higher number
of participants. The engagement of the citizens over time and the overall demonstrator-project appeal were periodically assessed,
allowing the engagement team to providing the individual demonstrator project research teams with triggers as to when to act and
ensure their project remained interesting and motivating for the volunteers.
Awareness campaigns and dissemination activities, such as the events and newsletters flanking the official launches of the
demonstrator projects on Zooniverse, appear to have had a high impact in terms of daily classifications, with a low number of
classifications per participant and a high drop-out rate. According to the data, online events with massive participation, such as
dedicated challenges and competitions, provided 41% of the total classifications of the demonstrators, with the majority of these
provided by a core team of dedicated participants. As examples, the REINFORCE Winter Challenge34 acted as a social trigger that
engaged 720 citizens to perform a total of 80,000 classifications, while the EPS Citizen Science Competition35 engaged 374 citizens,
who registered and performed a total of 201,000 classifications. The sustained participation of citizens in these events displays a
high dependence on these challenges, as well as the related prizes.
It is possible to argue that the engagement activities managed to balance inclusion and scientific efficiency by reaching out to
a large audience through dedicated large-scale events, while forming a core community of participants who contributed the bulk
of the project classifications. This community-building was made possible by the commitment and accessibility of the researchers
operating the demonstrator projects. The Zooniverse Talk forum was the place where day-to-day interactions between researchers
and volunteers took place. For the research teams, this included answering questions and discussing the work of the demonstrator
project, as well as more general topics, announcing events, recognising citizen participation, empowering citizens and announcing
project results. Beyond forum interactions, community building was further facilitated through meetings (both face-to-face and
34 https://reinforce.ea.gr/winter-challenge/.
35 https://reinforce.ea.gr/eps-citizen-science-competition/.
123
Fig. 15 Daily Classifications versus time overall (black) and per demonstrator project (blue: GWitchHunters, orange: Deep Sea Explorers, red: New Particle
Search at CERN, green: Cosmic Muon Images) during the piloting period. The peaks that correlate with the organisation of specific events have been labelled
accordingly
Table 2 Overview of the Metric GWitchHunters Deep sea explorers New particle search Cosmic muon imgs
engagement metrics for the
REINFORCE demonstrator Live time (months) 11.4 8.6 12.1 9.5
projects. Data correspond to the
Total number of participants 8223 4221 7983 3951
period starting on the 19th of
October, 2021, and ending on the Total number of 645,592 109,409 260,637 109,703
25th of October, 2022 classifications
Median classification time 7.0 12.6 26.0 19.0
(sec)
Median classifications per 20.0 6.0 5.0 7.0
participant
Median time per participant 11.8 6.3 13.7 8.3
performing classifications
(mins)
Gini coefficient 0.84 0.84 0.89 0.86
online) which helped citizens and researchers to also put a face to the person on the other side of their monitor; the person they had
been collaborating with for a long time.
5.2 Highlights of individual demonstrator-project characteristics
The effectiveness of the engagement activities heavily depended upon the respective characteristics of each demonstrator project.
Table 2 provides an overview of individual demonstrator-project metrics for the piloting period.
The differences between the four demonstrator projects stem from the convolution of different contributing factors, including the
nature and difficulty of the tasks (from pattern recognition of noise features to track reconstruction), the respective characteristics of
the interfaces, the live-time of each demonstrator, the community support, the knowledge background required, the popularity of the
respective science topic, the support from other dissemination channels, etc. As an example, the easy-to-use game-based interface
of the GWitchHunters demonstrator, as well as its launch in a mobile-friendly version, have contributed significantly to the success
of the demonstrator project. A major contributing factor was also the support of a large research infrastructure, which increased its
dissemination activities in relation to the project a lot. With a median classification time of seven seconds, the effort per classification
can also be argued to be lower than that of the other demonstrators. These facts contribute to the appealing nature of the topic of
this demonstrator. According to Spiers et al. [29], “All Zooniverse projects display unequal volunteer classification contribution”.
This characteristic is also present in the four REINFORCE demonstrator projects. In order to identify the inequality in terms of
citizen contributions, the Gini coefficient for each project was calculated. The Gini coefficients presented in Table 2 [0.84−0.89]
123
indicate an unequal distribution of classifications among the participants of each demonstrator project, with a maximum variation
of the order of 5% between demonstrators. Based on the results provided by Spiers et al. [29], the Gini coefficients reported for the
REINFORCE demonstrators are within the range of coefficients reported for the bulk of Zooniverse astronomy-oriented projects.
A further comparison of REINFORCE with other existing relevant Zooniverse projects is currently a research in progress.
As an epilogue, for one to reverse or delay the rapid fall of interest in the demonstrator and all online citizen-science projects
in general, the engagement model tested in REINFORCE has great added value, a fact that is shown in the shapes of classification
distributions over time and effectively in the classification output of the project. which ultimately resulted in more data for the
demonstrator project research teams. Furthermore, organisation of such engagement activities goes hand in hand with community-
building activities, which require the commitment of researchers and citizens and affects the overall sustainability of citizen-science
projects over time.
6 Art & Science
As Root-Bernstein et al. [33] remind us, more than fifty years ago, the Nobel Prize winner van’t Hoff [34] proposed a correlation
between creative activities outside of science and scientific imagination. Other studies have shown that a positive association exists
between creative activities and success in science [35]. More generally, adopting art practice in elementary schools, has proven to
enhance young children’s learning in science [36]. Indeed, by using art in science, children are stimulated to explore scientific facts
and concepts in a multi-dimensional way, referring not only to inquiry and analysis, but also to imagination, emotions, and creativity
[37, 38].
6.1 International Youth Art & Science competition
Starting from these assumptions, the REINFORCE project launched an International Youth Art & Science competition under the
theme Humans as observer and listener of the cosmos. The focus of the contest was on concepts of fundamental physics, such as
space, time and the nature and structure of the Universe. These same concepts underlay a cross-reflection between the artists and
the scientists involved in the project.
The aim of the contest was to stimulate the participation of young people in the re-questioning of the fundamental concepts of
physics that are mentioned above, by asking them to translate into artistic manifestations their understanding of these notions. To
this end, entries in the REINFORCE International Youth Art & Science contest were grouped into two different age categories, from
four-to-twelve years old and from twelve to eighteen years old. The individual calls for submission were designed accordingly, in
order to capture the attention of potential participants and stimulate their interests in the underlying science. More precisely, for
young children the project sought artwork representing humans observing space; listening to the sound of stars; discovering the
nature of space and time; and the structure of the Universe. On the other hand, participants in the 12–18 range were asked to represent
humanity observing space and more generally nature, through a multitude of senses, from vision to sound and touch; discovering
the embedding of REINFORCE research in the Universe, including the environment; wondering about the nature of space, time
and matter. Around twenty artworks were submitted by young artists aged five to seventeen. These were contributed from three
different continents (Europe, Asia and America) and six different countries (Italy, Greece, Bahrain, Ecuador, Argentina and North
Macedonia), with a prevalence of female artists (60%). All participants were awarded a certificate of participation and invited to
attend a remote or in-person tour of the Virgo interferometer, located at the European Gravitational Observatory.
Although the number of entries was not high (mainly because the contest was launched during the summer recess period, when it
becomes much more difficult to engage schools, teachers and students), the contest afforded the children the opportunity to explore
and learn about fundamental physics, the Virgo experiment and the project itself. For the youngest artists, the contest was also the
first time they were confronted with new concepts, such as the sound of the stars and the nature of space and time. The REINFORCE
project, on the other hand, built on this initiative to involve young citizens in science and organised a visit to Virgo, as well as a
Kid’s Lab on Messengers from Space,36 during the European Researcher’s Night events of 2022.
6.2 Art & Science events organised within REINFORCE
The REINFORCE project has shown very clearly how the tools of citizen science can involve anyone in fundamental physics
research, something that is so distant from people’s daily lives: from exploring the fundamental constituents of matter with particle
accelerators to listening to the most violent cosmic phenomena through gravitational waves, to the study of neutrinos and cosmic
rays. This is research that almost always generates applications that bring us back to the context in which our lives take place,
through the development of innovative technologies or monitoring the natural or human-made environment of experiments.
However, there is another aspect that establishes an important link between fundamental physics research and society: its profound
influence on thought, art and culture in general. The new visions of the world and the cosmos, often revolutionary and counter-intuitive,
imagined and then verified with experiments by physicists, answer ancestral questions and humanity’s irrepressible aspiration to
36 https://www.reinforceeu.eu/platform-for-artistic-intervention/kids-lab-messengers-space-during-european-researchers-night-2022.
123
investigate the origin and reflect on its place in the cosmos. Because of this, they inspire new artistic and philosophical visions,
generate stories that enter the common imagination at all levels, from cinema to comics, from science fiction to interactive and
artistic installations.
Research infrastructures operating at the frontier of our knowledge can be among the leading actors in this process, developing
collaborations with visual artists and musicians, stimulating the production of installations for the general public and organising
public events, theatrical performances, conference-shows, in which scientists and artists can dialogue directly with the public, using
different languages. These experiences have a dual significance: on the one hand, they make knowledge and ideas that are often
not yet widely disseminated, accessible to the general public; on the other, they are a true laboratory for the development of new
languages and syntheses between science and imagination, knowledge and art.
In this sense, the dimension of game and entertainment also plays a crucial role in communicating effectively and bringing the
public closer to subjects with which they are not familiar. Recently, it has been emphasised in several contexts how an intertwining
of education and basic research, with innovative languages, is increasingly urgent, not least to counter the negative and persistent
effects of the pandemic on the education of the youngest [39].
Finally, these actions related to public communication and cultural imagery are complementary, supporting and reinforcing the
motivation of citizens directly engaged in citizen-science initiatives. There is a virtuous circle between public communication and
citizen-science actions, which evidently reinforce each other.
In this context and with this inspiring vision, EGO, in collaboration with partners, developed, within the framework of the
REINFORCE project, a series of initiatives and events aimed at the general public and, in some cases, particularly at the youngest.
The events mainly explored topics related to gravitational and multi-messenger astronomy, i.e. listening to the cosmos via
gravitational waves. The theme of cosmic sounds was especially explored and developed, both as a metaphor and narrative suggestion,
and as an actual object of musical research, also through the sonification and elaboration of scientific data, in collaboration with
musicians and sound designers.
Several public events were organised in this direction, also taking advantage of the intensive research on the sonification of data
undertaken within the REINFORCE project. Below is a brief description of a selection of some of the art and science events and
initiatives organised within REINFORCE.
• Black holes and gravitational waves in the cosmic concert - Concert/Installation - Online and in Rome (31st of December,
2020) Black holes and gravitational waves in a cosmic concert by Tomàs Saraceno, organised as part of the Festa di Roma, the
New Year’s Eve celebration in Rome. The event involved a public concert and installation by Tomas Saraceno, along with a
discussion involving the artist, the curators and scientists.37
• E quindi uscimmo a riveder le stelle - EGO, Cascina (Italy) (1st of October, 2021) A special evening under the stars of Virgo to
discover Dante’s cosmos and how we listen to and narrate the Universe today, in the unique setting of the European Gravitational
Observatory. Accompanying Lina Bolzoni, historian of Italian literature at the Scuola Normale Superiore, Stavros Katsanevas,
physicist and director of the European Gravitational Observatory, and Riccardo Pratesi, professor of mathematics and scholar of
the Divine Comedy, were the canti cantati of the ensemble A Ricuccata and the irreverent humour of David Riondino.38
• Lo Spazio e i Sensi - Pisa (Italy) (24th of September, 2021) Event held at La Nunziatina, Pisa. A night of science and music on
the many ways we can explore and listen to the cosmos, as well as the world around us, in the name of diversity and inclusion. The
guests were two exceptional protagonists of scientific research: Marica Branchesi, internationally renowned for her contribution
to the birth of so-called multi-messenger astronomy, and Wanda Diaz-Merced, a blind astronomer of Puerto Rican origin, one
of the global leaders in research into the sonification of astronomical signals. Stavros Katsanevas spoke with them, while the
conversation between space, sounds and gravitational waves was accompanied by music from the Italian-Greek singer Marina
Mulopulos.39
• Il suono dell’Universo, Lajatico (Italy) (19th of July, 2022) The Sound of the Universe - Dialogue under the stars between
gravitational waves and cosmic sounds. During the event, EGO director Stavros Katsanevas and Wanda Diaz Merced, world
leader in the sonification of astronomical data, talked about data sonification, what it means to listen to the cosmos and how
this is done, accompanied by the music of sound designer Massimo Magrini. Magrini’s contribution was developed within the
framework of REINFORCE.40
• Il suono dell’Universo, Genova (Italy) (30th of October, 2022 Within the context of the Genoa Festival of Science, The Sound
of the Universe took place: a dialogue on what it means to listen to the cosmos with Wanda Diaz-Merced and Stavros Katsanevas,
and the musical contribution of sound designer Massimo Magrini.41
• Athens Science Festival, Athens (Greece) (22nd and 23rd of October, 2022 Over 600 children, students and people of all
ages discovered Virgo and the science of gravitational waves through many interactive activities organized by Ellinogermaniki
Agogi. A room dedicated to gravitational waves was available, including an art and science installation developed by the Italian
37 https://www.ego-gw.it/blog/2020/12/30/black-holes-and-gravitational-waves-in-the-cosmic-concert-by-tomas-saraceno/.
38 https://www.ego-gw.it/blog/2021/09/17/e-quindi-uscimmo-a-riveder-le-stelle/.
39 https://www.ego-gw.it/blog/2021/09/24/lo-spazio-e-i-sensi-science-and-music-beyond-the-senses/.
40 https://www.ego-gw.it/blog/2022/07/15/gravitational-waves-and-cosmic-sounds-during-the-summer-around-pisa/.
41 https://www.ego-gw.it/blog/2022/10/29/the-sound-of-the-universe-at-genoa-science-festival/.
123
Institute of Nuclear Physics (INFN) and a model interferometer. Visitors were able to find many work-stations at which they could
experiment with the GWitchHunters Zooniverse demonstrator project.42
7 Senior-citizen science
Another initiative implemented in REINFORCE designed and developed with the intention of increasing accessibility to the work
of the demonstrator project research teams was the Senior Citizen Science course. The course was co-designed with members of
the Università della libera età (University of the free age) group, based in Cascina, in the vicinity of the European Gravitational
Observatory (EGO). Sessions in the course were provided by members of the REINFORCE Collaboration based at the University
of Pisa and at EGO.
Running a course for senior citizens in the middle of a global pandemic was inevitably more complicated than it might otherwise
have been and required a certain logistic elasticity in terms of the calendar, which had to be shuffled and re-shuffled, as well as in
terms of participation numbers, with the number of people available to participate from one session to the next being relatively fluid.
Despite this, the implementation ultimately proved a success and even resulted in a subsequent edition for the 2022/23 academic
year, i.e. well beyond the lifetime of the REINFORCE project.
7.1 The course syllabus
The first edition of the course took place during the 2021/22 academic year and was formed of nine sessions, plus a concluding visit
to the Virgo interferometer, situated at EGO. The course took the following form:
• Classical particle physics was given by Giancarlo Cella, Researcher at the Italian National Institute for Nuclear Physics (INFN),
and Professor of Astroparticle Physics at Pisa University, on the 6th of October, 2021, and covered the concept of space and time
in Galileo, Descartes and Newton; particle trajectory, mass, speed, acceleration; and the Laws of Newton and the movement of
the planets;
• Particles & waves in the XX century was delivered by Massimiliano Razzano, Professor of Experimental Physics at the University
of Pisa, on the 8th of October, 2021, and covered: the scale and structure of the Universe; sub-nuclear structures; Newton, Maxwell,
Faraday, Hertz; electromagnetic waves; Einstein and Relativity; Planck, Schrödinger and the quantum revolution; Rutherford;
linear and circular accelerators and the Standard Model;
• Waves: concept and detection took place on the 10th of November, 2021, and was given by Stefano Rinaldi, PhD at the University
of Pisa, and covered: the concepts of wave in Huygens, amplitude, frequency, interference; electromagnetic theory and waves,
aether; their detection with the Michelson-Morley interferometer;
• The cosmology of the (in)visible universe took place on the 3rd of December, 2021, and was delivered by Stavros Katsanevas,
REINFORCE Coordinator and Professor of Physics (Exceptional Class) at the Université de Paris. The session ranged across a
broad spectrum of themes, covering: Newton and Maxwell; Einstein’s Theory of Gravity; quantum physics — Bohr, Schrödinger,
Dirac, Heisenberg, Feynman; and the Standard Model of subatomic particles; astroparticle physics; cosmic rays; Hubble; gravita-
tional lensing; redshift; neutrinos; quarks, gluons, protons, neutrons; Higgs theory and supersymmetry; ‘shadow’ particles; matter
and antimatter; nucleosynthesis; and gravitational waves.
• The fifth session, Citizen Science: from theory to practice, was delivered by Francesco Di Renzo, postdoctoral researcher at the
University of Pisa, on the 9th of March, 2022, and focussed on the work of the GWitchHunters demonstrator project;
• The General relativity session was delivered by Valerio Boschi, Researcher at the INFN, on the 12th of January, 2022, and
covered: spacetime, gravitational waves and their detection, LIGO and Virgo; sources of noise, analyses of waves and glitches;
• On the 29th of April, 2022, Gary Hemming, REINFORCE Technical Manager, of EGO, held a session under the heading Brain-
storming and resolution of technical and theoretical problems, with the aim of evaluating directly with course participants,
the ways in which the course had worked well up to that point, as well as to try to establish how and where it could be modified
to ensure it reached its full potential;
• The sonification of gravitational waves session was delivered by Francesco Di Renzo on the 6th of May, 2022, and covered:
signals and what they are; a particular example of a gravitational-wave signal: GW150914; signals as vehicles of information;
the characteristics of a sound; pentagrams and time-frequency representations; listening to gravitational-wave signals; handling
noise; and the voice of a glitch;
• The final formal session in the course, Art & Science was again delivered by Stavros Katasanevas and focussed on: space-time-
matter; the senses and beyond: the two infinities; the notion of cosmos, order and violence, the singularities; from multi-messenger
to multi-sensorial; and from Uranus to Gaia.
The course concluded with a final visit of the participants to the Virgo detector at EGO on the 18th of May, 2022 (Fig. 16). The tour
included a view of the interferometer from the top of the North Arm Technical Building; a view of the Main Experimental Hall from
the atrium of the Central Building of the interferometer; the Virgo Control Room; the Hall of the Main Building, which is home
42 https://www.ego-gw.it/blog/2022/10/27/ego-at-athens-science-festival/.
123
Fig. 16 Senior Citizen Science

course participants visiting the
Virgo gravitational-wave
interferometer, based at the
European Gravitational
Observatory
to a miniature model of a Michelson-Morley laser interferometer, an original prototype of the Superattenuator suspension system,
used to reduce seismic isolation at the level of sensitive optical components in Virgo and a mock-up of one of the mirrors used in
the detector; entering into the North Arm tunnel of the detector in order to see the interferometer vacuum tube at close quarters.
7.2 What needed to be worked on?
An analysis of the evaluation session that was held with course participants, combined with feedback from session providers, brought
to light some interesting outcomes. Participants were genuinely interested in the subject matter approached, although finding the
right way to pitch it to them required some calibration. It became clear that a classical lecture-based environment was not the best
approach in terms of ensuring the active participation of the group, which favoured a more relaxed, informal back-and-forth. This
way of delivering sessions was subsequently adopted, with significant success.
It was also clear from the evaluation session that some members of the group were concerned that the level of the course had
aimed too high. Again, this feedback proved useful in recalibrating the sessions slightly, in order to try and ensure that they remained
as accessible as possible to as wider number of participants as possible.
7.3 What worked well?
Certainly, the more informal approach that was adopted as the course evolved proved a real success, building enthusiasm and
motivation within the group and resulting in an enjoyable visit to EGO, followed by a request to continue the course into the
subsequent academic year. One fascinating discovery from the feedback sessions, however, related to the session contents and their
degree of accessibility. The The Cosmology of the Visible Universe session, delivered by Stavros Katsanevas, was the only session
of the entire course that was delivered in English. Despite the fact that all of the group were mother-tongue Italian speakers, with a
self-confessed limited knowledge of English, the feedback from the group showed that this session was the one that they had found
most interesting and inspiring. Accompanied by a useful presentation, containing many images for reference, the group were clearly
inspired by the session and felt that they had been able to put together the pieces, picking up enough information from Stavros,
together with the slides he presented, to build a picture that was enthusiastically received.
Above all, the course proved the importance of creating an environment in which participants feel comfortable, can ask questions
and not have to worry about getting things wrong.
8 Critical and Scientific Thinking course
The REINFORCE Course on Critical and Scientific Thinking took place over the course of five different days between the 24th of
October and the 2nd of November, 2022. On the first day, Nobel Prize laureate Professor Saul Perlmutter, professor at the University
123
Fig. 17 The REINFORCE Critical and Scientific Thinking podcast playlist on YouTube
of California, Berkeley, Lawrence Berkeley National Laboratory, delivered a session to an audience connected remotely, via Zoom.
The second and third days were delivered, also via Zoom, by Stefano Gattei, Professor of Science in Society, at the University
of Trento, Italy. The fourth and fifth days were again delivered by Prof. Gattei, but were in-person at the European Gravitational
Observatory, with participants following both on-site and from remote.
Following the delivery of the course, the sessions were divided into 18 separate sessions, which have been made available on the
REINFORCE YouTube channel.43 The individual sessions are also available in podcast format, via a dedicated playlist.44
These sessions were broken down into the following themes, with the first three dedicated directly to the concepts at the core of
critical thinking and the remainder providing a detailed history of the evolution of scientific thinking following the Second Scientific
Revolution.
• 01 - A Missing Ingredient in Science Education – collective critical thinking and the ways in which crises of scientific knowledge
are also crises of decision-making;
• 02 - The Four Rs – the ‘Can-Do’ aspects of science, such as Scientific Optimism, as well as the ideas of Confirmation Bias and
Blind Analysis; and Group Thinking; should the Three Rs evolve into the Four Rs: Reading, Writing, Arithmetic and Scientific
Reasoning;
• 03 - A Critical Thinking Q &A;
• 04 - What is this thing called science? – the so-called Second Scientific Revolution in a historical context, the key figures
involved and its foundations;
• 05 - How to provide science with solid foundations...at the origins of critical thinking (Part 1) – how the Second Scientific
Revolution intertwined with a revolution in the philosophy of science;
43 https://www.youtube.com/@reinforceeu9192.
44 https://www.youtube.com/playlist?list=PLW-MjYVWVgCXKCege85iA75CNVFCj7rJI.
123
• 06 - How to provide science with solid foundations...at the origins of critical thinking (Part 2) – Wittgenstein, Logical
Positivism and the life of the Vienna Circle;
• 07 - Verification versus Falsification – Popper and the publication of The Logic of Scientific Discovery; Logical Asymmetry
and metaphysics;
• 08 - Science as Falsification – Popper’s criticisms of Logical Positivism; the theories of Einstein, Marx and Adler;
• 09 - The Test of Experience – Popper’s approaches to induction and deduction;
• 10 - Falsifiability as a standard of evaluation of scientific theories – Popper’s work and the ideas of conjecture and refutation;
• 11 - Dogmatism versus Infinite Regress versus Pyschologism - Fries’s Trilemma;
• 12 - The Game of Science – Rules and the Supreme Rule;
• 13 - The New Philosophy of Science - The New Philosophy of Science; Kuhn; Feyerabend; Toulmin, Hanson and Polanyi;
• 14 - Divergent versus Convergent Thinking in Science – convergent and divergent thinking, scientific revolutions and normal
research, in the works of Thomas Kuhn;
• 15 - Normal Science as Puzzle Solving – Kuhn, Pre-Science and Normal Science, Paradigms and Puzzles, Ptolemy and Kepler;
• 16 - Normal Science versus Extraordinary Science – Puzzles and Anomalies and how crises in science cause scientific
revolutions and new Paradigms;
• 17 - Incommensurability in Science – do scientific revolutions lead progress and how can this be measured?;
• 18 - Truth and Scientific Progress – the evolution of the Paradigm into the Disciplinary Matrix.
9 Raising awareness and sustainability
Across the lifespan of the project, REINFORCE adopted a wide spectrum of communication, outreach and dissemination activities,
with a view to building up and nurturing a thriving community of engaged members and to ultimately pursue the primary objective
of minimising the knowledge gap between large research infrastructures and society through citizen science.
The execution of a wide-ranging communication plan, inclusive of a series of joint actions, allowed to achieve an effective overall
visibility of the project across the major stakeholders, summarised as follows:
• Potential citizen scientists;

• The scientific and research community;
• The citizen-science community;
• Policymakers;
• The academic/innovation community.
The overarching key goal of the REINFORCE dissemination activities was to meaningfully reach out to each of these groups, to
become engaged and to take part in the exchange of good practices, to liaise with other initiatives in the field of citizen science
and to promote the overall concept that the research community has to move beyond traditional outreach programmes and learn to
design effective citizen-science activities that benefit their research and can ensure citizen involvement.
9.1 Social media
Social-media channels represented the centerpiece of the projects’ dissemination plan; the aim being to efficiently provide brief
real-time updates and news, but also to be used as a tool to share information about citizen science on the whole, inform about the
projects’ outcomes, to reach scientific and research community players and promote critical events’ activities and deliver real-time
coverage of virtual events wherever the project was involved.
9.2 Webinar series
The webinar series proved to be one of the most effective and tailored dissemination activities and allowed the project to attract
the interest of interested communities and participants. A set of five specific webinars was carried out and broadcast to support
the most tangible results of each demonstrator project and to spark interest around the correlated Zooniverse theme: this brought a
considerable stream of community members to Zooniverse and supported the development of each demonstrator project.
The webinar series covered the following themes:
• (on the 1st of June, 2020) Bridging the gap between Science and Society through Citizen Science45 was the opening webinar
in the series and provided a general overview of the REINFORCE project, with interventions from members of the research
teams of the individual demonstrator projects, as well as an explanation of the ways in which the engagement strategy would be
implemented;
45 https://reinforceeu.eu/events/webinars/bridging-gap-between-science-and-society-through-citizen-science.
123
• (on the 16th of October, 2020) How to help scientists in the Gravitational Wave noise hunt46 provided an introduction to
Zooniverse and concentrated on the GWitchHunters demonstrator project and the work of the Virgo Collaboration;
• (on the 26th of February, 2021) Discovering the unexplored deep marine environment47 focussed on the work of the Deep
Sea Explorers demonstrator project, presenting the work of the KM3NeT collaboration along with interventions on the themes
of bioluminescence, bioacoustics and marine ecology;
• (on the 11th of May, 2021) How to use cosmic rays in the study of geosciences and archaeology48 provided an introduction to
the Cosmic Muon Images demonstrator project, along with an overview of multiple different muographical applications;
• (on the 7th of July, 2021) Identifying undiscovered particles at the Large Hadron Collider49 presented the New Particle
Search at CERN demonstrator project and an introduction to the work of the ATLAS collaboration;
• (on the 24th of March, 2022) Opening Research Infrastructures – How citizens can play an active role in the advance of
ground-breaking research?50 was a high-level round-table and interactive workshop aimed at policymakers, education-policy
experts, think-tankers and academics, and had the aim of contributing to the co-creation process for a policy roadmap on research
infrastructures for citizen science in Europe;
• (on the 21st of November, 2022) REINFORCE-ing citizen science in large research infrastructures – Delivery of a policy
roadmap51 concluded the series of webinars, bringing together the members of the collaboration to present the policy roadmap
and reflect on the legacy of the project and explore the future.
10 Evaluation activities
10.1 Overview of methodology
The evaluation framework of REINFORCE is based on the logic model developed by Kurz and Kubek [40] and presents the
methodology regarding the evaluation of the project processes, outputs, outcomes and impact. The logic model provides a systematic
overview of logical relationships between a project’s resources, activities and results and thus constitutes an excellent framework
for measuring the outcomes and impacts of a project. The distinction between these levels is important in preserving the direction
of the project and thus the impact orientation. In this paper the focus is on the outcomes on the side of the participants, as well as
on the wider impacts.
To operationalise the evaluative dimensions, different evaluation instruments were developed: i) The Zooniverse feedback survey
to measure participant satisfaction; ii) the pre- and post-questionnaires, for participants to understand the individual outcomes in
terms of gained knowledge, scientific skills, and change of attitudes towards science; and iii) a self-assessment questionnaire, based
on the work by Kieslinger et al. [41], for the research teams implementing the demonstrators to discuss processes & feasibility as
well as outcomes & impact on the scientific, citizen science, and socio-ecological dimensions (Fig. 18).
10.2 Data analysis and results
10.2.1 Participant satisfaction
In order to evaluate the participant experience on Zooniverse, a short online survey was developed, to be completed by the participants.
The survey consists of seven questions and basic demographic data (gender and age) and has been analysed (n=168).
On a scale from 1 (=strongly disagree) to 5 (=strongly agree), averages range from approximately 3 to above 4, indicating a
general moderate-to-high level of user satisfaction. When compared to one another, it is item 3 (Fig. 19), I find the project tasks
difficult, which has the lowest score. Considering the nature of the item, this is a satisfactory response. An average difficulty is ideal
as according to motivation theory [42], tasks should be neither too difficult nor too easy, in order to motivate people to continue with
them. Those with high motivation tend to choose moderately difficult tasks, feeling that they are challenging, but within reach [43].
10.2.2 Individual impact
The online pre and post survey is meant to detect new knowledge, new skills and changes of attitude, among participants in the
four Zooniverse demonstrator projects, with questions related to knowledge on the role of research infrastructures, basic physics
concepts, data recognition and analysis skills, awareness of science and scientific work. It also includes the assessment of change
46 https://reinforceeu.eu/events/webinars/how-help-scientists-gravitational-wave-noise-hunt.
47 https://reinforceeu.eu/events/webinars/discovering-unexplored-deep-marine-environment.
48 https://reinforceeu.eu/events/webinars/how-use-cosmic-rays-study-geosciences-and-archaeology.
49 https://reinforceeu.eu/events/webinars/identifying-undiscovered-particles-large-hadron-collider.
50 https://reinforceeu.eu/events/webinars/research-infrastructures-citizens-science.
51 https://reinforceeu.eu/citizen-science-webinar.
123
Fig. 18 The citizen-science evaluation framework used within the REINFORCE project
Fig. 19 Average scores for participant satisfaction
actions and behaviour: e.g. science career motivation; and living conditions: e.g. empowerment through contributing to citizen
science. In addition, the questionnaire contains some basic demographic information, such as age, gender, highest educational level
and a question regarding a background in science.
In the pre-post-test design as applied in our study, the pre-test aims to measure the baseline, in this case in three distinct areas:
knowledge, skills and attitudes. The following table shows the average scores on a five-point Likert scale (n 301) from the pre-test,
i.e. when first logging into the Zooniverse project and post-test, completed after one month. As the paired sample t-test values reveal,
three differences are significant (Fig. 20).
123
Fig. 20 Pre- and

Post-questionnaire scores
In a paired sample t-test, the T -value and df (degrees of freedom) are two important statistical values that help in the interpretation
of the results of the test:
• The T-value, also known as the T -statistic, measures the size of the difference relative to the variation in the sample data. It is
calculated by taking the difference between the mean of the paired samples and dividing it by the standard error of the differences.
Generally, the larger the T -value, the more likely it is that the difference between the means is statistically significant. A higher
absolute T -value indicates a larger difference between the two means. The sign of the T-value (±) indicates the direction of the
difference. If it is positive, the first sample mean is higher; if it is negative, the second sample mean is higher.
• Degrees of Freedom (df): in statistics, degrees of freedom represent the number of values in the final calculation of a statistic
that are free to vary. In the context of a paired sample t-test, the degrees of freedom represent the number of pairs of observations
minus 1. For a paired sample t-test, df n − 1, where n is the number of pairs of observations. Degrees of freedom are important
because they determine the critical value of the t-distribution, which is used to determine statistical significance.
• Sig stands for significance: values below.05 mean the difference is statistically significant and cannot be explained by coincidence.
In summary, the T -value tells you the size and direction of the difference between the means of the paired samples, while the degrees
of freedom indicate the number of independent pieces of information used to calculate the T -value and are important in determining
the critical value for significance testing.
There are significant changes in science-related skills, and in both reported and tested knowledge. In other words, participants
improve their knowledge and their science-related skills over the course of their participation in the demonstrator projects. They join
already with a very positive attitude towards science and this level does not change. The same applies to science-related self-efficacy.
10.2.3 Perceived outcomes and impacts by researcher teams
The self-assessment tool was meant to be conducted as a critical reflection exercise by the demonstrator project teams during the
lifetime of the projects. The exercise was conducted twice: first, one month after the official launch of the demonstrator project on
Zooniverse and, second, at a more advanced stage. This allowed the evaluation team to observe progress in reaching objectives and
reflect together with the researcher teams on the outcomes and possible impact of their projects.
The self-reflection exercises were set up as four separate Zoom calls with each of the individual research teams. In these, the
members of each component research team agreed, in a discussion, on joint values on the Likert scale that ranged from 1 (=strongly
disagree) to 7 (=strongly agree). The evaluation team led the discussion via the online questionnaire and recorded the discussions
and reflections, which added further details to the allocated scores.
The four reflection exercises with the demonstrators were meant to reflect on processes during the implementation of the citizen-
science demonstrator projects, to see how processes were set up to be feasible in a scientific dimension, as well as for participating
citizens in a wider socio-ecological dimension. Furthermore, the teams were asked to reflect on the impacts of their projects on
these different dimensions, i.e. science, citizens and society. As the results show (Fig. 21), the demonstrator-project teams perceived
the processes as well-designed and feasible and rather equally implemented in relation to citizens, science, and socio-ecological
systems. The teams recognised already manifested outcomes and impacts, again referring mostly to participating citizens and the
science side, and somewhat less to socio-ecological systems.
10.2.4 Evaluation conclusions
To conclude, in general we can say that the demonstrator projects have successfully reached their objectives. According to the
research teams in the self-assessment exercise, the science side has benefitted, as the algorithms used to categorise data have been
improved thanks to the manual classifications of the citizen scientists.
While the citizen scientists have enjoyed their participation, as the results of the Zooniverse experience survey show, and benefited
at the same time by gaining new scientific knowledge and scientific skills.
123
Fig. 21 Mean self-assessment

results
11 The policy roadmap
The policy roadmap [44] aims to promote the integration of citizen science within major research frameworks. The REINFORCE
project seeks to bridge the divide between scientific research and the broader public by emphasizing citizen science. Various factors
contribute to the existing status of citizen science in the academic realm. Over the past ten years, research bodies have diligently
pursued strategies to promote the approach, but, REINFORCE stands out as a pioneering and comprehensive venture in promoting
citizen science within major research infrastructures, specifically targeting astroparticle physics research.
Given REINFORCE’s unique stance compared to other initiatives, it is crucial for its participants to gather insights and reflect
on their experiences. Their objective is to provide effective guidelines that benefit policymakers, researchers, citizen scientists, and
the general public. The roadmap outlines the steps taken to execute these tasks and presents the findings of this policy initiative.
In detail, the roadmap incorporates results and evaluations from the REINFORCE demonstrator-project and monitoring feedback.
Key challenges are seen in the governance models of extensive research systems, especially concerning the recognition and resources
available to scientists for citizen-science endeavours. Fundamentally, the hurdles to integrating citizen science into these frameworks
revolve around capacity, capability, and motivation.
The roadmap provides suggestions to overcoming these barriers, capitalising on the chance to firmly establish citizen science
in expansive research environments. REINFORCE is uniquely positioned to serve as a mediator between scientific research and
society. While the adoption of citizen science in major research settings is on the rise, it is yet to become the norm. Nevertheless,
REINFORCE showcases that with a well-defined, strategic approach, there’s a promising path forward for the mainstream acceptance
of citizen science in astroparticle physics research infrastructures.
11.1 The Wider Landscape of the European Open Science Cloud
The European Open Science Cloud (EOSC) is a major strategic priority of the European Commission, who have invested over e500
million in making the data from all major European research facilities Findable, Accessible, Interoperable and Reusable (FAIR).
The aspiration of the EOSC Strategic Research and Innovation Agenda (SRIA [45]) is to deliver a working Web of FAIR data and
services, providing European scientists and social scientists with the benefits of data-driven research, with Europe taking a global
lead not just in scientific research but also in research data management.
This initiative is well-timed. The extraordinary recent growth in the volume and complexity of scientific data, across many
scientific disciplines, has opened up new avenues of scientific discovery, from genomics to astronomical imaging to social sciences
working with the firehose of data from social networks. However, the scale and complexity of the data avalanche poses significant
computational and operational challenges. Machine learning/artificial intelligence can alleviate some of these problems, but unless
the specific problems at hand are amenable to unsupervised ML techniques, the technologies still require labelled training sets.
Data mining by experts, or by volunteers, therefore still plays a central role in the science exploitation of open data. Furthermore,
human classifiers easily exceed the capabilities of even the most sophisticated ML, except usually in speed and scale. ML cannot take
a step back and un-ask the fundamental question being asked in response to oddities in the data. Serjeant, [46] gave a hypothetical
example of an ML algorithm tasked with recognising and counting faces in images of commuters. In this thought experiment, one
123
image contained a very prominent man in a clown costume. The machine vision would just count the face, but the human would
spot the outlier in the data and respond to it. Human volunteers can also identify features in the data that are too rare for training
ML, such as in the Snapshot Serengeti citizen-science project that measured biodiversity by examining camera-trap images. Over
100 thousand wildebeest were identified, more than sufficient for training ML, but the volunteers also found 17 examples of the
extremely elusive zorilla [47].
Citizen science is therefore key to many science applications of FAIR data in EOSC. Fundamentally, citizen science exists as a
tool for doing better research. For example, data-mining by citizen scientists can be likened to deploying a giant biological computer
to the research problem. It is a tool, in the same sense that a detector beam-line or a spectrometer is a research-enabling tool. As with
any other research facilities or capabilities, there are research objectives that are well-suited to the deployment of citizen science,
and objectives that are not. Citizen science can also help with education and public engagement, but it exists for the purposes of
conducting research and is optimised on that basis. If education or public engagement is the primary goal, there are often alternative
tools or activities that may perform better than citizen science against non-research metrics.
There is a subtle logical fallacy in trying to make citizen science optimal for public engagement. It is not logically possible
to optimise against two different things simultaneously. For example, there is no highest and hardest mountain to climb, because
the highest (Everest) is not the same as the hardest (K2, arguably). One can optimise subject to a constraint, e.g. the best research
subject to the constraint that it uses volunteer classifications, or the most educational benefit that happens to also use research, or the
highest point that is also accessible by car. However, one always has to choose what is being optimised for, and in citizen science it
is conducting scientific research. It is the highest point, or the nicest road, but it cannot be both. It has to be one or the other.
To encourage the growth of citizen science as a research tool, as opposed to viewing it as mechanism for education, the ESCAPE
and EOSC-Future Horizon 2020 projects have been building exemplar projects, similar to those developed in REINFORCE, as
well as several Jupyter notebooks and tutorials for creating citizen-science projects, managing them and folding in ML. These
demonstrator projects, in brief, are:
• Galaxy Zoo: Clump Scout. This project sought volunteers to identify clumps of star formation in images of galaxies, in order
to illuminate the physical causes of the rapid drop in comoving volume-averaged star formation rate since Cosmic Noon. The
science goal requires a human assessment for what features are regarded as clumps. Over 14 000 volunteers provided over 1.7
million classifications.52
• SuperWASP: Black Hole Hunters. This project used volunteers to identify the characteristic signature of a stellar-mass black
hole gravitationally lensing a companion star. The data are inhomogeneous and require a human assessment of data quality. Over
5,000 volunteers provided over 2.1 million classifications.53
• Radio Galaxy Zoo: LOFAR. This project used data from the Low Frequency Array (LOFAR) European radio telescope array.
LOFAR is a science and technology pathfinder for the Square Kilometer Array. The classification task asked volunteers to identify
regions of radio emission and to find the counterpart optical galaxy. Despite the complexity of the workflow, over 11 000 volunteers
contributed nearly one million classifications.54
• Galaxy Zoo: Cosmic Dawn. This was an incarnation of the longest-running project on the Zooniverse citizen science platform,
Galaxy Zoo. The project asked volunteers to classify the morphologies of galaxies in images from the HyperSuprimeCam camera
on the 8 m Subaru telescope in Hawaii, which had been produced by the Cosmic Dawn team in preparation for their future imaging
by the Euclid space telescope as one of its deep fields. The Galaxy Zoo: Cosmic Dawn project was completed in June, 2023, and
a general data release paper is currently in production.
• Knitting Leaflet Project. The ESCAPE project was focussed on the development of the astro/physics cell of EOSC, but there
were still aspirations through EOSC-Future to develop EOSC citizen science demonstrators to wider subject areas. EOSC-Future
therefore assisted in the development of a project to characterise historic knitting patterns in early Twentieth Century domestic
magazines from the UK. The project will provide insights into how versions of femininity connected to fashion and consumption,
and how they changed with time.55
• African Indigenous Knowledge. EOSC-Future also assisted in the development of this citizen-science demonstrator project,
aiming to capture, document and share African indigenous food system knowledge for promoting sustainable nutritious food
production, processing and consumption, initially in Sierra Leone.
The SRIA contains many mentions of citizens as stakeholders in various contexts of the EOSC, as well as several exemplars of good
EOSC practice involving citizen scientists. Perhaps partly as a result of our advocacy for citizen science as a research tool across
all disciplines, the latest Horizon work programme includes explicit EOSC support of citizens and citizen scientists for societal
challenges. However, one indicator of the barriers that still need to be overcome is that in the four key priority areas identified for
EOSC in the SRIA, citizen science features in Priority 2: Bridging the education gap: coordinating and aligning curricula for
students and researchers. It is true that all use of FAIR data in EOSC for research will involve educational materials, and the further
52 https://www.zooniverse.org/projects/hughdickinson/galaxy-zoo-clump-scout.
53 https://www.zooniverse.org/projects/hughdickinson/superwasp-black-hole-hunters.
54 https://www.zooniverse.org/projects/chrismrp/radio-galaxy-zoo-lofar.
55 https://www.zooniverse.org/projects/elliereed185/knitting-leaflet-project.
123
Fig. 22 Methodology for the development of the roadmap
that the data are from one’s discipline area, the more important these educational tools become. However, our policy aspiration is
that future versions of the SRIA have citizen science expressed as one of the research tools and methodologies.
11.2 Development
The Policy Roadmap on Research Infrastructures for Citizen Science in Europe seeks to offer policy guidelines to enhance the role
of research infrastructures in citizen science by consistently engaging the public.
This roadmapping process was divided into three primary phases:
1. Recognising policy voids that prevent the initiation and execution of citizen-science projects within research infrastructures;
2. Drafting potential future policy challenges and outlining plans related to advancing research infrastructures focused on citizen
science;
3. Establishing clear policy guidance and actionable suggestions for all participating entities.
In a more detailed context, the roadmap addresses the following considerations:
• What are the principal policy voids and hurdles to standardise the adoption of citizen science within research infrastructures?
• Which tools and incentives are essential to mitigate these challenges?
• How might these challenges impact various policy areas and society at large?
• What overarching advice can be given to policymakers, researchers, and civil society groups to hasten the integration of citizen
science in research settings?
• What measures need initiation to enact these recommendations?
A comprehensive methodology for the construction of the roadmap is depicted in the subsequent chart (Fig. 22). In it, blue boxes
depict specific executed activities, purple symbols indicate input sources for the roadmap, and the golden pointers signify progression
in the roadmap’s creation.
From the analysis, it is evident that the evaluation of the project outcomes and acquired insights played a crucial role in identifying
both the driving factors and obstacles, as well as the policy goals. Subsequently, aligning these policy objectives with current policy
measures allowed for a thorough examination of discrepancies and external challenges that impede the integration of citizen science
within research infrastructures. Based on these identified gaps, the research team formulated corresponding policy challenges,
recommendations, and steps for implementation.
11.3 Main recommendations
• Prioritise STEM in education. In REINFORCE and through a collaboration with the FRONTIERS56 network, a series of initia-
tives that brought the REINFORCE citizen-science projects into the realm of education was organised, including the production
of educational resources and teacher guidelines, as well as the organisation of dedicated workshops and courses for teachers.
• New rules of attribution of scientific discoveries and merits. REINFORCE provides an example of an array of effective tools for,
and approaches to, consultation. Within the REINFORCE demonstrator projects, several consultation activities were conducted
56 https://frontiers-project.eu/.
123
with both scientists and citizen scientists, such as reflection workshops, to identify meaningful actions that might be used in the
recognition of citizen-science effort.
• Foster design experience to ensure motivation, sustained engagement, and inclusivity. In REINFORCE, the participatory
engagement activities informed more than 1,000,000 citizens in a variety of events organised by the project and through a
systematic communication strategy aimed at targeted audiences, to involve around 30,000 citizens in the project activities, and to
collaborate with 21,437 citizens in the demonstrator work.
• Incentives to open data on the side of research infrastructures. REINFORCE clearly worked in this regard, as all data were
made available openly as part of a specific community on the Zenodo platform.57
• Boost the European Open Science Cloud. Many members of the REINFORCE consortium have relationships with EOSC
through its providers.
• Introduce citizen science into educational strategy. The REINFORCE consortium includes many education institutions that are
introducing citizen science into their programmes. This is in line with the European Astroparticle Physics Strategy 2017–2026.58
• Boost evaluation and monitoring of citizen science. The REINFORCE sustainability plan calls for the establishment of a think
tank, which would ultimately carry out the long-term monitoring and consultation, as well as mentoring for, new citizen-science
projects.
• Include educators in programme design. This has been shown to be an effective way of empowering citizens to become
ambassadors for projects. Through dedicated training courses, organised by REINFORCE partner EA and which partners of the
REINFORCE collaboration have co-organised, teachers and citizens were empowered through dedicated training activities.
• Community establishment, scale-up, sustain and engagement. With regard to this, the REINFORCE approach highlights the
importance of task analysis and close collaboration with boundary organisations to help understand the educational perspective
of any citizen-science initiative.
• Involve policymakers throughout the project life-cycle. In this regard, REINFORCE paved the way by adopting a strategic
approach towards policy and organising workshops aimed at creating common ground in the conversation between scientists and
policymakers through citizen science.
• Continue to pursue and encourage inclusion and diversity. In REINFORCE, CONICET and EGO have carried out a series
of activities to extend the senses used in scientific inference, beyond the visual, and included in the overall effort and scientific
community sense-disabled people (especially the visually disabled) and senior citizens, but have also profited from the different
points of view of artists and artisans.
12 Thoughts and conclusions at the end of the REINFORCE project
Reflecting on the three years of REINFORCE it is difficult not to feel that the project can be regarded as a success. All four
demonstrator projects were successfully officially launched on Zooniverse, despite one of them — New Particle Search at CERN
— being of such complexity that it was necessary to move the more advanced stages of the project out on to the separate HYPATIA
platform. All of the demonstrator projects have been successful at engaging volunteers in the scientific process with real-world
scientific results. Targets set within the REINFORCE project were demanding and the citizen-engagement strategy (Sect. 4) and
its subsequent implementation (Sect. 5) resulted in the successful delivery of more than 1 million classifications within the piloting
period, contributed by a community of more than 20,000 committed individual citizens.
Collaboration between the demonstrator project and sonification teams has led to the availability of new sonification forms via
the sonoUno library and the work of the sonification team has proven successful in engaging numerous volunteers in the further
development of the software.
The senior citizen science programme, following initial disruptions because of the pandemic and subsequent to an evaluation
activity undertaken with the participants, proved to be so successul that it led to a second edition, which took place, still under
the auspices of REINFORCE, throughout the 2023/24 academic year, well beyond the lifetime of REINFORCE itself, while the
art and science initiatives were very successful at bringing new participants into the world of the project and into the arena of the
demonstrator projects.
The policy roadmap provides a series of useful recommendations and objectives to pursue in order to effectively embed citizen
science in large research infrastructures. Implementation steps on the path to effective take-up are detailed, incorporating responsible
research and innovation at the heart of the process. A broad range of recommendations have been provided, covering a number of
different areas, including: open access to services; standards; collaboration; inclusion; science and society; sustainability; governance
models; scaling-up; and policy-making processes.
Exploring the potential of frontier citizen science for inclusion and diversity was at the very heart of the project and much work
has been done to this end, while the project also served to build a healthy international working collaboration. This in itself was not
guaranteed, given that the Covid-19 pandemic broke just three months after REINFORCE started, with all of the related tumult and
57 https://zenodo.org/communities/reinforceeu/.
58 https://www.appec.org/roadmap.
123
complication that it brought to an initiative that had been designed to heavily rely on in-person engagement activities. The fact that,
despite this, the REINFORCE collaboration—built from twelve partner organisations, based in seven different countries, across two
different continents, and involving, over the lifetime of the project, more than 70 people from twelve different countries—grew to
be a respectful and friendly environment, in which all members were able to contribute, was also a very happy outcome.
Acknowledgements The REINFORCE project has received funding from the European Union’s Horizon 2020 project call H2020-SwafS-2018-2020 funded
project Grant Agreement No. 872859.
Funding Open access funding provided by Università di Pisa within the CRUI-CARE Agreement.
Data availability In general, where data used in the production of this document is available, it can be found in the REINFORCE Zenodo community57 .
Non-publicly available data The data used to produce the spectrogram images supporting Figs. 1, 2 and 8 were provided by the Virgo Collaboration and
are not publicy-available. Their usage is subject to a memorandum of understanding between the REINFORCE and Virgo Collaborations. The data used
to produce the images supporting Figs. 6, 7 and 9, 10 were provided by the KM3NeT Collaboration and are not publicy-available. Their usage is subject
to a memorandum of understanding between the REINFORCE and KM3Net Collaborations. The data used to produce the images supporting Figs. 19–21
were collected via online questionnaires during the lifetime of the REINFORCE project and are not publicly-available. Publicly available data The data
used to produce the images supporting Figs. 4, 5 and 11 were provided by the ATLAS Collaboration and are publicy available in the ATLAS Open Data
repository: https://opendata.atlas.cern/. The anonymised Zooniverse classification data used to produce Tables 1-2 and Fig. 15 are available in the Zooniverse
Zenodo community. Separate data products are available for each individual demonstrator: GWitchHunters (https://doi.org/10.5281/zenodo.7729033), Deep
Sea Explorers (https://doi.org/10.5281/zenodo.7732577), New Particle Search at CERN (https://doi.org/10.5281/zenodo.7728999), Cosmic Muon Images
(https://doi.org/10.5281/zenodo.7728970).
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution
and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative
Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons
licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended
use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a
copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
References
1. M. Razzano, F. Di Renzo, F. Fidecaro, G. Hemming, S. Katsanevas, Gwitchhunters: machine learning and citizen science to improve the performance
of gravitational wave detector. Nucl. Instrum. Methods Phys. Res. Sect. A 1048, 167959 (2023). https://doi.org/10.1016/j.nima.2022.167959
2. F. Acernese et al., Advanced virgo: a second-generation interferometric gravitational wave detector. Class. Quantum Gravity 32(2), 024001 (2014).
https://doi.org/10.1088/0264-9381/32/2/024001
3. The LIGO Scientific Collaboration, et al.: Advanced ligo. Class. Quantum Gravity 32(7), 074001 (2015) https://doi.org/10.1088/0264-9381/32/7/074001
4. KAGRA Collaboration, et al.: Advanced ligo. Nat. Astron. 3, 35–40 (2019) https://doi.org/10.1038/s41550-018-0658-y
5. M. Zevin et al., Gravity spy: integrating advanced LIGO detector characterization, machine learning, and citizen science. Class. Quantum Gravity 34(6),
064003 (2017). https://doi.org/10.1088/1361-6382/aa5cea. arXiv:1611.04596 [gr-qc]
6. M. Razzano, E. Cuoco, Image-based deep learning for classification of noise transients in gravitational wave detectors. Class. Quantum Gravity 35(9),
095016 (2018). https://doi.org/10.1088/1361-6382/aab793
7. B.P. Abbott et al., Observation of gravitational waves from a binary black hole merger. Phys. Rev. Lett. 116(6), 061102 (2016). https://doi.org/10.1103/
PhysRevLett.116.061102. arXiv:1602.03837 [gr-qc] √
8. Atlas Collaboration, Search for massive, long-lived particles using multitrack displaced vertices or displaced lepton pairs in pp collisions at s 8
TeV with the ATLAS detector. Phys. Rev. D (2015). https://doi.org/10.1103/physrevd.92.072004
9. The ATLAS collaboration: Review of the 13 TeV ATLAS Open Data release. Technical report, CERN, Geneva (2020). All figures including auxil-
iary figures are available at https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PUBNOTES/ATL-OREACH-PUB-2020-001. http://cds.cern.ch/record/
2707171
10. C. Kourkoumelis, S. Vourakis, Hypatia-an online tool for atlas event visualization. Phys. Educ. 49(1), 21 (2014). https://doi.org/10.1088/0031-9120/
49/1/21
11. G. Bonomi et al., Applications of cosmic-ray muons. Prog. Part. Nucl. Phys. 112, 103768 (2020)
12. L.W. Alvarez et al., Search for hidden chambers in the pyramids. Sci. New Ser. 167, 832–839 (1970)
13. K. Nagamine et al., Method of probing ineer-structure of geophysical substance with the horizontal cosmic-ray muons and possible application to
volcanic eruption. Nucl. Instr. Methods Phys. Res. A 365, 585–595 (1995)
14. H. Tanaka et al., High resolution imaging in the inhomogeneous crust with cosmic-ray muon radiography: the density structure below the volcanic crater
floor of mt. Asama, Japan. Earth Planet. Sci. Lett. 263, 104–113 (2007)
15. S. Procureur, Muon imaging: principles, technologies and applications. Nuclear Inst. Methods Sec. A 878, 169–179 (2018)
16. T. Avgitas, S. Elles, C. Goy, Y. Karyotakis, J. Marteau, Muography applied to archaelogy (2022). https://doi.org/10.48550/arXiv.2203.00946
17. J. Marteau, T. Avgitas, D6.1 - CR Analysis Dataset. Zenodo (2023). https://doi.org/10.5281/zenodo.10017363
18. T. Avgitas, G. Wasseige, G. Hemming, D6.3 - Citizen Science Projects on Cosmic Rays. Zenodo (2023). https://doi.org/10.5281/zenodo.7576581
19. J. Marteau, T. Avgitas, D6.4 - Comparative study on CR data. Zenodo (2022). https://doi.org/10.5281/zenodo.7403362
20. A. Zanella, C.M. Harrison, S. Lenzi, J. Cooke, P. Damsma, S.W. Fleming, Sonification and sound design for astronomy research, education and public
engagement. Nat. Astron. 6(11), 1241–8 (2022)
21. B. García, W. Díaz-Merced, J. Casado, A. Cancio, Evolving from xsonify: a new digital platform for sonorization. EPJ Web Conf. 200, 01013 (2019)
22. J. Casado, W. Díaz-Merced, B. García, M. Schneps, K. Kolemberg, M. Rychtarikova, N.B. Roozen, Sonification as a tool for data analysis: usability
and compliance evaluation study. Int. J. Sociotechnol. Knowl. Dev. IJSKD 14(1), 1–27 (2022)
23. J. Casado, B. García, P. Gandhi, W. Díaz-Merced, A new approach to sonification of astrophysical data: the user centred design of sonouno. Am. J.
Astron. Astrophys. 9(4), 42–51 (2022)
123
24. W.L. Diaz Merced, Sound for the exploration of space physics data. PhD thesis, School of Computing Science, University of Glasgow (2014). https://
theses.gla.ac.uk/5804/
25. J. Peirce, J.R. Gray, S. Simpson, M. MacAskill, R. Höchenberger, H. Sogo, E. Kastman, J.K. Lindeløv, Psychopy2: experiments in behavior made easy.
Behav. Res. Methods 51(1), 195–203 (2019)
26. T. Schäfer, B. Kieslinger, Supporting emerging forms of citizen science: a plea for diversity, creativity and social innovation. JCOM (2016) https://doi.
org/10.22323/2.15020402
27. M.J. Raddick, G. Bracey, P.L. Gay, C.J. Lintott, P. Murray, K. Schawinski, A.S. Szalay, J. Vandenberg, Galaxy zoo: exploring the motivations of citizen
science volunteers. Astron. Educ. Rev. 9(1), 1–18 (2010). https://doi.org/10.3847/AER2009036. arXiv:0909.2925
28. J. Reed, M.J. Raddick, A. Lardner, K. Carney, An exploratory factor analysis of motivations for participating in zooniverse, a collection of virtual citizen
science projects. 46th Hawaii International Conference on System Sciences, Wailea, HI, USA, pp. 610–619 (2013) https://doi.org/10.1109/HICSS.2013.
85
29. H. Spiers, A. Swanson, L. Fortson, B. Simmons, L. Trouille, S. Blickhan, C. Lintott, Everyone counts? design considerations in online citizen science.
JCOM (2019) https://doi.org/10.22323/2.18010204
30. A. Greenhill, K. Holmes, G. Graham, J. Cox, J. Woodcock, K. Masters, E.Y. Oh, Crowdsourcing citizen science: Exploring the tensions between paid
professionals and users. J. Peer Prod. 1 (2017)
31. E. Chaniotakis, S. Sotiriou, J. Koslowsky, C.M. Fabian, E. Unterfrauner, J. Tschank, B. García, W. Diaz-Merced, D2.1 - citizen engagement plan (2022)
32. J. Cox, E.Y. Oh, B. Simmons et al., Defining and measuring success in online citizen science: a case study of zooniverse projects. Comput. Sci. Eng.
17(4), 28–41 (2015)
33. R. Root-Bernstein et al., Arts foster scientific success: avocations of nobel, national academy, royal society, and sigma xi members. J. Psychol. Sci.
Technol. 1(2), 51–63 (2008)
34. J.H. Van’t Hoff, Imagination in science (g. e. springer, trans). Mol. Biol. Biochem. Biophys. 1, 1–18 (1967)
35. R. Root-Bernstein, M. Bernstein, H. Garnier, Correlations between avocations, scientific style, work habits, and professional impact of scientists. Creat.
Res. J. 8(2), 115–37 (1995)
36. C. Caiman, B. Jakobson, The role of art practice in elementary school science. Sci. Edu. 28, 153–175 (2019)
37. C. Caiman, I. Lundegård, Young children’s imagination in science education and education for sustainability. Cult. Sci. Edu. 13(3), 687–705 (2018)
38. R. Root-Bernstein, Arts and crafts as adjuncts to stem education to foster creativity in gifted and talented students. Asia Pac. Educ. Rev. 16, 203–212
(2015)
39. R.D. Blandford, K.S. Thorne, Post-pandemic science and education. Am. J. Phys. 88(7), 518–520 (2020). https://doi.org/10.1119/10.0001390
40. B. Kurz, D. Kubek, Social impact navigator: the practical guide for organizations targeting better results (Phineo gAG, Berlin, 2016)
41. B. Kieslinger, T. Schäfer, F. Heigl, D. Dörler, A. Richter, A. Bonn, Evaluating citizen science-towards an open framework (UCL Press, London, 2018)
42. D.C. McClelland, Human Motivation (Cambridge University Press, Cambridge, 1988)
43. J.W. Atkinson, J.O. Raynor, Personality, Motivation, and Achievement (Hemisphere Publishing Corporation, London, 1978)
44. F. Mureddu, A. Paciaroni, N. Carvalho, E. Chaniotakis, B. Garcia, G. Hemming, A. Iordache, S. Katsanevas, G. Koekoek, C. Lintott, S. Ritz, S. Serjeant,
S. Sotiriou, F. Spagnuolo, D10.3 policy roadmap on research infrastructures for citizens science in Europe. Zenodo (2023). https://doi.org/10.5281/
zenodo.7753434
45. European Commission, Directorate-General for Research and Innovation: Strategic Research and Innovation Agenda (SRIA) of the European Open
Science Cloud (EOSC) (Publications Office of the European Union, Luxembourg, 2022). https://doi.org/10.2777/935288
46. S. Serjeant, Citizen science in the European open science cloud. Europhys. News 54(2), 20–23 (2023). https://doi.org/10.1051/epn/2023204. arXiv:
2307.06896 [astro-ph.IM]
47. A. Swanson, M. Kosmala, C. Lintott, R. Simpson, A. Smith, C. Packer, Snapshot serengeti, high-frequency annotated camera trap images of 40
mammalian species in an african savanna. Sci. Data 2(1), 150026 (2015). https://doi.org/10.1038/sdata.2015.26
123

Citizen Science, Accessibility, Art & Science, Critical Thinking, Policy and Engagement: Thoughts and Lessons Learned From The REINFORCE Experience

Uploaded by

Copyright:

Available Formats

Citizen Science, Accessibility, Art & Science, Critical Thinking, Policy and Engagement: Thoughts and Lessons Learned From The REINFORCE Experience

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Citizen Science, Accessibility, Art & Science, Critical Thinking, Policy and Engagement: Thoughts and Lessons Learned From The REINFORCE Experience

Uploaded by

Copyright:

Available Formats

Eur. Phys. J.

Plus (2024) 139:868

Citizen science, accessibility, art & science, critical thinking, policy

Received: 23 January 2024 / Accepted: 24 May 2024

a e-mail: gary.hemming@ego-gw.it (corresponding author)

0123456789().: V,-vol 123

2 Demonstrator projects on Zooniverse

Fig. 1 Identifying a glitch family in GWitchHunters

2.1.1 Sonification of gravitational-wave data

2.2 New Particle Search at CERN

Fig. 4 The first stage of New Particle Search at CERN on Zooniverse

Fig. 5 The second stage of new particle search at CERN on HYPATIA

2.2.1 Interaction with users, engagement and impacts

2.2.2 Outcomes and the future

2.3 Deep Sea Explorers

2.3.1 Successes: what worked and why

Fig. 6 Bioluminescence events

2.3.2 Areas that might be approached differently

2.3.3 Lessons learned

2.3.4 Interaction with the demonstrator project community

2.4 Cosmic Muon Images

2.4.1 Successes: what has worked and why

2.4.2 Improvements; what has not worked and why

2.4.3 Lessons learned

2.4.4 Interaction with the demonstrator project community

2.4.5 The future

3.1 The team working on sonification

3.2 User-centred design

3.3 sonoUno development

3.3.1 Desktop-based version

3.3.2 Web-based version

3.4 Sonifying the demonstrator-project data

Fig. 8 A Blip type glitch, with

Fig. 10 A data subject from the

3.5 Tactile models

3.7 What worked particularly well

3.7.1 Course on sonification

Fig. 11 A comparison of an event displayed in HYPATIA with its representation in sonoUno

3.8 Plans for the future

5.1 Impact of engagement activities and community building

Fig. 13 Details of the REINFORCE citizen-engagement strategy

Table 1 Metrics and results for Metric Value

5.2 Highlights of individual demonstrator-project characteristics

6 Art & Science

6.1 International Youth Art & Science competition

6.2 Art & Science events organised within REINFORCE

7.1 The course syllabus

Fig. 16 Senior Citizen Science

7.2 What needed to be worked on?

7.3 What worked well?

8 Critical and Scientific Thinking course

9 Raising awareness and sustainability

• Potential citizen scientists;

9.1 Social media

9.2 Webinar series

10.1 Overview of methodology

10.2 Data analysis and results

10.2.1 Participant satisfaction

10.2.2 Individual impact