Preprint: Please note that this article has not completed peer review.
Facilitating International Participation in Core
Outcome Set Development: A Proposed Approach for
Multi-language Delphi Surveys, a Review of Current
Practice and Issues to Consider
CURRENT STATUS:
UNDER REVIEW
Bilal Alkhaffaf
Salford Royal NHS Foundation Trust
bilal.alkhaffaf@srft.nhs.ukCorresponding Author
ORCiD: https://orcid.org/0000-0001-5751-1846
Paula Williamson
University of Liverpool
Jane M Blazeby
University of Bristol
Anne-Marie Glenny
University of Manchester
Ademola Adeyeye
University of Ilorin Teaching Hospital
Paulo Matos Costa
Hospital Garcia de Orta
Ismael Diez del Val
Hospital Universitario Basurto
Suzanne Gisbertz
Amsterdam UMC
Ali Guner
Karadeniz Teknik Universitesi
Simon Law
University of Hong Kong
1
Hyuk-Joon Lee
Seoul National University College of Medicine
Ziyu Li
Peking University Cancer Hospital and Institute
Koji Nakada
The Jikei University Daisan Hospital
Rafael Mauricio Restrepo Nunez
Hospital Rey Juan Carlos
Daniel Reim
TUM School of Medicine
John V Reynolds
Broomfield Hospital
Peter Vorwald
Hospital Universitario Fundacion Jimenez Diaz
Daniela Zanotti
University Hospitals Birmingham NHS Foundation Trust
William Allum
Royal Marsden NHS Foundation Trust
Asif Chaudry
Royal Marsden NHS Foundation Trust
Ewen Griffiths
University Hospitals Birmingham NHS Foundation Trust
Iain Bruce
Manchester University NHS Foundation Trust
DOI:
10.21203/rs.3.rs-16047/v1
SUBJECT AREAS
Health Economics & Outcomes Research
KEYWORDS
Stomach Neoplasms, Surgical Oncology, Patient Outcome Assessment, Consensus,
Outcome Assessment (Health Care), Core Outcome Set
2
Abstract
Background Core outcome sets (COS) should be relevant to key stakeholders and widely applicable so
that researchers are willing to use them when designing trials. Many COS developers have utilised
online Delphi surveys which enable wider international stakeholder participation. Despite broader
participation, most international COS projects have restricted Delphi surveys to a single language
potentially excluding important opinion from those who are not fluent in the survey language.
Methods A structured review of current approaches to translating Delphi surveys for COS
development was undertaken. We present a proposed methodology adapted from international
guidance. Results Four studies were identified from our structured review. Wide variation exists in the
methodological approaches to translating Delphi surveys for the developing of COS. Issues which
arise when developing multi-language Delphi surveys include establishing translation groups,
timelines, financial implications, strategies to maximise recruitment and regulatory approvals.
Conclusion Consideration of the issues described will improve planning by other COS developers and
can be used to widen international participation from both patients and healthcare professionals.
Main Article
Background
Core outcome sets (COS) aim to standardise the reporting of critically important outcomes within a
given research field1. COS should be relevant to key stakeholders and widely applicable such that
researchers are encouraged and willing to incorporate them in trials. Approaches to improve the
relevance of COS can take many forms, including involving stakeholders with lived experience of the
condition or intervention in question. Many COS developers are using online Delphi surveys during
stages to prioritise potentially important outcomes2. This approach enables overseas stakeholders to
participate more readily. Such broad participation can give COS greater validity across different
geographical regions and consequently make them more likely to be used in future trials regardless of
the location where trials are undertaken. Unless COS are widely used in trials within the same
research field, the challenge of inconsistent outcome reporting will persist3.
Most research groups developing ‘international’ Delphi surveys have restricted themselves to their
3
native language (usually, but not exclusively, English). This approach is less resource intensive than
translating the survey into multiple languages and overcomes issues with ambiguity or changes in
meaning - a recognized challenge with translation4. However, these methodological challenges are
not insurmountable and some COS developers are translating Delphi surveys to minimise the risk of
excluding important opinion from those not fluent in the study’s primary language.
Core outcome sets (COS) aim to standardise the reporting of critically important outcomes within a
given research field1. COS should be relevant to key stakeholders and widely applicable such that
researchers are encouraged and willing to incorporate them in trials. Approaches to improve the
relevance of COS can take many forms, including involving stakeholders with lived experience of the
condition or intervention in question. Many COS developers are using online Delphi surveys during
stages to prioritise potentially important outcomes2. This approach enables overseas stakeholders to
participate more readily. Such broad participation can give COS greater validity across different
geographical regions and consequently make them more likely to be used in future trials regardless of
the location where trials are undertaken. Unless COS are widely used in trials within the same
research field, the challenge of inconsistent outcome reporting will persist3.
Most research groups developing ‘international’ Delphi surveys have restricted themselves to their
native language (usually, but not exclusively, English). This approach is less resource intensive than
translating the survey into multiple languages and overcomes issues with ambiguity or changes in
meaning - a recognized challenge with translation4. However, these methodological challenges are
not insurmountable and some COS developers are translating Delphi surveys to minimise the risk of
excluding important opinion from those not fluent in the study’s primary language.
Aims and Objectives
Currently, there is no standardized method of translating Delphi surveys for use in the development
of international COS. This paper aims to address this need by presenting an approach to develop a
multi-language Delphi-survey adapted from international consensus guidelines on the translation and
cultural adaptation of patient-reported outcomes4. The survey forms an integral part of the GASTROS
study (GAstric Cancer Surgery TRials Reported Outcome Standardisation) which aims to develop an
4
international COS for surgical trials in gastric cancer5. The scope and design of the GASTROS study
has been previously detailed5. In summary, following a systematic review of randomized control
trials3 and a series of in-depth patient interviews6, a long-list of potentially important outcomes was
rationalized into a list of 56 outcomes. Following a consultative exercise with key stakeholders (see
below), these 56 outcomes were presented to patients and healthcare professionals in a two-round,
multi-language Delphi survey.
The objectives of this paper include:
1.
To describe the current methodological approaches used by COS developers in the
translation of Delphi surveys,
2.
To outline a pragmatic, robust and replicable approach to translating Delphi surveys
for use in COS development, and
3.
To outline important logistical considerations in preparation for an international
Delphi survey. These considerations are relevant to COS researchers seeking to
translate their surveys or undertake the Delphi process in a single language but in
multiple regions.
1. Current Approaches to translating Delphi Surveys
To gain an understanding of current translation approaches for multi-language Delphi-surveys, a
structured literature review of the COMET database was undertaken7. The COMET database is a
comprehensive registry which (as of 03/09/2019) contains 337 published and 280 ongoing COS
respectively dating back from 1981. The database is kept up-to-date through annual systematic
reviews of scientific databases (using MEDLINE via OVID and SCOPUS), automated alerts from
MEDLINE via OVID, SCOPUS and Google Scholar and direct submissions from COS developers8.
Search Strategy and Inclusion Criteria
The COMET database enables users to search for terms within the ’title’, ’abstract’ or ’author names’
categories. Searches can be restricted according to health area, target population, methods,
5
stakeholder involvement, study type and publication year. A broad search for the terms
‘international’, ‘language’ or ‘translat$’ in the title and abstract was undertaken with no other
restrictions. A flow chart illustrating how studies were included is summarized in figure 1.
Studies included in our review were those that used a multi-language Delphi survey in the
development of their respective COS. Only publications from completed studies were included - COS
methodology is a relatively new research field and so planned approaches may not accurately reflect
the final methodology used. The COMET database may contain several different references to COS
development for the same project. Any related publications were consolidated and handled as a
single COS study.
Examining Methodological Approaches to Translation
Four studies (summarized in table 1) were deemed eligible for inclusion in the structured review.
Corresponding authors were contacted and asked to participate in a questionnaire examining various
aspects of their respective methodological approaches (appendix 1). Our approach to translation was
based on international consensus guidance on translation and cultural adaptation of patient reported
outcome measures4, and so questions were developed to examine aspects of methodology presented
using this guidance. The questionnaire focused particularly on how items presented in the Delphi
surveys were translated and how discrepancies and conflict was resolved. Responses were received
from the corresponding authors of all four studies identified. Responses were combined with data
from the respective publications and are presented below.
General Approaches to Translation
All groups summarized their approach to translating surveys with one providing a reference to their
methodology and another referring to methodology described by the OMERACT group at a COS
development meeting
9,10.
In addition to forward translations, three groups undertook a backward
translation of the survey from the target to source language. The number of forward and backward
translations differed in each study. Two studies undertook a single forward translation whilst the
others undertook two and three. One group used no backward translations, one study undertook a
6
single backward translation whilst the other two undertook two backward translations. The
characteristics of those involved in the translation processes also differed amongst the groups (figure
2); no professional services were employed, and all translations were undertaken by healthcare
professionals or lay translators.
Discrepancies and Harmonization
All four groups described an approach to managing discrepancies in translations. Two groups reported
that discrepancies were discussed within the ‘research group’ until consensus was reached, whilst the
remaining two referred to individuals outside of the ‘research group’ who were fluent in the target
language to resolve any language issues.
Cognitive Debriefing
Three groups described undertaking an exercise to test alternative wording and check
understandability, interpretation and cultural relevance of the proposed Delphi survey in the target
language. Using cognitive interviews, they studied patients/relatives and health professionals’
interpretation of the translations to examine face validity. Two of these involved patients and/or their
relatives whilst the third was based on the opinion of healthcare professionals alone.
2. Proposed approach to translating Delphi surveys
As previously discussed, one of the principal aims of translating the Delphi survey in the development
of COS is to include the opinions of stakeholders who are not fluent in the source language. With
respect to the GASTROS study, this was especially important given that the highest incidence of
gastric cancer exists outside of English-speaking countries, in the Far East, Central and South America
and Southern Europe.
In developing our approach to translate the survey, the study management group was keen to ensure
that it was both methodologically sound yet pragmatic such that it could be easily reproduced by
multiple international collaborators within a relatively short period of time.
In 1999, the ISPOR-TCA group (The Professional Society for Health Economics and Outcomes Research
– Translation and Cultural Adaptation group) was formed to discuss and develop guidelines for
translating patient-reported outcome measures. The group highlighted inconsistencies with previous
7
methodologies and nomenclature in this field and sought to address these by developing guidance
setting out ‘principles of good practice’4. These principles were derived from 12 major sets of
translation guidelines from the following groups:
American Association of Orthopaedic Surgeons (AAOS)
Association of Test Publishers
EORTC group
Euro QoL group
Evidence: Clinical and Pharmaceutical Research
FACIT group
Health Outcomes group (HOG)
Health Utilities Inc. (HUInc)
International Quality of Life Assessment (IQOLA) group
Kidney Disease Quality of Life (KDQOL)
Medical Outcomes Trust (MOT)
World Health Organization
Other consensus guidelines have been developed for translating surveys. The Survey Research Centre
(SRC) guidelines provide broader consideration of the translation process and describe practical
support from expert contributors’ experience of different survey types9. There is much cross-over
between the two SRC guidelines. Given the focus of our work was primarily outcome related
translation, the principles as set out by the ISPOR-TCA group formed the basis of our methodology,
with references made to the SRC guidance and some pragmatic amendments which are explained in
further detail below.
Adopted Translation Approach
Below, we describe ten steps involved in translating the Delphi survey used in the development of a
COS for surgical trials in gastric cancer. The full rationale for each step, and the risks of omitting
them, is described in detail in the ISPOR-TCA guidance; we have stated the rationale for the steps
below (particularly in relation to pragmatic deviations) where we believed it was necessary to do so.
The source survey was developed in English and translated into seven target languages (Simplified
Chinese, Dutch, German, Italian, European Portuguese, European Spanish and Turkish). Appendix 2
details the instructions which were provided to each international collaborator responsible for leading
the translation process in their respective country. These outlined which files required translation,
how the translation should be undertaken, and by whom.
8
By using this approach in addition to considering other important issues (section 3), the GASTROS
study was able to recruit 1021 participants (463 surgeons, 313 patients, 245 nurses) in the first round
of the Delphi survey, with 349 participants using the English language version and 672 using one of
the seven other language versions.
1. Preparation
a. Cognitive Debriefing:
i. Cognitive debriefing aims to identify issues with comprehensibility of key concepts and
understanding amongst potential participants. As previously stated, we presented survey
participants with 56 outcomes which had been rationalized following a process that had
identified a long-list of potentially important outcomes from a systematic review and indepth patient interviews. This rationalization process from the long-list to the 56 survey
items involved key-stakeholders (members of the GASTROS study group, surgeons,
oncology nurses and patients) who also ensured that the outcomes were accompanied
with plain English-language explanations that could be understood by all participants
including patients. A further consultative exercise with an English-speaking patient-group
was held to ensure that the meaning of each outcome, in addition to other survey-related
files were clearly understood. Undertaking this work prior to translation was essential as it
minimized the possibility of ambiguous meanings which could result in a mistranslation.
b. Preparing documents for translation
i. Four documents were needed to run the Delphi survey; a participant information sheet
and three further files which were required to set up the web-based survey. We used
DelphiManager 3.0 platform, developed and maintained by the COMET Initiative
(http://www.comet-initiative.org/), to undertake the Delphi (section 3). The three files
included:
1. File 1 (appendix 3): An excel file containing details of each outcome,
accompanying meaning and the ‘outcome area’ under which the outcome was
categorised15.
9
2. File 2 (appendix 4): User-defined text: A file containing text specific to our
surveys (in this case the GASTROS Delphi survey).
3. File 3 (appendix 5): Static text: A file containing text common to all Delphi
surveys which was used in the setting up process by the DelphiManager team.
ii. Preparation for Round 2 of the Delphi survey: Additional translations were required to
support the second round of the survey. These included:
1. Outcomes identified by participants in round 1 as being important to consider
that were not identified from the systematic review or patient interviews.
2. Legends and terms required to produce charts which were presented to survey
participants in round 2.
3. Comments and feedback from study participants.
iii. Following Round 2 of the survey:
1. Participants who changed their scores between rounds were given the
opportunity to provide their reasons for doing so.
2. Participants were also given the opportunity to provide further comments after
completing the survey.
c. Understanding which methodological approaches to employ
Two approaches to translation were adopted:
1. “Two forward, one back translation”; this approach was the most comprehensive
and labour intensive as it required a further nine steps (below) before a final file
version was agreed. Following discussion amongst the study management team, it
was deemed content which could alter the meaning of the outcomes being
presented and ultimately influence how the overall aims of the survey was received
and understood by participants (file 1, file 2, and additional outcomes identified by
participants in round 1) underwent this approach. The steps involved in this process
are described in greater detail in points 2 to 10.
2. “One forward, dual independent proofreading”; File 3 consisted primarily of short
10
instructional phrases (e.g. ‘click here’, ‘register’, and ‘next page’) which were
necessary for the functionality of the survey. As these terms would not materially
influence the comprehension of the survey’s purpose or outcomes presented within
it, a simplified, less resource intensive approach was adopted. This file underwent a
single forward translation followed by two independent proof-readings by translators
who compared the translated and source files for accuracy and quality. Any
corrections or amendments were undertaken through discussion between the
translator and proof-readers. This approach was also adopted for the translation of
participant comments, feedback and reasons for changing scores between round 1
and round 2.
d. Setting up translation teams
To support the translation work, an international working group (IWG) was established
(see section 3). Each collaborator within the IWG was responsible for overseeing a team
which would undertake the translation and ensuring that the key concepts of the study
were appropriately communicated. The translation process was supported by the
GASTROS study Chief Investigator (BA) if any clarifications were required. The
characteristics of individuals involved in this process are described in greater detail in
Appendix 2. In summary, each team was made up of an IWG lead, two forward translators
and a single backward translator.
e. Developing instructions for translations
Setting out the methodology a priori in a clear and structured document ensured that
collaborators and their teams understood what would be required of them at each stage
of the translation. These instructions included ongoing responsibilities prior to and
following future rounds of the Delphi survey. This was essential given that one of our
primary aims was to ensure that our approach was easily replicable. Figure 3 is a flow
diagram which details these stages and the order in which they were to be undertaken.
Feedback from the IWG was positive in response to these instructions with collaborators
11
reporting that the document enabled them to undertake the translation process
efficiently.
f. Quality assurance
IWG collaborators were asked to provide documented evidence for each step of the
translation process. These could then be reviewed by the study management team as
required.
2. Forward Translation
Two independent forward translations by individuals who were native speakers of the target
language were undertaken. Culture is a primary determinant of language and therefore native
speakers have advantages with language abilities compared to second-language speakers.
Having two independent forward translations enables detection of errors and divergent
interpretations that could otherwise lead to bias.
3. Reconciliation
There are several approaches which can be used to reconcile the forward translations. We opted
to use the ‘in-country’ IWG collaborator who was also involved in cognitive debriefing and
piloting of the survey as this was pragmatic and would not require the identification of further
individuals to undertake this step. No issues arose from the reconciliation process, however, had
further clarifications been required, they would have been directed to the CI.
4. Back Translation
The ISPOR-TCA guidance states that ‘back translation’ is necessary, whilst the Survey Research
Centre guidance suggests that this is not required. We opted to undertake a single back
translation to provide quality-control of the forward translations. Whilst the ISPOR-TCA guidance
suggests that this should be undertaken by individuals who are native speakers of the source
language (i.e. English), we found it challenging to identify seven native English-speakers who
were also fluent in the required target languages and had an understanding of outcome
reporting without referring to a professional service. We opted to ensure that back-translators
were fluent in English and independent from the forward translators.
12
5. Back Translation Review
This step is important as it ensures that the cross-cultural adaptation needs of the translation is
met. Without it, there is a risk of that a mistranslation or omission would remain in the
translation. This was undertaken by the CI in combination with the IWG collaborator by
comparing the back translation to the source document. No significant discrepancies between
the source and back-translated files were identified across any of the translations.
6. Harmonization Across Different Languages
There is no agreed method to how harmonization across different translations should be
enforced; many approaches omit this step. However, our group opted to ensure harmonization
between each language at each step of the process. This was undertaken by the CI. We did not
encounter significant differences between translations. An example of a minor change that was
made across surveys was the term ‘last round scores’ which in the context of the survey meant
‘previous round scores’. Some teams translated this as ‘the final round scores’ which had to be
altered to ensure all versions contained the same meaning.
7. Cognitive Debriefing of the Translation
Following harmonization across translations, all survey versions were built using the
DelphiManager platform (section 3). A further cognitive debriefing exercise was undertaken by
asking IWG collaborators and their translation teams to complete a pilot version of the survey to
identify grammatical or stylistic errors and check understandability, interpretation and cultural
relevance of instructions and outcomes within the survey.
8. Review of Cognitive Debriefing Results and Finalization
There were no issues highlighted with comprehensibility or understanding. Spelling mistakes
and minor grammatical errors (e.g. pronouns ‘you’ formal and informal) were altered.
9. Proofreading
IWG collaborators were once again asked to examine the survey and ensure that any issues
highlighted in the previous steps had been addressed. No further changes were identified in any
of the language versions by this stage.
13
10. Final Report and ‘Start of Survey’
The ISPOR-TCA group guidance recommends that a report should be produced detailing the
methodological approach for translation and rationale for each step. The final report for
translations undertaken for the GASTROS study is represented by this paper.
3. Important Considerations
In this section, important considerations are presented to support the translation process and
maximise participation from stakeholder groups in different countries. Many of these considerations
can be applied by COS developers if they choose not to translate their Delphi survey, but are keen to
widen participation from those who speak the primary language. As in section 2, we describe the
rationale for each consideration and the potential risks of not applying these steps (where applicable).
International Working Group
The GASTROS study is a collaborative international initiative which sought to attract global
representation within the study group. Motivated, research-active collaborators from countries with a
significant incidence of gastric cancer were approached to form an IWG. Individuals signed a ‘terms of
reference’ document which outlined the benefits of their involvement in addition to the following
responsibilities:
To form a local team and oversee the translation of the Delphi survey (where applicable)
To drive recruitment locally, regionally, nationally and internationally through organisations and
personal networks
To garner and develop links specifically with patient groups who would be able to participate in
advertising the Delphi survey
To identify the need and apply for relevant local ethical and regulatory approvals
The IWG was made up of collaborators from the following countries:
Brazil
Mainland China and Hong Kong
Germany
Ireland
Italy
Japan
The Netherlands
Nigeria
Portugal
South Korea
Spain
14
Turkey
United Kingdom
Ensuring the IWG was set up early maximized our ability to develop translations in a timely manner
and recruit evenly across all stakeholder groups from a broad range of countries.
Patient and Public involvement
A Study Advisory Group (SAG) separate to the IWG forms part of the management structure of the
wider GASTROS study. The SAG is made up of key stakeholder representatives including patients. The
group provides advice on the methodology of the study, general delivery of the study against its
stated objectives and ensures that the viewpoints of all stakeholder groups are considered. In
addition, patient groups (see acknowledgements) were vitally important in reviewing and piloting the
translated surveys prior to recruitment to the Delphi. These groups were also instrumental in
recruiting patients (see below).
Who should undertake the translation work?
The GASTROS study management group opted to set up local translation teams made up of
healthcare professionals who met the rigorous criteria as set out by the ISPOR-TCA group. An
alternative approach would have been to employ a professional translation service to undertake this
work. One of the benefits of professional services is the ability to complete the translations in a
relatively short period of time, in addition to developing an unlimited number of translations which
may have resulted in wider participation in the Delphi survey. The main disadvantage to this
approach is cost. Quotes from three different professional translation services (all familiar with the
ISPOR TCA guidance) were requested to support rounds 1 and 2 of the survey. In April 2018, the
estimated costs were in the region of 3200GBP-4000GBP per language. All translations for rounds 1
and 2 of the survey would be finalized within 5 and 2 weeks respectively. Due to the financial
limitations of undertaking the survey in 7 languages, we did not pursue this option.
Milestone & timeline planning
Setting aside enough time for the translation process is of paramount importance, particularly if COS
developers are seeking to translate their surveys into more than one language. Some of the
translation steps required all language versions to have reached the same stage prior to moving onto
15
the next stage. For example, all initial translations had to have been completed before harmonization
across surveys could be achieved. Without this, we were unable to ask collaborators and their teams
to pilot their respective surveys. Furthermore, we chose to open recruitment to all language versions
simultaneously and so all translations needed to have been fully completed before participants could
complete their surveys. This was also the case for the second survey round. The impact of ethical
approval applications on timelines is discussed below in greater detail. Table 2 details timelines
involved in producing all versions of the survey for both rounds. The time to return the initial
translation documents and obtain ethical approvals resulted in the greatest variations with respect to
the overall timelines. We found that setting regular milestones and realistic timelines helped achieve
the required translation objectives. Regular communication between the CI and collaborators
underpinned this process.
Our aspiration was to translate the Delphi survey into Japanese and Korean to enable wider patient
participation from these countries. Due to challenges in identifying collaborators at an early stage,
assembling a translation team and meeting timelines, this could not be pursued. However, potential
participants were invited to complete the English language version of the survey.
Recruitment and retention targets
COS developers should consider minimum recruitment targets. Whilst there is no sample size
requirement for Delphi surveys, the GASTROS protocol initially set a conservative target of 100
participants in total to be recruited over a period of 6 to 8 weeks in round 1. However, as interest in
the study and international collaboration grew, it was clear to see that this target would easily be
surpassed. As described below, once the survey opened and momentum began to gather, we
witnessed a ‘snowballing’ effect amongst all three stakeholder groups. We therefore extended
recruitment to 13 weeks by which time new participation had plateaued (figure 4).
For round 2, an initial retention target of 80% was set following discussions with members of our
study management group who have extensive knowledge and experience of COS development.
Automated reminder emails were sent out on a weekly basis to participants and support from
professional bodies (in countries where round 2 responses were slow) was sought to encourage
16
completion of the survey. Personalised e-mails from the CI to professionals were also sent. Using this
strategy, we were able to retain 65% of participants from round 1 by week 13, by which time no
further responses were being received.
Paper and Internet-based Delphi survey versions
The GASTROS study used both internet-based and paper versions of the Delphi survey. The internetbased versions enabled us to reach participants in nearly 60 countries, the vast majority of which did
not have formal IWG collaborators. The paper-versions (printed versions of the internet-based survey
which were uploaded electronically by local collaboration teams) also enabled us to recruit
participants (particularly patients) who either did not readily have access to the internet or were not
‘internet-literate’ (e.g. China and Hong Kong, Nigeria and Turkey).
Several platforms exist to enable COS developers to run Delphi-surveys. These include platforms
specifically designed for Delphi surveys and other generic survey platforms which researchers can
use. When considering multi-language surveys, it is essential to ensure that the servers on which the
surveys are hosted meet the necessary data protection regulations and are accessible particularly
from countries where restrictions to certain ‘out-of-country’ domains exist. Furthermore, COS
developers must ensure that the platforms used are able to run surveys using different language
scripts and writing systems.
Our group used DelphiManager as it fulfils the required data protection criteria (as set out by our
United Kingdom ethical approval) and can work with all language systems including English, Chinese,
Japanese and Korean. Furthermore, the online survey domains are accessible from countries which
commonly restrict access to other foreign domains. DelphiManager has additional features which
simplified recruitment and completion of the surveys such as being able to send automated reminders
to individuals who had yet to complete all their answers.
Measures to maximize survey recruitment
One of the strengths of our Delphi survey was that it was able to recruit over 1000 patients and
healthcare professionals from nearly 60 countries in round one. From the study’s inception, the study
team recognised the importance of developing a clear networking and dissemination strategy5. We
17
hypothesised that this was necessary to achieve broad stakeholder participation both nationally and
internationally. Several strategies were employed to maximise recruitment:
1.
Dissemination of results from previous study stages
The study protocol and findings from previous study stages3,5,6 were presented at
targeted national and international meetings which were well-attended by potential
healthcare participants. This was integral to generating interest and support for our study
and ensured that participants understood the premise for GASTROS long before the Delphi
survey opened for recruitment. All presentations contained directions to the study website
and social media accounts (below).
2.
Local recruitment of patients through outpatient clinics
Ethical approval enabled the study team to recruit patients directly from outpatient
clinics. Our experience from the United Kingdom is that many patients regularly attend
patient support groups and are in contact with other eligible patients. As a result, a
snowballing effect resulted in patients being recruited by patients already within the
study.
3.
Support from stakeholder groups/associations and national research networks
Support from national and international professional associations and organisations (see
acknowledgements) was sought in the early stages of the study. Study group members
presented the study objectives at closed executive level meetings to gain support and
adoption from influential bodies including professional associations patient groups and
charities. Many of these organisations have large memberships (and corresponding
electronic mailing lists) through which the study was advertised. Most of the groups
through whom we sent out invitations followed up an initial e-mail with a further reminder
approximately 4 weeks later resulting in further recruitment spikes. Furthermore, the
GASTROS study was adopted onto the National Institute for Health Research (NIHR)
Portfolio (CPMS study ID 38318). This enabled us to advertise the study to healthcare
18
professionals and patient support groups within the United Kingdom through the national
Clinical Research Networks. Our experience suggests that recognition by respected
associations and groups results in a ‘snow-balling’ effect with subsequent support from
others becoming easier to harness.
4.
Collaborations
Standardising the reporting of outcomes can be achieved through several approaches.
The GASTROS study aims to identify critically important outcomes across the entire
spectrum of outcome types. Others have concentrated on the reporting of outcomes
within a defined area. For example, the GASTRODATA group (www.gastrodata.org) have
sought to standardise the reporting of all major post-gastrectomy complications16. Whilst
the goals of both studies are different, both teams have been able to work closely to
minimise duplication of work. In addition, the GASTRODATA group was able to promote
recruitment to the GASTROS Delphi survey through its membership and respective
networks. Such collaborations will also be vital for the future development of outcomes
research within the field of gastric cancer surgery.
5.
Personalised emails
a.
Most of the study management group, study advisory group and international
working group members have extensive research experience within the field of
gastric cancer surgery. Each member was asked to promote the study through
their personal research and clinical networks. Bulk e-mails through professional
bodies may be ignored by potential participants or diverted into ‘spam’ e-mail
folders, hence why this approach was employed.
b.
Corresponding email addresses for authors from previous trials and protocols
included in our systematic review3 were identified and personal invitations sent.
This captured research-active healthcare professionals from non-English
19
speaking regions where no formal national gastric cancer associations exist
(e.g. Eastern Europe).
6.
Social media and multimedia
a.
The study website (http://www.gastrosstudy.org) provides detailed information
about the GASTROS study aims as well as all its outputs. Prior to the
commencement of the Delphi survey, potential participants who had heard
about the study were able to register their interest to participate. In the
preceding 18 months before the survey opened, 150 healthcare professionals
and patients had registered.
b.
In addition to the study’s twitter account (https://twitter.com/GASTROSStudy),
members of the research team posted updates on their personal Twitter and
LinkedIn accounts. Regular study updates provided potential participants with
an opportunity to better understand the study aims and keep up to date with its
progress. Examination of analytics revealed that Twitter and LinkedIn posts in
the run-up to and during round 1 of the survey regularly received over 4000 and
3000 views respectively.
c.
A series of short videos were produced for the study. These provided potential
participants with an alternative way to engage with the study. At the time of
writing, these videos had been viewed over 600 times. In addition to an
introductory video on the study, a detailed step-by-step guide to completing the
online Delphi survey was developed. This created additional content for social
media platforms and the GASTROS website which in turn enabled the study to
maintain a regular online presence. COS developers may wish to produce
different language versions or translate video captions relatively easily to
expand their reach. Additional COS-related material is already available from
20
the COMET initiative YouTube site17 with versions available in Dutch,
Portuguese and Chinese. Work is underway to develop other language versions
as well.
Whilst advertising the study through these avenues aims to increase the number of recruits, care
must also be taken that potential participants are not ‘bombarded’ with requests to participate in the
survey. A small number of healthcare professionals highlighted that this was an issue. This coupled
with the well-recognised challenges of ‘survey-fatigue’, may in fact be counter-productive and result
in apathy amongst potential participants.
Ethical Approval
The requirement for regulatory or ethical approval varied across different regions. In the United
Kingdom, the approach to ethical approval has not been consistent; our group was asked to submit a
full application for ethical approval committee consideration, whilst other groups have been able to
gain approval through proportionate review18. Each IWG collaborator was responsible for
understanding local requirements and applying for ethics if it was required. They were asked to
enquire about these at the start of their agreement to participate in the study and applications were
made in parallel to the translation work. Two of our international collaborating centres did not require
ethical approval as local collaborators did not recruit patients directly from their clinical practice but
instead advertised the study through local patient groups and recruited healthcare professionals by
advertising through national Societies and networks. The time taken to complete this process varied
significantly (table 2) and was largely dependent on the frequency of and access to ethics committee
meetings, requirements to amend submitted materials and delays in final decisions reaching the
collaborators. COS developers should investigate the need for ethical approval as early as possible to
avoid unnecessary delays.
Financial Planning
Several aspects of undertaking multi-language Delphi surveys may potentially incur significant costs
depending on which approaches are adopted. COS developers should take these into account when
21
planning their studies. These include:
1.
Cost of professional translations. This represents the largest financial burden and has
been discussed above.
2.
Ethical approval. Some of our non-UK ethical approval applications required
payments of up to €250 Euros.
3.
Use of electronic mailing lists. Some stakeholder groups may charge administration
fees to send out invitations to their membership.
4.
Cost of Delphi survey platform. Whilst open-access platforms exist, our group opted
to pay to use a dedicated Delphi survey platform designed for the development of
COS.
5.
Statistical and qualitative methods support may be required when analysing scores in
rounds one and two, depending on the nature of feedback to be given.
Discussion
This paper is the first to address the topic of translation and cross-cultural adaptation in the context
of developing Delphi surveys for COS. We have presented a robust 10-step approach adapted from
international consensus guidelines which was easily and accurately replicated by seven different
translation teams within an acceptable timeframe.
Undertaking ‘international’ Delphi surveys has become easier as web-based platforms enable wider
participation across different geographical regions. Whether or not this is warranted depends on the
scope of the COS being developed and who is being targeted to use the COS. Very few pathologies or
interventions are limited to one geographical region and consequently trials in their respective fields
are undertaken all over the world, often collaboratively across different regions. In order to
standardise the reporting of critically important outcomes, it becomes necessary that a COS is
adopted by all trialists within the given research field. For this to be achieved, trialists need to be
confident that the COS that has been developed is robust and has considered the views of relevant
22
stakeholders from a wide range of backgrounds. In our case, it was essential that stakeholders from
the Far East as well as Asia, South America and Europe participated as this is where most gastric
cancer occurs and where most of the trials are undertaken.
Whilst translation of our Delphi survey aimed to widen participation and broaden the views taken into
consideration, the value of doing so is one which warrants further discussion. English is the most
commonly spoken language across the world19 and is used by most scientific and healthcare
publications. It is therefore unsurprising that most ‘international’ COS projects employing Delphi
surveys used an English-only version. The English version of our Delphi survey was offered to all
participants, however most preferred to complete the Delphi in their native language. Whilst for many
this choice would have been because they did not speak English, a significant proportion of bi-lingual
healthcare professionals known to the study team preferred to use a non-English version. One may
argue that this enabled participants to engage more confidently in the process and their
understanding of what was being asked of them and the quality of their responses was consequently
better. Such a high uptake in non-English surveys was not experienced by the four groups who
completed our questionnaire on methodology. However, this likely reflects challenges related to
recruitment in non-English speaking regions which we address in our ‘issues to consider’ section.
Undoubtedly, achieving high quality and accurate translations is resource intensive. They can take
time if undertaken by healthcare professionals or pose significant financial costs if study groups
employ professional services. However, restricting a consensus-seeking process in the development
of an international COS to a single language exposes studies to the risk of excluding important
opinion from those not fluent in this language. It may therefore be argued that for a COS to be truly
regarded as ‘international’, the consensus-seeking process should be undertaken in the native
language of the participant. Whilst we have demonstrated that non-English Delphi surveys can triple
the number of total participants, it is not known whether these additional participants bring a
different perspective that has not already been captured through the English-language version. It is
likely that there are many additional factors which may contribute to the validity of Delphi survey
results (e.g. cultural and geographical differences of participants, regional economic status and
23
quality of healthcare provision) and COS developers should consider these carefully during the
planning phase. These, along with other factors will be the focus of a future analysis by our research
group.
We have demonstrated that there is no standardised approach to translation in this field. So far, each
of the four COS groups in section 1 of this paper used different methods to forward translate and
utilised translation teams with different member characteristics. Whilst back translation was not
undertaken in a uniform manner, the issue of whether it is required is one on which there is
disagreement. Considering this, and the fact that, in our study, back translation did not identify
significant discrepancies, COS developers may be justified in omitting steps 4 and 5 of our approach.
This should however be done after careful consideration as the importance of back translation may
depend on the type of outcomes that are being translated. It is possible that certain outcomes are
conceptually alien between cultures or geographical regions and undergoing an added step to reduce
the risk of mistranslation is warranted. In the field of patient-reported outcome measurement (PROM),
it is common for questionnaires to undergo translations (for use in international trials). The methods
required for PROM translation is rigorous and includes back translation20. Whilst it may be argued that
less rigorous methods could be used in Delphi surveys for COS, to ensure optimal face validity of
items the same standards are recommended. In any case, we did not find that back translation
extended our timelines significantly, however, additional steps in translation may result in an added
financial burden should COS developers use professional translation services.
Conclusion
We present a robust method of translating Delphi surveys used in the development of COS adapted
from international consensus guidelines in the field of outcome reporting. Consideration of the issues
described will improve planning by other COS developers and can be used to widen international
participation from both patients and healthcare professionals.
Abbreviations
COS
Core outcome set
GASTROS
GAstric Cancer Surgery TRials Reported Outcome Standardisation
24
IWG
International working group
SAG
Study advisory group
ISPOR-TCA
The Professional Society for Health Economics and Outcomes Research – Translation
and Cultural Adaptation group
OMERACT
Outcome Measures in Rheumatology
Declarations
Ethical Approval and Consent to Participate
This study describes the methodology used to undertake a Delphi survey. The study was given ethical
approval by the North West - Greater Manchester East Research Ethics Committee (18/NW/0347) and
governance approvals by Manchester University Hospitals NHS Foundation Trust.
Consent for Publication
Not applicable
Availability of Data and Material
The datasets analysed during the current study available from the corresponding author on
reasonable request.
Competing Interests
The authors report no conflicts of interest.
Funding
This study is funded by the National Institute for Health Research (NIHR) Doctoral Research Fellowship
Grant (DRF-2015-08-023). JMB is partially funded by the NIHR Bristol Biomedical Research Centre and
the MRC ConDUCT-II Hub for Trials Methodology Research. PRW was funded by the MRC North West
Hub for Trials Methodology Research (Grant ref: MR/K025635/01).
This paper presents independent research funded by the National Institute for Health Research
(NIHR). The views expressed are those of the author(s) and not necessarily those of the NHS, the
NIHR or the Department of Health.
Reprints
Reprints will not be available from the authors.
25
Acknowledgements
The authors would like to highlight the role undertaken by Dr Aleksandra Metryka, Senior Clinical
Trials Coordinator, who facilitated the running of this study. In addition, the authors would like to
thank the following associations and groups for their support in facilitating recruitment to the
GASTROS study Delphi survey:
The International Gastric Cancer Association (www.igca.info)
The Association of Upper Gastro-Intestinal Surgeons of Great Britain and Ireland (www.augis.org)
The Brazilian Gastric Cancer Association (abcg.org.br)
The Canadian Gastric Cancer Association (www.gastriccancer.ca)
The Chinese Gastric Cancer Association
The Dutch Upper GI Cancer Group (ducg.nl)
The GASTRODATA group (www.gastrodata.org)
Italian Research Group for Gastric Cancer (gircg.it)
The Korean Gastric Cancer Association (kgca-i.or.kr)
Oesophago-Gastric Surgery Section of the Asociación Española de Cirujanos – Spain (aecirujanos.es)
Upper GI International Robotic Association (www.ugira.org)
United Kingdom Oncology Nursing Society (www.ukons.org.uk)
The European Oncology Nursing Society (cancernurse.eu)
The Oesophageal Patients Association – United Kingdom (www.opa.org.uk)
My Gut Feeling – Canada (mygutfeeling.ca)
No Stomach for Cancer – USA (nostomachforcancer.org)
Vivere Senza Stomaco - Italy (viveresenzastomaco.org)
Gastro/Oesophageal Support and Help Cancer Group (Bristol) – United Kingdom
GASTROS International Working Group Collaborators
(To be cited as collaborators in PUBMED as previously agreed by the study management
team)
Shuangxi Li
Yu-long He
Zekuan Xu
Yingwei Xue
Han Liang
Guoxin Li
Enhao Zhao
Philipp Neumann
Linda O'Neill
Emer Guinan
Gian Luca Baiocchi
Giovanni de Manzoni
Eliza R.C. Hagens
Mark I. van Berge Henegouwen
Patrícia Lages
26
Susana Onofre
Gabriel Salcedo Cabañas
Maria Posada Gonzalez
Cristina Marin Campos
Bahar Candas
Bahadır Emre Baki
Muhammed Selim Bodur
Reyyan Yildirim
Arif Burak Cekic
Jean-Baptiste Beuscart
Sophie Horbach
Christopher Mecoli
Toby O Smith
Author Contributions
Author contributions to the study - https://www.casrai.org/credit.html.
#
Role
Definition
1
Conceptualization
Ideas; formulation or evolution of overarching research goals and aims.
2
Data curation
Management activities to annotate (produce metadata), scrub data and maintain re
data (including software code, where it is necessary for interpreting the data itself)
initial use and later re-use.
3
Formal analysis
Application of statistical, mathematical, computational, or other formal techniques
analyse or synthesize study data.
4
Funding acquisition
Acquisition of the financial support for the project leading to this publication.
5
Investigation
Conducting a research and investigation process, specifically performing the exper
or data/evidence collection.
6
Methodology
Development or design of methodology; creation of models.
7
Project administration
Management and coordination responsibility for the research activity planning and
execution.
27
8
Resources
Provision of study materials, reagents, materials, patients, laboratory samples, anim
instrumentation, computing resources, or other analysis tools.
9
Software
Programming, software development; designing computer programs; implementati
computer code and supporting algorithms; testing of existing code components.
10
Supervision
Oversight and leadership responsibility for the research activity planning and execu
including mentorship external to the core team.
11
Validation
Verification, whether as a part of the activity or separate, of the overall
replication/reproducibility of results/experiments and other research outputs.
12
Visualization
Preparation, creation and/or presentation of the published work, specifically
visualization/data presentation.
13
Writing – original draft
Preparation, creation and/or presentation of the published work, specifically writing
initial draft (including substantive translation).
14
Writing – review &
Preparation, creation and/or presentation of the published work by those from the o
editing
research group, specifically critical review, commentary or revision – including prepublication stages.
References
1.
Williamson PR, Altman DG, Bagley H, et al. The COMET Handbook: version 1.0. Trials.
2017;18(S3):280. doi:10.1186/s13063-017-1978-4
2.
Gargon E, Gorst SL, Harman NL, Smith V, Matvienko-Sikar K, Williamson PR. Choosing
important health outcomes for comparative effectiveness research: 4th annual
update to a systematic review of core outcome sets for research. Gillies K, ed. PLoS
One. 2018;13(12):e0209869. doi:10.1371/journal.pone.0209869
3.
Alkhaffaf B, Blazeby JM, Williamson PR, Bruce IA, Glenny A-M. Reporting of outcomes
28
in gastric cancer surgery trials: a systematic review. BMJ Open. 2018;8(10):e021796.
doi:10.1136/BMJOPEN-2018-021796
4.
Wild D, Grove A, Martin M, et al. Principles of Good Practice for the Translation and
Cultural Adaptation Process for Patient-Reported Outcomes (PRO) Measures: Report
of the ISPOR Task Force for Translation and Cultural Adaptation Background and
Rationale.
https://www.ispor.org/workpaper/research_practices/PROTranslation_Adaptation.pdf.
Accessed June 17, 2018.
5.
Alkhaffaf B, Glenny A-M, Blazeby JM, Williamson P, Bruce IA. Standardising the
Reporting of Outcomes in Gastric Cancer Surgery Trials: Protocol for the
Development of a Core Outcome Set and Accompanying Outcome Measurement
Instrument Set (The GASTROS Study). Trials. 2017;18(1):370. doi:10.1186/s13063017-2100-7
6.
Alkhaffaf B, Blazeby JM, Bruce IA, Morris RL. Patient priorities in relation to surgery
for gastric cancer: qualitative interviews with gastric cancer surgery patients to
inform the development of a core outcome set. BMJ Open. 2020;10(2):e034782.
doi:10.1136/bmjopen-2019-034782
7.
Core Outcome Measures in Effectiveness Trials Initiative (COMET). COMET Database.
http://cometinitiative.org/studies/search. Accessed January 8, 2019.
8.
Gargon E, Gorst SL, Harman NL, Smith V, Matvienko-Sikar K, Williamson PR. Choosing
important health outcomes for comparative effectiveness research: 4th annual
update to a systematic review of core outcome sets for research. Gillies K, ed. PLoS
One. 2018;13(12):e0209869. doi:10.1371/journal.pone.0209869
9.
Ann Arbor, MI: Survey Research Center, Institute for Social Research U of M.
Guidelines for Best Practice in Cross-Cultural Surveys.
29
http://www.ccsg.isr.umich.edu/images/PDFs/CCSG_Full_Guidelines_2016_Version.pdf.
Accessed September 15, 2019.
10.
OMERACT: Outcome Measures in Rheumatology. Handbook | OMERACT Handbook.
https://omeracthandbook.org/handbook. Accessed October 5, 2019.
11.
Smith TO, Hawker GA, Hunter DJ, et al. The OMERACT-OARSI Core Domain Set for
Measurement in Clinical Trials of Hip and/or Knee Osteoarthritis. J Rheumatol.
2019;46(8):981-989. doi:10.3899/JRHEUM.181194
12.
Beuscart J-B, Knol W, Cullinan S, et al. International core outcome set for clinical
trials of medication review in multi-morbid older patients with polypharmacy. BMC
Med. 2018;16(1):21. doi:10.1186/s12916-018-1007-9
13.
Park JK, Mecoli CA, Alexanderson H, et al. Advancing the Development of Patientreported Outcomes for Adult Myositis at OMERACT 2016: An International Delphi
Study. J Rheumatol. 2017;44(11):1683-1687. doi:10.3899/jrheum.161252
14.
Horbach SER, van der Horst CMAM, Blei F, et al. Development of an international core
outcome set for peripheral vascular malformations: the OVAMA project. Br J
Dermatol. 2018;178(2):473-481. doi:10.1111/bjd.16029
15.
Dodd S, Clarke M, Becker L, Mavergames C, Fish R, Williamson PR. A taxonomy has
been developed for outcomes in medical research to help improve knowledge
discovery. J Clin Epidemiol. 2018;96:84-92. doi:10.1016/j.jclinepi.2017.12.020
16.
Baiocchi GL, Giacopuzzi S, Marrelli D, et al. International consensus on a
complications list after gastrectomy for cancer. Gastric Cancer. 2019;22(1):172-189.
doi:10.1007/s10120-018-0839-5
17.
COMET Initiative - YouTube. https://www.youtube.com/channel/UCxkjZgL12_QXAzn8cO2nsg?&ab_channel=COMETInitiative. Accessed September 7, 2019.
18.
Fish R, Sanders C, Adams R, et al. A core outcome set for clinical trials of
30
chemoradiotherapy interventions for anal cancer (CORMAC): a patient and healthcare professional consensus. lancet Gastroenterol Hepatol. 2018;3(12):865-873.
doi:10.1016/S2468-1253(18)30264-4
19.
What are the top 200 most spoken languages? | Ethnologue.
https://www.ethnologue.com/guides/ethnologue200. Accessed October 12, 2019.
20.
Kuliś D, Whittaker C, Greimel E, Bottomley A, Koller M. Reviewing back translation
reports of questionnaires: the EORTC conceptual framework and experience. Expert
Rev Pharmacoeconomics Outcomes Res. 2017;17(6):523-530.
doi:10.1080/14737167.2017.1384316
Tables
Table 1. Studies using multi-language Delphi surveys in the development of international COS.
31
Condition/Group
Original Language
Target Language(s)
Total Pa
Hip and Knee Osteoarthritis
English
Italian & Spanish
426
French
Dutch, German, English
150
English
Swedish, Dutch & Korean
500
English
Dutch
301
English
Chinese, Dutch, German, Italian,
1021
OMERACT-OARSI11
Medication review in multi-morbid older
patients with polypharmacy
OPERAM12
Idiopathic inflammatory myopathy
OMERACT13
Vascular Malformations
OVAMA Group14
GASTROS Study
Portuguese, Spanish, Turkish,
Table 2. Timeline related considerations in undertaking multi-language Delphi survey.
32
Language Version*
Translation 1
Docume
Time to return
Harmonization
Time to set
Time to pilot
Time
nt
completed
across language
up online
survey and
obta
Prepara
translations
versions
surveys
complete
appr
tion
for r1
2 weeks
6 weeks
amendments
2 weeks
1 week
1 week
25 we
Translation 2
10 weeks
1 week
29 we
Translation 3
3 weeks
1 week
Not r
Translation 4
10 weeks
1 week
Not r
Translation
18 weeks
1 week
***No
recei
Translation 6
12 weeks
3 weeks
40 W
Translation 7
2 weeks
1 week
2 wee
* The language versions are anonymized. **This represented the time the study management group
requested collaborators to begin ethical approval applications until IRB approval was received and not
necessarily the time between actual submission of the application and receiving approvals. ***Ethical
approval was not received before the end of round 1 of the Delphi survey. No patients were recruited
from this team’s country.
Figures
33
Figure 1
Flow diagram demonstrating which studies were included in the structured review.
34
Figure 2
a) Characteristics of translators undertaking forward translation(s). b) Characteristics of
translators undertaking backward translation(s).
35
Figure 3
Step by step translation process for multi-language Delphi surveys
Figure 4
Cumulative weekly recruitment figures for round 1 of GASTROS Delphi survey
36