Preprint: Please note that this article has not completed peer review. Facilitating International Participation in Core Outcome Set Development: A Proposed Approach for Multi-language Delphi Surveys, a Review of Current Practice and Issues to Consider CURRENT STATUS: UNDER REVIEW Bilal Alkhaffaf Salford Royal NHS Foundation Trust bilal.alkhaffaf@srft.nhs.ukCorresponding Author ORCiD: https://orcid.org/0000-0001-5751-1846 Paula Williamson University of Liverpool Jane M Blazeby University of Bristol Anne-Marie Glenny University of Manchester Ademola Adeyeye University of Ilorin Teaching Hospital Paulo Matos Costa Hospital Garcia de Orta Ismael Diez del Val Hospital Universitario Basurto Suzanne Gisbertz Amsterdam UMC Ali Guner Karadeniz Teknik Universitesi Simon Law University of Hong Kong 1 Hyuk-Joon Lee Seoul National University College of Medicine Ziyu Li Peking University Cancer Hospital and Institute Koji Nakada The Jikei University Daisan Hospital Rafael Mauricio Restrepo Nunez Hospital Rey Juan Carlos Daniel Reim TUM School of Medicine John V Reynolds Broomfield Hospital Peter Vorwald Hospital Universitario Fundacion Jimenez Diaz Daniela Zanotti University Hospitals Birmingham NHS Foundation Trust William Allum Royal Marsden NHS Foundation Trust Asif Chaudry Royal Marsden NHS Foundation Trust Ewen Griffiths University Hospitals Birmingham NHS Foundation Trust Iain Bruce Manchester University NHS Foundation Trust DOI: 10.21203/rs.3.rs-16047/v1 SUBJECT AREAS Health Economics & Outcomes Research KEYWORDS Stomach Neoplasms, Surgical Oncology, Patient Outcome Assessment, Consensus, Outcome Assessment (Health Care), Core Outcome Set 2 Abstract Background Core outcome sets (COS) should be relevant to key stakeholders and widely applicable so that researchers are willing to use them when designing trials. Many COS developers have utilised online Delphi surveys which enable wider international stakeholder participation. Despite broader participation, most international COS projects have restricted Delphi surveys to a single language potentially excluding important opinion from those who are not fluent in the survey language. Methods A structured review of current approaches to translating Delphi surveys for COS development was undertaken. We present a proposed methodology adapted from international guidance. Results Four studies were identified from our structured review. Wide variation exists in the methodological approaches to translating Delphi surveys for the developing of COS. Issues which arise when developing multi-language Delphi surveys include establishing translation groups, timelines, financial implications, strategies to maximise recruitment and regulatory approvals. Conclusion Consideration of the issues described will improve planning by other COS developers and can be used to widen international participation from both patients and healthcare professionals. Main Article Background Core outcome sets (COS) aim to standardise the reporting of critically important outcomes within a given research field1. COS should be relevant to key stakeholders and widely applicable such that researchers are encouraged and willing to incorporate them in trials. Approaches to improve the relevance of COS can take many forms, including involving stakeholders with lived experience of the condition or intervention in question. Many COS developers are using online Delphi surveys during stages to prioritise potentially important outcomes2. This approach enables overseas stakeholders to participate more readily. Such broad participation can give COS greater validity across different geographical regions and consequently make them more likely to be used in future trials regardless of the location where trials are undertaken. Unless COS are widely used in trials within the same research field, the challenge of inconsistent outcome reporting will persist3. Most research groups developing ‘international’ Delphi surveys have restricted themselves to their 3 native language (usually, but not exclusively, English). This approach is less resource intensive than translating the survey into multiple languages and overcomes issues with ambiguity or changes in meaning - a recognized challenge with translation4. However, these methodological challenges are not insurmountable and some COS developers are translating Delphi surveys to minimise the risk of excluding important opinion from those not fluent in the study’s primary language. Core outcome sets (COS) aim to standardise the reporting of critically important outcomes within a given research field1. COS should be relevant to key stakeholders and widely applicable such that researchers are encouraged and willing to incorporate them in trials. Approaches to improve the relevance of COS can take many forms, including involving stakeholders with lived experience of the condition or intervention in question. Many COS developers are using online Delphi surveys during stages to prioritise potentially important outcomes2. This approach enables overseas stakeholders to participate more readily. Such broad participation can give COS greater validity across different geographical regions and consequently make them more likely to be used in future trials regardless of the location where trials are undertaken. Unless COS are widely used in trials within the same research field, the challenge of inconsistent outcome reporting will persist3. Most research groups developing ‘international’ Delphi surveys have restricted themselves to their native language (usually, but not exclusively, English). This approach is less resource intensive than translating the survey into multiple languages and overcomes issues with ambiguity or changes in meaning - a recognized challenge with translation4. However, these methodological challenges are not insurmountable and some COS developers are translating Delphi surveys to minimise the risk of excluding important opinion from those not fluent in the study’s primary language. Aims and Objectives Currently, there is no standardized method of translating Delphi surveys for use in the development of international COS. This paper aims to address this need by presenting an approach to develop a multi-language Delphi-survey adapted from international consensus guidelines on the translation and cultural adaptation of patient-reported outcomes4. The survey forms an integral part of the GASTROS study (GAstric Cancer Surgery TRials Reported Outcome Standardisation) which aims to develop an 4 international COS for surgical trials in gastric cancer5. The scope and design of the GASTROS study has been previously detailed5. In summary, following a systematic review of randomized control trials3 and a series of in-depth patient interviews6, a long-list of potentially important outcomes was rationalized into a list of 56 outcomes. Following a consultative exercise with key stakeholders (see below), these 56 outcomes were presented to patients and healthcare professionals in a two-round, multi-language Delphi survey. The objectives of this paper include: 1. To describe the current methodological approaches used by COS developers in the translation of Delphi surveys, 2. To outline a pragmatic, robust and replicable approach to translating Delphi surveys for use in COS development, and 3. To outline important logistical considerations in preparation for an international Delphi survey. These considerations are relevant to COS researchers seeking to translate their surveys or undertake the Delphi process in a single language but in multiple regions. 1. Current Approaches to translating Delphi Surveys To gain an understanding of current translation approaches for multi-language Delphi-surveys, a structured literature review of the COMET database was undertaken7. The COMET database is a comprehensive registry which (as of 03/09/2019) contains 337 published and 280 ongoing COS respectively dating back from 1981. The database is kept up-to-date through annual systematic reviews of scientific databases (using MEDLINE via OVID and SCOPUS), automated alerts from MEDLINE via OVID, SCOPUS and Google Scholar and direct submissions from COS developers8. Search Strategy and Inclusion Criteria The COMET database enables users to search for terms within the ’title’, ’abstract’ or ’author names’ categories. Searches can be restricted according to health area, target population, methods, 5 stakeholder involvement, study type and publication year. A broad search for the terms ‘international’, ‘language’ or ‘translat$’ in the title and abstract was undertaken with no other restrictions. A flow chart illustrating how studies were included is summarized in figure 1. Studies included in our review were those that used a multi-language Delphi survey in the development of their respective COS. Only publications from completed studies were included - COS methodology is a relatively new research field and so planned approaches may not accurately reflect the final methodology used. The COMET database may contain several different references to COS development for the same project. Any related publications were consolidated and handled as a single COS study. Examining Methodological Approaches to Translation Four studies (summarized in table 1) were deemed eligible for inclusion in the structured review. Corresponding authors were contacted and asked to participate in a questionnaire examining various aspects of their respective methodological approaches (appendix 1). Our approach to translation was based on international consensus guidance on translation and cultural adaptation of patient reported outcome measures4, and so questions were developed to examine aspects of methodology presented using this guidance. The questionnaire focused particularly on how items presented in the Delphi surveys were translated and how discrepancies and conflict was resolved. Responses were received from the corresponding authors of all four studies identified. Responses were combined with data from the respective publications and are presented below. General Approaches to Translation All groups summarized their approach to translating surveys with one providing a reference to their methodology and another referring to methodology described by the OMERACT group at a COS development meeting 9,10. In addition to forward translations, three groups undertook a backward translation of the survey from the target to source language. The number of forward and backward translations differed in each study. Two studies undertook a single forward translation whilst the others undertook two and three. One group used no backward translations, one study undertook a 6 single backward translation whilst the other two undertook two backward translations. The characteristics of those involved in the translation processes also differed amongst the groups (figure 2); no professional services were employed, and all translations were undertaken by healthcare professionals or lay translators. Discrepancies and Harmonization All four groups described an approach to managing discrepancies in translations. Two groups reported that discrepancies were discussed within the ‘research group’ until consensus was reached, whilst the remaining two referred to individuals outside of the ‘research group’ who were fluent in the target language to resolve any language issues. Cognitive Debriefing Three groups described undertaking an exercise to test alternative wording and check understandability, interpretation and cultural relevance of the proposed Delphi survey in the target language. Using cognitive interviews, they studied patients/relatives and health professionals’ interpretation of the translations to examine face validity. Two of these involved patients and/or their relatives whilst the third was based on the opinion of healthcare professionals alone. 2. Proposed approach to translating Delphi surveys As previously discussed, one of the principal aims of translating the Delphi survey in the development of COS is to include the opinions of stakeholders who are not fluent in the source language. With respect to the GASTROS study, this was especially important given that the highest incidence of gastric cancer exists outside of English-speaking countries, in the Far East, Central and South America and Southern Europe. In developing our approach to translate the survey, the study management group was keen to ensure that it was both methodologically sound yet pragmatic such that it could be easily reproduced by multiple international collaborators within a relatively short period of time. In 1999, the ISPOR-TCA group (The Professional Society for Health Economics and Outcomes Research – Translation and Cultural Adaptation group) was formed to discuss and develop guidelines for translating patient-reported outcome measures. The group highlighted inconsistencies with previous 7 methodologies and nomenclature in this field and sought to address these by developing guidance setting out ‘principles of good practice’4. These principles were derived from 12 major sets of translation guidelines from the following groups: American Association of Orthopaedic Surgeons (AAOS) Association of Test Publishers EORTC group Euro QoL group Evidence: Clinical and Pharmaceutical Research FACIT group Health Outcomes group (HOG) Health Utilities Inc. (HUInc) International Quality of Life Assessment (IQOLA) group Kidney Disease Quality of Life (KDQOL) Medical Outcomes Trust (MOT) World Health Organization Other consensus guidelines have been developed for translating surveys. The Survey Research Centre (SRC) guidelines provide broader consideration of the translation process and describe practical support from expert contributors’ experience of different survey types9. There is much cross-over between the two SRC guidelines. Given the focus of our work was primarily outcome related translation, the principles as set out by the ISPOR-TCA group formed the basis of our methodology, with references made to the SRC guidance and some pragmatic amendments which are explained in further detail below. Adopted Translation Approach Below, we describe ten steps involved in translating the Delphi survey used in the development of a COS for surgical trials in gastric cancer. The full rationale for each step, and the risks of omitting them, is described in detail in the ISPOR-TCA guidance; we have stated the rationale for the steps below (particularly in relation to pragmatic deviations) where we believed it was necessary to do so. The source survey was developed in English and translated into seven target languages (Simplified Chinese, Dutch, German, Italian, European Portuguese, European Spanish and Turkish). Appendix 2 details the instructions which were provided to each international collaborator responsible for leading the translation process in their respective country. These outlined which files required translation, how the translation should be undertaken, and by whom. 8 By using this approach in addition to considering other important issues (section 3), the GASTROS study was able to recruit 1021 participants (463 surgeons, 313 patients, 245 nurses) in the first round of the Delphi survey, with 349 participants using the English language version and 672 using one of the seven other language versions. 1. Preparation a. Cognitive Debriefing: i. Cognitive debriefing aims to identify issues with comprehensibility of key concepts and understanding amongst potential participants. As previously stated, we presented survey participants with 56 outcomes which had been rationalized following a process that had identified a long-list of potentially important outcomes from a systematic review and indepth patient interviews. This rationalization process from the long-list to the 56 survey items involved key-stakeholders (members of the GASTROS study group, surgeons, oncology nurses and patients) who also ensured that the outcomes were accompanied with plain English-language explanations that could be understood by all participants including patients. A further consultative exercise with an English-speaking patient-group was held to ensure that the meaning of each outcome, in addition to other survey-related files were clearly understood. Undertaking this work prior to translation was essential as it minimized the possibility of ambiguous meanings which could result in a mistranslation. b. Preparing documents for translation i. Four documents were needed to run the Delphi survey; a participant information sheet and three further files which were required to set up the web-based survey. We used DelphiManager 3.0 platform, developed and maintained by the COMET Initiative (http://www.comet-initiative.org/), to undertake the Delphi (section 3). The three files included: 1. File 1 (appendix 3): An excel file containing details of each outcome, accompanying meaning and the ‘outcome area’ under which the outcome was categorised15. 9 2. File 2 (appendix 4): User-defined text: A file containing text specific to our surveys (in this case the GASTROS Delphi survey). 3. File 3 (appendix 5): Static text: A file containing text common to all Delphi surveys which was used in the setting up process by the DelphiManager team. ii. Preparation for Round 2 of the Delphi survey: Additional translations were required to support the second round of the survey. These included: 1. Outcomes identified by participants in round 1 as being important to consider that were not identified from the systematic review or patient interviews. 2. Legends and terms required to produce charts which were presented to survey participants in round 2. 3. Comments and feedback from study participants. iii. Following Round 2 of the survey: 1. Participants who changed their scores between rounds were given the opportunity to provide their reasons for doing so. 2. Participants were also given the opportunity to provide further comments after completing the survey. c. Understanding which methodological approaches to employ Two approaches to translation were adopted: 1. “Two forward, one back translation”; this approach was the most comprehensive and labour intensive as it required a further nine steps (below) before a final file version was agreed. Following discussion amongst the study management team, it was deemed content which could alter the meaning of the outcomes being presented and ultimately influence how the overall aims of the survey was received and understood by participants (file 1, file 2, and additional outcomes identified by participants in round 1) underwent this approach. The steps involved in this process are described in greater detail in points 2 to 10. 2. “One forward, dual independent proofreading”; File 3 consisted primarily of short 10 instructional phrases (e.g. ‘click here’, ‘register’, and ‘next page’) which were necessary for the functionality of the survey. As these terms would not materially influence the comprehension of the survey’s purpose or outcomes presented within it, a simplified, less resource intensive approach was adopted. This file underwent a single forward translation followed by two independent proof-readings by translators who compared the translated and source files for accuracy and quality. Any corrections or amendments were undertaken through discussion between the translator and proof-readers. This approach was also adopted for the translation of participant comments, feedback and reasons for changing scores between round 1 and round 2. d. Setting up translation teams To support the translation work, an international working group (IWG) was established (see section 3). Each collaborator within the IWG was responsible for overseeing a team which would undertake the translation and ensuring that the key concepts of the study were appropriately communicated. The translation process was supported by the GASTROS study Chief Investigator (BA) if any clarifications were required. The characteristics of individuals involved in this process are described in greater detail in Appendix 2. In summary, each team was made up of an IWG lead, two forward translators and a single backward translator. e. Developing instructions for translations Setting out the methodology a priori in a clear and structured document ensured that collaborators and their teams understood what would be required of them at each stage of the translation. These instructions included ongoing responsibilities prior to and following future rounds of the Delphi survey. This was essential given that one of our primary aims was to ensure that our approach was easily replicable. Figure 3 is a flow diagram which details these stages and the order in which they were to be undertaken. Feedback from the IWG was positive in response to these instructions with collaborators 11 reporting that the document enabled them to undertake the translation process efficiently. f. Quality assurance IWG collaborators were asked to provide documented evidence for each step of the translation process. These could then be reviewed by the study management team as required. 2. Forward Translation Two independent forward translations by individuals who were native speakers of the target language were undertaken. Culture is a primary determinant of language and therefore native speakers have advantages with language abilities compared to second-language speakers. Having two independent forward translations enables detection of errors and divergent interpretations that could otherwise lead to bias. 3. Reconciliation There are several approaches which can be used to reconcile the forward translations. We opted to use the ‘in-country’ IWG collaborator who was also involved in cognitive debriefing and piloting of the survey as this was pragmatic and would not require the identification of further individuals to undertake this step. No issues arose from the reconciliation process, however, had further clarifications been required, they would have been directed to the CI. 4. Back Translation The ISPOR-TCA guidance states that ‘back translation’ is necessary, whilst the Survey Research Centre guidance suggests that this is not required. We opted to undertake a single back translation to provide quality-control of the forward translations. Whilst the ISPOR-TCA guidance suggests that this should be undertaken by individuals who are native speakers of the source language (i.e. English), we found it challenging to identify seven native English-speakers who were also fluent in the required target languages and had an understanding of outcome reporting without referring to a professional service. We opted to ensure that back-translators were fluent in English and independent from the forward translators. 12 5. Back Translation Review This step is important as it ensures that the cross-cultural adaptation needs of the translation is met. Without it, there is a risk of that a mistranslation or omission would remain in the translation. This was undertaken by the CI in combination with the IWG collaborator by comparing the back translation to the source document. No significant discrepancies between the source and back-translated files were identified across any of the translations. 6. Harmonization Across Different Languages There is no agreed method to how harmonization across different translations should be enforced; many approaches omit this step. However, our group opted to ensure harmonization between each language at each step of the process. This was undertaken by the CI. We did not encounter significant differences between translations. An example of a minor change that was made across surveys was the term ‘last round scores’ which in the context of the survey meant ‘previous round scores’. Some teams translated this as ‘the final round scores’ which had to be altered to ensure all versions contained the same meaning. 7. Cognitive Debriefing of the Translation Following harmonization across translations, all survey versions were built using the DelphiManager platform (section 3). A further cognitive debriefing exercise was undertaken by asking IWG collaborators and their translation teams to complete a pilot version of the survey to identify grammatical or stylistic errors and check understandability, interpretation and cultural relevance of instructions and outcomes within the survey. 8. Review of Cognitive Debriefing Results and Finalization There were no issues highlighted with comprehensibility or understanding. Spelling mistakes and minor grammatical errors (e.g. pronouns ‘you’ formal and informal) were altered. 9. Proofreading IWG collaborators were once again asked to examine the survey and ensure that any issues highlighted in the previous steps had been addressed. No further changes were identified in any of the language versions by this stage. 13 10. Final Report and ‘Start of Survey’ The ISPOR-TCA group guidance recommends that a report should be produced detailing the methodological approach for translation and rationale for each step. The final report for translations undertaken for the GASTROS study is represented by this paper. 3. Important Considerations In this section, important considerations are presented to support the translation process and maximise participation from stakeholder groups in different countries. Many of these considerations can be applied by COS developers if they choose not to translate their Delphi survey, but are keen to widen participation from those who speak the primary language. As in section 2, we describe the rationale for each consideration and the potential risks of not applying these steps (where applicable). International Working Group The GASTROS study is a collaborative international initiative which sought to attract global representation within the study group. Motivated, research-active collaborators from countries with a significant incidence of gastric cancer were approached to form an IWG. Individuals signed a ‘terms of reference’ document which outlined the benefits of their involvement in addition to the following responsibilities: To form a local team and oversee the translation of the Delphi survey (where applicable) To drive recruitment locally, regionally, nationally and internationally through organisations and personal networks To garner and develop links specifically with patient groups who would be able to participate in advertising the Delphi survey To identify the need and apply for relevant local ethical and regulatory approvals The IWG was made up of collaborators from the following countries: Brazil Mainland China and Hong Kong Germany Ireland Italy Japan The Netherlands Nigeria Portugal South Korea Spain 14 Turkey United Kingdom Ensuring the IWG was set up early maximized our ability to develop translations in a timely manner and recruit evenly across all stakeholder groups from a broad range of countries. Patient and Public involvement A Study Advisory Group (SAG) separate to the IWG forms part of the management structure of the wider GASTROS study. The SAG is made up of key stakeholder representatives including patients. The group provides advice on the methodology of the study, general delivery of the study against its stated objectives and ensures that the viewpoints of all stakeholder groups are considered. In addition, patient groups (see acknowledgements) were vitally important in reviewing and piloting the translated surveys prior to recruitment to the Delphi. These groups were also instrumental in recruiting patients (see below). Who should undertake the translation work? The GASTROS study management group opted to set up local translation teams made up of healthcare professionals who met the rigorous criteria as set out by the ISPOR-TCA group. An alternative approach would have been to employ a professional translation service to undertake this work. One of the benefits of professional services is the ability to complete the translations in a relatively short period of time, in addition to developing an unlimited number of translations which may have resulted in wider participation in the Delphi survey. The main disadvantage to this approach is cost. Quotes from three different professional translation services (all familiar with the ISPOR TCA guidance) were requested to support rounds 1 and 2 of the survey. In April 2018, the estimated costs were in the region of 3200GBP-4000GBP per language. All translations for rounds 1 and 2 of the survey would be finalized within 5 and 2 weeks respectively. Due to the financial limitations of undertaking the survey in 7 languages, we did not pursue this option. Milestone & timeline planning Setting aside enough time for the translation process is of paramount importance, particularly if COS developers are seeking to translate their surveys into more than one language. Some of the translation steps required all language versions to have reached the same stage prior to moving onto 15 the next stage. For example, all initial translations had to have been completed before harmonization across surveys could be achieved. Without this, we were unable to ask collaborators and their teams to pilot their respective surveys. Furthermore, we chose to open recruitment to all language versions simultaneously and so all translations needed to have been fully completed before participants could complete their surveys. This was also the case for the second survey round. The impact of ethical approval applications on timelines is discussed below in greater detail. Table 2 details timelines involved in producing all versions of the survey for both rounds. The time to return the initial translation documents and obtain ethical approvals resulted in the greatest variations with respect to the overall timelines. We found that setting regular milestones and realistic timelines helped achieve the required translation objectives. Regular communication between the CI and collaborators underpinned this process. Our aspiration was to translate the Delphi survey into Japanese and Korean to enable wider patient participation from these countries. Due to challenges in identifying collaborators at an early stage, assembling a translation team and meeting timelines, this could not be pursued. However, potential participants were invited to complete the English language version of the survey. Recruitment and retention targets COS developers should consider minimum recruitment targets. Whilst there is no sample size requirement for Delphi surveys, the GASTROS protocol initially set a conservative target of 100 participants in total to be recruited over a period of 6 to 8 weeks in round 1. However, as interest in the study and international collaboration grew, it was clear to see that this target would easily be surpassed. As described below, once the survey opened and momentum began to gather, we witnessed a ‘snowballing’ effect amongst all three stakeholder groups. We therefore extended recruitment to 13 weeks by which time new participation had plateaued (figure 4). For round 2, an initial retention target of 80% was set following discussions with members of our study management group who have extensive knowledge and experience of COS development. Automated reminder emails were sent out on a weekly basis to participants and support from professional bodies (in countries where round 2 responses were slow) was sought to encourage 16 completion of the survey. Personalised e-mails from the CI to professionals were also sent. Using this strategy, we were able to retain 65% of participants from round 1 by week 13, by which time no further responses were being received. Paper and Internet-based Delphi survey versions The GASTROS study used both internet-based and paper versions of the Delphi survey. The internetbased versions enabled us to reach participants in nearly 60 countries, the vast majority of which did not have formal IWG collaborators. The paper-versions (printed versions of the internet-based survey which were uploaded electronically by local collaboration teams) also enabled us to recruit participants (particularly patients) who either did not readily have access to the internet or were not ‘internet-literate’ (e.g. China and Hong Kong, Nigeria and Turkey). Several platforms exist to enable COS developers to run Delphi-surveys. These include platforms specifically designed for Delphi surveys and other generic survey platforms which researchers can use. When considering multi-language surveys, it is essential to ensure that the servers on which the surveys are hosted meet the necessary data protection regulations and are accessible particularly from countries where restrictions to certain ‘out-of-country’ domains exist. Furthermore, COS developers must ensure that the platforms used are able to run surveys using different language scripts and writing systems. Our group used DelphiManager as it fulfils the required data protection criteria (as set out by our United Kingdom ethical approval) and can work with all language systems including English, Chinese, Japanese and Korean. Furthermore, the online survey domains are accessible from countries which commonly restrict access to other foreign domains. DelphiManager has additional features which simplified recruitment and completion of the surveys such as being able to send automated reminders to individuals who had yet to complete all their answers. Measures to maximize survey recruitment One of the strengths of our Delphi survey was that it was able to recruit over 1000 patients and healthcare professionals from nearly 60 countries in round one. From the study’s inception, the study team recognised the importance of developing a clear networking and dissemination strategy5. We 17 hypothesised that this was necessary to achieve broad stakeholder participation both nationally and internationally. Several strategies were employed to maximise recruitment: 1. Dissemination of results from previous study stages The study protocol and findings from previous study stages3,5,6 were presented at targeted national and international meetings which were well-attended by potential healthcare participants. This was integral to generating interest and support for our study and ensured that participants understood the premise for GASTROS long before the Delphi survey opened for recruitment. All presentations contained directions to the study website and social media accounts (below). 2. Local recruitment of patients through outpatient clinics Ethical approval enabled the study team to recruit patients directly from outpatient clinics. Our experience from the United Kingdom is that many patients regularly attend patient support groups and are in contact with other eligible patients. As a result, a snowballing effect resulted in patients being recruited by patients already within the study. 3. Support from stakeholder groups/associations and national research networks Support from national and international professional associations and organisations (see acknowledgements) was sought in the early stages of the study. Study group members presented the study objectives at closed executive level meetings to gain support and adoption from influential bodies including professional associations patient groups and charities. Many of these organisations have large memberships (and corresponding electronic mailing lists) through which the study was advertised. Most of the groups through whom we sent out invitations followed up an initial e-mail with a further reminder approximately 4 weeks later resulting in further recruitment spikes. Furthermore, the GASTROS study was adopted onto the National Institute for Health Research (NIHR) Portfolio (CPMS study ID 38318). This enabled us to advertise the study to healthcare 18 professionals and patient support groups within the United Kingdom through the national Clinical Research Networks. Our experience suggests that recognition by respected associations and groups results in a ‘snow-balling’ effect with subsequent support from others becoming easier to harness. 4. Collaborations Standardising the reporting of outcomes can be achieved through several approaches. The GASTROS study aims to identify critically important outcomes across the entire spectrum of outcome types. Others have concentrated on the reporting of outcomes within a defined area. For example, the GASTRODATA group (www.gastrodata.org) have sought to standardise the reporting of all major post-gastrectomy complications16. Whilst the goals of both studies are different, both teams have been able to work closely to minimise duplication of work. In addition, the GASTRODATA group was able to promote recruitment to the GASTROS Delphi survey through its membership and respective networks. Such collaborations will also be vital for the future development of outcomes research within the field of gastric cancer surgery. 5. Personalised emails a. Most of the study management group, study advisory group and international working group members have extensive research experience within the field of gastric cancer surgery. Each member was asked to promote the study through their personal research and clinical networks. Bulk e-mails through professional bodies may be ignored by potential participants or diverted into ‘spam’ e-mail folders, hence why this approach was employed. b. Corresponding email addresses for authors from previous trials and protocols included in our systematic review3 were identified and personal invitations sent. This captured research-active healthcare professionals from non-English 19 speaking regions where no formal national gastric cancer associations exist (e.g. Eastern Europe). 6. Social media and multimedia a. The study website (http://www.gastrosstudy.org) provides detailed information about the GASTROS study aims as well as all its outputs. Prior to the commencement of the Delphi survey, potential participants who had heard about the study were able to register their interest to participate. In the preceding 18 months before the survey opened, 150 healthcare professionals and patients had registered. b. In addition to the study’s twitter account (https://twitter.com/GASTROSStudy), members of the research team posted updates on their personal Twitter and LinkedIn accounts. Regular study updates provided potential participants with an opportunity to better understand the study aims and keep up to date with its progress. Examination of analytics revealed that Twitter and LinkedIn posts in the run-up to and during round 1 of the survey regularly received over 4000 and 3000 views respectively. c. A series of short videos were produced for the study. These provided potential participants with an alternative way to engage with the study. At the time of writing, these videos had been viewed over 600 times. In addition to an introductory video on the study, a detailed step-by-step guide to completing the online Delphi survey was developed. This created additional content for social media platforms and the GASTROS website which in turn enabled the study to maintain a regular online presence. COS developers may wish to produce different language versions or translate video captions relatively easily to expand their reach. Additional COS-related material is already available from 20 the COMET initiative YouTube site17 with versions available in Dutch, Portuguese and Chinese. Work is underway to develop other language versions as well. Whilst advertising the study through these avenues aims to increase the number of recruits, care must also be taken that potential participants are not ‘bombarded’ with requests to participate in the survey. A small number of healthcare professionals highlighted that this was an issue. This coupled with the well-recognised challenges of ‘survey-fatigue’, may in fact be counter-productive and result in apathy amongst potential participants. Ethical Approval The requirement for regulatory or ethical approval varied across different regions. In the United Kingdom, the approach to ethical approval has not been consistent; our group was asked to submit a full application for ethical approval committee consideration, whilst other groups have been able to gain approval through proportionate review18. Each IWG collaborator was responsible for understanding local requirements and applying for ethics if it was required. They were asked to enquire about these at the start of their agreement to participate in the study and applications were made in parallel to the translation work. Two of our international collaborating centres did not require ethical approval as local collaborators did not recruit patients directly from their clinical practice but instead advertised the study through local patient groups and recruited healthcare professionals by advertising through national Societies and networks. The time taken to complete this process varied significantly (table 2) and was largely dependent on the frequency of and access to ethics committee meetings, requirements to amend submitted materials and delays in final decisions reaching the collaborators. COS developers should investigate the need for ethical approval as early as possible to avoid unnecessary delays. Financial Planning Several aspects of undertaking multi-language Delphi surveys may potentially incur significant costs depending on which approaches are adopted. COS developers should take these into account when 21 planning their studies. These include: 1. Cost of professional translations. This represents the largest financial burden and has been discussed above. 2. Ethical approval. Some of our non-UK ethical approval applications required payments of up to €250 Euros. 3. Use of electronic mailing lists. Some stakeholder groups may charge administration fees to send out invitations to their membership. 4. Cost of Delphi survey platform. Whilst open-access platforms exist, our group opted to pay to use a dedicated Delphi survey platform designed for the development of COS. 5. Statistical and qualitative methods support may be required when analysing scores in rounds one and two, depending on the nature of feedback to be given. Discussion This paper is the first to address the topic of translation and cross-cultural adaptation in the context of developing Delphi surveys for COS. We have presented a robust 10-step approach adapted from international consensus guidelines which was easily and accurately replicated by seven different translation teams within an acceptable timeframe. Undertaking ‘international’ Delphi surveys has become easier as web-based platforms enable wider participation across different geographical regions. Whether or not this is warranted depends on the scope of the COS being developed and who is being targeted to use the COS. Very few pathologies or interventions are limited to one geographical region and consequently trials in their respective fields are undertaken all over the world, often collaboratively across different regions. In order to standardise the reporting of critically important outcomes, it becomes necessary that a COS is adopted by all trialists within the given research field. For this to be achieved, trialists need to be confident that the COS that has been developed is robust and has considered the views of relevant 22 stakeholders from a wide range of backgrounds. In our case, it was essential that stakeholders from the Far East as well as Asia, South America and Europe participated as this is where most gastric cancer occurs and where most of the trials are undertaken. Whilst translation of our Delphi survey aimed to widen participation and broaden the views taken into consideration, the value of doing so is one which warrants further discussion. English is the most commonly spoken language across the world19 and is used by most scientific and healthcare publications. It is therefore unsurprising that most ‘international’ COS projects employing Delphi surveys used an English-only version. The English version of our Delphi survey was offered to all participants, however most preferred to complete the Delphi in their native language. Whilst for many this choice would have been because they did not speak English, a significant proportion of bi-lingual healthcare professionals known to the study team preferred to use a non-English version. One may argue that this enabled participants to engage more confidently in the process and their understanding of what was being asked of them and the quality of their responses was consequently better. Such a high uptake in non-English surveys was not experienced by the four groups who completed our questionnaire on methodology. However, this likely reflects challenges related to recruitment in non-English speaking regions which we address in our ‘issues to consider’ section. Undoubtedly, achieving high quality and accurate translations is resource intensive. They can take time if undertaken by healthcare professionals or pose significant financial costs if study groups employ professional services. However, restricting a consensus-seeking process in the development of an international COS to a single language exposes studies to the risk of excluding important opinion from those not fluent in this language. It may therefore be argued that for a COS to be truly regarded as ‘international’, the consensus-seeking process should be undertaken in the native language of the participant. Whilst we have demonstrated that non-English Delphi surveys can triple the number of total participants, it is not known whether these additional participants bring a different perspective that has not already been captured through the English-language version. It is likely that there are many additional factors which may contribute to the validity of Delphi survey results (e.g. cultural and geographical differences of participants, regional economic status and 23 quality of healthcare provision) and COS developers should consider these carefully during the planning phase. These, along with other factors will be the focus of a future analysis by our research group. We have demonstrated that there is no standardised approach to translation in this field. So far, each of the four COS groups in section 1 of this paper used different methods to forward translate and utilised translation teams with different member characteristics. Whilst back translation was not undertaken in a uniform manner, the issue of whether it is required is one on which there is disagreement. Considering this, and the fact that, in our study, back translation did not identify significant discrepancies, COS developers may be justified in omitting steps 4 and 5 of our approach. This should however be done after careful consideration as the importance of back translation may depend on the type of outcomes that are being translated. It is possible that certain outcomes are conceptually alien between cultures or geographical regions and undergoing an added step to reduce the risk of mistranslation is warranted. In the field of patient-reported outcome measurement (PROM), it is common for questionnaires to undergo translations (for use in international trials). The methods required for PROM translation is rigorous and includes back translation20. Whilst it may be argued that less rigorous methods could be used in Delphi surveys for COS, to ensure optimal face validity of items the same standards are recommended. In any case, we did not find that back translation extended our timelines significantly, however, additional steps in translation may result in an added financial burden should COS developers use professional translation services. Conclusion We present a robust method of translating Delphi surveys used in the development of COS adapted from international consensus guidelines in the field of outcome reporting. Consideration of the issues described will improve planning by other COS developers and can be used to widen international participation from both patients and healthcare professionals. Abbreviations COS Core outcome set GASTROS GAstric Cancer Surgery TRials Reported Outcome Standardisation 24 IWG International working group SAG Study advisory group ISPOR-TCA The Professional Society for Health Economics and Outcomes Research – Translation and Cultural Adaptation group OMERACT Outcome Measures in Rheumatology Declarations Ethical Approval and Consent to Participate This study describes the methodology used to undertake a Delphi survey. The study was given ethical approval by the North West - Greater Manchester East Research Ethics Committee (18/NW/0347) and governance approvals by Manchester University Hospitals NHS Foundation Trust. Consent for Publication Not applicable Availability of Data and Material The datasets analysed during the current study available from the corresponding author on reasonable request. Competing Interests The authors report no conflicts of interest. Funding This study is funded by the National Institute for Health Research (NIHR) Doctoral Research Fellowship Grant (DRF-2015-08-023). JMB is partially funded by the NIHR Bristol Biomedical Research Centre and the MRC ConDUCT-II Hub for Trials Methodology Research. PRW was funded by the MRC North West Hub for Trials Methodology Research (Grant ref: MR/K025635/01). This paper presents independent research funded by the National Institute for Health Research (NIHR). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health. Reprints Reprints will not be available from the authors. 25 Acknowledgements The authors would like to highlight the role undertaken by Dr Aleksandra Metryka, Senior Clinical Trials Coordinator, who facilitated the running of this study. In addition, the authors would like to thank the following associations and groups for their support in facilitating recruitment to the GASTROS study Delphi survey: The International Gastric Cancer Association (www.igca.info) The Association of Upper Gastro-Intestinal Surgeons of Great Britain and Ireland (www.augis.org) The Brazilian Gastric Cancer Association (abcg.org.br) The Canadian Gastric Cancer Association (www.gastriccancer.ca) The Chinese Gastric Cancer Association The Dutch Upper GI Cancer Group (ducg.nl) The GASTRODATA group (www.gastrodata.org) Italian Research Group for Gastric Cancer (gircg.it) The Korean Gastric Cancer Association (kgca-i.or.kr) Oesophago-Gastric Surgery Section of the Asociación Española de Cirujanos – Spain (aecirujanos.es) Upper GI International Robotic Association (www.ugira.org) United Kingdom Oncology Nursing Society (www.ukons.org.uk) The European Oncology Nursing Society (cancernurse.eu) The Oesophageal Patients Association – United Kingdom (www.opa.org.uk) My Gut Feeling – Canada (mygutfeeling.ca) No Stomach for Cancer – USA (nostomachforcancer.org) Vivere Senza Stomaco - Italy (viveresenzastomaco.org) Gastro/Oesophageal Support and Help Cancer Group (Bristol) – United Kingdom GASTROS International Working Group Collaborators (To be cited as collaborators in PUBMED as previously agreed by the study management team) Shuangxi Li Yu-long He Zekuan Xu Yingwei Xue Han Liang Guoxin Li Enhao Zhao Philipp Neumann Linda O'Neill Emer Guinan Gian Luca Baiocchi Giovanni de Manzoni Eliza R.C. Hagens Mark I. van Berge Henegouwen Patrícia Lages 26 Susana Onofre Gabriel Salcedo Cabañas Maria Posada Gonzalez Cristina Marin Campos Bahar Candas Bahadır Emre Baki Muhammed Selim Bodur Reyyan Yildirim Arif Burak Cekic Jean-Baptiste Beuscart Sophie Horbach Christopher Mecoli Toby O Smith Author Contributions Author contributions to the study - https://www.casrai.org/credit.html. # Role Definition 1 Conceptualization Ideas; formulation or evolution of overarching research goals and aims. 2 Data curation Management activities to annotate (produce metadata), scrub data and maintain re data (including software code, where it is necessary for interpreting the data itself) initial use and later re-use. 3 Formal analysis Application of statistical, mathematical, computational, or other formal techniques analyse or synthesize study data. 4 Funding acquisition Acquisition of the financial support for the project leading to this publication. 5 Investigation Conducting a research and investigation process, specifically performing the exper or data/evidence collection. 6 Methodology Development or design of methodology; creation of models. 7 Project administration Management and coordination responsibility for the research activity planning and execution. 27 8 Resources Provision of study materials, reagents, materials, patients, laboratory samples, anim instrumentation, computing resources, or other analysis tools. 9 Software Programming, software development; designing computer programs; implementati computer code and supporting algorithms; testing of existing code components. 10 Supervision Oversight and leadership responsibility for the research activity planning and execu including mentorship external to the core team. 11 Validation Verification, whether as a part of the activity or separate, of the overall replication/reproducibility of results/experiments and other research outputs. 12 Visualization Preparation, creation and/or presentation of the published work, specifically visualization/data presentation. 13 Writing – original draft Preparation, creation and/or presentation of the published work, specifically writing initial draft (including substantive translation). 14 Writing – review & Preparation, creation and/or presentation of the published work by those from the o editing research group, specifically critical review, commentary or revision – including prepublication stages. References 1. Williamson PR, Altman DG, Bagley H, et al. The COMET Handbook: version 1.0. Trials. 2017;18(S3):280. doi:10.1186/s13063-017-1978-4 2. Gargon E, Gorst SL, Harman NL, Smith V, Matvienko-Sikar K, Williamson PR. Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research. Gillies K, ed. PLoS One. 2018;13(12):e0209869. doi:10.1371/journal.pone.0209869 3. Alkhaffaf B, Blazeby JM, Williamson PR, Bruce IA, Glenny A-M. Reporting of outcomes 28 in gastric cancer surgery trials: a systematic review. BMJ Open. 2018;8(10):e021796. doi:10.1136/BMJOPEN-2018-021796 4. Wild D, Grove A, Martin M, et al. Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported Outcomes (PRO) Measures: Report of the ISPOR Task Force for Translation and Cultural Adaptation Background and Rationale. https://www.ispor.org/workpaper/research_practices/PROTranslation_Adaptation.pdf. Accessed June 17, 2018. 5. Alkhaffaf B, Glenny A-M, Blazeby JM, Williamson P, Bruce IA. Standardising the Reporting of Outcomes in Gastric Cancer Surgery Trials: Protocol for the Development of a Core Outcome Set and Accompanying Outcome Measurement Instrument Set (The GASTROS Study). Trials. 2017;18(1):370. doi:10.1186/s13063017-2100-7 6. Alkhaffaf B, Blazeby JM, Bruce IA, Morris RL. Patient priorities in relation to surgery for gastric cancer: qualitative interviews with gastric cancer surgery patients to inform the development of a core outcome set. BMJ Open. 2020;10(2):e034782. doi:10.1136/bmjopen-2019-034782 7. Core Outcome Measures in Effectiveness Trials Initiative (COMET). COMET Database. http://cometinitiative.org/studies/search. Accessed January 8, 2019. 8. Gargon E, Gorst SL, Harman NL, Smith V, Matvienko-Sikar K, Williamson PR. Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research. Gillies K, ed. PLoS One. 2018;13(12):e0209869. doi:10.1371/journal.pone.0209869 9. Ann Arbor, MI: Survey Research Center, Institute for Social Research U of M. Guidelines for Best Practice in Cross-Cultural Surveys. 29 http://www.ccsg.isr.umich.edu/images/PDFs/CCSG_Full_Guidelines_2016_Version.pdf. Accessed September 15, 2019. 10. OMERACT: Outcome Measures in Rheumatology. Handbook | OMERACT Handbook. https://omeracthandbook.org/handbook. Accessed October 5, 2019. 11. Smith TO, Hawker GA, Hunter DJ, et al. The OMERACT-OARSI Core Domain Set for Measurement in Clinical Trials of Hip and/or Knee Osteoarthritis. J Rheumatol. 2019;46(8):981-989. doi:10.3899/JRHEUM.181194 12. Beuscart J-B, Knol W, Cullinan S, et al. International core outcome set for clinical trials of medication review in multi-morbid older patients with polypharmacy. BMC Med. 2018;16(1):21. doi:10.1186/s12916-018-1007-9 13. Park JK, Mecoli CA, Alexanderson H, et al. Advancing the Development of Patientreported Outcomes for Adult Myositis at OMERACT 2016: An International Delphi Study. J Rheumatol. 2017;44(11):1683-1687. doi:10.3899/jrheum.161252 14. Horbach SER, van der Horst CMAM, Blei F, et al. Development of an international core outcome set for peripheral vascular malformations: the OVAMA project. Br J Dermatol. 2018;178(2):473-481. doi:10.1111/bjd.16029 15. Dodd S, Clarke M, Becker L, Mavergames C, Fish R, Williamson PR. A taxonomy has been developed for outcomes in medical research to help improve knowledge discovery. J Clin Epidemiol. 2018;96:84-92. doi:10.1016/j.jclinepi.2017.12.020 16. Baiocchi GL, Giacopuzzi S, Marrelli D, et al. International consensus on a complications list after gastrectomy for cancer. Gastric Cancer. 2019;22(1):172-189. doi:10.1007/s10120-018-0839-5 17. COMET Initiative - YouTube. https://www.youtube.com/channel/UCxkjZgL12_QXAzn8cO2nsg?&ab_channel=COMETInitiative. Accessed September 7, 2019. 18. Fish R, Sanders C, Adams R, et al. A core outcome set for clinical trials of 30 chemoradiotherapy interventions for anal cancer (CORMAC): a patient and healthcare professional consensus. lancet Gastroenterol Hepatol. 2018;3(12):865-873. doi:10.1016/S2468-1253(18)30264-4 19. What are the top 200 most spoken languages? | Ethnologue. https://www.ethnologue.com/guides/ethnologue200. Accessed October 12, 2019. 20. Kuliś D, Whittaker C, Greimel E, Bottomley A, Koller M. Reviewing back translation reports of questionnaires: the EORTC conceptual framework and experience. Expert Rev Pharmacoeconomics Outcomes Res. 2017;17(6):523-530. doi:10.1080/14737167.2017.1384316 Tables Table 1. Studies using multi-language Delphi surveys in the development of international COS. 31 Condition/Group Original Language Target Language(s) Total Pa Hip and Knee Osteoarthritis English Italian & Spanish 426 French Dutch, German, English 150 English Swedish, Dutch & Korean 500 English Dutch 301 English Chinese, Dutch, German, Italian, 1021 OMERACT-OARSI11 Medication review in multi-morbid older patients with polypharmacy OPERAM12 Idiopathic inflammatory myopathy OMERACT13 Vascular Malformations OVAMA Group14 GASTROS Study Portuguese, Spanish, Turkish, Table 2. Timeline related considerations in undertaking multi-language Delphi survey. 32 Language Version* Translation 1 Docume Time to return Harmonization Time to set Time to pilot Time nt completed across language up online survey and obta Prepara translations versions surveys complete appr tion for r1 2 weeks 6 weeks amendments 2 weeks 1 week 1 week 25 we Translation 2 10 weeks 1 week 29 we Translation 3 3 weeks 1 week Not r Translation 4 10 weeks 1 week Not r Translation 18 weeks 1 week ***No recei Translation 6 12 weeks 3 weeks 40 W Translation 7 2 weeks 1 week 2 wee * The language versions are anonymized. **This represented the time the study management group requested collaborators to begin ethical approval applications until IRB approval was received and not necessarily the time between actual submission of the application and receiving approvals. ***Ethical approval was not received before the end of round 1 of the Delphi survey. No patients were recruited from this team’s country. Figures 33 Figure 1 Flow diagram demonstrating which studies were included in the structured review. 34 Figure 2 a) Characteristics of translators undertaking forward translation(s). b) Characteristics of translators undertaking backward translation(s). 35 Figure 3 Step by step translation process for multi-language Delphi surveys Figure 4 Cumulative weekly recruitment figures for round 1 of GASTROS Delphi survey 36