Translating ethnographic data into knowledge, skills, and attitude statements for medical scribes: a modified Delphi approach

(1) Conducted field study with interviews and observations (October 2017–January 2019)

To better understand the scribe role, we conducted ethnographic site visits using standard purposive sampling to identify suitable organizations across the United States and gathered data through interviews and observations about scribe activities in hospital, emergency department, and ambulatory settings.^20,21 These organizations represented the scope of scribe activities in disparate geographic settings with varied workflows as well as various scribing models (in-person, preprofessional, professional, and remote). See Table 1 for a breakdown of the varied attributes of each site and description of scribing models. Our research team was interdisciplinary, representing different perspectives and backgrounds, including social scientists, informaticians, providers, qualitative researchers, and scribe managers.

(2) Conducted fieldwork data analysis and created draft KSAs (February 2019–April 2019)

We transcribed and analyzed qualitative data from the site visits to inform the next stage of the study, which was convening an expert consensus conference. Three team members (SC, NS, and JA) extricated an initial list of attributes that were mentioned in the data as desirable KSAs for scribes to possess. Data were analyzed using an inductive grounded theory approach, meaning that we did not start with a list of preconceived terms to use while coding the data. Codes arose directly from the data; themes were generated after patterns were analyzed. The competency-related texts were further analyzed to identify subthemes. For example, if a theme had 50 text items categorized under that theme, those 50 items were coded again to identify patterns within that theme. During another exercise, the research team reviewed all coding and sorted individual items into 2 broad lists: those that were clearly EHR related and those that were not EHR related.

(3) Held expert consensus conference (April 2019)

In the consensus development phase, we invited a panel of subject matter experts (SMEs) to review the 2 lists, with the goal of reaching a consensus regarding optimal scribe-related KSAs. A two-day face-to-face in-person conference of SMEs was held to discuss, evaluate, and prioritize the fieldwork results, and provide input into the KSAs. Having previously utilized an expert consensus conference format during another study successfully,^22,23 we felt that this approach would benefit the process of achieving SME consensus about KSAs. The end goal of the conference was to produce a prioritized list of KSAs that could later be refined to inform a training toolkit for scribes and organizations to use.

The research team purposively selected and invited SMEs who were prominent in their professional domains within informatics, the scribe industry, patient safety, risk management, and medical education. We also invited scribe users (providers) and former scribes to gain the perspective of those on the front lines of clinical care delivery. A list of expert guests and their expertise as of 2019 is included in Supplementary Table S1. The group was presented with the qualitative study results and 5 breakout groups evaluated, clarified, and prioritized sections of the KSA list pertaining to the EHR. Each group was predetermined so that multiple perspectives were represented within each group. Participants then reconvened as a whole to review and refine the small group results. The same process was followed for evaluation of the non-EHR related KSA list.

(4) Research team reviewed and compiled data from conference (May 2019–June 2019)

The conference group sessions were recorded and audio transcripts were analyzed using an inductive approach and an open coding method for developing definitive KSA intent and wording. In addition to audio transcripts, the groups had generated flip charts during their discussions, and these were saved as artifacts and compared for analysis. The flip charts had bullet points of medical scribe KSAs that were discussed in each smaller group discussion. Each individual group then did a “dot” exercise, where they were given 5–10 dots and were told to put a dot next to the KSAs they felt were the most important ones discussed in the small group sessions to mark prioritization. Post conference, one research team member took the flipchart information and transformed it into Word documents. As part of the coding process, 8 team members (SC, JA, JB, JG, KW, BO, VM, and CH) worked in dyads to read and review transcripts and parse pertinent quotes regarding each KSA item. Dyads were chosen so at least 1 researcher that was present for each of the small group discussions was also a part of the dyad. The output from the dyads was correlated with the content of participant-generated flip charts to ensure continuity. We met as a team after dyad discussions to reach a team consensus on these lists.

(5–11) Created the Delphi survey (June 2019–May 2020)

We utilized a modified Delphi framework to iteratively develop the KSA lists in the months following the expert conference^24–26 (see Figure 1). A Delphi methodology is a structured approach to reaching agreement or consensus within a multidisciplinary group of SMEs, often across various geographical locations²⁵ and has been used in numerous disciplines, including medicine.^27–29 A Delphi approach typically consists of multiple iterations before a final consensus is achieved, thus producing both accurate and reliable information.²⁵

The first step involved the creation of the initial Delphi survey instrument. The surveys, developed using Qualtrics (Qualtrics, Seattle, WA), asked participants to denote each item as “Keep,” “Not Keep,” or “Change.” Branched-chain logic led to the next question. For example, if the experts clicked “Not Keep” or “Change,” they were prompted to the next box where they could comment on each of their choices. Please see Supplementary Appendix A for a copy of one of our Delphi survey instruments.

After piloting the survey with 3 informatics faculty members and graduate students, we sent the first round to the SMEs, and made edits based on the first round of initial feedback. Then, we iteratively deployed a second round of the Delphi survey and made edits to KSAs based on this iteration of feedback from the second round.

With the final lists of EHR-related and non-EHR related KSAs available, the research team assessed each statement and classified it within a framework used by the Accreditation Council for Graduate Medical Education (ACGME), which is universally utilized in the education and evaluation of house officers^30–33 in the United States. This framework has also been validated as a template for qualitative data analysis and crafting competency statements.^32–35 The ACGME framework is a standard format for framing medical competencies, and we chose to put the KSAs into this format for consistency purposes. The final iteration of both KSA lists was organized to conform to the standard ACGME format of core competencies i.e.: (1) patient care, (2) medical knowledge, (3) practice-based learning and improvement, (4) interpersonal and communication skills, (5) professionalism, and (6) system-based practice.³⁵ However, since medical scribes do not engage in direct patient care, the “patient care” category was replaced with a new category called “foundational skills.” After the research team completed the classification process, the KSAs were sent to the SMEs for a final review and there were no further comments or suggested changes.

(12) Developed the curriculum (February 2020–April 2020)

We needed to organize the KSAs in such a way that they could be taught. Therefore, the research team next assessed each KSA item for appropriateness for either simulation or didactic (lecture-based) training. We met as a team virtually multiple times to review our KSA items one by one and determine how we could best measure this KSA through simulation or through didactic lectures. We thoroughly discussed each item until a consensus was reached about how we would assess this item. Simulation is an ideal delivery method for allowing students to be taught effectively with controlled hands-on training.^36–38 Simulations allow for repeated iterations, so participants can run clinical scenarios numerous times. Simulations also allow participants to practice in a safe environment where no real patients can be harmed.^36,39–41

There are some types of knowledge, skills, and attitudes, that do not lend themselves to simulation, but can be best be presented in didactic lectures, which comprise the non-EHR KSAs lists. Didactic lectures are a common methodology for teaching medicine.^42,43

Hands-On Learning KSAs

Elements within the Hands-On Learning KSA list can best be taught using simulation-based activities. The research team divided this list using the ACGME categorization as well as an adapted version of the subcategorization strategy we created in Table 2. Please see Table 4, which describes the ACGME categorization and subcategorizations used for the Hands-On Learning list.

The first set of KSAs within this list focused on medical knowledge, including learning medical terminology, the spelling of clinically relevant words, common medication dosages, and abbreviations approved by the scribe’s organization. The second KSA category was practice-based learning and improvement, which involved scribe familiarity with organization and department-specific documentation standards (note types, sections, formatting, etc.). The third KSA category was that of interpersonal communication skills. Elements within this category included the need for scribes to be able to synthesize information from all sources during the patient-provider encounter, including verbal and nonverbal cues from both provider and patient. The fourth KSA category was professionalism, involving the need for scribes to be formally trained regarding concepts associated with compliance, HIPAA, patient privacy, and security. For the system-based practice KSA items, scribes need to be aware of organizational expectations and the limitations of their role and have the ability to focus and pay attention in all clinical environments. Finally, some EHR KSAs were background competencies on which the performance of many of those KSAs depended. For these foundational KSAs, scribes need to be proficient with basic technical skills such as typing with speed and accuracy, using a cogent writing style, and completing common computer-based tasks. Scribes also need to have expertise navigating the EHR. It is critical that a scribe’s documentation is consistent with standards and the style set forth by the organization and provider. For a full list of Hands-On Learning KSAs, please see Supplementary Appendix B.

Didactic KSAs

Elements within the Didactic KSA list can best be taught using lectures. For the final Didactic KSA list, the research team divided this list using the ACGME categorization as well as an adapted subcategorization strategy. Please see Table 5, which describes the ACGME categorization and subcategorizations used for the Didactic list.

The KSA elements within this didactically oriented list included specialty-specific knowledge such as the types of patients being seen as well as common medical issues for the demographics of their providers’ patients. Didactic KSA elements within the domain of practice-based learning and improvement primarily focused on the relationship between documentation and coding. For example, Didactic KSAs in the interpersonal and communication skills domain highlighted the need for scribes to possess the ability to collaborate in a team environment and seek clarification when necessary. Didactic KSAs in the professional domain emphasized the need for scribes to show respect for patients, exhibit emotional regulation, be honest with the health care team, and be flexible. In addition, the systems-based practice domain included the need for scribes to prepare themselves appropriately prior to patient visits and understand their scope of practice. For the entire list of Didactic KSAs, please see Supplementary Appendix C.

Prerequisite KSAs

With guidance from SMEs, the research team defined a set of foundational prerequisite KSAs, i.e. KSAs that scribes should possess very soon after hiring, or even prior to being hired. The research team determined that this would be best taught through a series of didactic lectures.

Please see Table 6, which describes the ACGME categorization and the team’s adapted subcategorizations used for the Prerequisite KSA list.

Practice based learning and improvement-related Prerequisite KSAs include the need for scribes to possess mental stamina and demonstrate an eagerness and passion to learn. Interpersonal communication-related Prerequisite KSAs involve the need for scribes to demonstrate patience and compassion when dealing with patients and equanimity when communicating with other members of their care delivery team. Professionalism-related Prerequisite KSAs highlight the need for scribes to have on-boarding training to learn about organizational rules as soon as possible after being hired. System-based practice-related Prerequisite KSAs include the importance of the scribe participating in activities that improve efficiency and quality of the clinical care that is being delivered. Scribes should also receive organization-specific training on how to perform their roles and know how to report safety and quality concerns. For a full list of Prerequisite KSAs, please see Supplementary Appendix D.