Why, when, and from whom: considerations for collecting and reporting race and ethnicity data in HCI

Racial categories have been included in every U.S. census since 1790, and the history of the U.S. census reveals the complexity of racial and ethnic data collection [88]. Firstly, prior to the 1960 census, an individual’s race was determined by census enumerators (i.e., professionals who are hired to visit and survey residents to compile data for the U.S. census), rather than through self-report. Moreover, categories used in the census changed almost every decade to reflect the politics and societal values at the time. For instance, “Native Hawaiian or Other Pacific Islander” was historically grouped with Asians and only became a new category in 2000. As another example, “Mexicans” were counted as a racial category in 1930, but that category has since disappeared. Since then, many of Mexican descents resort to the “other race” option and have been grouped under the Hispanic ethnicity only. The ability to identify as multi-racial (i.e. “mark one or more racial categories”) was only won through extensive advocacy in 2000 [121]. Racial categorization has always been extremely political, and the miscategorization and undercount of people from racial minority groups have contributed to systemic oppression and exclusions [5]. In particular, these examples from the U.S. census illustrate that racial and ethnic data collection does not automatically lead to racial and ethnic equality. While the Census Bureau now collects racial and ethnic information to “make policy decisions for civil rights, to promote equal employment opportunities, and to assess racial disparities in health and environmental risks” [113], such data was in fact exploited by the government to prosecute and oppress racial minorities in the past [13].

U.S.-based researchers may also be familiar with the standards published by the U.S. Office of Management and Budget (OMB) in 1997, which mandates minimum standards for collecting and presenting data on race and ethnicity to support data analysis across different racial and ethnic groups. The OMB standards have two categories for ethnicity (Hispanic versus Non-Hispanic) and five categories for racial data at a minimum — White, Black or African American, American Indian or Alaska Native, Asian, and Native Hawaiian or Other Pacific Islander. The OMB standards have been the guideline for collecting and presenting data on race and ethnicity for all federal reporting, including the decennial census and the mandates by certain U.S. grant funding agencies such as the National Institutes of Health (NIH) and the National Science Foundation (NSF) [79, 80]. Despite its general success in presenting data on race and ethnicity in federal programs, the OMB standards are far from a “gold standard”, and performed especially poor on historically marginalized groups and/or multi-racial populations [62, 63].

From the history of racial categories in the U.S., we see that racial and ethnic data not only reflects the present social and political environment but also directly influences the future socialization and construction of the concept of race and ethnicity. Therefore, we conclude this section by emphasizing that the goal of collecting and reporting racial and ethnic data is not to arrive at a list of “perfect” racial and ethnic classifications, but more importantly, to leverage such data to reflect, acknowledge, and reject racist and oppressive power structures in current research.

In this section, we look at how the social sciences, medical sciences, and computer sciences have collected and analyzed racial data as comparative case studies.

Social science scholars have long acknowledged the role of race in shaping individuals’ social status and everyday life experience [9, 39, 39, 69]. However, there is less consensus on whether the field should use racial classifications to assess the role and consequences of race. Some argue that collecting and reporting data on race and ethnicity would promote racial division and further the status quo of racial discrimination. In contrast, others take a “what we cannot measure, we cannot understand” approach and continue to report observed racial differences from profiling in law enforcement to disparity in healthcare systems [9, 14]. In 2003, the American Sociological Association (ASA) issued a statement in support of the continual collection and research of data on race [4]. Their reasoning is summarized as follows: (i) racial identities are central to the social organization and relationships and, therefore, the very core of social science research; (ii) taking a “colour-blind" approach and ignoring participants’ race in research does not eliminate the use of racial categories and racism in everyday life, as well as the resulting impact on societal outcomes; and (iii) understanding the role of race is central to challenging the existing systems of racial discrimination and stratification.

In contrast to the debates over racial data collection in social sciences, race and ethnicity of study participants are widely collected and used in healthcare databases to ascertain important group-level differences in healthcare outcomes in the U.S. [64, 86]. The basis of the observed race-associated differences in healthcare outcomes, however, remains under-explored, further illustrating that data collection itself does not automatically translate into racial and ethnic equality. One exception is Jones [59], who argued that as a social construct, race only serves as a very rough proxy for variables of interest such as social class and culture. Instead, race often appears predictive of healthcare outcomes because of the racism that has operated throughout U.S. history and to date. As an example, an analysis by Jones et al. [60] demonstrated that being classified by others as “White” is associated with better health status, regardless of one’s self-identification. In view of the complex interpretation of “race”, multiple threads of work have urged researchers in public health to take an interest in elucidating the underlying causes of the observed differences across race and ethnicity groups, e.g., by generating hypotheses about the basis and designing data collection and analysis plans to test the hypotheses. For instance, the difference in rates of estrogen-receptor-negative breast cancer between Black and White women in the U.S. is well-documented. Building on this observation, Krieger et al. [67] demonstrated that being born in the states that practiced Jim Crow laws (i.e., legal racial segregation) is associated with higher odds of cancer, thereby attributing the observed differences to racially discriminating laws.

In computer science, fair machine learning is one of many research areas that share similar interests in analyzing racial and ethnic data. With the growing role of predictive algorithms in critical fields such as credit reporting and employment assessment, a plethora of work has investigated the fairness of the algorithmic decisions across “sensitive attributes” such as different racial and ethnic groups [8, 11, 57]. Even though contextual discriminatory impacts of algorithms, such as the disparity between White versus Black Americans in the criminal justice system, are often cited as motivating examples for this line of research, researchers have largely treated racial or gender groups as an abstract feature of individuals. A few recent exceptions include Benthall and Haynes [8], Hanna et al. [42], and Hu and Kohler-Hausmann [50]. Benthall and Haynes [8] argued that people who are labelled as “Black" in the U.S. are subject to systemic differences through spatial, political, and social segregation. Therefore, the use of race categories such as “Black" might risk reifying racialized social inequality, if the observed differences are attributed to the race instead of to the underpinning systemic inequalities. Hence, Benthall and Haynes [8] proposed replacing racial categories with group labels learned through data to dynamically capture the underpinning inequalities, without furthering the status quo of disadvantaged racial groups. On the other hand, Hu and Kohler-Hausmann [50] investigated the social meaning of racial and gender membership. Their key insight is that prevailing analyses of socially salient categories such as race neglect the fact that many of the “effects" (e.g., career choices, family, and neighbourhood wealth) attributed to race are, in fact, its constitutive features. They implored for more care in conceptualizing and interpreting the “causal effects” of race. Finally, Hanna et al. [42] argued that most existing research ignores the “multi-dimensionality” of race and instead treated racial and ethnic categories as a fixed attribute. They urged researchers in the field to contextualize the meaning of race, and focus on the underlying mechanism that produces and reinforces racial inequality.

Within HCI, several authors have argued, through qualitative, mixed-methods, or quantitative methods, that there is a general lack of meaningful engagement with race and ethnicity [29, 47, 83, 91, 93, 104]. However, the current practice, as well as the motivations, for collecting and reporting study participants’ race and ethnicity remains under-explored. In this work, we analyzed recent CHI proceedings to understand the existing practice. Furthermore, we also surveyed authors to identify the motivations and methods for collecting racial and ethnic data of their participants. Motivated by parallel efforts on the gender of study participants [82, 102, 108], we drew from related disciplines and outlined considerations for HCI researchers when it comes to racial data collection.

We started with a total of 3,910 research articles published in 2016-2021 CHI proceedings on ACM digital library and proceeded with a keyword-search method informed by prior work [47, 104]. We initially filtered the collection using the keywords “race” OR “racial” OR “ethnicity”, which narrowed down the corpus to 663 articles. We then experimented with adding potentially defining keywords (race/racial, ethnicity/ethnic, White/Caucasian, African American/Black, Asian, Hispanic, Native American/American Indian, Pacific Islanders), and sampled the first 10 articles returned in each year to judge the quality of our search (the quality here is defined to be the number of search results that contained detailed race and ethnicity information of the study participants). The final keyword set that yielded the most relevant articles empirically is (i) “race” OR “racial” OR “ethnicity” AND (ii) “Hispanic” OR “Black American” OR “Asian” OR “Caucasian”. After identifying this initial collection of 340 articles, we then checked for the racial and ethnic composition of study participants in each article manually. Next, we aggregated the racial and ethnic information of study participants over all articles that detailed this information. Because of the focus on race and ethnicity in the U.S. context for this work, we further limit the corpus to the articles for which the authors had U.S. affiliations or the participants were recruited in the U.S. The dataset curation process is displayed in Figure 1.

For reference, we obtained racial and ethnic composition from two additional sources: (i) 2015–2019 demographics estimates of the U.S. collected by the United States Census Bureau [114]; and (ii) 2015–2019 demographics estimates of U.S.-based drug trials collected by the U.S. Food and Drug Administration (FDA) [12].

As stated in Section 2.1, categorizing race and ethnicity is extremely complex. In this paper, for simplicity, we adopted the following procedures to group the race and ethnicity categories across different studies. Firstly, since most of the surveyed CHI studies collected ethnicity and race using one single question for race and ethnicity, we make the assumption that the racial categories reported by studies in our final corpus refer to the corresponding Non-Hispanic subset (e.g., reported White participants in a paper refers to the non-Hispanic White participants). Second of all, we aggregated the participants into the following categories that roughly align with the OMB standards: White, Black or African American, Asian, and Others (including American Indian and Alaska Native, Mixed races, Native Hawaiian and Other Pacific Islander). This choice of analysis is largely driven by the existing racial categories in papers published in CHI, which certainly does not capture the full complexity of race and ethnicity of the study participants.

In addition to the aggregate analysis of studies in the final corpus, we will also report the following summary statistics: (i) the number of studies that collected ethnicity separately from race, and (ii) the racial and ethnic breakdown of large (> the median number participants across all studies in the final corpus) studies and small-to-medium-scale studies (≤ the median number of participants across all studies in the final corpus).

While all papers in our final corpus reported their participants’ race and ethnicity, many left the reason and process of collecting the race and ethnicity of their participants implicit. We, therefore, conducted an additional survey to find out why researchers collect racial and ethnic data and to learn about potential challenges they may have experienced. For each publication in the final curated corpus, we emailed the first and senior authors with the following list of open-ended questions on why and how they collected the race and ethnicity information of their participants.

After excluding those authors with an inactive email address (e.g., due to a change of affiliation), we emailed a total of 106 authors (counting both first and last authors) whose papers were in our final corpus. Among those, 15 (14.2%) participated in our survey, all of whom were currently affiliated with a U.S. institution or corporation.

In addition to the survey responses, we note that some papers already highlighted the importance of considering race and ethnicity for the piece of technology under study. For instance, Passmore et al. [87] surveyed gamers from diverse racial and ethnic backgrounds and established “significant differences between players of color and White players on the perception of racial norms in gaming, effects of behavior, emotions, player satisfaction, engagement, and beliefs stemming from a lack of diversity.” Moreover, they emphasized that the diverse recruitment amounted to “higher dissatisfaction [in diversity in digital games] than previous research.”

At the time of data collection, the ACM digital library includes 3,910 CHI papers published in 2016–2021 CHI proceedings. We analyzed 340 manuscripts that mentioned keywords related to race and ethnicity (see Figure 1 for details), of which only 93 provided descriptive statistics on the racial and ethnic breakdown of their study participants. In other words, our analysis showed that only 93 (2.4%) of 3,910 CHI papers included descriptive information on participants’ race and ethnicity. This is likely an undercount given that the final corpus only included studies with the specified keywords.

Out of the 93 manuscripts, the median number of reported racial and ethnic groups is 4 (IQR: 3–5). Only a small number (17; 18.2%) of studies mentioned (or was inferred of) using two separate questions for race and ethnicity. The median number of participants in the studies in the final corpus is 28 (IQR: 18–187); the largest study reported the racial and ethnic breakdown of 2,041 participants [115], and the smallest study in our corpus had only six participants [40]. In addition, almost all (>90%) of the authors in our final corpus were affiliated with a U.S. institution at the time of writing. The number of manuscripts is also not evenly distributed across time, with more than 43 out of 93 (46%) published in CHI 2021, followed by 14 in 2017, 11 in 2019, 11 in 2018, 9 in 2020, and 5 in 2016.

The aggregated studies reported 19,684 participants in total, 12,627 (64.1%) of whom are (Non-Hispanic) White; 2,028 (10.3%) Black; 1,766 (8.9%) Hispanic; 1,327 (6.7%) Asian; and 1,939 (4.6%) Others (with 205 Mixed races and 98 American Indian or Alaska Natives). By contrast, according to the estimated demographic data by the U.S. census for 2015–2019, 60.7% of the population in the U.S. is (Non-Hispanic) White; 12.3% Black; 18% Hispanic; 5.5% Asian; and 3.5% Others (2.4% Mixed and 0.7% American Indian or Alaska Natives). Regarding the U.S.-based FDA drug trials during 2015 and 2019, (Non-Hispanic) White accounted for 64.5% of the participants, followed by 16% for Black and 15% for Hispanics. Asians and Other groups account for 2% and 3.5% of the trial participants, respectively.

Racial and ethnic compositions from the three different sources (CHI, U.S. Census, and the FDA drug trials) are displayed in Figure 2. We see that compared to the U.S. Census, CHI studies in our final corpus have slightly more Non-Hispanic Whites and slightly fewer Hispanics. In addition, Figure 2 suggests that participants in neither CHI studies nor FDA trials are representative of the aggregated U.S. demographics. However, we note that many CHI studies actively recruited a representative sample of their interest, which may or may not agree with the aggregated demographics of the U.S. For instance, Lopez et al. [74] was a Non-Hispanic-White-focused study, and Dosono and Semaan [24] specifically looked at the engagement and dynamics of the Asian American and Pacific Islander online communities. As a result, both studies will not resemble the U.S. demographics by design, rather than by omission.

We also looked at the longitudinal trend of compositions of reported racial and ethnic groups across the six years of CHI proceedings. Overall, the racial and ethnic compositions of study participants appear stable over the course of six years, with a more noticeable increase of Non-White participants from 2020 onwards. Figure 3 displays the racial and ethnic composition over the six years, stratified by the size of the study, where a study is classified as “large” if it has more than 28 (i.e., the median number of participants across all studies) participants and “small-to-medium” otherwise. We see that large-scale studies tend to have more White participants. This is partly due to the use of online platforms (e.g., Twitter or Mechanical Turk) for participant recruitment, which has been known to skew towards White samples [70, 117]. On the other hand, small-to-medium studies are more likely to target specific populations of interest (e.g., studying particular technology of interest in low-income neighbourhoods or among Black females; see, e.g., Ogbonnaya-Ogburu et al. [83] and Wheeler and Dillahunt [120]). As a result, small-to-medium studies might appear to have a larger proportion of non-White participants than larger studies.

Responses from authors who participated in our open-ended survey are summarized in Table 1. Because the authors could cite multiple reasons in their open-ended responses, occurrences of the summarized categories in Table 1 may add up to over 15.

The most common reason (“Why” in Table 1) for collecting and reporting racial and ethnic information is external validity, that is, the degree to which the conclusion in one study would hold for other persons in other places and times [36]. For instance, one surveyed researcher noted in their response that “If my data is really only from a sample of white people, then I need to acknowledge that as a limitation of the study and ensure that my analysis is contextualized in that particular identity.” A majority of the surveyed researchers also named prior work (including a priori hypothesis on the differences across racial and ethnicity groups) as a driving factor. As an example, one respondent noted that “We were specifically interested in experiences of intra-community marginalization, which includes systemic biases such as racism, and we wanted to ensure that our sample could capture such dynamics.” In addition, the interplay between race, racism, and socioeconomic class in the U.S. motivates some researchers to “always collect these data” because when it comes to disparities as they affect technology, “race and income are so woefully correlated in this country [the U.S.] (and others) — it was important to collect racial/ethnic information.” Only two out of 15 responses mentioned “external requirement” as the primary reason for collecting and reporting participants’ race and ethnicity. Out of the two respondents, one stated that “I was in a Biomedical and Health Informatics program, and health studies often have people collect this data (perhaps tied to NIH funding/grant requirements)”; the other one mentioned that “our multi-year federal grant that funded this research had annual reporting requirements about, among other details, the demographics of our participants in each study”. Both responses speak to the possibility of leveraging practices and training from related disciplines, such as biomedical sciences, to improve the collection and report standards of race in HCI.

Regarding the method of collection (“How” in Table 1), all of the surveyed researchers administrated a questionnaire, but the details of the administration varied: The same number of authors used two separate questions (seven out of 15) versus one single question (seven out of 15) when obtaining the race and ethnicity information of their study participants. This is in contrary to the observation made from our systematic literature analysis, where less than 20% of the studies used two separate questions. Several researchers also used a combination of categorized and open-ended responses, where the study participants can provide their own race and ethnicity.

All responses mentioned existing resources that informed their data collection process (see the “Reference” column in Table 1). Among those responses, eight out of 15 cited the U.S. census as the primary reference for designing the categories in the questionnaires. However, researchers are aware of the limitations of U.S. census as a reference, “I found the categories somewhere — either NIH or census perhaps but unfortunately not sure. I know that I tried to find existing categories that the government recommended as I thought these would be the absolute best practice/best way of going about this and then later learned these may not tie best to how people identify themselves.”

Another response mentioned categories informed by the social sciences, partly due to their expertise in race and ethnicity research — “we reviewed a number of articles from top sociology journals in the years immediately prior to the data being collected (e.g., ASR [American Sociological Review], AJS [American Journal of Sociology]), as sociologists tend to think more about these issues, and also use representative survey data, than communication scholars.” In addition to the categories informed by the U.S. census, a few studies also identified a target population of interest via prior work, e.g., one respondent noted that “we looked at previous research in this area and consulted with our community partners who collect this information as part of their data collection and capacity building activities.”

Answering the question of whether to collect racial and ethnic data of study participants requires substantial care. On the one hand, researchers in HCI and many related scientific disciplines are trained and required to justify the collection and planned analysis of demographic variables such as race. Documented reasons for not collecting this data may be privacy concerns for the study participants [6], or preventing survey disengagement and fatigue [38, 51]. Moreover, some fear that collecting and reporting on the findings in different racial groups, when presented without a thorough and nuanced discussion, could reify the racial inequality and stereotype, and even delve into “scientific racism” (i.e., the pseudoscientific belief that empirical evidence justifies racial discrimination) [4]. When put together, these concerns result in recommendations such as “Researchers should not collect more information from participants than is needed to answer their research questions” by the Institutional Review Board (IRB), an institutional organization that approves, monitors, and reviews human-subject studies in U.S. academic research institutions [10]. By contrast, demographics including gender, age, and whether participants are in vulnerable groups (prisoners, minors) are routinely requested in human-subject studies applications to IRB. In practice, researchers often need to cite established differences in racial groups to justify their decisions. On the other hand, one cannot hope to find a documented “established difference” if no study is devoted to documenting and understanding potential differences of the piece of technology across racial groups. This “what we do not measure, we do not understand” argument is especially relevant, given that race has been understudied, and the historic impact of race (and racism) in the U.S. and many countries have been made invisible. For example, in the U.S., the NIH started mandating the recruiting and reporting of racial and ethnic minorities in all clinical trials after observing significant differences between racial groups across a wide range of health care outcomes [34, 65, 78]. While federal agencies in the U.S. (such as the NIH and the Census Bureau) have generally leaned towards collecting race and ethnicity of participants, the legality and regulations of the collection and analysis of racial and ethnic data differ greatly around the world. For instance, in the European Union (EU), the General Data Protection Regulation (GDPR) [18] mandates that “processing of personal data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs... shall be prohibited [in general].” Despite some listed exceptions, e.g., “archiving purposes in the public interest, scientific or historical research purposes or statistical purposes”, the legal and logistical challenges led to another important consideration for participants and researchers based in the EU (and other regions with similar laws and regulations).

Given the nuanced nature of this topic, we encourage HCI researchers to have an active discussion during the study design phase within the team. Some points of discussion (as framed through quotations from our survey respondents and findings from our review) may include:

Of course, given the nuanced nature of this question, we are not arguing to simply lower the barriers to collecting racial and ethnic data of participants for all HCI studies — as noted above, concerns for privacy and misinterpretation of the results should not be taken lightly. Instead, we advocate that whether or not to collect racial and ethnic data of participants should be a deliberate rather than a default decision.

Our findings show that researchers who have committed to collecting race and ethnicity of their participants desire a standardized method of collection and seek out templates and best practices. Because of our restriction to U.S.-based participants and research, the vast majority of researchers in our study used the U.S. census as their references when designing their surveys and questionnaires (see Table 1). A considerable subset of surveyed researchers also used the modified format to ask one combined question about race and ethnicity, rather than two separate questions. While this may be an effective starting place, the U.S. census is only conducted every 10 years, and its categories often represent the political values at the time rather than the best research practice [5, 69]. Therefore, there is a need for best practice that is research-driven and takes into account how participants actually want to be identified by race. In this section, we offer some specific, practical advice on how to collect participants’ race and ethnicity using surveys.

As an example, consider the collection of race and ethnicity in the U.S. census — census forms now have two separate questions about race (i.e., what is this person’s race) and ethnicity (i.e., whether this person is of Hispanic, Latino, or Spanish origin). This separation was largely motivated by the growing diversity within the Hispanic-American population. However, in the 2010 Census, 37% of surveyed Hispanic or Latinx chose not to identify with any of the provided race categories [52], which reveals the gap between the provided categories in the status quo and how people identify themselves.

While there is certainly no one-size-fits-all solution to collecting race and ethnicity of study participants, we hope the following pointers could be of help to researchers during data collection:

Even equipped with perfect survey questions, researchers are likely to run into a series of additional practical challenges — Who should be recruited for the study? Should we always strive for racially-representative samples? How to recruit and retain racial and ethnic minorities? While answers to these questions will be context- and study-specific, we briefly summarize some common motivations and barriers for recruiting racially- and ethnically-diverse or homogeneous participants in this section.

Recall that many of our surveyed researchers cited the external validity of their studies as a primary motivation for collecting participants’ race and ethnicity (see Table 1). Indeed, HCI researchers routinely draw inferences about populations by extrapolating findings based on data from small samples of people. Recent years have seen a surge of work on how studies in HCI, along with psychology, might have relied too much on samples from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) samples [7, 45, 73, 90] and among college students [41, 46, 90].

Therefore, we encourage researchers to assess the external validity, as it relates to participants’ race and ethnicity, of their specific studies. For instance, Obiorah et al. [81] proposed and evaluated a novel interactive multi-person tabletop exhibit in museums, and the race and ethnicity of participants they engaged “is reflective of typical museum visitors’ demographics in the U.S.”, which speaks to the applicability of their findings on the populations they are interested in. By contrast, Hamidi et al. [40] acknowledged the lack of diversity in their samples as a limitation and stated that “Our sample includes diverse age and gender perspectives but lacks diversity of race or ethnic identity [with respect to the U.S. demographics].”

Of course, researchers do not always need to recruit participants to match national (or regional) race and ethnicity composition — external validity should be assessed with respect to the population from whom we are drawing inferences. For instance, F. Maestre et al. [30] recruited a majority of White participants, in keeping with the demographics of the rural region of the U.S. Midwest. Other notable examples include studies with a focus on selected racial and ethnic groups: Lee and Rich [68] recruited primarily Black American participants because of the “substantial history and contemporary issues with medical racism towards the Black community”. Similarly, Dosono and Semaan [24] were interested in the dynamics of online Asian American and Pacific Islanders communities. In both cases, a lack of racial diversity does not necessarily pose threat to the external validity of the results.

We also want to acknowledge that relying on convenient (and, often WEIRD and non-racially-diverse) samples has its practical advantages, because recruiting and retaining racial and ethnic minorities presents logistical, financial, and sometimes even legal barriers [6, 15, 18, 85, 90]. Examining the effectiveness of sampling and recruitment strategies tailored to specific ethnic minority groups (e.g., F. Maestre et al. [30], Sadler et al. [100]) in HCI is an important thread of future work. In practice, the following questions summarized and adapted from IRB forms at various universities might be an effective starting point for researchers who want to reflect on their study design and sample collection [54, 55, 56, 97]:

While we focused on practical suggestions for collecting race in simplified settings such as surveys in this section, there is an array of important and exciting work dedicated to collecting complex, qualitative racial data through interviews, focus groups, and case studies [29, 83, 84, 93, 94, 120]. We encourage researchers to explore listed papers and references therein to engage with qualitative racial data collection.

HCI researchers might also find themselves analyzing race and ethnicity data of their participants, such as incorporating participants’ race in a regression model. While a well-conducted analysis will enrich our knowledge of the technology under study, such analyses, if done poorly, might give way to post hoc explanations that reinforce racial stereotypes and inequality [27, 32]. Below, we outline a few considerations for reporting and communicating the findings based on the race and ethnicity of study participants. These considerations are largely inspired and adapted from existing discussions in medical sciences [33, 59, 64], fair machine learning [6, 8, 50], HCI [29, 43, 93, 95, 102], and our own experience as researchers in the field.

Firstly, we encourage researchers to acknowledge that race is a social construct and proxy, rather than an objective measure of underlying traits [8, 50, 59] when presenting observed differences in racial groups. In particular, we caution against languages that suggest the differences in the measured outcomes can be attributed to participants’ race, a slippery slope to furthering the racial disparity and stereotypes. Instead, authors should provide qualifying statements when presenting their results across racial categories. As an example, in CHI 2021, K. Chua and Mazmanian [61] stated that “We recognize that the terms ‘Asian,’ ‘White,’ and ‘Hispanic’ are broad and homogenize the experiences of people from various races, ethnicities, and national backgrounds. We chose to use these terms because they are emic terms used by the vast majority of our participants in describing themselves.”

Second of all, we urge researchers to investigate the potential basis of the observed differences across racial and ethnic groups to the extent feasible, e.g., conducting a follow-up study or drawing from existing literature. For instance, when Dillahunt et al. [23] found that participants of different races had different levels of engagement and outcomes with online employment resources, they drew from the existing literature on Black-White income inequality [37, 118] and hypothesized that the observed differences across race could be attributed to the design of online resources websites with the “Us versus Them” thinking that marginalized the racial minorities.

Thirdly, discussions about the potential impact of the study should be sought after. This effort could take many forms: for example, the research artifact could include a paragraph on the societal impact of the research. Such practice would be widely applicable to research artifacts that propose new systems or software, paralleling efforts in the artificial intelligence community [2]. As an example, if a paper proposes a new piece of virtual reality technology, and researchers observe that the accuracies of the technology differ across racial groups, they should discuss the potential impact when the technology is deployed at a large scale. Similarly, for research outputs that incorporate machine learning models as part of their systems, potential disparate impacts on minority groups, as well as a toolkit for mitigating such ramifications in fair machine learning, could be discussed [6, 8].

Additionally, we want to call attention to the default practice of treating racial groups as categorical variables with “White” as the reference group (e.g., by classifying participants as “White” versus “Non-White”) in quantitative analyses in HCI. First of all, while easy to the implement, the grouping approach treats race as a non-overlapping attribute and ignores the heterogeneity within each racial and ethnic category [26, 42]. This could contribute to further reification of the groups under study, especially when the groups exhibit large internal differentiation such as the “Asian Americans and Pacific Islanders” category in the U.S. census [26]. As an alternative, we encourage quantitative analysis to account for the “multi-dimensionality” of race and carefully examine which dimension or representation is the most appropriate [42, 99]: for instance, researchers might adopt phenotypes such as skin type in investigating computer vision applications Buolamwini and Gebru [11]; or geography-based information such as those proposed in Benthall and Haynes [8] when looking at access to public resources. Furthermore, even when census-like racial categories are suitable, researchers should reflect whether they are defaulting to a culturally-dominant group (e.g., male, White, high income) as the reference could subtly establish and reify the notion that culturally-dominant groups are the most “normal” and “interesting”, thereby creating a nested system of importance over time [27, 32, 58]. Moreover, using the culturally dominant group as the reference does not always lead to the most interpretable results — for instance, if we are really interested in how different racial and ethnic groups compare against the average in the study population, coding the differences against a specific reference group is not the most informative approach (see more technical details and alternative modelling strategies in Dupree and Kraus [27] and references therein).

Last but not least, drawing from the unjust history of the U.S. census, as well as the thread of excellent work exposing the oppressive nature of current technology design in HCI [28, 43, 44, 94, 95, 103], we would like to emphasize that data collection and reporting is not sufficient in itself — researchers should always (i) consider narrating lived experiences of the racial and ethnic minorities; (ii) explore how oppression from race, gender, and class operates under the current technology design; and (iii) actively resist the current design and technology that perpetuates racism, whenever applicable.

In this paper, we primarily engage with race and ethnicity in the U.S. context. However, focusing on the North American experience does not fully embrace the issues of race and racism globally. Therefore, in this section, we describe how our analysis and reflections could shed light on how HCI researchers can consider why, when, and how to collect data about race and ethnicity globally: