Redesigning Privacy with User Feedback: The Case of Zoom Attendee Attention Tracking

Feedback Collection. The authors surveyed several social media platforms (including Reddit and Twitter) and online publications (including Hackernews [13] and Morning Consult [31]), resulting in 61 historical user comments referencing posts and threads on Zoom attendee attention tracking. Because this feedback was meant to generate ideas for feature requests for engineers, we only included potentially actionable feedback comments from the search, excluding general complaints and repeat comments from our analysis. Appendix Table 4 enumerates a breakdown of the user feedback sources we used. Zoom attendee attention tracking received more public critique during the onset of the COVID-19 pandemic, so the majority of the collected comments were from the spring of 2020. Given this feature was also deprecated over two years ago, availability of historical user feedback specifically about this feature was somewhat limited, but we did observe saturation with similar feedback before ending this collection. The two lead authors then independently conducted qualitative open coding analyses on these comments and discussed high-level categories for the resulting codes in order to choose representative comments.

Resulting Codes. After separately coding the 61 comments, we discovered our prevalent codes aligned with the Belmont Report’s ethical principles [29]. For instance, codes of “Loss of control/autonomy” and “Agency in usage” matched up with the “Respect for Persons” category, while the codes of “Risk of leaking personal data” and “Personal space/data usage” matched up with the “Beneficence” category. After deciding on these ethical principles as the new coding scheme, the authors re-coded the original feedback comments. To decide which comments to show to participants in our study, we selected representative comments that mostly centered around one principle over the others, 3-4 comments for each principle. While there were some positive comments lauding the feature’s usefulness in certain contexts, such as online education, the vast majority of comments were critiques of the feature’s shortcomings. For our representative comments we revealed to study participants, we focused on examples that were more critical, in order to more closely resemble presenting feature requests to our participants. Also, for these representative comments, we did not include minority comments that contradicted the majority comments. Table 1 displays the representative comments selected for each category.

Protocol. We conducted an online 60-minute semi-structured interview with each participant, where we started by asking if they were familiar with Zoom attendee attention tracking as a feature, introducing it if not. In our study, none of the participants were initially aware of its functionality. We clarified that Zoom later removed the feature and this study was meant to explore how this feature could have been different. We first asked participants to redesign the feature from the lens of a user, then introduced the representative comments from each category of user feedback (“Respect for Persons”, “Beneficence”, and “Justice”), one category at a time in a random order, asking the participant to redesign an improved feature addressing the request on top of their previous design (or explain why no change is necessary) (Figure 2). If time allowed, we also encouraged participants to sketch out their ideas, such as in the forms of user storyboards, information flow diagrams, or wireframes, to document how users would utilize the product feature and satisfy their privacy needs (see Figures 3, 4 for examples). In some cases, sketches were unnecessary, namely if explanations were not feature-specific or did not require illustration, for example if the participant mentioned deprecating the entire functionality. Some participants also preferred to verbally describe their ideas in more detail rather than sketching. After each redesign, the participant was also surveyed on how confidently they believed their current design addressed user needs and concerns, on a 5-point Likert scale. The guide for these semi-structured interviews is included in Appendix A.

= [thick,->,>=stealth]Recruitment. We recruited 18 software engineers through a snowball sampling strategy [32], where all three authors initially contacted software engineers from their professional and alumni networks then contacted eligible references from the initial participant set. While recruiting participants, we sampled for diversity in years of experience, company size, and privacy engineering experience. 3 participants (17%) identified as female, and the remaining identified as male. Most participants worked in large US-based companies. Participants had an average of 4.3 years of software work experience, and most participants (89%) had not worked with privacy frameworks directly as a collaborator. Two participants reported that their companies had separate security related software and legal teams, and most participants were aware of some approval steps related to privacy before deployment but didn’t actively consider privacy in their work, especially the more junior developers. While most participants had no experience working with privacy frameworks, some had indirect experience (if their product designs had security- or compliance-related considerations) and a couple directly collaborated on or led projects focused on privacy frameworks. Participants’ current employers were based in the United States and India and varied in size, but most participants (94%) worked at medium- (101-1000 people) to large-sized (over 1000 people) companies. Table 2 enumerates a breakdown of this participant information.

Interview Analysis. Utilizing recordings and transcripts of the interviews, two authors independently conducted inductive thematic analyses [10] on the data to identify patterns in participants’ responses and grouped those into higher-level descriptions and sub-groups (see Appendix Figure 6 for a general glimpse of our online coding process). We first analyzed the interview recordings to categorize participants’ design suggestions and coded those suggestions into high-level categories (see Table 3). We then dove into understanding the rationale behind each design suggestion and engaged in memo-writing, iterating on the codes to derive analytical themes.

Ethical Considerations. Our study was approved by our institution’s IRB. All participants verbally consented to having their data recorded and reported in a scholarly publication. Collected data was stored in a private location accessed only by the authors and anonymized during analysis and reporting. Email information was collected for the sole purpose of scheduling online interviews and distributing $20 USD compensation in the form of shopping gift cards and was deleted at the completion of the study.

User feedback as a checklist for edge cases. User privacy feedback allowed participants to cross-check their assumptions and anticipated concerns. This empowered participants to more confidently and efficiently validate their solutions and make clarifying adjustments in order to address the presented feedback similarly to a checklist. Participants’ reported confidences in their designs also generally increased after seeing user feedback and redesigning further (Figure 5). For example, P5 mentioned that “being able to compare [my design] against real user feedback made me realize it had accounted for a lot of the original feedback, so then it raised my confidence.” These responses were to the question, “How confident do you feel this product will satisfy user needs and concerns (on a scale of 1 being not at all confident to 5 being extremely confident)?” The difference in confidence between the distribution of responses on initial designs versus the distribution of responses after the third round of feedback redesigns is also statistically significant on a 95% confidence interval (α = 0.05) based on a Wilcoxon signed-rank test (p = 0.030).User feedback as evidence for high-level organizational privacy decisions. Participants recognized user feedback as a catalyst for product design changes at a higher organization level they might not normally be able to achieve on their own. As one participant mentioned, “It does give some more weight to have direct user feedback that goes along with [design decisions]. At least I feel that’s how it operates in [my company]” (P11). The same participant also mentioned the value of “some user studies with numbers, because people like numbers. [At my company] it would go a long way to have a more concrete percentage, more than anecdotes.” Other participants echoed similar sentiments, saying, “I would say those are pretty valid concerns. But do they really matter? I don’t know. So that’s the reason we want some metrics. We invent some quantitative metrics [...] We should listen to everyone but do the right things. And we need more data to do the right things” (P12) and, “In feature development, it’s worth just breaking out these slices of data and just seeing whether there’s a meaningful behavior difference before making decisions” (P3).

At the outset of the study, we anticipated a primary advantage of user privacy feedback to be the ability to enlighten engineers about user concerns and requirements they might not have initially foreseen and enable them to design scalable privacy solutions. After all, previous work identified that acknowledging the differences between developers’ perceived user privacy expectations and those actually expected by users was necessary for developers to successfully integrate privacy into software development [37]. However, we found that user privacy feedback in the form of raw text was unable to empower participants to devise feature updates which could scale for multiple user concerns. We observed a range of different design solutions (Figures 3, 4), but they were mostly focused around either minor, incremental interface changes (e.g. adding notice/consent) or complete deprecation of the feature.

Incremental interface changes. When participants saw user feedback they hadn’t originally anticipated, they were able to describe incremental technical solutions, usually focused on the front-end user interface. For example, many of these participants mentioned the best solution was to ensure that attendees understood the data being collected about them and that hosts understood the limitations of the data collected and reported (in a similar vein with purpose strings [12]). These were often in the form of notification interfaces for attendees describing the attention tracking process and “disclaimers” for hosts clarifying suggested usage and best practices for interpreting the reported data (see Figure 3a, 4a). As an example of messaging for hosts, P6 mentioned, “Instead of saying [attendees are] not paying attention, say they’re not watching their screen, they could still be listening to you.” In fact, transparency was the most-mentioned category of design suggestions (Table 3), with 16 participants (89%) bringing it up at least once

However, these bandage-like approaches often required many more patches to accommodate other feedback. This buildup of unanticipated changes inadvertently deprioritized usability in some cases. For instance, four participants proposed designs for meeting hosts to customize meeting expectations for different meeting types and different types of attendees (e.g. Figure 3b), but when asked to clarify specifics for the context of attention, a couple mentioned ideas for manually highlighting active times during the meeting to track attention only from certain types of attendee roles determined before the meeting. This type of customizability, while attempting to address a variety of concerns about accuracy in attention tracking, would certainly be a larger overhead for meeting hosts to configure before every meeting they wanted to track.

Participants who devised these patches for specific concerns also expressed lower confidence and some doubts that their solutions could handle all users’ edge cases; as P14 put it, “I still can’t completely foresee how it will be used [...] because of that, I’m not gonna give it a 5 [extremely confident].” P2 even commented, “I think the more and more I get educated on these corner cases, the more thornier it seems.” For these participants, it seemed exposure to novel concerns through the feedback humbled their original ideas and forced them to reconsider their privacy design strategies, while still limiting their solutions to patchwork fixes snowballing with front-end interface updates. Addressing concerns on an individual task level introduced issues of scale and usability.

Feature deprecation. The challenges of redesigning privacy with user feedback frequently drove participants to consider the opposite extreme solution, feature deprecation, echoing Zoom’s actual solution. Four participants consistently asked if they were allowed to deprecate the feature, or if the company was requiring them to maintain the feature. For instance, these participants asked, “Am I being pressured to keep this feature?” (P11) or mentioned “I don’t think I would implement that feature [...] unless I was forced to” (P7).

Another significant reason participants struggled to design a solution and considered abandoning the feature was the seemingly unsolvable conflict between their evident moral ground and the technical feasibility. Several participants expressed hesitancy to add more surveillance methods that “cross the line” for user privacy but struggled to design technical solutions to track attention as a result. For instance, participants mentioned, “The idea of surveillance in general tends not to be a great idea. And most people kinda just dislike that idea off the bat” (P1), “Despite all the attempts to frame it in a positive way, I don’t think people really like being tracked, just as a human nature kind of thing” (P5), and “I think that the act of tracking people inherently makes people suspicious, and I think people are naturally suspicious but have a right to be suspicious, and there will always be some amount of concern about how that data is being used, even if you demonstrate that it’s public and anonymized and aggregated” (P3). This concern led some participants to believe they couldn’t engineer a technical solution, for instance with some concluding, “I don’t know how you would be able to resolve this without even more invasive techniques” (P5), “I don’t think you’ll be able to capture [attention] digitally in any sort of digital platform or environment” (P1), and “I think anything that I would propose would be too invasive of privacy, and I would not be comfortable implementing that” (P7). It’s worth noting that several participants seemed to assume an inherent organizational or business pressure to keep the product feature. As P6 mentioned, “The fact that I built this feature in the first place meant that some other customer of mine asked for it, and if I don’t give them this feature, then they might just go somewhere else.”

Upon closer inspection of the participants’ reasons to deprecate the product, participants also seemed to pull in personal opinions and experiences outside of the displayed user feedback. Before seeing any feedback, P11 even noted, “I don’t think using it is good [...] it should just not exist. Because anything I can think of to, like, improve the quality of data is just more recording and analyzing. And they’re all very subjective to whoever creates the logic [...] so I think there’s a lot of human factor and decision, and I don’t think any of it is reliable and should not be presented to the end user.” As another example, P7 justified deprecating after seeing all three rounds of user feedback but for reasons beyond any specific feedback, mentioning, “I don’t think it effectively tracks performance, and I think it erodes trust and culture within the company. And it would cause micromanaging within the teams. And I think people would try to find ways to gamify the system [...] and I feel like there’d be more politicking involved. I think this would not be good for company culture. Yeah, I foresee it as like a detriment [...] I think it shuts down voices potentially.”

Some of these participants deferred evaluation of these concerns to others, such as product managers, to decide on task prioritization for engineers, while a couple of participants plainly refuted the value of addressing those concerns. For instance, for one user comment they disagreed with, P2 decided, “I think I would essentially pass the buck to like PM’s or UX researchers to essentially determine, is that a worthwhile business problem to solve? Like how much value does that generate for our users, whether it’s like they feel safer.” P16 refuted one feedback comment, mentioning, “That just makes this feature redundant. I don’t think this should be addressed [...] No one cares about that. So if I’m a developer at Zoom, I wouldn’t want to implement this feature, because there’s no point.” In general, participants who did not connect with user feedback seemed less motivated to take the initiative to fix them or looked to others to verify if they were worth addressing.

Another common challenge we observed was participants viewing product decision-making as outside of their primary role or acknowledging that outside factors could thwart their efforts to make a difference in their organization, even when presented with user feedback.

Engineering solutions accounting for different user applications of the product were limited to clear communications of feature functionality. There was a common pattern of participants considering some extent of product usage as outside the scope of their software design work. For instance, when considering how this tool would be used by employers for tracking attention of their employees in work meetings, many participants acknowledged how punitive action based on this tool could be seen as unfair and could even be abused to “make arguments in bad faith” (P3), but most of those participants mentioned this consideration as out of scope of the product design or struggled to think of ways to address it, other than to add clarity, such as with the previously-mentioned “disclaimers” about the data for users. For instance, participants mentioned, “You can’t guarantee that companies aren’t gonna [...] use this as justification for axing hybrid or fully remote work, right? Because that’s beyond a Zoom product designer’s control” (P3), “We provide the information, but [...] it’s really up to the meeting user, whatever the context of the meeting is [...] but as for what it means to be accountable, I don’t think the tool needs to answer that” (P1), and “Some things you just can’t control, like which environment [...] and what the employer is using this data for [...] that is the responsibility of the employer and the employee” (P4). Others echoed similar sentiments but mentioned clear communication, both from the product and from intermediate users such as organization heads and meeting hosts, as the best solution for this case, saying, “I don’t think we can address sort of, like, how it should be used. But we can provide enough information and enough clarity on how much information we do provide for users of it to make sort of those decisions kinda outside of the feature” (P1), or even, “Do companies have a right to use this information against you when they’re evaluating a performance? So I think the company should be very clear about this, only then the feature should be enabled” (P16). Other participants stated potential misuse as a factor to simply deprecate the feature, also struggling to think of other design solutions. For instance, P8 mentioned, “We built this tool that the employer may be trying to force on people, and it’s more on the employer for using it a certain way [...] But that raises the question of if this feature should even exist, if it can cause these sorts of controversies.”

Participants deferred to others to determine how the broader organization balanced feature prioritization with user privacy comfort. A number of participants mentioned a consideration of realistic business factors, namely that the product would likely cater to the highest-paying customers. P3 recognized that for many product decisions, “It depends on who your highest paying customer is and what they want,” and P13 similarly noted, “I think it really ultimately depends on whether or not that’s a feature that the paying customers want to put in, ’cause if you end up putting that in as a feature change, and the people that actually pay for the licenses don’t like it, then there’s no point actually putting it in.” These participants assumed that they as engineers would concede to what the company decided would be best for their paying customers, with a couple participants explicitly mentioning they would prioritize surveying preferences from premium users (or potential paying users) and defer to them to decide on product direction. In this sense, not only did engineers recognize their employer and the end user as stakeholders in the design process, but also the perceived relative importance (in this case, financial weight) of certain users as affecting the product design decisions.

Several other participants also cited external regulations as taking precedence when guiding product decisions. For example, P8 noted, “I would assume that Zoom has some sort of security council and legal counsel [...] to discuss this type of point, because there’s also different countries – I know the European Union has different security laws regarding user data,” and P12 mentioned, “First, we have to make sure we follow the laws, regulations – GDPR, some common legal codes, something like that.” In general, these broader balance considerations of employer and valued customer priorities as well as external regulations were seen as out of scope for these participants in the role of an engineer.

Hello! My name is ____ and I’m doing an interview about workplace surveillance as part of my studies at my institution. Thank you for agreeing to take the time to talk today. This interview will take 45-50 minutes of your time. It is also voluntary, so you may choose to withdraw from the interview at any point for any reason. And please don’t hesitate to ask questions at any point.

For the purposes of my study, I will be recording our conversation as well as any sketches I may ask you to do during this interview. I will anonymize your identity and will report the results only in the context of academic publications. Do I have your consent to use your data, and would you still like to participate in this study?

Thank you! To get things rolling, let’s start by going through a few questions:

Have you heard of Zoom’s attendee attention tracking feature? It’s a feature on the Zoom video calling platform that isn’t active anymore, but basically if someone was sharing their screen in a group call, the hosts could see which participants weren’t actively on the presentation window for more than 30 seconds. Inactivity was marked by a gray timer icon next to their name on the participants list (visible only to the hosts). After the meeting, Zoom would generate a report listing the percentage of time each participant had Zoom in focus during the meeting, as well as how long they were in the meeting. [Show screenshots.]

I’d like you to first imagine you were a user in a meeting utilizing Zoom Attention. Can you anticipate any needs and concerns as a user?

How might these issues be addressed in a feature update?

How confident do you feel this product will satisfy user needs and concerns (on a scale of 1 being not at all confident to 5 being extremely confident)?

Now I’d like you to act as a software developer for this feature on your product, Zoom. Taking the original Zoom Attention feature, let’s say you were provided the following request to update and change the feature:

[Choose one of the following 3 based on the random order chosen for the protocol for this participant - can paste into a separate doc and share so the participant can refer to these]

Now I would like you to draw up a change to this feature that would address this request (and any other needs that you might think of). If you feel your previous idea is sufficient, please explain how it addresses this request.

How confident do you feel this product will satisfy user needs and concerns (on a scale of 1 being not at all confident to 5 being extremely confident)?

Now let’s say you as the developer also receive this feature request:

[Choose one of the 3 listed above based on the random order from the protocol for this participant - can paste into a separate doc and share so the participant can refer to these]

As before, please draw up any fixes or feature improvements you would add to address this request (and any other needs you might think of), on top of your previous ideas. If you feel your previous design is sufficient, please explain how it addresses this request.

How confident do you feel this product will satisfy user needs and concerns (on a scale of 1 being not at all confident to 5 being extremely confident)?

That’s all the questions I had for you today! Is there anything else about your ideas or thoughts about anything we’ve talked about that you didn’t get to mention yet?

Thank you very much for your time! I’ll collect your information for paying out your incentive. If you would also like to stay in touch and see the potential results of this study, let me know and I’ll reach out once I have updates to share!