Alert Now or Never: Understanding and Predicting Notification Preferences of Smartphone Users

We sought a diverse sample of participants during recruitment. In our formal study, 40 participants successfully completed the onboarding process and contributed valid data. Of these, 17 people self-identified as male, 22 as female, and 1 as non-binary gender. The youngest participants were in the 18–24 age group and the oldest participants were in the 65 or older age group. Most subjects were in the 35–54 age group (17 out of 40). The self-reported education level ranged from high school degree to doctoral degree with diverse occupations. More than half of the participants identified as a “heavy user” of their smartphone, defined as using their phone more than 6 hours per day.

The participants of both the pilot study and the formal study were recruited through Google’s internal user study recruitment service. We created a screener to identify candidates, and then we selected and contacted the final participants ourselves. We required participants to be located in the US and be at least 18 years of age. We also required participants to have an Android phone as their primary phone, running at least Android Oreo, due to technical constraints. The participants were required to be single language (English) users so that all apps on the phone would mainly target English users and notifications would be written mainly in English. We also excluded users who had more than one phone or used a smartwatch in order to reduce potential interference of receiving notifications on multiple devices.

To encourage participants to stay in our study, which requires constant engagement and collects relatively invasive data, the compensation followed a tiered incentive structure. Participants who answered questionnaires for the study tasks in the first three days received $35 USD. Participants who answered questionnaires in the last four days received an additional $45 USD. There was a $20 USD bonus for participants who completed the entire 7-day study, for a maximum compensation of $100 USD. 47 participants signed up initially, though seven dropped out or did not complete onboarding, and five did not respond to any questionnaires during the study. Phone usage and behavior data were still collected for the five subjects that did not drop out but did not respond to questionnaires. The structure and amount of the incentives for the study were determined by Google’s user research compensation policy. This study was approved by Google’s privacy working group and complied with all internal corporate policies for the collection and storage of user data.

Data collection for the study was conducted remotely via a custom Android app. The study app first guided participants through an onboarding process. This process instructed participants to grant permission to the notification listener service and the accessibility service so that the app could listen to all incoming notification traffic and log phone usage traces. Over the course of the study, participants responded to two types of ESM tasks within the app, which were triggered by notification events. These tasks, denoted Task 1 and Task 2, are described in more detail below (also see Figure 1). Throughout the study, the app also collected all notification content, how users handled notifications and phone usage traces. All collected data was synchronized to a secured Firebase instance that we operated specifically for this study.

Our study began on April 2nd, 2019 (Tuesday) and ended on April 9th, 2019 (Monday) for all but three participants. Those three started and ended two days later due to recruiting issues. Before the study, all participants signed a consent form. On the morning of the first day, we sent an e-mail to each participant which contained a link to the study app and study instructions. Once installed, the study app guided users through an onboarding process that requested the necessary permissions to collect data, demonstrated the two ESM tasks, and described the privacy features of the study app.

After seven days, the app automatically stopped collecting data and posted a notification to inform users that the study had ended. Before participants removed the app, they were asked to initiate a manual upload from the app to ensure all data had been uploaded to our remote server.

Over the course of the week-long study, we received phone usage data from 40 participants in total, including how and when users handled notifications and when they opened what apps. Five of these participants only contributed passive data (see Section 2.2.4), while the other 35 participants completed some of the ESM tasks. For our analysis, we removed Task 1 responses that were marked as “not seen” and Task 2 responses with “I don’t know” to the first question (“Did you immediately discard the notification upon seeing it?”).

Before we started the qualitative coding analysis and the experiments about predicting notification preferences, we split the entire dataset into two parts, based on notification delivery time. Notifications delivered in the first five days of the study were treated as the training set, and notifications delivered in the last two days were treated as the test set. The dataset was divided for all analyses, including both the evaluation of the statistical models and the qualitative coding, in order to avoid overfitting to the test set. Dividing the dataset based on time is a common practice in user behavior modeling research, which emulates the real-world situations where past data of the same user or different users is used to understand human behaviors and build predictive models.

Figure 3 shows the number of participants that responded to ESM tasks each day of the study (35 on the first day, 16 on the last day). Due to this dropoff in participation, our training set consists of data from 35 people, while the test set only consists of data from 20 people.

Table 1 presents the number of notifications that were collected (All), have a Task 1 response (Task 1), have a Task 2 response (Task 2), have either a Task 1 or Task 2 response (Either task or All sampled), and have both Task 1 and Task 2 responses (Both tasks), separated into the Training set and the Test set.

We performed qualitative thematic analysis [4] on two types of textual data collected in the study: (1) explanations of delivery preference from Task 1, and (2) explanations of notification-related activity from Task 2.

In our sample, 56% notifications were expected to be delivered with an alert immediately, indicating a potential to improve user experience by reducing interruption. The distribution of the other three choices demonstrates users’ preferences of methods for reducing interruptions. The ratios of “Send now, never alert” (26%) and “never send” (12%) were higher than “send now, alert later” (6.3%).

We conducted pairwise comparisons of the ratios of the four delivery methods and found significant difference for all the six pairs under McNemar’s test (p-values have been adjusted using Bonferroni correction): alert now vs. alert later: \(\chi ^2(1, N\!=\!525)\!=\!147\) , \(p\lt .001\) ; alert now vs. never alert: \(\chi ^2(1, N\!=\!525)\!=\!24\) , \(p\lt .001\) ; alert now vs. never send: \(\chi ^2(1, N\!=\!525)\!=\!98\) , \(p\lt .001\) ; alert later vs. never alert: \(\chi ^2(1, N\!=\!525)\!=\!199\) , \(p\lt .001\) ; alert later vs. never send: \(\chi ^2(1, N\!=\!525)\!=\!334\) , \(p\lt .001\) ; never alert vs. never send: \(\chi ^2(1, N\!=\!525)\!=\!236\) , \(p\lt .001\) .

Figure 4 shows that our subjects are evenly distributed across the spectrum of the tolerance to interruption caused by notifications. Two subjects (user1 and user2) did not want any of their notifications to interrupt them immediately, while three others (user33, user34, and user35) preferred to receive immediate alerts for every notification. A total of 13 out of 35 participants (user1, 2, 4, 11, 12, 13, 15, 16, 17, 18, 20, 21, 26) chose “send now, alert later” for some of their notifications, while others did not select this option at all. This suggests that individual differences affect people’s attitudes towards deferring the alert.

Based on the qualitative coding of Task 1 user responses, we generated a list of factors (Table 2) that influenced users’ notification delivery preferences. We categorize the factors into two categories: content and context. Factors related to the notification itself are considered content factors. These factors are mostly intrinsic characteristics of notifications, varying from objective factors such as notification type to subjective factors such as users’ perceived notification importance. Factors related to the user’s status (e.g., interruptibility) or the environment (e.g., location) are considered context factors. Figure 5 shows the breakdown of delivery preferences for some content and contextual factors with relatively high occurrence.

Although both content characteristics and contextual factors emerged, content-related factors (91.6% of notifications) were mentioned much more frequently than contextual factors (15.3%). The ratio of content-related factors is significantly higher than context-related factors under McNemar’s test ( \(\chi ^2(1, N=464)=8.7\) , \(p=.003\) ).

In the following, we present the definitions, examples and the relationship of each factor to notification delivery preferences (Figure 5).

Using the results of the qualitative coding analysis of users’ explanations as collected in Task 2, we generated a list of reasons that caused users to act upon notifications in a certain way (Table 3). Figure 6 plots the breakdown of delivery preferences, removal action types, and self-reported promptness for each of the 11 codes. We will use this to help unravel the relationship between subjective preferences and objective behaviors through grouping the 11 codes into four sub-categories and two categories. Overall, our analysis provides an in-depth understanding of the decision process that users may go through when handling notifications. We learned that although users may take similar actions on two notifications on the surface, the underlying reason and their preferences about how the notification should be delivered can be completely different.

Response time [3, 33, 34, 38] and the action the user took to remove a notification [19, 30, 40, 51] are two common objective metrics that prior notification management algorithms have optimized for or used as a proxy to label the desirability or importance of notifications. In particular, prior work often associates shorter response time and click-on-notification actions with positive attitudes towards the notification. Therefore, we want to quantitatively study how these two variables correlate with our self-reported notification delivery preferences.

Our baseline is a generic model that can be used in this scenario: the notification management system relies on a pre-trained, static classifier to predict users’ preferences of a new incoming notification. The key idea of the predicting algorithm is to find notifications with similar content from the Task 1 training set (525 notifications, from the first five days), and use the ratio of the “alert now” and “not alert now” labels previously specified for these similar notifications to predict the preference for notifications from the Task 1 test set (102 notifications, from the last two days, see Table 1 and Section 2.4 for more detail about how we split the data).

We define similarity as notifications issued by the same app, because of the universal availability of the source app feature in our dataset and the relatively small training set (525 instances). A larger labeled dataset would allow us to relax this definition and also consider other features.

During the classification process, the model uses the preferences of similar notifications labeled by other users in the training set to calculate the preference for a single user’s notification in the test set. The algorithm compares the ratio of “Send now, alert now” among all matched notifications with a predefined threshold, and predicts “Send now, alert now” if the ratio is higher than the threshold. We experimented with different thresholds in the evaluation results sub-section.

Similarly, the second model also only uses subjective preferences labeled by users. This setting is based on a different fictional use scenario: the system requests the user label their preferred delivery method after receiving a notification, and when a new notification arrives, the model combines other users’ data and the labels provided by the same user to predict the preferred delivery method of the new notification.

The classification algorithm works similarly to the algorithm in the baseline model, except that the algorithm uses similar notifications labeled by the same user in the training set when available, and otherwise falls back to preferences labeled by other users, as in the baseline model.

In this model, we explore personalized prediction of notification preferences when users’ subjective labels are not available for previous notifications. This is an important problem since soliciting subjective preference labels can impose extra burdens on users and is therefore unrealistic for real-world deployment. Following the analysis in Section 4, we studied predicting delivery preferences based on how users handled similar notifications in the past.

We evaluated the three models on the “Task 1” test set and present the results below. Since our models rely on finding similar notifications from the same package to make predictions, only 79 out of the 102 notifications in the test set was used in the evaluation. We selected the 79 notifications based on the coverage of the generic model, which is a subset of the two other personalized models (92 and 96 notifications for the user-label and user-behavior personalized model respectively).

Because the direct output of all three models is a score, we thoroughly enumerated different threshold values for each model to convert the score to a prediction. This allows us to study the tradeoff between the true prediction rate of alert now instances (recall) and the true prediction of other instances (specificity) by plotting Receiver Operating Characteristic (ROC) curves and calculating the Area Under the Curve (AUC) for each algorithm. This is among the most commonly used performance measurements for classification problems that require predetermined thresholds. AUC varies from 0 to 1. The AUC of an uninformative classifier is 0.5. For AUC above 0.5, a higher AUC indicates the model is better at differentiating between classes [5].

Table 4 shows the AUC of different models examined in our experiments. We can see that the personalized models have much better predictive powers than the generic model. Figure 8 plots the ROC curves. Note that we only plot the user-behavior personalized model built with the removal action type and the response time, which achieves the best performance among the three behavior-based personalized models we tested. The best performance of the two personalized models are very close, which improve the performance of the generic model by 42% and 39%, respectively.

With factors that naturally emerged from the free-form explanations of notification delivery preferences, we are able to gain a better understanding of what factors may be more important to users in determining how to deliver a notification (RQ1). In the following, we discuss some lessons learned about designing an intelligent notification system that fits users’ expectations.

Users seem to be fairly risk-averse when making delivery decisions for their notifications. People seem to err on the side of being notified (56% preferred an immediate alert, 26% preferred suppressing the alert, while only 6.3% preferred deferring the alert), seemingly due to the fear of missing out on important or useful information. For example, some subjects were fine receiving promotional notifications if there was a small chance that they could benefit. This echoes the dilemma of the positive and negative effects of notifications discussed in prior work [23, 41, 43]. Specifically, prior research in e-mail spam filtering has a similar assumption that the consequences of information loss or delay depends on the nature of the message [7]. Furthermore, the preference of suppressing alerts over deferring alerts may be related to previous observations that users often consciously leave some or all notifications in the notification tray until a later time [52] and users snooze notifications [53]. We speculate that users want to gain more control over their notifications and therefore are more inclined to manually defer reading or taking actions on notifications rather than let the system do so. Note that our study was focused on the timing of the alert rather than the delivery.

The frequent reference to notification content in survey explanations suggests that notification content plays an important on users’ notification preferences. This result echoes findings of prior work [11, 33], while we revealed more nuances about how content and contextual factors affect users’ mental models of how an ideal intelligent notification system might work. As it seems to be more natural for subjects to reason about their preferences in terms of content features, we suggest notification systems should provide more support to help users manage notifications based on notification content, such as grouping notifications with similar characteristics across apps to allow for more convenient control and identifying different notification patterns within an app to allow for more flexible control. It may also be useful for these systems to automatically detect intrinsic characteristics of a notification such as time-sensitivity, user’s relationship with people mentioned in it, whether driven by the user and digital/physical actions that can be triggered by it. Since these factors were frequently used as justifications of notification delivery preferences, existing algorithms that mine notification management rules [30] may improve the interpretability of the system’s decisions by providing explanations of the generated rules based on these factors.

A focus on content features is further called for because of the availability of contextual signals. Many contextual features mentioned by users in their explanations, such as the knowledge of whether the user has already seen the information in the notification, or the user’s current task when it does not involve the phone, are difficult if not impossible for the notification system to obtain. Fortunately, our classification experiments suggest that reasonable results may be possible relying only on the content that is available and not on contextual information that may not be.

Taking a step back, we want to stress that although users mentioned contextual factors less often, this does not mean contextual factors are less important. First, our analysis has shown that users’ perceptions of certain notification content may vary in different contexts and from person to person. Second, we chose a relatively brief expiration threshold (5 minutes) for our ESM tasks, which may bias users toward responding to ESM surveys when they are available and less affected by context. Third, given that we were using open-ended questions, the less frequent reference to contextual factors could be because our subjects were not fully aware of what we sought to obtain from the question. Hence, we want to caution that the actual influence of contextual factors may be higher than what was measured in our study.

In Section 4 (RQ2), our qualitative analysis showed that similar actions can be driven by different reasons associated with different delivery preferences. Then we quantitatively showed that the response time and action type were both weak predictors of delivery preference for the same notification. These results offer takeaways for using observable user actions in designing intelligent notification systems.

First, the relationship between user actions and user preferences seems to be moderated by the notification content. That is, it may be helpful to take into account notification types when using user actions to infer delivery preferences. We have seen Mehrotra et al. [30] specifically discussed reminder notifications that do not require follow-up actions. Our systematic analysis of users’ rationales behind their actions allowed us to expand on this special case and identify more related scenarios. For example, some notifications contained enough information so users did not need to click on it to read further, such as a news notification or a notification containing a verification code. Some notifications were a confirmation or a notice about an expected action, such as a download complete notice or a credit card payment confirmation.

Another important takeaway is that users may not use notification systems as they were designed, which suggests the intentions behind users’ direct interaction with notifications may be clarified when more digital traces are collected and analyzed. We use three examples to demonstrate this point. First, the code “Want to take action but not via the notification” showed that users sometimes chose to directly open the app to complete actions instigated by a notification. Second, users appropriated notification systems for a much wider range of applications than initially intended such as using the notification tray as a type of to do list, which is related to the code “Intentionally delay the action”. Third, the delivery of notification in a multi-device condition has been discussed in some work [8, 31] and we further demonstrated that users still preferred two-thirds of notifications coded as “Seen elsewhere” to be delivered with an immediate alert.

Then in Section 5 (RQ3), we demonstrated that although the response time and action type of a notification do not seem to be effective features for inferring delivery preferences, using actions aggregated on similar notifications received in the past can be an effective substitute for user labels to build personalized models with decent performance. Our current models only use two basic types of features: notification type and response time. Given that our qualitative analysis has identified other aspects of user behavior that may correlate with their delivery preferences, such as app usage before or after receiving and dismissing the notification, we expect incorporating these measures may further improve the prediction accuracy. In addition, as people seemed to use different strategies to handle different types of notifications, NLP features could be used to automatically identify high-level categories of a notification [15] and we can feed this information to the model to make more targeted prediction. Due to the small size of our sample, these sophisticated model designs can only be left for future work to evaluate.

Given the above findings, we argue algorithms predicting when and how to deliver notifications may need to be incorporated into a more nuanced, holistic, mixed-initiative notification management approach to ensure accuracy and user control. For example, it may be a promising approach to provide a flexible interface for users to configure notification preferences based on the app and the notification content, coupled with automatic notification preference tuning.

We therefore propose an alternate design for an intelligent notification system. Unlike other methods that predict the opportune time to send notifications based mainly on contextual factors [1, 11, 20, 37, 41, 45] and prevent missing information via a bounded deferral mechanism [20], our approach primarily focuses on notification content. Users could set the urgency of notifications using hand-crafted keyword-/template-based filters, or adjusting a filtering-aggressiveness threshold to choose filters automatically created by the system. The threshold-adjusting idea could be supported by our template identification process and the context-agnostic preference prediction models introduced in Section 5, and we create a preliminary user interface design to help illustrate this concept in Figure 9.

Current mainstream smartphone operating systems already provide notification management features, such as specifying the importance of notifications per app and per notification channel and snoozing notifications.⁷ However, our study results suggest that they still fall short for adoption, as seen for example in the extremely low utilization rate of SNOOZE. This further illustrates the importance of automatic notification preference tuning. Our initial exploration in preference prediction has demonstrated that we could achieve substantial improvement upon the generic model in multiple aspects if a few preference labels on previous notifications are available or some user behavior data is available. We believe there is a large space for future work to study better data collection methods and better prediction algorithm design.

As with any study, our work has limitations that must be considered when evaluating its results.

One limitation of including a question about a proposed system behavior, in our case delaying the time of the alert, is that users may not necessarily realize the potential benefits of a feature they have not experienced. Thus, we want to note that this result should not be interpreted as proving that opportune moment detection is not valuable. Instead, we argue that the deferment option may be more acceptable to users when the concerns of the risk of information loss are addressed, e.g., making sure important notifications will always be delivered on time.

As noted by previous research [46], the ESM has an inherent bias: being able to answer the survey already implies that the subject is likely to be more tolerant of an interruption. Data collected in this way may underestimate the frequency of the moment when people find a notification to be disruptive. Although we granted a 5-minute time window for answering the surveys, this bias may still exist. As a result, we might expect a larger portion of notifications to be deemed inappropriate because of the disruption caused by the study, and thus delivery preferences to favor no alerts or even complete suppression of the notification.

The duration of our study is relatively short (1 week), although the same length of study can be found in prior work with similar study design (logging + daily diary surveys [41], logging + ESM surveys [54]). The week-long study design results in more notifications received on weekdays than on weekends in the training set, which may affect the qualitative and quantitative analysis in Sections 3 and 4. However, the drop-off rates in Figure 3 seem to justify our choice of study duration, which suggests that drop off may have reached 100% by day 11 of a longer study.

We were limited in the number of surveys that we believed users would respond to, especially because our surveys required free-text answers and typing long passages on a phone can be tiring. However, the fact that we were able to obtain statistically significant difference for the main results suggests the sample size is sufficient for our goals.

In this study, we did not analyze the modalities used for sending the alert and could not take into account how users configured modalities to manage interruption [9] and its implications on users’ delivery preferences. As previous research showed that different people prefer different modalities for notifications with varying priority and from different people [6, 29], we would like to further investigate factors that affect users’ preferences of different modalities and the corresponding prediction task in the future.

Some early work borrowed the concept of the “breakpoint” from classic psychology literature and proposed a “defer-to-breakpoint” policy to manage notifications [1]. The main idea is to exploit the natural transition between two units of the primary tasks (i.e., the breakpoint) to deliver the notification. Breakpoint detection has been studied for both physical activities and interactions with corresponding digital devices. Research has found that this mechanism has positive effects on reducing mental load and task resumption lag [22, 35, 36, 37].

Another line of work has studied interruptibility, asking users to evaluate whether a particular moment is good for an external interruption and aimed at scheduling notifications at an opportune moment. Past research that falls into this category usually follows a traditional machine learning research methodology, where ESMs [17] are first used to collect users’ feedback on incoming notifications and then predictive models are built of users’ interruptibility using the dataset. The predictive model can be used by an intelligent system to automatically defer notifications to more appropriate moments [1, 11, 20, 37, 41, 45].

Both the defer-to-breakpoint policy and opportune moment detection models heavily focus on understanding the user’s context when receiving the notification, reflected by the use of contextual features such as location, time, physical activity, and phone usage when building the classifier [10, 13, 14, 38, 44, 48, 55]. Conversely, most of this previous work did not consider the content of the incoming notification when making predictions. The effect of notification content on users’ perception of the interruption was usually studied by sending artificial notifications from the study app [38] or only studying a certain type of notification (e.g., phone call [49] or news [11, 37]), which leads to a gap between these interruptibility models and working solutions of notification systems.

Recent work has begun to systematically examine the notifications received on today’s mobile systems and how content affects users’ perception of the interruption. Fischer et al. [11] controlled topics of news pushed to users during a study via notifications, and found that the entertainment value, relevance, and actionability of the content seem to have more effect than delivery timing on users’ impressions of notifications. Mehrotra et al. [34] analyzed a large-scale notification dataset collected in the wild. Their results showed that the perceived disruption can be influenced by the sender-recipient relationship. Iqbal et al. [23] found that people were willing to tolerate mobile interruptions in order not to miss important updates, which resonates with our study. As more work has pointed out the important role that notification content plays in users’ subjective feelings about the interruption caused by a certain notification, researchers started expanding the meaning of interruptibility and integrating content features into predictive models [8, 32, 33].

Although notification content is considered more frequently by intelligent notification management systems, its use has also exposed more issues. For example, inferences based on context and content may not always be consistent. When contextual-based interruptibility is low but notification importance is high, what should the system do? Some work suggests that notification content may be more influential than the context [11], while other research observed the opposite [26] that context overshadowed the sender-recipient relationship in predicting the attentiveness, responsiveness, interruptibility, and opportuneness of the notification.

In our work, we framed our query to users from a different perspective, as a chance to adjust how the system delivered the notification (see Section 3.2). Unlike prior research that fixed dependent variables before running the study [11, 26, 34], we asked users to explain their choices in a free form response. In this way, we attempted to eliminate the potential bias of priming users with certain factors, so we could learn about the dominant factors that naturally emerged in their responses. Our results showed that content factors (e.g., importance, notification type, time-sensitivity) were mentioned much more frequently than contextual factors (e.g., interruptibility, location, time, phone usage), which suggests that smartphone users expect more system-level control on their notifications based on what the notification is, as compared to when it is delivered to them.

A critical challenge in modeling user preferences is that collecting subjective labels usually incurs a considerable cost to users because these queries are also interruptions. This cost directly leads to the sparsity of the labeled data [34], making it hard for the model to fit users’ preferences in a supervised manner. The intrinsic disruptive nature of the queries may also bias the dataset towards moments that are more appropriate to users, where users have time to handle the study queries [46].

Therefore, some work adopted an alternative approach, which used observable user behaviors as the source of feedback. Engagement and responsiveness are two common metrics in past literature. Engagement is determined by the way the notification is handled, specifically focusing on whether a notification is accepted (click to open) or discarded (swipe to remove). Some work [30, 40, 51] directly used whether the user engages with the notification by accepting it as the prediction target. Others applied reinforcement learning and treated the user action as the reward function [19]. Responsiveness is usually quantified by the response time, the time from the user receiving or seeing a notification to the time they removed the notification. Mehrotra et al. [33] set a hard response time threshold and considered all instances that had a shorter response time than the threshold to have been appropriate and instances with a longer response time to be inappropriate.

In contrast, we find in our work that nuanced user preferences cannot be characterized by response time or the action type alone. Unstructured reasons provided by our participants offer some explanation of this difference. For example, users delayed acting on important notifications because sufficient information was available in the notification preview text. In other cases, they dismissed a notification quickly if it was completely irrelevant, or chose to defer the bulk clean up of irrelevant notifications to a later time. We further learned in the predicting algorithm experiments that although some user actions only have very weak predictive power for a single notification, aggregating them for a group of similar notifications received in the past can generate a much more stable and accurate prediction for a new incoming notification.

Empirical studies and predictive modeling have provided an important foundation for designing intelligent notification systems. Past research has proposed various system designs, which aimed at either reducing the disruption or improving the engagement by deferring the notification to a more appropriate time [1, 2, 11, 20, 37, 41, 45], filter out unimportant notifications [2, 30], or automatically choose the appropriate modulation for the notification alert [12, 28, 46].

However, due to the nuances of individual’s different delivery preferences, current physical and mental status, and future plans, it is exceptionally difficult to achieve near perfect accuracy on statistical modeling tasks [32, 50]. Furthermore, a misprediction may cause a large disruption and users would not accept a system that defers or stops an important notification [34]. This means that a system for notification management is unlikely to be able to rely on modeling alone in order to provide a high-quality experience. As a result, in our study we do not focus our data collection solely on the opportuneness of notification timing but on the larger space of notification management.

In the same spirit, some previous research focused not only on improving accuracy, but also aimed at building explainable intelligent notification systems to give users a better understanding and stronger feelings of trust in the system [27]. PrefMiner [30] is a system that aims at automatically dismissing unimportant notifications. The system mines human-readable rules to manage notifications and requests for user approvals before using the rules. In this way, users can benefit from the automatic management and also stay in the control. In this article, we propose algorithms that can be used to automatically initiate the notification delivery preference configurations for notifications from different apps and notifications using different templates from the same app, which also aims at reducing the manual labor when still keep the systems’ actions intelligible and controllable.

Recently, the designs of the commercial systems and proposals from academic papers [2, 8, 47, 48] also show a trend to provide users with more fine-grained ability to control notifications. However, our study demonstrates that some existing features in commercial system (e.g., snooze notification) has an extremely low utilization rate. We consider proper automation may still be an indispensable part for such systems to achieve success.

Overall, subjects believed they handled the sampled notifications immediately after seeing them in most cases, with only 22.1% notifications marked as not immediately discarded/accepted/snoozed (83 notifications). We also found a consistent trend in the response time, the time between seeing and acting on a notification, which has a long-tail distribution (Figure 10).

Table 5 shows the raw count and the ratio of different actions that caused a notification to be removed. The first four rows (“CLICK”, “CANCEL”, “CANCEL_ALL”, “SNOOZE”) represent user-driven actions that remove notifications, and the next two rows (“AUTOMATIC”, “OVERWRITE”) represent two situations where the app automatically removed the notification before the user took any action. The last row (“Not removed”) represents a special situation where the notification was never removed during the study.

Below are some notes on the impact of our experience sampling methodology on this data. Overall, the impact of ESM tasks and sampling rules seem to be small, which suggests that the usage data collected in our study could reflect users’ natural behavior.