The Internet with Privacy Policies: Measuring The Web Upon Consent

In this tangled picture, legislators started to regulate the ecosystem to avoid massive indiscriminate tracking that may threaten users’ privacy. In 2013, the European Cookie Law [22] entered into force, which mandates websites to ask for informed consent before using any profiling technology. Later, in May 2018, the General Data Protection Regulation (GDPR) [32] entered into force in all European member states. It is an extensive regulation on privacy, aiming at protecting users’ privacy by imposing strict rules when handling personal information. Unlike previous regulations, it sets severe fines and infringements that could result in a fine of up to \(€\) 10 million, or 2% of the firm’s worldwide annual revenue, whichever amount is higher. Some websites have already been caught presenting legal violations in their Consent Banner implementation [40] and a large fraction have been shown to use tracking technologies before user consent [50, 54]. In the US, the California Consumer Privacy Act (CCPA) [21] enhances privacy rights and consumer protection for California residents by requiring businesses to give consumers notices about their privacy practices.

As a result, most of the websites now provide explicit Consent Banners [28] and many adopt Consent Management toolsets [36], making the website content difficult to access until visitors accept the privacy policy. For example, Figure 1 shows the same news website homepage before and after accepting the privacy policy. Only upon pressing the “Got it” button, the website content is fully loaded and visible.

Despite cases of misuse, the new regulations had a large impact on the web users and complicate the measurement of the tracking ecosystem. A simple Web crawler visiting the websites without accepting the privacy policies would offer a biased picture, with no tracker and no ad being loaded. Hu et al. [37] already found that the number of third-parties dropped by more than 10% after GDPR when visiting websites automatically. Conversely, when using a dataset from 15 real users, they measure no significant reduction in long-term numbers of third-party cookies. Dabrowski et al. [25] draw similar conclusions, finding an apparent decrease in the use of persistent cookies from 2016 to 2018. Sorensen et al. [52] testify a decreasing trend in the number of third parties during 2018. We quantify this phenomenon in Figure 2, using the HTTPArchive open dataset [6]. The curators of this dataset maintain a list of top websites worldwide that they automatically visit using the Google Chrome browser from a US-based server to store a copy of each visited webpage. Using the tracker list detailed in Section 3, we report the percentage of websites embedding one or more trackers for five European countries (simply using the Top-Level Domain to identify the country).² We restrict the analysis on those websites that exist for the whole six years-long periods ( \(9\,196\) website in total).

Figure 2 could suggest that the introduction of the GDPR (the black vertical line in May 2018) results in an abrupt decrease in the number of tracker-embedding websites, a trend that continues up to the moment we write. However, as we will show, these measurements are an artifact due to the GDPR itself. Indeed, the Web crawler used by HTTPArchive can only capture the behavior of the websites as a “first-time visitor”, before the user accepts any privacy policy. The crawler thus misses third-party trackers and ads.

Research papers that rely on crawling large portions of the Web for different reasons could be affected by the same bias in their measurements. For instance, this would challenge the automatic measurement of the Web ecosystem on privacy [14, 16, 30, 33, 37, 38, 43, 45, 47, 48, 55] and counter-measurements [41, 47, 53]. Moreover, this will also impact those works that rely on crawlers and headless browsers [18] to quantify the impact in the wild of new technologies like SPDY, HTTP/2 [20, 27, 31, 59], 4G/5G [15, 17], accelerating proxies [49, 51, 60], or generic benchmark solutions [44]. At last, even spiders and mirroring tools like Wayback Machine and HTTPArchive may be affected if the website allows the visitor to access its content only after accepting the privacy policy.

Vallina et al. [56] are the first to consider the impact of the Consent Banner presence. First, they instruct a custom OpenWPM crawler to identify specific Consent Banners, and then they manually verify the results. Unfortunately, they solely focus on the pornographic ecosystem, which they acknowledge to be rather different from the Web at large, and thus their work can hardly generalize.

Recently, authors of [16] demonstrated that it is fundamental to consider the complexity of the Web ecosystem and include internal pages in every measurement study. They find a number of recent works that neglect internal pages and, as such, might provide biased results. Yet, they ignore the complications due to Consent Banners. Here, we aim at providing an extensive and thorough study of their impact on the Web. Our goal is to enable the automatic study of webpage characteristics as visitors would experience, assuming that most of them accept the default privacy setting as offered by the Consent Banner. Indeed, it has been shown that most users tend to ignore privacy-related notices [23, 34, 58]. Considering GDPR Consent Banners, users tend to accept privacy policies when offered a default button via intrusive banners that nudge users [19, 29], which is often the case [35] with websites presenting large pop-ups or wall-style banners that cover most of the webpage as seen in Figure 1.

For completeness, notice that cookies are among the simplest tracking mechanisms. Authors of [45] show how practices like cookie synchronization, cookie leaking, and other profiling techniques like canvas fingerprinting are common in today’s Web. Similarly, authors of [39] show how the crawling context, in terms of vantage point and browser configuration, has a significant impact on the results. Our work is orthogonal to these to obtain automatic, realistic, reliable, and user-centric measurements of the Web.

Focusing on automatic management of Consent Banners, some browser add-ons try to hide them by using a list of CSS selectors of known Consent Banners. The most popular add-ons of this kind are “I don’t care about cookies” [7] and “Remove Cookie Banners” [9]. Unfortunately, hiding the Consent Banners has an unpredictable behavior, in some cases falling back to privacy policies acceptance, while, in other cases, triggering an opt-out choice. Other proposals, again in the form of browser add-ons, try to explicitly opt-in or opt-out to cookies. For example, “Ninja Cookie” [8] approves only cookies strictly needed to proceed on the website. Conversely, Autoconsent [2] and Consent-O-Matic [3] use a set of predefined rules to either opt-in or opt-out to cookies, according to the user configuration. These two are the most similar solutions to Priv-Accept. However, they are based on a list of actions the browser automatically runs when finding a set of popular Consent Management Platforms (CMPs), limiting their effectiveness. In Section 3.2, we compare Priv-Accept with Consent-O-Matic–the most mature tool–showing that Priv-Accept offers a much higher coverage. Indeed, the diversity of the Web ecosystem, the presence of multiple languages and the fully customizable choice of Consent Banner buttons make the engineering of Priv-Accept not trivial.

At the core of Priv-Accept there is the list of keywords to be matched against the webpage content to localize the clickable DOM element for accepting the privacy policy. We thoroughly build this list manually in an iterative way. To handle different languages, we build a list that includes keywords for each country we are interested in. For this work, we focus on five European countries, namely France, Germany, Italy, Spain, and the UK,⁵ plus the US–which we use as an example of a large, extra-EU country were privacy laws are in force. For each country, we pick the most popular websites according to the Similarweb lists [10], a website-ranking service analogous to Alexa.

We compare the effectiveness of Priv-Accept with Consent-O-Matic, the most mature browser plugin designed to offer/deny consent to privacy policies automatically. Unlike our tool, Consent-O-Matic exploits the presence of popular Consent Management Platforms (CMP), services that take care of the management of users’ choices on behalf of the website. At the time of writing, Consent-O-Matic allows managing Consent Banners for 35 CMPs. To gauge its performance, we visit the top-100 most popular websites with a Consent Banner for the five countries using a Chrome browser with the Consent-O-Matic plugin enabled. Consent-O-Matic accepts the privacy policies in less than 35% of websites with Consent Banner, and as little as 17% and 20% for websites in Italy and UK, respectively. Here Priv-Accept accepts the privacy policies on all websites by construction.

We then run a second experiment considering another set of 100 websites randomly picked from the Similarweb per country lists. We visit each website with Priv-Accept and a Consent-O-Matic-enabled browser. Figure 5 summarizes the comparison. Priv-Accept accepts the privacy policies in more than 50% of websites, more than twice the success rate of Consent-O-Matic. These results are in line with those of Figure 3. The remaining websites may not have a Consent Banner, fail to load, or use an unknown keyword. This testifies that the customization of Consent Banners makes it difficult to engineer a generic and simple solution. The keyword-based strategy provides results more robust than the CMP-based approach (with similar complexity in curating the lists).

In the following, we use Priv-Accept to check the impact of using Priv-Accept when doing large web measurement experiments. We target a large set of websites popular in France, Germany, Italy, Spain and US, using a test server located in our university campus. For each of the six countries, we use the Similarweb lists to select the top-100 websites from 24 different categories–see Figure 10. These are the top-level unique categories listed in the Similarweb page [13]. In total, we include \(12\,277\) unique websites to visit (as the lists in different countries partially overlap). When visiting websites of a given country, we set the Accept-Language header to indicate the appropriate locale and country language. This behavior can be configured in the Priv-Accept configuration to allow further experimentation.

We run Priv-Accept on a single high-end server running 16 parallel instances to speed up the crawl. We instrument it to run a test sequence, which consists in a Warm-up visit, Before-Accept and After-Accept to the landing page, followed by Additional-Visits to five randomly chosen internal pages–previous studies indeed show that internal and landing pages have different properties [16]. For each website, we repeat the test sequence five times, randomizing the order of websites to visit in each repetition. Our main experimental campaign took place for two weeks on April 2021.

We run additional measurement campaigns to investigate specific aspects. To understand whether Consent Banners appear or have a different impact depending on the visitor location, we repeat the above experiments using servers located in the US, Brazil, and Japan. We use Amazon Web Services to deploy on-demand servers on the desired availability zone. Here, we aim to check if websites behave differently based on the location of the visitors. Since we are using cloud servers, targeted websites may wrongly recognise the test machines as not regular users and locate them in a generic or wrong country. While we cannot check this, we verified that the two most popular commercial IP location databases (IP2Location⁸ and MaxMind⁹) map the IP addresses of our crawlers to the correct country.

To offer a view on a larger number of websites, we visit the top-100 000 websites according to the Tranco list [46]. Unfortunately, the Tranco list does not offer a per-category and per-country rank. We run two separate test sequences: with warm caches, doing (i) Warm-up visit, (ii) Before-Accept, and (iii) After-Accept. And with cold caches, (i) Before-Accept, (ii) erase HTTP cache and clean socket pool, and (iii) After-Accept. Following this procedure, we ensure a fair comparison between Before-Accept and After-Accept in the two scenarios. Recall that Priv-Accept allows one to generate any combination of test sequence with warm/cold cache.

To observe how the presence of trackers changes, we rely on publicly-available lists provided by Whotracksme [12] (a tracking-related open-data provider), EasyPrivacy [5] (one of the lists at the core of AdBlock tracker-blocking strategy) and AdGuard [1] (a popular ad-blocking tool). For robustness, we merge the three lists and consider as a potential tracker any third-party domain that appears in at least two lists. In total, we obtain 1,497 domains that we consider tracking services.¹⁰ We finally record the presence of a tracker during a visit if the webpage embeds an object from a tracking domain, and the latter installs a cookie with a lifetime longer than one month [54]–commonly referred to as profiling cookie. As such, we divide the HTTP transactions carried out during a visit in:

We first study the pervasiveness of Third-Parties and Trackers and check how it varies when we measure it in a Before-Accept or After-Accept. Priv-Accept found and accepted a Consent Banner on \(63.2\%\) of websites. Here, we aim at quantifying the impact of privacy policy acceptance on European websites (10,542 in total) and we exclude those websites exclusively popular in the US.

We first detail the top-15 most pervasive Third-Parties in Figure 6. The GDPR mandates to obtain informed consent before starting to collect any personal data. As such, Third-Parties may be seen as possibly offending services if activated before accepting the privacy policy.¹¹ With little surprise, the most pervasive Third-Party is google-analytics.com. It grows from \(61\%\) to \(74\%\) in popularity on the After-Accept. This value is surprisingly similar to what Metwalley et al. [42] found in 2016, when they found google-analytics.com appearing in 71% of websites. The growth is also sizeable for other Google services such as googleadservices.com and googlesyndication.com. Conversely, domains belonging to Content Delivery Networks, such as cloudflare.com and cloudflare.net do not increase their pervasiveness on the After-Accept, likely being not included in the mechanisms of Consent Banners. Interestingly, only three out of the top-15 Third-Parties are Trackers–i.e., present in our tracker list and setting a persistent cookie. doubleclick.net and facebook.com are the most popular ones, with pervasiveness growing from \(41\%\) to \(58\%\) and from \(24\%\) to \(39\%\) on the After-Accept, respectively. They are present in more than twice the number of websites than their first competitor (quantserve.com). In Figure 6, we also report 95% confidence intervals. It results in that the sample proportion (in percentage) of pervasiveness of Third-Parties is an unbiased estimator of the probability p of a Bernoulli random variable. Therefore, by repeating a number of occurrences of a Bernoulli random variable equal to the number of samples, we obtain the number of successes of a binomial random variable. The confidence intervals become the classical binomial proportion confidence intervals. For the sake of completeness, we report error bars also in the following plots. Note, that, given the large number of samples, the confidence intervals are very narrow and not overlapping between Before-Accept and After-Accept, except for the case of cloudflare.com.

Focusing now on Trackers only, we show their pervasiveness in Figure 7. We count 342 of them. The red curve shows the pervasiveness on the Before-Accept, which is what a naive crawler would report. The blue curve shows how the figure changes on the After-Accept. The Trackers on the x-axis are sorted in descending order according to their pervasiveness on the Before-Accept–hence the Before-Accept curve is monotonically decreasing, while the After-Accept is not. The increase in pervasiveness is general and includes both popular and infrequent Trackers, reaching one order of magnitude in some cases. On the After-Accept, the number of Trackers that are present on \(1\%\) or more of websites grows from 40 to 90. Here, the Spearman’s rank correlation is 0.90, indicating that the Tracker popularity order is approximately the same before and after the privacy policy acceptance. The difference is that their pervasiveness increases.

As it emerges from Figure 7, many Trackers are widespread even on the Before-Accept. This hints at a possibly wrong implementation of the GPDR regulation, which mandates acquiring the visitor’s explicit consent before activating any tracking mechanisms. To be precise, the presence of Trackers on the Before-Accept does not necessarily entail a violation of the law. An analysis of the most popular cookies reveals the presence of test cookies during the Before-Accept using a form similar to test_cookie = CheckForPermission. Google Analytics is a notable example. These cookies are just a check for the possibility of installing profiling cookies upon the user’s acceptance. It is thus possible that the Before-Accept pervasiveness of some Trackers includes cases in which only test cookies are actually used (curiously with expiration date longer than a month). Here we limit to observe that often Trackers set some (potentially) profiling cookies even on the Before-Accept.

Take away: Collecting measurements with or without consent to privacy policies leads to a largely different picture. Upon consent, Trackers are far more pervasive than it appears beforehand. Priv-Accept is instrumental for this goal, thanks to its ability to handle Consent Banners and accept website privacy policies.

We now detail the impact of accepting privacy policies on the number of Trackers found in each website, breaking down our results by country and website category.

We now consider additional measurement campaigns using crawling servers in the Amazon AWS data centers located in the US (Ohio and California), Japan, and Brazil. Figure 11 summarizes our findings. First, notice how Priv-Accept accepted privacy policies on around \(10\%\) fewer websites (about 1,150–1,200) when run from outside Europe, as reported on top x-labels. Checking the screenshot taken by Priv-Accept during the visit on a random subset of these websites, we confirm that no Consent Banner is displayed. We can conclude that some websites customize the Consent Banners based on visitors’ properties, such as their location. If the visit comes from a non-EU country, no Consent Banner is shown.

This different behaviour of websites affects also the statistics of the fraction of websites that embed trackers in the Before-Accept and After-Accept visits. Visiting from outside Europe leads to an increase of Tracking on the Before-Accept in all cases, while, on the After-Accept, changes are limited.

Take away: The crawling location has some impact on the results. This is mostly due to websites that show or not show the Consent Banner based on the user’s location, thus not enabling or enabling tracking on the Before-Accept.

We focus on the webpage complexity in terms of bytes and objects to download. We compute the ratio R between the measurement on the Before-Accept and After-Accept, i.e., \(R = x_{{\it After}}/x_{{\it Before}}\) , where x is the metric of interest. We show the results in Figure 13(a), separately for total downloaded bytes and objects. As expected, accepting the privacy policy increases the webpage size ( \(R\gt 1\) ) by a sizeable factor. For instance, about 9% of websites download more than twice the number objects, and about 5% of websites sees an increase of 3 times or more.

Interestingly, we also observe some websites that are lighter in the After-Accept than in the Before-Accept. Investigating further, these cases are mostly due to the lack of additional content upon acceptance coupled with the savings of not loading the CMP objects on the After-Accept. This happens commonly on those websites that either add a Consent Banner despite not using tracking mechanisms, or that contact Trackers and Third-Parties even before the user has accepted the privacy policies. While the former might be seen as an excess of caution, the latter cases are likely violating the privacy regulations.

To better characterize the differences, we quantify the number of Third-Parties seen in the Before-Accept and After-Accept. We show the Empirical Complementary Cumulative Distribution Function (ECCDF) in Figure 13(b). On median, websites rely on 12 Third-Parties on the Before-Accept. This figure grows to 17 on the After-Accept. The ECCDF highlights the tail of the distribution where we observe those websites that rely on a very large number of Third-Parties: the percentage of websites with more than 50 grows from \(1.8\%\) to \(9.2\%\) , with \(3.0\%\) including more than 75 Third-Parties upon acceptance. This growth in the number of Third-Parties is mostly due to an increase of Trackers and objects related to advertisements that gets loaded after accepting the privacy policy. We also perform statistical tests to compare whether the mean and median of the two sample distributions are statistically different at level 0.05 (t-Test for the mean and Mood’s test for the median). Both result statistically significant in After-Accept. In the appendix we include the picture as above, plotting the number of Trackers instead of Third-Parties, leading to similar conclusions.

Take away: When the Consent Banner is accepted, websites are larger, with 9% of them containing more than twice as many objects. Websites including more than 50 Third-Parties increase from \(1.8\%\) to \(9.2\%\) .

The Third-Party domains appearing after acceptance are generally devoted to advertisements, analytics, and Web tracking. Contacting them has direct implications on the page load time and, indirectly, on the users’ QoE [24]. We thus expect this to cause an increase on the page load time because the browser has to resolve the server name via DNS and contact more servers. For instance, this ultimately limits the advantages offered by new protocols like the stream multiplexing and the header compression offered by HTTP/2 and HTTP/3.

To gauge this, we dissect the webpage load time in Figure 14, comparing separately visits with a warm cache (Figure 14(a)) and with a cold cache (Figure 14(b)). In case of warm cache, we run a Warm-up visit, then the Before-Accept and After-Accept. In case of of cold cache, we run the Before-Accept without a Warm-up visit. Then we erase the HTTP cache and socket pool, then we run the After-Accept.

We report the distributions of the onLoad time for websites with a similar number of additional Third-Parties that are loaded in the After-Accept. We use boxplots, where the boxes span from the first to the third quartile and whiskers from the \(10{th}\) to \(90{th}\) percentile. The central stroke represents the median. The number of websites in each set is detailed on the top the respective boxplot. As expected, the more Third-Parties are loaded upon acceptance, the larger the time needed to load the webpage and the larger its variability. Especially for the websites that add more than 10 Third-Parties, the distributions are remarkably different on the Before-Accept and After-Accept. Considering visits with cold browser cache (Figure 14(a)), those website with 20–50 additional Third-Parties, the median onLoad time passes from 0.91 to 1.41 seconds. The difference increases for the 632 websites adding more than 50 Third-Parties upon acceptance. Here, the median onLoad time increases from 1.35 to 3.38 seconds, more than doubling. Notice also the tail of \(25\%\) of websites loading in more than 4.8 seconds, which happens in less than \(2\%\) of cases during the Before-Accept. We already observed such an increase in our previous study [53], where we measured that median onLoad time increases by \(1.3s\) when cookies policies are accepted. We statistically compare all these couples of sample distributions between Before-Accept and After-Accept, testing differences in the median at a significance level 0.05 (Mood’s test). The test is passed in all cases, showing statistically significant differences.

Similar considerations hold for visits with a cold browser cache (Figure 14(b)). As expected, with the clean cache, websites load time increases–compare values in Figures 14(a) and 14(b). Those that do not add new Third-Parties tend to load slightly faster on the After-Accept, potentially due to the absence of the Consent Banner. In fact, differences are statistically significant in the median of the distributions between Before-Accept and After-Accept, except for the group 1–10 additional Third-Parties. Again, we observe that those adding several Third-Parties after acceptance have a much higher onLoad time on the After-Accept than on the Before-Accept: The median onLoad time increases from 1.8 to 5.2 seconds. Finally, we observe that the onLoad time values tend to be lower than what measured in older works, potentially because of the advances of content delivery network and increased hardware and software performance. Bocchi et al. [20] measured a median onLoad time of 3s in 2016 on a similar albeit smaller set of websites.

Take away: Measuring the webpage load time of websites without considering the implications of accepting the Consent Banners would result in a very biased measurement. Websites that include many more Third-Parties upon acceptance are significantly slower to load.

We here complement the analysis we carried out on the last paragraph of Section 4.2.1. Web tracking involves a number of mechanisms (real-time bidding among all) that result in the same page containing different Trackers on multiple visits. To obtain a reliable picture, we repeat each test five times. In Figure 15, we show how two macroscopic tracking measurements vary with different numbers of repeated visits for each website. The blue line in the figure shows the fraction of websites that contain at least one Tracker when measured with an increasing number of test repetitions. It is moderately affected by the number of tests, increasing from 69.1% with a single repetition to 70.0% with five repetitions. Similarly, the average number of Trackers, increases from 6.5 to 7.8.

We here report the same analyses depicted in Figures 12(b) and 13(b), showing the number of Trackers instead of the number of Third-Parties. The two pictures lead to similar conclusions.