A Solution toward Transparent and Practical AI Regulation: Privacy Nutrition Labels for Open-source Generative AI-based Applications

Generative Artificial Intelligence [88] refers to AI systems that utilize existing media to generate new content [67, 81, 82]. These systems leverage large datasets to learn patterns and structures, enabling them to create original outputs that resemble the input data. This capability spans various modalities, including text, images, audio, and video, making GAI a versatile tool in numerous apps [58]. A variety of apps with different functionality are popping up on the market. For instance, it can be used to produce realistic images from textual descriptions, generate emails, write news, and even simulate voices. The potential of GAI lies in its ability to extend human creativity and productivity by automating the creation of high-quality, innovative content.

Recent advancements in GAI, especially in the domain of LLMs, have substantially improved the ability of these systems to understand and process textual information. The leaps in this field have empowered GAI apps to interpret user inputs with greater precision and generate contextually relevant responses. This progress has markedly reduced the accessibility barriers to such GAI apps, enabling users, even those with limited expertise in prompts or instructions, to effectively utilize text-guided generation capabilities [10].

Third-party GAI app collection websites are independent online marketplaces established by external developers or organizations. The gpt3demo⁴⁴4https://gpt3demo.com and gpt4demo⁵⁵5https://gpt4demo.com are two popular third-party websites that actively collect and curate GAI apps and demonstrations since the mid of 2022, as pioneers of their kind. We use those two popular third-party collections to harvest mainstream GAI apps on the market. Through the deployment of customized web scrapers, we successfully retrieved information including the name, category tags, code repository link, and privacy notices (e.g., privacy policies). For the gpt3demo site, a total of 887 GAI apps are categorized into 228 fine-grained categories. Meanwhile, the gpt4demo site lists 87 GAI apps, spread across 47 distinct categories. Most of these GAI apps position themselves as domain experts, incorporating professional domain knowledge to act as advisors for users in specific fields, including but not limited to Robot Lawyers and Coding Assistants. We manually checked and selected open-source GAI apps that explicitly provided a GitHub repository address. After excluding three inaccessible repositories, we successfully collected 148 open-source GAI apps with a valid GitHub repository link, with the proportions being 15.2% of all curated GAI apps. As shown in Table I.c, the average stars and the average forks are 11.4k and 1.9k, respectively, reflecting the significant popularity of those open-source GAI Apps. According to Repositories Ranking [7], the median stars of the top 10,000 GitHub repositories is about 5.8k. GAI apps provide expertise and consultation to support a wide range of app scenarios. However, the complexity and capability of these apps lead to potentially significant privacy risks, which remain to be adequately addressed [93].

Notably, as this is a super dynamic market, our data collection is a snapshot from January 2024. This subset is sufficient to demonstrate some characteristics of GAI apps discussed in this section. OpenAI official GPTs store⁶⁶6https://chatgpt.com/gpts is another GAI app repository that attracts increasing attention, but all apps are solely based on GPT foundation models and most of them are not open-source.

The privacy notice is an essential component of software, including the emerging GAI apps and tools. The privacy policy is the most common type of privacy notice and is widely required by privacy regulations and industry standards [76, 75]. In the mobile app ecosystem, two of the largest app stores, Google Play and Apple App Store, both mandate that app developers provide a privacy policy before publishing an app. Fig. 2.a shows that the privacy policies of mobile apps in the Google Play app store are curated in a visually obvious and easily accessible position (in the sidebar) on the homepage. However, the GAI apps market is an emerging sector, and there are no such industry standards to enforce developers to follow. Therefore, we conduct an empirical study to investigate the current state and practices of privacy notices for open-source GAI apps, as follows.

For the 148 GAI apps, we conducted manual scrutinization about their GitHub repositories, official websites (if available), and tool interfaces (if available) to check the provision of privacy notices, especially privacy policies. Specifically, we visited all the aforementioned websites and searched for relevant information in their HTML sources. The keywords used for the search included “privacy”, “privacy policy”, “notices”, “terms”, “terms of services”, etc. Although the terms of services (ToS) are not privacy notices, they are commonly put aside in practice. Those keywords might help us to find privacy notices. Additionally, we employed the Google search engine to broadly search the privacy notices for GAI apps. The search terms are “[GAI application name] + [keyword] (e.g., privacy policy)”, and for each search keyword, we evaluated all entries on the first result page. For example, the privacy policies of AgentGPT⁷⁷7Its official website is https://agentgpt.reworkd.ai, and its privacy policy is available at https://agentgpt.reworkd.ai/privacypolicy.html and PromptLayer⁸⁸8Its official website is https://promptlayer.com/, and its privacy policy is available at https://promptlayer.com/privacy_policy.pdf are discovered by additional searching, but their privacy policies are not directly included in their repositories or main interfaces. Fig. 2.b shows the privacy policy link of AutoGPT, located in the footer of the advertisement website, is not easily accessible to users. In total, only 12.2% (18/148) of examined GAI apps offer a privacy policy as listed in Table I.a, indicating a significant transparency deficiency in providing essential privacy information to end users. Our findings also highlight the issue that existing privacy policies can be difficult for users to find without extra effort.

Among the small portion of GAI apps, the readability of their privacy policies is as unsatisfactory as reported in previous studies [76, 51]. These policies are often lengthy, filled with jargon, and contain frequent hyperlinks and cross-references. Table I.b shows the statistics of GAI app privacy policies we collected. The average length of examined privacy policies is about 3,000 words, and the average reading time is about 12 minutes, according to [13]. We then calculated the readability based on the Flesch Reading-Ease Test for privacy policies [43, 76], and the average readability score is 39.9. This number indicates that fully comprehending the privacy policy of GAI apps requires at least a college-level education [43], which contradicts the mission of GAI apps to mitigate the expertise gap.

Surprisingly, through additional searching, we noticed only two (1.3%) GAI apps provide privacy labels as their privacy notices, instead of privacy policies. Privacy labels are short-form, clear, table-like disclosures that enable users to quickly understand how their data is collected and utilized [39, 41, 24]. Privacy labels have been proven useful over the past decade, and the concept has now been widely applied in the industry, such as Apple and Google. Comparably, 60% of apps on the Apple App Store and 44% of apps on the Google Play Store have completed the necessary forms to generate privacy labels, as of August 26, 2022 [50]. The two existing privacy labels for GAI apps are compulsorily required privacy disclosures (similar to the Data Safety section in the Google Play app store) in the Chrome web store⁹⁹9https://chromewebstore.google.com/category/extensions for Chrome extensions, as both GAI apps are implemented and launched in the Chrome store. Such privacy label format is tailored for browser extensions, neglecting the unique challenges and regulatory requirements of GAI apps. We detail the current challenges of privacy labels in the next section and propose a GAI-specific privacy label format in Section IV.

Given a GAI app, our ultimate objective is to analyze its corresponding GitHub repository, determining the value of each GAI privacy label field mentioned in Section IV. Fig. 3 illustrates the overview of our Repo2Label framework, including four stages. First, we extract the code repository from a given GAI app for subsequent analysis (Section V-B). Then, we design multiple AI units to extract information about GAI privacy labels from the code repository and generate explanations for each field (Section V-C). Next, we conduct label verification to assess the accuracy of the provided references and implement reflection processes to correct any hallucinated labels identified by the model (Section V-D). Finally, we combine individually generated GAI privacy labels (for each code file) into a comprehensive repository-level GAI privacy label, with references to further explain the value of each field.(Section V-E).

The only input of the Repo2Label framework is the link of the GitHub repository of the given GAI apps. Based on the link, we can access all files contained in the repository by the GitHub API [4]. Normally, these code repositories contain a semi-structured textual document, README, that demonstrate the implementation, and various source code files. We exclude irrelevant art files (e.g., images), embedding vectors (e.g., the weights of models), and datasets, since they do not provide any informative hint about the behaviors of GAI apps. We filter out those files based on the file type.

This is the core stage of the whole framework. We aim to analyze the content in the repository to obtain the behaviors that are related to the GAI privacy label. This task requires understanding the functionality of individual functions within code files and the semantics of natural language descriptions. Recent studies have shown that large language models (LLMs) are exceptionally capable of software code understanding tasks [33, 54, 30, 73]. Considering our need to analyze the semantics of both code and natural language for extracting privacy nutrition label information, we designed several AI units by employing LLMs as our foundational model.

Privacy Nutrition Label Extraction AI Unit. For each AI unit, there are two inputs: a file from the repository and the definition of GAI privacy label fields. Specifically, all content in the file will be treated as character strings and the definition is the field explanations in Table IV. We then strategically design the prompt templates to exploit the LLMs’ capabilities, enabling the effective extraction of GAI privacy label information from code and textual files. The prompt template mainly comprises three main components: task description, input, and output. The task description utilizes a structured prompt to clearly outline the process. There are three key components of this prompt as follows. 1) @persona: This component instructs the model to assume the role of an expert data analyzer, ensuring clarity and focus on the task’s specific requirements. 2) @terminology: Here, essential terms are defined in a keyword format to clarify the context. For instance, the privacy label field, such as Base Model and its corresponding definition, such as “The names of foundation models that are embedded in this tool. (e.g., GPT-4, GPT-3.5, Ernie, etc)” 3) @instruction: This section furnishes explicit instructions for executing the task:

• •

  @command: Offers an overarching strategy for the model to follow, ensuring that the extraction process adheres to defined objectives.

• •

  @rule: Outlines the fundamental constraints and considerations that must be observed during the GAI privacy label extraction process.

• •

  @Input_format: Specifies the format of the File Content to be inputted.

• •

  @Output_format: Details the expected presentation format for information extracted from the file content.

In the @command, we guide the model through the extraction process step-by-step by employing a chain-of-thought approach, as detailed in the works of Wei et al.[87], Wang et al.[86], and Lyu et al. [57]. This structured methodology helps in unfolding the reasoning required for each step of the task, enhancing the model’s ability to generate coherent and contextually accurate outputs. Additionally, in the @rule section, we establish specific guidelines to govern the extraction of nutrition labels by the model, ensuring clarity and accuracy in its tasks:

• •

  @rule1: Limit the model’s task exclusively to information extraction.

• •

  @rule2: Require the model to provide a reference for each extraction.

• •

  @rule3: Ensure that all references provided by the model originate exclusively from the file content. The validity of these references will be evaluated in Section V-D to determine if any model-generated hallucinations occur during the extraction process and to enable necessary corrective reflections.

• •

  @rule4: Reiterate that all references come solely from the file content. Emphasizing this rule multiple times underlines its importance and reinforces the model’s adherence to rigorous standards.

• •

  @rule5: Reiterate that the model generates results according to the specified output format.

In addition to the privacy label content, the second and third rules also fetch the references from the file content that supports the extracted label, serving for the next stage for verification, self-correcting potential hallucinations. Based on the aforementioned prompt template, we develop four specialized AI units, each designed to target specific sections of the privacy labels:

• •

  Basic Info Extractor: This unit utilizes six key labels: Base Model, Tools Modality, Tool Functionality, Tool’s Working Details, Controller Contact, and Target Users. It systematically extracts these labels from the input content, each supported by detailed explanations.

• •

  Data Rights Info Extractor: Integrating four critical labels: Data Retention, Right to Access, Right to be Forgotten, and Right to Lodge Complaints. This unit extracts information pertinent to data rights from the input content, ensuring each label is clearly explained.

• •

  Risk Related Info Extractor: This unit focuses on extracting three risk-related labels: AI-generated Watermarking, Prompt Guardrail, and Risk Notification. Each label is detailed within the input to provide a clear contextual understanding.

• •

  Additional Info Extractor: Targeting additional factors, this unit processes two labels: Data Encryption and Protection of Minors.

LLMs are widely criticized for their tendency to produce hallucinations [37, 35, 95]. This phenomenon occurs because the inherently probabilistic nature of LLMs can result in outputs that appear convincingly accurate but are actually incorrect, making it difficult for humans to discern the inaccuracies. Given this, privacy labels, which serve as crucial privacy notices for GAI app users, must strive to be as accurate as possible. Studies also show that LLMs can self-correct the hallucinations by proper strategies [72]. Therefore, we design the Label Verification to double-check the potentially incorrect extracted privacy labels. Specifically, instead of directly checking the labels, we employ the AI unit to review the correctness of the references used by the model during the label-generation process. We use the string matching techniques to confirm whether the reference indeed exists within the file content. If the reference is indeed in the file content, we consider the corresponding label generated by the model to be likely accurate and thus retain it. Conversely, if the reference is incorrect (i.e., not exist in the original file), we ask the AI unit to proceed with the reflection prompt to regenerate the label with the specific instruction: “You previously extracted a label with an incorrect reference that does not exist in the file content. Please ensure that the reference provided this time is present in the file content.” If the model provides incorrect references more than three times for the same label, we categorize the label as N/A, indicating its non-applicability or unreliability.

After the previous stages, we obtain all extracted GAI privacy labels and references from each file. We then aggregate them as a complete GAI privacy label based on the whole repository. For each nutrition label, from a set of n files, we collect all labels that are not N/A. We record the source code path and the corresponding reference for each valid label. The format used for this recording is $<$file_path, nutrition_label, reference$>$. For example,

• •

  codeRepos/babyagi/babyagi.py

• •

  gpt-3.5-turbo

• •

  LLM_MODEL = os.getenv(‘LLM_MODEL’,os.getenv (‘OPENAI_API_MODEL’, ‘gpt-3.5-turbo’)).lower()

To generate a repository-level nutrition label, we merge the labels from all files, taking their union. For instance, if the base model label across different files includes versions like gpt-4, gpt-3.5-turbo, and text-embedding-ada-002, the final label for the repository would be the combination of these three. Additionally, the final repository-level nutrition label features an expansion window that allows users to view all the references for each label. This feature provides a transparent view of the evidence supporting each label, enhancing the credibility and usability of our extracted information.

To examine the design of our proposed GAI privacy label format, we conducted a comprehensive online survey as a human evaluation. Ethical approval for this research was secured from our institution’s Institutional Review Board (IRB). It is important to note that before the formal experiment, we conducted a small-scale pilot study. This pilot study allowed us to preliminarily assess the duration required for the formal study, which averaged 11 minutes and 53 seconds. More critically, through actual task performance and participant feedback, we identified and refined ambiguous statements within the scales.

The accuracy of privacy labels is crucial, as inaccuracies can lead to information overload for users and severely undermine their trust in the software.

By observing the manually annotated dataset of privacy practices of GAI apps, we notice a significant discrepancy between GAI app implementation and regulations. Notably, a label marked as “No” does not necessarily indicate that the GAI app violates regulations, and vice versa. There are several reasons for this: 1) Open-source GAI apps are not commercial software, and they may not be subject to certain regulations; 2) Regulations vary by region, and their interpretation could be subjective and controversial; and 3) The implementation of privacy practices may not rigorously comply against regulatory requirements. Nonetheless, our findings still reflect that the open-source GAI apps community commonly lacks awareness and responsiveness to those regulations. At the early stage of the market, compliance often cannot be guaranteed, but we advocate for the community to pay attention to these requirements.

To establish transparency surrounding AI-enabled products (e.g., GAI apps) and thus yield trust, researchers have also proposed various types of privacy notices in addition to privacy policies and privacy labels, such as the Model Cards [65, 53] and AI Bills of Materials (AIBOMs) [89]. Model cards are files that accompany the foundation models (e.g., Llama) and provide handy information. They are essential for discoverability, reproducibility, and sharing. Typically, Model Cards coefficiently provide critical information about the model’s dataset, evaluation results, and potential ethical considerations. However, Model Cards do not provide privacy disclosures and the dynamic nature of model development presents challenges in maintaining the model card content timely. AIBOMs offers a more specialized disclosure aimed at experts, focusing on elements such as software dependency, version & licenses, etc. While model cards and AIBOM play a crucial role in enhancing transparency and promoting responsible AI usage, they currently exist only as a community norm and are not proactively aligned with privacy and GAI-specific laws and regulations.

Privacy labels can enhance data controllers’ internal compliance routines [68]. Hence, GAI privacy labels contribute to enhancing the security of the entire open-source software ecosystem. Our privacy label design serves as a reference model for building a responsible AI. Drawing from research in related fields, open-source projects with Model Cards have seen a significant increase in downloads [53], primarily due to improved transparency. This provides a theoretical basis for the expectation that GAI apps with privacy labels in the open-source community will also experience higher user adoption. Our regulation-driven privacy label ensures that privacy practices and technologies are aligned with current legal standards.

1) Internal Validity: We follow OpenAI’s official API to access the aforementioned GPT models in our experiments. We evaluate the performance of our framework on the GAI app dataset. The results from GPT models may deviate due to the probabilistic nature of the model. We have taken extensive measures to mitigate this risk, including providing references when generating labels and implementing a verification step to ensure accuracy. We believe that with the continued development of more advanced foundation models, this issue can be further alleviated.

2) External Validity: Our privacy label design incorporates common requirements from various regulations to emphasize the current regulatory focus on generative AI tools. However, for our privacy labels to be deployed in individual countries, they must adhere to specific local regulations. In addition, although our framework is evaluated on a dataset based on GitHub code repositories, the core analysis focuses on individual code files, making it can be easily generalized to other code repositories.