A Distributed System for The Management of Fine-grained Provenance

Existing provenance systems operate at a single layer of abstraction workflow/process/OS at which they record and store provenance. However, the provenance captured from different layers provides the highest benefit when integrated through a unified provenance framework. To build such a framework, a comprehensive provenance model able to represent the provenance of data objects with various semantics and granularity is the first step. In this paper, the authors propose a provenance model able to represent the provenance of any data object captured at any abstraction layer and present an abstract schema of the model. The expressive nature of the model enables a wide range of provenance queries. The authors also illustrate the utility of their model in real world data processing systems. In the paper, they also introduce a data provenance distributed middleware system composed of several different components and services that capture provenance according to their model and securely stores it in a central repository. As part of our middleware, the authors present a thin stackable file system, called FiPS, for capturing local provenance in a portable manner. FiPS is able to capture provenance at various degrees of granularity, transform provenance records into secure information, and direct the resulting provenance data to various persistent storage systems.