Organizing and compressing collections of files using differences
Article No.: 14, Pages 1 - 10
Abstract
A collection of related files often exhibits strong similarities among its constituents. These similarities, and the dual differences, may be used for both compressing the collection and for organizing it in a manner that reveals human-readable structure and relationships. This paper motivates and studies methods for such organizing and compression of file collections using inter-file differences. It presents an algorithm based on computing a minimum-weight spanning tree of a graph that has vertices corresponding to files and edges with weights corresponding to the size of the difference between the documents of its incident vertices. It describes the design and implementation of a prototype system called diboc (for difference-based organization and compression) that uses these methods to enable both compression and graphical organization and interactive exploration of a file collection. It illustrates the benefits of this system by presenting examples of its operation on a widely deployed and publicly available corpus of file collections (collections of PPD files used to configure the CUPS printing system as packaged by the Debian GNU/Linux distribution). In addition to these qualitative measures, some quantitative experimental results of applying the methods to the same corpus are also presented.
References
[1]
Jyrki Alakuijala and Zoltan Szabadka. 2016. Brotli Compressed Data Format. Internet Engineering Task Force (IETF). Request for Comments 7932. https://tools.ietf.org/html/rfc7932
[2]
Jon Bentley and Douglas McIlroy. 1999. Data Compression Using Long Common Strings. In Proceedings of the IEEE Data Compression Conference (DCC'99). IEEE Computer Society, 287--295.
[3]
Per Bothner. 1998. Kawa: Compiling Dynamic Languages to the Java VM. In Proceedings of the USENIX Annual Technical Conference, FREENIX Track (New Orleans, Louisiana). USENIX Association, Berkeley, CA, USA, 41--41. http://dl.acm.org/citation.cfm?id=1268256.1268297
[4]
Jon Butler, Wei-Hsin Lee, Bryan McQuade, and Kenneth Mixter. 2008. A Proposal for Shared Dictionary Compression over HTTP. Technical Report. Google, Inc. https://lists.w3.org/Archives/Public/ietf-http-wg/2008JulSep/att-0441/Shared_Dictionary_Compression_over_HTTP.pdf IETF-HTTP-WG mailing-list message.
[5]
Sudarshan S. Chawathe. 2005. Differencing Data Streams. In Proceedings of the 9th International Database Engineering and Applications Symposium (IDEAS '05). IEEE Computer Society, USA, 273--284.
[6]
Sudarshan S. Chawathe. 2020. Efficient File Collections for Embedded Devices. In Proceedings of the 8th Workshop on Communications in Critical Embedded Systems (WoCCES 2020). IEEE, Rennes, France.
[7]
Yann Collet and Murray S. Kucherawy. 2018. Zstandard Compression and the application/zstd Media Type. Internet Engineering Task Force (IETF). Request for Comments 8478. https://tools.ietf.org/html/rfc8478
[8]
Lasse Collin et al. 2020. The Tukaani Project: XZ Utils. https://tukaani.org/xz/.
[9]
Christoph Doblander, Tanuj Ghinaiya, Kaiwen Zhang, and Hans-Arno Jacobsen. 2016. Shared Dictionary Compression in Publish/Subscribe Systems. In Proceedings of the 10th ACM International Conference on Distributed and Event-Based Systems (Irvine, California) (DEBS '16). Association for Computing Machinery, New York, NY, USA, 117--124.
[10]
Christoph Doblander, Arash Khatayee, and Hans-Arno Jacobsen. 2018. PreDict: Predictive Dictionary Maintenance for Message Compression in Publish/Subscribe. In Proceedings of the 19th International Middleware Conference (Rennes, France) (Middleware '18). Association for Computing Machinery, New York, NY, USA, 174--186.
[11]
Emden R. Gansner and Stephen C. North. 2000. An open graph visualization system and its applications to software engineering. Software: Practice and Experience 30, 11 (2000), 1203--1233. <1203::AID-SPE338>3.0.CO;2-N
[12]
James W. Hunt and Thomas G. Szymanski. 1977. A Fast Algorithm for Computing Longest Common Subsequences. Commun. ACM 20, 5 (May 1977), 350--353.
[13]
Till Kamppeter et al. 2020. pyppd: A CUPS PostScript Printer Driver's compressor and generator. https://github.com/OpenPrinting/pyppd.
[14]
David G. Korn, Joshua P. MacDonald, Jeffrey C. Mogul, and Kiem-Phong Vo. 2002. The VCDIFF Generic Differencing and Compression Data Format. Internet Engineering Task Force (IETF). Request for Comments 3284. https://tools.ietf.org/html/rfc3284
[15]
Debra A. Lelewer and Daniel S. Hirschberg. 1987. Data Compression. Comput. Surveys 19, 3 (Sept. 1987), 261âĂŞ296.
[16]
David MacKenzie, Paul Eggert, and Richard Stallman. 2018. GNU diffutils---Comparing and Merging Files. Free Software Foundation. https://www.gnu.org/software/diffutils/manual/diffutils.html Manual for diffutils version 3.7.
[17]
Dimitrios Michail, Joris Kinable, Barak Naveh, and John V. Sichi. 2020. JGraphT---A Java Library for Graph Data Structures and Algorithms. ACM Trans. Math. Softw. 46, 2, Article 16 (May 2020), 29 pages.
[18]
Eugene W. Myers. 1986. AnO(ND) difference algorithm and its variations. Algorithmica 1, 1 (1986), 251--266.
[19]
java-diff-utils contributors on GitHub. 2020. Java Diff Utils. https://github.com/java-diff-utils/java-diff-utils/.
[20]
G. Toubassi. 2012. How FemtoZip Works (In Painful Detail). Software system documentation. https://github.com/gtoubassi/femtozip/wiki/How-FemtoZip-Works-(In-Painful-Detail).
Index Terms
- Organizing and compressing collections of files using differences
Recommendations
Compressing Inverted Files
AbstractResearch into inverted file compression has focused on compression ratio—how small the indexes can be. Compression ratio is important for fast interactive searching. It is taken as read, the smaller the index, the faster the search.
The premise “...
Data compression techniques for economic processing of large commercial files
SIGIR '71: Proceedings of the 1971 international ACM SIGIR conference on Information storage and retrievalThe application of compact coding, differencing and other techniques to indexed sequential files is discussed. The effects on system performance are discussed and reductions of almost 80% in mass storage requirements for a particular file are reported.
Compressing Java class files
Java class files are often distributed as jar files, which are collections of individually compressed class files (and possibility other files). Jar files are typically about 1/2 the size of the original class files due to compression. I have developed ...
Comments
Information & Contributors
Information
Published In
Copyright © 2020 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 25 August 2020
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
IDEAS 2020
IDEAS 2020: 24th International Database Engineering & Applications Symposium
August 12 - 14, 2020
Seoul, Republic of Korea
Acceptance Rates
IDEAS '20 Paper Acceptance Rate 27 of 57 submissions, 47%;
Overall Acceptance Rate 74 of 210 submissions, 35%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 32Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)0
Reflects downloads up to 27 Jul 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in