11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

DEFLATE
FromWikipedia,thefreeencyclopedia

Incomputing,deflateisadatacompressionalgorithmthatusesacombinationoftheLZ77algorithmand
Huffmancoding.ItwasoriginallydefinedbyPhilKatzforversion2ofhisPKZIParchivingtoolandwas
laterspecifiedinRFC1951.[1]
TheoriginalalgorithmasdesignedbyKatzwaspatentedasU.S.Patent5,051,745
(https://www.google.com/patents/US5051745)andassignedtoPKWARE,Inc.[2][3]AsstatedintheRFC
document,Deflateiswidelythoughttobeimplementableinamannernotcoveredbypatents.[1]Thishas
ledtoitswidespreaduse,forexampleingzipcompressedfiles,PNGimagefilesandthe.ZIPfileformat
forwhichKatzoriginallydesignedit.

Contents
1Streamformat
1.1Duplicatestringelimination
1.2Bitreduction
2Encoder/compressor
2.1Deflate64/EnhancedDeflate
3UsingDeflateinnewsoftware
3.1Encoderimplementations
3.2Hardwareencoders
4Decoder/decompressor
4.1Inflateonlyimplementations
4.2Hardwaredecoders
5Seealso
6References
7Externallinks

Streamformat
ADeflatestreamconsistsofaseriesofblocks.Eachblockisprecededbya3bitheader:
Firstbit:Lastblockinstreammarker:
1:thisisthelastblockinthestream.
0:therearemoreblockstoprocessafterthisone.

Secondandthirdbits:Encodingmethodusedforthisblocktype:
http://en.wikipedia.org/wiki/DEFLATE

1/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

00:astored/raw/literalsection,between0and65,535bytesinlength.
01:astaticHuffmancompressedblock,usingapreagreedHuffmantree.
10:acompressedblockcompletewiththeHuffmantablesupplied.
11:reserved,don'tuse.

Mostblockswillendupbeingencodedusingmethod10,thedynamicHuffmanencoding,whichproduces
anoptimisedHuffmantreecustomisedforeachblockofdataindividually.Instructionstogeneratethe
necessaryHuffmantreeimmediatelyfollowtheblockheader.
Compressionisachievedthroughtwosteps
Thematchingandreplacementofduplicatestringswithpointers.
Replacingsymbolswithnew,weightedsymbolsbasedonfrequencyofuse.

Duplicatestringelimination
Withincompressedblocks,ifaduplicateseriesofbytesisspotted(arepeatedstring),thenabackreference
isinserted,linkingtothepreviouslocationofthatidenticalstringinstead.Anencodedmatchtoanearlier
stringconsistsofalength(3258bytes)andadistance(132,768bytes).Relativebackreferencescanbe
madeacrossanynumberofblocks,aslongasthedistanceappearswithinthelast32kBofuncompressed
datadecoded(termedtheslidingwindow).

Bitreduction
Thesecondcompressionstageconsistsofreplacingcommonlyusedsymbolswithshorterrepresentations
andlesscommonlyusedsymbolswithlongerrepresentations.ThemethodusedisHuffmancodingwhich
createsanunprefixedtreeofnonoverlappingintervals,wherethelengthofeachsequenceisinversely
proportionaltotheprobabilityofthatsymbolneedingtobeencoded.Themorelikelyasymbolhastobe
encoded,theshorteritsbitsequencewillbe.
Atreeiscreated,containingspacefor288symbols:
0255:representtheliteralbytes/symbols0255.
256:endofblockstopprocessingiflastblock,otherwisestartprocessingnextblock.
257285:combinedwithextrabits,amatchlengthof3258bytes.
286,287:notused,reservedandillegalbutstillpartofthetree.
Amatchlengthcodewillalwaysbefollowedbyadistancecode.Basedonthedistancecoderead,further
"extra"bitsmaybereadinordertoproducethefinaldistance.Thedistancetreecontainsspacefor32
symbols:
03:distances14
45:distances58,1extrabit
67:distances916,2extrabits
http://en.wikipedia.org/wiki/DEFLATE

2/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

89:distances1732,3extrabits
...
2627:distances8,19316,384,12extrabits
2829:distances16,38532,768,13extrabits
3031:notused,reservedandillegalbutstillpartofthetree.
Notethatforthematchdistancesymbols229,thenumberofextrabitscanbecalculatedas

Encoder/compressor
Duringthecompressionstage,itistheencoderthatchoosestheamountoftimespentlookingformatching
strings.Thezlib/gzipreferenceimplementationallowstheusertoselectfromaslidingscaleoflikely
resultingcompressionlevelvs.speedofencoding.Optionsrangefrom0(donotattemptcompression,just
storeuncompressed)to9representingthemaximumcapabilityofthereferenceimplementationin
zlib/gzip.
OtherDeflateencodershavebeenproduced,allofwhichwillalsoproduceacompatiblebitstreamcapable
ofbeingdecompressedbyanyexistingDeflatedecoder.Differingimplementationswilllikelyproduce
variationsonthefinalencodedbitstreamproduced.Thefocuswithnonzlibversionsofanencoderhas
normallybeentoproduceamoreefficientlycompressedandsmallerencodedstream.

Deflate64/EnhancedDeflate
Deflate64,specifiedbyPKWare,isaproprietaryvariantoftheDeflateprocedure.Thefundamental
mechanismsremainthesame.Whathaschangedistheincreaseindictionarysizefrom32kBto64kB,an
additionof14bitstothedistancecodessothattheymayaddressarangeof64kB,andthelengthcodehas
beenextendedby16bitssothatitmaydefinelengthsof3to65538bytes.[4]ThisleadstoDeflate64having
aslightlyhighercompressionratioandaslightlylowercompressiontimethanDeflate.[5]Severalfree
and/oropensourceprojectssupportDeflate64,suchas7Zip,[6]whileothers,suchaszlib,donot,asa
resultoftheproprietarynatureoftheprocedure[7]andtheverymodestperformanceincreaseover
Deflate.[8]

UsingDeflateinnewsoftware
ImplementationsofDeflatearefreelyavailableinmanylanguages.Cprogramstypicallyusethezlib
library(undertheoldBSDlicensewithoutadvertisingclause).ProgramswrittenusingtheBorlanddialects
ofPascalcanusepaszlibaC++libraryisincludedaspartof7Zip/AdvanceCOMP.Javaincludessupport
aspartofthestandardlibrary(injava.util.zip).Microsoft.NETFramework2.0baseclasslibrarysupportsit
intheSystem.IO.Compressionnamespace.

Encoderimplementations
PKZIP:thefirstimplementation,originallydonebyPhilKatzaspartofPKZip.
http://en.wikipedia.org/wiki/DEFLATE

3/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

zlib/gzip:standardreferenceimplementationusedinahugeamountofsoftware,owingtopublic
availabilityofthesourcecodeandalicenseallowinginclusionintoothersoftware.
jzlib(http://www.jcraft.com/jzlib/):Rewrite/reimplementation/portofthezlibencoderinto
pureJavaanddistributedunderaBSDlicense.(Fullyfeaturedreplacementforjava.util.zip).
PasZLIB(http://www.nomssi.de/paszlib/paszlib.html):Translation/portofthezlibcodeinto
PascalsourcecodebyJacquesNomssiNzali.
gziplite(http://sourceforge.net/projects/gziplite/):Minimalistreworkofgzip/gunzipwith
minimalmemoryrequirement,alsosupportingontheflydatacompression/decompression(no
needtobufferizeallinput)andinput/outputto/frommemory.
pako(https://github.com/nodeca/pako):fullfeaturedzlibporttoJavaScript,optimizedforhigh
speed.Worksinbrowsersandnode.js.
miniz(http://code.google.com/p/miniz/)PublicdomainDeflate/Inflateimplementationwithazlib
compatibleAPIinasingle.Csourcefile
lodepng(http://lodev.org/lodepng/)byLodeVandevenne.ABSDlicensedsinglefilePNGfilereader
withbuiltinC++Inflateimplementationandnoexternaldependencies.
KZIP(http://advsys.net/ken/utils.htm#kzip)/PNGOUT:anencoderbythegameprogrammerKen
Silvermanusing"anexhaustivesearchofallpatterns"and"[an]advancedblocksplitter".
PuZip(http://www.cs.tut.fi/~albert/Dev/puzip/):designedforCommodore64/C128computers.PuZip
islimitedtoan8kBLZ77windowsize,withonlythestore(type00)andfixedHuffman(type01)
methods.
BigSpeedDeflate(http://www.bigspeed.net/index.php?page=bsdefdll):"Tinyinmemorycompression
library"availableasaMSWindowsDLLlimitedto32kBblocksatatimeandthreecompression
settings.
BJWFlate&DeflOpt(http://www.walbeehm.com/download/)/DeflOpt:BenJosWalbeehm'sutilities
"designedtoattempttosqueezeeverypossiblebyteoutofthefilesitcompresses".Notethatthe
authorhasstoppeddevelopmentonBJWFlate(butnotDeflOpt)inMarch2004.
Crypto++:containsapublicdomainimplementationinC++aimedmainlyatreducingpotential
securityvulnerabilities.Theauthor,WeiDaistates"Thiscodeislessclever,buthopefullymore
understandableandmaintainable[thanzlib]".
DeflateStream(http://msdn.microsoft.com/enus/library/system.io.compression.deflatestream.aspx)
animplementationofastreamthatperformsDEFLATEcompression,itispackagedwiththeBase
ClassLibraryincludedwiththe.NETFramework.
ParallelDeflateOutputStream(http://cheeso.members.winisp.net/DotNetZipHelp/html/26cbdba2
021accf1a9c9b7ae55f6ecb8.htm)anopensourcestreamthatimplementsaparallel(multithread)
deflatingstream,forusein.NETprograms.
7Zip/AdvanceCOMP:writtenbyIgorPavlovinC++,thisversionisfreelylicensedandtendsto
http://en.wikipedia.org/wiki/DEFLATE

4/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

achievehighercompressionthanzlibattheexpenseofCPUusage.Hasanoptiontousethe
DEFLATE64storageformat.
deflate.s7i(http://seed7.sourceforge.net/libraries/deflate.htm)/gzip.s7i
(http://seed7.sourceforge.net/libraries/gzip.htm),apureSeed7implementationofDeflateandgzip
compression,byThomasMertes.MadeavailableundertheGNULGPLlicense.
PuTTY`sshzlib.c`:astandaloneimplementation,capableoffulldecode,butstatictreeonlycreation,
bySimonTatham.MITlicensed.
Halibut(http://www.chiark.greenend.org.uk/~sgtatham/halibut/)`deflate.c`:astandalone
implementationcapableoffulldecode.ForkedfromPuTTY's`sshzlib.c`,butextendedtowrite
dynamicHuffmantreesandprovidesAdler32andCRC32checksumsupport.
Plan9fromBellLabsoperatingsystem'slibflate(http://plan9.bell
labs.com/sources/plan9/sys/src/libflate/)implementsdeflatecompression.
Hyperbac:usesitsownproprietarylosslesscompressionlibrary(writteninC++andAssembly)with
anoptiontoimplementtheDEFLATE64storageformat.
zip.js(http://gildaslormeau.github.com/zip.js/):JavaScriptimplementation.
Zopfli:publicdomainCimplementationbyGooglethatachieveshighestcompressionattheexpense
ofCPUusage.
AdvanceCOMPusesthehighercompressionratioversionofDeflateasimplementedby7Zip(or
optionallyZopfliinrecentversions)toenablerecompressionofgzip,PNG,MNGandZIPfileswiththe
possibilityofachievingsmallerfilesizesthanzlibisabletoatmaximumsettings.Anevenmoreeffective
(butalsomoreuserinputdemandingandCPUintensive)DeflateencoderisemployedinsideKen
Silverman'sKZIPandPNGOUTutilities,althoughrecentversionsofAdvanceCOMPhavesurpassedKZIP
andPNGOUTwhenusingAdvanceCOMP'sZopflimode.

Hardwareencoders
AHA361PCIX/AHA362PCIXfromComtechAHA(http://www.aha.com/).Comtechproduceda
PCIXcard(PCIID:193f:0001)capableofcompressingstreamsusingDeflateatarateofupto3.0
Gbit/s(375MB/s)forincominguncompresseddata.AccompanyingtheLinuxkerneldriverforthe
AHA361PCIXisan"ahagzip"utilityandcustomised"mod_deflate_aha"capableofusingthe
hardwarecompressionfromApache.ThehardwareisbasedonaXilinxVirtexFPGAandfour
customAHA3601ASICs.TheAHA361/AHA362boardsarelimitedtoonlyhandlingstaticHuffman
blocksandrequiresoftwaretobemodifiedtoaddsupportthecardswerenotabletosupportthe
fullDeflatespecification,meaningtheycouldonlyreliablydecodetheirownoutput(astreamthat
didnotcontainanydynamicHuffmantype2blocks).
StorCompress300(http://www.indranetworks.com/SC300.html)/MX3
(http://www.indranetworks.com/SCMX3.html)fromIndraNetworks
(http://www.indranetworks.com/).ThisisarangeofPCI(PCIID:17b4:0011)orPCIXcards
http://en.wikipedia.org/wiki/DEFLATE

5/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

featuringbetweenoneandsixcompressionengineswithclaimedprocessingspeedsofupto3.6
Gbit/s(450MB/s).AversionofthecardsareavailablewiththeseparatebrandWebEnhance
specificallydesignedforwebservinguseratherthanSANorbackupuseaPCIerevision,theMX4E
(http://www.indranetworks.com/SCMX4E.html)isalsoproduced.
AHA363PCIe(http://www.aha.com/show_prod.php?id=36)/AHA364PCIe
(http://www.aha.com/show_prod.php?id=37)/AHA367PCIe(http://www.aha.com/show_prod.php?
id=38).In2008,ComtechstartedproducingtwoPCIecards(PCIID:193f:0363/193f:0364)witha
newhardwareAHA3610encoderchip.Thenewchipwasdesignedtobecapableofasustained
2.5Gbit/s.Usingtwoofthesechips,theAHA363PCIeboardcanprocessDeflateatarateofupto
5.0Gbit/s(625MB/s)usingthetwochannels(twocompressionandtwodecompression).The
AHA364PCIevariantisanencodeonlyversionofthecarddesignedforoutgoingloadbalancers
andinsteadhasmultipleregistersetstoallow32independentvirtualcompressionchannelsfeeding
twophysicalcompressionengines.Linux,MicrosoftWindows,andOpenSolariskerneldevice
driversareavailableforbothofthenewcards,alongwithamodifiedzlibsystemlibrarysothat
dynamicallylinkedapplicationscanautomaticallyusethehardwaresupportwithoutinternal
modification.TheAHA367PCIeboard(PCIID:193f:0367)issimilartotheAHA363PCIebutuses
fourAHA3610chipsforasustainedcompressionrateof10Gbit/s(1250MB/s).Unlikethe
AHA362PCIX,thedecompressionenginesontheAHA363PCIeandAHA367PCIeboardsare
fullydeflatecompliant.

Decoder/decompressor
InflateisthedecodingprocessthattakesaDeflatebitstreamfordecompressionandcorrectlyproducesthe
originalfullsizedataorfile.

Inflateonlyimplementations
ThenormalintentwithanalternativeInflateimplementationishighlyoptimiseddecodingspeed,or
extremelypredictableRAMusageformicrocontrollerembeddedsystems.
Assembly
6502inflate(https://github.com/pfusik/zlib6502),writtenbyPiotrFusikin6502assembly
language.
ElektronikaMK90inflate(http://www.pisi.com.pl/piotr433/mk90mc1e.htm#inflate),the
above6502programportedbyPiotrPiatektothePDP11architecture.
SAMflate(http://sourceforge.net/projects/samflate/),writtenbyAndrewCollierinZ80
assemblylanguagewithoptionalmemorypagingsupportfortheSAMCoup,andmade
availableundertheBSD/GPL/LGPL/DFSGlicenses.
http://en.wikipedia.org/wiki/DEFLATE

6/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

C/C++
kunzip(http://www.mikekohn.net/file_formats/kunzip.php)byMichaelKohnandunrelatedto
"KZIP".ComeswithCsourcecodeundertheGNULGPLlicense.UsedintheGIMPinstaller.
puff.c(zlib),asmall,unencumbered,singlefilereferenceimplementationincludedinthe
/contrib/puffdirectoryofthezlibdistribution.
tinf(http://www.ibsensoftware.com/download.html)writtenbyJrgenIbseninANSICand
comeswithzliblicense.Addsabout2kcode.
tinfl.c(http://code.google.com/p/miniz/source/browse/trunk/tinfl.c)(miniz
(http://code.google.com/p/miniz/)),PublicdomainInflateimplementationcontainedentirelyin
asingleCfunction.
PCDEZIP,BobFlandersandMichaelHolmes,publishedinPCMagazine19940111.

inflate.cl(http://opensource.franz.com/deflate/)byJohnFoderaro.SelfstandingCommonLisp
decoderdistributedwithaGNULGPLlicense.
inflate.s7i(http://seed7.sourceforge.net/libraries/inflate.htm)/gzip.s7i
(http://seed7.sourceforge.net/libraries/gzip.htm),apureSeed7implementationofDeflateandgzip
decompression,byThomasMertes.MadeavailableundertheGNULGPLlicense.
pyflate(http://www.paul.sladen.org/projects/pyflate/),apurePythonstandaloneDeflate(gzip)and
bzip2decoderbyPaulSladen.Writtenforresearch/prototypingandmadeavailableunderthe
BSD/GPL/LGPL/DFSGlicenses.
deflatelua(http://luausers.org/wiki/ModuleCompressDeflateLua),apureLuaimplementationof
Deflateandgzip/zlibdecompression,byDavidManura.
inflate(https://github.com/chrisdickinson/inflate)apureJavascriptimplementationofInflateby
ChrisDickinson
pako(https://github.com/nodeca/pako):JavaScriptspeedoptimizedportofzlib.Containsseparate
buildwithinflateonly.

Hardwaredecoders
SerialInflateGPU(http://www.bitsim.com/en/badge.htm)fromBitSim.Hardwareimplementationof
Inflate.PartofBitSim'sBADGE(BitsimAcceleratedDisplayGraphicsEngine)controlleroffering
forembeddedsystems.

Seealso
Listofarchiveformats
Listoffilearchivers
http://en.wikipedia.org/wiki/DEFLATE

7/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

Comparisonoffilearchivers

References
1. ^abL.PeterDeutsch(May1996).DEFLATECompressedDataFormatSpecificationversion1.3
(https://tools.ietf.org/html/rfc1951#sectionAbstract).IETF.p.1.sec.Abstract.RFC1951.
https://tools.ietf.org/html/rfc1951#sectionAbstract.Retrieved23April2014.
2. ^USpatent5051745(http://worldwide.espacenet.com/textdoc?DB=EPODOC&IDX=US5051745),Katz,Phillip
W.,"Stringsearcher,andcompressorusingsame",published24September1991,issued24September1991
3. ^David,Salomon(2007).DataCompression:TheCompleteReference(http://books.google.com/books?
id=ujnQogzx_2EC&pg=PA241)(4ed.).Springer.p.241.ISBN9781846286025.
4. ^BinaryEssenceDeflate64(http://www.binaryessence.com/dct/imp/en000225.htm)
5. ^BinaryEssence"CalgaryCorpus"compressioncomparisons
(http://www.binaryessence.com/dct/apc/en000263.htm)
6. ^7ZipManualandDocumentationcompressionMethod
(http://sevenzip.sourceforge.jp/chm/cmdline/switches/method.htm)
7. ^HistoryofLosslessDataCompressionAlgorithmsDeflate64
(http://ieeeghn.org/wiki/index.php/History_of_Lossless_Data_Compression_Algorithms#DEFLATE64)
8. ^zlibFAQDoeszlibsupportthenew"Deflate64"formatintroducedbyPKWare?
(http://www.zlib.net/zlib_faq.html#faq40)

Externallinks
PKWARE,Inc.'sappnote.txt,.ZIPFileFormatSpecification
(http://www.pkware.com/documents/casestudies/APPNOTE.TXT)Section10,X.DeflatingMethod
8.
RFC1951DeflateCompressedDataFormatSpecificationversion1.3
zlibHomePage(http://www.zlib.net)
AnExplanationoftheDeflateAlgorithm(http://zlib.net/feldspar.html)byAntaeusFeldspar.
ExtendedApplicationofSuffixTreestoDataCompression
(http://www.larsson.dogma.net/dccpaper.pdf)AnexcellentalgorithmtoimplementDeflatebyJesper
Larsson
Retrievedfrom"http://en.wikipedia.org/w/index.php?title=DEFLATE&oldid=623213190"
Categories: Losslesscompressionalgorithms
Thispagewaslastmodifiedon28August2014at19:19.
TextisavailableundertheCreativeCommonsAttributionShareAlikeLicenseadditionaltermsmay
http://en.wikipedia.org/wiki/DEFLATE

8/9

11/22/2014

DEFLATEWikipedia,thefreeencyclopedia

apply.Byusingthissite,youagreetotheTermsofUseandPrivacyPolicy.Wikipediaisa
registeredtrademarkoftheWikimediaFoundation,Inc.,anonprofitorganization.

http://en.wikipedia.org/wiki/DEFLATE

9/9