This package provides various compression algorithms.
github.com/golang/snappy offering better compression and concurrent streams.
Go Reference Go Sourcegraph Badge
Use go get github.com/klauspost/compress@latest to add it to your project.
This package will support the current Go version and 2 versions back.
nounsafe tag to disable all use of the "unsafe" package.
noasm tag to disable all assembly across packages.
Use the links above for more information on each.
* zstd: Add simple zstd EncodeTo/DecodeTo functions https://github.com/klauspost/compress/pull/1079 * zstd: Fix incorrect buffer size in dictionary encodes https://github.com/klauspost/compress/pull/1059 * s2: check for cap, not len of buffer in EncodeBetter/Best by @vdarulis in https://github.com/klauspost/compress/pull/1080 * zlib: Avoiding extra allocation in zlib.reader.Reset by @travelpolicy in https://github.com/klauspost/compress/pull/1086 * gzhttp: remove redundant err check in zstdReader by @ryanfowler in https://github.com/klauspost/compress/pull/1090 * flate: Faster load+store https://github.com/klauspost/compress/pull/1104 * flate: Simplify matchlen https://github.com/klauspost/compress/pull/1101 * flate: Use exact sizes for huffman tables https://github.com/klauspost/compress/pull/1103
* Add unsafe little endian loaders https://github.com/klauspost/compress/pull/1036
* fix: check r.err != nil but return a nil value error err by @alingse in https://github.com/klauspost/compress/pull/1028
* flate: Simplify L4-6 loading https://github.com/klauspost/compress/pull/1043
* flate: Simplify matchlen (remove asm) https://github.com/klauspost/compress/pull/1045
* s2: Improve small block compression speed w/o asm https://github.com/klauspost/compress/pull/1048
* flate: Fix matchlen L5+L6 https://github.com/klauspost/compress/pull/1049
* flate: Cleanup & reduce casts https://github.com/klauspost/compress/pull/1050
<details> <summary>See changes to v1.17.x</summary>
* zstd: Fix extra CRC written with multiple Close calls https://github.com/klauspost/compress/pull/1017 * s2: Don't use stack for index tables https://github.com/klauspost/compress/pull/1014 * gzhttp: No content-type on no body response code by @juliens in https://github.com/klauspost/compress/pull/1011 * gzhttp: Do not set the content-type when response has no body by @kevinpollet in https://github.com/klauspost/compress/pull/1013
* gzhttp: Add TransportAlwaysDecompress option. https://github.com/klauspost/compress/pull/978 * gzhttp: Add supported decompress request body by @mirecl in https://github.com/klauspost/compress/pull/1002 * s2: Add EncodeBuffer buffer recycling callback https://github.com/klauspost/compress/pull/982 * zstd: Improve memory usage on small streaming encodes https://github.com/klauspost/compress/pull/1007 * flate: read data written with partial flush by @vajexal in https://github.com/klauspost/compress/pull/996
* s2: Reduce ReadFrom temporary allocations https://github.com/klauspost/compress/pull/949 * flate, zstd: Shave some bytes off amd64 matchLen by @greatroar in https://github.com/klauspost/compress/pull/963 * Upgrade zip/zlib to 1.22.4 upstream https://github.com/klauspost/compress/pull/970 https://github.com/klauspost/compress/pull/971 * zstd: BuildDict fails with RLE table https://github.com/klauspost/compress/pull/951
* zstd: Reject blocks where reserved values are not 0 https://github.com/klauspost/compress/pull/885 * zstd: Add RLE detection+encoding https://github.com/klauspost/compress/pull/938
* s2: Add AsyncFlush method: Complete the block without flushing by @Jille in https://github.com/klauspost/compress/pull/927 * s2: Fix literal+repeat exceeds dst crash https://github.com/klauspost/compress/pull/930
* zstd: Fix incorrect repeat coding in best mode https://github.com/klauspost/compress/pull/923 * s2: Fix DecodeConcurrent deadlock on errors https://github.com/klauspost/compress/pull/925
* flate: Fix reset with dictionary on custom window encodes https://github.com/klauspost/compress/pull/912 * zstd: Add Frame header encoding and stripping https://github.com/klauspost/compress/pull/908 * zstd: Limit better/best default window to 8MB https://github.com/klauspost/compress/pull/913 * zstd: Speed improvements by @greatroar in https://github.com/klauspost/compress/pull/896 https://github.com/klauspost/compress/pull/910 * s2: Fix callbacks for skippable blocks and disallow 0xfe (Padding) by @Jille in https://github.com/klauspost/compress/pull/916 https://github.com/klauspost/compress/pull/917 https://github.com/klauspost/compress/pull/919 https://github.com/klauspost/compress/pull/918
* huff0: Speed up symbol counting by @greatroar in https://github.com/klauspost/compress/pull/887 * huff0: Remove byteReader by @greatroar in https://github.com/klauspost/compress/pull/886 * gzhttp: Allow overriding decompression on transport https://github.com/klauspost/compress/pull/892 * gzhttp: Clamp compression level https://github.com/klauspost/compress/pull/890 * gzip: Error out if reserved bits are set https://github.com/klauspost/compress/pull/891
* fse: Fix max header size https://github.com/klauspost/compress/pull/881 * zstd: Improve better/best compression https://github.com/klauspost/compress/pull/877 * gzhttp: Fix missing content type on Close https://github.com/klauspost/compress/pull/883
zstd: Fix rare CORRUPTION* output in "best" mode. See https://github.com/klauspost/compress/pull/876
* s2: Fix S2 "best" dictionary wrong encoding https://github.com/klauspost/compress/pull/871 * flate: Reduce allocations in decompressor and minor code improvements by @fakefloordiv in https://github.com/klauspost/compress/pull/869 * s2: Fix EstimateBlockSize on 6&7 length input https://github.com/klauspost/compress/pull/867
* Add experimental dictionary builder https://github.com/klauspost/compress/pull/853 * Add xerial snappy read/writer https://github.com/klauspost/compress/pull/838 * flate: Add limited window compression https://github.com/klauspost/compress/pull/843 * s2: Do 2 overlapping match checks https://github.com/klauspost/compress/pull/839 * flate: Add amd64 assembly matchlen https://github.com/klauspost/compress/pull/837 * gzip: Copy bufio.Reader on Reset by @thatguystone in https://github.com/klauspost/compress/pull/860
</details> <details> <summary>See changes to v1.16.x</summary>
* zstd: Fix default level first dictionary encode https://github.com/klauspost/compress/pull/829 * s2: add GetBufferCapacity() method by @GiedriusS in https://github.com/klauspost/compress/pull/832
* zstd: correctly ignore WithEncoderPadding(1) by @ianlancetaylor in https://github.com/klauspost/compress/pull/806 * zstd: Add amd64 match length assembly https://github.com/klauspost/compress/pull/824 * gzhttp: Handle informational headers by @rtribotte in https://github.com/klauspost/compress/pull/815 * s2: Improve Better compression slightly https://github.com/klauspost/compress/pull/663
* zstd: readByte needs to use io.ReadFull by @jnoxon in https://github.com/klauspost/compress/pull/802 * gzip: Fix WriterTo after initial read https://github.com/klauspost/compress/pull/804
* zstd: Improve zstd best efficiency by @greatroar and @klauspost in https://github.com/klauspost/compress/pull/784 * zstd: Respect WithAllLitEntropyCompression https://github.com/klauspost/compress/pull/792 * zstd: Fix amd64 not always detecting corrupt data https://github.com/klauspost/compress/pull/785 * zstd: Various minor improvements by @greatroar in https://github.com/klauspost/compress/pull/788 https://github.com/klauspost/compress/pull/794 https://github.com/klauspost/compress/pull/795 * s2: Fix huge block overflow https://github.com/klauspost/compress/pull/779 * s2: Allow CustomEncoder fallback https://github.com/klauspost/compress/pull/780 * gzhttp: Support ResponseWriter Unwrap() in gzhttp handler by @jgimenez in https://github.com/klauspost/compress/pull/799
* zstd: Speed up + improve best encoder by @greatroar in https://github.com/klauspost/compress/pull/776 * gzhttp: Add optional BREACH mitigation. https://github.com/klauspost/compress/pull/762 https://github.com/klauspost/compress/pull/768 https://github.com/klauspost/compress/pull/769 https://github.com/klauspost/compress/pull/770 https://github.com/klauspost/compress/pull/767 * s2: Add Intel LZ4s converter https://github.com/klauspost/compress/pull/766 * zstd: Minor bug fixes https://github.com/klauspost/compress/pull/771 https://github.com/klauspost/compress/pull/772 https://github.com/klauspost/compress/pull/773 * huff0: Speed up compress1xDo by @greatroar in https://github.com/klauspost/compress/pull/774
* s2: Add Dictionary support. https://github.com/klauspost/compress/pull/685 * s2: Add Compression Size Estimate. https://github.com/klauspost/compress/pull/752 * s2: Add support for custom stream encoder. https://github.com/klauspost/compress/pull/755 * s2: Add LZ4 block converter. https://github.com/klauspost/compress/pull/748 * s2: Support io.ReaderAt in ReadSeeker. https://github.com/klauspost/compress/pull/747 * s2c/s2sx: Use concurrent decoding. https://github.com/klauspost/compress/pull/746 </details>
<details> <summary>See changes to v1.15.x</summary>
* deflate: Improve level 7-9 https://github.com/klauspost/compress/pull/739 * zstd: Add delta encoding support by @greatroar in https://github.com/klauspost/compress/pull/728 * zstd: Various speed improvements by @greatroar https://github.com/klauspost/compress/pull/741 https://github.com/klauspost/compress/pull/734 https://github.com/klauspost/compress/pull/736 https://github.com/klauspost/compress/pull/744 https://github.com/klauspost/compress/pull/743 https://github.com/klauspost/compress/pull/745 * gzhttp: Add SuffixETag() and DropETag() options to prevent ETag collisions on compressed responses by @willbicks in https://github.com/klauspost/compress/pull/740
* flate: Improve speed in big stateless blocks https://github.com/klauspost/compress/pull/718 * zstd: Minor speed tweaks by @greatroar in https://github.com/klauspost/compress/pull/716 https://github.com/klauspost/compress/pull/720 * export NoGzipResponseWriter for custom ResponseWriter wrappers by @harshavardhana in https://github.com/klauspost/compress/pull/722 * s2: Add example for indexing and existing stream https://github.com/klauspost/compress/pull/723
* zstd: Add MaxEncodedSize to encoder https://github.com/klauspost/compress/pull/691 * zstd: Various tweaks and improvements https://github.com/klauspost/compress/pull/693 https://github.com/klauspost/compress/pull/695 https://github.com/klauspost/compress/pull/696 https://github.com/klauspost/compress/pull/701 https://github.com/klauspost/compress/pull/702 https://github.com/klauspost/compress/pull/703 https://github.com/klauspost/compress/pull/704 https://github.com/klauspost/compress/pull/705 https://github.com/klauspost/compress/pull/706 https://github.com/klauspost/compress/pull/707 https://github.com/klauspost/compress/pull/708
* zstd: Tweak decoder allocs. https://github.com/klauspost/compress/pull/680
* gzhttp: Always delete HeaderNoCompression https://github.com/klauspost/compress/pull/683
* flate: Improve level 1-3 compression https://github.com/klauspost/compress/pull/678 * zstd: Improve "best" compression by @nightwolfz in https://github.com/klauspost/compress/pull/677 * zstd: Fix+reduce decompression allocations https://github.com/klauspost/compress/pull/668 * zstd: Fix non-effective noescape tag https://github.com/klauspost/compress/pull/667
* zstd: Add WithDecodeAllCapLimit https://github.com/klauspost/compress/pull/649 * Add Go 1.19 - deprecate Go 1.16 https://github.com/klauspost/compress/pull/651 * flate: Improve level 5+6 compression https://github.com/klauspost/compress/pull/656 * zstd: Improve "better" compression https://github.com/klauspost/compress/pull/657 * s2: Improve "best" compression https://github.com/klauspost/compress/pull/658 * s2: Improve "better" compression. https://github.com/klauspost/compress/pull/635 * s2: Slightly faster non-assembly decompression https://github.com/klauspost/compress/pull/646 * Use arrays for constant size copies https://github.com/klauspost/compress/pull/659
* zstd: Fix decoder crash on amd64 (no BMI) on invalid input https://github.com/klauspost/compress/pull/645 * zstd: Disable decoder extended memory copies (amd64) due to possible crashes https://github.com/klauspost/compress/pull/644 * zstd: Allow single segments up to "max decoded size" https://github.com/klauspost/compress/pull/643
* gzip: fix stack exhaustion bug in Reader.Read https://github.com/klauspost/compress/pull/641 * s2: Add Index header trim/restore https://github.com/klauspost/compress/pull/638 * zstd: Optimize seqdeq amd64 asm by @greatroar in https://github.com/klauspost/compress/pull/636 * zstd: Improve decoder memcopy https://github.com/klauspost/compress/pull/637 * huff0: Pass a single bitReader pointer to asm by @greatroar in https://github.com/klauspost/compress/pull/634 * zstd: Branchless getBits for amd64 w/o BMI2 by @greatroar in https://github.com/klauspost/compress/pull/640 * gzhttp: Remove header before writing https://github.com/klauspost/compress/pull/639
* s2: Fix absolute forward seeks https://github.com/klauspost/compress/pull/633 * zip: Merge upstream https://github.com/klauspost/compress/pull/631 * zip: Re-add zip64 fix https://github.com/klauspost/compress/pull/624 * zstd: translate fseDecoder.buildDtable into asm by @WojciechMula in https://github.com/klauspost/compress/pull/598 * flate: Faster histograms https://github.com/klauspost/compress/pull/620 * deflate: Use compound hcode https://github.com/klauspost/compress/pull/622
* s2: Improve coding for long, close matches https://github.com/klauspost/compress/pull/613 * s2c: Add Snappy/S2 stream recompression https://github.com/klauspost/compress/pull/611 * zstd: Always use configured block size https://github.com/klauspost/compress/pull/605 * zstd: Fix incorrect hash table placement for dict encoding in default https://github.com/klauspost/compress/pull/606 * zstd: Apply default config to ZipDecompressor without options https://github.com/klauspost/compress/pull/608 * gzhttp: Exclude more common archive formats https://github.com/klauspost/compress/pull/612 * s2: Add ReaderIgnoreCRC https://github.com/klauspost/compress/pull/609 * s2: Remove sanity load on index creation https://github.com/klauspost/compress/pull/607 * snappy: Use dedicated function for scoring https://github.com/klauspost/compress/pull/614 * s2c+s2d: Use official snappy framed extension https://github.com/klauspost/compress/pull/610
* s2: Add concurrent stream decompression https://github.com/klauspost/compress/pull/602 * s2: Fix final emit oob read crash on amd64 https://github.com/klauspost/compress/pull/601 * huff0: asm implementation of Decompress1X by @WojciechMula https://github.com/klauspost/compress/pull/596 * zstd: Use 1 less goroutine for stream decoding https://github.com/klauspost/compress/pull/588 * zstd: Copy literal in 16 byte blocks when possible https://github.com/klauspost/compress/pull/592 * zstd: Speed up when WithDecoderLowmem(false) https://github.com/klauspost/compress/pull/599 * zstd: faster next state update in BMI2 version of decode by @WojciechMula in https://github.com/klauspost/compress/pull/593 * huff0: Do not check max size when reading table. https://github.com/klauspost/compress/pull/586 * flate: Inplace hashing for level 7-9 https://github.com/klauspost/compress/pull/590
* huff0: decompress directly into output by @WojciechMula in #577 * inflate: Keep dict on stack #581 * zstd: Faster decoding memcopy in asm #583 * zstd: Fix ignored crc #580
* zstd: Allow to ignore checksum checking by @WojciechMula #572 * s2: Fix incorrect seek for io.SeekEnd in #575
* zstd: Add x86-64 assembly for decompression on streams and blocks. Contributed by @WojciechMula. Typically 2x faster. #528 #531 #545 #537 * zstd: Add options to ZipDecompressor and fixes #539 * s2: Use sorted search for index #555 * Minimum version is Go 1.16, added CI test on 1.18.
* huff0: Add x86 assembly of Decode4X by @WojciechMula in #512 * zstd: Reuse zip decoders in #514 * zstd: Detect extra block data and report as corrupted in #520 * zstd: Handle zero sized frame content size stricter in #521 * zstd: Add stricter block size checks in #523
* zstd: Refactor decoder #498 * zstd: Add stream encoding without goroutines #505 * huff0: Prevent single blocks exceeding 16 bits by @klauspost in#507 * flate: Inline literal emission #509 * gzhttp: Add zstd to transport #400 * gzhttp: Make content-type optional #510
Both compression and decompression now supports "synchronous" stream operations. This means that whenever "concurrency" is set to 1, they will operate without spawning goroutines.
Stream decompression is now faster on asynchronous, since the goroutine allocation much more effectively splits the workload. On typical streams this will typically use 2 cores fully for decompression. When a stream has finished decoding no goroutines will be left over, so decoders can now safely be pooled and still be garbage collected.
While the release has been extensively tested, it is recommended to testing when upgrading.
</details>
<details> <summary>See changes to v1.14.x</summary>
* flate: Fix rare huffman only (-2) corruption. #503 * zip: Update deprecated CreateHeaderRaw to correctly call CreateRaw by @saracen in #502 * zip: don't read data descriptor early by @saracen in #501 #501 * huff0: Use static decompression buffer up to 30% faster #499 #500
* flate: Improve fastest levels compression speed ~10% more throughput. #482 #489 #490 #491 #494 #478 * flate: Faster decompression speed, ~5-10%. #483 * s2: Faster compression with Go v1.18 and amd64 microarch level 3+. #484 #486
* zstd: improve header decoder by @dsnet #476 * zstd: Add bigger default blocks #469 * zstd: Remove unused decompression buffer #470 * zstd: Fix logically dead code by @ningmingxiao #472 * flate: Improve level 7-9 #471 #473 * zstd: Add noasm tag for xxhash #475
* s2: Add stream index in #462 * flate: Speed and efficiency improvements in #439 #461 #455 #452 #458 * zstd: Performance improvement in #420 #456 #437 #467 #468 * zstd: add arm64 xxhash assembly in #464 * Add garbled for binaries for s2 in #445 </details>
<details> <summary>See changes to v1.13.x</summary>
* gz/zlib/flate: Alias stdlib errors #425 * s2: Add block support to commandline tools #413 * zstd: pooledZipWriter should return Writers to the same pool #426 * Removed golang/snappy as external dependency for tests #421
* Add snappy replacement package. * zstd: Fix incorrect encoding in "best" mode #415
* zstd: Improve Best compression #404 * zstd: Fix WriteTo error forwarding #411 * gzhttp: Return http.HandlerFunc instead of http.Handler. Unlikely breaking change. #406 * s2sx: Fix max size error #399 * zstd: Add optional stream content size on reset #401 * zstd: use SpeedBestCompression for level >= 10 #410
* s2: Add full Snappy output support #396 * zstd: Add configurable Decoder window size #394 * gzhttp: Add header to skip compression #389 * s2: Improve speed with bigger output margin #395
* Added gzhttp which allows wrapping HTTP servers and clients with GZIP compressors. * zstd: Detect short invalid signatures #382 * zstd: Spawn decoder goroutine only if needed. #380 </details>
<details> <summary>See changes to v1.12.x</summary>
* deflate: Better/faster Huffman encoding #374 * deflate: Allocate less for history. #375 * zstd: Forward read errors #373
* zstd: Improve better/best compression #360 #364 #365 * zstd: Add helpers to compress/decompress zstd inside zip files #363 * deflate: Improve level 5+6 compression #367 * s2: Improve better/best compression #358 #359 * s2: Load after checking src limit on amd64. #362 * s2sx: Limit max executable size #368
* snappy package removed. Upstream added as dependency. * s2: Better compression in "best" mode #353 * s2sx: Add stdin input and detect pre-compressed from signature #352 * s2c/s2d: Add http as possible input #348 * s2c/s2d/s2sx: Always truncate when writing files #352 * zstd: Reduce memory usage further when using WithLowerEncoderMem #346 * s2: Fix potential problem with amd64 assembly and profilers #349 </details>
<details> <summary>See changes to v1.11.x</summary>
* zstd: Big speedup on small dictionary encodes #344 #345 * zstd: Add WithLowerEncoderMem encoder option #336 * deflate: Improve entropy compression #338 * s2: Clean up and minor performance improvement in best #341
* s2: Add s2sx binary that creates self extracting archives.
* s2: Speed up decompression on non-assembly platforms #328
* s2: Add ARM64 decompression assembly. Around 2x output speed. #324 * s2: Improve "better" speed and efficiency. #325 * s2: Fix binaries.
* s2: Fixed occasional out-of-bounds write on amd64. Upgrade recommended. * s2: Add AMD64 assembly for better mode. 25-50% faster. #315 * s2: Less upfront decoder allocation. #322 * zstd: Faster "compression" of incompressible data. #314 * zip: Fix zip64 headers. #313
* Use Bytes() interface to get bytes across packages. #309
* s2: Add 'best' compression option. #310
* s2: Add ReaderMaxBlockSize, changes s2.NewReader signature to include varargs. #311
* s2: Fix crash on small better buffers. #308
* s2: Clean up decoder. #312
* zstd: Make decoder allocations smaller #306 * zstd: Free Decoder resources when Reset is called with a nil io.Reader #305
* zstd: Add Best compression mode #304 * Add header decoder #299 * s2: Add uncompressed stream option #297 * Simplify/speed up small blocks with known max size. #300 * zstd: Always reset literal dict encoder #303
* inflate: 10-15% faster decompression #293 * zstd: Tweak DecodeAll default allocation #295
* s2: Fix out of bounds read in "better" block compression #291
* zstd: Set allLitEntropy true in default configuration #286
* zstd: Add experimental compression dictionaries #281 * zstd: Fix mixed Write and ReadFrom calls #282 * inflate/gz: Limit variable shifts, ~5% faster decompression #274 </details>
<details> <summary>See changes to v1.10.x</summary>
* zstd: Fix extra block when compressing with ReadFrom. #278 * huff0: Also populate compression table when reading decoding table. #275
* zstd: Skip entropy compression in fastest mode when no matches. #270
* zstd: API change for specifying dictionaries. See #268 * zip: update CreateHeaderRaw to handle zip64 fields. #266 * Fuzzit tests removed. The service has been purchased and is no longer available.
* 1.15x faster zstd block decompression. #265
* Added zstd decompression dictionary support * Increase zstd decompression speed up to 1.19x. #259 * Remove internal reset call in zstd compression and reduce allocations. #263
* zstd: Reduce allocations while decoding. #258, #252 * zstd: Stricter decompression checks.
* s2-commands: Flush output when receiving SIGINT. #239
* zstd: Minor/special case optimizations. #251, #250, #249, #247
* s2: Use S2 encoder in pure Go mode for Snappy output as well. #245 * s2: Fix pure Go block encoder. #244 * zstd: Added "better compression" mode. #240 * zstd: Improve speed of fastest compression mode by 5-10% #241 * zstd: Skip creating encoders when not needed. #238
* Close to 50% speedup in inflate (gzip/zip decompression). #236 #234 #232 * Reduce deflate level 1-6 memory usage up to 59%. #227
* Fix zstd crash when resetting multiple times without sending data. #226 * deflate: Fix dictionary use on level 1-6. #224 * Remove deflate writer reference when closing. #224
* Add optional dictionary to stateless deflate. Breaking change, send nil for previous behaviour. #216
* Fix buffer overflow on repeated small block deflate. #218
* Allow copying content from an existing ZIP file without decompressing+compressing. #214
* Added S2 AMD64 assembler and various optimizations. Stream speed >10GB/s. #186
</details>
<details> <summary>See changes prior to v1.10.0</summary>
</details>
<details> <summary>See changes prior to v1.9.0</summary>
-rm (remove source files) and -q (no output except errors) to s2c and s2d commands
s2c and s2d commandline tools.
</details>
The packages are drop-in replacements for standard libraries. Simply replace the import path to use them:
Typical speed is about 2x of the standard library packages.
| old import | new import | Documentation |
|---|---|---|
compress/gzip | github.com/klauspost/compress/gzip | gzip |
compress/zlib | github.com/klauspost/compress/zlib | zlib |
archive/zip | github.com/klauspost/compress/zip | zip |
compress/flate | github.com/klauspost/compress/flate | flate |
You may also be interested in pgzip, which is a drop in replacement for gzip, which support multithreaded compression on big files and the optimized crc32 package used by these packages.
The packages contains the same as the standard library, so you can use the godoc for that: gzip, zip, zlib, flate.
Currently there is only minor speedup on decompression (mostly CRC32 calculation).
Memory usage is typically 1MB for a Writer. stdlib is in the same range. If you expect to have a lot of concurrently allocated Writers consider using the stateless compress described below.
For compression performance, see: this spreadsheet.
To disable all assembly add -tags=noasm. This works across all packages.
This package offers stateless compression as a special option for gzip/deflate. It will do compression but without maintaining any state between Write calls.
This means there will be no memory kept between Write calls, but compression and speed will be suboptimal.
This is only relevant in cases where you expect to run many thousands of compressors concurrently, but with very little activity. This is not intended for regular web servers serving individual requests.
Because of this, the size of actual Write calls will affect output size.
In gzip, specify level -3 / gzip.StatelessCompression to enable.
For direct deflate use, NewStatelessWriter and StatelessDeflate are available. See documentation
A bufio.Writer can of course be used to control write sizes. For example, to use a 4KB buffer:
// replace 'ioutil.Discard' with your output.
gzw, err := gzip.NewWriterLevel(ioutil.Discard, gzip.StatelessCompression)
if err != nil {
return err
}
defer gzw.Close()
w := bufio.NewWriterSize(gzw, 4096)
defer w.Flush()
// Write to 'w'
This will only use up to 4KB in memory when the writer is idle.
Compression is almost always worse than the fastest compression level and each write will allocate (a little) memory.
Here are other packages of good quality and pure Go (no cgo wrappers or autoconverted code):
This code is licensed under the same conditions as the original Go code. See LICENSE file.