Skip to content

[reopen] Add std::compression::zlib - #3663

Open
Berat121 wants to merge 9 commits into
c3lang:masterfrom
Berat121:faster-zlib
Open

Berat121 wants to merge 9 commits into
c3lang:masterfrom
Berat121:faster-zlib

Conversation

@Berat121

Copy link
Copy Markdown
Contributor

Adds std::compression::zlib (RFC 1950): compress, compress_stream, uncompress, uncompress_stream, and ZlibWriter/ZlibReader streams. It is a thin layer over the DEFLATE code from #3624 (it uses its levels) and the adler32 from #3621. zlib streams are what PNG and PDF FlateDecode use, and what most network protocols and file formats mean by "deflate". Nothing existing changes, this only adds a module.

Changes
  • lib/std/compression/zlib.c3 (new), release notes line, test/unit/stdlib/compression/zlib.c3 (new).
  • API mirrors gzip: compress(Allocator, char[] src, int level = deflate::DEFAULT_LEVEL), compress_stream(OutStream, char[], int level = ...), uncompress(Allocator, char[]), uncompress_stream(OutStream, InStream).
  • Header is 78 xx (deflate, 32 KiB window) with a valid FCHECK. FLEVEL is a hint derived from level: 1 -> 0, below the default -> 1, the default -> 2 (78 9C), above -> 3. The writer appends the big endian Adler-32 on close.
  • The reader checks CM, CINFO and FCHECK, and verifies the Adler-32 at the end. Truncated or corrupt data is an error (CORRUPT_HEADER, CORRUPT_DATA, UNSUPPORTED_METHOD, named like the gzip ones), never a wrong result.
  • It reads one stream. When the source can seek, it is left right after the Adler-32, so data that follows the stream can be read by the caller.
  • A preset dictionary (FDICT) is not supported: reading such a stream returns DICTIONARY_REQUIRED, and the writer never sets FDICT.
Benchmark

MB/s of uncompressed data, one shot, mean over 4 files (1.8 MB C3 sources, 3.0 MB C sources, 16 MiB ELF (libLLVM), 16 MB tar of /usr/include), best of 5 runs per file, -O3. 1 core x86-64 (Xeon, default c3c x86-64 target), Ubuntu 24.04, LLVM 20, c3c 0.8.6-pre. Run to run noise on this VM is about 5-10%.

zlib::compress deflate::compress zlib::uncompress deflate decompress zlib 1.3 uncompress
MB/s 85.7 90.9 675.5 686.7 450.5

The zlib layer is about the same speed as raw deflate, the extra cost is the Adler-32 and the header/trailer handling. The tool is zbench.c3 in the gist from #3624: https://gist.github.com/Berat121/8f7bacb73adc4f684a41e550ba229e20

Testing
  • 9 new unit tests: header errors and the FLEVEL hint, vectors made by Python's zlib, round trips at all levels and several sizes, writer with chunked writes, reader with tiny reads, a source without seek, data left in the source after the stream, truncated and corrupt data.
  • Full unit suite (compile-test unit -O1 -D SLOW_TESTS -D RUN_PROCESS_TESTS) on current master plus this commit: 1948/1948, no warnings. The zlib tests also pass at -O0, -O3 and with ASAN (--sanitize=address -O1).
  • Python zlib as an oracle (zlib_interop.py in the gist): our output at levels 1-9 and the default (one shot and chunked), streams from Python compressobj (levels 0-9, windows 9/12/15, also read with 1 byte reads), and 1800 corrupted / truncated / trailing-garbage streams: success or failure always matched Python, 0 differences.
  • linux-x86, linux-aarch64 and wasm32 compile-only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants