Project Pack made an attempt to create a file archiving format based on the SQLite library and the ZSTD (Zstandard) compression algorithm. Prepared prototype written in Pascal and distributed under the Apache 2.0 license, it outperformed the most common archivers in the speed of creating archives, despite the fact that its work was limited to reading data, compressing it with a library libzstd and performing SQL operations on adding compressed data into a file with a SQLite database.
When compressing a directory with 81 thousand files with a total size of 1.25 GB, pack was 112 times faster than the ZIP utility, completing the operation in 1.3 seconds versus 146 seconds for ZIP. The archive size for pack was 23% smaller (194 MB for Pack and 253 MB for ZIP). For comparison, the tar utility completed the packaging in 4.7 seconds without compression and in 28.5 seconds with gzip compression, the RAR archiver completed the test in 27.5 seconds, and 7z in 54.2 seconds. The archive sizes were: tar.gz – 214 MB, RAR – 235 MB, 7z – 135 MB. It is noted that in terms of speed of unpacking and random access to files, Pack is also ahead of other archivers, while consuming less RAM.
ZIP: 253 MB, 146 s 7z: 135 MB, 54.2 s быстрее ZIP в 2.7 раза tar.gz: 214 MB, 28.5 s x 5.1 RAR: 235 MB, 27.5 s x 5.3 tar: 1345 MB, 4.7 s x 31 Pack: 194 MB, 1.3 s x 112
There is no mention of the effect of the file cache on the test results. Probably, the low speed of ZIP is due to the order in which the tests were launched without regard to data caching in memory – the test with zip was launched with a cold cache, and the rest of the tests with a warm cache. Under normal conditions, Zstandard exhibits 3-5 times faster compression speeds than zlib and 2 times faster decompression, with 10-15% higher compression levels.
Thanks for reading: