fbpx
Wikipedia

Zstandard

".zst" redirects here. For the game console emulator also using this file extension, see ZSNES.

This article relies too much on references to primary sources. Please improve this by adding secondary or tertiary sources.(September 2017) ()

Zstandard (or zstd) is a lossless data compression algorithm developed by Yann Collet at Facebook. Zstd is the reference implementation in C. Version 1 of this implementation was released as open-source software on 31 August 2016.

Zstandard
Original author(s)Yann Collet
Developer(s)Yann Collet, Nick Terrell, Przemysław Skibiński
Initial release23 January 2015 (2015-01-23)
Stable release
1.5.0 / 14 May 2021; 5 months ago (2021-05-14)
Repository
Written inC
Operating systemCross-platform
PlatformPortable
TypeData compression
LicenseDual: BSD License, GPLv2
Websitefacebook.github.io/zstd/

Contents

Zstandard was designed to give a compression ratio comparable to that of the DEFLATE algorithm (developed in 1991 and used in the original ZIP and gzip programs), but faster, especially for decompression. It is tunable with compression levels ranging from negative 7 (fastest) to 22 (slowest in compression speed, but best compression ratio).

The zstd package includes parallel (multi-threaded) implementations of both compression and decompression[citation needed]. Starting from version 1.3.2 (October 2017), zstd optionally implements very long range search and deduplication (--long, 128 MiB window) similar to rzip or lrzip.

Compression speed can vary by a factor of 20 or more between the fastest and slowest levels, while decompression is uniformly fast, varying by less than 20% between the fastest and slowest levels. Zstandard command-line has an "adaptive" (--adapt) mode that varies compression level depending on I/O conditions, mainly how fast it can write the output.

Zstd at its maximum compression level gives a compression ratio close to lzma, lzham, and ppmx, and performs better than lza, or bzip2. Zstandard reaches the current Pareto frontier, as it decompresses faster than any other currently-available algorithm with similar or better compression ratio.

Dictionaries can have a large impact on the compression ratio of small files, so Zstandard can use a user-provided compression dictionary. It also offers a training mode, able to generate a dictionary from a set of samples. In particular, one dictionary can be loaded to process large sets of files with redundancy between files, but not necessarily within each file, e.g., log files.

Zstandard combines a dictionary-matching stage (LZ77) with a large search window and a fast entropy coding stage, using both Finite State Entropy (a fast tabled version of ANS, tANS, used for entries in the Sequences section), and Huffman coding (used for entries in the Literals section).

Because of the way that FSE carries over state between symbols, decompression involves processing symbols within the Sequences section of each block in reverse order (from last to first).

The Linux kernel has included Zstandard since November 2017 (version 4.14) as a compression method for the btrfs and squashfs filesystems.

In 2017, Allan Jude integrated Zstandard into the FreeBSD kernel and it was subsequently integrated as a compressor option for core dumps (both user programs and kernel panics). It was also used to create a proof of concept OpenZFS compression method which was integrated in 2020.

The AWS Redshift and RocksDB databases include support for field compression using Zstandard.

In March 2018, Canonical tested the use of zstd as a deb package compression method by default for the Ubuntu Linux distribution. Compared with xz compression of deb packages, zstd at level 19 decompresses significantly faster, but at the cost of 6% larger package files. Debian developer Ian Jackson favored waiting several years before official adoption.

In 2018 the algorithm was published as RFC 8478, which also defines an associated media type "application/zstd", filename extension "zst", and HTTP content encoding "zstd".

Arch Linux added support for zstd as a package compression method in October 2019 with the release of the pacman 5.2 package manager, and in January 2020 switched from xz to zstd for the packages in the official repository. Arch uses zstd -c -T0 --ultra -20 -, the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 14 times faster, decompression memory increased by 50 MiB when using multiple threads, compression memory increases but scales with the number of threads used.

Fedora added ZStandard support to RPM in May 2018 (Fedora release 28), and used it for packaging the release in October 2019 (Fedora 31)

Full implementation of the algorithm with an option to choose the compression level is used in the .NSZ / .XCZ file formats, developed by the homebrew community for the Nintendo Switch hybrid game console.

7-Zip ZS, a port of 7-Zip FM with Zstandard (and other formats) support, is developed by Tino Reichardt.

Modern7z, a Zstandard (and other formats) plugin for 7-Zip FM is developed by Denis Anisimov (TC4shell).

7-zip-zstd, a fork of 7-zip with support for Zstandard.

The reference implementation is licensed under the BSD license, published at GitHub. Since version 1.0, it had an additional Grant of Patent Rights.

From version 1.3.1, this patent grant was dropped and the license was changed to a BSD + GPLv2 dual license.

  1. "Contributors to facebook/zstd". github.com. Archived from the original on 27 January 2021. Retrieved26 January 2021.
  2. "Releases - facebook/zstd". Retrieved16 May 2021 – via GitHub.
  3. Sergio De Simone (2 September 2016). "Facebook Open-Sources New Compression Algorithm Outperforming Zlib". InfoQ. Retrieved20 April 2019.
  4. "Life imitates satire: Facebook touts zlib killer just like Silicon Valley's Pied Piper". The Register. 31 August 2016. Retrieved6 September 2016.
  5. https://github.com/facebook/zstd/releases/tag/v1.3.4 Faster compression levels
  6. "Command Line Interface for Zstandard library". GitHub. 28 October 2021.
  7. "ZStandard in ZFS"(PDF). open-zfs.org. 2017. Retrieved20 April 2019.
  8. Matt Mahoney. "Silesia Open Source Compression Benchmark". Retrieved10 May 2019.
  9. Matt Mahoney (29 August 2016). "Large Text Compression Benchmark, .2157 zstd". Retrieved1 September 2016.
  10. TurboBench: Static/Dynamic web content compression benchmark, PowTurbo
  11. Matt Mahoney, Silesia Open Source Compression Benchmark
  12. "Facebook developers report massive speedups and compression ratio improvements when using dictionaries"(PDF).
  13. "Smaller and faster data compression with Zstandard". Facebook. 31 August 2016.
  14. "facebook/zstd". GitHub. 28 October 2021.
  15. Collet, Yann (October 2018). Kucherawy, Murray S. (ed.). Zstandard Compression and the application/zstd Media Type. Internet Engineering Task Force Request for Comments. doi:10.17487/RFC8478. RFC 8478. Retrieved7 October 2020.
  16. "The rest of the 4.14 merge window [LWN.net]". lwn.net.
  17. "Linux_4.14 - Linux Kernel Newbies". Kernelnewbies.org. Retrieved16 August 2018.
  18. "Zstd Compression For Btrfs & Squashfs Set For Linux 4.14, Already Used Within Facebook - Phoronix". www.phoronix.com.
  19. https://github.com/freebsd/freebsd/commit/28ef16535cde21eeeaf75f6006b3a77952b3b51
  20. "Add ZSTD support to ZFS · openzfs/[email protected]".
  21. "Zstandard Encoding - Amazon Redshift". 20 April 2019.
  22. Larabel, Michael (12 March 2018). "Canonical Working On Zstd-Compressed Debian Packages For Ubuntu". phoronix.com. Phoronix Media. Retrieved29 October 2019. The developers at Canonical are considering a feature freeze exception to get this newly-developed Zstd Apt/Dpkg support in Ubuntu 18.04 LTS. In doing so, they mention they would be looking at enabling Zstd compression for packages by default in Ubuntu 18.10.
  23. "New Ubuntu Installs Could Be Speed Up by 10% with the Zstd Compression Algorithm". Softpedia. 12 March 2018. Retrieved13 August 2018.
  24. "Canonical Working On Zstd-Compressed Debian Packages For Ubuntu". phoronix. 12 March 2018. Retrieved13 August 2018.
  25. Jackson, Ian (27 April 2018). "RFC: Support for zstd in .deb packages?". debian-devel (Mailing list).
  26. "Arch Linux Nears Roll-Out of ZSTD Compressed Packages for Faster Pacman Installs - Phoronix".
  27. Broda, Robin (4 January 2020). "Now using Zstandard instead of xz for package compression". Arch Linux. Retrieved5 January 2020.
  28. Broda, Robin (25 March 2019). "RFC: (devtools) Changing default compression method to zstd". arch-dev-public (Mailing list).
  29. Broda, Robin; Polyak, Levente (27 December 2019). "makepkg.conf: change default compression method to zstd".
  30. "Changes/Switch RPMS to ZSTD compression - Fedora Project Wiki".
  31. "RELEASE - nsZip - NSP compressor/decompressor to reduce storage". GBAtemp.net - The Independent Video Game Community. Retrieved3 November 2019.
  32. Bosshard, Nico (31 October 2019), nsZip is a tool to compress/decompress Nintendo Switch games using the here specified NSZ file format: nicoboss/nsZip, retrieved3 November 2019
  33. "Milkys Homepage - 7-Zip with support for Zstandard, Brotli, Lz4, Lz5 and Lizard Compression".
  34. "Modern7z".
  35. "README". 28 October 2021.
  36. "Facebook open sources Zstandard data compression algorithm, aims to replace technology behind Zip". ZDnet. 31 August 2016. Retrieved1 September 2016.
  37. zstd/PATENTS "Additional Grant of Patent Rights Version 2", Facebook
  38. "Zstd v1.3.1 release", GitHub "facebook/zstd"
  39. "New license", GitHub "facebook/zstd"

Zstandard
Zstandard Language Watch Edit zst redirects here For the game console emulator also using this file extension see ZSNES This article relies too much on references to primary sources Please improve this by adding secondary or tertiary sources September 2017 Learn how and when to remove this template message Zstandard or zstd is a lossless data compression algorithm developed by Yann Collet at Facebook Zstd is the reference implementation in C Version 1 of this implementation was released as open source software on 31 August 2016 3 4 ZstandardOriginal author s Yann ColletDeveloper s Yann Collet Nick Terrell Przemyslaw Skibinski 1 Initial release23 January 2015 2015 01 23 Stable release1 5 0 14 May 2021 5 months ago 2021 05 14 2 Repositorygithub wbr com wbr facebook wbr zstd wbr gitWritten inCOperating systemCross platformPlatformPortableTypeData compressionLicenseDual BSD License GPLv2Websitefacebook wbr github wbr io wbr zstd wbr Contents 1 Features 2 Design 3 Usage 4 License 5 See also 6 References 7 External linksFeatures EditZstandard was designed to give a compression ratio comparable to that of the DEFLATE algorithm developed in 1991 and used in the original ZIP and gzip programs but faster especially for decompression It is tunable with compression levels ranging from negative 7 fastest 5 to 22 slowest in compression speed but best compression ratio The zstd package includes parallel multi threaded implementations of both compression and decompression citation needed Starting from version 1 3 2 October 2017 zstd optionally implements very long range search and deduplication long 128 MiB window similar to rzip or lrzip 6 Compression speed can vary by a factor of 20 or more between the fastest and slowest levels while decompression is uniformly fast varying by less than 20 between the fastest and slowest levels 7 Zstandard command line has an adaptive adapt mode that varies compression level depending on I O conditions mainly how fast it can write the output Zstd at its maximum compression level gives a compression ratio close to lzma lzham and ppmx and performs better than lza or bzip2 8 9 Zstandard reaches the current Pareto frontier as it decompresses faster than any other currently available algorithm with similar or better compression ratio 10 11 Dictionaries can have a large impact on the compression ratio of small files so Zstandard can use a user provided compression dictionary It also offers a training mode able to generate a dictionary from a set of samples 12 13 In particular one dictionary can be loaded to process large sets of files with redundancy between files but not necessarily within each file e g log files Design EditZstandard combines a dictionary matching stage LZ77 with a large search window and a fast entropy coding stage using both Finite State Entropy a fast tabled version of ANS tANS used for entries in the Sequences section and Huffman coding used for entries in the Literals section 14 Because of the way that FSE carries over state between symbols decompression involves processing symbols within the Sequences section of each block in reverse order from last to first Usage EditZstandardFilename extension zst 15 Internet media typeapplication zstd 15 Magic number28 b5 2f fd 15 Type of formatData compressionStandardRFC 8478Websitehttps github com facebook zstd blob dev doc zstd compression format mdZstandard DictionaryInternet media typeapplication zstandardMagic number37 a4 30 ec 15 StandardRFC 8478Websitehttps github com facebook zstd blob dev doc zstd compression format md dictionary format The Linux kernel has included Zstandard since November 2017 version 4 14 as a compression method for the btrfs and squashfs filesystems 16 17 18 In 2017 Allan Jude integrated Zstandard into the FreeBSD kernel 19 and it was subsequently integrated as a compressor option for core dumps both user programs and kernel panics It was also used to create a proof of concept OpenZFS compression method 7 which was integrated in 2020 20 The AWS Redshift and RocksDB databases include support for field compression using Zstandard 21 In March 2018 Canonical tested 22 the use of zstd as a deb package compression method by default for the Ubuntu Linux distribution Compared with xz compression of deb packages zstd at level 19 decompresses significantly faster but at the cost of 6 larger package files Debian developer Ian Jackson favored waiting several years before official adoption 23 24 25 In 2018 the algorithm was published as RFC 8478 which also defines an associated media type application zstd filename extension zst and HTTP content encoding zstd 15 Arch Linux added support for zstd as a package compression method in October 2019 with the release of the pacman 5 2 package manager 26 and in January 2020 switched from xz to zstd for the packages in the official repository Arch uses zstd c T0 ultra 20 the size of all compressed packages combined increased by 0 8 compared to xz the decompression speed is 14 times faster decompression memory increased by 50 MiB when using multiple threads compression memory increases but scales with the number of threads used 27 28 29 Fedora added ZStandard support to RPM in May 2018 Fedora release 28 and used it for packaging the release in October 2019 Fedora 31 30 Full implementation of the algorithm with an option to choose the compression level is used in the NSZ XCZ 31 file formats developed by the homebrew community for the Nintendo Switch hybrid game console 32 7 Zip ZS a port of 7 Zip FM with Zstandard and other formats support is developed by Tino Reichardt 33 Modern7z a Zstandard and other formats plugin for 7 Zip FM is developed by Denis Anisimov TC4shell 34 7 zip zstd a fork of 7 zip with support for Zstandard 35 License EditThe reference implementation is licensed under the BSD license published at GitHub 36 Since version 1 0 it had an additional Grant of Patent Rights 37 From version 1 3 1 38 this patent grant was dropped and the license was changed to a BSD GPLv2 dual license 39 See also EditZlib LZFSE a similar algorithm by Apple used since iOS 9 and OS X 10 11 made open source on 1 June 2016 LZ4 compression algorithm a fast member of the LZ77 familyReferences Edit Contributors to facebook zstd github com Archived from the original on 27 January 2021 Retrieved 26 January 2021 Releases facebook zstd Retrieved 16 May 2021 via GitHub Sergio De Simone 2 September 2016 Facebook Open Sources New Compression Algorithm Outperforming Zlib InfoQ Retrieved 20 April 2019 Life imitates satire Facebook touts zlib killer just like Silicon Valley s Pied Piper The Register 31 August 2016 Retrieved 6 September 2016 https github com facebook zstd releases tag v1 3 4 Faster compression levels Command Line Interface for Zstandard library GitHub 28 October 2021 a b ZStandard in ZFS PDF open zfs org 2017 Retrieved 20 April 2019 Matt Mahoney Silesia Open Source Compression Benchmark Retrieved 10 May 2019 Matt Mahoney 29 August 2016 Large Text Compression Benchmark 2157 zstd Retrieved 1 September 2016 TurboBench Static Dynamic web content compression benchmark PowTurbo Matt Mahoney Silesia Open Source Compression Benchmark Facebook developers report massive speedups and compression ratio improvements when using dictionaries PDF Smaller and faster data compression with Zstandard Facebook 31 August 2016 facebook zstd GitHub 28 October 2021 a b c d e Collet Yann October 2018 Kucherawy Murray S ed Zstandard Compression and the application zstd Media Type Internet Engineering Task Force Request for Comments doi 10 17487 RFC8478 RFC 8478 Retrieved 7 October 2020 The rest of the 4 14 merge window LWN net lwn net Linux 4 14 Linux Kernel Newbies Kernelnewbies org Retrieved 16 August 2018 Zstd Compression For Btrfs amp Squashfs Set For Linux 4 14 Already Used Within Facebook Phoronix www phoronix com https github com freebsd freebsd commit 28ef16535cde21eeeaf75f6006b3a77952b3b51 Add ZSTD support to ZFS openzfs ZFS 10b3c7f Zstandard Encoding Amazon Redshift 20 April 2019 Larabel Michael 12 March 2018 Canonical Working On Zstd Compressed Debian Packages For Ubuntu phoronix com Phoronix Media Retrieved 29 October 2019 The developers at Canonical are considering a feature freeze exception to get this newly developed Zstd Apt Dpkg support in Ubuntu 18 04 LTS In doing so they mention they would be looking at enabling Zstd compression for packages by default in Ubuntu 18 10 New Ubuntu Installs Could Be Speed Up by 10 with the Zstd Compression Algorithm Softpedia 12 March 2018 Retrieved 13 August 2018 Canonical Working On Zstd Compressed Debian Packages For Ubuntu phoronix 12 March 2018 Retrieved 13 August 2018 Jackson Ian 27 April 2018 RFC Support for zstd in deb packages debian devel Mailing list Arch Linux Nears Roll Out of ZSTD Compressed Packages for Faster Pacman Installs Phoronix Broda Robin 4 January 2020 Now using Zstandard instead of xz for package compression Arch Linux Retrieved 5 January 2020 Broda Robin 25 March 2019 RFC devtools Changing default compression method to zstd arch dev public Mailing list Broda Robin Polyak Levente 27 December 2019 makepkg conf change default compression method to zstd Changes Switch RPMS to ZSTD compression Fedora Project Wiki RELEASE nsZip NSP compressor decompressor to reduce storage GBAtemp net The Independent Video Game Community Retrieved 3 November 2019 Bosshard Nico 31 October 2019 nsZip is a tool to compress decompress Nintendo Switch games using the here specified NSZ file format nicoboss nsZip retrieved 3 November 2019 Milkys Homepage 7 Zip with support for Zstandard Brotli Lz4 Lz5 and Lizard Compression Modern7z README 28 October 2021 Facebook open sources Zstandard data compression algorithm aims to replace technology behind Zip ZDnet 31 August 2016 Retrieved 1 September 2016 zstd PATENTS Additional Grant of Patent Rights Version 2 Facebook Zstd v1 3 1 release GitHub facebook zstd New license GitHub facebook zstd External links EditOfficial website zstd on GitHub 7zip with Zstandard on GitHub Smaller and faster data compression with Zstandard Yann Collet and Chip Turner 31 August 2016 Facebook Announcement The Guardian is using ZStandard instead of zlib Retrieved from https en wikipedia org w index php title Zstandard amp oldid 1052387534, wikipedia, wiki, book,

books

, library,

article

, read, download, free, free download, mp3, video, mp4, 3gp, jpg, jpeg, gif, png, picture, music, song, movie, book, game, games.