Zstd announcement by erikjee · Pull Request #76 · trifectatechfoundation/trifectatech-website

erikjee · 2026-05-27T10:01:01Z

No description provided.

erikjee · 2026-05-27T10:06:46Z

+
+Using zstd in rust is already supported via the [zstd](https://crates.io/crates/zstd) crate, so why bother with a whole new implementation?
+
+One practical advantage of rust is that it is straightforward to write portable software.  Currently the `zstd` crate compiles the C code from source, which requires a C toolchain for the target and for the target to be supported at all. Setting up a C toolchain on windows or for webassembly can be a challenge, and isn't needed with a pure-rust implementation. For rust programmers, using dependencies written in rust is a superior experience.


Suggested change

One practical advantage of rust is that it is straightforward to write portable software. Currently the `zstd` crate compiles the C code from source, which requires a C toolchain for the target and for the target to be supported at all. Setting up a C toolchain on windows or for webassembly can be a challenge, and isn't needed with a pure-rust implementation. For rust programmers, using dependencies written in rust is a superior experience.

### Portability

One practical advantage of rust is that it is straightforward to write portable software. Currently the `zstd` crate compiles the C code from source, which requires a C toolchain for the target and for the target to be supported at all. Setting up a C toolchain on windows or for webassembly can be a challenge, and isn't needed with a pure-rust implementation. For rust programmers, using dependencies written in rust is a superior experience.

erikjee · 2026-05-27T10:10:23Z

+
+By default decompression performance of our implementation is a couple percent slower than the C reference implementation. We believe we can justify this though, because with the `unsafe-performance-experimental` feature flag enabled we match C performance.
+
+This feature flag disables 4 bounds checks where data from the input is used to index  into a data structure. For most users a ~3% performance regression is likely an acceptable price to pay for increased memory safety. If you really need that last bit of performance, you can enable the flag at your own risk. Its behavior matches C which also does not check the bounds and appears to run just fine in many production systems.


Maybe note how the number of unsafe operations relates to the unsafety of the C impl.
This can read as "you're back to the unsafety of the C impl"

erikjee · 2026-05-27T10:10:49Z

+
+### The cost of memory safety
+
+By default decompression performance of our implementation is a couple percent slower than the C reference implementation. We believe we can justify this though, because with the `unsafe-performance-experimental` feature flag enabled we match C performance.


Link to benchmark suite?

LisetRoos · 2026-05-27T10:21:11Z

+
+The reference implementation was initially translated using [`c2rust`](https://c2rust.com/) , and we have since completed the cleanup work for decompression and the dictionary builder. 
+
+We test our rust code (compiled into a C static library) with the the reference implementation's test suite. We additionally use fuzz testing and Miri, so we're fairly confident in the correctness of our implementation.


Suggested change

We test our rust code (compiled into a C static library) with the the reference implementation's test suite. We additionally use fuzz testing and Miri, so we're fairly confident in the correctness of our implementation.

We test our Rust code (compiled into a C static library) with the the reference implementation's test suite. We additionally use fuzz testing and Miri, so we're fairly confident in the correctness of our implementation.

Co-authored-by: Erik Jonkers <erik@tweedegolf.com> Co-authored-by: Liset <116883252+LisetRoos@users.noreply.github.com>

folkertdev · 2026-05-28T09:21:08Z

+
+We test our rust code (compiled into a C static library) with the the reference implementation's test suite. We additionally use fuzz testing and Miri, so we're confident in the correctness of our implementation.
+
+The release is available here: <TODO: link to release>.


So we have https://crates.io/crates/libzstd-rs-sys/0.0.1-prerelease.1 which is what google uses. Do we want something more official? I'd still classify the current code as beta, there is still more unsafe in there than we'd like, and it's not yet seen serious third-party testing (google is working on it, from what I understand)

erikjee · 2026-05-29T09:17:39Z

+
+We test our Rust code (compiled into a C static library) with the the reference implementation's test suite. We additionally use fuzz testing and Miri, so we're confident in the correctness of our implementation.
+
+The pre-release is available here: <TODO: link to release>.


Add link after release of ~ 0.0.1.prelease2

erikjee added 2 commits May 27, 2026 11:47

Initial version. Folkert's draft + sponsor section

6d139bb

Updates to data compression and libzstd pages

d2261eb

erikjee requested a review from LisetRoos May 27, 2026 10:01

erikjee commented May 27, 2026

View reviewed changes

erikjee requested a review from folkertdev May 27, 2026 10:11

LisetRoos approved these changes May 27, 2026

View reviewed changes

Apply suggestions from code review

037d136

Co-authored-by: Erik Jonkers <erik@tweedegolf.com> Co-authored-by: Liset <116883252+LisetRoos@users.noreply.github.com>

folkertdev reviewed May 28, 2026

View reviewed changes

tweaks

50bd389

folkertdev force-pushed the zstd-announcement branch from 9f83091 to 50bd389 Compare May 28, 2026 09:22

Apply suggestion from @erikjee

de34149

erikjee commented May 29, 2026

View reviewed changes

Comment thread content/blog/Announcing Zstandard in Rust.md Outdated

Apply suggestion from @erikjee

e8d35e1

erikjee commented May 29, 2026

View reviewed changes

Comment thread content/blog/Announcing Zstandard in Rust.md Outdated

Apply suggestion from @erikjee

afa6d5b


		Using zstd in rust is already supported via the [zstd](https://crates.io/crates/zstd) crate, so why bother with a whole new implementation?

		One practical advantage of rust is that it is straightforward to write portable software. Currently the `zstd` crate compiles the C code from source, which requires a C toolchain for the target and for the target to be supported at all. Setting up a C toolchain on windows or for webassembly can be a challenge, and isn't needed with a pure-rust implementation. For rust programmers, using dependencies written in rust is a superior experience.


		By default decompression performance of our implementation is a couple percent slower than the C reference implementation. We believe we can justify this though, because with the `unsafe-performance-experimental` feature flag enabled we match C performance.

		This feature flag disables 4 bounds checks where data from the input is used to index into a data structure. For most users a ~3% performance regression is likely an acceptable price to pay for increased memory safety. If you really need that last bit of performance, you can enable the flag at your own risk. Its behavior matches C which also does not check the bounds and appears to run just fine in many production systems.


		### The cost of memory safety

		By default decompression performance of our implementation is a couple percent slower than the C reference implementation. We believe we can justify this though, because with the `unsafe-performance-experimental` feature flag enabled we match C performance.


		The reference implementation was initially translated using [`c2rust`](https://c2rust.com/) , and we have since completed the cleanup work for decompression and the dictionary builder.

		We test our rust code (compiled into a C static library) with the the reference implementation's test suite. We additionally use fuzz testing and Miri, so we're fairly confident in the correctness of our implementation.


		We test our rust code (compiled into a C static library) with the the reference implementation's test suite. We additionally use fuzz testing and Miri, so we're confident in the correctness of our implementation.

		The release is available here: <TODO: link to release>.


		We test our Rust code (compiled into a C static library) with the the reference implementation's test suite. We additionally use fuzz testing and Miri, so we're confident in the correctness of our implementation.

		The pre-release is available here: <TODO: link to release>.

Uh oh!

Conversation

erikjee commented May 27, 2026

Uh oh!

Uh oh!

Uh oh!

erikjee May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erikjee May 27, 2026

Choose a reason for hiding this comment

Uh oh!

erikjee May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LisetRoos May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

folkertdev May 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erikjee May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants