Discussion: Which way to go for enabling a wasip3 sysroot?

[cpetig]

Update :
MR #700 by Alex answered most of these questions and contains a link to my initial implementation for stdout, file read, tcp, and poll.

Dear all,

to enable the use of cooperative threads from C++ we need a wasip3 implementation of files and sockets. I have some initial bits and pieces working together and would like to discuss a way to upstream these.

My thoughts and questions are roughly:

• A wasip3 compiler target isn't necessary for this as all WASI calls happen from within libc. The available wasm32-wasip2 target can already handle this with a custom sysroot. Do we need a wasip3 compiler target? And does a non-cooperative-threaded libc target for p3 make any sense?
• Using the classical Makefile doesn't make much sense with CMake ready in Migrate the Makefile to CMake #685 , but CMake uses the target triple to deduct the WASI variable (perhaps CMakefile.txt should be changed to not override WASI from the target triple?) My working example specifies WASI=p3 on the make command line. How should this work with cmake?
• I fully separated the names for p3 (wall_clock3_datetime_t) from the names for p2 (wall_clock_datetime_t) by binding renames to detect remaining uses of the wrong symbols. I see both upsides and downsides to having a disjunct set of symbols between wasip2.h and wasip3.h. Any thoughts?
• Converting the blocking C API to using asynchronous p3 calls together with waitable-set.wait calls works just fine for my purpose. But perhaps the p3 (stream) handles could enable uses outside of libc, e.g. via __get_osfhandle.
• I understood that @TartanLlama is currently working on the threading infrastructure for p3, so there should be little overlap between our work, but having a common compilation target in place simplifies cooperation. This is what this aims for, I will prepare a PR but would like to avoid unnecessary re-work by discussing here first.

[alexcrichton]

Personally I've been assuming that we'll be adding a wasm32-wasip3 target to wasi-libc, wasi-sdk, rustc, etc. That's the signal for what the standard library can do as well as an explicit signal for using component-model-async-related-things. It's true that it's not strictly necessary but I think it'll mesh well with the ecosystem better with a new target.

In terms of libc implementation, what I'd roughly expect is that the wasip1/wasip2 headers wouldn't be available at all. That's not the current state of things but there's refactorings that I've been delaying until after the cmake build was landed to implement. In general though what I'd expect is that wasi-libc would provide something like wasi/version.h which would ensure __wasi__ and __wasip3__ were defined (or something like that). That would then guide the internal implementations in libc to use wasip3 primitives instead of wasip2.

I unfortunately don't know of a great way to incrementally move to wasip3 within libc itself. I think it'll need to have a "big bang" where everything is done at once because of how tightly intertwined everything is due to poll, for example. Well, in any case, the exact transition plan is something I think still needs to be worked out.

For accessing internal handles, I've been assuming that some __wasilibc_* functions would be added or something like that to have specific type signatures for wasi. I don't know what exactly they'd look like though, but we could bikeshed that over time too.

[TartanLlama]

I think the wasm32-wasip3 target is definitely needed: that's what I'm putting my current work on supporting TLS and multiple stack pointers for coop threading.

[ricochet]

provide something like wasi/version.h which would ensure wasi and wasip3

Related to request: #688

[cpetig]

I think the wasm32-wasip3 target is definitely needed: that's what I'm putting my current work on supporting TLS and multiple stack pointers for coop threading.

This convinced me that a p3 compiler target is necessary. For my use case a single thread local instance could work though.

I will take a look into adding a preliminary p3 compiler target and continue from there, if you don't beat me to it (I know Alex already added it for Rust, so llvm might already accept it)

[cpetig]

I unfortunately don't know of a great way to incrementally move to wasip3 within libc itself. I think it'll need to have a "big bang" where everything is done at once because of how tightly intertwined everything is due to poll, for example. Well, in any case, the exact transition plan is something I think still needs to be worked out.

This matches my experience, I tried incremental and ended up implementing all of p3 because of handles to poll (now waitable-set.X) .

For accessing internal handles, I've been assuming that some __wasilibc_* functions would be added or something like that to have specific type signatures for wasi. I don't know what exactly they'd look like though, but we could bikeshed that over time too.

Makes sense.

[alexcrichton]

#700 is my initial stab at creating support for the target

[cpetig]

PS: alexcrichton#1 contains my initial implementation (and there is some very minimalist pthread in my adjacent wasip3_threads branch)

[TartanLlama]

You can see my current progress on LLVM here and on wasi-libc here

Current status is that I can get a simple pthreads example running as long as I disable a check in wasmtime related to changes in the component model spec about when sync calls can block. I'll be chatting to Luke about what to do there.

I've done some work on TLS support, but it's not there yet

My expectation is that, once I get things working, I'll start refactoring these changes to fit into the work the rest of you have been doing and start landing them piece-by-piece

[cpetig]

You can see my current progress on LLVM here and on wasi-libc here

🤔 I guess this issue could be a reasonable place to voice a question to this implementation.

First, I was shocked to see that, with these patches, wasm32-wasip3 code becomes binary incompatible with any other wasm32 target, because the long standing convention of using a global for the current stack pointer and TLS base was replaced by a call to the context built-ins in every compiled function. That means even if code never calls into any WASI API it will be incompatible across this change. So any small (external) library needs recompilation (or a potential re-coding).

If the libc functions would save the stack pointer+TLS base via the context built-ins before switching to a different thread and restoring them before returning, old code following the old convention would continue to work.

I guess this change is made to prepare for future preemptive (shared everything) threads, although shared functions and unshared globals would require another binary incompatibility anyway.

In the past binary incompatibilities between wasm binaries (e.g. p1 to p2) have let to community confusion and fragmentation (some tools still only work with p1) - I always found the reason for the change reasonable, but it creates so much pain ...

[TartanLlama]

Unfortunately libc boundaries aren't the only places where those globals would be switched out. For example, a task could spawn a thread that yields, the implicit thread is rescheduled and calls an import that blocks. The spawned thread would be rescheduled without libc being touched.

Also, an indirect call doesn't encode anything about whether the callee is an import, so I believe switching code would need to be injected around every indirect call in the general case.

[TartanLlama]

FWIW I spent a bunch of time trying to do exactly what you suggested @cpetig and hit all of these issues 😅

@alexcrichton and @lukewagner also put some thought into this approach and may have other context to add

[alexcrichton]

The lynchpin feature for me was for the runtime to be able to automatically switch to some other thread during one thread's waitable-set.wait, for example. This means that the threads behave pretty close to what one might actually expect real native threads to do which means that the green threading bits in the guest are surprisingly light. For example wasi-libc won't be the single point of being able to context switch, it just manages its own threads. Like on native platforms it should be possible to link, in one wasm binary, two runtimes doing threads each in their own way.

This requirement though means that it can't be assumed that implicit handling of the stack pointer global is handled by the guest. That'd require a huge amount more infrastructure in each guest with instrumentation around any and every blocking or async call that could trigger some sort of yield. While possible it seemed much more desirable to have a much lighter guest with a slightly more heavyweight host.

In the past binary incompatibilities between wasm binaries (e.g. p1 to p2) have let to community confusion and fragmentation

While it's tough to say how exactly this change will play out and be received, my hunch is that this won't be as much of a problem from p2 to p3. The p1->p2 transition was particularly extreme due to:

• The long transition period during which a lot of it produced core wasm modules for a "p2 target"
• Lack of existence of a p2 target for quite some time
• p1 -> p2 completely changes the binary format (core wasm -> component)

In contrast p2 -> p3 will have virtually no difference from an "external" point of view except for using different WASI interfaces. Internally the wasm will be much different, but in theory that's "just an implementation detail".

We still have work to do in Wasmtime as well to make calls to context.{get,set} "just" a load/store, but with Nick's inlining work in Wasmtime I don't expect that this will be a herculean task, just a nontrivial one.

Otherwise this is also, in some respects, a stepping stone to preemptible threads. This is intended to be future-polyfillable with the direction the shared-everything-threads proposal is going in (roughly) at least to a certain degree. Frobbing the global stack pointer will definitely not be workable with preemptible threads at least.

---

Now I don't want to discount how big of a change this is going to be. I don't think we'll be able to grasp the breadth/depth of this change and the disruption it causes until we're much further down the road of implementing it. I'm hopeful we can keep the fallout contained, but I can't say I'm 100% confident it'll be a reasonable ride.

[cpetig]

Thank you @TartanLlama and @alexcrichton for the detailed explanation, and especially thank you for showing that this incompatibility was indeed the result of a long term investigation where alternatives were tried. (If there is a place I would have been able to see the work you put into it please tell me about it)

As you can see from the amount of work I put into wasip3 libc in the past weeks I believe that wasip3, especially with cooperative threading, is a huge leap forward and unlocks many important use cases.

Thank you all for making this real!

[TartanLlama]

I've made a PR to wit-bindgen for adding threading builtins, which will help with the wasip3 work: bytecodealliance/wit-bindgen#1491