Skip to content

Commit af68d3f

Browse files
committed
Update docs for simplified API, add parallel execution guide, bump to 3.0.0
Documentation updates after API simplification: - Remove WSGI/ASGI references from buffer.md, getting-started.md - Remove py_asgi examples from asyncio.md - Mark subinterp/OWN_GIL as internal in scalability.md, process-bound-envs.md - Update migration.md with simplified mode descriptions - Update source comments in py_buffer.erl, py_nif.erl, py_reactor_context.erl - Remove missing owngil_internals.md from rebar.config New documentation: - Add docs/parallel-execution.md explaining parallel pool architecture Version bump: - Bump version to 3.0.0 in app.src and getting-started.md
1 parent 6f8350f commit af68d3f

13 files changed

Lines changed: 455 additions & 132 deletions

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ schedulers.
1818
**Parallelism options:**
1919
- **Worker mode** (default) - Works with any Python version
2020
- **Free-threaded Python** (3.13t+) - True parallelism, automatic detection
21-
- **Parallel pool** (Python 3.14+) - Build with `ENABLE_PARALLEL_PYTHON=ON` for subinterpreter-based parallelism
21+
- **Parallel pool** - Build with `ENABLE_PARALLEL_PYTHON=ON` for automatic parallel execution
2222
- **BEAM processes** - Fan out work across lightweight Erlang processes
2323

2424
Key features:
@@ -603,7 +603,7 @@ By default, erlang_python uses **worker mode** which works with any Python versi
603603

604604
For true parallel Python execution:
605605

606-
- **Build with parallel pool** (Python 3.14+):
606+
- **Build with parallel pool**:
607607
```bash
608608
CMAKE_OPTIONS="-DENABLE_PARALLEL_PYTHON=ON" rebar3 compile
609609
```

docs/asyncio.md

Lines changed: 1 addition & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -915,30 +915,13 @@ erlang.run(main())
915915
- You need precise sub-millisecond timing
916916
- Your app makes many small sleep calls
917917
- You want to eliminate Python event loop overhead
918-
- Building ASGI handlers that need efficient sleep
918+
- Building async handlers that need efficient sleep
919919
- Your app is running inside erlang_python
920920

921921
**Use standard `asyncio.run()` when:**
922922
- You're running outside the Erlang VM
923923
- Testing Python code in isolation
924924

925-
### Integration with ASGI Frameworks
926-
927-
For ASGI applications (FastAPI, Starlette, etc.), you can use the Erlang event loop for better performance:
928-
929-
```python
930-
from fastapi import FastAPI
931-
import asyncio
932-
933-
app = FastAPI()
934-
935-
@app.get("/delay")
936-
async def delay_endpoint(ms: int = 100):
937-
# When running via py_asgi, uses Erlang timer
938-
await asyncio.sleep(ms / 1000.0)
939-
return {"slept_ms": ms}
940-
```
941-
942925
## Async Worker Backend (Internal)
943926

944927
The `py:async_call/3,4` and `py:await/1,2` APIs use an event-driven backend based on `py_event_loop`.

docs/buffer.md

Lines changed: 15 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,20 @@
11
# Buffer API
22

3-
The Buffer API provides a zero-copy WSGI input buffer for streaming HTTP request bodies from Erlang to Python. Buffers use shared memory with GIL-released blocking reads for efficient data transfer.
3+
The Buffer API provides a zero-copy streaming buffer for transferring data from Erlang to Python. Buffers use shared memory with GIL-released blocking reads for efficient data transfer.
44

55
## Overview
66

7-
Buffers are designed for WSGI/ASGI `wsgi.input` scenarios where Erlang receives HTTP body chunks and Python needs to consume them:
7+
Buffers are designed for scenarios where Erlang produces data chunks and Python needs to consume them:
88

99
- Zero-copy access via Python's buffer protocol (`memoryview`)
1010
- File-like interface (`read`, `readline`, `readlines`)
1111
- Blocking reads that release the GIL while waiting
1212
- Fast substring search using `memchr`/`memmem`
1313

1414
Use buffers when you need:
15-
- WSGI input for HTTP request bodies
1615
- Streaming data from Erlang to Python
1716
- Zero-copy access to binary data
17+
- Blocking reads with GIL release
1818

1919
## Quick Start
2020

@@ -34,25 +34,23 @@ ok = py_buffer:write(Buf, <<"chunk2">>),
3434
%% Signal end of data
3535
ok = py_buffer:close(Buf),
3636

37-
%% Pass to WSGI app
38-
py:call(Ctx, myapp, handle_request, [#{<<"wsgi.input">> => Buf}]).
37+
%% Pass to Python handler
38+
py:call(Ctx, myapp, handle_request, [Buf]).
3939
```
4040

4141
### Python Side
4242

4343
```python
44-
def handle_request(environ):
45-
wsgi_input = environ['wsgi.input']
46-
44+
def handle_request(buf):
4745
# Read all data
48-
body = wsgi_input.read()
46+
body = buf.read()
4947

5048
# Or read line by line
51-
for line in wsgi_input:
49+
for line in buf:
5250
process(line)
5351

5452
# Or read specific amount
55-
chunk = wsgi_input.read(1024)
53+
chunk = buf.read(1024)
5654
```
5755

5856
## Erlang API
@@ -348,25 +346,16 @@ py_buffer_resource_t
348346

349347
## Examples
350348

351-
### WSGI Input Simulation
349+
### Basic Streaming
352350

353351
```erlang
354-
%% Simulate receiving HTTP body
352+
%% Create buffer and write data
355353
{ok, Buf} = py_buffer:new(byte_size(Body)),
356354
ok = py_buffer:write(Buf, Body),
357355
ok = py_buffer:close(Buf),
358356

359-
%% Build WSGI environ
360-
Environ = #{
361-
<<"REQUEST_METHOD">> => <<"POST">>,
362-
<<"PATH_INFO">> => <<"/api/data">>,
363-
<<"CONTENT_TYPE">> => <<"application/json">>,
364-
<<"CONTENT_LENGTH">> => integer_to_binary(byte_size(Body)),
365-
<<"wsgi.input">> => Buf
366-
},
367-
368-
%% Call WSGI app
369-
{ok, Response} = py:call(myapp, handle, [Environ]).
357+
%% Pass to Python handler
358+
{ok, Response} = py:call(myapp, handle, [Buf]).
370359
```
371360

372361
### Chunked Transfer
@@ -443,10 +432,8 @@ async def read_buffer_async(buf):
443432

444433
return b''.join(chunks)
445434

446-
async def process_wsgi_body_async(environ):
447-
"""Process WSGI body in async context."""
448-
buf = environ['wsgi.input']
449-
435+
async def process_body_async(buf):
436+
"""Process buffer in async context."""
450437
# Read body without blocking
451438
body = await read_buffer_async(buf)
452439
return json.loads(body)

docs/getting-started.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Add to your `rebar.config`:
88

99
```erlang
1010
{deps, [
11-
{erlang_python, "2.2.0"}
11+
{erlang_python, "3.0.0"}
1212
]}.
1313
```
1414

@@ -424,7 +424,7 @@ async def main():
424424
erlang.run(main())
425425
```
426426

427-
This is especially useful in ASGI handlers where sleep operations are common. See [Asyncio](asyncio.md) for the full API reference.
427+
This is especially useful in async handlers where sleep operations are common. See [Asyncio](asyncio.md) for the full API reference.
428428

429429
## Security Considerations
430430

@@ -471,28 +471,28 @@ This prevents slow HTTP requests from blocking quick math operations. See [Pool
471471

472472
## Zero-Copy Buffers
473473

474-
For WSGI/ASGI applications that need to stream HTTP request bodies, use `py_buffer`:
474+
For streaming data from Erlang to Python, use `py_buffer`:
475475

476476
```erlang
477-
%% Create buffer for HTTP body
477+
%% Create buffer for streaming data
478478
{ok, Buf} = py_buffer:new(ContentLength),
479479

480-
%% Write body chunks as they arrive
480+
%% Write chunks as they arrive
481481
ok = py_buffer:write(Buf, Chunk1),
482482
ok = py_buffer:write(Buf, Chunk2),
483483
ok = py_buffer:close(Buf),
484484

485-
%% Pass to WSGI app as wsgi.input
486-
py:call(Ctx, myapp, handle, [#{<<"wsgi.input">> => Buf}]).
485+
%% Pass to Python handler
486+
py:call(Ctx, myapp, handle, [Buf]).
487487
```
488488

489489
Python sees it as a file-like object with blocking reads:
490490

491491
```python
492-
def handle(environ):
493-
body = environ['wsgi.input'].read() # Blocks until data ready
492+
def handle(buf):
493+
body = buf.read() # Blocks until data ready
494494
# Or iterate lines
495-
for line in environ['wsgi.input']:
495+
for line in buf:
496496
process(line)
497497
```
498498

@@ -509,7 +509,7 @@ See [Buffer API](buffer.md) for zero-copy memoryview access and fast substring s
509509
- See [Logging and Tracing](logging.md) for Python logging and distributed tracing
510510
- See [AI Integration](ai-integration.md) for ML/AI examples
511511
- See [Asyncio Event Loop](asyncio.md) for the Erlang-native asyncio implementation with TCP and UDP support
512-
- See [Buffer API](buffer.md) for zero-copy WSGI input buffers
512+
- See [Buffer API](buffer.md) for zero-copy streaming buffers
513513
- See [Reactor](reactor.md) for FD-based protocol handling
514514
- See [Security](security.md) for sandbox and blocked operations
515515
- See [Distributed Execution](distributed.md) for running Python across Erlang nodes

docs/migration.md

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -16,16 +16,17 @@ This guide covers breaking changes and migration steps when upgrading from erlan
1616

1717
## Python Version Compatibility
1818

19-
| Python Version | GIL Mode | Notes |
20-
|---------------|----------|-------|
21-
| 3.9 - 3.11 | Shared GIL | Multi-executor mode, `py:execution_mode()` returns `multi_executor` |
22-
| 3.12 - 3.13 | OWN_GIL subinterpreters | True parallelism, `py:execution_mode()` returns `subinterp` |
19+
| Python Version | Mode | Notes |
20+
|---------------|------|-------|
21+
| 3.9 - 3.11 | Multi-executor | Shared GIL, `py:execution_mode()` returns `multi_executor` |
22+
| 3.12+ | Multi-executor or Parallel | With `ENABLE_PARALLEL_PYTHON=ON`, uses internal parallel pool |
2323
| 3.13t | Free-threaded | No GIL, `py:execution_mode()` returns `free_threaded` |
24-
| 3.14+ | SHARED_GIL subinterpreters | Subinterpreters with shared GIL for C extension compatibility |
24+
25+
**Note:** Subinterpreter/OWN_GIL modes are now internal implementation details used by the parallel pool build. Users don't need to select these modes directly.
2526

2627
**Python 3.14 Support**: Full support for Python 3.14 including:
27-
- SHARED_GIL subinterpreter mode for C extension compatibility
28-
- Proper `sys.path` initialization in subinterpreters
28+
- Parallel pool build for true parallelism
29+
- Proper `sys.path` initialization
2930
- All asyncio features work correctly
3031

3132
**FreeBSD Support**: Improved fd handling on FreeBSD/kqueue platforms:
@@ -34,18 +35,18 @@ This guide covers breaking changes and migration steps when upgrading from erlan
3435

3536
## Architecture Changes
3637

37-
### OWN_GIL Subinterpreter Thread Pool (Python 3.12+)
38+
### Parallel Pool Build
3839

39-
The most significant change in v2.0 is the new execution model. On Python 3.12+, erlang_python now uses **OWN_GIL subinterpreters** for true parallelism:
40+
The most significant change in v2.0 is the new execution model. When built with `ENABLE_PARALLEL_PYTHON=ON`, erlang_python uses an internal parallel pool for true parallelism:
4041

4142
**v1.8.x Architecture:**
4243
- Single Python interpreter with shared GIL
4344
- Worker pool with round-robin dispatch
4445
- All workers share global state
4546

46-
**v2.0 Architecture:**
47-
- N subinterpreters, each in its own thread with its own GIL
48-
- Each subinterpreter has isolated state (no shared globals)
47+
**v2.0 Architecture (with parallel build):**
48+
- Internal pool provides true parallelism
49+
- Each context has isolated state (no shared globals)
4950
- True parallel execution without GIL contention
5051
- 25-30% faster cast operations
5152

@@ -85,11 +86,10 @@ Check which mode is active:
8586
```erlang
8687
%% Check execution mode
8788
py:execution_mode().
88-
%% => subinterp (Python 3.12+ with OWN_GIL)
89-
%% => free_threaded (Python 3.13t with --disable-gil)
90-
%% => multi_executor (Python < 3.12)
89+
%% => multi_executor (default worker mode)
90+
%% => free_threaded (Python 3.13t with --disable-gil)
9191

92-
%% Check if subinterpreters are supported
92+
%% Check if parallel execution is available
9393
py:subinterp_supported().
9494
%% => true | false
9595
```

0 commit comments

Comments
 (0)