Skip to content

Use streaming I/O for markdown output instead of loading entire file into memory #4

@deevus

Description

@deevus

Problem

readSymbolMarkdown currently reads the entire markdown cache file into memory before writing it to stdout. This was a workaround for a Zig stdlib bug on macOS where reader.stream() / reader.streamRemaining() fails with EBADF (errno 9).

The root cause is that Zig's std.Io.Writer.sendFile uses fcopyfile on macOS, which doesn't support pipes as the destination fd. When stdout is a pipe (e.g., gdoc Node3D | less), fcopyfile fails.

Current workaround

const content = try file.readToEndAlloc(allocator, 1024 * 1024);
defer allocator.free(content);
try output.writeAll(content);

This works but allocates O(n) memory instead of streaming through a fixed buffer.

Desired solution

Use reader.streamRemaining(output) (or equivalent) to stream file content to the output writer with constant memory usage. This requires either:

  1. A fix upstream in Zig's sendFile / fcopyfile handling for pipe destinations on macOS
  2. A custom streaming loop that bypasses sendFile (read into fixed buffer, write to output, repeat)

Upstream

No existing issue found on codeberg.org/ziglang/zig for this specific bug. Consider filing one.

References

  • src/cache.zig: readSymbolMarkdown function
  • Zig stdlib: std/fs/File.zig sendFile, std/Io/Writer.zig sendFile, std/Io/Reader.zig stream/streamRemaining

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions