Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
163 changes: 163 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,12 @@ second sibling: a `ToolRanker` backed by TypeSafe's Jev decision model through
`tinyjevclient`, kept out of the vocabulary crate because it carries an HTTP
transport.

[`tinytools-code`](crates/tinytools-code/README.md) is a pure sibling for bounded
syntax analysis, revision-checked edits and typed boilerplate rendering. The
optional `code` feature in [`tinytools-std`](crates/tinytools-std/README.md) exposes
the six filesystem-backed tools through a host's `FsGate`. Neither package adds
parser or runtime dependencies to the base vocabulary crate.

## What is deliberately not here

**No enforcement.** Nothing in this crate checks a `PermissionLevel`, applies a
Expand Down
32 changes: 32 additions & 0 deletions crates/tinytools-code/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
[package]
name = "tinytools-code"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
description = "Bounded syntax analysis, revision-checked patches and typed boilerplate rendering without I/O."
readme = "README.md"
publish = false

[dependencies]
serde = { workspace = true }
serde_json = { workspace = true }
# Content revisions detect stale source and template snapshots without I/O.
blake3 = "1"
# Compiled, explicit grammars provide bounded structural parsing and queries.
tree-sitter = "0.25"
tree-sitter-rust = "0.24"
tree-sitter-typescript = "0.23"
tree-sitter-javascript = "0.25"
tree-sitter-python = "0.25"
tree-sitter-go = "0.25"
tree-sitter-json = "0.24"
tree-sitter-toml-ng = "0.7"

[lints]
workspace = true

[[bench]]
name = "structural"
harness = false
52 changes: 52 additions & 0 deletions crates/tinytools-code/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# TinyTools Code

Pure, bounded syntax analysis and boilerplate rendering. The engine accepts
caller-provided UTF-8 and an explicit Rust, TypeScript, TSX, JavaScript, Python,
Go, JSON or TOML grammar. It performs no filesystem or network access.

`CodeEngine::analyze` returns lexical declarations/references, syntax diagnostics,
content revisions and a folded view. Cache outcomes disclose fresh, unchanged or
incremental parsing. Cache byte limits measure retained source, excluding native
tree allocations. A language hint can be obtained with `Language::from_path`.

`CodeEngine::patch` requires an exact expected revision. Operations replace bodies
or whole declarations, insert imports, append members and wrap one selected node.
Original and resulting code must parse. Exact IDs identify a revision and source
range. Name selectors must be provably unique; truncated outlines only admit IDs.
No operation promises typechecking, semantic name resolution or behavior changes
equivalent to a compiler-assisted refactor.

`ChunkCatalog::builtin` contains twelve patterns. Search and inspect are metadata
operations. Templates are separate and `builtin_source` only serves genuine
built-in entries. Custom template paths are portable relative paths, with actual
filesystem containment enforced by the native adapter.

`render_chunk` expands typed `{{parameter}}` values and original standalone
`// @slot name`, `# @slot name` or `/* @slot name */` syntax comments once. It
refuses unknown, missing, duplicate or unused declarations. Supplied text is not
rescanned for directives. Parameter kinds are conservative ASCII identifiers,
uppercase-leading identifiers, syntax-checked type expressions, escaped strings,
JSON numbers and booleans. Required slots need nonempty code; optional slots can
be empty. A slot can require exactly one grammar node kind. Forbidden node kinds
apply to generated code, excluding artificial validation wrappers.

File chunks validate as complete sources. Body/member chunks use explicit syntax
contexts. Rust members can be fields or impl items, Go members can be fields or
interface items, JavaScript/TypeScript members use classes, and Python members
use class suites. JSON and TOML support file chunks only. Dependency declarations
are metadata and never install packages.

Default source/result limit: 1 MiB. Cache: 32 entries and 8 MiB of source.
Symbols/references: 2048 each. Diagnostics: 64. Manifests: 64 KiB; 32 parameters,
32 slots and 512 catalog entries. Configured limits have hard caps. All expansion
paths check limits before output growth, including indentation.

The optional `tinytools-std` `code` feature provides FsGate-backed native tools.
Host permission, preview, result paging and atomic publication belong there.

## Validation

Run the repository's four contract commands and 90% per-file coverage gate.
Focused tests use `cargo test -p tinytools-code`. Run the fixed-fixture benchmark
with `cargo bench -p tinytools-code --bench structural`. It reports byte counts
and cold/cache/incremental timings, without token-savings or speed claims.
40 changes: 40 additions & 0 deletions crates/tinytools-code/benches/structural.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
//! Fixed-fixture cold, unchanged and incremental parse observations.

use std::{hint::black_box, time::Instant};
use tinytools_code::{CodeEngine, Language};

fn main() -> Result<(), tinytools_code::CodeError> {
let source = include_str!("../tests/fixtures/analysis.rs");
let iterations = 100;
let start = Instant::now();
for _ in 0..iterations {
black_box(CodeEngine::default().analyze("fixture", Language::Rust, source)?);
}
let cold = start.elapsed();
let mut engine = CodeEngine::default();
let report = engine.analyze("fixture", Language::Rust, source)?;
let start = Instant::now();
for _ in 0..iterations {
black_box(engine.analyze("fixture", Language::Rust, source)?);
}
let cached = start.elapsed();
let changed = source.replace("value * 2", "value * 3");
let start = Instant::now();
for index in 0..iterations {
black_box(engine.analyze(
"fixture",
Language::Rust,
if index % 2 == 0 { &changed } else { source },
)?);
}
println!(
"iterations={iterations} source_bytes={} folded_bytes={} symbols={} cold_us={} cached_us={} incremental_us={}",
source.len(),
report.skeleton.len(),
report.symbols.len(),
cold.as_micros(),
cached.as_micros(),
start.elapsed().as_micros()
);
Ok(())
}
3 changes: 3 additions & 0 deletions crates/tinytools-code/src/analysis/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Lexical analysis

Bundled query captures describe declarations, imports and calls. Names are lexical, not resolved compiler symbols. Results and diagnostics disclose truncation. Exact IDs bind a content revision to byte ranges; name-based selection refuses truncated outlines. Folded views preserve surrounding text and Python docstrings.
Loading
Loading