Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions docs/knowledge-base.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,13 @@ flowchart TB
P -. "source view" .-> M1
C -. "coverage view" .-> M2["SearchSpace Matrix"]
O["K2: normalized reads / Collect"] -. "reuse" .-> R
W["Follow-up: Create / Update"] -. "authorized local store only" .-> B["Complete revision bundles"]
W["K2: Create / Update"] -. "store-owned candidates only" .-> B["Complete revision bundles"]
```

Solid edges describe the K0/K1 foundation; the dashed
[normalized reads / Collect](knowledge-operations.md) box reuses those readers,
and Create / Update remains a follow-up. No mandatory ContextPack, conversation manager, crawler or universal
Solid edges describe the K0/K1 foundation; dashed boxes are the K2
[normalized reads / Collect](knowledge-operations.md) and
[authorized local Create / Update](knowledge-store.md) operations, which reuse
those readers. No mandatory ContextPack, conversation manager, crawler or universal
performance loop is introduced. A generated catalog is a projection, not a third
metadata authority or maintained matrix.

Expand Down Expand Up @@ -82,7 +83,9 @@ dependency and peer remain independent repository relationships.

The legacy readers retain their command contracts. [Normalized reads](knowledge-operations.md)
identify complete declared dependency bundles and reject unavailable old snapshot
revisions. An old revision never silently resolves newer resource bytes. Typed source
revisions. The [authorized store](knowledge-store.md) retains old metadata/resources
and verifies their bundle digest. An old revision never silently resolves newer
resource bytes. Typed source
examples are present for `audit-vllm-kv-lifetimes` (repository),
`calibrate-serving-simulation` (reviewed Vidur v2 mechanism), and the internal case.
Other source records retain their existing evidence rather than fabricated fields.
Expand Down Expand Up @@ -150,7 +153,7 @@ available after noneditable installation; local `skills/` is not wheel data.
|---|---|---|
| PR #12, K0/K1 | Ownership, preserved content, metadata projection, explicit instruction/resource/reference reads, both matrices and reusable case | No normalized Collect or mutation API claim |
| K2 reads | [`KnowledgeReader`](knowledge-operations.md) facade, typed sources, byte-derived bundle identity, authority-qualified refs with ambiguity rejection, exact revisions/resources and bounded Collect | Historical snapshot material must exist; no crawling or standalone export |
| PR 3, K2 writes | Separate follow-up | Supplied store-owned candidate Create/Update in one authorized store, adapter-identity collision rejection, duplicate/stale rejection, serialized writes, complete publication and immutable history |
| K2 writes | Python `KnowledgeStore` for store-owned candidates, adapter-identity reservation, `derived_from` adaptations, SQLite writer serialization, duplicate/stale rejection, complete publication and immutable history | One authorized local directory only; no repository writer, promotion or distributed writer guarantee |

The writer will not edit site-packages, frozen evidence or source checkouts.
A version field alone provides no concurrency guarantee. No automatic promotion
Expand Down
85 changes: 85 additions & 0 deletions docs/knowledge-store.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Authorized local Create / Update

This K2 write slice builds on the [normalized read contract](knowledge-operations.md).
It adds one backend: SQLite in an **existing, explicitly authorized local directory**.
Create and Update operate only on **store-owned candidate bundles**. Package and
checkout content stay read-only; they are never written, and their IDs cannot be
reused in the store. Amending repository-owned content follows the normal scoped
PR path.

```python
from pathlib import Path
from omnirsi.knowledge_operations import Bundle, KnowledgeReader
from omnirsi.knowledge_store import KnowledgeStore

skills = Path("skills")
store = KnowledgeStore(Path("/authorized/local/knowledge-workspace"), skill_root=skills)
source = KnowledgeReader(skill_root=skills).read("calibrate-serving-simulation")
draft = Bundle(
{**source.metadata, "id": "calibrate-serving-simulation-local", "state": "candidate",
"derived_from": [{"ref": "checkout:calibrate-serving-simulation",
"revision": source.revision}]},
dict(source.content), dict(source.dependencies),
)
revision = store.create(draft)
# Supply a complete amended Bundle, preserving source attribution and resources:
# next_revision = store.update(amended, expected=revision,
# reason="Clarify applicability", evidence=[source_pointer])
old = store.read("calibrate-serving-simulation-local", revision=revision)
collected = store.collect(ids=["store:calibrate-serving-simulation-local"],
revisions={"store:calibrate-serving-simulation-local": revision},
material=True)
```

`skill_root` is required: pass the checkout skills root whose IDs must be reserved,
or `None` to declare package-only use. `store.reader()` / `store.collect` combine
package, checkout and store items with qualified `store:<id>` refs. An ID owned by
the store and a read-only adapter is reported as ambiguous, never shadowed.

## Create and Update

- Create registers a supplied **candidate** and rejects duplicate identity. It
rejects an ID owned by package or checkout content. An adaptation of existing
content uses a distinct ID and records `derived_from` entries with a qualified
`ref` and the source content revision (or `null` if unknown).
- Create does not relabel imported content or generate guidance. Update requires
the expected current revision, a reason and an explicit evidence list (empty
means none supplied). It publishes a successor without changing the content's
state; promotion is not implemented.
- An Update whose bundle reproduces a retained revision is an explicit conflict.
Existing history preserves old metadata, provenance, resources, parent revision,
reason and evidence even if a later interpretation changes.
- If a read-only adapter later gains a store item's ID, reads report ambiguity and
further Updates fail with a collision until the owners resolve it.

## Writer and publication guarantees

- Scope: cooperating processes/threads using this API on one local SQLite file;
no network-filesystem, distributed-store or hostile filesystem-owner guarantee.
- The workspace must already exist and lie outside the installed `omnirsi`
package, the checkout containing `skill_root` and, for a source or editable
install, the checkout containing the package (these hold evidence and both
matrices). That package checkout is the nearest ancestor holding both
`pyproject.toml` and `.git`; with `skill_root=None` it is the only checkout
guarded, so a checkout lacking either marker is not protected. The entrypoint of a stored bundle must be UTF-8 text. Workspace and database symlinks are rejected. No destination
directory is guessed or automatically created; the caller controls permissions.
- SQLite `BEGIN IMMEDIATE` serializes writers before duplicate/stale checks.
Resources, revision metadata and the current pointer commit in one transaction.
A database error rolls the whole transaction back; partial usable bundles are
never published. A five-second lock timeout raises an explicit SQLite error.
- Content/resource/dependency validation runs before writing. Dependencies are
included as bytes and checked against declared digests. Old reads load retained
bytes and verify their complete bundle identity; they never fall through to a
current checkout. No external mutable dependency lookup occurs during reads.
- Resource keys cannot be absolute paths or traversal paths; they are database
keys, never filesystem write destinations.
- Runtime operations write only the store database. They do not edit adapter
sources, raw evidence or either matrix, and a store candidate is not repository
coverage until exported through a reviewed repository change.
- Normal SQLite local-filesystem transaction guarantees apply. This implementation
does not claim tested power-loss durability, malicious tamper resistance or an
automatic recovery campaign. Integrity mismatches fail visibly.

Tests use real bundles, SQLite transactions, competing writer connections and a
real rejecting database trigger to prove rollback after an insert starts. They do
not use hardware, synthetic performance claims or a fake storage implementation.
13 changes: 11 additions & 2 deletions src/omnirsi/knowledge_operations.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import hashlib
import json
import re
import sqlite3
from dataclasses import dataclass
from importlib.resources import files
from pathlib import Path, PurePosixPath
Expand Down Expand Up @@ -177,8 +178,11 @@ def _tree(node, prefix: str) -> dict[str, bytes]:


class KnowledgeReader:
def __init__(self, *, skill_root: Path | None = None):
def __init__(self, *, skill_root: Path | None = None, store=None):
self.skill_root = skill_root.resolve() if skill_root is not None else None
if store is not None and store.skill_root != self.skill_root:
raise ValueError("Store must reserve identities from the same checkout")
self.store = store
self._records: dict[str, dict] | None = None
self._library: tuple[dict[str, bytes], str] | None = None

Expand Down Expand Up @@ -210,6 +214,9 @@ def owners(self) -> dict[str, list[str]]:
raise IdentityError("collision", "Duplicate identity inside the checkout")
for identity in local:
result.setdefault(identity, []).append("checkout")
if self.store is not None:
for identity in self.store.identities():
result.setdefault(identity, []).append("store")
return result

def identities(self) -> list[str]:
Expand Down Expand Up @@ -250,6 +257,8 @@ def read(self, reference: str, *, revision: str | None = None) -> Bundle:

def _read(self, qualified: str, revision: str | None) -> Bundle:
authority, _, identity = qualified.partition(":")
if authority == "store":
return self.store.read(identity, revision=revision)
bundle = self._checkout(identity) if authority == "checkout" else self._knowledge(identity)
if revision is not None and bundle.revision != revision:
raise RevisionConflict(
Expand Down Expand Up @@ -455,7 +464,7 @@ def matches(metadata: dict, authority: str) -> bool:
{"id": reference, "reason": "unavailable_revision", "detail": str(exc)}
)
continue
except (ValueError, KeyError, TypeError, AttributeError, OSError) as exc:
except (ValueError, KeyError, TypeError, AttributeError, OSError, sqlite3.Error) as exc:
omitted.append({"id": reference, "reason": "invalid", "detail": str(exc)})
continue
if not matches(view, authority):
Expand Down
Loading
Loading