diff --git a/AGENTS.md b/AGENTS.md index 52ce6361..880dd8a2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -86,10 +86,10 @@ Every public module has a committed API report at `packages//api-report PRs are squash-merged with the PR title as the commit message, and release-please builds each package's changelog from those messages. -- **A change to an experimental feature uses that feature's scope:** `skills` (Agent Skills) and `evaluations` (Evaluations from Code). `feat`, `fix` and `perf` commits with one of these scopes are listed under **Experimental** (`changelog-sections` in `release-please-config.json`), for example `feat(skills): watch skills for revocations`. Any other scope, such as `client`, lists the change as a core feature or fix. -- **Use exactly one scope.** `feat(skills,client):` matches no Experimental row, so it lands under Features. +- **A change to an experimental feature uses that feature's scope:** `evaluations` (Evaluations from Code). `feat`, `fix` and `perf` commits with that scope are listed under **Experimental** (`changelog-sections` in `release-please-config.json`), for example `feat(evaluations): score rows in parallel`. Any other scope, such as `client` or `skills`, lists the change as a core feature or fix. Agent Skills is core, so `feat(skills):` lands under Features. +- **Use exactly one scope.** `feat(evaluations,client):` matches no Experimental row, so it lands under Features. - **A PR that changes both core and experimental behaviour** should be split. If it can't be, add a `BEGIN_COMMIT_OVERRIDE` block to its description with one conventional-commit line per change. An override also corrects a merged PR's title, as long as the release PR hasn't merged yet. -- **A breaking experimental change** uses `!` while the package is 0.x: `feat(skills)!:` bumps the minor version, which is what an experimental break is allowed. After 1.0, `!` would force a major release. Use `feat(skills):` without `!` instead, and say the change is breaking in the title. +- **A breaking experimental change** uses `!` while the package is 0.x: `feat(evaluations)!:` bumps the minor version, which is what an experimental break is allowed. After 1.0, `!` would force a major release. Use `feat(evaluations):` without `!` instead, and say the change is breaking in the title. - **When a feature is promoted to core,** remove its rows from `changelog-sections`. The rule is specified in `TESTING.md` §0.3 in `launchdarkly/ai-sdks-monorepo`. diff --git a/packages/ai/README.md b/packages/ai/README.md index e0b57d02..391ce7b8 100644 --- a/packages/ai/README.md +++ b/packages/ai/README.md @@ -52,16 +52,26 @@ if result["enabled"]: Never raises. Returns `{"enabled": bool, "config": dict | None, "meta": dict | None}`. +## Agent Skills + +Agent Skills (`get_skill`, `write_skills`, `watch_skills` and the rest) are re-exported from this +package's root. They need `launchdarkly-ai-server` 1.0.0 or later, the first release with Agent +Skills in core. This package does not pin `launchdarkly-ai-server`, so upgrading it alone can leave +an older one installed, and `from launchdarkly_ai_python import get_skill` then raises +`ImportError`. Upgrade both: + +```bash +pip install --upgrade launchdarkly-ai-python "launchdarkly-ai-server>=1.0.0" +``` + ## Experimental features Experimental features are not re-exported from this package. They live in `launchdarkly-ai-server`, the package this one depends on, and need a `launchdarkly-ai-server` release that includes them. Import them from `launchdarkly_ai_server.experimental`; if that import fails, upgrade `launchdarkly-ai-server` itself, since upgrading this package alone can -leave an older `launchdarkly-ai-server` in place. The experimental features are: +leave an older `launchdarkly-ai-server` in place. The experimental feature is: -- **Agent Skills**, in `launchdarkly_ai_server.experimental.skills`; see the - [Agent Skills guide](https://github.com/launchdarkly/python-ai-sdk/blob/main/packages/client/README.md#agent-skills-experimental). - **Evaluations from Code**, in `launchdarkly_ai_server.experimental.evaluations`; see the [evaluations guide](https://github.com/launchdarkly/python-ai-sdk/blob/main/packages/client/README.md#run-an-evaluation-from-code-experimental). diff --git a/packages/client/README.md b/packages/client/README.md index 924aefdf..91939e9f 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -471,11 +471,7 @@ asyncio.run(main()) --- -### Agent Skills (experimental) - -> **Experimental.** Import Agent Skills from `launchdarkly_ai_server.experimental.skills`; -> none of these names is exported from the package root. They may change in a minor release, -> so review the changelog when you upgrade. +### Agent Skills Skills are versioned `SKILL.md` documents managed in LaunchDarkly and attached to AI Config variations by reference. The SDK tells you which skills a config references, retrieves their @@ -487,9 +483,9 @@ import asyncio import hashlib from pathlib import Path -from launchdarkly_ai_server import init_client, inspect_config -from launchdarkly_ai_server.experimental.skills import ( - InMemorySkillStore, get_skill, set_skill_store, skill_refs, write_skills, +from launchdarkly_ai_server import ( + InMemorySkillStore, get_skill, init_client, inspect_config, set_skill_store, + skill_refs, write_skills, ) SKILL_MD = "---\nname: PDF Extraction\n---\nExtract text from PDFs.\n" @@ -623,9 +619,8 @@ ERROR ld.skills.integrity_failure {"action":"withheld","event":"ld.skills.integr ``` **Match on `ld.skills.integrity_failure`.** It is the name LaunchDarkly's AI SDKs share for this -record. While Agent Skills is experimental, the event name, its fields, and its `reason_code` -values may change in a minor release; any such change gets a changelog entry under -**Experimental**, so review it before upgrading. JSON keys are sorted, so the line is byte-identical across LaunchDarkly's +record. A change to the event name, its fields, or its `reason_code` values gets a changelog +entry, so review the changelog before upgrading. JSON keys are sorted, so the line is byte-identical across LaunchDarkly's AI SDKs for the same input. | Field | Description | @@ -683,7 +678,7 @@ retrieval, verification, and telemetry as `get_skill`, but reports which of five happened instead of collapsing them all to `None`. ```python -from launchdarkly_ai_server.experimental.skills import get_skill_result +from launchdarkly_ai_server import get_skill_result outcome = await get_skill_result("pdf-extraction") @@ -731,9 +726,7 @@ authenticated with the environment's server-side SDK key. ```python import os -from launchdarkly_ai_server.experimental.skills import ( - FDv2SkillStore, set_skill_store, watch_skills, -) +from launchdarkly_ai_server import FDv2SkillStore, set_skill_store, watch_skills store = FDv2SkillStore(os.environ["LD_SDK_KEY"]).start() if not store.wait_for_skills(timeout=10): diff --git a/packages/client/agents.md b/packages/client/agents.md index b93877a7..13ff1a95 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -79,8 +79,8 @@ from launchdarkly_ai_server import execute_and_track, execute_and_stream, wrap_t from launchdarkly_ai_server import config, graph, resolve_graph from launchdarkly_ai_server.experimental.evaluations import init_evaluations -# Agent Skills (experimental: none of these is exported from the package root) -from launchdarkly_ai_server.experimental.skills import ( +# Agent Skills +from launchdarkly_ai_server import ( set_skill_store, SkillStore, InMemorySkillStore, FDv2SkillStore, StoreDiagnostics, skill_refs, get_skill, get_skill_result, get_skills, all_skills, write_skills, watch_skills, SkillWatcher, @@ -90,12 +90,12 @@ from launchdarkly_ai_server.experimental.skills import ( ) ``` -An experimental feature is exported only from its module under -`launchdarkly_ai_server.experimental`, and its names may change in a minor release. No core -public type, function or `init_client` option may name it. Core reaches it only through an -internal hook (as `shutdown` clears the skill store), and an error there is logged, never -raised into the core call. The experimental part is the SDK's API, not the LaunchDarkly data -model: `AiConfigRep` still documents the `skills` references a config carries. +An experimental feature (today only Evaluations from Code) is exported only from its module +under `launchdarkly_ai_server.experimental`, and its names may change in a minor release. No +core public type, function or `init_client` option may name it. Agent Skills is core and is +exported from the package root; `shutdown` still reaches its state only through an internal +hook (it clears the skill store), and an error there is logged, never raised into +`shutdown`. `MAX_SKILL_CONTENT_BYTES` and `SKILL_OBJECT_KIND` are deliberately **not** exported; both stay internal to `skills_core`: @@ -106,7 +106,7 @@ internal to `skills_core`: would imply otherwise. An adapter that needs it imports it from `launchdarkly_ai_server.skills_core`. -When adding a new core export, add it to `__init__.py`'s imports and `__all__`. An Agent Skills export goes in `experimental/skills.py` instead, and in `EXPERIMENTAL_SKILLS_SURFACE` in `tests/test_skills.py`. Handler packages must never import from sub-paths (e.g. `launchdarkly_ai_server.client`). +When adding a new core export, add it to `__init__.py`'s imports and `__all__`, and to `ROOT_SURFACE` in `tests/test_public_surface.py`. An Agent Skills export also goes in `SKILLS_SURFACE` in `tests/test_skills.py`. Handler packages must never import from sub-paths (e.g. `launchdarkly_ai_server.client`). --- @@ -212,7 +212,7 @@ Three layers, in increasing order of blast radius: (including `null`, or one bad entry) raises `ValueError` rather than returning a partial list that would authorize a prune. So does a config that is not a dict, including the `None` a failed `inspect_config` returns; only a dict with no `skills` key yields `[]`. `parse_ai_config` deliberately does not check `skills`, - so an experimental field cannot fail a core config call (TESTING.md §0.3). + so a malformed field cannot fail a config call that does not use skills. 2. **Content accessors** — `get_skill`, `get_skill_result`, `get_skills`, `all_skills` read through the `SkillStore` seam. Configure a store with `set_skill_store(store)`; with none configured the accessors raise @@ -318,12 +318,12 @@ inside its known-event branches. Empty data on a known event is not JSON either, connection the same way: read as `None`, a `delete-object` would be ignored and a `payload-transferred` would commit with no selector. -**A put or delete that is valid JSON but has no usable `key` is warned and ignored, for the -experimental stage.** Its transfer still commits, so a keyless `delete-object` loses its +**A put or delete that is valid JSON but has no usable `key` is warned and ignored, and +delivery keeps going.** This is the accepted behaviour, not a provisional one. Its transfer still commits, so a keyless `delete-object` loses its revocation and a keyless `put-object` in an `xfer-full` revokes that skill by omission (and `write_skills("*")` prunes it). `ldclient` interrupts the stream instead. Interrupting is not strictly safer, because a server that keeps resending the object would then block every later -change, so the choice is to be revisited before 1.0 (TESTING.md §3.25). Pinned by +change. Pinned by `test_a_delete_with_no_usable_key_is_ignored_and_its_transfer_commits` and `test_a_put_with_no_usable_key_in_an_xfer_full_revokes_by_omission`; changing the rule should fail them. @@ -625,8 +625,9 @@ Do not undo any of these as a simplification: store), and the stdlib's default formatter drops `extra`, so an `extra`-only record is invisible under a plain `logging.basicConfig()`. - **`ld.skills.integrity_failure` is documented for customers to match on.** Do not rename it - casually. The experimental stage allows a rename in a minor release, but only in every SDK - at once and with a changelog entry, because a rename silently breaks customers' alerts. + casually. Agent Skills is core, so a rename is a breaking change: it needs a `!` commit (a + minor bump while on 0.x, a major after 1.0) in every SDK at once and a changelog entry, + because a rename silently breaks customers' alerts. - **`sort_keys=True` makes the line byte-identical across SDKs** (modulo `language`), since the other implementations build the object in alphabetical key order. - **Optional fields are omitted, never nulled**, so a SIEM field-existence check means @@ -689,10 +690,9 @@ the allowlist is enforced in one place. `_clear_state` delegate to `skills_core`. `set_skill_store` and tests use those; none should reach into `skills_core` directly. -**Core never imports this feature.** `skills_core` registers `clear_state` with -`shutdown_hooks.register_shutdown_hook` when it loads, and `shutdown()` runs the registered -hooks. `lifecycle.py` imports no skills module (TESTING.md §0.3, pinned by -`test_core_does_not_import_experimental_code`). +**Lifecycle never imports this feature.** `skills_core` registers `clear_state` with +`shutdown_hooks.register_shutdown_hook` when it loads (the package root imports it), and +`shutdown()` runs the registered hooks. `lifecycle.py` imports no skills module. ### Descriptor-pinned filesystem access @@ -984,6 +984,6 @@ a conversation is out of reach at this layer either way. - `Skill.content` is opaque `bytes`. Do not add anything that parses or interprets it — no YAML library in this package's dependencies at any tier, and no accessor that decodes content. - Do not route skills telemetry through `client.track()`, and do not introduce an LD context anywhere in the skills path. Signals go through the `skills_core.py` emitter seam, whose default is a no-op, and only via its `record_*` functions. - Do not add a signal name outside the three in the Agent Skills table above — the list is an allowlist. `AgentControl Skill SDK Reference Returned` and `AgentControl Skill Content Retrieved` were considered and deliberately excluded from SDK emission. -- Do not rename `ld.skills.integrity_failure` in one language only or without a changelog entry, and do not add an eleventh `reason_code` in one language only — both are documented compatibility surfaces, though the experimental stage lets either change in a minor release. See "The integrity-failure log record" above. +- Do not rename `ld.skills.integrity_failure` in one language only or without a changelog entry, and do not add an eleventh `reason_code` in one language only — both are documented compatibility surfaces. See "The integrity-failure log record" above. - Do not relax any of the `write_skills` filesystem defenses (local key re-validation, symlink refusal, manifest-authorized destruction, corrupt-manifest fail-closed, atomic `0644` writes). Each is a deliberate security property with abuse-case tests attached. - Do not make `SkillStore` lookups key-only. Version is part of the lookup identity because a payload holds several versions of one key; a key-only seam cannot express a version-pinned reference. diff --git a/packages/client/api-report/launchdarkly_ai_server.experimental.skills.txt b/packages/client/api-report/launchdarkly_ai_server.experimental.skills.txt deleted file mode 100644 index 11f3fee8..00000000 --- a/packages/client/api-report/launchdarkly_ai_server.experimental.skills.txt +++ /dev/null @@ -1,85 +0,0 @@ -# Public API of launchdarkly_ai_server.experimental.skills. Generated by `make api-report`; do not edit. -all_skills() -> list[Skill] [async] -FDv2SkillStore [class] - __init__(sdk_key: str, *, base_uri: str | None = None, stream_uri: str | None = None, mode: Mode = 'stream', poll_interval: float = 30.0, read_timeout: float | None = None, initial_backoff: float = 1.0, max_backoff: float = 30.0, _requester: Any = None) - __enter__() -> FDv2SkillStore - __exit__(*_exc: Any) -> None - add_listener(kind: str, fn: Callable[[dict[str, Any]], Any]) -> None - all_objects(kind: str) -> dict[str, dict[str, Any]] - close(timeout: float = 5.0) -> None - diagnostics: StoreDiagnostics [property] - failed: str | None [property] - get_object(kind: str, key: str, version: int | None = None) -> dict[str, Any] | None - is_initialized() -> bool - remove_listener(kind: str, fn: Callable[[dict[str, Any]], Any]) -> None - start() -> FDv2SkillStore - wait_for_skills(timeout: float = 10.0) -> bool -get_skill(key: str, *, version: int | None = None) -> Skill | None [async] -get_skill_result(key: str, *, version: int | None = None) -> SkillOutcome [async] -get_skills(refs: Sequence[SkillReference | str]) -> list[Skill] [async] -InMemorySkillStore [class] - __init__(objects: dict[str, dict[str, Any]] | None = None) - add_listener(kind: str, fn: Callable[[dict[str, Any]], Any]) -> None - all_objects(kind: str) -> dict[str, dict[str, Any]] - get_object(kind: str, key: str, version: int | None = None) -> dict[str, Any] | None - put(raw: dict[str, Any]) -> None - remove_listener(kind: str, fn: Callable[[dict[str, Any]], Any]) -> None -MANIFEST_FILENAME = '.launchdarkly-skills.json' -MANIFEST_VERSION = 1 -OnUnavailable = Literal['keep', 'raise'] -ReconcileAction [class, @dataclass(frozen=True)] - __init__(key: str, action: ReconcileActionKind, version: int | None = None, path: str | None = None, error: str | None = None) - action: ReconcileActionKind - error: str | None = None - key: str - path: str | None = None - version: int | None = None -ReconcileActionKind = Literal['written', 'updated', 'skipped_current', 'removed', 'error'] -ReconcileReport [class, @dataclass(frozen=True)] - __init__(actions: list[ReconcileAction] = list()) - actions: list[ReconcileAction] = field(default_factory=list) - errors: list[ReconcileAction] [property] - ok: bool [property] -set_skill_store(store: SkillStore | None) -> None -Skill [class, @dataclass(frozen=True)] - __init__(key: str, version: int, content: bytes, content_hash: str, name: str | None = None, description: str | None = None) - content: bytes - content_hash: str - description: str | None = None - key: str - name: str | None = None - version: int -SKILL_FILENAME = 'SKILL.md' -skill_refs(config: AiConfigRep | None) -> list[SkillReference] -SkillOutcome [class, @dataclass(frozen=True)] - __init__(skill: Skill | None, reason: SkillOutcomeReason, detail: str | None) - detail: str | None - reason: SkillOutcomeReason - skill: Skill | None -SkillOutcomeReason = Literal['absent', 'integrity_failure', 'ok', 'store_unavailable', 'wrong_version'] -SkillReference [class, @dataclass(frozen=True)] - __init__(key: str, version: int) - key: str - version: int -SkillStore(Protocol) [class] - all_objects(kind: str) -> dict[str, dict[str, Any]] - get_object(kind: str, key: str, version: int | None = None) -> dict[str, Any] | None -SkillWatcher [class] - __init__(request: Sequence[Skill | SkillReference | str] | str, root: str | os.PathLike[str], store: Any, *, prune: bool, timeout: float, on_unavailable: OnUnavailable, debounce: float, on_reconcile: Callable[[ReconcileReport], Any] | None) - __enter__() -> SkillWatcher - __exit__(*_exc: Any) -> None - close(timeout: float = 15.0) -> None - notify(_raw: Any = None) -> None - reconciles: int [property] -StoreDiagnostics [class, @dataclass] - __init__(payloads_transferred: int = 0, skill_objects_received: int = 0, objects_ignored: int = 0, objects_revoked: int = 0, payloads_ignored: int = 0, hashless_objects: int = 0, connection_failures: int = 0, last_error: str | None = None) - connection_failures: int = 0 - hashless_objects: int = 0 - last_error: str | None = None - objects_ignored: int = 0 - objects_revoked: int = 0 - payloads_ignored: int = 0 - payloads_transferred: int = 0 - skill_objects_received: int = 0 -watch_skills(skills: Sequence[Skill | SkillReference | str] | str, root: str | os.PathLike[str], *, prune: bool = True, timeout: float = 10.0, on_unavailable: OnUnavailable = 'keep', debounce: float = DEFAULT_DEBOUNCE_SECONDS, on_reconcile: Callable[[ReconcileReport], Any] | None = None) -> tuple[ReconcileReport, SkillWatcher] [async] -write_skills(skills: Sequence[Skill | SkillReference | str] | str, root: str | os.PathLike[str], *, prune: bool = True, timeout: float = 10.0, on_unavailable: OnUnavailable = 'keep') -> ReconcileReport [async] diff --git a/packages/client/api-report/launchdarkly_ai_server.txt b/packages/client/api-report/launchdarkly_ai_server.txt index d1792118..36e76e69 100644 --- a/packages/client/api-report/launchdarkly_ai_server.txt +++ b/packages/client/api-report/launchdarkly_ai_server.txt @@ -1,6 +1,7 @@ # Public API of launchdarkly_ai_server. Generated by `make api-report`; do not edit. add_cached_tokens_to_input(raw_usage: dict[str, Any]) -> SpanUsage AiConfigRep = dict[str, Any] +all_skills() -> list[Skill] [async] any_multimodal(turns: list[dict[str, Any]]) -> bool build_judge_tasks(*, config: AiConfigRep, user_context: LDContext, handler: ProviderHandler, handlers: list[ProviderHandler] | None = None, llm_response: str, base_track_data: TrackData, user_input: str | None = None, trajectory: str = '') -> list[JudgeTask] [async] compose(a: Registry, b: Registry) -> Registry @@ -27,7 +28,24 @@ execute_and_track(*, config_key: str, config: AiConfigRep, meta: VariationMeta, ExecuteStreamDoneEvent = dict[str, Any] ExecuteStreamEvent = dict[str, Any] extract_variation(config_key: str, context: dict[str, Any]) -> dict[str, Any] [async] +FDv2SkillStore [class] + __init__(sdk_key: str, *, base_uri: str | None = None, stream_uri: str | None = None, mode: Mode = 'stream', poll_interval: float = 30.0, read_timeout: float | None = None, initial_backoff: float = 1.0, max_backoff: float = 30.0, _requester: Any = None) + __enter__() -> FDv2SkillStore + __exit__(*_exc: Any) -> None + add_listener(kind: str, fn: Callable[[dict[str, Any]], Any]) -> None + all_objects(kind: str) -> dict[str, dict[str, Any]] + close(timeout: float = 5.0) -> None + diagnostics: StoreDiagnostics [property] + failed: str | None [property] + get_object(kind: str, key: str, version: int | None = None) -> dict[str, Any] | None + is_initialized() -> bool + remove_listener(kind: str, fn: Callable[[dict[str, Any]], Any]) -> None + start() -> FDv2SkillStore + wait_for_skills(timeout: float = 10.0) -> bool get_client() -> Any +get_skill(key: str, *, version: int | None = None) -> Skill | None [async] +get_skill_result(key: str, *, version: int | None = None) -> SkillOutcome [async] +get_skills(refs: Sequence[SkillReference | str]) -> list[Skill] [async] global_registry = Registry() graph(key: str, *, handlers: list[ProviderHandler] | None = None, tool_handlers: dict[str, Callable[..., Any] | NativeTool] | None = None, registry: Any = None, graph_judge: str | None = None) -> GraphInstance GraphArgs = dict[str, Any] @@ -72,6 +90,13 @@ has_multimodal_content(content: MessageContent) -> bool image_block_to_url(block: dict[str, Any]) -> str init_client(options: InitClientOptions | None = None, client: Any = None) -> Any [async] InitClientOptions = dict[str, Any] +InMemorySkillStore [class] + __init__(objects: dict[str, dict[str, Any]] | None = None) + add_listener(kind: str, fn: Callable[[dict[str, Any]], Any]) -> None + all_objects(kind: str) -> dict[str, dict[str, Any]] + get_object(kind: str, key: str, version: int | None = None) -> dict[str, Any] | None + put(raw: dict[str, Any]) -> None + remove_listener(kind: str, fn: Callable[[dict[str, Any]], Any]) -> None InputTokenDetails [class, @dataclass] __init__(uncached: int = 0, cache_read: int = 0, cache_creation: int = 0) cache_creation: int = 0 @@ -120,6 +145,8 @@ LDClientInterface [class] variation(key: str, context: LDContext, default_value: Any) -> Any [async] LDContext = dict[str, Any] make_track_data(node: GraphNode, graph_key: str, run_id: str) -> dict[str, Any] +MANIFEST_FILENAME = '.launchdarkly-skills.json' +MANIFEST_VERSION = 1 Message [class] content: str role: Literal['user', 'assistant', 'system'] @@ -132,6 +159,7 @@ NativeTool [class] normalize_mode(mode: str | None) -> Literal['agent', 'messages'] number_or_zero(value: Any) -> int omit_model_stamps(track_data: Mapping[str, Any]) -> dict[str, Any] +OnUnavailable = Literal['keep', 'raise'] parse_ai_config(raw: Any) -> ParseResult parse_json_with_possible_fences(raw_text: str) -> Any | None parse_template(template: str, variables: dict[str, Any]) -> str @@ -164,6 +192,19 @@ ProviderResponse(Generic[T]) [class, @dataclass] response: T track_data: TrackData | None = None usage: UsageDict +ReconcileAction [class, @dataclass(frozen=True)] + __init__(key: str, action: ReconcileActionKind, version: int | None = None, path: str | None = None, error: str | None = None) + action: ReconcileActionKind + error: str | None = None + key: str + path: str | None = None + version: int | None = None +ReconcileActionKind = Literal['written', 'updated', 'skipped_current', 'removed', 'error'] +ReconcileReport [class, @dataclass(frozen=True)] + __init__(actions: list[ReconcileAction] = list()) + actions: list[ReconcileAction] = field(default_factory=list) + errors: list[ReconcileAction] [property] + ok: bool [property] register_ai_sdk_package(name: str, version: str) -> None Registry [class] __init__(*, handlers: list[ProviderHandler] | None = None, tools: dict[str, Callable[..., Any] | NativeTool] | None = None) @@ -192,10 +233,41 @@ set_input_content_attributes(span: Any, capture: bool, *, system_instructions: s set_ld_span_attributes(span: Any, variables: dict[str, Any] | None) -> None set_model_identity_attributes(span: Any, provider_name: str, request_model: str, legacy_system: str | None = None) -> None set_output_content_attributes(span: Any, capture: bool, messages: list[SpanMessage]) -> None +set_skill_store(store: SkillStore | None) -> None set_tool_call_content_attributes(span: Any, capture: bool, *, arguments: Any = None, result: Any = None) -> None set_tool_definition_attributes(span: Any, capture: bool, tools: list[ToolDefinitionInput]) -> None set_usage_span_attributes(span: Any, usage: SpanUsage) -> None shutdown() -> None [async] +Skill [class, @dataclass(frozen=True)] + __init__(key: str, version: int, content: bytes, content_hash: str, name: str | None = None, description: str | None = None) + content: bytes + content_hash: str + description: str | None = None + key: str + name: str | None = None + version: int +SKILL_FILENAME = 'SKILL.md' +skill_refs(config: AiConfigRep | None) -> list[SkillReference] +SkillOutcome [class, @dataclass(frozen=True)] + __init__(skill: Skill | None, reason: SkillOutcomeReason, detail: str | None) + detail: str | None + reason: SkillOutcomeReason + skill: Skill | None +SkillOutcomeReason = Literal['absent', 'integrity_failure', 'ok', 'store_unavailable', 'wrong_version'] +SkillReference [class, @dataclass(frozen=True)] + __init__(key: str, version: int) + key: str + version: int +SkillStore(Protocol) [class] + all_objects(kind: str) -> dict[str, dict[str, Any]] + get_object(kind: str, key: str, version: int | None = None) -> dict[str, Any] | None +SkillWatcher [class] + __init__(request: Sequence[Skill | SkillReference | str] | str, root: str | os.PathLike[str], store: Any, *, prune: bool, timeout: float, on_unavailable: OnUnavailable, debounce: float, on_reconcile: Callable[[ReconcileReport], Any] | None) + __enter__() -> SkillWatcher + __exit__(*_exc: Any) -> None + close(timeout: float = 15.0) -> None + notify(_raw: Any = None) -> None + reconciles: int [property] SpanMessage [class, @dataclass] __init__(role: str, parts: list[SpanMessagePart] = list(), finish_reason: str | None = None) finish_reason: str | None = None @@ -219,6 +291,16 @@ SpanUsage [class, @dataclass] cache_read: int = 0 input: int = 0 output: int = 0 +StoreDiagnostics [class, @dataclass] + __init__(payloads_transferred: int = 0, skill_objects_received: int = 0, objects_ignored: int = 0, objects_revoked: int = 0, payloads_ignored: int = 0, hashless_objects: int = 0, connection_failures: int = 0, last_error: str | None = None) + connection_failures: int = 0 + hashless_objects: int = 0 + last_error: str | None = None + objects_ignored: int = 0 + objects_revoked: int = 0 + payloads_ignored: int = 0 + payloads_transferred: int = 0 + skill_objects_received: int = 0 StreamChunkEvent = dict[str, Any] StreamDoneEvent = dict[str, Any] StreamEvent = dict[str, Any] @@ -240,4 +322,6 @@ UsageDict [class, @dataclass] output: int = 0 total: int = 0 VariationMeta = dict[str, Any] +watch_skills(skills: Sequence[Skill | SkillReference | str] | str, root: str | os.PathLike[str], *, prune: bool = True, timeout: float = 10.0, on_unavailable: OnUnavailable = 'keep', debounce: float = DEFAULT_DEBOUNCE_SECONDS, on_reconcile: Callable[[ReconcileReport], Any] | None = None) -> tuple[ReconcileReport, SkillWatcher] [async] wrap_tool_handlers(tool_handlers: dict[str, Callable[..., Any] | NativeTool] | None, user_context: LDContext, track_data: TrackData) -> dict[str, Callable[..., Any]] +write_skills(skills: Sequence[Skill | SkillReference | str] | str, root: str | os.PathLike[str], *, prune: bool = True, timeout: float = 10.0, on_unavailable: OnUnavailable = 'keep') -> ReconcileReport [async] diff --git a/packages/client/src/launchdarkly_ai_server/__init__.py b/packages/client/src/launchdarkly_ai_server/__init__.py index 952cb8ed..edd26b63 100644 --- a/packages/client/src/launchdarkly_ai_server/__init__.py +++ b/packages/client/src/launchdarkly_ai_server/__init__.py @@ -55,6 +55,25 @@ ) from .sdk_info import SDK_INFO_CONTEXT, SDK_INFO_EVENT, register_ai_sdk_package from .sdk_usage import report_usage +from .skills import ( + InMemorySkillStore, + all_skills, + get_skill, + get_skill_result, + get_skills, + set_skill_store, + skill_refs, +) +from .skills_core import SkillStore +from .skills_fdv2 import FDv2SkillStore, StoreDiagnostics +from .skills_fs import ( + MANIFEST_FILENAME, + MANIFEST_VERSION, + SKILL_FILENAME, + OnUnavailable, + write_skills, +) +from .skills_watch import SkillWatcher, watch_skills from .tracking import execute_and_stream, execute_and_track, wrap_tool_handlers from .types import ( NATIVE_TOOL_KEY, @@ -85,7 +104,14 @@ ProviderGraphResponse, ProviderHandler, ProviderResponse, + ReconcileAction, + ReconcileActionKind, + ReconcileReport, SDKOptions, + Skill, + SkillOutcome, + SkillOutcomeReason, + SkillReference, StreamChunkEvent, StreamDoneEvent, StreamEvent, @@ -236,6 +262,36 @@ "graph", "resolve_graph", "GraphInstance", + # Agent Skills: configuration + "set_skill_store", + "SkillStore", + "InMemorySkillStore", + # Agent Skills: LaunchDarkly delivery store and on-change re-reconcile + "FDv2SkillStore", + "StoreDiagnostics", + "watch_skills", + "SkillWatcher", + # Agent Skills: accessors + "skill_refs", + "get_skill", + "get_skill_result", + "get_skills", + "all_skills", + "write_skills", + # Agent Skills: value types + "Skill", + "SkillReference", + "SkillOutcome", + "ReconcileAction", + "ReconcileReport", + # Agent Skills: literal types for typed consumers + "ReconcileActionKind", + "OnUnavailable", + "SkillOutcomeReason", + # Agent Skills: on-disk filenames and manifest version + "SKILL_FILENAME", + "MANIFEST_FILENAME", + "MANIFEST_VERSION", ] register_ai_sdk_package("launchdarkly-ai-server", __version__) diff --git a/packages/client/src/launchdarkly_ai_server/experimental/__init__.py b/packages/client/src/launchdarkly_ai_server/experimental/__init__.py index 80ecfd7d..1d2e1e28 100644 --- a/packages/client/src/launchdarkly_ai_server/experimental/__init__.py +++ b/packages/client/src/launchdarkly_ai_server/experimental/__init__.py @@ -1,7 +1,7 @@ """ Experimental features of the LaunchDarkly AI SDK. -Each feature is a submodule, for example ``launchdarkly_ai_server.experimental.skills``. +Each feature is a submodule, for example ``launchdarkly_ai_server.experimental.evaluations``. Names imported from here may change in a minor release, so review the changelog when you upgrade. Names from the package root change only in a major release. """ diff --git a/packages/client/src/launchdarkly_ai_server/experimental/skills.py b/packages/client/src/launchdarkly_ai_server/experimental/skills.py deleted file mode 100644 index 18a2c9c6..00000000 --- a/packages/client/src/launchdarkly_ai_server/experimental/skills.py +++ /dev/null @@ -1,70 +0,0 @@ -""" -Agent Skills (experimental). - -Configure a store with ``set_skill_store``, then read verified skill content with -``get_skill``, ``get_skills`` or ``all_skills``, or write it to disk with -``write_skills``. These names may change in a minor release, so review the -changelog when you upgrade. -""" - -from ..skills import ( - InMemorySkillStore, - all_skills, - get_skill, - get_skill_result, - get_skills, - set_skill_store, - skill_refs, -) -from ..skills_core import SkillStore -from ..skills_fdv2 import FDv2SkillStore, StoreDiagnostics -from ..skills_fs import ( - MANIFEST_FILENAME, - MANIFEST_VERSION, - SKILL_FILENAME, - OnUnavailable, - write_skills, -) -from ..skills_watch import SkillWatcher, watch_skills -from ..types import ( - ReconcileAction, - ReconcileActionKind, - ReconcileReport, - Skill, - SkillOutcome, - SkillOutcomeReason, - SkillReference, -) - -__all__ = [ # noqa: RUF022 - # configuration - "set_skill_store", - "SkillStore", - "InMemorySkillStore", - # LaunchDarkly delivery store and on-change re-reconcile - "FDv2SkillStore", - "StoreDiagnostics", - "watch_skills", - "SkillWatcher", - # accessors - "skill_refs", - "get_skill", - "get_skill_result", - "get_skills", - "all_skills", - "write_skills", - # value types - "Skill", - "SkillReference", - "SkillOutcome", - "ReconcileAction", - "ReconcileReport", - # literal types for typed consumers - "ReconcileActionKind", - "OnUnavailable", - "SkillOutcomeReason", - # on-disk filenames and manifest version - "SKILL_FILENAME", - "MANIFEST_FILENAME", - "MANIFEST_VERSION", -] diff --git a/packages/client/src/launchdarkly_ai_server/lifecycle.py b/packages/client/src/launchdarkly_ai_server/lifecycle.py index 0d626f2d..f9ad8f9a 100644 --- a/packages/client/src/launchdarkly_ai_server/lifecycle.py +++ b/packages/client/src/launchdarkly_ai_server/lifecycle.py @@ -401,8 +401,8 @@ async def shutdown() -> None: Shuts down the singleton client. Idempotent — safe to call multiple times even if the client was never initialized or already shut down. - Also runs the registered shutdown hooks, which clear the state of any - experimental feature in use, and restores process-wide SDK defaults from + Also runs the registered shutdown hooks, which clear feature state such as + the configured skill store, and restores process-wide SDK defaults from ``configure``. When telemetry was running, this also releases the process-global tracer diff --git a/packages/client/src/launchdarkly_ai_server/shutdown_hooks.py b/packages/client/src/launchdarkly_ai_server/shutdown_hooks.py index 06f42a7a..4bb53add 100644 --- a/packages/client/src/launchdarkly_ai_server/shutdown_hooks.py +++ b/packages/client/src/launchdarkly_ai_server/shutdown_hooks.py @@ -1,10 +1,10 @@ """ Cleanup hooks that ``shutdown()`` runs. -This is how core reaches experimental state without importing it: an -experimental module registers its own cleanup when it loads, and ``shutdown()`` -runs whatever is registered. Core never imports, names or calls experimental -code (TESTING.md §0.3), so a hook's *name* comes from the module that +This is how ``shutdown()`` reaches a feature's state without importing it: a +feature module (Agent Skills, say) registers its own cleanup when it loads, and +``shutdown()`` runs whatever is registered. Lifecycle code never imports, names +or calls a feature module, so a hook's *name* comes from the module that registers it. """ @@ -30,8 +30,8 @@ def register_shutdown_hook(name: str, hook: Callable[[], None]) -> None: def run_shutdown_hooks(*, raise_errors: bool = False) -> None: """ - Runs every registered hook. Experimental code must not break a core call, so - a hook that raises is logged and the rest still run. *raise_errors* is for + Runs every registered hook. A feature's cleanup must not break + ``shutdown()``, so a hook that raises is logged and the rest still run. *raise_errors* is for the test reset, where a failed cleanup should fail the test instead. """ for name, hook in list(_hooks.items()): diff --git a/packages/client/src/launchdarkly_ai_server/skills.py b/packages/client/src/launchdarkly_ai_server/skills.py index c44b9ff7..a1fc06e9 100644 --- a/packages/client/src/launchdarkly_ai_server/skills.py +++ b/packages/client/src/launchdarkly_ai_server/skills.py @@ -16,6 +16,7 @@ from typing import Any from . import skills_core +from .sdk_usage import report_usage from .skills_core import ( SKILL_OBJECT_KIND, SkillStore, @@ -220,6 +221,7 @@ def skill_refs(config: AiConfigRep | None) -> list[SkillReference]: objects with a valid key and an integer version >= 1. This includes ``skills: null``. """ + report_usage("client.skillRefs") if not isinstance(config, dict): raise ValueError( f"skill_refs was given {type(config).__name__}, not an AI Config. " @@ -261,6 +263,7 @@ async def get_skill(key: str, *, version: int | None = None) -> Skill | None: Raises: RuntimeError: If no skill store is configured. """ + report_usage("client.getSkill") return resolve_from_store(require_store(), key, version).skill @@ -287,6 +290,7 @@ async def get_skill_result(key: str, *, version: int | None = None) -> SkillOutc Raises: RuntimeError: If no skill store is configured. """ + report_usage("client.getSkillResult") resolved = resolve_from_store(require_store(), key, version) return SkillOutcome( skill=resolved.skill, reason=resolved.reason, detail=resolved.error @@ -310,6 +314,7 @@ async def get_skills(refs: Sequence[SkillReference | str]) -> list[Skill]: TypeError: If *refs* is a single string; pass ``[key]`` instead. RuntimeError: If no skill store is configured. """ + report_usage("client.getSkills") if isinstance(refs, str): # A str type-checks as Sequence[str] but would be iterated per character. raise TypeError( @@ -344,6 +349,7 @@ async def all_skills() -> list[Skill]: Raises: RuntimeError: If no skill store is configured. """ + report_usage("client.allSkills") objects, error = list_raw_objects(require_store()) if error is not None: return [] diff --git a/packages/client/src/launchdarkly_ai_server/skills_core.py b/packages/client/src/launchdarkly_ai_server/skills_core.py index e5cb6cbc..101fcd8c 100644 --- a/packages/client/src/launchdarkly_ai_server/skills_core.py +++ b/packages/client/src/launchdarkly_ai_server/skills_core.py @@ -3,7 +3,7 @@ interface and configured store, telemetry, integrity verification, and store resolution. -Package-internal except ``SkillStore``, which ``experimental.skills`` re-exports. This +Package-internal except ``SkillStore``, which the package root re-exports. This module imports neither ``skills`` nor ``skills_fs``. **Everything a store returns is untrusted.** Key, version, size and content hash @@ -94,7 +94,7 @@ NO_STORE_MESSAGE = ( "No skill store is configured, so skill content cannot be retrieved. Configure " - "one with set_skill_store(store) from launchdarkly_ai_server.experimental.skills " + "one with set_skill_store(store) from launchdarkly_ai_server " "— FDv2SkillStore receives content from LaunchDarkly, and InMemorySkillStore is " "available for local development and testing." ) diff --git a/packages/client/src/launchdarkly_ai_server/skills_fs.py b/packages/client/src/launchdarkly_ai_server/skills_fs.py index 2d7bf29d..34d8607e 100644 --- a/packages/client/src/launchdarkly_ai_server/skills_fs.py +++ b/packages/client/src/launchdarkly_ai_server/skills_fs.py @@ -43,6 +43,7 @@ pinned_directory, unlink_file, ) +from .sdk_usage import report_usage from .skills_core import ( NO_STORE_MESSAGE, Resolution, @@ -166,6 +167,21 @@ async def write_skills( large reconcile blocks the event loop. Wrap it in ``asyncio.to_thread`` if that matters. Descriptor pinning of the root requires POSIX; see ``safe_fs``. """ + report_usage("client.writeSkills") + return await _write_skills( + skills, root, prune=prune, timeout=timeout, on_unavailable=on_unavailable + ) + + +async def _write_skills( + skills: Sequence[Skill | SkillReference | str] | str, + root: str | os.PathLike[str], + *, + prune: bool = True, + timeout: float = 10.0, + on_unavailable: OnUnavailable = "keep", +) -> ReconcileReport: + """``write_skills`` without the usage report, for the watcher's reconciles.""" # Typed as closed sets, but untyped callers can pass anything. if on_unavailable not in ("keep", "raise"): raise ValueError( diff --git a/packages/client/src/launchdarkly_ai_server/skills_watch.py b/packages/client/src/launchdarkly_ai_server/skills_watch.py index 0283a2d3..c1d7c6bf 100644 --- a/packages/client/src/launchdarkly_ai_server/skills_watch.py +++ b/packages/client/src/launchdarkly_ai_server/skills_watch.py @@ -22,8 +22,9 @@ from collections.abc import Callable, Sequence from typing import Any +from .sdk_usage import report_usage from .skills_core import SKILL_OBJECT_KIND, get_store -from .skills_fs import OnUnavailable, write_skills +from .skills_fs import OnUnavailable, _write_skills from .types import ReconcileReport, Skill, SkillReference logger = logging.getLogger(__name__) @@ -114,7 +115,7 @@ def _run(self) -> None: def _reconcile_once(self) -> None: try: report = asyncio.run( - write_skills( + _write_skills( self._request, self._root, prune=self._prune, @@ -224,11 +225,12 @@ async def watch_skills( ``add_listener`` (use ``write_skills`` for a one-shot reconcile). ValueError: If *debounce* is negative, ``NaN``, or infinite. """ + report_usage("client.watchSkills") store = get_store() if store is None: raise RuntimeError( "watch_skills needs a configured skill store. Configure one with " - "set_skill_store(store) from launchdarkly_ai_server.experimental.skills." + "set_skill_store(store) from launchdarkly_ai_server." ) add_listener = getattr(store, "add_listener", None) if not callable(add_listener): @@ -260,7 +262,7 @@ async def watch_skills( ) try: # Run on the caller's task, so a bad root raises to the caller. - report = await write_skills( + report = await _write_skills( skills, root, prune=prune, timeout=timeout, on_unavailable=on_unavailable ) except BaseException: diff --git a/packages/client/src/launchdarkly_ai_server/types.py b/packages/client/src/launchdarkly_ai_server/types.py index 45139325..80bf1119 100644 --- a/packages/client/src/launchdarkly_ai_server/types.py +++ b/packages/client/src/launchdarkly_ai_server/types.py @@ -95,7 +95,7 @@ class Message: ``model``, ``provider``, at least one of ``instructions``, ``messages``, or a non-empty ``classifiers`` list, and an optional ``skills`` array of ``{key, version}`` references. Parsing does not validate ``skills``; read it -with ``launchdarkly_ai_server.experimental.skills.skill_refs``, which does. +with ``launchdarkly_ai_server.skill_refs``, which does. """ VariationMeta = dict[str, Any] diff --git a/packages/client/src/launchdarkly_ai_server/types_validation.py b/packages/client/src/launchdarkly_ai_server/types_validation.py index c7850bc1..a03305dd 100644 --- a/packages/client/src/launchdarkly_ai_server/types_validation.py +++ b/packages/client/src/launchdarkly_ai_server/types_validation.py @@ -67,8 +67,8 @@ def skills_field_rejection_reason(raw: Any) -> str | None: """ Why a present ``skills`` field is malformed, or ``None`` when it is valid. - Used by ``skill_refs``, not by ``parse_ai_config``: Agent Skills is - experimental, and a malformed field must not fail a core config call. + Used by ``skill_refs``, not by ``parse_ai_config``: a malformed field must + not fail a config call that does not use skills. One malformed reference rejects the whole field, never a partial list. """ if not isinstance(raw, list): @@ -151,8 +151,8 @@ def parse_ai_config(raw: Any) -> ParseResult: error={"message": "outputFormat must be an object (JSON Schema)"}, ) - # ``skills`` is passed through unvalidated. Agent Skills is experimental, so - # a malformed field must not fail a core config call (TESTING.md §0.3); - # ``skill_refs`` rejects it where the references are used. + # ``skills`` is passed through unvalidated, so a malformed field does not fail + # a config call that does not use skills; ``skill_refs`` rejects it where the + # references are used. return ParseSuccess(success=True, data=raw) diff --git a/packages/client/tests/conftest.py b/packages/client/tests/conftest.py index 14f7a7dd..de69af0c 100644 --- a/packages/client/tests/conftest.py +++ b/packages/client/tests/conftest.py @@ -7,7 +7,7 @@ import launchdarkly_ai_server.lifecycle as lifecycle_module import launchdarkly_ai_server.skills as skills_module -from launchdarkly_ai_server.experimental.skills import InMemorySkillStore +from launchdarkly_ai_server import InMemorySkillStore @pytest.fixture diff --git a/packages/client/tests/test_lifecycle.py b/packages/client/tests/test_lifecycle.py index b9ddb4e1..78cb2ed9 100644 --- a/packages/client/tests/test_lifecycle.py +++ b/packages/client/tests/test_lifecycle.py @@ -3,7 +3,9 @@ Reference: TESTING.md §3.9 """ +import ast import asyncio +import inspect import os import subprocess import sys @@ -613,14 +615,30 @@ def failing() -> None: assert "Could not clear the Failing Feature state" in caplog.messages later.assert_called_once() # one failing hook does not skip the rest + def test_lifecycle_imports_no_skills_module(self) -> None: + """agents.md: lifecycle reaches Agent Skills only through the shutdown hooks.""" + tree = ast.parse(inspect.getsource(lifecycle_module)) + imported: set[str] = set() + for node in ast.walk(tree): + if isinstance(node, ast.Import): + imported.update(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom): + imported.add(node.module or "") + # `from . import skills_core` names the module in the alias, not in `module`. + imported.update(alias.name for alias in node.names) + skills = sorted( + name for name in imported if name.rsplit(".", 1)[-1].startswith("skills") + ) + assert skills == [] + def test_core_does_not_import_experimental_code(self) -> None: - """Core never hooks into experimental code (TESTING.md §0.3).""" + """Importing and shutting down core loads no experimental module.""" script = ( "import asyncio, sys\n" "import launchdarkly_ai_server\n" "asyncio.run(launchdarkly_ai_server.shutdown())\n" "loaded = sorted(m for m in sys.modules if m.startswith(" - "('launchdarkly_ai_server.skills', 'launchdarkly_ai_server.evaluations'," + "('launchdarkly_ai_server.evaluations'," " 'launchdarkly_ai_server.experimental')))\n" "print(','.join(loaded))\n" ) @@ -870,8 +888,8 @@ async def test_returns_enabled_true_and_config_when_variation_is_enabled( assert result["meta"] is not None async def test_a_malformed_skills_field_does_not_fail_core_calls(self) -> None: - """Agent Skills is experimental, so its field cannot break a core call - (TESTING.md §0.3). ``skill_refs`` rejects it instead.""" + """A malformed ``skills`` field cannot break a config call that does + not use skills. ``skill_refs`` rejects it instead.""" stub = _make_stub_client() stub.variation = AsyncMock( return_value={ diff --git a/packages/client/tests/test_public_surface.py b/packages/client/tests/test_public_surface.py index ef8c06d7..ec8972e5 100644 --- a/packages/client/tests/test_public_surface.py +++ b/packages/client/tests/test_public_surface.py @@ -116,6 +116,31 @@ "to_usage_dict", "tool_result_string", "wrap_tool_handlers", + # Agent Skills + "set_skill_store", + "SkillStore", + "InMemorySkillStore", + "FDv2SkillStore", + "StoreDiagnostics", + "watch_skills", + "SkillWatcher", + "skill_refs", + "get_skill", + "get_skill_result", + "get_skills", + "all_skills", + "write_skills", + "Skill", + "SkillReference", + "SkillOutcome", + "ReconcileAction", + "ReconcileReport", + "ReconcileActionKind", + "OnUnavailable", + "SkillOutcomeReason", + "SKILL_FILENAME", + "MANIFEST_FILENAME", + "MANIFEST_VERSION", } ) diff --git a/packages/client/tests/test_schema.py b/packages/client/tests/test_schema.py index e58164d4..22f00924 100644 --- a/packages/client/tests/test_schema.py +++ b/packages/client/tests/test_schema.py @@ -106,9 +106,9 @@ class TestParseAiConfigSkills: """ ``skills`` is passed through unvalidated. - Agent Skills is experimental, so a malformed ``skills`` field must not fail - a core config call (TESTING.md §0.3). ``skill_refs`` validates the field - where the references are used; see ``TestSkillRefs`` in test_skills.py. + A malformed ``skills`` field must not fail a config call that does not use + skills. ``skill_refs`` validates the field where the references are used; + see ``TestSkillRefs`` in test_skills.py. """ def _base(self, **extra: Any) -> dict[str, Any]: diff --git a/packages/client/tests/test_sdk_usage.py b/packages/client/tests/test_sdk_usage.py index 5d3c1990..fdf99121 100644 --- a/packages/client/tests/test_sdk_usage.py +++ b/packages/client/tests/test_sdk_usage.py @@ -1,5 +1,7 @@ """Tests for TESTING.md §3.27 helper usage events.""" +import time +from pathlib import Path from typing import Any from unittest.mock import AsyncMock, MagicMock, patch @@ -9,17 +11,28 @@ import launchdarkly_ai_server.sdk_usage as sdk_usage_module from launchdarkly_ai_server import ( SDK_INFO_CONTEXT, + FDv2SkillStore, + InMemorySkillStore, ProviderHandler, + SkillReference, __version__, + all_skills, build_judge_tasks, config, create_handler, + get_skill, + get_skill_result, + get_skills, graph, init_client, inspect_config, resolve_graph, run_judge, + set_skill_store, shutdown, + skill_refs, + watch_skills, + write_skills, ) from launchdarkly_ai_server.experimental.evaluations import init_evaluations @@ -499,3 +512,148 @@ async def test_package_name_without_version_reports_the_core( assert len(calls) == 1 assert calls[0].args[2]["helperPackageName"] == "launchdarkly-ai-server" assert calls[0].args[2]["helperPackageVersion"] == __version__ + + +# --------------------------------------------------------------------------- +# Agent Skills helpers (§3.27) +# --------------------------------------------------------------------------- + +SKILL_KEY = "test-skill" +_SKILL_CONFIG: dict[str, Any] = { + **_ENABLED_CONFIG, + "skills": [{"key": SKILL_KEY, "version": 1}], +} + + +def _skill_store(make_raw_skill: Any) -> InMemorySkillStore: + store = InMemorySkillStore() + store.put(make_raw_skill(key=SKILL_KEY, version=1)) + set_skill_store(store) + return store + + +async def _call_skill_helper(helper: str, root: Path) -> None: + if helper == "client.allSkills": + await all_skills() + elif helper == "client.getSkill": + await get_skill(SKILL_KEY) + elif helper == "client.getSkillResult": + await get_skill_result(SKILL_KEY) + elif helper == "client.getSkills": + await get_skills([SKILL_KEY]) + elif helper == "client.skillRefs": + skill_refs(_SKILL_CONFIG) + elif helper == "client.watchSkills": + _report, watcher = await watch_skills("*", root, debounce=0.05) + watcher.close() + else: + assert helper == "client.writeSkills", helper + await write_skills("*", root) + + +@pytest.mark.parametrize( + "helper", + [ + "client.allSkills", + "client.getSkill", + "client.getSkillResult", + "client.getSkills", + "client.skillRefs", + "client.watchSkills", + "client.writeSkills", + ], +) +async def test_skill_helper_sends_one_usage_event( + client: MagicMock, helper: str, tmp_path: Path, make_raw_skill: Any +) -> None: + _skill_store(make_raw_skill) + await _call_skill_helper(helper, tmp_path / "skills") + _expect_usage(client, helper) + # One call emits only its own helper, and never an experimental.* skills ID. + assert _helpers(client) == [helper] + + +async def test_get_skill_reports_once_per_client( + client: MagicMock, make_raw_skill: Any +) -> None: + _skill_store(make_raw_skill) + await get_skill(SKILL_KEY) + await get_skill(SKILL_KEY) + _expect_usage(client, "client.getSkill") + + await shutdown() + fresh = await _install() + # shutdown clears the store, so configure it again. + _skill_store(make_raw_skill) + await get_skill(SKILL_KEY) + _expect_usage(fresh, "client.getSkill") + + +async def test_skill_refs_before_client_is_sent_on_init() -> None: + await shutdown() + lifecycle_module._reset_for_testing() + skill_refs(_SKILL_CONFIG) + client = _client() + assert _usage(client) == [] + with patch.object(lifecycle_module, "_setup_telemetry", return_value=None): + await init_client(client=client) + _expect_usage(client, "client.skillRefs") + await shutdown() + + +async def test_get_skill_without_store_still_reports(client: MagicMock) -> None: + with pytest.raises(RuntimeError): + await get_skill(SKILL_KEY) + _expect_usage(client, "client.getSkill") + + +async def test_watch_skills_reconciles_report_no_write_skills( + client: MagicMock, tmp_path: Path, make_raw_skill: Any +) -> None: + store = _skill_store(make_raw_skill) + _report, watcher = await watch_skills("*", tmp_path / "skills", debounce=0.05) + try: + assert _helpers(client) == ["client.watchSkills"] + store.put(make_raw_skill(key="other", version=1)) + deadline = time.monotonic() + 10 + while watcher.reconciles < 1 and time.monotonic() < deadline: + time.sleep(0.01) + assert watcher.reconciles >= 1 + assert _helpers(client) == ["client.watchSkills"] + finally: + watcher.close() + + +@pytest.mark.parametrize( + "request_", + [[SkillReference(key=SKILL_KEY, version=1)], [SKILL_KEY], "*"], + ids=["reference", "string", "star"], +) +async def test_write_skills_resolves_without_reporting_accessors( + client: MagicMock, tmp_path: Path, make_raw_skill: Any, request_: Any +) -> None: + _skill_store(make_raw_skill) + await write_skills(request_, tmp_path / "skills") + assert _helpers(client) == ["client.writeSkills"] + + +@pytest.mark.parametrize( + ("helper", "silent"), + [ + ("client.getSkills", "client.getSkill"), + ("client.getSkillResult", "client.getSkill"), + ("client.getSkill", "client.getSkillResult"), + ], +) +async def test_accessors_do_not_report_each_other( + client: MagicMock, make_raw_skill: Any, helper: str, silent: str, tmp_path: Path +) -> None: + _skill_store(make_raw_skill) + await _call_skill_helper(helper, tmp_path) + assert _usage(client, silent) == [] + + +async def test_store_setup_is_not_a_helper(client: MagicMock) -> None: + set_skill_store(InMemorySkillStore()) + FDv2SkillStore("sdk-test-key") + assert _usage(client) == [] diff --git a/packages/client/tests/test_skills.py b/packages/client/tests/test_skills.py index ebfe133a..4b67c382 100644 --- a/packages/client/tests/test_skills.py +++ b/packages/client/tests/test_skills.py @@ -7,6 +7,7 @@ import dataclasses import hashlib +import importlib import json import unicodedata from types import SimpleNamespace @@ -17,8 +18,7 @@ import launchdarkly_ai_server.lifecycle as lifecycle_module import launchdarkly_ai_server.skills as skills_module -from launchdarkly_ai_server import get_client, init_client, shutdown -from launchdarkly_ai_server.experimental.skills import ( +from launchdarkly_ai_server import ( InMemorySkillStore, ReconcileAction, ReconcileReport, @@ -26,10 +26,13 @@ SkillOutcome, SkillReference, all_skills, + get_client, get_skill, get_skill_result, get_skills, + init_client, set_skill_store, + shutdown, skill_refs, ) from launchdarkly_ai_server.skills_core import list_raw_objects, require_store @@ -363,7 +366,7 @@ def test_requires_no_client_or_store(self, mock_ld_client: Any) -> None: mock_ld_client.track.assert_not_called() -EXPERIMENTAL_SKILLS_SURFACE = frozenset( +SKILLS_SURFACE = frozenset( { "set_skill_store", "SkillStore", @@ -391,58 +394,47 @@ def test_requires_no_client_or_store(self, mock_ld_client: Any) -> None: "MANIFEST_VERSION", } ) -"""Every name ``launchdarkly_ai_server.experimental.skills`` exports, spelled out -so that any addition — an accessor that interprets ``Skill.content``, say — -fails here and is seen in review.""" +"""Every Agent Skills name the package root exports, spelled out. The check here is +that each one is exported. What catches an addition (an accessor that interprets +``Skill.content``, say) is ``test_root_surface_is_exactly_the_snapshot`` in +``test_public_surface.py``, which compares the whole of ``__all__``.""" class TestPackageExports: """ What is and is not part of the public surface. - Agent Skills is experimental: every name is exported from - ``launchdarkly_ai_server.experimental.skills`` and none from the package - root. The literal values are spelled out on purpose: this is the one place - the constants themselves are asserted, so importing them to build the - expectation would make the assertion circular. + Agent Skills is core: every name is exported from the package root, and + ``launchdarkly_ai_server.experimental.skills`` no longer exists. The literal + values are spelled out on purpose: this is the one place the constants + themselves are asserted, so importing them to build the expectation would + make the assertion circular. """ - def test_experimental_surface_is_exactly_the_documented_set(self) -> None: - """An allowlist, not a denylist of guessed names: a new export of any - name, including a reader over ``Skill.content``, fails this test.""" - from launchdarkly_ai_server.experimental import skills as experimental - - assert set(experimental.__all__) == EXPERIMENTAL_SKILLS_SURFACE - for name in EXPERIMENTAL_SKILLS_SURFACE: - assert hasattr(experimental, name), name - - def test_experimental_module_defines_no_other_public_name(self) -> None: - """``__all__`` is what ``import *`` and the docs follow, but a public - name defined on the module and left out of it is still importable.""" - from launchdarkly_ai_server.experimental import skills as experimental + def test_every_skills_name_is_exported_from_the_package_root(self) -> None: + import launchdarkly_ai_server as package - public = {name for name in dir(experimental) if not name.startswith("_")} - assert public == EXPERIMENTAL_SKILLS_SURFACE + assert SKILLS_SURFACE <= set(package.__all__) + for name in SKILLS_SURFACE: + assert hasattr(package, name), name - def test_no_skills_name_is_exported_from_the_package_root(self) -> None: - import launchdarkly_ai_server as package - import launchdarkly_ai_server.experimental.skills + def test_the_experimental_skills_module_is_gone(self) -> None: + import launchdarkly_ai_server.experimental as experimental - for name in EXPERIMENTAL_SKILLS_SURFACE: - assert name not in package.__all__, name - assert not hasattr(package, name), name + with pytest.raises(ImportError): + importlib.import_module("launchdarkly_ai_server.experimental.skills") + for name in SKILLS_SURFACE: + assert not hasattr(experimental, name), name def test_content_cap_is_not_public_api(self) -> None: """The content cap stays internal to ``skills_core`` — see the ``MAX_SKILL_CONTENT_BYTES`` docstring there for why it is not exported.""" import launchdarkly_ai_server as package from launchdarkly_ai_server import skills_core - from launchdarkly_ai_server.experimental import skills as experimental assert skills_core.MAX_SKILL_CONTENT_BYTES == 10485760 - for module in (package, experimental): - assert "MAX_SKILL_CONTENT_BYTES" not in module.__all__ - assert not hasattr(module, "MAX_SKILL_CONTENT_BYTES") + assert "MAX_SKILL_CONTENT_BYTES" not in package.__all__ + assert not hasattr(package, "MAX_SKILL_CONTENT_BYTES") def test_object_kind_is_not_public_api(self) -> None: """The kind is an SDK-side interface value, not the wire contract. @@ -455,19 +447,17 @@ def test_object_kind_is_not_public_api(self) -> None: """ import launchdarkly_ai_server as package from launchdarkly_ai_server import skills_core - from launchdarkly_ai_server.experimental import skills as experimental assert skills_core.SKILL_OBJECT_KIND == "skill" - for module in (package, experimental): - assert "SKILL_OBJECT_KIND" not in module.__all__ - assert not hasattr(module, "SKILL_OBJECT_KIND") + assert "SKILL_OBJECT_KIND" not in package.__all__ + assert not hasattr(package, "SKILL_OBJECT_KIND") def test_constants_have_their_documented_values(self) -> None: - from launchdarkly_ai_server.experimental import skills as experimental + import launchdarkly_ai_server as package - assert experimental.SKILL_FILENAME == "SKILL.md" - assert experimental.MANIFEST_FILENAME == ".launchdarkly-skills.json" - assert experimental.MANIFEST_VERSION == 1 + assert package.SKILL_FILENAME == "SKILL.md" + assert package.MANIFEST_FILENAME == ".launchdarkly-skills.json" + assert package.MANIFEST_VERSION == 1 def test_exported_closed_set_types_admit_exactly_their_tokens(self) -> None: """Each union must match the values it actually types. @@ -478,17 +468,17 @@ def test_exported_closed_set_types_admit_exactly_their_tokens(self) -> None: """ import typing - from launchdarkly_ai_server.experimental import skills as experimental + import launchdarkly_ai_server as package - assert set(typing.get_args(experimental.ReconcileActionKind)) == { + assert set(typing.get_args(package.ReconcileActionKind)) == { "written", "updated", "skipped_current", "removed", "error", } - assert set(typing.get_args(experimental.OnUnavailable)) == {"keep", "raise"} - assert set(typing.get_args(experimental.SkillOutcomeReason)) == { + assert set(typing.get_args(package.OnUnavailable)) == {"keep", "raise"} + assert set(typing.get_args(package.SkillOutcomeReason)) == { "absent", "integrity_failure", "ok", @@ -857,7 +847,7 @@ async def test_set_skill_store_rejects_a_non_store( async def test_init_client_ignores_a_skill_store_option( self, make_raw_skill: Any, caplog: pytest.LogCaptureFixture ) -> None: - """Core client options do not configure an experimental feature, and + """Core client options do not configure the skill store, and the ignored option is reported rather than dropped silently.""" store = InMemorySkillStore() store.put(make_raw_skill(key="a")) diff --git a/packages/client/tests/test_skills_fdv2.py b/packages/client/tests/test_skills_fdv2.py index b35c5d72..99eab3c9 100644 --- a/packages/client/tests/test_skills_fdv2.py +++ b/packages/client/tests/test_skills_fdv2.py @@ -34,9 +34,12 @@ import pytest -from launchdarkly_ai_server import skills as skills_module -from launchdarkly_ai_server import skills_core, skills_fdv2 -from launchdarkly_ai_server.experimental.skills import ( +import launchdarkly_ai_server.skills as skills_module + +# Internal submodules, kept out of the public-name import below so they don't read as root exports. +import launchdarkly_ai_server.skills_core as skills_core +import launchdarkly_ai_server.skills_fdv2 as skills_fdv2 +from launchdarkly_ai_server import ( FDv2SkillStore, InMemorySkillStore, all_skills, @@ -913,11 +916,12 @@ def test_an_unknown_event_name_is_ignored(self) -> None: def test_a_delete_with_no_usable_key_is_ignored_and_its_transfer_commits( self, ) -> None: - """Pins TESTING.md §3.25's experimental-stage rule: ignored, not fatal. + """Pins the keyless-object rule: ignored, not fatal, and delivery keeps + going. The revocation it carried is lost, so ``revoked`` stays served. Base - ``ldclient`` interrupts the stream instead; the choice is to be - revisited before 1.0, and changing it should fail this test. + ``ldclient`` interrupts the stream instead; this SDK's choice is the + accepted behaviour, and changing it should fail this test. """ held = _SkillObjectSet() reader = _ProtocolReader(held) @@ -4363,7 +4367,7 @@ def test_it_adds_no_dependency(self) -> None: assert _module_imports(skills_fdv2) == stdlib | within_the_feature def test_nothing_in_the_feature_imports_it(self) -> None: - """Only the experimental entry point names it, to re-export + """Only the package root names it, to re-export ``FDv2SkillStore`` and ``StoreDiagnostics``. That re-export is the published surface rather than a dependency: it is @@ -4390,9 +4394,9 @@ def test_nothing_in_the_feature_imports_it(self) -> None: ] assert importers == [] - # The positive control: the entry point does name it, so an empty result + # The positive control: the package root does name it, so an empty result # above cannot be a misspelt filename list finding nothing anywhere. - entry_point = source_dir / "experimental" / "skills.py" + entry_point = source_dir / "__init__.py" assert "skills_fdv2" in entry_point.read_text(encoding="utf-8") diff --git a/packages/client/tests/test_skills_fs.py b/packages/client/tests/test_skills_fs.py index 6eeb08a2..5fb48d8e 100644 --- a/packages/client/tests/test_skills_fs.py +++ b/packages/client/tests/test_skills_fs.py @@ -22,12 +22,13 @@ import launchdarkly_ai_server.skills as skills_module import launchdarkly_ai_server.skills_core as skills_core_module import launchdarkly_ai_server.skills_fs as skills_fs_module -from launchdarkly_ai_server import init_client, parse_ai_config -from launchdarkly_ai_server.experimental.skills import ( +from launchdarkly_ai_server import ( InMemorySkillStore, Skill, SkillReference, get_skill, + init_client, + parse_ai_config, skill_refs, write_skills, ) diff --git a/packages/client/tests/test_skills_watch.py b/packages/client/tests/test_skills_watch.py index 72dee6d6..7f80767d 100644 --- a/packages/client/tests/test_skills_watch.py +++ b/packages/client/tests/test_skills_watch.py @@ -22,7 +22,7 @@ import pytest -from launchdarkly_ai_server.experimental.skills import ( +from launchdarkly_ai_server import ( InMemorySkillStore, set_skill_store, watch_skills, diff --git a/release-please-config.json b/release-please-config.json index e370a841..9c94b155 100644 --- a/release-please-config.json +++ b/release-please-config.json @@ -158,9 +158,6 @@ } }, "changelog-sections": [ - { "type": "feat", "scope": "skills", "section": "Experimental" }, - { "type": "fix", "scope": "skills", "section": "Experimental" }, - { "type": "perf", "scope": "skills", "section": "Experimental" }, { "type": "feat", "scope": "evaluations", "section": "Experimental" }, { "type": "fix", "scope": "evaluations", "section": "Experimental" }, { "type": "perf", "scope": "evaluations", "section": "Experimental" },