docs(client-contract): state build joining, the fetch wait and the base URL rule

python/client-contract/CHANGELOG.md

@@ -21,6 +21,7 @@ All notable changes to the `dfo-spec-kit-client-contract` Python package will be
 - `without_party_totals` in the ref's artifact rendering, `true` for the version rendered with party totals left out and absent for a plain rendering, with `ref-without-party-totals` and `ref-without-party-totals-false-rejected` as its examples.
 - `tools/capture_client_contract.py`, which captures `served/openapi.json` from a dfo-apis checkout in process and records the scenarios against a base URL, dev by default, or with `record --offline` derives their outcomes again from the records with no request. A local app's signed URLs are written as the storage URLs a deployment signs.
 - `tools/validate_client_contract.py`, which checks the contract against its schemas, the served document and `registries/codes.json`, checks `client-codes.json` against them and against the reasons the key-lookup rule raises, and checks each scenario's `expect` against the outcomes its exchanges, `actions.json` and `client-codes.json` give, each Search request against the input its steps derive, a sentence lookup's `entity_groups` included, each build's `input.scope` against the narrowing its step names over the spec's bound roles, each build's options against its layout, each Search request's whole body, each download against its artifact's digest, each recorded scenario's commit against `served/SOURCE.json`, each problem document's code against `registries/codes.json` or, for the website proxy's, `proxy-codes.json`, each proxy problem document's content type, members and `retryable: true`, each 202 against an operation that reads `done`, and each absolute URL against the storage host, each ref example's members against the dataset members and downloaded files of the scenario it names, and the scenarios directory against a 16,000,000-byte budget for records and bodies together. A record that ends a run of retries while a retry remains does not fit. The outcomes include each long build's operation id, poll waits and resubmissions, and a `VersionUnavailableError`'s newest version.
-- The client rules and their vectors: the credentials path and file format, the key lookup order, with an empty `api_key` refused as `key_empty` and an empty profile key as `profile_without_key`, the key format check, and the cache directory and layout.
+- The contract README states that a POST for a build identical to one pending or running returns that operation, so a client that stops waiting posts the same build to join it, and that `fetch` waits for a build with no whole-call deadline by default, with a download that delivers no byte for 30 s a retryable `network_error`.
+- The client rules and their vectors: the credentials path and file format, the key lookup order, with an empty `api_key` refused as `key_empty` and an empty profile key as `profile_without_key`, the key format check, the base URL order, `base_url_invalid` for an invalid value, and the cache directory and layout.
 
 [unreleased]: https://github.com/DevelopmentFinanceObservatory/spec-kit/commits/main/python

python/client-contract/README.md

@@ -176,15 +176,15 @@ The records also fix these client rules:
 - Fetching an answer with a gap raises `IncompleteAnswerError` before any request, unless the fetch is partial and some part has a reading.
 - A fetch of an answer issues one `SearchWarning` when a definite part has an operation Search did not run, a departure, or a warning at severity `warn` or `degraded` with a `message`, or the answer's `diagnostics.warnings` holds such a warning. An `info` warning, a warning with no `message`, and an operation that does not read under Strict fields issue nothing. Each way is one line a client shows: `Not done: <verb> (<phrase as JSON>)` coded by the verb, the phrase quoted as a JSON string that keeps non-ASCII characters as they are, the departure's `text` coded by its code, or `Note: <message>` coded by the warning's code, in part order with the diagnostics warnings last, and ways that show the same line count once, under the first one's code. A strict fetch raises the `SearchWarning` before any build, with the code of its one line, or a null code when there are several lines. A strict fetch raises it before the `frame` layout's several-member check.
 - A ref's version is found in this order: the snapshot route and the files route at the stored digest; the file selector, `GET /datasets/{id}/{content_version}/file` with the ref's rendering as the query; and a build of the ref's spec in the ref's rendering, which must answer the ref's `spec_id` and the same `content_version`. Both recovery steps keep the ref's identity, so a recovered member carries the ref's `spec_id` and `content_version`: a snapshot manifest whose `definition.spec_id` or `content_version` is not the ref's, or a rebuild manifest whose `definition.spec_id` is not the ref's, raises `ContractError` `contract_violation` before any download, which `fetch-ref-expired-rebuild-other-spec` pins for the rebuild. A snapshot refusal other than `artifact_expired` ends the lookup, so `snapshot_not_found` raises `NotFoundError` with no further request. A files route or selector refusal of `artifact_expired`, `artifact_not_found` or `not_found` takes the next step, and a selector file of another digest is not the stored file. A snapshot manifest whose artifacts list no artifact of the ref member's digest sends the client to the selector in the ref member's own rendering, and a file that hashes to that digest gives the dataset, which `fetch-ref-artifact-unlisted` pins. A rebuild that answers another version, or that raises `InvalidRequestError`, `NotFoundError` or `VersionUnavailableError`, ends in `VersionUnavailableError` with the first refusal's code, or `version_unavailable` when nothing refused. Its `newest_version` is the version the rebuild answered, or else the newest stored version, as a `newest_version` step finds it: the first version `GET /datasets/{id}/versions?limit=100` lists with a null `base_content_version`, the newest plain build, read page by page through `&cursor=` with each page's `next_cursor` until one is listed. The route lists versions newest first and needs no key. A version with a `base_content_version` is derived from a plain build, a groupby collapse or a party-totals cut, and is skipped, which `fetch-ref-newest-version` pins. The newest version is null when the list ends without a plain version or the route refuses with `NotFoundError`. Any other refusal of the versions route raises, and a page whose `versions` is not a list of versions each with a `content_version` and a `base_content_version` that is a string or null, or whose `next_cursor` is neither a string nor null, raises `ContractError` `contract_violation`, which `newest-version-unreadable-page` pins.
-- A build answers 200 with the manifest, or 202 with an operation the client polls at `/datasets/operations/{operation_id}`, the id of the operation it holds. The 202 reads `pending` or `running`, because a submission starts a build or joins one under way, and a `done` operation left by an earlier submission of a rejected head is built again. A deterministic failure answers a 202 that reads `failed`. `fetch-spec-async-build` pins `pending` and `fetch-spec-accepted-running` pins `running`. A mint's build answers 202. A record whose mint answers 200 fits no mint step. Before each poll it waits the `Retry-After` of the refusal that answered the last poll, whatever its status, as every retry of a refused request waits; else the last operation resource's `retry_after_ms`, else the `Retry-After` of the last answered poll, the response that carried that resource, else a backoff of 1, 2, 4 and then 8 seconds that advances one step each time it is used, a poll after a refusal included. `fetch-spec-poll-pacing-changes` pins a `retry_after_ms` that changes from one poll to the next, so a client waits each poll's own value.
+- A build answers 200 with the manifest, or 202 with an operation the client polls at `/datasets/operations/{operation_id}`, the id of the operation it holds. The 202 reads `pending` or `running`, because a submission starts a build or joins one under way, and a `done` operation left by an earlier submission of a rejected head is built again. A deterministic failure answers a 202 that reads `failed`. `fetch-spec-async-build` pins `pending` and `fetch-spec-accepted-running` pins `running`. A POST for a build identical to one that is pending or running returns that same operation, because the server derives the operation id from the request: dfo-apis `src/datasets_api/operation_store.py` `compute_operation_id` hashes the request identity, the origin catalogue build id and the rendering. A client that stops waiting for a build can post the same build again to join it. A mint's build answers 202. A record whose mint answers 200 fits no mint step. Before each poll it waits the `Retry-After` of the refusal that answered the last poll, whatever its status, as every retry of a refused request waits; else the last operation resource's `retry_after_ms`, else the `Retry-After` of the last answered poll, the response that carried that resource, else a backoff of 1, 2, 4 and then 8 seconds that advances one step each time it is used, a poll after a refusal included. `fetch-spec-poll-pacing-changes` pins a `retry_after_ms` that changes from one poll to the next, so a client waits each poll's own value.
 - A retryable 429 poll spends no retry, and a 429 whose body says it is not retryable raises at once, as a refusal's `retryable` outranks its status, which `fetch-spec-poll-rate-limited-not-retryable` pins. Any other retryable refused poll spends one of the two retries, which a poll that answers restores, and a refusal that is not retryable raises. `fetch-spec-poll-refused` pins two 503s polled again after an answered poll, `fetch-spec-poll-refused-backoff` a 503 with no `Retry-After` among polls no response paces, after which the backoff advances, `fetch-spec-poll-refused-retry-after` a 503 whose `Retry-After` paces the next poll, and `fetch-spec-poll-not-found` a 404 that raises at once. A poll that answers a failed operation whose error is retryable posts the same build once more and polls the operation it starts; a second failure, an error that is not retryable, and a resumed operation, whose build request the client does not hold, raise the operation's error. A refused poll whose code is `operation_failed` is a failed operation with that error. A failed operation whose resource carries no `error` raises `ServerError` with `operation_failed`, retryable by status 500 as a refusal without a document is, which `operation-failed-without-error` pins. A 202 or resumed operation that already reads `failed` raises its error before any poll.
 - Any other refusal is retried when it is retryable: as its problem document's `retryable` says, else when its status is 408, 429, 500, 502, 503 or 504. A client makes two retries by default, and the scenarios assume that default. A build's POST shares those two retries with every other refusal of it, and once a 504 cuts the build off the client posts it once more only, whatever that post answers: a build refused 504 then 503 raises the 503, a build refused 503 then 504 is posted once more after the 504, and a second 504 raises. `fetch-spec-504-then-503`, `fetch-spec-503-then-504` and `fetch-spec-504-twice` pin the three. A refusal that is not retryable, or one past the retries, raises its error.
 - Each retry waits the refusal's `Retry-After` when it carries one, and otherwise the client's backoff for that attempt: 0.5 seconds before the first retry, doubling to at most 8. The build's 504 cut-off retry follows the same rule. `retry_waits` pins these waits, and `fetch-spec-504-retry-after` pins a cut-off that carries `Retry-After`.
 - A 401 resolves the key again by the key-lookup rule. When the key changed, the client sends the request once more at once, with no wait and spending no retry. When it did not, or the second answer is a 401 too, the 401 raises `AuthenticationError`. A replay's key never changes, so `search-unauthorized` pins the second case and no scenario the first.
 - A `request(method, path)` whose `path` is not a path on the base URL, because it starts with `//` or names a scheme, raises `InvalidRequestError` `invalid_argument` before any request. No scenario step makes such a request.
 - A 2xx response of a JSON route with no JSON body raises `ContractError` `unexpected_response`. `search-no-json-body` pins it for Search.
 - A `Deprecation` or `Sunset` header on an API response, either alone, issues `route_deprecated` once per process per distinct pair of their values, and an `x-dfo-warning` header issues `server_warning` once per process per message. A signed fetch's headers issue nothing. `search-sunset-warning` pins a Sunset header alone, issued once over two responses that carry the same value, and `search-server-warning` pins a repeated `x-dfo-warning` message issued once, a new message issued again, and a Deprecation beside the same Sunset issued as a new pair.
-- Python synchronous platform name resolution can exceed the following timeout and deadline limits. Asynchronous name resolution is cancellable. Timeouts follow the clients' design and are not replayed, since the replay advances the clock only by the waits the client asks for: the client's `timeout`, 30 s by default, bounds one request attempt; `search(timeout=60)` bounds the whole search, each attempt reading until that deadline, and None means no limit; `fetch(timeout=1800)` bounds the whole fetch, each download getting the time left on the fetch's deadline; a synchronous build POST, and mint's POST, read for up to 150 s within any outer deadline; `result(timeout=1800)` bounds the wait, and None means no limit. A poll attempt that hangs past the per-attempt timeout spends a retry, and a deadline that passes during a body read raises `DfoTimeoutError` `deadline_exceeded`. In TypeScript an infinite remaining time arms no timer, and a finite one above 2^31-1 ms is clamped. No scenario step takes a timeout.
+- Python synchronous platform name resolution can exceed the following timeout and deadline limits. Asynchronous name resolution is cancellable. Timeouts follow the clients' design and are not replayed, since the replay advances the clock only by the waits the client asks for: the client's `timeout`, 30 s by default, bounds one request attempt; `search(timeout=60)` bounds the whole search, each attempt reading until that deadline, and None means no limit; `fetch` waits for a build with no whole-call deadline by default, and `fetch(timeout=60)` bounds the whole fetch, each download getting the time left on the fetch's deadline; a download that delivers no byte for 30 s raises the retryable `network_error`; a synchronous build POST, and mint's POST, read for up to 150 s within any outer deadline; `result(timeout=1800)` bounds the wait, and None means no limit. A poll attempt that hangs past the per-attempt timeout spends a retry, and a deadline that passes during a body read raises `DfoTimeoutError` `deadline_exceeded`. In TypeScript an infinite remaining time arms no timer, and a finite one above 2^31-1 ms is clamped. No scenario step takes a timeout.
 
 ## Recording
 
@@ -215,7 +215,7 @@ A record's `origin.base_url` is the base URL the run was given, a local app's in
 
 ## Rules
 
-`rules/README.md` states the shared rules for credentials, key lookup and format, and disk caching. A rule applies where the client and runtime implement its capability. Python implements disk caching; TypeScript currently keeps no disk cache, so `cache-dir` and `cache-layout` apply to Python. Credentials-file rules apply to runtimes with local credentials support. Each rule's vectors are in `rules/<rule>.json`.
+`rules/README.md` states the shared rules for credentials, key lookup and format, the base URL, and disk caching. A rule applies where the client and runtime implement its capability. Python implements disk caching; TypeScript currently keeps no disk cache, so `cache-dir` and `cache-layout` apply to Python. Credentials-file rules apply to runtimes with local credentials support. Each rule's vectors are in `rules/<rule>.json`.
 
 ## Validation
 

python/client-contract/src/dfo_client_contract/contract.py

@@ -13,6 +13,7 @@
     "credentials-path",
     "credentials-file",
     "key-lookup",
+    "base-url",
     "cache-dir",
     "cache-layout",
 )

python/client-contract/src/dfo_client_contract/rules/README.md

@@ -1,6 +1,6 @@
 # Client rules
 
-Clients implement these rules for the capabilities available in their runtime. Python implements every rule. TypeScript implements the key and credentials rules in Node; browser runtimes use the key sources available there. TypeScript currently keeps no disk cache, so `cache-dir` and `cache-layout` apply to Python. Clients implementing the same capability use the same rules and formats. Each rule's vectors are in `<rule>.json`, validated by `rules.schema.json`. In a vector, `env` lists the environment variables set, and an empty value counts as unset everywhere below.
+Clients implement these rules for the capabilities available in their runtime. Python implements every rule. TypeScript implements the key, credentials and base-URL rules in Node; browser runtimes use the key sources available there. TypeScript currently keeps no disk cache, so `cache-dir` and `cache-layout` apply to Python. Clients implementing the same capability use the same rules and formats. Each rule's vectors are in `<rule>.json`, validated by `rules.schema.json`. In a vector, `env` lists the environment variables set, and an empty value counts as unset everywhere below.
 
 ## Credentials path (`credentials-path`)
 
@@ -34,6 +34,16 @@ A client resolves its key when it is built, and again after a 401. The first sou
 
 A missing file goes on to the next source, and so does a missing `default` profile. A file that fails to parse raises `credentials_file_invalid`, a profile named by `DFO_PROFILE` that the file lacks raises `profile_not_found`, and a chosen profile with no `key`, or an empty one, raises `profile_without_key`. The file is read only when no earlier source wins. The argument may also be a callable or a key object, called or unwrapped when the client resolves the key. The vectors pass strings, and the Colab source has no vectors.
 
+## Base URL (`base-url`)
+
+A client resolves its base URL when it is built. The first source found wins:
+
+1. the `base_url` argument of the call or constructor, as given;
+1. `DFO_BASE_URL`;
+1. the public default, `https://developmentfinance.org/api/v1`.
+
+Only the chosen source is checked. A base URL is valid when it is an absolute `http` or `https` URL with a host and no userinfo. An invalid one raises `base_url_invalid`, so an empty `base_url` argument raises it where an empty `DFO_BASE_URL` counts as unset. A valid value is used as given; the rule fixes the source and the validity, and leaves the URL's form to the client. A key sent over plain `http` to a host that is not loopback is refused by `insecure_transport`, which this rule does not decide.
+
 ## Key format (`key-format`)
 
 A secret key is `dfo_k_`, then a base62 body, then a 6-character checksum; `dfo_p_` marks a browser-safe key. Base62 is `0-9A-Za-z` in that order. The checksum is the CRC32 (ISO-HDLC, as zlib computes it) of the body's ASCII bytes, written in base62 and left-padded with `0` to six characters. Issued keys have a 32-character body, 44 characters in all, but the check ignores length. The last six characters after the prefix are the checksum and the rest is the body.