Repo structure: split provider-node, break the storage-client dependency, tidy crate layout

[bkontur]

Why

• provider-node depends on storage-client — an inverted edge (server pulling in the client SDK).
• provider-node is one ~8.4k-LOC crate mixing storage, HTTP, and on-chain coordinators.
• Crate layout is inconsistent (runtime/ vs runtimes/, dir names != crate names, flat vs nested).

Also sets up a clean home for utils/storage-cli (#175).

1. Break provider-node -> storage-client

What it actually pulls from storage-client is two unrelated things:

• Pure wire types (no I/O): negotiate agreement::{NegotiateRequest, AgreementTermsOf, SignedTerms} + sign_terms + discovery::ProviderInfo, used server-side in the /negotiate handler.
• A second chain client: chain_state_coordinator.rs uses storage_client::ProviderClient to talk to the parachain over WS — get_provider_info, fetch_replay_hsn, fetch_request_timeout, plus a finalized-block subscription parsed with StorageProviderEventParser + substrate::PALLET_NAME.

⚠️ The provider already has its own chain client. subxt_client.rs (SubxtChainClient, ~917 LOC) talks to the same parachain directly via subxt. So today there are two parallel chain-access paths in one node — the node's own SubxtChainClient and the reads/subscription routed through storage_client::ProviderClient. Step 1 isn't just deleting a types dependency; it collapses those two clients into one.

• Extract storage-negotiation primitive crate (wire types + MultiSignature serde + sign_terms); AgreementTerms etc. already live in primitives.
• Move the chain reads + event subscription (get_provider_info, fetch_replay_hsn, fetch_request_timeout, finalized-block stream) onto the node's own SubxtChainClient (post-Reuse one SubxtChainClient struct that implements all chain client traits in provider-node #159), retiring the ProviderClient path.
• Drop storage-client from provider-node.

2. Split provider-node -> provider/{storage,rpc,node}

Crate	Modules	subxt?
provider/storage	storage/*, fs_index, s3_index, mmr	no
provider/rpc (HTTP API)	api, fs_api, s3_api, auth, negotiate, error, types	no
provider/node (bin)	main, command, cli, lib, subxt_client, coordinators	yes

Direction: node -> rpc -> storage -> primitives; only node touches the chain. rpc gets chain data via traits it defines, with subxt-backed impls supplied by node (extends #145).

3. Crate layout

Changed (Rust crates that move / get created)

pallets/                   <- consolidates scattered pallets
  storage-provider/        was pallet/                                   (pallet-storage-provider)
  drive-registry/          was storage-interfaces/file-system/pallet-registry
  s3-registry/             was storage-interfaces/s3/pallet-s3-registry
primitives/
  storage/                 was primitives/                               (storage-primitives)
  file-system/             was storage-interfaces/file-system/primitives
  s3/                      was storage-interfaces/s3/primitives
  negotiation/             NEW — wire types, MultiSignature serde, sign_terms (step 1)
provider/                  <- was the single provider-node/ crate
  storage/                 storage/*, fs_index, s3_index, mmr
  rpc/                     api, fs_api, s3_api, auth(core), negotiate, error, types
  node/                    main, command, cli, lib, subxt_client, coordinators  (the bin)
runtimes/
  parachain/               was runtimes/web3-storage-local               (storage-parachain-runtime)
  paseo/                   was runtimes/web3-storage-paseo
clients/                   <- was client/ + storage-interfaces/*/client
  storage/                 was client/                                   (storage-client)
  file-system/             was storage-interfaces/file-system/client
  s3/                      was storage-interfaces/s3/client
utils/
  storage-cli/             NEW (#175)

Net effect: pallet/, primitives/, client/, provider-node/, and the whole
storage-interfaces/ tree are dissolved into the role-based groups above;
runtimes/ subdirs are renamed. Align dir names to crate names while moving
(today every dir mismatches its crate, e.g. client/ -> storage-client).

Unchanged (stay where they are)

precompiles/               drive-registry, s3-registry, storage-provider
packages/                  JS/TS SDK: core, layer0, layer1, papi, sdk (the old `sdk/ ???` slot)
user-interfaces/           JS/TS UIs: landing, drive-ui, provider, s3-ui, console-ui, shared
zombienet/                 zombienet.toml + local network config
chain-specs/  docs/  examples/  templates/  scripts/  .bin/  .config/  .github/

Their contents don't move, but the mechanical reorg PR must still fix path
references inside scripts/*.sh, zombienet.toml, the justfile, CI
workflows under .github/, and docs that cite moved crate paths.

Note: runtime/ vs runtimes/ (in "Why") is already resolved — only runtimes/ exists today.

Plan (each its own PR; reorg last)

• Break the coupling (step 1)
• Split provider-node — storage first, then rpc, leaving node as the bin
• Directory reorg — one mechanical git mv PR fixing Cargo paths + CI + scripts together

[naijauser]

Hi @bkontur, can I pick this up?

[bkontur]

Hi @bkontur, can I pick this up?

@naijauser Hey, not yet, please, still some pieces are moving around