Skip to content

feat(storage): locate and explicitly export one Seafile library offline - #40

Draft
servicememeai wants to merge 14 commits into
devfrom
feat/offline-seafile-library-inspect-export-20261024
Draft

servicememeai wants to merge 14 commits into
devfrom
feat/offline-seafile-library-inspect-export-20261024

Conversation

@servicememeai

Copy link
Copy Markdown

Implements a bounded offline maintenance slice of #33. This is not a new backup engine.

Why

V2 per-library local storage keeps Seafile's storage/{commits,fs,blocks}/<repo_uuid> object layout beneath a stable repo_key. The original Seafile seaf-fsck --export ... <repo-id> reads that layout without Seafile DB, but default native -d cannot find the extra root when CloudFile is unavailable.

Changes

  • New standalone Python 3 stdlib CLI tools/ops/seafile-library-offline.py:
    • inspect --repo-id UUID --seafile-data ABS --library-root ABS: read-only single-library physical locator; no database, API, full filesystem scan, or copy. Identifies either original CE or one managed per-key layout, rejects ambiguous/missing/symlink routes.
    • export --repo-id UUID --seafile-data ABS --library-root ABS --seaf-fsck ABS --output-dir NEW_ABS: explicit potentially heavy copy of only the requested library's logical files, reusing unmodified native Seafile seaf-fsck --export. A temporary standard Seafile config/object view uses read-only source path links so the compiled Seafile binary works without CloudFile DB/Hub. Never overwrites existing output.
    • verify-export --repo-id UUID --output-dir ABS [--hash]: explicit export file inventory / optional byte hashing. Reports that this is not a proof of source completeness.
  • Standalone regression test (old CE, managed custom key, duplicate/invalid routes, no implicit export, mocked Seafile binary and verification).
  • Operator-only documentation describing Seafile CE14 native export restrictions.

Non-goals / safety

No default bulk backup, no scheduling, no DDL, no archive format, no new CF table, no custom Commit/Block decoder, no file writing into Seafile object storage, no implicit all-library operation. S3 not supported by this filesystem-only tool. This is separate from whole-instance disaster recovery.

Validation

5 isolated synthetic tests passed locally, including the native-binary invocation contract via a simulated seaf-fsck. Real CE14 binary export and encrypted/corrupted library cases remain to be tested before release; the quick source CI is not runtime E2E. No deployment or data changes.

@servicememeai
servicememeai marked this pull request as draft October 24, 2026 07:57
servicememeai and others added 4 commits October 24, 2026 17:11
… order

os.walk descends in filesystem readdir order, so the same logical export
hashed differently after a copy/re-export, which makes tree_sha256 useless
for comparison. Sort each directory list before recursing and cover it with
a deterministic regression test.
@opendev9

Copy link
Copy Markdown

Independent review fix pushed in b966489 (review-only commit on top of the PR head):

  • tools/ops/seafile-library-offline.py: verify-export computed tree_sha256 by walking with os.walk and sorting only file names; subdirectory descent followed readdir order, so the same logical export hashed differently after a copy/re-export, defeating the point of --hash. Sort directories before descending; added a deterministic regression test that fails before the fix and passes after (suite now 8/8).
  • Verified: 8/8 tests pass, py_compile OK, quick static CI passes.

Review verdict unchanged: KEEP DRAFT until one real CE14 seaf-fsck --export run (native CE and auto-local layouts). Unverified risks found by review: the tool calls the seaf-fsck binary directly without SEAFILE_LD_LIBRARY_PATH (the shipped seaf-fsck.sh sets it), and native export can exit 0 after internal partial failures, so a damaged library can be published as EXPORTED_UNVERIFIED with missing files — README line 51 over-promises on that.

Copy link
Copy Markdown
Author

2026-10-24 targeted recovery export correctness

The native CE14 server/gc/fsck.c implementation has void create_file() /void export_file(): after a missing/failed Block read it can print Failed to export file ... but the top-level seaf-fsck --export still exits 0. This could cause a wrapper to publish an incomplete single-repo export as a success.

This Draft PR now reads the native export output log streamingly and refuses publication on native Failed to export file, Failed to mkdir or Failed to open block, even with exit code zero. Added an isolated fake-CE regression producing one matching output directory plus a soft file error; wrapper must fail with no published destination. Latest Quick CI success.

This makes the explicit repo_id export safer but does not prove source-complete recovery: missing top-level/descendant directories may still lack a definitive machine-readable native error. True CE14 binary recovery acceptance is pending. No automatic backup or all-library copy is introduced.

seaf-fsck's export_file()/create_file() are void: CE can skip one file, a whole
directory subtree (unreadable FS dir object) or an uncreatable destination file,
still print "Finish exporting files" and exit 0. The previous guard only matched
three per-file messages, so a missing subdir/root dir object or a g_open()
failure (e.g. ENOSPC) was published as EXPORTED_UNVERIFIED.

Treat any native [WARNING]/[ERROR] record plus the known INFO failure signatures
("Failed to export file", "Failed to mkdir", "... is missing.") as failure: no
destination is published, exit code is 2, and the matched native log lines are
surfaced for diagnosis. The scan stays incremental.

Also mirror the shipped seaf-fsck.sh: prepend the seafile/lib and seafile/lib64
siblings of the given binary to LD_LIBRARY_PATH (keeping any inherited value).
A direct call into 14.0.8 images otherwise dies with "libevent-2.1.so.7: cannot
open shared object file". No-op when that install layout is absent.

Six regression tests use log lines captured verbatim from real CE14
seaf-fsck --export runs plus the library-path handoff; a benign INFO-only
progress log still publishes.

README states the precise publish conditions, the direct-binary invocation and
its library-path handling, and that log-based detection cannot prove content
completeness.

Verified against real 14.0 images: valid export publishes; missing block,
missing subdir object and missing root dir object are refused; the 14.0.8 image
now exports without any caller LD_LIBRARY_PATH.
@opendev9

Copy link
Copy Markdown

Second-round review: independent fix pushed in e05a7f2 (review-only commit on top of the PR head, no history rewrite). The delta 20b10b4 fixed real partial-export cases, but review against the real seaf-fsck binary from real 14.0 images found two reproducible zero-exit false-successes with hidden identical lib/14.0.8 images (C export_file()/create_file() are void, seaf-fsck.c returns 0 unconditionally):

  • missing sub-directory FS object -> whole subtree omitted, tool published exit 0 with an empty sub/;
  • missing root FS object -> empty library directory published as EXPORTED_UNVERIFIED (CE logs only Dir <id> is missing., which the 3-string allowlist did not match);
  • uncreatable destination file -> Open file ... failed warning, file skipped, exit 0.

The pushed fix:

  1. fails closed on the native export log: any [WARNING]/[ERROR] record or known failure signature (Failed to export file, Failed to mkdir, is missing.) refuses with exit 2 and publishes nothing (up to 8 matched lines surfaced in stderr); drops the dead Failed to open block pattern;
  2. mirrors seaf-fsck.sh by prepending <fsck>/../../lib{,64} to the child's LD_LIBRARY_PATH (inherited value kept; no-op if the layout is absent) — this makes the documented direct-binary example work on 14.0.8-v0.2-rc-app, where a bare call fails with libevent-2.1.so.7: cannot open shared object file;
  3. corrects the README publish conditions and states that log-based detection still cannot prove completeness (verify-export + native fsck remain the integrity path).

Verification: 15/15 tests (6 new; 4 use log lines captured verbatim from the real binary), py_compile OK; real-binary matrix re-run in cloudfile/cloudfile:14.0.0-cf.0-local and 14.0.8-v0.2-rc-app: valid export succeeds, missing block / missing subdir / missing root all refuse with nothing published. Trade-off: the strict warning rule can refuse a complete export that logs a benign warning (e.g. destination cannot store mtimes) — fail-safe direction, narrow later only with evidence.

Verdict after this fix: MERGE AFTER ① one end-to-end export on a server-created CE14 library in the target deployed image, then verify-export --hash; ② one deliberate corruption (delete a block, then a non-root dir FS object) on that real library to confirm refusal + no published dir; ③ confirm the deployed image/arch matrix (14.0.0-cf vs 14.0.8-era). Real-library provenance (synthetic vs server-created) is the only remaining evidence gap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants