refactor(frontend): extract the multipart upload engine into a shared service - #7142
Draft
tanishqgandhi1908 wants to merge 29 commits into
Draft
refactor(frontend): extract the multipart upload engine into a shared service#7142tanishqgandhi1908 wants to merge 29 commits into
tanishqgandhi1908 wants to merge 29 commits into
Conversation
… tests Review fixes on apache#5911: use dataset owner email in retrieveLatestDatasetVersion, replace brittle Option.get/.head with headOption, strip datasets prefix on selection-modal reopen, clarify FileResolver docs; add FileResolver/DatasetFileNode and frontend path-helper tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The datasets logical-path prefix strips four leading segments (datasets/owner/dataset/version); update the cover-image test's input path to include the prefix so the extracted relative path is the file name, not an empty string. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Now that models are coming as a separate resource/table, the leading resource-type segment is what selects the backing table, so an unprefixed path can no longer be routed unambiguously. Make the "datasets" prefix required instead of a tolerated fallback, and migrate existing data. - FileResolver: a dataset path must start with the "datasets" segment; unprefixed paths are no longer treated as dataset paths. - pytexera DatasetFileDocument: same rule, mirroring the backend. - FileListerSourceOpExec: parse the now-prefixed datasetVersionPath (skip the "datasets" segment); extracted into a testable helper. - Migration (sql/updates/29.sql): prepend "datasets/" to legacy paths stored in workflow.content and workflow_version.content, covering both the fileName (scan sources) and datasetVersionPath (file lister) operator properties. Only values whose first two segments match an existing (user.email, dataset.name) are rewritten, so local paths and URLs are left untouched; email format is irrelevant (owner may be a username without "@"). Uses create_missing=false and is idempotent. - Example workflows: use datasets-prefixed paths. Tests: FileResolverSpec and WorkflowExecutionsResourceSpec updated; test_dataset_file_document.py updated; new FileListerSourceOpExecSpec. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…l-path-prefix # Conflicts: # frontend/src/app/common/type/datasetVersionFileTree.spec.ts
…format python test - dataset-selection-modal.component.spec.ts: expect the datasets-prefixed selectedPath in version (non-file) mode, matching the emitted path. - test_dataset_file_document.py: apply ruff format (wrap an over-length line) so `ruff format --check` passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The cover-image endpoints resolve a dataset path built as
{owner}/{name}/{coverImage}. With the prefix now required by FileResolver,
these must carry the datasets/ segment; add it in the set-cover,
get-cover redirect, and cover-url handlers so cover images resolve again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…storage # Conflicts: # sql/changelog.xml
…rage # Conflicts: # common/workflow-core/src/test/scala/org/apache/texera/amber/core/storage/DocumentFactorySpec.scala
The explicit-access branch of listModels hardcoded size 0 while the public branch computed it, so a user's own models reported no size. Compute it in both, mirroring the dataset listing, and degrade to 0 rather than dropping the row when LakeFS cannot answer.
DatasetFileNode.fromLakeFSRepositoryCommittedObjects hardcoded ResourceType.Datasets as the tree's leading segment. Models reuse that builder, so model file nodes reported paths under /datasets/... . FileResolver dispatches on that segment, so presigning a model file by its logical path resolved against the DATASET table -- a 404, or the wrong resource's file when a user owns a same-named dataset. Make resourceType a required parameter so every call site declares which resource it builds, and extract the scaffolding the two builders duplicated into buildVersionSkeleton and sortChildrenRecursively. fromPhysicalFileNodes stays pinned to Datasets: physical on-disk trees are dataset-only. Datasets are unaffected -- the file-population loop is moved verbatim and the terminal sort makes creation order unobservable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-actions
Bot
added
feature
engine
ddl-change
Contributor
Automated Reviewer SuggestionsBased on the
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
The multipart upload engine was welded into DatasetService. The model upload UI (#6499) needs the same engine, and copying it would have duplicated all of that.
It now lives in MultipartUploadService, parameterized by a small FileResourceEndpoint ({ baseUrl, nameParamKey }), since the dataset and model endpoint families differ only in base path and in the query-param name carrying the resource. DatasetService keeps all four methods with byte-identical signatures as one-line delegates and re-exports MultipartUploadProgress, so no caller changes.
FilesUploaderComponent is generalized rather than forked. It takes resourceName/resourceId (renamed from datasetName/did) plus an endpoint input defaulting to the dataset one, so its single call site changes three bindings.
No behavioural change. Abort/teardown, resume, retry and the progress/ETA arithmetic all already existed and are moved verbatim — this PR adds no upload behaviour, only relocates it. Production code is net-negative: −300 from DatasetService, +404 in the new service and endpoint config.
One latent bug fixed along the way. multipartUpload builds the per-part URL by hand rather than through HttpParams, and that string hardcoded &datasetName=. Parameterizing only the HttpParams call sites would have left the part upload sending datasetName for every resource — correct for datasets, and a 400 "Model not found" the moment a model used it, since /model/multipart-upload/part expects modelName. Both addressing paths now go through endpoint.nameParamKey.
Any related issues, documentation, discussions?
Part of #6494. Preparation for the upload half of #6499 — this is the shared engine that PR consumes.
How was this PR tested?
The correctness claim is that four pre-existing specs pass with no assertion changes. The uploader spec's changes are mock rewiring only — it constructs the component positionally and asserts banner state and emitted items, never call arguments.
The engine previously had no direct test — only indirect coverage through DatasetService's public surface. It now has multipart-upload.service.spec.ts with 13 cases: the endpoint seam via a synthetic { baseUrl: "widget", nameParamKey: "widgetName" } endpoint; init → part → finish addressing; resume from missingParts; non-2xx and transport-level part failures; unsubscribe → abort; auth header present and absent; null-payload tolerance; and the double-encoding convention on filePath (pre-encoded for HttpParams, encoded once for the hand-built part URL).
That synthetic endpoint is what catches the hardcoded-param bug above — verified by mutation: reintroducing it fails exactly one test and nothing else.
cd frontend && npx ng test --watch=false --include src/app/dashboard/service/user/file-resource/multipart-upload.service.spec.ts
203 tests pass across the seven affected spec files. tsc --noEmit, eslint and prettier are clean.
Manually verified against a local stack, uploading into a dataset: single-part and multi-part uploads (a 55 MiB file across two 50 MiB chunks), version creation, a dragged folder preserving nested paths (churn-predictor/tokenizer/vocab.txt), the "Matching File Found" conflict modal for an already-committed file, cancelling an in-flight upload, and staging then finalizing a file deletion.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)