[WIP] Investigate intermittent failures of jobs#5985
Closed
janniklasrose wants to merge 4 commits into
Closed
Conversation
executeNotebookTask and executeSparkPythonTask looked up files with `len(s.files[path].Data) == 0` to mean "not found". A file that exists but is empty has the same zero-length Data as a missing key, so an empty deployed file was reported as "not found" and its task failed spuriously. Check the map's comma-ok for existence instead of byte length. A present empty file now executes (with empty output) rather than erroring. Updates the state-wiped acceptance golden, whose foo.py fixture is empty and now runs. Co-authored-by: Isaac
…cess On the RUNNING->TERMINATED transition, JobsGetRun hardcoded every run and task to SUCCESS, discarding the per-task result that JobsRunNow already computed. A task that genuinely failed (e.g. its file was missing) still reported a successful run. Roll the per-task states up into the run-level result, and carry the failed task's error into state_message as the cloud does. The immutable acceptance test deploys to a content-addressed snapshot path the local testserver cannot resolve, so its (intentionally unasserted) run now fails locally; wrap that run in errcode to tolerate the exit. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: 942a524
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 7 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Why
Tests