Establish architectural design process for orchestration rework#74
Open
Establish architectural design process for orchestration rework#74
Conversation
Bumps [astro](https://github.com/withastro/astro/tree/HEAD/packages/astro) from 5.18.1 to 6.0.6. - [Release notes](https://github.com/withastro/astro/releases) - [Changelog](https://github.com/withastro/astro/blob/main/packages/astro/CHANGELOG.md) - [Commits](https://github.com/withastro/astro/commits/astro@6.0.6/packages/astro) --- updated-dependencies: - dependency-name: astro dependency-version: 6.0.6 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Collaborator
|
@lythx fix the conflicts |
grzanka
reviewed
Apr 12, 2026
Comment on lines
+16
to
+17
| 3. Optimized result transport using S3-compatible object storage and efficient binary formats instead of large JSON payloads through the message broker. | ||
| 4. Optimized result merge by replacing pure-Python averaging with high-performance numerical libraries. |
Collaborator
There was a problem hiding this comment.
S3 storage and binary formats are the technical details which would come up after many more interations and discussions of all pros and cons. In this document I will focus more on the goals and user stories.
we also need a lot of estimations for many items:
- typical times needed to connect to HPC cluster (like Ares)
- typical time the job sits in the waiting queue
- number of parallel jobs per simulation users would like to HPC
- number of simulations single user would like to run in parallel
- time needed to dump single binary results to disk
- how many files per simulation
- how many pages per file
I suspect mine input would be needed in many cases here.
You can ask LLM to formulate many (i.e. >20) open questions, you can commit them here. I could help you and provide my input for that.
Take a look at the bottom of https://github.com/APTG/dedx_web/blob/master/docs/04-feature-specs/entity-selection.md - these are leftovers of similar process I had
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a docs/rework-orchestration directory which will be used for documenting research, decisions and progress related to the orchestration rework as described in this issue. I used GPT 5.3 Codex to generate the initial pages, focusing on the current architecture, bottlenecks and profiling plan.