Skip to content

Refactor metadata_manager to move to versions provider based model#244

Open
shiv-tyagi wants to merge 8 commits into
ArduPilot:mainfrom
shiv-tyagi:feat/manifest-provider
Open

Refactor metadata_manager to move to versions provider based model#244
shiv-tyagi wants to merge 8 commits into
ArduPilot:mainfrom
shiv-tyagi:feat/manifest-provider

Conversation

@shiv-tyagi

Copy link
Copy Markdown
Member

This PR refactors the versions manager to move to a versions provider based model.

We introduce three providers:-

  1. ManifestJson provider which parses manifest.json from the firmware server and list officially released versions on custom build server based on that. We are moving away from a release git tags based method, to make sure we are in sync with official firmware server and also make sure we have features.txt for all versions we have.
  2. A whitelisted-tags provider which pulls tags in repo of whitelisted forks and list them for building on the custom build server.
  3. remotes.json based provider to allow manually listing versions on custom build server by adding entries into the file.

I have tested this locally and it works good.

I will update #242 to use the same manifest json client we introduced in this PR to help listing the standard builds for a vehicle-version-release combination.

@shiv-tyagi shiv-tyagi changed the title Refactor metadata_manager to move to versions provider based model Refactor metadata_manager to move to versions provider based model Jul 11, 2026
@shiv-tyagi shiv-tyagi requested a review from peterbarker July 11, 2026 04:30
@shiv-tyagi shiv-tyagi force-pushed the feat/manifest-provider branch from 7ab3df1 to 23eb457 Compare July 11, 2026 04:59

@peterbarker peterbarker left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Introduces a race condition where the SHA a user is dealing with can disappear while they're playing in the interface. Should be explicitly commented on somewhere. Even better if we can find a neat solution :-)

LGTM

@@ -0,0 +1,9 @@
from .manifest import ManifestJson

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably be

Suggested change
from .manifest import ManifestJson
from .manifest import ManifestJSON

self._providers = providers
self.repo = ap_repo
self._remotes_json_path = remotes_json_path
self._access_lock = Lock()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never used?

Comment thread builder/requirements.txt
redis==5.2.1
dill==0.3.8
requests==2.31.0
packaging==26.2

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a mismatch with the web requirements.txt

Comment thread web/core/config.py
def ap_firmware_manifest_url(self) -> str:
"""URL for the ArduPilot firmware manifest."""
return os.getenv(
'AP_FIRMWARE_MANIFEST_URL',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
'AP_FIRMWARE_MANIFEST_URL',
'CBS_AP_FIRMWARE_MANIFEST_URL',

maybe?


def _write_cache(self, raw: bytes, meta: _CacheMeta) -> None:
self.cache_dir.mkdir(parents=True, exist_ok=True)
self.cache_path.write_bytes(raw)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider writing into a separate file and os.replace instead - i.e. try to make both these file-writes closer to atomic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants