-
Notifications
You must be signed in to change notification settings - Fork 30
mkdocs. Mapping tables apis. Fix #1032 #1043
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
dgarijo
merged 6 commits into
KnowledgeCaptureAndDiscovery:dev
from
juanjemdIos:fix-1032
Jul 3, 2026
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
933a60c
Merge pull request #49 from KnowledgeCaptureAndDiscovery/dev
juanjemdIos b3d7ad3
mapping tables apis
juanjemdIos 773b4a1
add enrichment documentation
juanjemdIos 6391e55
mkdocs. Api docs.
juanjemdIos 9538f0f
Merge branch 'dev' into fix-1032
juanjemdIos 3bc7078
dani comments
juanjemdIos File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,29 +1,46 @@ | ||
| When analyzing a Codeberg repository, SOMEF uses the [Codeberg API](https://codeberg.org/api/v1/swagger) | ||
| (`GET /api/v1/repos/{owner}/{repo}`) to retrieve metadata. The table below shows how Codeberg API | ||
| When analyzing a Codeberg repository, SOMEF uses the [Codeberg API](https://codeberg.org/api/v1/swagger) | ||
| (`GET /api/v1/repos/{owner}/{repo}`) to retrieve metadata. The table below shows how Codeberg API | ||
| fields map to SOMEF categories: | ||
|
|
||
| | SOMEF category | Codeberg API field | Notes | | ||
| |---|---|---| | ||
| | `name` | `name` | | | ||
| | `description` | `description` | | | ||
| | `code_repository` | `html_url` | | | ||
| | `owner` | `owner.login` | | | ||
| | `full_name` | `full_name` | Format: `{owner}/{repo}` | | ||
| | `name` | `name` | | | ||
| | `description` | `description` | | | ||
| | `date_created` | `created_at` | | | ||
| | `date_updated` | `updated_at` | | | ||
| | `stars` | `stars_count` | In GitHub this field is `stargazers_count` | | ||
| | `stargazers_count` | `stars_count` | Called `stars_count` in Codeberg | | ||
| | `forks_count` | `forks_count` | | | ||
| | `homepage` | `website` | In GitHub this field is `homepage` | | ||
| | `homepage` | `website` | Called `website` in Codeberg, `homepage` in GitHub | | ||
| | `download_url` | *(built from URL)* | `https://codeberg.org/{owner}/{repo}/releases` | | ||
| | `keywords` | `topics` | | | ||
| | `issue_tracker` | *(constructed)* | Built as `{html_url}/issues` | | ||
| | `license` | *(content API)* | *1* | | ||
| | `programming_languages` | `languages_url` | Additional GET request to the languages endpoint | | ||
| | `releases` | `/repos/{owner}/{repo}/releases` | Additional GET request | | ||
| | `issue_tracker` | *(built from URL)* | `{html_url}/issues` | | ||
| | `programming_languages` | `languages_url` | Additional GET request to the languages endpoint, returns byte counts per language | | ||
| | `releases` | `/repos/{owner}/{repo}/releases` | Additional GET request, mapped via `release_codeberg_crosswalk_table` | | ||
|
|
||
| For releases, the field mapping is identical to GitHub. The only differences are that Codeberg | ||
| uses `attachments` instead of `assets` for release files, and it does not provide | ||
| For releases, the field mapping is identical to GitHub. The only differences are that Codeberg | ||
| uses `attachments` instead of `assets` for release files, and it does not provide | ||
| `author.type` (`AGENT_TYPE`) for release authors. | ||
|
|
||
| --------------- | ||
|
|
||
| *1* | ||
| Extracted by fetching the LICENSE file via `GET /api/v1/repos/{owner}/{repo}/contents/{filename}` (tries `LICENSE`, `LICENSE.md`, `LICENCE`, `COPYING`). The content is base64-decoded and analyzed with `detect_license_spdx()` to obtain the SPDX identifier, name and URL. Technique: `Codeberg_API`. | ||
| ### Archive download | ||
|
|
||
| Repository archives are downloaded from: | ||
| `https://codeberg.org/{owner}/{repo}/archive/{branch}.zip` | ||
|
|
||
| Codeberg archive URLs typically include a `Content-Length` header, so the size limit check | ||
| can be performed before downloading. | ||
|
|
||
| ### Enrichment via CODEOWNERS | ||
|
|
||
| When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details | ||
| (full name, email) from `GET https://codeberg.org/api/v1/users/{username}` for the | ||
| repository owner and for each CODEOWNERS entry. | ||
|
|
||
| ### Limitations | ||
|
|
||
| - **License detection**: Codeberg does not provide a `license` field in the repository | ||
| API response. SOMEF falls back to fetching LICENSE/COPYING files directly via the | ||
| content API. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| When analyzing a GitHub repository, SOMEF uses the [GitHub REST API](https://docs.github.com/en/rest/repos/repos) | ||
| (`GET /repos/{owner}/{repo}`) to retrieve metadata. The table below shows how GitHub API | ||
| fields map to SOMEF categories: | ||
|
|
||
| | SOMEF category | GitHub API field | Notes | | ||
| |---|---|---| | ||
| | `code_repository` | `html_url` | | | ||
| | `owner` | `owner.login` | `agent_type` is extracted from `owner.type` (User or Organization) | | ||
| | `date_created` | `created_at` | | | ||
| | `date_updated` | `updated_at` | | | ||
| | `license` | `license` | Nested object with `spdx_id`, `name`, `url` | | ||
| | `description` | `description` | | | ||
| | `name` | `name` | | | ||
| | `full_name` | `full_name` | Format: `{owner}/{repo}` | | ||
| | `issue_tracker` | `issues_url` | The `{/number}` suffix is stripped | | ||
| | `forks_url` | `forks_url` | | | ||
| | `stargazers_count` | `stargazers_count` | | | ||
| | `keywords` | `topics` | | | ||
| | `forks_count` | `forks_count` | | | ||
| | `homepage` | `homepage` | | | ||
| | `programming_languages` | `languages` | Additional GET to `/repos/{owner}/{repo}/languages`. Returns a dictionary with byte counts per language | | ||
| | `releases` | `/repos/{owner}/{repo}/releases` | Paginated results, mapped via `release_crosswalk_table` | | ||
| | `download_url` | *(constructed)* | Built as `https://github.com/{owner}/{repo}/releases` | | ||
|
|
||
| ### Archive download | ||
|
|
||
| SOMEF downloads the repository archive from `https://github.com/{owner}/{repo}/archive/{ref}.zip`. | ||
| GitHub archive URLs do not include a `Content-Length` header, so SOMEF uses a streaming check: | ||
| it reads the response in 1 MB chunks and aborts if the total exceeds the configured size limit | ||
| (see `--download-limit`). | ||
|
|
||
| If the ref name is ambiguous (a branch and a tag share the same name), GitHub returns | ||
| HTTP 300. SOMEF handles this by trying the following fallback URLs in order: | ||
|
|
||
| 1. `https://github.com/{owner}/{repo}/archive/{ref}.zip` (short form) | ||
| 2. `https://github.com/{owner}/{repo}/archive/refs/heads/{ref}.zip` (explicit branch) | ||
| 3. `https://github.com/{owner}/{repo}/archive/refs/tags/{ref}.zip` (explicit tag) | ||
| 4. `https://github.com/{owner}/{repo}/archive/main.zip` (legacy rename fallback) | ||
|
|
||
| ### Limitations | ||
|
|
||
| - **Private repositories**: SOMEF cannot access private repositories without a valid token. | ||
|
|
||
|
|
||
| ### Enrichment via CODEOWNERS | ||
|
|
||
| When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details | ||
| (name, company, email) from `GET https://api.github.com/users/{username}` for the | ||
| repository owner and for each CODEOWNERS entry. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| When analyzing a GitLab repository, SOMEF uses the [GitLab REST API](https://docs.gitlab.com/ee/api/projects.html) | ||
| (`GET /api/v4/projects/{project_id}`) to retrieve metadata. The table below shows how GitLab API | ||
| fields map to SOMEF categories: | ||
|
|
||
| | SOMEF category | GitLab API field | Notes | | ||
| |---|---|---| | ||
| | `code_repository` | *(built from URL)* | `https://{host}/{project_path}/` | | ||
| | `download_url` | *(built from URL)* | `https://{host}/{project_path}/-/branches` | | ||
| | `defaultBranch` | `default_branch` | Also checks `defaultBranch` (legacy key) | | ||
| | `description` | `description` | | | ||
| | `name` | `name` | | | ||
| | `full_name` | `path_with_namespace` | | | ||
| | `owner` | `owner.username` | `agent_type` deduced from `namespace.kind` ("group" → Organization, "user" → Person) | | ||
| | `date_created` | `created_at` | | | ||
| | `date_updated` | `last_activity_at` | | | ||
| | `issue_tracker` | *(built from URL)* | `https://{host}/{project_path}/-/issues` | | ||
| | `license` | `license` | `name` and `url` with SPDX detection. Falls back to fetching raw LICENSE file | | ||
| | `keywords` | `topics` | | | ||
| | `stargazers_count` | `star_count` | Called `star_count` in GitLab | | ||
| | `forks_count` | `forks_count` | | | ||
| | `programming_languages` | `languages` | Only language names (keys), no byte counts | | ||
| | `readme_url` | `readme_url` | | | ||
| | `releases` | `/projects/{id}/releases` | Paginated via `X-Next-Page`, mapped via `release_gitlab_crosswalk_table` | | ||
|
|
||
|
|
||
| ### Self-hosted GitLab instances | ||
|
|
||
| SOMEF supports self-hosted GitLab instances (e.g., `gitlab.in2p3.fr`). The project path is | ||
| URL-encoded and used to query the instance's API at `https://{host}/api/v4/projects/{encoded_path}`. | ||
| SOMEF detects self-hosted instances by checking if the URL contains `gitlab.com`. | ||
|
|
||
| ### Archive download | ||
|
|
||
| Repository archives are downloaded from: | ||
| `https://{host}/{project_path}/-/archive/{branch}/{repo_name}-{branch}.zip` | ||
|
|
||
| GitLab archive URLs typically include a `Content-Length` header, so the size limit check | ||
| can be performed before downloading. | ||
|
|
||
| ### Enrichment via CODEOWNERS | ||
|
|
||
| When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details | ||
| (name, organization, public email) from `GET {server_url}/api/v4/users?username={username}` | ||
| for the repository owner and for each CODEOWNERS entry. Self-hosted instances use their | ||
| own API endpoint; `gitlab.com` uses the standard `https://gitlab.com/api/v4/users`. | ||
|
|
||
| ### Limitations | ||
|
|
||
| - **Release assets**: GitLab releases include sources (tar.gz, zip) and links, but do not | ||
| provide `download_count` or `content_size` like GitHub. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # OpenAIRE and Zenodo API Mapping | ||
|
|
||
| When the enrichment flag (`-e` / `--enrichment`) is enabled, SOMEF integrates with the **OpenAIRE** and **Zenodo** APIs to fetch project context, funding background, and ecosystem identifiers. | ||
|
|
||
| ## Funding Enrichment (OpenAIRE) | ||
|
|
||
| These fields are injected into the **Funding** category when a matching grant or project is reconciled via OpenAIRE keywords or grant identifiers. | ||
|
|
||
| | SOMEF Property | Source API Field | Description | | ||
| | :--- | :--- | :--- | | ||
| | `project_code` | `code` | The official project or grant code assigned by the funding body. | | ||
| | `project_title` | `title` | The full title of the funded research project. | | ||
| | `project_acronym` | `acronym` | The official acronym of the project. | | ||
| | `grant_id` | `callidentifier` | Unique identifier for the specific call or grant agreement. | | ||
|
|
||
| ## Identifier Enrichment (OpenAIRE & Zenodo) | ||
|
|
||
| When processing repository identifiers or DOIs found in citations, SOMEF resolves them against OpenAIRE to construct an explore landing page link. Additionally, if the DOI belongs to Zenodo, SOMEF queries the Zenodo API to extract its corresponding Software Heritage identifier (`swhid`). | ||
|
|
||
| | SOMEF Property | Source API / Function | Description | | ||
| | :--- | :--- | :--- | | ||
| | `openaire_id` | `get_openaire_id(doi)` | Direct link to the OpenAIRE explore page dedicated to this artifact. | | ||
| | `swhid` | `get_zenodo_swhid(doi)` | The Software Heritage Identifier (SWHID) associated with a Zenodo record. | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # OpenAlex API Mapping | ||
|
|
||
| When the enrichment flag (`-e` / `--enrichment`) is enabled, SOMEF integrates with the **OpenAlex API** to resolve and complete scholarly publication data and missing author identities. | ||
|
|
||
| ## Citation Enrichment | ||
| If OpenAIRE fails to resolve a publication's DOI, SOMEF queries OpenAlex as a fallback to fetch the publication's unique ID. | ||
|
|
||
| | SOMEF Property | Source API Function | Description | | ||
| | :--- | :--- | :--- | | ||
| | `openalex_id` | `get_openalex_id(doi)` | The unique OpenAlex URL identifier assigned to the publication. | | ||
|
|
||
| ## Author/Contributor Enrichment | ||
| For extracted authors or contributors who lack a structured identifier, SOMEF searches OpenAlex by name to retrieve their professional ORCID. | ||
|
|
||
| | SOMEF Property | Source API Function | Description | | ||
| | :--- | :--- | :--- | | ||
| | `identifier` | `search_openalex_author(name)` | The author's verified ORCID URL (used as fallback when missing in local files). | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Per above, this should be removed