From b3d7ad37b23ff3084da28dcd62767f603522af4d Mon Sep 17 00:00:00 2001 From: Juanje Mendoza Date: Thu, 2 Jul 2026 10:42:01 +0200 Subject: [PATCH 1/4] mapping tables apis --- docs/bitbucket.md | 40 ++++++++++++++++++++--- docs/codeberg.md | 50 ++++++++++++++++++++++------- docs/github.md | 54 +++++++++++++++++++++++++++++++ docs/gitlab.md | 57 +++++++++++++++++++++++++++++++++ mkdocs.yml | 9 ++++-- src/somef/process_repository.py | 51 +++++++++++++++++++++++++++++ 6 files changed, 243 insertions(+), 18 deletions(-) create mode 100644 docs/github.md create mode 100644 docs/gitlab.md diff --git a/docs/bitbucket.md b/docs/bitbucket.md index 7c8eeba2..93709631 100644 --- a/docs/bitbucket.md +++ b/docs/bitbucket.md @@ -13,9 +13,41 @@ fields map to SOMEF categories: | `date_updated` | `updated_on` | | | `homepage` | `website` | | | `forks_url` | `links.forks.href` | | -| `download_url` | *(constructed)* | Built as `{html_url}/downloads` | -| `issue_tracker` | *(constructed)* | Built as `{html_url}/issues` when `has_issues` is true | +| `download_url` | *(built from URL)*| Built as `{html_url}/downloads` | +| `issue_tracker` | *(built from URL)* | Built as `{html_url}/issues` when `has_issues` is true | | `programming_languages` | `language` | Single string, not a dictionary with sizes | | `releases` | `/refs/tags` | Bitbucket has no dedicated releases endpoint; uses the tags endpoint | -| `stars` | *(not available)* | Bitbucket does not have a stargazers feature | -| `forks_count` | *(not available)* | Bitbucket does not expose fork counts in its API | \ No newline at end of file +| `stargazers_count` | *(not available)* | Bitbucket does not have a stargazers feature | +| `forks_count` | *(not available)* | Bitbucket does not expose fork counts in its API | + + +### Authentication + +Bitbucket uses HTTP Basic authentication. The token is encoded as +`base64(email:token)` and sent as the `Authorization` header. +Provide both the Bitbucket app password and your Atlassian account email +via `--bitbucket-token` and `--bitbucket-email`, or by running `somef configure`. + +### Archive download + +Repository archives are downloaded from: +`https://bitbucket.org/{owner}/{repo}/get/{branch}.zip` + +Bitbucket archive URLs typically include a `Content-Length` header, so the size limit check +can be performed before downloading. + +### Limitations + +- **No stargazers**: Bitbucket does not have a stargazers feature. +- **No forks count**: Bitbucket does not expose fork counts in its API. +- **Tags-only releases**: Bitbucket has no dedicated releases endpoint. + SOMEF uses the `/refs/tags` endpoint instead, which does not include release + descriptions or assets. +- **Programming languages**: Bitbucket returns only a single language string, + without byte counts per language. +- **CODEOWNERS enrichment**: Not supported for Bitbucket, as the platform does not + expose a public user API. +- **Rate limits**: Unauthenticated requests are limited to 60 requests/hour. + Authenticated requests (via `bitbucket-token` and `bitbucket-email` ) have higher limits) have higher limits. + Create an app password at `https://bitbucket.org/account/settings/api-tokens/` + with `read:repository:bitbucket` and `read:account` scopes. \ No newline at end of file diff --git a/docs/codeberg.md b/docs/codeberg.md index 302826f3..652d1677 100644 --- a/docs/codeberg.md +++ b/docs/codeberg.md @@ -1,29 +1,55 @@ -When analyzing a Codeberg repository, SOMEF uses the [Codeberg API](https://codeberg.org/api/v1/swagger) -(`GET /api/v1/repos/{owner}/{repo}`) to retrieve metadata. The table below shows how Codeberg API +When analyzing a Codeberg repository, SOMEF uses the [Codeberg API](https://codeberg.org/api/v1/swagger) +(`GET /api/v1/repos/{owner}/{repo}`) to retrieve metadata. The table below shows how Codeberg API fields map to SOMEF categories: | SOMEF category | Codeberg API field | Notes | |---|---|---| -| `name` | `name` | | -| `description` | `description` | | | `code_repository` | `html_url` | | | `owner` | `owner.login` | | +| `full_name` | `full_name` | Format: `{owner}/{repo}` | +| `name` | `name` | | +| `description` | `description` | | | `date_created` | `created_at` | | | `date_updated` | `updated_at` | | -| `stars` | `stars_count` | In GitHub this field is `stargazers_count` | +| `stargazers_count` | `stars_count` | Called `stars_count` in Codeberg | | `forks_count` | `forks_count` | | -| `homepage` | `website` | In GitHub this field is `homepage` | +| `homepage` | `website` | Called `website` in Codeberg, `homepage` in GitHub | +| `download_url` | *(built from URL)* | `https://codeberg.org/{owner}/{repo}/releases` | | `keywords` | `topics` | | -| `issue_tracker` | *(constructed)* | Built as `{html_url}/issues` | +| `issue_tracker` | *(built from URL)* | `{html_url}/issues` | | `license` | *(content API)* | *1* | -| `programming_languages` | `languages_url` | Additional GET request to the languages endpoint | -| `releases` | `/repos/{owner}/{repo}/releases` | Additional GET request | +| `programming_languages` | `languages_url` | Additional GET request to the languages endpoint, returns byte counts per language | +| `releases` | `/repos/{owner}/{repo}/releases` | Additional GET request, mapped via `release_codeberg_crosswalk_table` | -For releases, the field mapping is identical to GitHub. The only differences are that Codeberg -uses `attachments` instead of `assets` for release files, and it does not provide +For releases, the field mapping is identical to GitHub. The only differences are that Codeberg +uses `attachments` instead of `assets` for release files, and it does not provide `author.type` (`AGENT_TYPE`) for release authors. + --------------- *1* -Extracted by fetching the LICENSE file via `GET /api/v1/repos/{owner}/{repo}/contents/{filename}` (tries `LICENSE`, `LICENSE.md`, `LICENCE`, `COPYING`). The content is base64-decoded and analyzed with `detect_license_spdx()` to obtain the SPDX identifier, name and URL. Technique: `Codeberg_API`. \ No newline at end of file +Extracted by fetching the LICENSE file via `GET /api/v1/repos/{owner}/{repo}/contents/{filename}` (tries `LICENSE`, `LICENSE.md`, `LICENCE`, `COPYING`). The content is base64-decoded and analyzed with `detect_license_spdx()` to obtain the SPDX identifier, name and URL. + +### Archive download + +Repository archives are downloaded from: +`https://codeberg.org/{owner}/{repo}/archive/{branch}.zip` + +Codeberg archive URLs typically include a `Content-Length` header, so the size limit check +can be performed before downloading. + +### Enrichment via CODEOWNERS + +When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details +(full name, email) from `GET https://codeberg.org/api/v1/users/{username}` for the +repository owner and for each CODEOWNERS entry. + +### Limitations + +- **Rate limits**: Unauthenticated requests are limited to 60 requests/hour. Authenticated + requests (via `codeberg-token`) have higher limits. Create a token at + `https://codeberg.org/user/settings/applications`. +- **License detection**: Codeberg does not provide a `license` field in the repository + API response. SOMEF falls back to fetching LICENSE/COPYING files directly via the + content API. \ No newline at end of file diff --git a/docs/github.md b/docs/github.md new file mode 100644 index 00000000..c8f05e83 --- /dev/null +++ b/docs/github.md @@ -0,0 +1,54 @@ +When analyzing a GitHub repository, SOMEF uses the [GitHub REST API](https://docs.github.com/en/rest/repos/repos) +(`GET /repos/{owner}/{repo}`) to retrieve metadata. The table below shows how GitHub API +fields map to SOMEF categories: + +| SOMEF category | GitHub API field | Notes | +|---|---|---| +| `code_repository` | `html_url` | | +| `owner` | `owner.login` | `agent_type` is extracted from `owner.type` (User or Organization) | +| `date_created` | `created_at` | | +| `date_updated` | `updated_at` | | +| `license` | `license` | Nested object with `spdx_id`, `name`, `url` | +| `description` | `description` | | +| `name` | `name` | | +| `full_name` | `full_name` | Format: `{owner}/{repo}` | +| `issue_tracker` | `issues_url` | The `{/number}` suffix is stripped | +| `forks_url` | `forks_url` | | +| `stargazers_count` | `stargazers_count` | | +| `keywords` | `topics` | | +| `forks_count` | `forks_count` | | +| `homepage` | `homepage` | | +| `programming_languages` | `languages` | Additional GET to `/repos/{owner}/{repo}/languages`. Returns a dictionary with byte counts per language | +| `releases` | `/repos/{owner}/{repo}/releases` | Paginated results, mapped via `release_crosswalk_table` | +| `download_url` | *(constructed)* | Built as `https://github.com/{owner}/{repo}/releases` | + +### Archive download + +SOMEF downloads the repository archive from `https://github.com/{owner}/{repo}/archive/{ref}.zip`. +GitHub archive URLs do not include a `Content-Length` header, so SOMEF uses a streaming check: +it reads the response in 1 MB chunks and aborts if the total exceeds the configured size limit +(see `--download-limit`). + +If the ref name is ambiguous (a branch and a tag share the same name), GitHub returns +HTTP 300. SOMEF handles this by trying the following fallback URLs in order: + +1. `https://github.com/{owner}/{repo}/archive/{ref}.zip` (short form) +2. `https://github.com/{owner}/{repo}/archive/refs/heads/{ref}.zip` (explicit branch) +3. `https://github.com/{owner}/{repo}/archive/refs/tags/{ref}.zip` (explicit tag) +4. `https://github.com/{owner}/{repo}/archive/main.zip` (legacy rename fallback) + +### Limitations + +- **No Content-Length**: GitHub archive downloads lack a `Content-Length` header, so the + size limit check relies on streaming (reading 1 MB chunks until the limit is exceeded). +- **Rate limits**: Unauthenticated requests are limited to 60 requests/hour. Authenticated + requests (via `github-token`) are limited to 5,000 requests/hour. +- **Private repositories**: SOMEF cannot access private repositories without a valid token. +- **Asset download count**: GitHub provides `download_count` for release assets; other + providers may not offer this field. + +### Enrichment via CODEOWNERS + +When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details +(name, company, email) from `GET https://api.github.com/users/{username}` for the +repository owner and for each CODEOWNERS entry. \ No newline at end of file diff --git a/docs/gitlab.md b/docs/gitlab.md new file mode 100644 index 00000000..81502db0 --- /dev/null +++ b/docs/gitlab.md @@ -0,0 +1,57 @@ +When analyzing a GitLab repository, SOMEF uses the [GitLab REST API](https://docs.gitlab.com/ee/api/projects.html) +(`GET /api/v4/projects/{project_id}`) to retrieve metadata. The table below shows how GitLab API +fields map to SOMEF categories: + +| SOMEF category | GitLab API field | Notes | +|---|---|---| +| `code_repository` | *(built from URL)* | `https://{host}/{project_path}/` | +| `download_url` | *(built from URL)* | `https://{host}/{project_path}/-/branches` | +| `defaultBranch` | `default_branch` | Also checks `defaultBranch` (legacy key) | +| `description` | `description` | | +| `name` | `name` | | +| `full_name` | `path_with_namespace` | | +| `owner` | `owner.username` | `agent_type` deduced from `namespace.kind` ("group" → Organization, "user" → Person) | +| `date_created` | `created_at` | | +| `date_updated` | `last_activity_at` | | +| `issue_tracker` | *(built from URL)* | `https://{host}/{project_path}/-/issues` | +| `license` | `license` | `name` and `url` with SPDX detection. Falls back to fetching raw LICENSE file | +| `keywords` | `topics` | | +| `stargazers_count` | `star_count` | Called `star_count` in GitLab | +| `forks_count` | `forks_count` | | +| `programming_languages` | `languages` | Only language names (keys), no byte counts | +| `readme_url` | `readme_url` | | +| `releases` | `/projects/{id}/releases` | Paginated via `X-Next-Page`, mapped via `release_gitlab_crosswalk_table` | + + +### Self-hosted GitLab instances + +SOMEF supports self-hosted GitLab instances (e.g., `gitlab.in2p3.fr`). The project path is +URL-encoded and used to query the instance's API at `https://{host}/api/v4/projects/{encoded_path}`. +SOMEF detects self-hosted instances by checking if the URL contains `gitlab.com`. + +### Archive download + +Repository archives are downloaded from: +`https://{host}/{project_path}/-/archive/{branch}/{repo_name}-{branch}.zip` + +GitLab archive URLs typically include a `Content-Length` header, so the size limit check +can be performed before downloading. + +### Enrichment via CODEOWNERS + +When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details +(name, organization, public email) from `GET {server_url}/api/v4/users?username={username}` +for the repository owner and for each CODEOWNERS entry. Self-hosted instances use their +own API endpoint; `gitlab.com` uses the standard `https://gitlab.com/api/v4/users`. + +### Limitations + +- **Self-hosted detection**: SOMEF detects self-hosted instances by checking for `gitlab.com` + in the URL. If a self-hosted instance uses a custom domain without a GitLab API endpoint, + detection may fail. +- **Rate limits**: GitLab.com unauthenticated requests are limited to 600 requests/hour. + Authenticated requests (via `gitlab-token`)have higher limits depending on the token type. Self-hosted + instances have their own rate limits. +- **Programming languages**: GitLab returns only language names (no byte counts). +- **Release assets**: GitLab releases include sources (tar.gz, zip) and links, but do not + provide `download_count` or `content_size` like GitHub. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 4756d54c..e5baa108 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -8,8 +8,13 @@ nav: - Output: output.md - Supported metadata files: supported_metadata_files.md - Supported languages: supported_languages.md - -theme: + - Supported APIs: + - GitHub: github.md + - GitLab: gitlab.md + - Codeberg: codeberg.md + - Bitbucket: bitbucket.md + + theme: name: material features: - navigation.expand diff --git a/src/somef/process_repository.py b/src/somef/process_repository.py index d77347f6..1fd161fa 100644 --- a/src/somef/process_repository.py +++ b/src/somef/process_repository.py @@ -398,6 +398,57 @@ def load_gitlab_repository_metadata(repo_metadata: Result, repository_url, autho } repo_metadata.add_result(constants.CAT_KEYWORDS, result, 1, constants.TECHNIQUE_GITLAB_API) + # extract name + if project_details.get('name'): + repo_metadata.add_result(constants.CAT_NAME, { + constants.PROP_VALUE: project_details['name'], + constants.PROP_TYPE: constants.STRING + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract full_name + if project_details.get('path_with_namespace'): + repo_metadata.add_result(constants.CAT_FULL_NAME, { + constants.PROP_VALUE: project_details['path_with_namespace'], + constants.PROP_TYPE: constants.STRING + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract description + if project_details.get('description'): + repo_metadata.add_result(constants.CAT_DESCRIPTION, { + constants.PROP_VALUE: project_details['description'], + constants.PROP_TYPE: constants.STRING + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract owner + if project_details.get('owner') and project_details['owner'].get('username'): + owner_type = "Person" + if project_details.get('namespace') and project_details['namespace'].get('kind'): + owner_type = "Organization" if project_details['namespace']['kind'] == 'group' else "Person" + repo_metadata.add_result(constants.CAT_OWNER, { + constants.PROP_VALUE: project_details['owner']['username'], + constants.PROP_TYPE: owner_type + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract date_created + if project_details.get('created_at'): + repo_metadata.add_result(constants.CAT_DATE_CREATED, { + constants.PROP_VALUE: project_details['created_at'], + constants.PROP_TYPE: constants.DATE + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract date_updated + if project_details.get('last_activity_at'): + repo_metadata.add_result(constants.CAT_DATE_UPDATED, { + constants.PROP_VALUE: project_details['last_activity_at'], + constants.PROP_TYPE: constants.DATE + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract issue_tracker + repo_metadata.add_result(constants.CAT_ISSUE_TRACKER, { + constants.PROP_VALUE: f"https://{url.netloc}/{project_path}/-/issues", + constants.PROP_TYPE: constants.URL + }, 1, constants.TECHNIQUE_GITLAB_API) + # get social features: stargazers_count if project_details['star_count'] is not None: result = { From 773b4a15cacf6e0fe880e067706b34fa51e6092c Mon Sep 17 00:00:00 2001 From: Juanje Mendoza Date: Thu, 2 Jul 2026 11:06:38 +0200 Subject: [PATCH 2/4] add enrichment documentation --- README.md | 29 +++++++++++++++++++++++++++-- docs/usage.md | 2 ++ 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8e7f625c..9eb9e693 100644 --- a/README.md +++ b/README.md @@ -55,8 +55,22 @@ We recognize the following properties: - **Forks url**: Links to forks made of the project - **Full name**: Name + owner (owner/name) - **Full title**: If the repository is a short name, we will attempt to extract the longer version of the repository name -- **Funding**: Funding information associated with the project. **Note**: Currently, this information is only extracted from existing `codemeta.json` files within the repository. -- **Identifier**: Identifier associated with the software (if any), such as Digital Object Identifiers and Software Heritage identifiers (SWH). DOIs associated with publications will also be detected. +- **Funding**: Funding information associated with the project. **Note**: This information is extracted from existing `codemeta.json` files within the repository. When using `-e`, the project data is enriched with OpenAIRE, adding: + +- `project_code`: Project code + +- `project_title`: Project title + +- `project_acronym`: Project acronym + +- `grant_id`: Call/grant identifier +- **Identifier**: Identifier associated with the software (if any), such as Digital Object Identifiers and Software Heritage identifiers (SWH). DOIs associated with publications will also be detected. When using `-e`, the following enrichment identifiers are also added: + +- `openalex_id`: OpenAlex ID of the publication + +- `openaire_id`: URL to the OpenAIRE explore page for the software + +- `swhid`: Software Heritage identifier (for Zenodo DOIs) - **Images**: Images used to illustrate the software component - **Installation instructions**: A set of instructions that indicate how to install a target repository - **Invocation**: Execution command(s) needed to run a scientific software component @@ -372,6 +386,8 @@ Options: -h, --help Show this message and exit. + -e, --enrichment Enrich metadata with external APIs (OpenAlex, OpenAIRE, Zenodo) + --github-token TEXT GitHub personal access token (if invalid, stored config is used instead) @@ -397,6 +413,15 @@ Alternatively, you can set tokens via environment variables or by running `somef The CLI flags take precedence over stored config when valid. +### Enrichment with `-e` + +The `-e` (or `--enrichment`) flag queries external APIs to complete the extracted metadata: +- **OpenAlex**: adds `openalex_id` to DOIs of publications. +- **OpenAIRE**: adds `openaire_id` and enriches funding information (project code, title, acronym, grant id). +- **Zenodo**: adds `swhid` (Software Heritage ID) for Zenodo DOIs. + +**Note:** Enrichment makes additional network requests to external services, which may slow down the overall execution time. Use this flag only when you need the extra metadata. + ## Usage example: The following command extracts all metadata available from [https://github.com/dgarijo/Widoco/](https://github.com/dgarijo/Widoco/). diff --git a/docs/usage.md b/docs/usage.md index fa68fbbc..fcb5f475 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -82,6 +82,8 @@ Options: -h, --help Show this message and exit. + -e, --enrichment Enrich metadata with external APIs (OpenAlex, OpenAIRE, Zenodo) + --github-token TEXT GitHub personal access token (if invalid, stored config is used instead) From 6391e5532e0f1f737b96e8ad509b82fa85fd28bd Mon Sep 17 00:00:00 2001 From: Juanje Mendoza Date: Thu, 2 Jul 2026 15:32:06 +0200 Subject: [PATCH 3/4] mkdocs. Api docs. --- README.md | 18 +++++++++++------- docs/openaire.md | 23 +++++++++++++++++++++++ docs/openalex.md | 17 +++++++++++++++++ docs/usage.md | 15 ++++++++++++++- mkdocs.yml | 6 +++++- 5 files changed, 70 insertions(+), 9 deletions(-) create mode 100644 docs/openaire.md create mode 100644 docs/openalex.md diff --git a/README.md b/README.md index 9eb9e693..bd7a5c6a 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ Given a readme file (or a GitHub/Gitlab/Codeberg/Bitbucket repository) SOMEF wil - **Citation**: Preferred citation(s) as the authors have stated in their readme file. SOMEF recognizes Bibtex, Citation File Format files and other means by which authors cite their papers (e.g., by in-text citation). For CITATION.cff files, SOMEF now generates two separate entries: one for the software tool and another for the preferred citation (if available). This ensures metadata like DOI or version is correctly assigned to each entity. SOMEF now performs citation reconciliation: scholarly publications (articles) are assigned in codemeta to `referencePublication`, while the software itself is credited in `creditText`. (See https://somef.readthedocs.io/en/latest/output/#codemeta-format). -We recognize the following properties: +When using `-e`, publication metadata is enriched via OpenAlex. We recognize the following properties: - Title: Title of the publication - Author: list of author names in the publication - URL: URL of the publication @@ -37,6 +37,7 @@ We recognize the following properties: - Journal: Journal name where the paper was published - Year: Year of publication - Pages: Page range in the journal + - `openalex_id`: OpenAlex ID of the publication - **Code of conduct**: Link to the code of conduct of the project - **Code repository**: Link to the GitHub/GitLab/Codeberg and Bitbucket repository used for the extraction - **Contact**: Contact person responsible for maintaining a software component @@ -66,8 +67,6 @@ We recognize the following properties: - `grant_id`: Call/grant identifier - **Identifier**: Identifier associated with the software (if any), such as Digital Object Identifiers and Software Heritage identifiers (SWH). DOIs associated with publications will also be detected. When using `-e`, the following enrichment identifiers are also added: -- `openalex_id`: OpenAlex ID of the publication - - `openaire_id`: URL to the OpenAIRE explore page for the software - `swhid`: Software Heritage identifier (for Zenodo DOIs) @@ -386,7 +385,8 @@ Options: -h, --help Show this message and exit. - -e, --enrichment Enrich metadata with external APIs (OpenAlex, OpenAIRE, Zenodo) + -e, --enrichment Enrich metadata with external APIs (OpenAlex, + OpenAIRE, Zenodo) --github-token TEXT GitHub personal access token (if invalid, stored config is used instead) @@ -416,9 +416,13 @@ The CLI flags take precedence over stored config when valid. ### Enrichment with `-e` The `-e` (or `--enrichment`) flag queries external APIs to complete the extracted metadata: -- **OpenAlex**: adds `openalex_id` to DOIs of publications. -- **OpenAIRE**: adds `openaire_id` and enriches funding information (project code, title, acronym, grant id). -- **Zenodo**: adds `swhid` (Software Heritage ID) for Zenodo DOIs. +- **OpenAlex**: Adds `openalex_id` to DOIs of publications and reconciles missing author ORCIDs. +- **OpenAIRE**: Adds `openaire_id` to publications/identifiers and enriches project funding metadata. +- **Zenodo**: Adds `swhid` (Software Heritage ID) for records matching Zenodo DOIs. + +For a detailed technical breakdown of the fields mapped by each external service, please refer to the specific documentation pages: +- See the [OpenAlex Mapping Guide](openalex.md) for citation and author properties. +- See the [OpenAIRE and Zenodo Mapping Guide](openaire.md) for funding and identifier properties. **Note:** Enrichment makes additional network requests to external services, which may slow down the overall execution time. Use this flag only when you need the extra metadata. diff --git a/docs/openaire.md b/docs/openaire.md new file mode 100644 index 00000000..953685e5 --- /dev/null +++ b/docs/openaire.md @@ -0,0 +1,23 @@ +# OpenAIRE and Zenodo API Mapping + +When the enrichment flag (`-e` / `--enrichment`) is enabled, SOMEF integrates with the **OpenAIRE** and **Zenodo** APIs to fetch project context, funding background, and ecosystem identifiers. + +## Funding Enrichment (OpenAIRE) + +These fields are injected into the **Funding** category when a matching grant or project is reconciled via OpenAIRE keywords or grant identifiers. + +| SOMEF Property | Source API Field | Description | +| :--- | :--- | :--- | +| `project_code` | `code` | The official project or grant code assigned by the funding body. | +| `project_title` | `title` | The full title of the funded research project. | +| `project_acronym` | `acronym` | The official acronym of the project. | +| `grant_id` | `callidentifier` | Unique identifier for the specific call or grant agreement. | + +## Identifier Enrichment (OpenAIRE & Zenodo) + +When processing repository identifiers or DOIs found in citations, SOMEF resolves them against OpenAIRE to construct an explore landing page link. Additionally, if the DOI belongs to Zenodo, SOMEF queries the Zenodo API to extract its corresponding Software Heritage identifier (`swhid`). + +| SOMEF Property | Source API / Function | Description | +| :--- | :--- | :--- | +| `openaire_id` | `get_openaire_id(doi)` | Direct link to the OpenAIRE explore page dedicated to this artifact. | +| `swhid` | `get_zenodo_swhid(doi)` | The Software Heritage Identifier (SWHID) associated with a Zenodo record. | \ No newline at end of file diff --git a/docs/openalex.md b/docs/openalex.md new file mode 100644 index 00000000..ed96bce0 --- /dev/null +++ b/docs/openalex.md @@ -0,0 +1,17 @@ +# OpenAlex API Mapping + +When the enrichment flag (`-e` / `--enrichment`) is enabled, SOMEF integrates with the **OpenAlex API** to resolve and complete scholarly publication data and missing author identities. + +## Citation Enrichment +If OpenAIRE fails to resolve a publication's DOI, SOMEF queries OpenAlex as a fallback to fetch the publication's unique ID. + +| SOMEF Property | Source API Function | Description | +| :--- | :--- | :--- | +| `openalex_id` | `get_openalex_id(doi)` | The unique OpenAlex URL identifier assigned to the publication. | + +## Author/Contributor Enrichment +For extracted authors or contributors who lack a structured identifier, SOMEF searches OpenAlex by name to retrieve their professional ORCID. + +| SOMEF Property | Source API Function | Description | +| :--- | :--- | :--- | +| `identifier` | `search_openalex_author(name)` | The author's verified ORCID URL (used as fallback when missing in local files). | \ No newline at end of file diff --git a/docs/usage.md b/docs/usage.md index fcb5f475..f89c4a69 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -82,7 +82,8 @@ Options: -h, --help Show this message and exit. - -e, --enrichment Enrich metadata with external APIs (OpenAlex, OpenAIRE, Zenodo) + -e, --enrichment Enrich metadata with external APIs (OpenAlex, + OpenAIRE, Zenodo) --github-token TEXT GitHub personal access token (if invalid, stored config is used instead) @@ -111,6 +112,18 @@ GitHub, GitLab and Codeberg; Bitbucket does not expose a public user API, so enr Alternatively, you can set tokens via environment variables or by running `somef configure`, which stores them permanently. The CLI flags take precedence over stored config when valid. +### Enrichment with `-e` + +The `-e` (or `--enrichment`) flag queries external APIs to complete the extracted metadata: +- **OpenAlex**: Adds `openalex_id` to DOIs of publications and reconciles missing author ORCIDs. +- **OpenAIRE**: Adds `openaire_id` to publications/identifiers and enriches project funding metadata. +- **Zenodo**: Adds `swhid` (Software Heritage ID) for records matching Zenodo DOIs. + +For a detailed technical breakdown of the fields mapped by each external service, please refer to the specific documentation pages: +- See the [OpenAlex Mapping Guide](openalex.md) for citation and author properties. +- See the [OpenAIRE and Zenodo Mapping Guide](openaire.md) for funding and identifier properties. + +**Note:** Enrichment makes additional network requests to external services, which may slow down the overall execution time. Use this flag only when you need the extra metadata. ## Usage example: The following command extracts all metadata available from [https://github.com/dgarijo/Widoco/](https://github.com/dgarijo/Widoco/). diff --git a/mkdocs.yml b/mkdocs.yml index e5baa108..124ba4c8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,6 +1,7 @@ extra_css: - stylesheets/mkdocs.css site_name: SOMEF documentation + nav: - Home: index.md - Install: install.md @@ -13,8 +14,11 @@ nav: - GitLab: gitlab.md - Codeberg: codeberg.md - Bitbucket: bitbucket.md + - Enrichment: + - OpenAlex: openalex.md + - OpenAIRE / Zenodo: openaire.md - theme: +theme: name: material features: - navigation.expand From 3bc7078a78fa95920dcf7085eede48b6e374f468 Mon Sep 17 00:00:00 2001 From: Juanje Mendoza Date: Fri, 3 Jul 2026 09:05:06 +0200 Subject: [PATCH 4/4] dani comments --- docs/bitbucket.md | 9 +-------- docs/codeberg.md | 9 --------- docs/github.md | 7 +------ docs/gitlab.md | 7 ------- 4 files changed, 2 insertions(+), 30 deletions(-) diff --git a/docs/bitbucket.md b/docs/bitbucket.md index 93709631..46cbaf2a 100644 --- a/docs/bitbucket.md +++ b/docs/bitbucket.md @@ -33,9 +33,6 @@ via `--bitbucket-token` and `--bitbucket-email`, or by running `somef configure` Repository archives are downloaded from: `https://bitbucket.org/{owner}/{repo}/get/{branch}.zip` -Bitbucket archive URLs typically include a `Content-Length` header, so the size limit check -can be performed before downloading. - ### Limitations - **No stargazers**: Bitbucket does not have a stargazers feature. @@ -46,8 +43,4 @@ can be performed before downloading. - **Programming languages**: Bitbucket returns only a single language string, without byte counts per language. - **CODEOWNERS enrichment**: Not supported for Bitbucket, as the platform does not - expose a public user API. -- **Rate limits**: Unauthenticated requests are limited to 60 requests/hour. - Authenticated requests (via `bitbucket-token` and `bitbucket-email` ) have higher limits) have higher limits. - Create an app password at `https://bitbucket.org/account/settings/api-tokens/` - with `read:repository:bitbucket` and `read:account` scopes. \ No newline at end of file + expose a public user API. \ No newline at end of file diff --git a/docs/codeberg.md b/docs/codeberg.md index 652d1677..1861f844 100644 --- a/docs/codeberg.md +++ b/docs/codeberg.md @@ -17,7 +17,6 @@ fields map to SOMEF categories: | `download_url` | *(built from URL)* | `https://codeberg.org/{owner}/{repo}/releases` | | `keywords` | `topics` | | | `issue_tracker` | *(built from URL)* | `{html_url}/issues` | -| `license` | *(content API)* | *1* | | `programming_languages` | `languages_url` | Additional GET request to the languages endpoint, returns byte counts per language | | `releases` | `/repos/{owner}/{repo}/releases` | Additional GET request, mapped via `release_codeberg_crosswalk_table` | @@ -26,11 +25,6 @@ uses `attachments` instead of `assets` for release files, and it does not provid `author.type` (`AGENT_TYPE`) for release authors. ---------------- - -*1* -Extracted by fetching the LICENSE file via `GET /api/v1/repos/{owner}/{repo}/contents/{filename}` (tries `LICENSE`, `LICENSE.md`, `LICENCE`, `COPYING`). The content is base64-decoded and analyzed with `detect_license_spdx()` to obtain the SPDX identifier, name and URL. - ### Archive download Repository archives are downloaded from: @@ -47,9 +41,6 @@ repository owner and for each CODEOWNERS entry. ### Limitations -- **Rate limits**: Unauthenticated requests are limited to 60 requests/hour. Authenticated - requests (via `codeberg-token`) have higher limits. Create a token at - `https://codeberg.org/user/settings/applications`. - **License detection**: Codeberg does not provide a `license` field in the repository API response. SOMEF falls back to fetching LICENSE/COPYING files directly via the content API. \ No newline at end of file diff --git a/docs/github.md b/docs/github.md index c8f05e83..5c087120 100644 --- a/docs/github.md +++ b/docs/github.md @@ -39,13 +39,8 @@ HTTP 300. SOMEF handles this by trying the following fallback URLs in order: ### Limitations -- **No Content-Length**: GitHub archive downloads lack a `Content-Length` header, so the - size limit check relies on streaming (reading 1 MB chunks until the limit is exceeded). -- **Rate limits**: Unauthenticated requests are limited to 60 requests/hour. Authenticated - requests (via `github-token`) are limited to 5,000 requests/hour. - **Private repositories**: SOMEF cannot access private repositories without a valid token. -- **Asset download count**: GitHub provides `download_count` for release assets; other - providers may not offer this field. + ### Enrichment via CODEOWNERS diff --git a/docs/gitlab.md b/docs/gitlab.md index 81502db0..b0605b69 100644 --- a/docs/gitlab.md +++ b/docs/gitlab.md @@ -46,12 +46,5 @@ own API endpoint; `gitlab.com` uses the standard `https://gitlab.com/api/v4/user ### Limitations -- **Self-hosted detection**: SOMEF detects self-hosted instances by checking for `gitlab.com` - in the URL. If a self-hosted instance uses a custom domain without a GitLab API endpoint, - detection may fail. -- **Rate limits**: GitLab.com unauthenticated requests are limited to 600 requests/hour. - Authenticated requests (via `gitlab-token`)have higher limits depending on the token type. Self-hosted - instances have their own rate limits. -- **Programming languages**: GitLab returns only language names (no byte counts). - **Release assets**: GitLab releases include sources (tar.gz, zip) and links, but do not provide `download_count` or `content_size` like GitHub. \ No newline at end of file