diff --git a/README.md b/README.md index 9644fd86..70060a94 100644 --- a/README.md +++ b/README.md @@ -57,13 +57,13 @@ When using `-e`, publication metadata is enriched via OpenAlex. We recognize the - **Full name**: Name + owner (owner/name) - **Full title**: If the repository is a short name, we will attempt to extract the longer version of the repository name - **Funding**: Funding information associated with the project. **Note**: This information is extracted from existing `codemeta.json` files within the repository. When using `-e`, the project data is enriched with OpenAIRE, adding: - - `project_code`: Project code - - `project_title`: Project title - - `project_acronym`: Project acronym - - `grant_id`: Call/grant identifier +- `project_code`: Project code +- `project_title`: Project title +- `project_acronym`: Project acronym +- `grant_id`: Call/grant identifier - **Identifier**: Identifier associated with the software (if any), such as Digital Object Identifiers and Software Heritage identifiers (SWH). DOIs associated with publications will also be detected. When using `-e`, the following enrichment identifiers are also added: - - `openaire_id`: URL to the OpenAIRE explore page for the software - - `swhid`: Software Heritage identifier (for Zenodo DOIs) +- `openaire_id`: URL to the OpenAIRE explore page for the software +- `swhid`: Software Heritage identifier (for Zenodo DOIs) - **Images**: Images used to illustrate the software component - **Installation instructions**: A set of instructions that indicate how to install a target repository - **Invocation**: Execution command(s) needed to run a scientific software component @@ -379,7 +379,8 @@ Options: -h, --help Show this message and exit. - -e, --enrichment Enrich metadata with external APIs (OpenAlex, OpenAIRE, Zenodo) + -e, --enrichment Enrich metadata with external APIs (OpenAlex, + OpenAIRE, Zenodo) --github-token TEXT GitHub personal access token (if invalid, stored config is used instead) @@ -407,6 +408,19 @@ Alternatively, you can set tokens via environment variables or by running `somef The CLI flags take precedence over stored config when valid. +### Enrichment with `-e` + +The `-e` (or `--enrichment`) flag queries external APIs to complete the extracted metadata: +- **OpenAlex**: Adds `openalex_id` to DOIs of publications and reconciles missing author ORCIDs. +- **OpenAIRE**: Adds `openaire_id` to publications/identifiers and enriches project funding metadata. +- **Zenodo**: Adds `swhid` (Software Heritage ID) for records matching Zenodo DOIs. + +For a detailed technical breakdown of the fields mapped by each external service, please refer to the specific documentation pages: +- See the [OpenAlex Mapping Guide](openalex.md) for citation and author properties. +- See the [OpenAIRE and Zenodo Mapping Guide](openaire.md) for funding and identifier properties. + +**Note:** Enrichment makes additional network requests to external services, which may slow down the overall execution time. Use this flag only when you need the extra metadata. + ## Usage example: The following command extracts all metadata available from [https://github.com/dgarijo/Widoco/](https://github.com/dgarijo/Widoco/). diff --git a/docs/bitbucket.md b/docs/bitbucket.md index 7c8eeba2..46cbaf2a 100644 --- a/docs/bitbucket.md +++ b/docs/bitbucket.md @@ -13,9 +13,34 @@ fields map to SOMEF categories: | `date_updated` | `updated_on` | | | `homepage` | `website` | | | `forks_url` | `links.forks.href` | | -| `download_url` | *(constructed)* | Built as `{html_url}/downloads` | -| `issue_tracker` | *(constructed)* | Built as `{html_url}/issues` when `has_issues` is true | +| `download_url` | *(built from URL)*| Built as `{html_url}/downloads` | +| `issue_tracker` | *(built from URL)* | Built as `{html_url}/issues` when `has_issues` is true | | `programming_languages` | `language` | Single string, not a dictionary with sizes | | `releases` | `/refs/tags` | Bitbucket has no dedicated releases endpoint; uses the tags endpoint | -| `stars` | *(not available)* | Bitbucket does not have a stargazers feature | -| `forks_count` | *(not available)* | Bitbucket does not expose fork counts in its API | \ No newline at end of file +| `stargazers_count` | *(not available)* | Bitbucket does not have a stargazers feature | +| `forks_count` | *(not available)* | Bitbucket does not expose fork counts in its API | + + +### Authentication + +Bitbucket uses HTTP Basic authentication. The token is encoded as +`base64(email:token)` and sent as the `Authorization` header. +Provide both the Bitbucket app password and your Atlassian account email +via `--bitbucket-token` and `--bitbucket-email`, or by running `somef configure`. + +### Archive download + +Repository archives are downloaded from: +`https://bitbucket.org/{owner}/{repo}/get/{branch}.zip` + +### Limitations + +- **No stargazers**: Bitbucket does not have a stargazers feature. +- **No forks count**: Bitbucket does not expose fork counts in its API. +- **Tags-only releases**: Bitbucket has no dedicated releases endpoint. + SOMEF uses the `/refs/tags` endpoint instead, which does not include release + descriptions or assets. +- **Programming languages**: Bitbucket returns only a single language string, + without byte counts per language. +- **CODEOWNERS enrichment**: Not supported for Bitbucket, as the platform does not + expose a public user API. \ No newline at end of file diff --git a/docs/codeberg.md b/docs/codeberg.md index 302826f3..1861f844 100644 --- a/docs/codeberg.md +++ b/docs/codeberg.md @@ -1,29 +1,46 @@ -When analyzing a Codeberg repository, SOMEF uses the [Codeberg API](https://codeberg.org/api/v1/swagger) -(`GET /api/v1/repos/{owner}/{repo}`) to retrieve metadata. The table below shows how Codeberg API +When analyzing a Codeberg repository, SOMEF uses the [Codeberg API](https://codeberg.org/api/v1/swagger) +(`GET /api/v1/repos/{owner}/{repo}`) to retrieve metadata. The table below shows how Codeberg API fields map to SOMEF categories: | SOMEF category | Codeberg API field | Notes | |---|---|---| -| `name` | `name` | | -| `description` | `description` | | | `code_repository` | `html_url` | | | `owner` | `owner.login` | | +| `full_name` | `full_name` | Format: `{owner}/{repo}` | +| `name` | `name` | | +| `description` | `description` | | | `date_created` | `created_at` | | | `date_updated` | `updated_at` | | -| `stars` | `stars_count` | In GitHub this field is `stargazers_count` | +| `stargazers_count` | `stars_count` | Called `stars_count` in Codeberg | | `forks_count` | `forks_count` | | -| `homepage` | `website` | In GitHub this field is `homepage` | +| `homepage` | `website` | Called `website` in Codeberg, `homepage` in GitHub | +| `download_url` | *(built from URL)* | `https://codeberg.org/{owner}/{repo}/releases` | | `keywords` | `topics` | | -| `issue_tracker` | *(constructed)* | Built as `{html_url}/issues` | -| `license` | *(content API)* | *1* | -| `programming_languages` | `languages_url` | Additional GET request to the languages endpoint | -| `releases` | `/repos/{owner}/{repo}/releases` | Additional GET request | +| `issue_tracker` | *(built from URL)* | `{html_url}/issues` | +| `programming_languages` | `languages_url` | Additional GET request to the languages endpoint, returns byte counts per language | +| `releases` | `/repos/{owner}/{repo}/releases` | Additional GET request, mapped via `release_codeberg_crosswalk_table` | -For releases, the field mapping is identical to GitHub. The only differences are that Codeberg -uses `attachments` instead of `assets` for release files, and it does not provide +For releases, the field mapping is identical to GitHub. The only differences are that Codeberg +uses `attachments` instead of `assets` for release files, and it does not provide `author.type` (`AGENT_TYPE`) for release authors. ---------------- -*1* -Extracted by fetching the LICENSE file via `GET /api/v1/repos/{owner}/{repo}/contents/{filename}` (tries `LICENSE`, `LICENSE.md`, `LICENCE`, `COPYING`). The content is base64-decoded and analyzed with `detect_license_spdx()` to obtain the SPDX identifier, name and URL. Technique: `Codeberg_API`. \ No newline at end of file +### Archive download + +Repository archives are downloaded from: +`https://codeberg.org/{owner}/{repo}/archive/{branch}.zip` + +Codeberg archive URLs typically include a `Content-Length` header, so the size limit check +can be performed before downloading. + +### Enrichment via CODEOWNERS + +When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details +(full name, email) from `GET https://codeberg.org/api/v1/users/{username}` for the +repository owner and for each CODEOWNERS entry. + +### Limitations + +- **License detection**: Codeberg does not provide a `license` field in the repository + API response. SOMEF falls back to fetching LICENSE/COPYING files directly via the + content API. \ No newline at end of file diff --git a/docs/github.md b/docs/github.md new file mode 100644 index 00000000..5c087120 --- /dev/null +++ b/docs/github.md @@ -0,0 +1,49 @@ +When analyzing a GitHub repository, SOMEF uses the [GitHub REST API](https://docs.github.com/en/rest/repos/repos) +(`GET /repos/{owner}/{repo}`) to retrieve metadata. The table below shows how GitHub API +fields map to SOMEF categories: + +| SOMEF category | GitHub API field | Notes | +|---|---|---| +| `code_repository` | `html_url` | | +| `owner` | `owner.login` | `agent_type` is extracted from `owner.type` (User or Organization) | +| `date_created` | `created_at` | | +| `date_updated` | `updated_at` | | +| `license` | `license` | Nested object with `spdx_id`, `name`, `url` | +| `description` | `description` | | +| `name` | `name` | | +| `full_name` | `full_name` | Format: `{owner}/{repo}` | +| `issue_tracker` | `issues_url` | The `{/number}` suffix is stripped | +| `forks_url` | `forks_url` | | +| `stargazers_count` | `stargazers_count` | | +| `keywords` | `topics` | | +| `forks_count` | `forks_count` | | +| `homepage` | `homepage` | | +| `programming_languages` | `languages` | Additional GET to `/repos/{owner}/{repo}/languages`. Returns a dictionary with byte counts per language | +| `releases` | `/repos/{owner}/{repo}/releases` | Paginated results, mapped via `release_crosswalk_table` | +| `download_url` | *(constructed)* | Built as `https://github.com/{owner}/{repo}/releases` | + +### Archive download + +SOMEF downloads the repository archive from `https://github.com/{owner}/{repo}/archive/{ref}.zip`. +GitHub archive URLs do not include a `Content-Length` header, so SOMEF uses a streaming check: +it reads the response in 1 MB chunks and aborts if the total exceeds the configured size limit +(see `--download-limit`). + +If the ref name is ambiguous (a branch and a tag share the same name), GitHub returns +HTTP 300. SOMEF handles this by trying the following fallback URLs in order: + +1. `https://github.com/{owner}/{repo}/archive/{ref}.zip` (short form) +2. `https://github.com/{owner}/{repo}/archive/refs/heads/{ref}.zip` (explicit branch) +3. `https://github.com/{owner}/{repo}/archive/refs/tags/{ref}.zip` (explicit tag) +4. `https://github.com/{owner}/{repo}/archive/main.zip` (legacy rename fallback) + +### Limitations + +- **Private repositories**: SOMEF cannot access private repositories without a valid token. + + +### Enrichment via CODEOWNERS + +When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details +(name, company, email) from `GET https://api.github.com/users/{username}` for the +repository owner and for each CODEOWNERS entry. \ No newline at end of file diff --git a/docs/gitlab.md b/docs/gitlab.md new file mode 100644 index 00000000..b0605b69 --- /dev/null +++ b/docs/gitlab.md @@ -0,0 +1,50 @@ +When analyzing a GitLab repository, SOMEF uses the [GitLab REST API](https://docs.gitlab.com/ee/api/projects.html) +(`GET /api/v4/projects/{project_id}`) to retrieve metadata. The table below shows how GitLab API +fields map to SOMEF categories: + +| SOMEF category | GitLab API field | Notes | +|---|---|---| +| `code_repository` | *(built from URL)* | `https://{host}/{project_path}/` | +| `download_url` | *(built from URL)* | `https://{host}/{project_path}/-/branches` | +| `defaultBranch` | `default_branch` | Also checks `defaultBranch` (legacy key) | +| `description` | `description` | | +| `name` | `name` | | +| `full_name` | `path_with_namespace` | | +| `owner` | `owner.username` | `agent_type` deduced from `namespace.kind` ("group" → Organization, "user" → Person) | +| `date_created` | `created_at` | | +| `date_updated` | `last_activity_at` | | +| `issue_tracker` | *(built from URL)* | `https://{host}/{project_path}/-/issues` | +| `license` | `license` | `name` and `url` with SPDX detection. Falls back to fetching raw LICENSE file | +| `keywords` | `topics` | | +| `stargazers_count` | `star_count` | Called `star_count` in GitLab | +| `forks_count` | `forks_count` | | +| `programming_languages` | `languages` | Only language names (keys), no byte counts | +| `readme_url` | `readme_url` | | +| `releases` | `/projects/{id}/releases` | Paginated via `X-Next-Page`, mapped via `release_gitlab_crosswalk_table` | + + +### Self-hosted GitLab instances + +SOMEF supports self-hosted GitLab instances (e.g., `gitlab.in2p3.fr`). The project path is +URL-encoded and used to query the instance's API at `https://{host}/api/v4/projects/{encoded_path}`. +SOMEF detects self-hosted instances by checking if the URL contains `gitlab.com`. + +### Archive download + +Repository archives are downloaded from: +`https://{host}/{project_path}/-/archive/{branch}/{repo_name}-{branch}.zip` + +GitLab archive URLs typically include a `Content-Length` header, so the size limit check +can be performed before downloading. + +### Enrichment via CODEOWNERS + +When `--reconcile_authors` (`-ra`) is enabled, SOMEF fetches additional user details +(name, organization, public email) from `GET {server_url}/api/v4/users?username={username}` +for the repository owner and for each CODEOWNERS entry. Self-hosted instances use their +own API endpoint; `gitlab.com` uses the standard `https://gitlab.com/api/v4/users`. + +### Limitations + +- **Release assets**: GitLab releases include sources (tar.gz, zip) and links, but do not + provide `download_count` or `content_size` like GitHub. \ No newline at end of file diff --git a/docs/openaire.md b/docs/openaire.md new file mode 100644 index 00000000..953685e5 --- /dev/null +++ b/docs/openaire.md @@ -0,0 +1,23 @@ +# OpenAIRE and Zenodo API Mapping + +When the enrichment flag (`-e` / `--enrichment`) is enabled, SOMEF integrates with the **OpenAIRE** and **Zenodo** APIs to fetch project context, funding background, and ecosystem identifiers. + +## Funding Enrichment (OpenAIRE) + +These fields are injected into the **Funding** category when a matching grant or project is reconciled via OpenAIRE keywords or grant identifiers. + +| SOMEF Property | Source API Field | Description | +| :--- | :--- | :--- | +| `project_code` | `code` | The official project or grant code assigned by the funding body. | +| `project_title` | `title` | The full title of the funded research project. | +| `project_acronym` | `acronym` | The official acronym of the project. | +| `grant_id` | `callidentifier` | Unique identifier for the specific call or grant agreement. | + +## Identifier Enrichment (OpenAIRE & Zenodo) + +When processing repository identifiers or DOIs found in citations, SOMEF resolves them against OpenAIRE to construct an explore landing page link. Additionally, if the DOI belongs to Zenodo, SOMEF queries the Zenodo API to extract its corresponding Software Heritage identifier (`swhid`). + +| SOMEF Property | Source API / Function | Description | +| :--- | :--- | :--- | +| `openaire_id` | `get_openaire_id(doi)` | Direct link to the OpenAIRE explore page dedicated to this artifact. | +| `swhid` | `get_zenodo_swhid(doi)` | The Software Heritage Identifier (SWHID) associated with a Zenodo record. | \ No newline at end of file diff --git a/docs/openalex.md b/docs/openalex.md new file mode 100644 index 00000000..ed96bce0 --- /dev/null +++ b/docs/openalex.md @@ -0,0 +1,17 @@ +# OpenAlex API Mapping + +When the enrichment flag (`-e` / `--enrichment`) is enabled, SOMEF integrates with the **OpenAlex API** to resolve and complete scholarly publication data and missing author identities. + +## Citation Enrichment +If OpenAIRE fails to resolve a publication's DOI, SOMEF queries OpenAlex as a fallback to fetch the publication's unique ID. + +| SOMEF Property | Source API Function | Description | +| :--- | :--- | :--- | +| `openalex_id` | `get_openalex_id(doi)` | The unique OpenAlex URL identifier assigned to the publication. | + +## Author/Contributor Enrichment +For extracted authors or contributors who lack a structured identifier, SOMEF searches OpenAlex by name to retrieve their professional ORCID. + +| SOMEF Property | Source API Function | Description | +| :--- | :--- | :--- | +| `identifier` | `search_openalex_author(name)` | The author's verified ORCID URL (used as fallback when missing in local files). | \ No newline at end of file diff --git a/docs/usage.md b/docs/usage.md index fa68fbbc..f89c4a69 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -82,6 +82,9 @@ Options: -h, --help Show this message and exit. + -e, --enrichment Enrich metadata with external APIs (OpenAlex, + OpenAIRE, Zenodo) + --github-token TEXT GitHub personal access token (if invalid, stored config is used instead) @@ -109,6 +112,18 @@ GitHub, GitLab and Codeberg; Bitbucket does not expose a public user API, so enr Alternatively, you can set tokens via environment variables or by running `somef configure`, which stores them permanently. The CLI flags take precedence over stored config when valid. +### Enrichment with `-e` + +The `-e` (or `--enrichment`) flag queries external APIs to complete the extracted metadata: +- **OpenAlex**: Adds `openalex_id` to DOIs of publications and reconciles missing author ORCIDs. +- **OpenAIRE**: Adds `openaire_id` to publications/identifiers and enriches project funding metadata. +- **Zenodo**: Adds `swhid` (Software Heritage ID) for records matching Zenodo DOIs. + +For a detailed technical breakdown of the fields mapped by each external service, please refer to the specific documentation pages: +- See the [OpenAlex Mapping Guide](openalex.md) for citation and author properties. +- See the [OpenAIRE and Zenodo Mapping Guide](openaire.md) for funding and identifier properties. + +**Note:** Enrichment makes additional network requests to external services, which may slow down the overall execution time. Use this flag only when you need the extra metadata. ## Usage example: The following command extracts all metadata available from [https://github.com/dgarijo/Widoco/](https://github.com/dgarijo/Widoco/). diff --git a/mkdocs.yml b/mkdocs.yml index 4756d54c..124ba4c8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,6 +1,7 @@ extra_css: - stylesheets/mkdocs.css site_name: SOMEF documentation + nav: - Home: index.md - Install: install.md @@ -8,7 +9,15 @@ nav: - Output: output.md - Supported metadata files: supported_metadata_files.md - Supported languages: supported_languages.md - + - Supported APIs: + - GitHub: github.md + - GitLab: gitlab.md + - Codeberg: codeberg.md + - Bitbucket: bitbucket.md + - Enrichment: + - OpenAlex: openalex.md + - OpenAIRE / Zenodo: openaire.md + theme: name: material features: diff --git a/src/somef/process_repository.py b/src/somef/process_repository.py index d77347f6..1fd161fa 100644 --- a/src/somef/process_repository.py +++ b/src/somef/process_repository.py @@ -398,6 +398,57 @@ def load_gitlab_repository_metadata(repo_metadata: Result, repository_url, autho } repo_metadata.add_result(constants.CAT_KEYWORDS, result, 1, constants.TECHNIQUE_GITLAB_API) + # extract name + if project_details.get('name'): + repo_metadata.add_result(constants.CAT_NAME, { + constants.PROP_VALUE: project_details['name'], + constants.PROP_TYPE: constants.STRING + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract full_name + if project_details.get('path_with_namespace'): + repo_metadata.add_result(constants.CAT_FULL_NAME, { + constants.PROP_VALUE: project_details['path_with_namespace'], + constants.PROP_TYPE: constants.STRING + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract description + if project_details.get('description'): + repo_metadata.add_result(constants.CAT_DESCRIPTION, { + constants.PROP_VALUE: project_details['description'], + constants.PROP_TYPE: constants.STRING + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract owner + if project_details.get('owner') and project_details['owner'].get('username'): + owner_type = "Person" + if project_details.get('namespace') and project_details['namespace'].get('kind'): + owner_type = "Organization" if project_details['namespace']['kind'] == 'group' else "Person" + repo_metadata.add_result(constants.CAT_OWNER, { + constants.PROP_VALUE: project_details['owner']['username'], + constants.PROP_TYPE: owner_type + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract date_created + if project_details.get('created_at'): + repo_metadata.add_result(constants.CAT_DATE_CREATED, { + constants.PROP_VALUE: project_details['created_at'], + constants.PROP_TYPE: constants.DATE + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract date_updated + if project_details.get('last_activity_at'): + repo_metadata.add_result(constants.CAT_DATE_UPDATED, { + constants.PROP_VALUE: project_details['last_activity_at'], + constants.PROP_TYPE: constants.DATE + }, 1, constants.TECHNIQUE_GITLAB_API) + + # extract issue_tracker + repo_metadata.add_result(constants.CAT_ISSUE_TRACKER, { + constants.PROP_VALUE: f"https://{url.netloc}/{project_path}/-/issues", + constants.PROP_TYPE: constants.URL + }, 1, constants.TECHNIQUE_GITLAB_API) + # get social features: stargazers_count if project_details['star_count'] is not None: result = {