Codeberg and Bitbucket repos. Fixes #1007

.gitignore

@@ -24,5 +24,7 @@ repos.txt
 !package_neors.json
 !package_npm.json
 !test_data/api_responses/*.json
+!**/test_data/api_responses/codeberg/*.json
+!**/test_data/api_responses/bitbucket/*.json
 uv.lock
 .python-version

README.md

@@ -13,7 +13,7 @@ A command line interface for automatically extracting relevant metadata from cod
 
 ## Features
 
-Given a readme file (or a GitHub/Gitlab repository) SOMEF will extract the following categories (if present), listed in alphabetical order:
+Given a readme file (or a GitHub/Gitlab/Codeberg/Bitbucket repository) SOMEF will extract the following categories (if present), listed in alphabetical order:
 
 - **Acknowledgement**: Text acknowledging funding sources or contributors
 - **Application domain**: The application domain of the repository. Current supported domains include: Astrophysics, Audio, Computer vision, Graphs, Natural language processing, Reinforcement learning, Semantc web, Sequential. Domains are not mutually exclusive. These domains have been extracted from [awesome lists](https://github.com/topics/awesome-list) and [Papers with code](https://paperswithcode.com/). Find more information in our [documentation](https://somef.readthedocs.io/en/latest/)
@@ -38,7 +38,7 @@ We recognize the following properties:
   - Year: Year of publication
   - Pages: Page range in the journal
 - **Code of conduct**: Link to the code of conduct of the project
-- **Code repository**: Link to the GitHub/GitLab repository used for the extraction
+- **Code repository**: Link to the GitHub/GitLab/Codeberg and Bitbucket repository used for the extraction
 - **Contact**: Contact person responsible for maintaining a software component
 - **Continuous integration**: Link to continuous integration service(s)
 - **Contribution guidelines**: Text indicating how to contribute to this code repository
@@ -72,7 +72,7 @@ We recognize the following properties:
 - **Package files**: Links to package files used to wrap the project in a package.
 - **Programming languages**: Languages used in the repository
 - **Related papers**: URL to possible related papers within the repository stated within the readme file (from Arxiv)
-- **Releases** (GitHub only): Pointer to the available versions of a software component. For each release, somef will track the following properties:
+- **Releases**: Pointer to the available versions of a software component. For each release, somef will track the following properties:
   - Description: Release notes
   - Author: Agent responsible of creating the release
   - Name: Name of the release
@@ -93,7 +93,7 @@ We recognize the following properties:
 - **Usage examples**: Assumptions and considerations recorded by the authors when executing a software component, or examples on how to use it
 - **Workflows**: URL and path to the computational workflow files present in the repository
 
-We use different supervised classifiers, header analysis, regular expressions, the GitHub/Gitlab API to retrieve all these fields (more than one technique may be used for each field) and language specific metadata parsers (e.g., for package files). Each extraction records its provenance, with the confidence and technique used on each step. For more information check the [output format description](https://somef.readthedocs.io/en/latest/output/)
+We use different supervised classifiers, header analysis, regular expressions, the GitHub/Gitlab/Codeberg and Bitbucket API to retrieve all these fields (more than one technique may be used for each field) and language specific metadata parsers (e.g., for package files). Each extraction records its provenance, with the confidence and technique used on each step. For more information check the [output format description](https://somef.readthedocs.io/en/latest/output/)
 
 ### Confidence values in header analysis
 
@@ -265,10 +265,14 @@ somef configure
 
 And you will be asked to provide the following:
 
-- A GitHub authentication token [**optional, leave blank if not used**], which SOMEF uses to retrieve metadata from GitHub. If you don't include an authentication token, you can still use SOMEF. However, you may be limited to a series of requests per hour. For more information, see [https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line)
-- The path to the trained classifiers (pickle files). If you have your own classifiers, you can provide them here. Otherwise, you can leave it blank
+- A **GitHub** authentication token [**optional, leave blank if not used**], which SOMEF uses to retrieve metadata from GitHub. If you don't include an authentication token, you can still use SOMEF. However, you may be limited to a series of requests per hour. For more information, see [https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line)
+- A **GitLab** authentication token [**optional**], used for GitLab.com and self-hosted GitLab instances (e.g., `gitlab.in2p3.fr`). Tokens are per-instance. Note: **a token from GitLab.com does not work for self-hosted servers**. Create one at `https://gitlab.com/-/user_settings/personal_access_tokens` (scope: `read_api`). Without a token, some self-hosted GitLab instances may not return rate limit information.
+- A **Codeberg** authentication token [**optional**], used to retrieve metadata from Codeberg. Create one at `https://codeberg.org/user/settings/applications` (permissions: `read:repository`, `read:user`). Codeberg (Forgejo) does not expose rate limit headers even with a token.
+- A **Bitbucket** authentication token [**optional**], used for Bitbucket Cloud. Create an API token with scopes at `https://bitbucket.org/account/settings/api-tokens/` (permissions: `read:repository:bitbucket`, `read:account`). You will also need to provide your Atlassian account email, as Bitbucket API tokens use Basic authentication (`email:token` encoded in base64). Without a token you are limited to 60 requests/hour.
+- The path to the trained classifiers (pickle files). If you have your own classifiers, you can provide them here. Otherwise, you can leave it blank.
 
-If you want somef to be automatically configured (without GitHUb authentication key and using the default classifiers) just type:
+
+If you want SOMEF to be automatically configured (without any tokens and using the default classifiers) just type:
 
 ```bash
 somef configure -a
@@ -310,10 +314,10 @@ Usage: somef describe [OPTIONS]
 Options:
   -t, --threshold FLOAT           Threshold to classify the text  [required]
   Input: [mutually_exclusive, required]
-    -r, --repo_url URL            Github/Gitlab Repository URL
+    -r, --repo_url URL            Github/Gitlab/Codeberg/Bitbucket Repository URL
     -d, --doc_src PATH            Path to the README file source
     -i, --in_file PATH            A file of newline separated links to GitHub/
-                                  Gitlab repositories
+                                  Gitlab/Codeberg/Bitbucket repositories
     -l, --local_repo PATH         Path to the local repository source. No APIs will be used
 
   Output: [required_any]
@@ -360,13 +364,32 @@ Options:
                                   requests and increase execution time
 
   -h, --help                      Show this message and exit.
+
+  --github-token TEXT             GitHub personal access token (if invalid,
+                                  stored config is used instead)
+
+  --gitlab-token TEXT             GitLab personal access token (if invalid,
+                                  stored config is used instead)
+
+  --codeberg-token TEXT           Codeberg personal access token (if invalid,
+                                  stored config is used instead)
+
+  --bitbucket-token TEXT          Bitbucket app password (if invalid, stored 
+                                  config is used instead)
+
+  --bitbucket-email TEXT          Bitbucket Atlassian account email (required
+                                  with --bitbucket-token)
 
   Repoository versions [mutually_exclusive] (see section *Repository versions*t):
   -b, --branch name branch        Branch of the repository to analyze. Overrides the default branch.
 
       --tag text                  Tag of the repository to analyze. Cannot be used together with --branch.
 ```
 
+Alternatively, you can set tokens via environment variables or by running `somef configure`, which stores them permanently.
+The CLI flags take precedence over stored config when valid.
+
+
 ## Usage example:
 
 The following command extracts all metadata available from [https://github.com/dgarijo/Widoco/](https://github.com/dgarijo/Widoco/).

docs/bitbucket.md

@@ -0,0 +1,21 @@
+When analyzing a Bitbucket repository, SOMEF uses the [Bitbucket Cloud API](https://developer.atlassian.com/cloud/bitbucket/rest/api-group-repositories/)
+(`GET /2.0/repositories/{workspace}/{repo_slug}`) to retrieve metadata. The table below shows how Bitbucket API
+fields map to SOMEF categories:
+
+| SOMEF category | Bitbucket API field | Notes |
+|---|---|---|
+| `name` | `slug` | |
+| `description` | `description` | |
+| `full_name` | `full_name` | Format: `{workspace}/{slug}` |
+| `code_repository` | `links.html.href` | |
+| `owner` | `owner.nickname` | Falls back to `owner.username` for team workspaces |
+| `date_created` | `created_on` | |
+| `date_updated` | `updated_on` | |
+| `homepage` | `website` | |
+| `forks_url` | `links.forks.href` | |
+| `download_url` | *(constructed)* | Built as `{html_url}/downloads` |
+| `issue_tracker` | *(constructed)* | Built as `{html_url}/issues` when `has_issues` is true |
+| `programming_languages` | `language` | Single string, not a dictionary with sizes |
+| `releases` | `/refs/tags` | Bitbucket has no dedicated releases endpoint; uses the tags endpoint |
+| `stars` | *(not available)* | Bitbucket does not have a stargazers feature |
+| `forks_count` | *(not available)* | Bitbucket does not expose fork counts in its API |

docs/codeberg.md

@@ -0,0 +1,29 @@
+When analyzing a Codeberg repository, SOMEF uses the [Codeberg API](https://codeberg.org/api/v1/swagger) 
+(`GET /api/v1/repos/{owner}/{repo}`) to retrieve metadata. The table below shows how Codeberg API 
+fields map to SOMEF categories:
+
+| SOMEF category | Codeberg API field | Notes |
+|---|---|---|
+| `name` | `name` | |
+| `description` | `description` | |
+| `code_repository` | `html_url` | |
+| `owner` | `owner.login` | |
+| `date_created` | `created_at` | |
+| `date_updated` | `updated_at` | |
+| `stars` | `stars_count` | In GitHub this field is `stargazers_count` |
+| `forks_count` | `forks_count` | |
+| `homepage` | `website` | In GitHub this field is `homepage` |
+| `keywords` | `topics` | |
+| `issue_tracker` | *(constructed)* | Built as `{html_url}/issues` |
+| `license` | *(content API)* | *1* |
+| `programming_languages` | `languages_url` | Additional GET request to the languages endpoint |
+| `releases` | `/repos/{owner}/{repo}/releases` | Additional GET request |
+
+For releases, the field mapping is identical to GitHub. The only differences are that Codeberg 
+uses `attachments` instead of `assets` for release files, and it does not provide 
+`author.type` (`AGENT_TYPE`) for release authors.
+
+--------------- 
+
+*1*
+Extracted by fetching the LICENSE file via `GET /api/v1/repos/{owner}/{repo}/contents/{filename}` (tries `LICENSE`, `LICENSE.md`, `LICENCE`, `COPYING`). The content is base64-decoded and analyzed with `detect_license_spdx()` to obtain the SPDX identifier, name and URL. Technique: `Codeberg_API`.

docs/index.md

@@ -46,7 +46,7 @@ We recognize the following properties:
   - Year: Year of publication
   - Pages: Page range in the journal
 - **Code of conduct**: Link to the code of conduct of the project
-- **Code repository**: Link to the GitHub/GitLab repository used for the extraction
+- **Code repository**: Link to the GitHub/GitLab/Codeberg/Bitbucket repository used for the extraction
 - **Contact**: Contact person responsible for maintaining a software component
 - **Continuous integration**: Link to continuous integration service(s)
 - **Contribution guidelines**: Text indicating how to contribute to this code repository
@@ -80,7 +80,7 @@ We recognize the following properties:
 - **Package files**: Links to package files used to wrap the project in a package.
 - **Programming languages**: Languages used in the repository
 - **Related papers**: URL to possible related papers within the repository stated within the readme file (from Arxiv)
-- **Releases** (GitHub and Gitlab): Pointer to the available versions of a software component. For each release, somef will track the following properties:
+- **Releases** (GitHub, Gitlab, Codeberg and Bitbucket): Pointer to the available versions of a software component. For each release, somef will track the following properties:
   - Assets: files attached to the release
   - Description: Release notes
   - Author: Agent responsible of creating the release
@@ -102,7 +102,7 @@ We recognize the following properties:
 - **Usage examples**: Assumptions and considerations recorded by the authors when executing a software component, or examples on how to use it
 - **Workflows**: URL and path to the computational workflow files present in the repository
 
-We use different supervised classifiers, header analysis, regular expressions, the GitHub/Gitlab API to retrieve all these fields (more than one technique may be used for each field) and language specific metadata parsers (e.g., for package files). Each extraction records its provenance, with the confidence and technique used on each step. For more information check the [output format description](https://somef.readthedocs.io/en/latest/output/)
+We use different supervised classifiers, header analysis, regular expressions, the GitHub/Gitlab/Codeberg/Bitbucket API to retrieve all these fields (more than one technique may be used for each field) and language specific metadata parsers (e.g., for package files). Each extraction records its provenance, with the confidence and technique used on each step. For more information check the [output format description](https://somef.readthedocs.io/en/latest/output/)
 
 <a name="myfootnote1">1</a> The available application domains currently are: 
 

docs/install.md

@@ -83,25 +83,25 @@ docker run -it --rm -v $PWD/:/out kcapd/somef /bin/bash
 ```
 If you move any files produced by somef into `/out`, then you will be able to see them in your current directory.
 
+
 ## Configure
-Before running SOMEF, you must configure it appropriately. Run:
 
-```bash
-python -m nltk.downloader wordnet
-python -m nltk.downloader omw-1.4
-```
-To download two wordnet modules needed. Then run:
+Before running SOMEF for the first time, you must **configure** it appropriately (you only need to do this once). Run:
 
 ```bash
 somef configure
 ```
 
-And you will be asked to provide the following: 
+And you will be asked to provide the following:
 
-- A GitHub authentication token [**optional, leave blank if not used**], which SOMEF uses to retrieve metadata from GitHub. If you don't include an authentication token, you can still use SOMEF. However, you may be limited to a series of requests per hour. For more information, see [https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line) 
-- The path to the trained classifiers (pickle files). If you have your own classifiers, you can provide them here. Otherwise, you can leave it blank
+- A **GitHub** authentication token [**optional, leave blank if not used**], which SOMEF uses to retrieve metadata from GitHub. If you don't include an authentication token, you can still use SOMEF. However, you may be limited to a series of requests per hour. For more information, see [https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line](https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line)
+- A **GitLab** authentication token [**optional**], used for GitLab.com and self-hosted GitLab instances (e.g., `gitlab.in2p3.fr`). Tokens are per-instance. Note: **a token from GitLab.com does not work for self-hosted servers**. Create one at `https://gitlab.com/-/user_settings/personal_access_tokens` (scope: `read_api`). Without a token, some self-hosted GitLab instances may not return rate limit information.
+- A **Codeberg** authentication token [**optional**], used to retrieve metadata from Codeberg. Create one at `https://codeberg.org/user/settings/applications` (permissions: `read:repository`, `read:user`). Codeberg (Forgejo) does not expose rate limit headers even with a token.
+- A **Bitbucket** authentication token [**optional**], used for Bitbucket Cloud. Create an API token with scopes at `https://bitbucket.org/account/settings/api-tokens/` (permissions: `read:repository:bitbucket`, `read:account`). You will also need to provide your Atlassian account email, as Bitbucket API tokens use Basic authentication (`email:token` encoded in base64). Without a token you are limited to 60 requests/hour.
+- The path to the trained classifiers (pickle files). If you have your own classifiers, you can provide them here. Otherwise, you can leave it blank.
 
-If you want to configure SOMEF with the default parameters, just type:
+
+If you want SOMEF to be automatically configured (without any tokens and using the default classifiers) just type:
 
 ```bash
 somef configure -a
@@ -112,6 +112,7 @@ For showing help about the available options, run:
 ```bash
 somef configure --help
 ```
+
 Which displays:
 
 ```bash

docs/output.md

@@ -73,7 +73,7 @@ SOMEF aims to recognize the following categories (in alphabetical order):
 - `code_of_conduct`: Link to the code of conduct file of the project
 - `code_repository`: Link to the source code (typically the repository where the readme can be found)
 - `contact`: Contact person responsible for maintaining a software component.
-- `continuous_integration`: Link to continuous integration service, supported on GitHub as well as in GitLab.
+- `continuous_integration`: Link to continuous integration service, supported on GitHub as well as in GitLab, Codeberg and Bitbucket.
 - `contributing guidelines`: Guidelines indicating how to contribute to a software component.
 - `contributor`: Contributors to this software. Note: Contributor metadata is exported from metadata files (e.g., CodeMeta, CONTRIBUTORS, etc.) not from git logs.
 - `copyright_holder`: Entity or individual owning the rights to the software. The year is also extracted, if available.
@@ -180,7 +180,7 @@ Depending on the `type` of the result, additional properties may be found.
 
 The following object `types` are currently supported:
 
-- `Release`: software releases of the current code repository, as available from GitHub.
+- `Release`: software releases of the current code repository, as available from GitHub, GitLab and Codeberg
 - `Programming_language`: Programming language used in the repository. 
 - `License`: object representing all the metadata SOMEF extracts from a license.
 - `Agent`: user (typically, a person) or organization responsible for authoring a software release or a paper.
@@ -330,6 +330,8 @@ The techniques can be of several types:
 - `file_exploration`: the result comes from an exploration of the files in the repository
 - `GitHub_API`: the result was obtained from the GitHub API.
 - `GitLab_API`: the result was obtained from the GitLab API.
+- `Codeberg_API`: the result was obtained from the Codeberg API.
+- `Bitbucket_API`: the result was obtained from the Bitbucket API.
 - `regular_expression`: the result was obtained after performing regular expressions on the files in the repository.
 - `software_type_heuristics`: the result was obtained from analysis of the repository based on various heuristics from the README, code and extension analysis. 
 - `supervised_classification`: the results were obtained after running text classifiers trained for detecting that type of header.