Add support for extra options in the fullDomFetcher (#1173)

Author: Ndpnt

.env.example

@@ -1,8 +1,21 @@
-OTA_ENGINE_SENDINBLUE_API_KEY='xkeysib-3f51c…'
-OTA_ENGINE_SMTP_PASSWORD='password'
+# Open Terms Archive Engine - Environment Variables Example
+# Copy this file to .env and fill in your actual values
 
-# If both GitHub and GitLab tokens are defined, GitHub takes precedence for dataset publishing
-OTA_ENGINE_GITHUB_TOKEN=ghp_XXXXXXXXX
+OTA_ENGINE_GITHUB_TOKEN=your_github_token_here
+OTA_ENGINE_GITLAB_TOKEN=your_gitlab_token_here
+OTA_ENGINE_GITLAB_RELEASES_TOKEN=your_gitlab_releases_token_here
+OTA_ENGINE_SENDINBLUE_API_KEY=your_sendinblue_api_key_here
+OTA_ENGINE_SMTP_PASSWORD=your_smtp_password_here
 
-OTA_ENGINE_GITLAB_TOKEN=XXXXXXXXXX
-OTA_ENGINE_GITLAB_RELEASES_TOKEN=XXXXXXXXXX
+HTTP_PROXY=http://proxy.example.com:8080
+HTTPS_PROXY=https://proxy.example.com:8080
+
+# Alternative lowercase versions (some systems prefer these)
+# http_proxy=http://proxy.example.com:8080
+# https_proxy=https://proxy.example.com:8080
+
+# Disable headless mode for Puppeteer (shows browser window)
+OTA_ENGINE_FETCHER_NO_HEADLESS=1
+
+# Disable Chrome sandbox (required for some Docker environments)
+OTA_ENGINE_FETCHER_NO_SANDBOX=1

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 All changes that impact users of this module are documented in this file, in the [Common Changelog](https://common-changelog.org) format with some additional specifications defined in the CONTRIBUTING file. This codebase adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## Unreleased [minor]
+
+> Development of this release was supported by the [European Commission](https://commission.europa.eu/) for its [VLOPs/VLOSEs instance](https://code.europa.eu/dsa/terms-and-conditions-database/vlops-and-vloses/).
+
+### Added
+
+- Add proxy support for fetching documents behind firewalls or restricted networks; configure using `HTTP_PROXY` and `HTTPS_PROXY` (or `http_proxy` and `https_proxy`) environment variables
+- Add debugging options to disable headless mode for visual troubleshooting during development; set `OTA_ENGINE_FETCHER_NO_HEADLESS=1` to show browser window
+- Add sandbox control for improved compatibility with Docker and containerized environments; set `OTA_ENGINE_FETCHER_NO_SANDBOX=1` when running in containers
+
 ## 9.1.2 - 2025-10-30
 
 _Full changeset and discussions: [#1199](https://github.com/OpenTermsArchive/engine/pull/1199)._

CONTRIBUTING.md

@@ -8,6 +8,7 @@ First of all, thanks for taking the time to contribute! 🎉👍
   - [Commit messages](#commit-messages)
   - [Changelog](#changelog)
 - [Development](#development)
+  - [Configuration and environment variables](#configuration-and-environment-variables)
   - [Documentation](#documentation)
 - [Naming](#naming)
   - [Instances and repositories](#instances-and-repositories)
@@ -75,7 +76,7 @@ Changes that require an adjustment in the infrastructure, they are considered as
 
 4. Since each release is produced automatically from a single pull request, the [notice](https://common-changelog.org/#23-notice) links to the source pull request rather than [references](https://common-changelog.org/#242-references), which would always reference the same pull request. References can link to relevant parts of an RFC, decision record, or diff. **This notice is automatically generated by the CI during the release process and should not be added manually.**
 
-5. The [notice](https://common-changelog.org/#23-notice) is also used to present sponsor information and it is required. Since the development of this project is funded by different actors, and following discussions with sponsors, financial contributions are acknowledged in the changelog itself. The format of the notice thus diverges from the Common Changelog specification in that it is not “a single-sentence paragraph”. Sponsor information is in quote format, starts with “Development of this release was supported by <funding_from>”, and provides the name and link to the sponsor, as well as information on the specific funding instrument, as specified by the sponsor itself or as required by law. A short message from the sponsor might also be added, as long as it abides by the community’s [Code of Conduct](./CODE_OF_CONDUCT.md) and aligns with the project’s goals. For volunteer contributions, the sentence should start with: “Development of this release was made on a volunteer basis by <contributor_name>”
+5. The [notice](https://common-changelog.org/#23-notice) is also used to present sponsor information and it is required. Since the development of this project is funded by different actors, as a matter of transparency and recognition, financial contributions and contributions supported by employers are acknowledged in the changelog itself. The format of the notice thus diverges from the Common Changelog specification in that it is not “a single-sentence paragraph”. Sponsor information is in quote format, starts with “Development of this release was supported by <funding_from>”, and provides the name and link to the sponsor, as well as information on the specific funding instrument, as specified by the sponsor itself or as required by law. A short message from the sponsor might also be added, as long as it abides by the community’s [Code of Conduct](./CODE_OF_CONDUCT.md) and aligns with the project’s goals. For volunteer contributions, the sentence should start with: “Development of this release was made on a volunteer basis by <contributor_name>”
 
 #### Changes that do not impact users
 
@@ -91,6 +92,40 @@ This content will be automatically deleted by the CI after merging.
 
 ## Development
 
+### Configuration and environment variables
+
+The choice between environment variables and configuration files should be made based on the nature of the data and how it will be used.
+
+**Use environment variables for:**
+
+- Secrets: API keys, passwords, tokens, or any sensitive data that should not be committed to version control. Examples:
+  - `OTA_ENGINE_GITHUB_TOKEN`: GitHub API token for creating issues and managing repositories
+  - `OTA_ENGINE_SMTP_PASSWORD`: password for SMTP server authentication
+- Debugging flags: toggles for development features. Examples:
+  - `OTA_ENGINE_FETCHER_NO_HEADLESS`: disables headless mode in Puppeteer to show the browser window during fetching
+- Unix standards: system-level settings following Unix conventions. Examples:
+  - `HTTP_PROXY`, `HTTPS_PROXY`, `http_proxy`, `https_proxy`: proxy server configuration for HTTP/HTTPS requests
+- Runtime overrides: container-specific or deployment-specific settings that vary between environments. Examples:
+  - `OTA_ENGINE_FETCHER_NO_SANDBOX`: disables Chrome sandbox (required in some Docker environments)
+
+**Use configuration files for:**
+
+- Engine behavior: Core functionality settings that define how the application operates. Examples:
+  - `trackingSchedule`: Cron expression defining when to track terms (e.g., `"30 */12 * * *"` for every 12 hours)
+  - `fetcher.language`: Language code for Accept-Language header in HTTP requests
+- Service settings: External service endpoints and integration parameters. Examples:
+  - `versionsRepositoryURL`: URL of the GitHub repository storing document versions
+  - `logger.smtp.host`: SMTP server hostname for sending error notifications
+- Static infrastructure: Deployment-independent paths and identifiers. Examples:
+  - `recorder.versions.storage.git.path`: File system path where Git repository for versions is stored
+  - `recorder.versions.storage.git.author`: Git commit author name and email for automated commits
+
+When uncertain whether to use an environment variable or a configuration file, consider:
+
+- Does it contain sensitive information? → Environment variable
+- Should it be version-controlled and reviewed? → Configuration file
+- Is it a stable setting that defines application behavior? → Configuration file
+
 ### Documentation
 
 #### Copywriting

src/archivist/fetcher/fullDomFetcher.js

@@ -1,6 +1,8 @@
 import puppeteer from 'puppeteer-extra';
 import stealthPlugin from 'puppeteer-extra-plugin-stealth';
 
+import { resolveProxyConfiguration, extractProxyCredentials } from './proxyUtils.js';
+
 puppeteer.use(stealthPlugin());
 
 let browser;
@@ -25,6 +27,10 @@ export default async function fetch(url, cssSelectors, config) {
 
     await client.send('Network.clearBrowserCookies'); // Clear cookies to ensure clean state between fetches and prevent session persistence across different URLs
 
+    if (browser.proxyCredentials?.username && browser.proxyCredentials?.password) {
+      await page.authenticate(browser.proxyCredentials);
+    }
+
     response = await page.goto(url, { waitUntil: 'load' }); // Using `load` instead of `networkidle0` as it's more reliable and faster. The 'load' event fires when the page and all its resources (stylesheets, scripts, images) have finished loading. `networkidle0` can be problematic as it waits for 500ms of network inactivity, which may never occur on dynamic pages and then triggers a navigation timeout.
 
     if (!response) {
@@ -86,7 +92,34 @@ export async function launchHeadlessBrowser() {
     return browser;
   }
 
-  browser = await puppeteer.launch({ headless: true });
+  const options = {
+    args: [],
+    headless: !process.env.OTA_ENGINE_FETCHER_NO_HEADLESS,
+  };
+
+  const { httpProxy, httpsProxy } = resolveProxyConfiguration();
+
+  let proxyCredentials = null;
+
+  if (httpProxy) {
+    const httpProxyUrl = new URL(httpProxy);
+    const httpsProxyUrl = new URL(httpsProxy);
+
+    proxyCredentials = extractProxyCredentials(httpProxy, httpsProxy);
+
+    options.args.push(`--proxy-server=http=${httpProxyUrl.host};https=${httpsProxyUrl.host}`);
+  }
+
+  if (process.env.OTA_ENGINE_FETCHER_NO_SANDBOX) {
+    options.args.push('--no-sandbox');
+    options.args.push('--disable-setuid-sandbox');
+  }
+
+  browser = await puppeteer.launch(options);
+
+  if (proxyCredentials) {
+    browser.proxyCredentials = proxyCredentials;
+  }
 
   return browser;
 }

src/archivist/fetcher/htmlOnlyFetcher.js

@@ -4,6 +4,8 @@ import HttpProxyAgent from 'http-proxy-agent';
 import HttpsProxyAgent from 'https-proxy-agent';
 import nodeFetch, { AbortError } from 'node-fetch';
 
+import { resolveProxyConfiguration } from './proxyUtils.js';
+
 export default async function fetch(url, config) {
   const controller = new AbortController();
   const timeout = setTimeout(() => controller.abort(), config.navigationTimeout);
@@ -14,10 +16,12 @@ export default async function fetch(url, config) {
     headers: { 'Accept-Language': config.language },
   };
 
-  if (url.startsWith('https:') && process.env.HTTPS_PROXY) {
-    nodeFetchOptions.agent = new HttpsProxyAgent(process.env.HTTPS_PROXY);
-  } else if (url.startsWith('http:') && process.env.HTTP_PROXY) {
-    nodeFetchOptions.agent = new HttpProxyAgent(process.env.HTTP_PROXY);
+  const { httpProxy, httpsProxy } = resolveProxyConfiguration();
+
+  if (url.startsWith('https:') && httpsProxy) {
+    nodeFetchOptions.agent = new HttpsProxyAgent(httpsProxy);
+  } else if (url.startsWith('http:') && httpProxy) {
+    nodeFetchOptions.agent = new HttpProxyAgent(httpProxy);
   }
 
   let response;

src/archivist/fetcher/proxyUtils.js

@@ -0,0 +1,30 @@
+export function resolveProxyConfiguration() {
+  const httpProxy = process.env.http_proxy || process.env.HTTP_PROXY;
+  const httpsProxy = process.env.https_proxy || process.env.HTTPS_PROXY || httpProxy;
+
+  return {
+    httpProxy,
+    httpsProxy,
+  };
+}
+
+export function extractProxyCredentials(httpProxy, httpsProxy) {
+  if (!httpProxy) {
+    return null;
+  }
+
+  const httpProxyUrl = new URL(httpProxy);
+  const httpsProxyUrl = new URL(httpsProxy);
+
+  const { username, password } = httpProxyUrl;
+
+  if (!username || !password) {
+    return null;
+  }
+
+  if (httpProxyUrl.username !== httpsProxyUrl.username || httpProxyUrl.password !== httpsProxyUrl.password) {
+    throw new Error('Unsupported proxies specified, http and https proxy should have the same credentials.');
+  }
+
+  return { username, password };
+}