Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@
implicit source auto-priority are intentionally removed.
- Sitemap ingestion source discovery (`SitemapSourceAdapter`) sets
browser-like request headers for `advertools.sitemaps.sitemap_to_df`
(`User-Agent`, `Accept`, `Accept-Language`, `Referer`,
(`User-Agent`, `Accept`, `Accept-Language`,
`Upgrade-Insecure-Requests`, `Sec-CH-*`) so sitemap fetch identity is closer
to Playwright/browser ingestion defaults.
- `wordlift_sdk.kg_build` callback patch preparation annotates first-level
Expand Down
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Changelog

## 8.3.8 - 2026-07-15

### Changed

- Removed the `Referer: https://wordlift.io` header from browser-like fetch
requests (`simple` / `playwright` loaders and sitemap discovery), so client
sites no longer see `wordlift.io` referral traffic in their analytics.

## 8.3.7 - 2026-07-13

### Fixed
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Current release: see `CHANGELOG.md`.

## Features
- URL sources: XML sitemaps, Google Sheets (`url` column), or Python lists, with global optional `URL_REGEX` filtering (also enforced in graph-sync source selection before `new_or_changed` GraphQL lookup).
- Sitemap discovery requests use a browser-like header bundle aligned with Playwright defaults (including `User-Agent`, `Accept`, `Accept-Language`, `Referer`, and `Sec-CH-*` headers).
- Sitemap discovery requests use a browser-like header bundle aligned with Playwright defaults (including `User-Agent`, `Accept`, `Accept-Language`, and `Sec-CH-*` headers).
- Change detection: skips URLs that are already imported unless `OVERWRITE` is enabled; re-imports when `lastmod` is newer.
- Web page imports: sends URLs to WordLift with embedding requests, output types, retry logic, and pluggable callbacks.
- Python 3.14 compatibility: retry filters use `pydantic_core.ValidationError` via the public API.
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "wordlift-sdk"
version = "8.3.7"
version = "8.3.8"
description = "Python toolkit for orchestrating WordLift imports and structured data workflows."
authors = ["David Riccitelli <david@wordlift.io>"]
readme = "README.md"
Expand Down
38 changes: 38 additions & 0 deletions tests/ingestion/test_loaders.py
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,44 @@ def _urlopen(request, timeout):
}


def test_simple_loader_sends_no_referer(monkeypatch: pytest.MonkeyPatch) -> None:
"""Regression: a real fetch must not carry a wordlift.io Referer.

The equality check above only proves the loader forwards the builder's
headers; it would still pass if the referer were re-added to the builder.
This asserts absence on the actual outgoing request.
"""
loader = SimpleLoaderAdapter()
observed: dict[str, str] = {}

class _Resp(BytesIO):
status = 200

def __init__(self) -> None:
super().__init__(b"<html>ok</html>")

def geturl(self) -> str:
return "https://example.com/final"

def __enter__(self):
return self

def __exit__(self, exc_type, exc, tb):
return False

def _urlopen(request, timeout):
observed.update(
{key.lower(): value for key, value in dict(request.header_items()).items()}
)
return _Resp()

monkeypatch.setattr("urllib.request.urlopen", _urlopen)
loader.load(SourceItem(id="1", url="https://example.com"), _config())

assert "referer" not in observed
assert "https://wordlift.io" not in observed.values()


def test_simple_loader_adapter_wraps_url_errors(
monkeypatch: pytest.MonkeyPatch,
) -> None:
Expand Down
1 change: 0 additions & 1 deletion wordlift_sdk/render/render_options.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
DEFAULT_BROWSER_REQUEST_HEADERS = {
"Accept": DEFAULT_ACCEPT_HEADER,
"Accept-Language": DEFAULT_ACCEPT_LANGUAGE_HEADER,
"Referer": "https://wordlift.io",
"Upgrade-Insecure-Requests": "1",
"Sec-CH-UA": '"Not A(Brand";v="99", "Chromium";v="120", "Google Chrome";v="120"',
"Sec-CH-UA-Mobile": "?0",
Expand Down
Loading