SERP Scraper

A powerful, async Python library for scraping Google and Bing Search Engine Results Pages (SERPs) with SeleniumBase UC Mode (Chromium), proxy rotation, stealth browsing, and intelligent caching.

Features

• SeleniumBase UC Mode: Stealth Chromium browser automation with hex-edited chromedriver (navigator.webdriver=false)
• Multi-layered Stealth: Driver-level, Chromium args, CDP overrides, WebRTC spoofing, JS fingerprint injection, and IP metadata sync
• Google News RSS: Scrape news articles via Google News RSS feeds (httpx-based)
• Google Scholar: Search academic papers from Google Scholar
• Proxy Rotation: DataImpulse, Decodo, and custom proxy support with automatic rotation
• Intelligent Caching: Disk-based caching with configurable TTL
• CAPTCHA Handling: Automatic detection with retry logic and exponential backoff
• Type Safety: Full type annotations with Pydantic validation, py.typed marker included
• Async/Await: Modern asynchronous API design
• YAML Configuration: config.yaml file support (see config.yaml.example)
• CLI Tool: Interactive command-line interface for testing
• Dual Output Format: Text (human-readable) and JSON (LLM-friendly) output
• REST API: Optional FastAPI-based REST API with rate limiting and authentication
• Content Compression: Built-in compress_content() utility for truncating long content — available in both the core library and the REST API

Installation

Basic Installation

pip install serp-scraper

From Source

git clone https://github.com/neuronaline/serp-scraper.git
cd serp-scraper
pip install -e .

With Optional Dependencies

pip install serp-scraper[dev]   # Dev tools (ruff, mypy, build, twine)
pip install serp-scraper[test]  # Test dependencies (pytest, pytest-asyncio, etc.)
pip install serp-scraper[api]   # REST API (FastAPI + uvicorn)

CLI Entry Point

After installation, the interactive CLI is available as a command:

serp-cli

Or run directly:

python main.py

Requirements

• Python 3.10 or higher
• Chromium browser — automatically managed by SeleniumBase (downloads chromedriver + browser on first run)
• Virtual display (for non-headless mode, the default):
  • Linux/headless servers: Install Xvfb (sudo apt install xvfb) and use DISPLAY=:99 or run with xvfb-run
  • macOS: No additional setup needed (has built-in display)
  • Windows: No additional setup needed (has built-in display)

Quick Start

Recommended: Using SerpClient

import asyncio
from serp import SerpClient

async def main():
    async with SerpClient() as client:
        results = await client.search("python programming")

        for r in results:
            print(f"{r.rank}. {r.title}")
            print(f"   {r.url}")
            print(f"   {r.description[:100]}...")

asyncio.run(main())

Running on Headless Servers

By default, the browser runs in non-headless mode (visible window) which requires a display. For headless servers or CI/CD environments, use one of these approaches:

Option 1: Use xvfb-run (recommended for Linux):

xvfb-run -a python your_script.py

Option 2: Set DISPLAY environment variable:

DISPLAY=:99 python your_script.py

Option 3: Run in headless mode:

async with SerpClient(headless=True) as client:
    results = await client.search("python programming")

Note: The VirtualScreenRequiredError exception is raised when running non-headless without a display.

Google News RSS

Scrape news articles using Google News RSS feeds:

import asyncio
from serp import GoogleNewsClient

async def main():
    async with GoogleNewsClient(language="en", country="US") as client:
        news = await client.get_news("Tesla", max_results=20)

        for r in news:
            print(f"{r.title}")
            print(f"  Source: {r.source}")
            print(f"  URL: {r.url}")
            print(f"  Date: {r.published}")

asyncio.run(main())

Or use the quick function:

from serp import quick_news

news = await quick_news("Tesla", language="en", country="US", max_results=20)

Google Scholar

Search academic papers from Google Scholar:

import asyncio
from serp import ScholarClient

async def main():
    async with ScholarClient() as client:
        papers = await client.search_scholar("machine learning", max_results=10)

        for p in papers:
            print(f"{p.title}")
            print(f"  Authors: {p.authors}")
            print(f"  URL: {p.url}")
            print(f"  Citations: {p.citations}")

asyncio.run(main())

Or use the quick function:

from serp import quick_scholar

papers = await quick_scholar("machine learning")

URL Fetching

import asyncio
from serp import SerpClient

async def main():
    async with SerpClient() as client:
        # Fetch page content as Markdown using SeleniumBase UC Mode browser
        content = await client.fetch("https://example.com")
        print(content)

asyncio.run(main())

Fetch Strategy:

• All URL fetches use SeleniumBase UC Mode browser for full JavaScript-rendered page content
• This ensures JavaScript-rendered content is never missed and incomplete data is never collected
• The browser waits for the load event plus additional time for JS rendering

Content Compression

Compress long content into head, middle, and tail portions — useful for LLM contexts where you want to retain key information while reducing token count:

from serp import compress_content, CompressionMeta

long_text = "..."  # e.g. 20,000+ characters
compressed, meta = compress_content(long_text)

if meta.was_truncated:
    print(f"Reduced from {meta.original_length:,} to {meta.compressed_length:,} chars")
    print(f"Truncated {meta.truncated_chars:,} characters")

Compression can also be enabled directly during fetch:

content = await client.fetch("https://example.com", compress=True)

The standalone compress_content() function gives you full control over thresholds and metadata, while the compress=True parameter on fetch() is a convenience shortcut.

Quick Functions

For simple use cases without creating a client:

import asyncio
from serp import quick_search, quick_fetch

async def main():
    # Search
    results = await quick_search("web scraping")
    print(f"Found {len(results)} results")

    # Fetch URL (with optional compression)
    content = await quick_fetch("https://example.com", compress=True)
    print(content[:500])

asyncio.run(main())

Using with Configuration

import asyncio
from serp import SerpClient, SerpConfig

# Create configured client directly
config = SerpConfig(
    log_level="DEBUG",
    max_retries=5,
    cache_ttl=3600,  # 1 hour
    cache_enabled=True,
)

async with SerpClient(config) as client:
    results = await client.search("python tutorial")

Or load from config.yaml:

config = SerpConfig.from_yaml()
# Or specify path:
config = SerpConfig.from_yaml("/path/to/config.yaml")

client = SerpClient(config)

YAML Configuration (config.yaml)

Create a config.yaml file in your project root. Copy from config.yaml.example for all options:

serp:
  log_level: INFO

  proxy:
    provider: dataimpulse
    dataimpulse:
      gateway: http://gw.dataimpulse.com:823
      user: myuser
      pass: mypass
      protocol: http
      country: us
    # decodo:
    #   user: myuser
    #   pass: mypass
    #   country: us
    # custom:
    #   - http://user:pass@proxy1.com:8080
    #   - socks5://proxy2.com:1080

  cache:
    enabled: true
    dir: .cache/serp
    ttl: 86400

  retry:
    max_retries: 3
    delay_min: 0.5
    delay_max: 2.0
    exponential_backoff: false

  search:
    source: auto    # "google", "bing", or "auto"
    headless: false
    timeout: 30
    human_behavior: true
    block_images: false

Configuration

SerpConfig Options

Parameter	Type	Default	Description
Proxy Settings
proxy_provider	str	"dataimpulse"	Proxy provider: "dataimpulse", "decodo", "custom", or "none"
dataimpulse_gateway	str	None	DataImpulse gateway URL
dataimpulse_user	str	None	DataImpulse username
dataimpulse_pass	str	None	DataImpulse password
dataimpulse_protocol	str	"http"	DataImpulse proxy protocol ("http" or "socks5")
dataimpulse_country	str	"us"	DataImpulse country code (e.g., "us", "de")
dataimpulse_sessid	str	None	DataImpulse session ID for sticky proxy
dataimpulse_sessttl	int	None	DataImpulse session TTL in minutes (1-120)
dataimpulse_nocr	str	None	Exclude countries (comma-separated ISO codes)
dataimpulse_noasn	str	None	Exclude ASNs (comma-separated)
dataimpulse_state	str	None	US state targeting (e.g., "arizona")
dataimpulse_city	str	None	City targeting (e.g., "berlin")
dataimpulse_zip	str	None	ZIP code targeting (e.g., "10001")
dataimpulse_asn	str	None	Specific ASN targeting
dataimpulse_anon	bool	false	High anonymity filter (elite proxies)
decodo_user	str	None	Decodo username
decodo_pass	str	None	Decodo password
decodo_country	str	None	ISO country code (e.g., "us", "de")
decodo_city	str	None	City name
decodo_state	str	None	State format "us_california"
decodo_session	str	None	Sticky session ID
decodo_sessionduration	int	None	Session TTL in minutes (1-1440)
decodo_protocol	str	"http"	Proxy protocol ("http" or "socks5")
custom_proxies	str	""	Comma-separated proxy URLs
Cache Settings
cache_enabled	bool	true	Enable/disable caching
cache_dir	str	".cache/serp"	Cache directory path
cache_ttl	int	86400	Cache TTL in seconds (min 60)
Retry Settings
max_retries	int	3	Maximum retry attempts (1-10)
retry_delay_min	float	5.0	Minimum retry delay in seconds
retry_delay_max	float	15.0	Maximum retry delay in seconds
exponential_backoff	bool	true	Use exponential backoff
Search Settings
default_source	str	"auto"	Default search source: "google", "bing", or "auto"
headless	bool	false	Run browser in headless mode
timeout	int	30	Request timeout in seconds (5-120)
user_agent	str	None	Custom user agent string (None = natural UA from SeleniumBase UC Mode)
human_behavior	bool	true	Simulate human browsing behavior
block_images	bool	false	Block image loading to save bandwidth
Logging
log_level	str	"WARNING"	Logging level (DEBUG, INFO, WARNING, ERROR)

API Reference

SerpClient

The recommended high-level interface for using the library.

from serp import SerpClient

client = SerpClient(
    headless=False,              # Run browser in headless mode
    use_cache=True,              # Enable/disable caching
    cache_ttl=86400,             # Cache TTL in seconds
    source=None,                 # "google", "bing", or None (auto)
    max_retries=3,               # Max retry attempts
    timeout=30,                  # Request timeout in seconds
    log_level="WARNING",         # Logging level
)

# Or with a SerpConfig instance:
config = SerpConfig(log_level="DEBUG", max_retries=5)
client = SerpClient(config)

SerpClient supports both async and sync context managers:

# Async (recommended)
async with SerpClient() as client:
    results = await client.search("query")

# Sync (backward compatibility)
with SerpClient() as client:
    results = client.search("query")  # still async, needs await

Methods

client.search(query, page_num=1, source=None, use_cache=None)

Search for a query and return results.

Note: Google and Bing SERP searches always use SeleniumBase UC Mode browser because search engines require JavaScript to render results.

Parameters:

• query (str): Search query string
• page_num (int): Page number (1-based), defaults to 1
• source (str): Search engine — "google", "bing", or None (auto: google first, bing fallback)
• use_cache (bool): Whether to use cache. None uses client default.

Returns:

• list[SearchResult]: List of SearchResult objects

Raises:

• ProxyError: All proxies failed
• CaptchaError: CAPTCHA detected after all retries
• PageTimeoutError: Page load timeout
• ParseError: Failed to parse results

---

client.fetch(url, use_cache=None, prefer_browser=True, compress=False)

Fetch a URL and return content as Markdown.

Fetch Strategy:

• All fetches use SeleniumBase UC Mode browser for full JavaScript-rendered page content
• Browser waits for load event plus additional time for JS rendering

Parameters:

• url (str): Target URL
• use_cache (bool): Whether to use cache. None uses client default.
• prefer_browser (bool): Deprecated, ignored (browser is always used).
• compress (bool): If True and content exceeds ~10K chars, compress by taking head, middle, and tail portions. (Default: False)

Returns:

• str: Page content converted to Markdown (optionally compressed)

---

SearchResult

Typed result object returned by search operations.

from serp import SearchResult

result = SearchResult(
    rank=1,
    title="Example Title",
    url="https://example.com",
    description="Example description...",
    source="google"  # or "bing"
)

Attributes:

• rank (int): Position in search results (1-based)
• title (str): Result title
• url (str): Target URL
• description (str): Result snippet/description
• source (str): Search engine source ("google" or "bing")

Methods:

• to_dict(): Convert to dictionary

---

GoogleSERPPage / BingSERPPage

Rich parsed SERP page objects with metadata:

from serp import SerpClient

async with SerpClient() as client:
    page = await client.search_with_metadata("python tutorial")
    # page.results   — list of organic results
    # page.stats     — SearchStats (result count, search time)
    # page.pagination_has_next — bool
    # page.pagination_next_url — str

Additional rich types (available from serp):

• SearchStats — Google search statistics (result count, search time)
• PeopleAlsoAskItem — People Also Ask question/answer pair
• VideoResultItem — Video carousel result
• KnowledgeGraphInfo — Knowledge Graph panel data
• BingSearchStats — Bing search statistics
• BingRelatedSearchItem — Bing related search suggestion

---

Quick Functions

Module-level convenience functions using a default client:

from serp import quick_search, quick_fetch

# Quick search
results = await quick_search("query")

# Fetch URL (with optional compression)
content = await quick_fetch("https://example.com", compress=True)

---

GoogleNewsClient

Client for scraping Google News via RSS feeds.

from serp import GoogleNewsClient

client = GoogleNewsClient(
    language="tr",        # Language code (tr, en, etc.)
    country="TR",        # Country code (TR, US, etc.)
    time_range="d",      # Time range: "h" (hour), "d" (day), "w" (week), "m" (month)
)

Methods

client.get_news(query, max_results=50, queries=None)

Get news articles for a search term.

Parameters:

• query (str): Search term to find news for
• max_results (int): Maximum number of results (default: 50)
• queries (list[str]): Custom list of queries to use

Returns:

• list[NewsResult]: List of NewsResult objects

quick_news(query, max_results=50, language="tr", country="TR")

Convenience function for quick news retrieval.

from serp import quick_news

news = await quick_news("Tesla", max_results=20, language="en", country="US")

---

NewsResult

Typed result object for Google News articles.

from serp import NewsResult

result = NewsResult(
    title="Tesla announces new model",
    url="https://news.google.com/rss/articles/...",
    original_url="https://example.com/article",
    published=datetime(2026, 5, 11, 8, 0, 0),
    source="BBC",
    description="Tesla unveiled...",
    query="Tesla"
)

Attributes:

• title (str): News headline
• url (str): Google News RSS URL
• original_url (str): Original article URL (extracted from description)
• published (datetime): Publication date
• source (str): News source name (e.g., "BBC", "NTV")
• description (str): News summary/snippet
• query (str): Search query that returned this result

---

ScholarClient

Client for searching Google Scholar academic papers.

from serp import ScholarClient

client = ScholarClient()
papers = await client.search_scholar("machine learning", max_results=10)

Methods:

• search_scholar(query, max_results=10, page_num=1) — Search for academic papers
• quick_scholar(query, max_results=10) — Convenience function

Returns: list[ScholarResult] with attributes:

• title (str): Paper title
• authors (str): Author names
• url (str): Paper URL
• snippet (str): Abstract/excerpt
• citations (int): Citation count
• year (str): Publication year

---

OutputFormatter

Structured output formatting for CLI tools and LLM integration.

from serp import OutputFormatter, OUTPUT_TEXT, OUTPUT_JSON

# Format search results
output = OutputFormatter.format_search_results(
    results=search_results,
    mode=OUTPUT_JSON,
    query="python tutorial",
    source="google",
)

# Format news results
output = OutputFormatter.format_news(
    news_list=news_results,
    mode=OUTPUT_JSON,
    search_term="Tesla",
    language="en",
    country="US",
)

# Format fetch results
output = OutputFormatter.format_fetch(
    content=page_content,
    url="https://example.com",
    char_count=len(page_content),
    mode=OUTPUT_JSON,
)

Constants:

• OUTPUT_TEXT: Text output mode (human-readable)
• OUTPUT_JSON: JSON output mode (machine-parseable)

---

Utility Functions

set_log_level(level)

Set the log level for all serp loggers.

from serp import set_log_level

set_log_level("DEBUG")   # Enable debug logging
set_log_level("WARNING") # Only show warnings and errors

---

compress_content(content, threshold=10000, head_pct=0.35, middle_pct=0.15, tail_pct=0.50)

Compress long content by extracting head, middle, and tail portions, joined with a truncation marker.

from serp import compress_content, CompressionMeta

compressed, meta = compress_content(
    content,
    threshold=10000,      # Content longer than this gets compressed
    head_pct=0.35,        # 35% of target for the head
    middle_pct=0.15,      # 15% of target for the middle
    tail_pct=0.50,        # 50% of target for the tail
)

if meta.was_truncated:
    print(f"Original: {meta.original_length:,} chars")
    print(f"Compressed: {meta.compressed_length:,} chars")
    print(f"Truncated: {meta.truncated_chars:,} chars")

Parameters:

Param	Type	Default	Description
content	str	-	The content string to compress
threshold	int	10000	Char length threshold. Content shorter than this is unchanged.
head_pct	float	0.35	Fraction of target length for the head portion
middle_pct	float	0.15	Fraction of target length for the middle portion
tail_pct	float	0.50	Fraction of target length for the tail portion

Returns: tuple[str, CompressionMeta] — (compressed_content, metadata)

CompressionMeta attributes:

Attribute	Type	Description
original_length	int	Character count before compression
compressed_length	int	Character count after compression
truncated_chars	int	Number of characters removed
was_truncated	bool	Whether content exceeded threshold and was truncated

---

Exceptions

Exception	Description
ProxyError	All configured proxies failed or returned errors
CaptchaError	Search engine detected automation and presented CAPTCHA
PageTimeoutError	Page load timeout
ParseError	Page loaded but results could not be parsed
BrowserCrashError	Browser process crashed unexpectedly
VirtualScreenRequiredError	No virtual display available for non-headless mode

---

Constants

Constant	Description
MAX_RETRIES	Maximum retry attempts (default: 3)
TIMEOUT_MS	Page timeout in milliseconds (default: 30000)

---

Interactive CLI

The package includes an interactive CLI tool for testing:

# Via entry point (after pip install)
serp-cli

# Or run directly
python main.py
python main.py --format json   # JSON output for LLM integration
python main.py -f text         # Human-readable text output (default)
python main.py --compress      # Enable content compression for URL fetch
python main.py --compress -f json  # Compress + JSON output

Features:

• SERP Search testing (Google & Bing)
• URL Fetch testing (with optional --compress flag)
• Google News RSS testing
• Google Scholar testing
• Dual output format (text/JSON)
• Proxy status checking

REST API (Optional)

The package includes an optional FastAPI-based REST API for programmatic access.

Installation

pip install serp-scraper[api]

Running the API

python -m api.main
# Or with uvicorn directly:
uvicorn api.main:app --host 0.0.0.0 --port 8000

API Endpoints

Method	Path	Description
GET	/health	Health check
POST	/api/v1/search	Search SERP results
POST	/api/v1/fetch	Fetch URL content (supports compress field)
POST	/api/v1/news	Get Google News articles
POST	/api/v1/scholar	Search Google Scholar articles

Authentication

The API uses API key authentication. Set the X-API-Key header in your requests.

Rate Limiting

The API includes rate limiting middleware to prevent abuse.

API Environment Variables

Variable	Description	Default
API_HOST	API server host	0.0.0.0
API_PORT	API server port	8000
API_DEBUG	API debug mode	false
API_MAX_CONCURRENT_REQUESTS	Max concurrent requests (pool size)	15
API_REQUEST_TIMEOUT	Request timeout in seconds	60
API_RATE_LIMIT_SEARCH	Rate limit for /search (req/min)	30
API_RATE_LIMIT_FETCH	Rate limit for /fetch (req/min)	60
API_RATE_LIMIT_NEWS	Rate limit for /news (req/min)	30
API_RATE_LIMIT_SCHOLAR	Rate limit for /scholar (req/min)	30
API_RATE_LIMIT_DEFAULT	Default rate limit (req/min)	100
API_KEYS_HASHED	Comma-separated hashed API keys	-
API_ALLOW_NO_AUTH	Allow unauthenticated access	false
API_LOG_LEVEL	API logging level	INFO
API_LOG_DIR	API log directory	logs
API_LOG_RETENTION_DAYS	Log retention days	7
API_CORS_ORIGINS	CORS allowed origins (comma-separated)	-

Project Structure

serp-scraper/
├── serp/                        # Core library
│   ├── __init__.py              # Public API exports, __version__
│   ├── client.py                # SerpClient and quick functions
│   ├── sb_browser.py            # SeleniumBase UC Mode browser lifecycle
│   ├── ip_metadata.py           # IP metadata sync (ipinfo.io → locale/currency)
│   ├── browser_stealth.py       # Chromium/CDP stealth JS + human behavior
│   ├── parsers.py               # SERP page parsing (Google, Bing)
│   ├── cleaning.py              # Regex-based HTML/Markdown cleaning
│   ├── cache.py                 # Disk-based caching (DiskCache, NullCache)
│   ├── compression.py           # Content compression (compress_content, CompressionMeta)
│   ├── config.py                # Configuration constants (BING_URL_TEMPLATE)
│   ├── config_pydantic.py       # Pydantic-based SerpConfig (from_yaml)
│   ├── proxy_providers.py       # DataImpulse, Decodo, Custom proxy providers
│   ├── google_news.py           # Google News RSS client
│   ├── google_scholar.py        # Google Scholar client
│   ├── output_formatter.py      # Text and JSON output formatting
│   ├── types.py                 # Type definitions (SearchResult, RetryPolicy, etc.)
│   ├── utils.py                 # Exceptions, helpers, constants
│   └── py.typed                 # PEP 561 typing marker
├── api/                         # REST API (optional)
│   ├── __init__.py
│   ├── main.py                  # FastAPI application
│   ├── config.py                # API configuration (APISettings, RateLimitConfig)
│   ├── deps.py                  # DI: auth, rate limit, semaphore
│   ├── exceptions.py            # Custom exception classes
│   ├── models/                  # Pydantic models
│   │   ├── __init__.py
│   │   ├── requests.py          # Request schemas
│   │   └── responses.py         # Response schemas
│   ├── routers/                 # API routes
│   │   ├── __init__.py
│   │   ├── health.py            # GET /health
│   │   ├── search.py            # POST /api/v1/search
│   │   ├── fetch.py             # POST /api/v1/fetch
│   │   ├── news.py              # POST /api/v1/news
│   │   └── scholar.py           # POST /api/v1/scholar
│   ├── middleware/              # Middleware
│   │   ├── __init__.py
│   │   ├── rate_limit.py        # Sliding window rate limiter
│   │   └── logging_middleware.py
│   ├── utils/                   # Backward-compat re-exports
│   │   ├── __init__.py
│   │   └── compression.py       # Re-exports from serp.compression
│   └── cli/                     # API CLI tools
│       ├── __init__.py
│       └── keys.py              # API key generation
├── tests/                       # Test suite
│   ├── __init__.py
│   ├── conftest.py              # Fixtures, global state reset
│   ├── test_serp.py             # Core library tests
│   ├── test_cache.py            # Cache tests
│   ├── test_google_news.py      # Google News tests
│   └── api/                     # API tests
│       └── __init__.py
├── main.py                      # Interactive CLI tool
├── config.yaml                  # Active configuration
├── config.yaml.example          # Configuration template
├── pyproject.toml               # Project metadata & build config
├── requirements.txt             # Pinned dependencies
└── README.md                    # This file

Architecture

Search Flow

1. Check cache for existing results (if use_cache=True)
2. Load proxy configuration from config.yaml or programmatic settings
3. Select proxy via provider abstraction (DataImpulse, Decodo, or Custom)
4. Create SeleniumBase UC Mode browser instance with stealth settings
5. Apply CDP overrides (timezone, geolocation, locale, hardware concurrency)
6. Inject stealth JS (WebRTC sanitization, fingerprint spoof, Intl overrides)
7. Navigate to search URL with human-like timing
8. Wait for results to load
9. Check for CAPTCHA
10. Parse organic results (Google/Bing specific parsers)
11. Cache results before returning

Browser-Only Fetch Strategy

All fetching uses SeleniumBase UC Mode (Chromium) browser exclusively:

• SERP Search (Google/Bing): Browser required — JavaScript renders results
• URL Fetch: Browser ensures JavaScript-rendered content is captured
• Google Scholar: Browser-based scraping with JS evaluation

This ensures complete, accurate data collection and avoids incomplete results from HTTP-only approaches.

Multi-layered Stealth

Layer	Technology	Description
Driver	SeleniumBase UC Mode	Hex-edited chromedriver, navigator.webdriver=false
Chromium args	--disable-blink-features=AutomationControlled	Various anti-detection flags
CDP	Emulation.setTimezoneOverride, setGeolocationOverride	Emulation overrides
WebRTC	JSON prefs injection	disable_non_proxied_udp
JS	WebRTC ICE sanitization, fingerprint spoof	Runtime fingerprint masking
Monkey-patch	sb.get override	Auto-inject CDP + JS on navigation
IP metadata	ipinfo.io API	Country → locale/currency/timezone derivation

Asyncio Bridge

SeleniumBase is synchronous. All WebDriver calls run via asyncio.to_thread():

await asyncio.to_thread(sb.get, url)
await asyncio.to_thread(sb.execute_script, script)
await asyncio.to_thread(lambda: sb.get_page_source())

Caching

The caching system uses a disk-based approach:

• Cache entries stored as JSON files in .cache/serp/
• Keys are SHA256 hashes of query parameters
• Automatic expiration based on TTL
• Can be disabled via cache_enabled=False

Content Compression

The library includes a built-in content compression utility for truncating long content while preserving key information:

Algorithm:

1. If content length ≤ threshold (default: 10,000 chars), return unchanged
2. Target compressed length = 45% of threshold (~4,500 chars)
3. Extract three portions: head (35%), middle (15%), tail (50%)
4. Join with a truncation marker: \n\n-- X,XXX chars truncated --\n\n

Testing

Run the test suite:

pytest

Run with coverage:

pytest --cov=serp --cov-report=html

Run specific test file:

pytest tests/test_serp.py

Dependencies

Package	Version	Purpose
seleniumbase	>=4.30.0	Stealth Chromium browser automation (UC Mode)
markdownify	>=0.12.0	HTML to Markdown conversion
httpx	>=0.25.0	Async HTTP client (Google News RSS)
pyyaml	>=6.0	YAML config file parsing
pydantic	>=2.0.0	Configuration validation
pydantic-settings	>=2.0.0	Pydantic settings integration

API Dependencies (Optional)

Install with pip install serp-scraper[api]:

Package	Version	Purpose
fastapi	>=0.100.0	REST API framework
uvicorn[standard]	>=0.23.0	ASGI server
passlib[bcrypt]	>=1.7.4	Password hashing
bcrypt	<4.1.0	Password hashing

Development Dependencies

Package	Version	Purpose
pytest	>=7.0.0	Testing framework
pytest-asyncio	>=0.21.0	Async test support
pytest-cov	>=4.0.0	Coverage reporting
pytest-mock	>=3.10.0	Mocking utilities
pytest-httpserver	>=1.0.0	HTTP server for testing
ruff	>=0.1.0	Linting
mypy	>=1.0.0	Type checking
build	>=1.0.0	Package building
twine	>=4.0.0	Package publishing

License

MIT License - see LICENSE file for details.

Author

neuronaline - flashneuron@proton.me

Links

• Repository
• Issue Tracker
• Changelog

Disclaimer

This software is provided for educational and legitimate purposes only. Users are responsible for ensuring their use complies with search engine Terms of Service and applicable laws. The authors assume no liability for misuse of this software.