๐ Key Opportunities
+Key Opportunities
- **Data Capitalization**: Transform digital footprints into measurable, valuable assets. - **Authentic AI Data**: Provide AI systems with real human insights. diff --git a/README.md b/README.md index 733678be3..fb22e087a 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ #### ๐ Crawl4AI Cloud API โ Closed Beta (Launching Soon) Reliable, large-scale web extraction, now built to be _**drastically more cost-effective**_ than any of the existing solutions. -๐ **Apply [here](https://forms.gle/E9MyPaNXACnAMaqG7) for early access** + **Apply [here](https://forms.gle/E9MyPaNXACnAMaqG7) for early access** _Weโll be onboarding in phases and working closely with early users. Limited slots._ @@ -37,20 +37,20 @@ Limited slots._ Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community. -[โจ Check out latest update v0.9](#-recent-updates) +[ Check out latest update v0.9](#-recent-updates) -โจ **New in v0.9**: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Breaking changes for the self-hosted server only; the pip library is unchanged. If you self-host the Docker API, read the [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) before upgrading. [Release notes โ](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.0.md) + **New in v0.9**: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Breaking changes for the self-hosted server only; the pip library is unchanged. If you self-host the Docker API, read the [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) before upgrading. [Release notes โ](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.0.md) -โจ Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities (RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret), adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. [Release notes โ](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.7.md) + Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities (RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret), adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. [Release notes โ](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.7.md) -โจ Recent v0.8.6: Security hotfix that replaced `litellm` with `unclecode-litellm` due to a PyPI supply chain compromise. + Recent v0.8.6: Security hotfix that replaced `litellm` with `unclecode-litellm` due to a PyPI supply chain compromise. -โจ Previous v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with `resume_state` and `on_state_change` callbacks for long-running crawls. New `prefetch=True` mode for 5-10x faster URL discovery. [Release notes โ](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.0.md) + Previous v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with `resume_state` and `on_state_change` callbacks for long-running crawls. New `prefetch=True` mode for 5-10x faster URL discovery. [Release notes โ](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.0.md) -โจ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. [Release notes โ](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.8.md) + Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. [Release notes โ](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.8.md)
-
๐ค My Personal Story
+My Personal Story
I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. Thatโs where I learned how much extraction matters. @@ -121,9 +121,9 @@ crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10 crwl https://www.example.com/products -q "Extract all product prices" ``` -## ๐ Support Crawl4AI +## Support Crawl4AI -> ๐ **Sponsorship Program Now Open!** After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for **startups** and **enterprises**. Be among the first 50 **Founding Sponsors** for permanent recognition in our Hall of Fame. +> **Sponsorship Program Now Open!** After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for **startups** and **enterprises**. Be among the first 50 **Founding Sponsors** for permanent recognition in our Hall of Fame. Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community โ while giving you direct access to premium benefits. @@ -134,12 +134,12 @@ Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keep -### ๐ค Sponsorship Tiers +### Sponsorship Tiers -- **๐ฑ Believer ($5/mo)** โ Join the movement for data democratization +- ** Believer ($5/mo)** โ Join the movement for data democratization - **๐ Builder ($50/mo)** โ Priority support & early access to features -- **๐ผ Growing Team ($500/mo)** โ Bi-weekly syncs & optimization help -- **๐ข Data Infrastructure Partner ($2000/mo)** โ Full partnership with dedicated support +- ** Growing Team ($500/mo)** โ Bi-weekly syncs & optimization help +- ** Data Infrastructure Partner ($2000/mo)** โ Full partnership with dedicated support *Custom arrangements available - see [SPONSORS.md](SPONSORS.md) for details & contact* **Why sponsor?** @@ -148,96 +148,96 @@ No rate-limited APIs. No lock-in. Build and own your data pipeline with direct g [See All Tiers & Benefits โ](https://github.com/sponsors/unclecode) -## โจ Features +## Features
-
๐ Markdown Generation
+Markdown Generation
-- ๐งน **Clean Markdown**: Generates clean, structured Markdown with accurate formatting. -- ๐ฏ **Fit Markdown**: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing. +- **Clean Markdown**: Generates clean, structured Markdown with accurate formatting. +- **Fit Markdown**: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing. - ๐ **Citations and References**: Converts page links into a numbered reference list with clean citations. -- ๐ ๏ธ **Custom Strategies**: Users can create their own Markdown generation strategies tailored to specific needs. -- ๐ **BM25 Algorithm**: Employs BM25-based filtering for extracting core information and removing irrelevant content. +- **Custom Strategies**: Users can create their own Markdown generation strategies tailored to specific needs. +- **BM25 Algorithm**: Employs BM25-based filtering for extracting core information and removing irrelevant content.
-
๐ Structured Data Extraction
+Structured Data Extraction
- ๐ค **LLM-Driven Extraction**: Supports all LLMs (open-source and proprietary) for structured data extraction. -- ๐งฑ **Chunking Strategies**: Implements chunking (topic-based, regex, sentence-level) for targeted content processing. -- ๐ **Cosine Similarity**: Find relevant content chunks based on user queries for semantic extraction. -- ๐ **CSS-Based Extraction**: Fast schema-based data extraction using XPath and CSS selectors. -- ๐ง **Schema Definition**: Define custom schemas for extracting structured JSON from repetitive patterns. +- **Chunking Strategies**: Implements chunking (topic-based, regex, sentence-level) for targeted content processing. +- **Cosine Similarity**: Find relevant content chunks based on user queries for semantic extraction. +- **CSS-Based Extraction**: Fast schema-based data extraction using XPath and CSS selectors. +- **Schema Definition**: Define custom schemas for extracting structured JSON from repetitive patterns.
-
๐ Browser Integration
+Browser Integration
-- ๐ฅ๏ธ **Managed Browser**: Use user-owned browsers with full control, avoiding bot detection. -- ๐ **Remote Browser Control**: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction. -- ๐ค **Browser Profiler**: Create and manage persistent profiles with saved authentication states, cookies, and settings. +- **Managed Browser**: Use user-owned browsers with full control, avoiding bot detection. +- **Remote Browser Control**: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction. +- **Browser Profiler**: Create and manage persistent profiles with saved authentication states, cookies, and settings. - ๐ **Session Management**: Preserve browser states and reuse them for multi-step crawling. -- ๐งฉ **Proxy Support**: Seamlessly connect to proxies with authentication for secure access. -- โ๏ธ **Full Browser Control**: Modify headers, cookies, user agents, and more for tailored crawling setups. -- ๐ **Multi-Browser Support**: Compatible with Chromium, Firefox, and WebKit. -- ๐ **Dynamic Viewport Adjustment**: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements. +- **Proxy Support**: Seamlessly connect to proxies with authentication for secure access. +- **Full Browser Control**: Modify headers, cookies, user agents, and more for tailored crawling setups. +- **Multi-Browser Support**: Compatible with Chromium, Firefox, and WebKit. +- **Dynamic Viewport Adjustment**: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.
-
๐ Crawling & Scraping
+Crawling & Scraping
-- ๐ผ๏ธ **Media Support**: Extract images, audio, videos, and responsive image formats like `srcset` and `picture`. +- **Media Support**: Extract images, audio, videos, and responsive image formats like `srcset` and `picture`. - ๐ **Dynamic Crawling**: Execute JS and wait for async or sync for dynamic content extraction. -- ๐ธ **Screenshots**: Capture page screenshots during crawling for debugging or analysis. -- ๐ **Raw Data Crawling**: Directly process raw HTML (`raw:`) or local files (`file://`). +- **Screenshots**: Capture page screenshots during crawling for debugging or analysis. +- **Raw Data Crawling**: Directly process raw HTML (`raw:`) or local files (`file://`). - ๐ **Comprehensive Link Extraction**: Extracts internal, external links, and embedded iframe content. -- ๐ ๏ธ **Customizable Hooks**: Define hooks at every step to customize crawling behavior (supports both string and function-based APIs). -- ๐พ **Caching**: Cache data for improved speed and to avoid redundant fetches. -- ๐ **Metadata Extraction**: Retrieve structured metadata from web pages. -- ๐ก **IFrame Content Extraction**: Seamless extraction from embedded iframe content. -- ๐ต๏ธ **Lazy Load Handling**: Waits for images to fully load, ensuring no content is missed due to lazy loading. -- ๐ **Full-Page Scanning**: Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages. +- **Customizable Hooks**: Define hooks at every step to customize crawling behavior (supports both string and function-based APIs). +- **Caching**: Cache data for improved speed and to avoid redundant fetches. +- **Metadata Extraction**: Retrieve structured metadata from web pages. +- **IFrame Content Extraction**: Seamless extraction from embedded iframe content. +- **Lazy Load Handling**: Waits for images to fully load, ensuring no content is missed due to lazy loading. +- **Full-Page Scanning**: Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.๐ Deployment
-- ๐ณ **Dockerized Setup**: Optimized Docker image with FastAPI server for easy deployment. -- ๐ **Secure Authentication**: Built-in JWT token authentication for API security. -- ๐ **API Gateway**: One-click deployment with secure token authentication for API-based workflows. -- ๐ **Scalable Architecture**: Designed for mass-scale production and optimized server performance. -- โ๏ธ **Cloud Deployment**: Ready-to-deploy configurations for major cloud platforms. +- **Dockerized Setup**: Optimized Docker image with FastAPI server for easy deployment. +- **Secure Authentication**: Built-in JWT token authentication for API security. +- **API Gateway**: One-click deployment with secure token authentication for API-based workflows. +- **Scalable Architecture**: Designed for mass-scale production and optimized server performance. +- **Cloud Deployment**: Ready-to-deploy configurations for major cloud platforms.
-
## Try it Now!
-โจ Play around with this [](https://colab.research.google.com/drive/1SgRPrByQLzjRfwoRNq1wSGE9nYY_EE8C?usp=sharing)
+ Play around with this [](https://colab.research.google.com/drive/1SgRPrByQLzjRfwoRNq1wSGE9nYY_EE8C?usp=sharing)
-โจ Visit our [Documentation Website](https://docs.crawl4ai.com/)
+ Visit our [Documentation Website](https://docs.crawl4ai.com/)
-## Installation ๐ ๏ธ
+## Installation
Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.
๐ฏ Additional Features
+Additional Features
-- ๐ถ๏ธ **Stealth Mode**: Avoid bot detection by mimicking real users. -- ๐ท๏ธ **Tag-Based Content Extraction**: Refine crawling based on custom tags, headers, or metadata. +- **Stealth Mode**: Avoid bot detection by mimicking real users. +- **Tag-Based Content Extraction**: Refine crawling based on custom tags, headers, or metadata. - ๐ **Link Analysis**: Extract and analyze all links for detailed data exploration. - ๐ก๏ธ **Error Handling**: Robust error management for seamless execution. -- ๐ **CORS & Static Serving**: Supports filesystem-based caching and cross-origin requests. -- ๐ **Clear Documentation**: Simplified and updated guides for onboarding and advanced usage. -- ๐ **Community Recognition**: Acknowledges contributors and pull requests for transparency. +- **CORS & Static Serving**: Supports filesystem-based caching and cross-origin requests. +- **Clear Documentation**: Simplified and updated guides for onboarding and advanced usage. +- **Community Recognition**: Acknowledges contributors and pull requests for transparency.
-
๐ Using pip
+Using pip
Choose the installation option that best fits your needs: @@ -252,7 +252,7 @@ crawl4ai-setup # Setup the browser By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling. -๐ **Note**: When you install Crawl4AI, the `crawl4ai-setup` should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods: + **Note**: When you install Crawl4AI, the `crawl4ai-setup` should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods: 1. Through the command line: @@ -303,7 +303,7 @@ pip install -e ".[all]" # Install all optional features
-
๐ณ Docker Deployment
+Docker Deployment
> ๐ **Now Available!** Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever. @@ -361,12 +361,12 @@ For more examples, see our [Docker Examples](https://github.com/unclecode/crawl4 --- -## ๐ฌ Advanced Usage Examples ๐ฌ +## Advanced Usage Examples You can check the project structure in the directory [docs/examples](https://github.com/unclecode/crawl4ai/tree/main/docs/examples). Over there, you can find a variety of examples; here, some popular examples are shared.
-
๐ Heuristic Markdown Generation with Clean and Fit Markdown
+Heuristic Markdown Generation with Clean and Fit Markdown
```python import asyncio @@ -404,7 +404,7 @@ if __name__ == "__main__":
-
๐ฅ๏ธ Executing JavaScript & Extract Structured Data without LLMs
+Executing JavaScript & Extract Structured Data without LLMs
```python import asyncio @@ -477,7 +477,7 @@ if __name__ == "__main__":
-
| Leveraging Thordata ensures seamless compatibility with any AI/ML workflows and data infrastructure, massively accessing web data with 99.9% uptime, backed by one-on-one customer support. | ๐ฅ Silver |
-| 
| Helps engineers and buyers find, compare, and source electronic & industrial parts in seconds, with specs, pricing, lead times & alternatives.| ๐ฅ Gold |
-| 
| Singapore-based Aleph Null is Asiaโs leading edtech hub, dedicated to student-centric, AI-driven educationโempowering learners with the tools to thrive in a fast-changing world. | ๐ฅ Gold |
+| 
| Leveraging Thordata ensures seamless compatibility with any AI/ML workflows and data infrastructure, massively accessing web data with 99.9% uptime, backed by one-on-one customer support. | Silver |
+| 
| NstProxy is a trusted proxy provider with over 110M+ real residential IPs, city-level targeting, 99.99% uptime, and low pricing at $0.1/GB, it delivers unmatched stability, scale, and cost-efficiency. | Silver |
+|
๐ Extracting Structured Data with LLMs
+Extracting Structured Data with LLMs
```python import os @@ -562,9 +562,9 @@ async def test_news_crawl(): --- -> **๐ก Tip:** Some websites may use **CAPTCHA** based verification mechanisms to prevent automated access. If your workflow encounters such challenges, you may optionally integrate a third-party CAPTCHA-handling service such as [CapSolver](https://www.capsolver.com/blog/Partners/crawl4ai-capsolver/?utm_source=crawl4ai&utm_medium=github_pr&utm_campaign=crawl4ai_integration). They support reCAPTCHA v2/v3, Cloudflare Turnstile, Challenge, AWS WAF, and more. Please ensure that your usage complies with the target websiteโs terms of service and applicable laws. +> ** Tip:** Some websites may use **CAPTCHA** based verification mechanisms to prevent automated access. If your workflow encounters such challenges, you may optionally integrate a third-party CAPTCHA-handling service such as [CapSolver](https://www.capsolver.com/blog/Partners/crawl4ai-capsolver/?utm_source=crawl4ai&utm_medium=github_pr&utm_campaign=crawl4ai_integration). They support reCAPTCHA v2/v3, Cloudflare Turnstile, Challenge, AWS WAF, and more. Please ensure that your usage complies with the target websiteโs terms of service and applicable laws. -## โจ Recent Updates +## Recent UpdatesVersion 0.9.0 Release Highlights - Secure-by-Default Docker Server
@@ -624,17 +624,17 @@ Our biggest release since v0.8.0. Anti-bot detection with proxy escalation, Shad ) ``` -- **๐ Shadow DOM Flattening**: +- ** Shadow DOM Flattening**: - Extract content hidden inside shadow DOM components ```python config = CrawlerRunConfig(flatten_shadow_dom=True) ``` -- **๐ Deep Crawl Cancellation**: +- ** Deep Crawl Cancellation**: - Stop long crawls gracefully with `cancel()` or `should_cancel` callback - Works with BFS, DFS, and BestFirst strategies -- **โ๏ธ Config Defaults API**: +- ** Config Defaults API**: - `set_defaults()` / `get_defaults()` / `reset_defaults()` on BrowserConfig and CrawlerRunConfig - **๐ Critical Security Fixes**: @@ -652,7 +652,7 @@ Our biggest release since v0.8.0. Anti-bot detection with proxy escalation, Shad This release introduces crash recovery for deep crawls, a new prefetch mode for fast URL discovery, and critical security fixes for Docker deployments. -- **๐ Deep Crawl Crash Recovery**: +- ** Deep Crawl Crash Recovery**: - `on_state_change` callback fires after each URL for real-time state persistence - `resume_state` parameter to continue from a saved checkpoint - JSON-serializable state for Redis/database storage @@ -667,7 +667,7 @@ This release introduces crash recovery for deep crawls, a new prefetch mode for ) ``` -- **โก Prefetch Mode for Fast URL Discovery**: +- ** Prefetch Mode for Fast URL Discovery**: - `prefetch=True` skips markdown, extraction, and media processing - 5-10x faster than full processing - Perfect for two-phase crawling: discover first, process selectively @@ -691,7 +691,7 @@ This release introduces crash recovery for deep crawls, a new prefetch mode for This release focuses on stability with 11 bug fixes addressing issues reported by the community. No new features, but significant improvements to reliability. -- **๐ณ Docker API Fixes**: +- ** Docker API Fixes**: - Fixed `ContentRelevanceFilter` deserialization in deep crawl requests (#1642) - Fixed `ProxyConfig` JSON serialization in `BrowserConfig.to_dict()` (#1629) - Fixed `.cache` folder permissions in Docker image (#1638) @@ -724,11 +724,11 @@ This release focuses on stability with 11 bug fixes addressing issues reported b - Fixed relative URL resolution after JavaScript redirects (#1268) - Fixed import statement formatting in extracted code (#1181) -- **๐ฆ Dependency Updates**: +- ** Dependency Updates**: - Replaced deprecated PyPDF2 with pypdf (#1412) - Pydantic v2 ConfigDict compatibility - no more deprecation warnings (#678) -- **๐ง AdaptiveCrawler**: +- ** AdaptiveCrawler**: - Fixed query expansion to actually use LLM instead of hardcoded mock data (#1621) [Full v0.7.8 Release Notes โ](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.8.md) @@ -738,7 +738,7 @@ This release focuses on stability with 11 bug fixes addressing issues reported bVersion 0.7.7 Release Highlights - The Self-Hosting & Monitoring Update
-- **๐ Real-time Monitoring Dashboard**: Interactive web UI with live system metrics and browser pool visibility +- ** Real-time Monitoring Dashboard**: Interactive web UI with live system metrics and browser pool visibility ```python # Access the monitoring dashboard # Visit: http://localhost:11235/dashboard @@ -751,7 +751,7 @@ This release focuses on stability with 11 bug fixes addressing issues reported b # - Error monitoring with full context ``` -- **๐ Comprehensive Monitor API**: Complete REST API for programmatic access to all monitoring data +- ** Comprehensive Monitor API**: Complete REST API for programmatic access to all monitoring data ```python import httpx @@ -769,12 +769,12 @@ This release focuses on stability with 11 bug fixes addressing issues reported b stats = await client.get("http://localhost:11235/monitor/endpoints/stats") ``` -- **โก WebSocket Streaming**: Real-time updates every 2 seconds for custom dashboards -- **๐ฅ Smart Browser Pool**: 3-tier architecture (permanent/hot/cold) with automatic promotion and cleanup -- **๐งน Janitor System**: Automatic resource management with event logging -- **๐ฎ Control Actions**: Manual browser management (kill, restart, cleanup) via API -- **๐ Production Metrics**: 6 critical metrics for operational excellence with Prometheus integration -- **๐ Critical Bug Fixes**: +- ** WebSocket Streaming**: Real-time updates every 2 seconds for custom dashboards +- ** Smart Browser Pool**: 3-tier architecture (permanent/hot/cold) with automatic promotion and cleanup +- ** Janitor System**: Automatic resource management with event logging +- ** Control Actions**: Manual browser management (kill, restart, cleanup) via API +- ** Production Metrics**: 6 critical metrics for operational excellence with Prometheus integration +- ** Critical Bug Fixes**: - Fixed async LLM extraction blocking issue (#1055) - Enhanced DFS deep crawl strategy (#1607) - Fixed sitemap parsing in AsyncUrlSeeder (#1598) @@ -789,8 +789,8 @@ This release focuses on stability with 11 bug fixes addressing issues reported bVersion 0.7.5 Release Highlights - The Docker Hooks & Security Update
-- **๐ง Docker Hooks System**: Complete pipeline customization with user-provided Python functions at 8 key points -- **โจ Function-Based Hooks API (NEW)**: Write hooks as regular Python functions with full IDE support: +- ** Docker Hooks System**: Complete pipeline customization with user-provided Python functions at 8 key points +- ** Function-Based Hooks API (NEW)**: Write hooks as regular Python functions with full IDE support: ```python from crawl4ai import hooks_to_string from crawl4ai.docker_client import Crawl4aiDockerClient @@ -827,8 +827,8 @@ This release focuses on stability with 11 bug fixes addressing issues reported b - **๐ค Enhanced LLM Integration**: Custom providers with temperature control and base_url configuration - **๐ HTTPS Preservation**: Secure internal link handling with `preserve_https_for_internal_links=True` -- **๐ Python 3.10+ Support**: Modern language features and enhanced performance -- **๐ ๏ธ Bug Fixes**: Resolved multiple community-reported issues including URL processing, JWT authentication, and proxy configuration +- ** Python 3.10+ Support**: Modern language features and enhanced performance +- ** Bug Fixes**: Resolved multiple community-reported issues including URL processing, JWT authentication, and proxy configuration [Full v0.7.5 Release Notes โ](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.5.md) @@ -858,9 +858,9 @@ This release focuses on stability with 11 bug fixes addressing issues reported b print(f"Extracted table: {len(table['data'])} rows") ``` -- **โก Dispatcher Bug Fix**: Fixed sequential processing bottleneck in arun_many for fast-completing tasks -- **๐งน Memory Management Refactor**: Consolidated memory utilities into main utils module for cleaner architecture -- **๐ง Browser Manager Fixes**: Resolved race conditions in concurrent page creation with thread-safe locking +- ** Dispatcher Bug Fix**: Fixed sequential processing bottleneck in arun_many for fast-completing tasks +- ** Memory Management Refactor**: Consolidated memory utilities into main utils module for cleaner architecture +- ** Browser Manager Fixes**: Resolved race conditions in concurrent page creation with thread-safe locking - **๐ Advanced URL Processing**: Better handling of raw:// URLs and base tag link resolution - **๐ก๏ธ Enhanced Proxy Support**: Flexible proxy configuration supporting both dict and string formats @@ -871,7 +871,7 @@ This release focuses on stability with 11 bug fixes addressing issues reported bVersion 0.7.3 Release Highlights - The Multi-Config Intelligence Update
-- **๐ต๏ธ Undetected Browser Support**: Bypass sophisticated bot detection systems: +- ** Undetected Browser Support**: Bypass sophisticated bot detection systems: ```python from crawl4ai import AsyncWebCrawler, BrowserConfig @@ -889,7 +889,7 @@ This release focuses on stability with 11 bug fixes addressing issues reported b # Successfully bypass Cloudflare, Akamai, and custom bot detection ``` -- **๐จ Multi-URL Configuration**: Different strategies for different URL patterns in one batch: +- ** Multi-URL Configuration**: Different strategies for different URL patterns in one batch: ```python from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode @@ -915,7 +915,7 @@ from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode # Each URL gets the perfect configuration automatically ``` -- **๐ง Memory Monitoring**: Track and optimize memory usage during crawling: +- ** Memory Monitoring**: Track and optimize memory usage during crawling: ```python from crawl4ai.memory_utils import MemoryMonitor @@ -930,7 +930,7 @@ from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode # Get optimization recommendations ``` -- **๐ Enhanced Table Extraction**: Direct DataFrame conversion from web tables: +- ** Enhanced Table Extraction**: Direct DataFrame conversion from web tables: ```python result = await crawler.arun("https://site-with-tables.com") @@ -942,8 +942,8 @@ from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode print(f"Table: {df.shape[0]} rows ร {df.shape[1]} columns") ``` -- **๐ฐ GitHub Sponsors**: 4-tier sponsorship system for project sustainability -- **๐ณ Docker LLM Flexibility**: Configure providers via environment variables +- ** GitHub Sponsors**: 4-tier sponsorship system for project sustainability +- ** Docker LLM Flexibility**: Configure providers via environment variables [Full v0.7.3 Release Notes โ](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.7.3.md) @@ -952,7 +952,7 @@ from crawl4ai import CrawlerRunConfig, MatchMode, CacheModeVersion 0.7.0 Release Highlights - The Adaptive Intelligence Update
-- **๐ง Adaptive Crawling**: Your crawler now learns and adapts to website patterns automatically: +- ** Adaptive Crawling**: Your crawler now learns and adapts to website patterns automatically: ```python config = AdaptiveConfig( confidence_threshold=0.7, # Min confidence to stop crawling @@ -970,7 +970,7 @@ from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode # Crawler learns patterns and improves extraction over time ``` -- **๐ Virtual Scroll Support**: Complete content extraction from infinite scroll pages: +- ** Virtual Scroll Support**: Complete content extraction from infinite scroll pages: ```python scroll_config = VirtualScrollConfig( container_selector="[data-testid='feed']", @@ -999,7 +999,7 @@ from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode # Links ranked by relevance and quality ``` -- **๐ฃ Async URL Seeder**: Discover thousands of URLs in seconds: +- ** Async URL Seeder**: Discover thousands of URLs in seconds: ```python seeder = AsyncUrlSeeder(SeedingConfig( source="sitemap+cc", @@ -1011,7 +1011,7 @@ from crawl4ai import CrawlerRunConfig, MatchMode, CacheMode urls = await seeder.discover("https://example.com") ``` -- **โก Performance Boost**: Up to 3x faster with optimized resource handling and memory efficiency +- ** Performance Boost**: Up to 3x faster with optimized resource handling and memory efficiency Read the full details in our [0.7.0 Release Notes](https://docs.crawl4ai.com/blog/release-v0.7.0) or check the [CHANGELOG](https://github.com/unclecode/crawl4ai/blob/main/CHANGELOG.md). @@ -1022,7 +1022,7 @@ Read the full details in our [0.7.0 Release Notes](https://docs.crawl4ai.com/blo Crawl4AI follows standard Python version numbering conventions (PEP 440) to help users understand the stability and features of each release.
-
-## ๐ Documentation & Roadmap
+## Documentation & Roadmap
-> ๐จ **Documentation Update Alert**: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!
+> **Documentation Update Alert**: We're undertaking a major documentation overhaul next week to reflect recent updates and improvements. Stay tuned for a more comprehensive and up-to-date guide!
For current documentation, including installation instructions, advanced features, and API reference, visit our [Documentation Website](https://docs.crawl4ai.com/).
To check our development plans and upcoming features, visit our [Roadmap](https://github.com/unclecode/crawl4ai/blob/main/ROADMAP.md).
๐ Version Numbers Explained
+Version Numbers Explained
Our version numbers follow this pattern: `MAJOR.MINOR.PATCH` (e.g., 0.4.3) @@ -1061,16 +1061,16 @@ For production environments, we recommend using the stable version. For testing
-
-## ๐ค Contributing
+## Contributing
We welcome contributions from the open-source community. Check out our [contribution guidelines](https://github.com/unclecode/crawl4ai/blob/main/CONTRIBUTORS.md) for more information.
@@ -1095,7 +1095,7 @@ I'll help modify the license section with badges. For the halftone effect, here'
Here's the updated license section:
-## ๐ License & Attribution
+## License & Attribution
This project is licensed under the Apache License 2.0, attribution is recommended via the badges below. See the [Apache 2.0 License](https://github.com/unclecode/crawl4ai/blob/main/LICENSE) file for details.
@@ -1103,7 +1103,7 @@ This project is licensed under the Apache License 2.0, attribution is recommende
When using Crawl4AI, you must include one of the following attribution methods:
๐ Development TODOs
+Development TODOs
- [x] 0. Graph Crawler: Smart website traversal using graph search algorithms for comprehensive nested page extraction - [x] 1. Question-Based Crawler: Natural language driven web discovery and content extraction @@ -1087,7 +1087,7 @@ To check our development plans and upcoming features, visit our [Roadmap](https:
-
๐ 1. Badge Attribution (Recommended)
+1. Badge Attribution (Recommended)
Add one of these badges to your README, documentation, or website: | Theme | Badge | @@ -1145,14 +1145,14 @@ HTML code for adding the badges:
-
-## ๐ Citation
+## Citation
If you use Crawl4AI in your research or project, please cite:
@@ -1174,7 +1174,7 @@ UncleCode. (2024). Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper [Com
GitHub. https://github.com/unclecode/crawl4ai
```
-## ๐ง Contact
+## Contact
For questions, suggestions, or feedback, feel free to reach out:
@@ -1182,16 +1182,16 @@ For questions, suggestions, or feedback, feel free to reach out:
- Twitter: [@unclecode](https://twitter.com/unclecode)
- Website: [crawl4ai.com](https://crawl4ai.com)
-Happy Crawling! ๐ธ๏ธ๐
+Happy Crawling! ๐
-## ๐พ Mission
+## Mission
Our mission is to unlock the value of personal and enterprise data by transforming digital footprints into structured, tradeable assets. Crawl4AI empowers individuals and organizations with open-source tools to extract and structure data, fostering a shared data economy.
We envision a future where AI is powered by real human knowledge, ensuring data creators directly benefit from their contributions. By democratizing data and enabling ethical sharing, we are laying the foundation for authentic AI advancement.
๐ 2. Text Attribution
+2. Text Attribution
Add this line to your documentation: ``` This project uses Crawl4AI (https://github.com/unclecode/crawl4ai) for web data extraction. ```
-
-## ๐ Current Sponsors
+## Current Sponsors
-### ๐ข Enterprise Sponsors & Partners
+### Enterprise Sponsors & Partners
Our enterprise sponsors and technology partners help scale Crawl4AI to power production-grade data pipelines.
| Company | About | Sponsorship Tier |
|------|------|----------------------------|
-| ๐ Key Opportunities
+Key Opportunities
- **Data Capitalization**: Transform digital footprints into measurable, valuable assets. - **Authentic AI Data**: Provide AI systems with real human insights. @@ -1209,25 +1209,25 @@ We envision a future where AI is powered by real human knowledge, ensuring data For more details, see our [full mission statement](./MISSION.md). | Leveraging Thordata ensures seamless compatibility with any AI/ML workflows and data infrastructure, massively accessing web data with 99.9% uptime, backed by one-on-one customer support. | ๐ฅ Silver |
-| | Helps engineers and buyers find, compare, and source electronic & industrial parts in seconds, with specs, pricing, lead times & alternatives.| ๐ฅ Gold |
-| KidoCode
| Kidocode is a hybrid technology and entrepreneurship school for kids aged 5โ18, offering both online and on-campus education. | ๐ฅ Gold | -| | Singapore-based Aleph Null is Asiaโs leading edtech hub, dedicated to student-centric, AI-driven educationโempowering learners with the tools to thrive in a fast-changing world. | ๐ฅ Gold |
+| | Leveraging Thordata ensures seamless compatibility with any AI/ML workflows and data infrastructure, massively accessing web data with 99.9% uptime, backed by one-on-one customer support. | Silver |
+| | NstProxy is a trusted proxy provider with over 110M+ real residential IPs, city-level targeting, 99.99% uptime, and low pricing at $0.1/GB, it delivers unmatched stability, scale, and cost-efficiency. | Silver |
+| | Helps engineers and buyers find, compare, and source electronic & industrial parts in seconds, with specs, pricing, lead times & alternatives.| Gold |
+| KidoCode
| Kidocode is a hybrid technology and entrepreneurship school for kids aged 5โ18, offering both online and on-campus education. | Gold | +| | Singapore-based Aleph Null is Asiaโs leading edtech hub, dedicated to student-centric, AI-driven educationโempowering learners with the tools to thrive in a fast-changing world. | Gold |
-### ๐งโ๐ค Individual Sponsors
+### โ Individual Sponsors
A heartfelt thanks to our individual supporters! Every contribution helps us keep our opensource mission alive and thriving!
diff --git a/ROADMAP.md b/ROADMAP.md
index 0fd784c13..b55dc40a7 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -51,7 +51,7 @@ graph TD
Crawl4AI is evolving to provide more intelligent, efficient, and versatile web crawling capabilities. This roadmap outlines the key developments and features planned for the project, organized into strategic sections that build upon our current foundation.
-## 1. Advanced Crawling Systems ๐ง
+## 1. Advanced Crawling Systems
This section introduces three powerful crawling systems that extend Crawl4AI's capabilities from basic web crawling to intelligent, purpose-driven data extraction.
@@ -163,7 +163,7 @@ async with AsyncWebCrawler() as crawler:
print("Success Rate:", result.success_rate)
```
-# Section 2: Specialized Features ๐ ๏ธ
+# Section 2: Specialized Features
This section introduces specialized tools and features that enhance Crawl4AI's capabilities for specific use cases and data extraction needs.
@@ -308,15 +308,15 @@ async with AsyncWebCrawler() as crawler:
Each of these specialized features builds upon Crawl4AI's core functionality while providing targeted solutions for specific use cases. They can be used independently or combined for more complex data extraction and processing needs.
-# Section 3: Development Tools ๐ง
+# Section 3: Development Tools
This section covers tools designed to enhance the development experience, monitoring, and deployment of Crawl4AI applications.
-### 3.1 Crawl4AI Playground ๐ฎ
+### 3.1 Crawl4AI Playground
The Crawl4AI Playground is an interactive web-based development environment that simplifies web scraping experimentation, development, and deployment. With its intuitive interface and AI-powered assistance, users can quickly prototype, test, and deploy web scraping solutions.
-#### Key Features ๐
+#### Key Features
##### Visual Strategy Builder
- Interactive point-and-click interface for building extraction strategies
@@ -441,7 +441,7 @@ print(f"Monitor URL: {deployment.monitor_url}")
These development tools work together to provide a comprehensive environment for developing, testing, monitoring, and deploying Crawl4AI applications. The Playground helps users experiment and generate optimal configurations, the Performance Monitor ensures smooth operation, and the Cloud Integration tools simplify deployment and scaling.
-# Section 4: Community & Growth ๐ฑ
+# Section 4: Community & Growth
This section outlines initiatives designed to build and support the Crawl4AI community, provide educational resources, and ensure sustainable project growth.
diff --git a/SPONSORS.md b/SPONSORS.md
index 6503cdb76..4f3edcaf5 100644
--- a/SPONSORS.md
+++ b/SPONSORS.md
@@ -1,16 +1,16 @@
-# ๐ Sponsors & Supporters
+# Sponsors & Supporters
Thank you to everyone supporting Crawl4AI! Your sponsorship helps keep this project open-source and actively maintained.
-## ๐ Founding Sponsors
+## Founding Sponsors
*The first 50 sponsors who believed in our vision - permanently recognized*
-๐ **Become a Founding Sponsor!** Only [X/50] spots remaining! [Join now โ](https://github.com/sponsors/unclecode)
+ **Become a Founding Sponsor!** Only [X/50] spots remaining! [Join now โ](https://github.com/sponsors/unclecode)
---
-## ๐ข Data Infrastructure Partners ($2000/month)
+## Data Infrastructure Partners ($2000/month)
*These organizations are building their data sovereignty with Crawl4AI at the core*
@@ -18,7 +18,7 @@ Thank you to everyone supporting Crawl4AI! Your sponsorship helps keep this proj
---
-## ๐ผ Growing Teams ($500/month)
+## Growing Teams ($500/month)
*Teams scaling their data extraction with Crawl4AI*
@@ -34,7 +34,7 @@ Thank you to everyone supporting Crawl4AI! Your sponsorship helps keep this proj
---
-## ๐ฑ Believers ($5/month)
+## Believers ($5/month)
*The community supporting data democratization*
@@ -42,15 +42,15 @@ Thank you to everyone supporting Crawl4AI! Your sponsorship helps keep this proj
---
-## ๐ค Want to Sponsor?
+## Want to Sponsor?
Crawl4AI is the #1 trending open-source web crawler. We're building the future of data extraction - where organizations own their data pipelines instead of relying on rate-limited APIs.
### Available Sponsorship Tiers:
-- **๐ฑ Believer** ($5/mo) - Support the movement
+- ** Believer** ($5/mo) - Support the movement
- **๐ Builder** ($50/mo) - Priority support & early access
-- **๐ผ Growing Team** ($500/mo) - Bi-weekly syncs & optimization
-- **๐ข Data Infrastructure Partner** ($2000/mo) - Full partnership & dedicated support
+- ** Growing Team** ($500/mo) - Bi-weekly syncs & optimization
+- ** Data Infrastructure Partner** ($2000/mo) - Full partnership & dedicated support
[View all tiers and benefits โ](https://github.com/sponsors/unclecode)
@@ -58,7 +58,7 @@ Crawl4AI is the #1 trending open-source web crawler. We're building the future o
Building data extraction at scale? Need dedicated support or infrastructure? Let's talk about a custom partnership.
-๐ง Contact: [hello@crawl4ai.com](mailto:hello@crawl4ai.com) | ๐
[Schedule a call](https://calendar.app.google/rEpvi2UBgUQjWHfJ9)
+ Contact: [hello@crawl4ai.com](mailto:hello@crawl4ai.com) | [Schedule a call](https://calendar.app.google/rEpvi2UBgUQjWHfJ9)
---
diff --git a/crawl4ai/adaptive_crawler copy.py b/crawl4ai/adaptive_crawler copy.py
deleted file mode 100644
index 294a292d4..000000000
--- a/crawl4ai/adaptive_crawler copy.py
+++ /dev/null
@@ -1,1847 +0,0 @@
-"""
-Adaptive Web Crawler for Crawl4AI
-
-This module implements adaptive information foraging for efficient web crawling.
-It determines when sufficient information has been gathered to answer a query,
-avoiding unnecessary crawls while ensuring comprehensive coverage.
-"""
-
-from abc import ABC, abstractmethod
-from typing import Dict, List, Optional, Set, Tuple, Any, Union
-from dataclasses import dataclass, field
-import asyncio
-import pickle
-import os
-import json
-import math
-from collections import defaultdict, Counter
-import re
-from pathlib import Path
-
-from crawl4ai.async_webcrawler import AsyncWebCrawler
-from crawl4ai.async_configs import CrawlerRunConfig, LinkPreviewConfig
-from crawl4ai.models import Link, CrawlResult
-
-
-@dataclass
-class CrawlState:
- """Tracks the current state of adaptive crawling"""
- crawled_urls: Set[str] = field(default_factory=set)
- knowledge_base: List[CrawlResult] = field(default_factory=list)
- pending_links: List[Link] = field(default_factory=list)
- query: str = ""
- metrics: Dict[str, float] = field(default_factory=dict)
-
- # Statistical tracking
- term_frequencies: Dict[str, int] = field(default_factory=lambda: defaultdict(int))
- document_frequencies: Dict[str, int] = field(default_factory=lambda: defaultdict(int))
- documents_with_terms: Dict[str, Set[int]] = field(default_factory=lambda: defaultdict(set))
- total_documents: int = 0
-
- # History tracking for saturation
- new_terms_history: List[int] = field(default_factory=list)
- crawl_order: List[str] = field(default_factory=list)
-
- # Embedding-specific tracking (only if strategy is embedding)
- kb_embeddings: Optional[Any] = None # Will be numpy array
- query_embeddings: Optional[Any] = None # Will be numpy array
- expanded_queries: List[str] = field(default_factory=list)
- coverage_shape: Optional[Any] = None # Alpha shape
- semantic_gaps: List[Tuple[List[float], float]] = field(default_factory=list) # Serializable
- embedding_model: str = ""
-
- def save(self, path: Union[str, Path]):
- """Save state to disk for persistence"""
- path = Path(path)
- path.parent.mkdir(parents=True, exist_ok=True)
-
- # Convert CrawlResult objects to dicts for serialization
- state_dict = {
- 'crawled_urls': list(self.crawled_urls),
- 'knowledge_base': [self._crawl_result_to_dict(cr) for cr in self.knowledge_base],
- 'pending_links': [link.model_dump() for link in self.pending_links],
- 'query': self.query,
- 'metrics': self.metrics,
- 'term_frequencies': dict(self.term_frequencies),
- 'document_frequencies': dict(self.document_frequencies),
- 'documents_with_terms': {k: list(v) for k, v in self.documents_with_terms.items()},
- 'total_documents': self.total_documents,
- 'new_terms_history': self.new_terms_history,
- 'crawl_order': self.crawl_order,
- # Embedding-specific fields (convert numpy arrays to lists for JSON)
- 'kb_embeddings': self.kb_embeddings.tolist() if self.kb_embeddings is not None else None,
- 'query_embeddings': self.query_embeddings.tolist() if self.query_embeddings is not None else None,
- 'expanded_queries': self.expanded_queries,
- 'semantic_gaps': self.semantic_gaps,
- 'embedding_model': self.embedding_model
- }
-
- with open(path, 'w') as f:
- json.dump(state_dict, f, indent=2)
-
- @classmethod
- def load(cls, path: Union[str, Path]) -> 'CrawlState':
- """Load state from disk"""
- path = Path(path)
- with open(path, 'r') as f:
- state_dict = json.load(f)
-
- state = cls()
- state.crawled_urls = set(state_dict['crawled_urls'])
- state.knowledge_base = [cls._dict_to_crawl_result(d) for d in state_dict['knowledge_base']]
- state.pending_links = [Link(**link_dict) for link_dict in state_dict['pending_links']]
- state.query = state_dict['query']
- state.metrics = state_dict['metrics']
- state.term_frequencies = defaultdict(int, state_dict['term_frequencies'])
- state.document_frequencies = defaultdict(int, state_dict['document_frequencies'])
- state.documents_with_terms = defaultdict(set, {k: set(v) for k, v in state_dict['documents_with_terms'].items()})
- state.total_documents = state_dict['total_documents']
- state.new_terms_history = state_dict['new_terms_history']
- state.crawl_order = state_dict['crawl_order']
-
- # Load embedding-specific fields (convert lists back to numpy arrays)
- import numpy as np
- state.kb_embeddings = np.array(state_dict['kb_embeddings']) if state_dict.get('kb_embeddings') is not None else None
- state.query_embeddings = np.array(state_dict['query_embeddings']) if state_dict.get('query_embeddings') is not None else None
- state.expanded_queries = state_dict.get('expanded_queries', [])
- state.semantic_gaps = state_dict.get('semantic_gaps', [])
- state.embedding_model = state_dict.get('embedding_model', '')
-
- return state
-
- @staticmethod
- def _crawl_result_to_dict(cr: CrawlResult) -> Dict:
- """Convert CrawlResult to serializable dict"""
- # Extract markdown content safely
- markdown_content = ""
- if hasattr(cr, 'markdown') and cr.markdown:
- if hasattr(cr.markdown, 'raw_markdown'):
- markdown_content = cr.markdown.raw_markdown
- else:
- markdown_content = str(cr.markdown)
-
- return {
- 'url': cr.url,
- 'content': markdown_content,
- 'links': cr.links if hasattr(cr, 'links') else {},
- 'metadata': cr.metadata if hasattr(cr, 'metadata') else {}
- }
-
- @staticmethod
- def _dict_to_crawl_result(d: Dict):
- """Convert dict back to CrawlResult"""
- # Create a mock object that has the minimal interface we need
- class MockMarkdown:
- def __init__(self, content):
- self.raw_markdown = content
-
- class MockCrawlResult:
- def __init__(self, url, content, links, metadata):
- self.url = url
- self.markdown = MockMarkdown(content)
- self.links = links
- self.metadata = metadata
-
- return MockCrawlResult(
- url=d['url'],
- content=d.get('content', ''),
- links=d.get('links', {}),
- metadata=d.get('metadata', {})
- )
-
-
-@dataclass
-class AdaptiveConfig:
- """Configuration for adaptive crawling"""
- confidence_threshold: float = 0.7
- max_depth: int = 5
- max_pages: int = 20
- top_k_links: int = 3
- min_gain_threshold: float = 0.1
- strategy: str = "statistical" # statistical, embedding, llm
-
- # Advanced parameters
- saturation_threshold: float = 0.8
- consistency_threshold: float = 0.7
- coverage_weight: float = 0.4
- consistency_weight: float = 0.3
- saturation_weight: float = 0.3
-
- # Link scoring parameters
- relevance_weight: float = 0.5
- novelty_weight: float = 0.3
- authority_weight: float = 0.2
-
- # Persistence
- save_state: bool = False
- state_path: Optional[str] = None
-
- # Embedding strategy parameters
- embedding_model: str = "sentence-transformers/all-MiniLM-L6-v2"
- embedding_llm_config: Optional[Dict] = None # Separate config for embeddings
- n_query_variations: int = 10
- coverage_threshold: float = 0.85
- alpha_shape_alpha: float = 0.5
-
- # Embedding confidence calculation parameters
- embedding_coverage_radius: float = 0.2 # Distance threshold for "covered" query points
- # Example: With radius=0.2, a query point is considered covered if ANY document
- # is within cosine distance 0.2 (very similar). Smaller = stricter coverage requirement
-
- embedding_k_exp: float = 3.0 # Exponential decay factor for distance-to-score mapping
- # Example: score = exp(-k_exp * distance). With k_exp=1, distance 0.2 โ score 0.82,
- # distance 0.5 โ score 0.61. Higher k_exp = steeper decay = more emphasis on very close matches
-
- embedding_nearest_weight: float = 0.7 # Weight for nearest neighbor in hybrid scoring
- embedding_top_k_weight: float = 0.3 # Weight for top-k average in hybrid scoring
- # Example: If nearest doc has score 0.9 and top-3 avg is 0.6, final = 0.7*0.9 + 0.3*0.6 = 0.81
- # Higher nearest_weight = more focus on best match vs neighborhood density
-
- # Embedding link selection parameters
- embedding_overlap_threshold: float = 0.85 # Similarity threshold for penalizing redundant links
- # Example: Links with >0.85 similarity to existing KB get penalized to avoid redundancy
- # Lower = more aggressive deduplication, Higher = allow more similar content
-
- # Embedding stopping criteria parameters
- embedding_min_relative_improvement: float = 0.1 # Minimum relative improvement to continue
- # Example: If confidence is 0.6, need improvement > 0.06 per batch to continue crawling
- # Lower = more patient crawling, Higher = stop earlier when progress slows
-
- embedding_validation_min_score: float = 0.4 # Minimum validation score to trust convergence
- # Example: Even if learning converged, keep crawling if validation score < 0.4
- # This prevents premature stopping when we haven't truly covered the query space
-
- # Quality confidence mapping parameters (for display to user)
- embedding_quality_min_confidence: float = 0.7 # Minimum confidence for validated systems
- embedding_quality_max_confidence: float = 0.95 # Maximum realistic confidence
- embedding_quality_scale_factor: float = 0.833 # Scaling factor for confidence mapping
- # Example: Validated system with learning_score=0.5 โ confidence = 0.7 + (0.5-0.4)*0.833 = 0.78
- # These control how internal scores map to user-friendly confidence percentages
-
- def validate(self):
- """Validate configuration parameters"""
- assert 0 <= self.confidence_threshold <= 1, "confidence_threshold must be between 0 and 1"
- assert self.max_depth > 0, "max_depth must be positive"
- assert self.max_pages > 0, "max_pages must be positive"
- assert self.top_k_links > 0, "top_k_links must be positive"
- assert 0 <= self.min_gain_threshold <= 1, "min_gain_threshold must be between 0 and 1"
-
- # Check weights sum to 1
- weight_sum = self.coverage_weight + self.consistency_weight + self.saturation_weight
- assert abs(weight_sum - 1.0) < 0.001, f"Coverage weights must sum to 1, got {weight_sum}"
-
- weight_sum = self.relevance_weight + self.novelty_weight + self.authority_weight
- assert abs(weight_sum - 1.0) < 0.001, f"Link scoring weights must sum to 1, got {weight_sum}"
-
- # Validate embedding parameters
- assert 0 < self.embedding_coverage_radius < 1, "embedding_coverage_radius must be between 0 and 1"
- assert self.embedding_k_exp > 0, "embedding_k_exp must be positive"
- assert 0 <= self.embedding_nearest_weight <= 1, "embedding_nearest_weight must be between 0 and 1"
- assert 0 <= self.embedding_top_k_weight <= 1, "embedding_top_k_weight must be between 0 and 1"
- assert abs(self.embedding_nearest_weight + self.embedding_top_k_weight - 1.0) < 0.001, "Embedding weights must sum to 1"
- assert 0 <= self.embedding_overlap_threshold <= 1, "embedding_overlap_threshold must be between 0 and 1"
- assert 0 < self.embedding_min_relative_improvement < 1, "embedding_min_relative_improvement must be between 0 and 1"
- assert 0 <= self.embedding_validation_min_score <= 1, "embedding_validation_min_score must be between 0 and 1"
- assert 0 <= self.embedding_quality_min_confidence <= 1, "embedding_quality_min_confidence must be between 0 and 1"
- assert 0 <= self.embedding_quality_max_confidence <= 1, "embedding_quality_max_confidence must be between 0 and 1"
- assert self.embedding_quality_scale_factor > 0, "embedding_quality_scale_factor must be positive"
-
-
-class CrawlStrategy(ABC):
- """Abstract base class for crawling strategies"""
-
- @abstractmethod
- async def calculate_confidence(self, state: CrawlState) -> float:
- """Calculate overall confidence that we have sufficient information"""
- pass
-
- @abstractmethod
- async def rank_links(self, state: CrawlState, config: AdaptiveConfig) -> List[Tuple[Link, float]]:
- """Rank pending links by expected information gain"""
- pass
-
- @abstractmethod
- async def should_stop(self, state: CrawlState, config: AdaptiveConfig) -> bool:
- """Determine if crawling should stop"""
- pass
-
- @abstractmethod
- async def update_state(self, state: CrawlState, new_results: List[CrawlResult]) -> None:
- """Update state with new crawl results"""
- pass
-
-
-class StatisticalStrategy(CrawlStrategy):
- """Pure statistical approach - no LLM, no embeddings"""
-
- def __init__(self):
- self.idf_cache = {}
- self.bm25_k1 = 1.2 # BM25 parameter
- self.bm25_b = 0.75 # BM25 parameter
-
- async def calculate_confidence(self, state: CrawlState) -> float:
- """Calculate confidence using coverage, consistency, and saturation"""
- if not state.knowledge_base:
- return 0.0
-
- coverage = self._calculate_coverage(state)
- consistency = self._calculate_consistency(state)
- saturation = self._calculate_saturation(state)
-
- # Store individual metrics
- state.metrics['coverage'] = coverage
- state.metrics['consistency'] = consistency
- state.metrics['saturation'] = saturation
-
- # Weighted combination (weights from config not accessible here, using defaults)
- confidence = 0.4 * coverage + 0.3 * consistency + 0.3 * saturation
-
- return confidence
-
- def _calculate_coverage(self, state: CrawlState) -> float:
- """Coverage scoring - measures query term presence across knowledge base
-
- Returns a score between 0 and 1, where:
- - 0 means no query terms found
- - 1 means excellent coverage of all query terms
- """
- if not state.query or state.total_documents == 0:
- return 0.0
-
- query_terms = self._tokenize(state.query.lower())
- if not query_terms:
- return 0.0
-
- term_scores = []
- max_tf = max(state.term_frequencies.values()) if state.term_frequencies else 1
-
- for term in query_terms:
- tf = state.term_frequencies.get(term, 0)
- df = state.document_frequencies.get(term, 0)
-
- if df > 0:
- # Document coverage: what fraction of docs contain this term
- doc_coverage = df / state.total_documents
-
- # Frequency signal: normalized log frequency
- freq_signal = math.log(1 + tf) / math.log(1 + max_tf) if max_tf > 0 else 0
-
- # Combined score: document coverage with frequency boost
- term_score = doc_coverage * (1 + 0.5 * freq_signal)
- term_scores.append(term_score)
- else:
- term_scores.append(0.0)
-
- # Average across all query terms
- coverage = sum(term_scores) / len(term_scores)
-
- # Apply square root curve to make score more intuitive
- # This helps differentiate between partial and good coverage
- return min(1.0, math.sqrt(coverage))
-
- def _calculate_consistency(self, state: CrawlState) -> float:
- """Information overlap between pages - high overlap suggests coherent topic coverage"""
- if len(state.knowledge_base) < 2:
- return 1.0 # Single or no documents are perfectly consistent
-
- # Calculate pairwise term overlap
- overlaps = []
-
- for i in range(len(state.knowledge_base)):
- for j in range(i + 1, len(state.knowledge_base)):
- # Get terms from both documents
- terms_i = set(self._get_document_terms(state.knowledge_base[i]))
- terms_j = set(self._get_document_terms(state.knowledge_base[j]))
-
- if terms_i and terms_j:
- # Jaccard similarity
- overlap = len(terms_i & terms_j) / len(terms_i | terms_j)
- overlaps.append(overlap)
-
- if overlaps:
- # Average overlap as consistency measure
- consistency = sum(overlaps) / len(overlaps)
- else:
- consistency = 0.0
-
- return consistency
-
- def _calculate_saturation(self, state: CrawlState) -> float:
- """Diminishing returns indicator - are we still discovering new information?"""
- if not state.new_terms_history:
- return 0.0
-
- if len(state.new_terms_history) < 2:
- return 0.0 # Not enough history
-
- # Calculate rate of new term discovery
- recent_rate = state.new_terms_history[-1] if state.new_terms_history[-1] > 0 else 1
- initial_rate = state.new_terms_history[0] if state.new_terms_history[0] > 0 else 1
-
- # Saturation increases as rate decreases
- saturation = 1 - (recent_rate / initial_rate)
-
- return max(0.0, min(saturation, 1.0))
-
- async def rank_links(self, state: CrawlState, config: AdaptiveConfig) -> List[Tuple[Link, float]]:
- """Rank links by expected information gain"""
- scored_links = []
-
- for link in state.pending_links:
- # Skip already crawled URLs
- if link.href in state.crawled_urls:
- continue
-
- # Calculate component scores
- relevance = self._calculate_relevance(link, state)
- novelty = self._calculate_novelty(link, state)
- authority = 1.0
- # authority = self._calculate_authority(link)
-
- # Combined score
- score = (config.relevance_weight * relevance +
- config.novelty_weight * novelty +
- config.authority_weight * authority)
-
- scored_links.append((link, score))
-
- # Sort by score descending
- scored_links.sort(key=lambda x: x[1], reverse=True)
-
- return scored_links
-
- def _calculate_relevance(self, link: Link, state: CrawlState) -> float:
- """BM25 relevance score between link preview and query"""
- if not state.query or not link:
- return 0.0
-
- # Combine available text from link
- link_text = ' '.join(filter(None, [
- link.text or '',
- link.title or '',
- link.head_data.get('meta', {}).get('title', '') if link.head_data else '',
- link.head_data.get('meta', {}).get('description', '') if link.head_data else '',
- link.head_data.get('meta', {}).get('keywords', '') if link.head_data else ''
- ])).lower()
-
- if not link_text:
- return 0.0
-
- # Use contextual score if available (from BM25 scoring during crawl)
- # if link.contextual_score is not None:
- if link.contextual_score and link.contextual_score > 0:
- return link.contextual_score
-
- # Otherwise, calculate simple term overlap
- query_terms = set(self._tokenize(state.query.lower()))
- link_terms = set(self._tokenize(link_text))
-
- if not query_terms:
- return 0.0
-
- overlap = len(query_terms & link_terms) / len(query_terms)
- return overlap
-
- def _calculate_novelty(self, link: Link, state: CrawlState) -> float:
- """Estimate how much new information this link might provide"""
- if not state.knowledge_base:
- return 1.0 # First links are maximally novel
-
- # Get terms from link preview
- link_text = ' '.join(filter(None, [
- link.text or '',
- link.title or '',
- link.head_data.get('title', '') if link.head_data else '',
- link.head_data.get('description', '') if link.head_data else '',
- link.head_data.get('keywords', '') if link.head_data else ''
- ])).lower()
-
- link_terms = set(self._tokenize(link_text))
- if not link_terms:
- return 0.5 # Unknown novelty
-
- # Calculate what percentage of link terms are new
- existing_terms = set(state.term_frequencies.keys())
- new_terms = link_terms - existing_terms
-
- novelty = len(new_terms) / len(link_terms) if link_terms else 0.0
-
- return novelty
-
- def _calculate_authority(self, link: Link) -> float:
- """Simple authority score based on URL structure and link attributes"""
- score = 0.5 # Base score
-
- if not link.href:
- return 0.0
-
- url = link.href.lower()
-
- # Positive indicators
- if '/docs/' in url or '/documentation/' in url:
- score += 0.2
- if '/api/' in url or '/reference/' in url:
- score += 0.2
- if '/guide/' in url or '/tutorial/' in url:
- score += 0.1
-
- # Check for file extensions
- if url.endswith('.pdf'):
- score += 0.1
- elif url.endswith(('.jpg', '.png', '.gif')):
- score -= 0.3 # Reduce score for images
-
- # Use intrinsic score if available
- if link.intrinsic_score is not None:
- score = 0.7 * score + 0.3 * link.intrinsic_score
-
- return min(score, 1.0)
-
- async def should_stop(self, state: CrawlState, config: AdaptiveConfig) -> bool:
- """Determine if crawling should stop"""
- # Check confidence threshold
- confidence = state.metrics.get('confidence', 0.0)
- if confidence >= config.confidence_threshold:
- return True
-
- # Check resource limits
- if len(state.crawled_urls) >= config.max_pages:
- return True
-
- # Check if we have any links left
- if not state.pending_links:
- return True
-
- # Check saturation
- if state.metrics.get('saturation', 0.0) >= config.saturation_threshold:
- return True
-
- return False
-
- async def update_state(self, state: CrawlState, new_results: List[CrawlResult]) -> None:
- """Update state with new crawl results"""
- for result in new_results:
- # Track new terms
- old_term_count = len(state.term_frequencies)
-
- # Extract and process content - try multiple fields
- try:
- content = result.markdown.raw_markdown
- except AttributeError:
- print(f"Warning: CrawlResult {result.url} has no markdown content")
- content = ""
- # content = ""
- # if hasattr(result, 'extracted_content') and result.extracted_content:
- # content = result.extracted_content
- # elif hasattr(result, 'markdown') and result.markdown:
- # content = result.markdown.raw_markdown
- # elif hasattr(result, 'cleaned_html') and result.cleaned_html:
- # content = result.cleaned_html
- # elif hasattr(result, 'html') and result.html:
- # # Use raw HTML as last resort
- # content = result.html
-
-
- terms = self._tokenize(content.lower())
-
- # Update term frequencies
- term_set = set()
- for term in terms:
- state.term_frequencies[term] += 1
- term_set.add(term)
-
- # Update document frequencies
- doc_id = state.total_documents
- for term in term_set:
- if term not in state.documents_with_terms[term]:
- state.document_frequencies[term] += 1
- state.documents_with_terms[term].add(doc_id)
-
- # Track new terms discovered
- new_term_count = len(state.term_frequencies)
- new_terms = new_term_count - old_term_count
- state.new_terms_history.append(new_terms)
-
- # Update document count
- state.total_documents += 1
-
- # Add to crawl order
- state.crawl_order.append(result.url)
-
- def _tokenize(self, text: str) -> List[str]:
- """Simple tokenization - can be enhanced"""
- # Remove punctuation and split
- text = re.sub(r'[^\w\s]', ' ', text)
- tokens = text.split()
-
- # Filter short tokens and stop words (basic)
- tokens = [t for t in tokens if len(t) > 2]
-
- return tokens
-
- def _get_document_terms(self, crawl_result: CrawlResult) -> List[str]:
- """Extract terms from a crawl result"""
- content = crawl_result.markdown.raw_markdown or ""
- return self._tokenize(content.lower())
-
-
-class EmbeddingStrategy(CrawlStrategy):
- """Embedding-based adaptive crawling using semantic space coverage"""
-
- def __init__(self, embedding_model: str = None, llm_config: Dict = None):
- self.embedding_model = embedding_model or "sentence-transformers/all-MiniLM-L6-v2"
- self.llm_config = llm_config
- self._embedding_cache = {}
- self._link_embedding_cache = {} # Cache for link embeddings
- self._validation_passed = False # Track if validation passed
-
- # Performance optimization caches
- self._distance_matrix_cache = None # Cache for query-KB distances
- self._kb_embeddings_hash = None # Track KB changes
- self._validation_embeddings_cache = None # Cache validation query embeddings
- self._kb_similarity_threshold = 0.95 # Threshold for deduplication
-
- async def _get_embeddings(self, texts: List[str]) -> Any:
- """Get embeddings using configured method"""
- from .utils import get_text_embeddings
- embedding_llm_config = {
- 'provider': 'openai/text-embedding-3-small',
- 'api_token': os.getenv('OPENAI_API_KEY')
- }
- return await get_text_embeddings(
- texts,
- embedding_llm_config,
- self.embedding_model
- )
-
- def _compute_distance_matrix(self, query_embeddings: Any, kb_embeddings: Any) -> Any:
- """Compute distance matrix using vectorized operations"""
- import numpy as np
-
- if kb_embeddings is None or len(kb_embeddings) == 0:
- return None
-
- # Ensure proper shapes
- if len(query_embeddings.shape) == 1:
- query_embeddings = query_embeddings.reshape(1, -1)
- if len(kb_embeddings.shape) == 1:
- kb_embeddings = kb_embeddings.reshape(1, -1)
-
- # Vectorized cosine distance: 1 - cosine_similarity
- # Normalize vectors
- query_norm = query_embeddings / np.linalg.norm(query_embeddings, axis=1, keepdims=True)
- kb_norm = kb_embeddings / np.linalg.norm(kb_embeddings, axis=1, keepdims=True)
-
- # Compute cosine similarity matrix
- similarity_matrix = np.dot(query_norm, kb_norm.T)
-
- # Convert to distance
- distance_matrix = 1 - similarity_matrix
-
- return distance_matrix
-
- def _get_cached_distance_matrix(self, query_embeddings: Any, kb_embeddings: Any) -> Any:
- """Get distance matrix with caching"""
- import numpy as np
-
- if kb_embeddings is None or len(kb_embeddings) == 0:
- return None
-
- # Check if KB has changed
- kb_hash = hash(kb_embeddings.tobytes()) if kb_embeddings is not None else None
-
- if (self._distance_matrix_cache is None or
- kb_hash != self._kb_embeddings_hash):
- # Recompute matrix
- self._distance_matrix_cache = self._compute_distance_matrix(query_embeddings, kb_embeddings)
- self._kb_embeddings_hash = kb_hash
-
- return self._distance_matrix_cache
-
- async def map_query_semantic_space(self, query: str, n_synthetic: int = 10) -> Any:
- """Generate a point cloud representing the semantic neighborhood of the query"""
- from .utils import perform_completion_with_backoff
-
- # Generate more variations than needed for train/val split
- n_total = int(n_synthetic * 1.3) # Generate 30% more for validation
-
- # Generate variations using LLM
- prompt = f"""Generate {n_total} variations of this query that explore different aspects: '{query}'
-
- These should be queries a user might ask when looking for similar information.
- Include different phrasings, related concepts, and specific aspects.
-
- Return as a JSON array of strings."""
-
- # Use the LLM for query generation
- provider = self.llm_config.get('provider', 'openai/gpt-4o-mini') if self.llm_config else 'openai/gpt-4o-mini'
- api_token = self.llm_config.get('api_token') if self.llm_config else None
-
- # response = perform_completion_with_backoff(
- # provider=provider,
- # prompt_with_variables=prompt,
- # api_token=api_token,
- # json_response=True
- # )
-
- # variations = json.loads(response.choices[0].message.content)
-
-
- # # Mock data with more variations for split
- variations ={'queries': ['what are the best vegetables to use in fried rice?', 'how do I make vegetable fried rice from scratch?', 'can you provide a quick recipe for vegetable fried rice?', 'what cooking techniques are essential for perfect fried rice with vegetables?', 'how to add flavor to vegetable fried rice?', 'are there any tips for making healthy fried rice with vegetables?']}
-
-
- variations = {'queries': [
- 'How do async and await work with coroutines in Python?',
- 'What is the role of event loops in asynchronous programming?',
- 'Can you explain the differences between async/await and traditional callback methods?',
- 'How do coroutines interact with event loops in JavaScript?',
- 'What are the benefits of using async await over promises in Node.js?',
- # 'How to manage multiple coroutines with an event loop?',
- # 'What are some common pitfalls when using async await with coroutines?',
- # 'How do different programming languages implement async await and event loops?',
- # 'What happens when an async function is called without await?',
- # 'How does the event loop handle blocking operations?',
- 'Can you nest async functions and how does that affect the event loop?',
- 'What is the performance impact of using async/await?'
- ]}
-
- # Split into train and validation
- # all_queries = [query] + variations['queries']
-
- # Randomly shuffle for proper train/val split (keeping original query in training)
- import random
-
- # Keep original query always in training
- other_queries = variations['queries'].copy()
- random.shuffle(other_queries)
-
- # Split: 80% for training, 20% for validation
- n_validation = max(2, int(len(other_queries) * 0.2)) # At least 2 for validation
- val_queries = other_queries[-n_validation:]
- train_queries = [query] + other_queries[:-n_validation]
-
- # Embed only training queries for now (faster)
- train_embeddings = await self._get_embeddings(train_queries)
-
- # Store validation queries for later (don't embed yet to save time)
- self._validation_queries = val_queries
-
- return train_embeddings, train_queries
-
- def compute_coverage_shape(self, query_points: Any, alpha: float = 0.5):
- """Find the minimal shape that covers all query points using alpha shape"""
- try:
- import numpy as np
-
- if len(query_points) < 3:
- return None
-
- # For high-dimensional embeddings (e.g., 384-dim, 768-dim),
- # alpha shapes require exponentially more points than available.
- # Instead, use a statistical coverage model
- query_points = np.array(query_points)
-
- # Store coverage as centroid + radius model
- coverage = {
- 'center': np.mean(query_points, axis=0),
- 'std': np.std(query_points, axis=0),
- 'points': query_points,
- 'radius': np.max(np.linalg.norm(query_points - np.mean(query_points, axis=0), axis=1))
- }
- return coverage
- except Exception:
- # Fallback if computation fails
- return None
-
- def _sample_boundary_points(self, shape, n_samples: int = 20) -> List[Any]:
- """Sample points from the boundary of a shape"""
- import numpy as np
-
- # Simplified implementation - in practice would sample from actual shape boundary
- # For now, return empty list if shape is None
- if shape is None:
- return []
-
- # This is a placeholder - actual implementation would depend on shape type
- return []
-
- def find_coverage_gaps(self, kb_embeddings: Any, query_embeddings: Any) -> List[Tuple[Any, float]]:
- """Calculate gap distances for all query variations using vectorized operations"""
- import numpy as np
-
- gaps = []
-
- if kb_embeddings is None or len(kb_embeddings) == 0:
- # If no KB yet, all query points have maximum gap
- for q_emb in query_embeddings:
- gaps.append((q_emb, 1.0))
- return gaps
-
- # Use cached distance matrix
- distance_matrix = self._get_cached_distance_matrix(query_embeddings, kb_embeddings)
-
- if distance_matrix is None:
- # Fallback
- for q_emb in query_embeddings:
- gaps.append((q_emb, 1.0))
- return gaps
-
- # Find minimum distance for each query (vectorized)
- min_distances = np.min(distance_matrix, axis=1)
-
- # Create gaps list
- for i, q_emb in enumerate(query_embeddings):
- gaps.append((q_emb, min_distances[i]))
-
- return gaps
-
- async def select_links_for_expansion(
- self,
- candidate_links: List[Link],
- gaps: List[Tuple[Any, float]],
- kb_embeddings: Any
- ) -> List[Tuple[Link, float]]:
- """Select links that most efficiently fill the gaps"""
- from .utils import cosine_distance, cosine_similarity, get_text_embeddings
- import numpy as np
- import hashlib
-
- scored_links = []
-
- # Prepare for embedding - separate cached vs uncached
- links_to_embed = []
- texts_to_embed = []
- link_embeddings_map = {}
-
- for link in candidate_links:
- # Extract text from link
- link_text = ' '.join(filter(None, [
- link.text or '',
- link.title or '',
- link.meta.get('description', '') if hasattr(link, 'meta') and link.meta else '',
- link.head_data.get('meta', {}).get('description', '') if link.head_data else ''
- ]))
-
- if not link_text.strip():
- continue
-
- # Create cache key from URL + text content
- cache_key = hashlib.md5(f"{link.href}:{link_text}".encode()).hexdigest()
-
- # Check cache
- if cache_key in self._link_embedding_cache:
- link_embeddings_map[link.href] = self._link_embedding_cache[cache_key]
- else:
- links_to_embed.append(link)
- texts_to_embed.append(link_text)
-
- # Batch embed only uncached links
- if texts_to_embed:
- embedding_llm_config = {
- 'provider': 'openai/text-embedding-3-small',
- 'api_token': os.getenv('OPENAI_API_KEY')
- }
- new_embeddings = await get_text_embeddings(texts_to_embed, embedding_llm_config, self.embedding_model)
-
- # Cache the new embeddings
- for link, text, embedding in zip(links_to_embed, texts_to_embed, new_embeddings):
- cache_key = hashlib.md5(f"{link.href}:{text}".encode()).hexdigest()
- self._link_embedding_cache[cache_key] = embedding
- link_embeddings_map[link.href] = embedding
-
- # Get coverage radius from config
- coverage_radius = self.config.embedding_coverage_radius if hasattr(self, 'config') else 0.2
-
- # Score each link
- for link in candidate_links:
- if link.href not in link_embeddings_map:
- continue # Skip links without embeddings
-
- link_embedding = link_embeddings_map[link.href]
-
- if not gaps:
- score = 0.0
- else:
- # Calculate how many gaps this link helps with
- gaps_helped = 0
- total_improvement = 0
-
- for gap_point, gap_distance in gaps:
- # Only consider gaps that actually need filling (outside coverage radius)
- if gap_distance > coverage_radius:
- new_distance = cosine_distance(link_embedding, gap_point)
- if new_distance < gap_distance:
- # This link helps this gap
- improvement = gap_distance - new_distance
- # Scale improvement - moving from 0.5 to 0.3 is valuable
- scaled_improvement = improvement * 2 # Amplify the signal
- total_improvement += scaled_improvement
- gaps_helped += 1
-
- # Average improvement per gap that needs help
- gaps_needing_help = sum(1 for _, d in gaps if d > coverage_radius)
- if gaps_needing_help > 0:
- gap_reduction_score = total_improvement / gaps_needing_help
- else:
- gap_reduction_score = 0
-
- # Check overlap with existing KB (vectorized)
- if kb_embeddings is not None and len(kb_embeddings) > 0:
- # Normalize embeddings
- link_norm = link_embedding / np.linalg.norm(link_embedding)
- kb_norm = kb_embeddings / np.linalg.norm(kb_embeddings, axis=1, keepdims=True)
-
- # Compute all similarities at once
- similarities = np.dot(kb_norm, link_norm)
- max_similarity = np.max(similarities)
-
- # Only penalize if very similar (above threshold)
- overlap_threshold = self.config.embedding_overlap_threshold if hasattr(self, 'config') else 0.85
- if max_similarity > overlap_threshold:
- overlap_penalty = (max_similarity - overlap_threshold) * 2 # 0 to 0.3 range
- else:
- overlap_penalty = 0
- else:
- overlap_penalty = 0
-
- # Final score - emphasize gap reduction
- score = gap_reduction_score * (1 - overlap_penalty)
-
- # Add contextual score boost if available
- if hasattr(link, 'contextual_score') and link.contextual_score:
- score = score * 0.8 + link.contextual_score * 0.2
-
- scored_links.append((link, score))
-
- return sorted(scored_links, key=lambda x: x[1], reverse=True)
-
- async def calculate_confidence(self, state: CrawlState) -> float:
- """Coverage-based learning score (0โ1)."""
- import numpy as np
-
- # Guard clauses
- if state.kb_embeddings is None or state.query_embeddings is None:
- return 0.0
- if len(state.kb_embeddings) == 0 or len(state.query_embeddings) == 0:
- return 0.0
-
- # Prepare L2-normalised arrays
- Q = np.asarray(state.query_embeddings, dtype=np.float32)
- D = np.asarray(state.kb_embeddings, dtype=np.float32)
- Q /= np.linalg.norm(Q, axis=1, keepdims=True) + 1e-8
- D /= np.linalg.norm(D, axis=1, keepdims=True) + 1e-8
-
- # Best cosine per query
- best = (Q @ D.T).max(axis=1)
-
- # Mean similarity or hit-rate above tau
- tau = getattr(self.config, 'coverage_tau', None)
- score = float((best >= tau).mean()) if tau is not None else float(best.mean())
-
- # Store quick metrics
- state.metrics['coverage_score'] = score
- state.metrics['avg_best_similarity'] = float(best.mean())
- state.metrics['median_best_similarity'] = float(np.median(best))
-
- return score
-
-
-
- # async def calculate_confidence(self, state: CrawlState) -> float:
- # """Calculate learning score for adaptive crawling (used for stopping)"""
- # import numpy as np
-
- # if state.kb_embeddings is None or state.query_embeddings is None:
- # return 0.0
-
- # if len(state.kb_embeddings) == 0:
- # return 0.0
-
- # # Get cached distance matrix
- # distance_matrix = self._get_cached_distance_matrix(state.query_embeddings, state.kb_embeddings)
-
- # if distance_matrix is None:
- # return 0.0
-
- # # Vectorized analysis for all queries at once
- # all_query_metrics = []
-
- # for i in range(len(state.query_embeddings)):
- # # Get distances for this query
- # distances = distance_matrix[i]
- # sorted_distances = np.sort(distances)
-
- # # Store metrics for this query
- # query_metric = {
- # 'min_distance': sorted_distances[0],
- # 'top_3_distances': sorted_distances[:3],
- # 'top_5_distances': sorted_distances[:5],
- # 'close_neighbors': np.sum(distances < 0.3),
- # 'very_close_neighbors': np.sum(distances < 0.2),
- # 'all_distances': distances
- # }
- # all_query_metrics.append(query_metric)
-
- # # Hybrid approach with density (exponential base)
- # k_exp = self.config.embedding_k_exp if hasattr(self, 'config') else 1.0
- # coverage_scores_hybrid_exp = []
-
- # for metric in all_query_metrics:
- # # Base score from nearest neighbor
- # nearest_score = np.exp(-k_exp * metric['min_distance'])
-
- # # Top-k average (top 3)
- # top_k = min(3, len(metric['all_distances']))
- # top_k_avg = np.mean([np.exp(-k_exp * d) for d in metric['top_3_distances'][:top_k]])
-
- # # Combine using configured weights
- # nearest_weight = self.config.embedding_nearest_weight if hasattr(self, 'config') else 0.7
- # top_k_weight = self.config.embedding_top_k_weight if hasattr(self, 'config') else 0.3
- # hybrid_score = nearest_weight * nearest_score + top_k_weight * top_k_avg
- # coverage_scores_hybrid_exp.append(hybrid_score)
-
- # learning_score = np.mean(coverage_scores_hybrid_exp)
-
- # # Store as learning score
- # state.metrics['learning_score'] = learning_score
-
- # # Store embedding-specific metrics
- # state.metrics['avg_min_distance'] = np.mean([m['min_distance'] for m in all_query_metrics])
- # state.metrics['avg_close_neighbors'] = np.mean([m['close_neighbors'] for m in all_query_metrics])
- # state.metrics['avg_very_close_neighbors'] = np.mean([m['very_close_neighbors'] for m in all_query_metrics])
- # state.metrics['total_kb_docs'] = len(state.kb_embeddings)
-
- # # Store query-level metrics for detailed analysis
- # self._query_metrics = all_query_metrics
-
- # # For stopping criteria, return learning score
- # return float(learning_score)
-
- async def rank_links(self, state: CrawlState, config: AdaptiveConfig) -> List[Tuple[Link, float]]:
- """Main entry point for link ranking"""
- # Store config for use in other methods
- self.config = config
-
- # Filter out already crawled URLs and remove duplicates
- seen_urls = set()
- uncrawled_links = []
-
- for link in state.pending_links:
- if link.href not in state.crawled_urls and link.href not in seen_urls:
- uncrawled_links.append(link)
- seen_urls.add(link.href)
-
- if not uncrawled_links:
- return []
-
- # Get gaps in coverage (no threshold needed anymore)
- gaps = self.find_coverage_gaps(
- state.kb_embeddings,
- state.query_embeddings
- )
- state.semantic_gaps = [(g[0].tolist(), g[1]) for g in gaps] # Store as list for serialization
-
- # Select links that fill gaps (only from uncrawled)
- return await self.select_links_for_expansion(
- uncrawled_links,
- gaps,
- state.kb_embeddings
- )
-
- async def validate_coverage(self, state: CrawlState) -> float:
- """Validate coverage using held-out queries with caching"""
- if not hasattr(self, '_validation_queries') or not self._validation_queries:
- return state.metrics.get('confidence', 0.0)
-
- import numpy as np
-
- # Cache validation embeddings (only embed once!)
- if self._validation_embeddings_cache is None:
- self._validation_embeddings_cache = await self._get_embeddings(self._validation_queries)
-
- val_embeddings = self._validation_embeddings_cache
-
- # Use vectorized distance computation
- if state.kb_embeddings is None or len(state.kb_embeddings) == 0:
- return 0.0
-
- # Compute distance matrix for validation queries
- distance_matrix = self._compute_distance_matrix(val_embeddings, state.kb_embeddings)
-
- if distance_matrix is None:
- return 0.0
-
- # Find minimum distance for each validation query (vectorized)
- min_distances = np.min(distance_matrix, axis=1)
-
- # Compute scores using same exponential as training
- k_exp = self.config.embedding_k_exp if hasattr(self, 'config') else 1.0
- scores = np.exp(-k_exp * min_distances)
-
- validation_confidence = np.mean(scores)
- state.metrics['validation_confidence'] = validation_confidence
-
- return validation_confidence
-
- async def should_stop(self, state: CrawlState, config: AdaptiveConfig) -> bool:
- """Stop based on learning curve convergence"""
- confidence = state.metrics.get('confidence', 0.0)
-
- # Basic limits
- if len(state.crawled_urls) >= config.max_pages or not state.pending_links:
- return True
-
- # Track confidence history
- if not hasattr(state, 'confidence_history'):
- state.confidence_history = []
-
- state.confidence_history.append(confidence)
-
- # Need at least 3 iterations to check convergence
- if len(state.confidence_history) < 2:
- return False
-
- improvement_diffs = list(zip(state.confidence_history[:-1], state.confidence_history[1:]))
-
- # Calculate average improvement
- avg_improvement = sum(abs(b - a) for a, b in improvement_diffs) / len(improvement_diffs)
- state.metrics['avg_improvement'] = avg_improvement
-
- min_relative_improvement = self.config.embedding_min_relative_improvement * confidence if hasattr(self, 'config') else 0.1 * confidence
- if avg_improvement < min_relative_improvement:
- # Converged - validate before stopping
- val_score = await self.validate_coverage(state)
-
- # Only stop if validation is reasonable
- validation_min = self.config.embedding_validation_min_score if hasattr(self, 'config') else 0.4
- if val_score > validation_min:
- state.metrics['stopped_reason'] = 'converged_validated'
- self._validation_passed = True
- return True
- else:
- state.metrics['stopped_reason'] = 'low_validation'
- # Continue crawling despite convergence
-
- return False
-
- def get_quality_confidence(self, state: CrawlState) -> float:
- """Calculate quality-based confidence score for display"""
- learning_score = state.metrics.get('learning_score', 0.0)
- validation_score = state.metrics.get('validation_confidence', 0.0)
-
- # Get config values
- validation_min = self.config.embedding_validation_min_score if hasattr(self, 'config') else 0.4
- quality_min = self.config.embedding_quality_min_confidence if hasattr(self, 'config') else 0.7
- quality_max = self.config.embedding_quality_max_confidence if hasattr(self, 'config') else 0.95
- scale_factor = self.config.embedding_quality_scale_factor if hasattr(self, 'config') else 0.833
-
- if self._validation_passed and validation_score > validation_min:
- # Validated systems get boosted scores
- # Map 0.4-0.7 learning โ quality_min-quality_max confidence
- if learning_score < 0.4:
- confidence = quality_min # Minimum for validated systems
- elif learning_score > 0.7:
- confidence = quality_max # Maximum realistic confidence
- else:
- # Linear mapping in between
- confidence = quality_min + (learning_score - 0.4) * scale_factor
- else:
- # Not validated = conservative mapping
- confidence = learning_score * 0.8
-
- return confidence
-
- async def update_state(self, state: CrawlState, new_results: List[CrawlResult]) -> None:
- """Update embeddings and coverage metrics with deduplication"""
- from .utils import get_text_embeddings
- import numpy as np
-
- # Extract text from results
- new_texts = []
- valid_results = []
- for result in new_results:
- content = result.markdown.raw_markdown if hasattr(result, 'markdown') and result.markdown else ""
- if content: # Only process non-empty content
- new_texts.append(content[:5000]) # Limit text length
- valid_results.append(result)
-
- if not new_texts:
- return
-
- # Get embeddings for new texts
- embedding_llm_config = {
- 'provider': 'openai/text-embedding-3-small',
- 'api_token': os.getenv('OPENAI_API_KEY')
- }
- new_embeddings = await get_text_embeddings(new_texts, embedding_llm_config, self.embedding_model)
-
- # Deduplicate embeddings before adding to KB
- if state.kb_embeddings is None:
- # First batch - no deduplication needed
- state.kb_embeddings = new_embeddings
- deduplicated_indices = list(range(len(new_embeddings)))
- else:
- # Check for duplicates using vectorized similarity
- deduplicated_embeddings = []
- deduplicated_indices = []
-
- for i, new_emb in enumerate(new_embeddings):
- # Compute similarities with existing KB
- new_emb_normalized = new_emb / np.linalg.norm(new_emb)
- kb_normalized = state.kb_embeddings / np.linalg.norm(state.kb_embeddings, axis=1, keepdims=True)
- similarities = np.dot(kb_normalized, new_emb_normalized)
-
- # Only add if not too similar to existing content
- if np.max(similarities) < self._kb_similarity_threshold:
- deduplicated_embeddings.append(new_emb)
- deduplicated_indices.append(i)
-
- # Add deduplicated embeddings
- if deduplicated_embeddings:
- state.kb_embeddings = np.vstack([state.kb_embeddings, np.array(deduplicated_embeddings)])
-
- # Update crawl order only for non-duplicate results
- for idx in deduplicated_indices:
- state.crawl_order.append(valid_results[idx].url)
-
- # Invalidate distance matrix cache since KB changed
- self._kb_embeddings_hash = None
- self._distance_matrix_cache = None
-
- # Update coverage shape if needed
- if hasattr(state, 'query_embeddings') and state.query_embeddings is not None:
- state.coverage_shape = self.compute_coverage_shape(state.query_embeddings, self.config.alpha_shape_alpha if hasattr(self, 'config') else 0.5)
-
-
-class AdaptiveCrawler:
- """Main adaptive crawler that orchestrates the crawling process"""
-
- def __init__(self,
- crawler: Optional[AsyncWebCrawler] = None,
- config: Optional[AdaptiveConfig] = None,
- strategy: Optional[CrawlStrategy] = None):
- self.crawler = crawler
- self.config = config or AdaptiveConfig()
- self.config.validate()
-
- # Create strategy based on config
- if strategy:
- self.strategy = strategy
- else:
- self.strategy = self._create_strategy(self.config.strategy)
-
- # Initialize state
- self.state: Optional[CrawlState] = None
-
- # Track if we own the crawler (for cleanup)
- self._owns_crawler = crawler is None
-
- def _create_strategy(self, strategy_name: str) -> CrawlStrategy:
- """Create strategy instance based on name"""
- if strategy_name == "statistical":
- return StatisticalStrategy()
- elif strategy_name == "embedding":
- return EmbeddingStrategy(
- embedding_model=self.config.embedding_model,
- llm_config=self.config.embedding_llm_config
- )
- else:
- raise ValueError(f"Unknown strategy: {strategy_name}")
-
- async def digest(self,
- start_url: str,
- query: str,
- resume_from: Optional[str] = None) -> CrawlState:
- """Main entry point for adaptive crawling"""
- # Initialize or resume state
- if resume_from:
- self.state = CrawlState.load(resume_from)
- self.state.query = query # Update query in case it changed
- else:
- self.state = CrawlState(
- crawled_urls=set(),
- knowledge_base=[],
- pending_links=[],
- query=query,
- metrics={}
- )
-
- # Create crawler if needed
- if not self.crawler:
- self.crawler = AsyncWebCrawler()
- await self.crawler.__aenter__()
-
- self.strategy.config = self.config # Pass config to strategy
-
- # If using embedding strategy and not resuming, expand query space
- if isinstance(self.strategy, EmbeddingStrategy) and not resume_from:
- # Generate query space
- query_embeddings, expanded_queries = await self.strategy.map_query_semantic_space(
- query,
- self.config.n_query_variations
- )
- self.state.query_embeddings = query_embeddings
- self.state.expanded_queries = expanded_queries[1:] # Skip original query
- self.state.embedding_model = self.strategy.embedding_model
-
- try:
- # Initial crawl if not resuming
- if start_url not in self.state.crawled_urls:
- result = await self._crawl_with_preview(start_url, query)
- if result and hasattr(result, 'success') and result.success:
- self.state.knowledge_base.append(result)
- self.state.crawled_urls.add(start_url)
- # Extract links from result - handle both dict and Links object formats
- if hasattr(result, 'links') and result.links:
- if isinstance(result.links, dict):
- # Extract internal and external links from dict
- internal_links = [Link(**link) for link in result.links.get('internal', [])]
- external_links = [Link(**link) for link in result.links.get('external', [])]
- self.state.pending_links.extend(internal_links + external_links)
- else:
- # Handle Links object
- self.state.pending_links.extend(result.links.internal + result.links.external)
-
- # Update state
- await self.strategy.update_state(self.state, [result])
-
- # adaptive expansion
- depth = 0
- while depth < self.config.max_depth:
- # Calculate confidence
- confidence = await self.strategy.calculate_confidence(self.state)
- self.state.metrics['confidence'] = confidence
-
- # Check stopping criteria
- if await self.strategy.should_stop(self.state, self.config):
- break
-
- # Rank candidate links
- ranked_links = await self.strategy.rank_links(self.state, self.config)
-
- if not ranked_links:
- break
-
- # Check minimum gain threshold
- if ranked_links[0][1] < self.config.min_gain_threshold:
- break
-
- # Select top K links
- to_crawl = [(link, score) for link, score in ranked_links[:self.config.top_k_links]
- if link.href not in self.state.crawled_urls]
-
- if not to_crawl:
- break
-
- # Crawl selected links
- new_results = await self._crawl_batch(to_crawl, query)
-
- if new_results:
- # Update knowledge base
- self.state.knowledge_base.extend(new_results)
-
- # Update crawled URLs and pending links
- for result, (link, _) in zip(new_results, to_crawl):
- if result:
- self.state.crawled_urls.add(link.href)
- # Extract links from result - handle both dict and Links object formats
- if hasattr(result, 'links') and result.links:
- new_links = []
- if isinstance(result.links, dict):
- # Extract internal and external links from dict
- internal_links = [Link(**link_data) for link_data in result.links.get('internal', [])]
- external_links = [Link(**link_data) for link_data in result.links.get('external', [])]
- new_links = internal_links + external_links
- else:
- # Handle Links object
- new_links = result.links.internal + result.links.external
-
- # Add new links to pending
- for new_link in new_links:
- if new_link.href not in self.state.crawled_urls:
- self.state.pending_links.append(new_link)
-
- # Update state with new results
- await self.strategy.update_state(self.state, new_results)
-
- depth += 1
-
- # Save state if configured
- if self.config.save_state and self.config.state_path:
- self.state.save(self.config.state_path)
-
- # Final confidence calculation
- learning_score = await self.strategy.calculate_confidence(self.state)
-
- # For embedding strategy, get quality-based confidence
- if isinstance(self.strategy, EmbeddingStrategy):
- self.state.metrics['confidence'] = self.strategy.get_quality_confidence(self.state)
- else:
- # For statistical strategy, use the same as before
- self.state.metrics['confidence'] = learning_score
-
- self.state.metrics['pages_crawled'] = len(self.state.crawled_urls)
- self.state.metrics['depth_reached'] = depth
-
- # Final save
- if self.config.save_state and self.config.state_path:
- self.state.save(self.config.state_path)
-
- return self.state
-
- finally:
- # Cleanup if we created the crawler
- if self._owns_crawler and self.crawler:
- await self.crawler.__aexit__(None, None, None)
-
- async def _crawl_with_preview(self, url: str, query: str) -> Optional[CrawlResult]:
- """Crawl a URL with link preview enabled"""
- config = CrawlerRunConfig(
- link_preview_config=LinkPreviewConfig(
- include_internal=True,
- include_external=False,
- query=query, # For BM25 scoring
- concurrency=5,
- timeout=5,
- max_links=50, # Reasonable limit
- verbose=False
- ),
- score_links=True # Enable intrinsic scoring
- )
-
- try:
- result = await self.crawler.arun(url=url, config=config)
- # Extract the actual CrawlResult from the container
- if hasattr(result, '_results') and result._results:
- result = result._results[0]
-
- # Filter our all links do not have head_date
- if hasattr(result, 'links') and result.links:
- result.links['internal'] = [link for link in result.links['internal'] if link.get('head_data')]
- # For now let's ignore external links without head_data
- # result.links['external'] = [link for link in result.links['external'] if link.get('head_data')]
-
- return result
- except Exception as e:
- print(f"Error crawling {url}: {e}")
- return None
-
- async def _crawl_batch(self, links_with_scores: List[Tuple[Link, float]], query: str) -> List[CrawlResult]:
- """Crawl multiple URLs in parallel"""
- tasks = []
- for link, score in links_with_scores:
- task = self._crawl_with_preview(link.href, query)
- tasks.append(task)
-
- results = await asyncio.gather(*tasks, return_exceptions=True)
-
- # Filter out exceptions and failed crawls
- valid_results = []
- for result in results:
- if isinstance(result, CrawlResult):
- # Only include successful crawls
- if hasattr(result, 'success') and result.success:
- valid_results.append(result)
- else:
- print(f"Skipping failed crawl: {result.url if hasattr(result, 'url') else 'unknown'}")
- elif isinstance(result, Exception):
- print(f"Error in batch crawl: {result}")
-
- return valid_results
-
- # Status properties
- @property
- def confidence(self) -> float:
- """Current confidence level"""
- if self.state:
- return self.state.metrics.get('confidence', 0.0)
- return 0.0
-
- @property
- def coverage_stats(self) -> Dict[str, Any]:
- """Detailed coverage statistics"""
- if not self.state:
- return {}
-
- total_content_length = sum(
- len(result.markdown.raw_markdown or "")
- for result in self.state.knowledge_base
- )
-
- return {
- 'pages_crawled': len(self.state.crawled_urls),
- 'total_content_length': total_content_length,
- 'unique_terms': len(self.state.term_frequencies),
- 'total_terms': sum(self.state.term_frequencies.values()),
- 'pending_links': len(self.state.pending_links),
- 'confidence': self.confidence,
- 'coverage': self.state.metrics.get('coverage', 0.0),
- 'consistency': self.state.metrics.get('consistency', 0.0),
- 'saturation': self.state.metrics.get('saturation', 0.0)
- }
-
- @property
- def is_sufficient(self) -> bool:
- """Check if current knowledge is sufficient"""
- if isinstance(self.strategy, EmbeddingStrategy):
- # For embedding strategy, sufficient = validation passed
- return self.strategy._validation_passed
- else:
- # For statistical strategy, use threshold
- return self.confidence >= self.config.confidence_threshold
-
- def print_stats(self, detailed: bool = False) -> None:
- """Print comprehensive statistics about the knowledge base
-
- Args:
- detailed: If True, show detailed statistics including top terms
- """
- if not self.state:
- print("No crawling state available.")
- return
-
- # Import here to avoid circular imports
- try:
- from rich.console import Console
- from rich.table import Table
- console = Console()
- use_rich = True
- except ImportError:
- use_rich = False
-
- if not detailed and use_rich:
- # Summary view with nice table (like original)
- table = Table(title=f"Adaptive Crawl Stats - Query: '{self.state.query}'")
- table.add_column("Metric", style="cyan", no_wrap=True)
- table.add_column("Value", style="magenta")
-
- # Basic stats
- stats = self.coverage_stats
- table.add_row("Pages Crawled", str(stats.get('pages_crawled', 0)))
- table.add_row("Unique Terms", str(stats.get('unique_terms', 0)))
- table.add_row("Total Terms", str(stats.get('total_terms', 0)))
- table.add_row("Content Length", f"{stats.get('total_content_length', 0):,} chars")
- table.add_row("Pending Links", str(stats.get('pending_links', 0)))
- table.add_row("", "") # Spacer
-
- # Strategy-specific metrics
- if isinstance(self.strategy, EmbeddingStrategy):
- # Embedding-specific metrics
- table.add_row("Confidence", f"{stats.get('confidence', 0):.2%}")
- table.add_row("Avg Min Distance", f"{self.state.metrics.get('avg_min_distance', 0):.3f}")
- table.add_row("Avg Close Neighbors", f"{self.state.metrics.get('avg_close_neighbors', 0):.1f}")
- table.add_row("Validation Score", f"{self.state.metrics.get('validation_confidence', 0):.2%}")
- table.add_row("", "") # Spacer
- table.add_row("Is Sufficient?", "[green]Yes (Validated)[/green]" if self.is_sufficient else "[red]No[/red]")
- else:
- # Statistical strategy metrics
- table.add_row("Confidence", f"{stats.get('confidence', 0):.2%}")
- table.add_row("Coverage", f"{stats.get('coverage', 0):.2%}")
- table.add_row("Consistency", f"{stats.get('consistency', 0):.2%}")
- table.add_row("Saturation", f"{stats.get('saturation', 0):.2%}")
- table.add_row("", "") # Spacer
- table.add_row("Is Sufficient?", "[green]Yes[/green]" if self.is_sufficient else "[red]No[/red]")
-
- console.print(table)
- else:
- # Detailed view or fallback when rich not available
- print("\n" + "="*80)
- print(f"Adaptive Crawl Statistics - Query: '{self.state.query}'")
- print("="*80)
-
- # Basic stats
- print("\n[*] Basic Statistics:")
- print(f" Pages Crawled: {len(self.state.crawled_urls)}")
- print(f" Pending Links: {len(self.state.pending_links)}")
- print(f" Total Documents: {self.state.total_documents}")
-
- # Content stats
- total_content_length = sum(
- len(self._get_content_from_result(result))
- for result in self.state.knowledge_base
- )
- total_words = sum(self.state.term_frequencies.values())
- unique_terms = len(self.state.term_frequencies)
-
- print(f"\n[*] Content Statistics:")
- print(f" Total Content: {total_content_length:,} characters")
- print(f" Total Words: {total_words:,}")
- print(f" Unique Terms: {unique_terms:,}")
- if total_words > 0:
- print(f" Vocabulary Richness: {unique_terms/total_words:.2%}")
-
- # Strategy-specific output
- if isinstance(self.strategy, EmbeddingStrategy):
- # Semantic coverage for embedding strategy
- print(f"\n[*] Semantic Coverage Analysis:")
- print(f" Average Min Distance: {self.state.metrics.get('avg_min_distance', 0):.3f}")
- print(f" Avg Close Neighbors (< 0.3): {self.state.metrics.get('avg_close_neighbors', 0):.1f}")
- print(f" Avg Very Close Neighbors (< 0.2): {self.state.metrics.get('avg_very_close_neighbors', 0):.1f}")
-
- # Confidence metrics
- print(f"\n[*] Confidence Metrics:")
- if self.is_sufficient:
- if use_rich:
- console.print(f" Overall Confidence: {self.confidence:.2%} [green][VALIDATED][/green]")
- else:
- print(f" Overall Confidence: {self.confidence:.2%} [VALIDATED]")
- else:
- if use_rich:
- console.print(f" Overall Confidence: {self.confidence:.2%} [red][NOT VALIDATED][/red]")
- else:
- print(f" Overall Confidence: {self.confidence:.2%} [NOT VALIDATED]")
-
- print(f" Learning Score: {self.state.metrics.get('learning_score', 0):.2%}")
- print(f" Validation Score: {self.state.metrics.get('validation_confidence', 0):.2%}")
-
- else:
- # Query coverage for statistical strategy
- print(f"\n[*] Query Coverage:")
- query_terms = self.strategy._tokenize(self.state.query.lower())
- for term in query_terms:
- tf = self.state.term_frequencies.get(term, 0)
- df = self.state.document_frequencies.get(term, 0)
- if df > 0:
- if use_rich:
- console.print(f" '{term}': found in {df}/{self.state.total_documents} docs ([green]{df/self.state.total_documents:.0%}[/green]), {tf} occurrences")
- else:
- print(f" '{term}': found in {df}/{self.state.total_documents} docs ({df/self.state.total_documents:.0%}), {tf} occurrences")
- else:
- if use_rich:
- console.print(f" '{term}': [red][X] not found[/red]")
- else:
- print(f" '{term}': [X] not found")
-
- # Confidence metrics
- print(f"\n[*] Confidence Metrics:")
- status = "[OK]" if self.is_sufficient else "[!!]"
- if use_rich:
- status_colored = "[green][OK][/green]" if self.is_sufficient else "[red][!!][/red]"
- console.print(f" Overall Confidence: {self.confidence:.2%} {status_colored}")
- else:
- print(f" Overall Confidence: {self.confidence:.2%} {status}")
- print(f" Coverage Score: {self.state.metrics.get('coverage', 0):.2%}")
- print(f" Consistency Score: {self.state.metrics.get('consistency', 0):.2%}")
- print(f" Saturation Score: {self.state.metrics.get('saturation', 0):.2%}")
-
- # Crawl efficiency
- if self.state.new_terms_history:
- avg_new_terms = sum(self.state.new_terms_history) / len(self.state.new_terms_history)
- print(f"\n[*] Crawl Efficiency:")
- print(f" Avg New Terms per Page: {avg_new_terms:.1f}")
- print(f" Information Saturation: {self.state.metrics.get('saturation', 0):.2%}")
-
- if detailed:
- print("\n" + "-"*80)
- if use_rich:
- console.print("[bold cyan]DETAILED STATISTICS[/bold cyan]")
- else:
- print("DETAILED STATISTICS")
- print("-"*80)
-
- # Top terms
- print("\n[+] Top 20 Terms by Frequency:")
- top_terms = sorted(self.state.term_frequencies.items(), key=lambda x: x[1], reverse=True)[:20]
- for i, (term, freq) in enumerate(top_terms, 1):
- df = self.state.document_frequencies.get(term, 0)
- if use_rich:
- console.print(f" {i:2d}. [yellow]'{term}'[/yellow]: {freq} occurrences in {df} docs")
- else:
- print(f" {i:2d}. '{term}': {freq} occurrences in {df} docs")
-
- # URLs crawled
- print(f"\n[+] URLs Crawled ({len(self.state.crawled_urls)}):")
- for i, url in enumerate(self.state.crawl_order, 1):
- new_terms = self.state.new_terms_history[i-1] if i <= len(self.state.new_terms_history) else 0
- if use_rich:
- console.print(f" {i}. [cyan]{url}[/cyan]")
- console.print(f" -> Added [green]{new_terms}[/green] new terms")
- else:
- print(f" {i}. {url}")
- print(f" -> Added {new_terms} new terms")
-
- # Document frequency distribution
- print("\n[+] Document Frequency Distribution:")
- df_counts = {}
- for df in self.state.document_frequencies.values():
- df_counts[df] = df_counts.get(df, 0) + 1
-
- for df in sorted(df_counts.keys()):
- count = df_counts[df]
- print(f" Terms in {df} docs: {count} terms")
-
- # Embedding stats
- if self.state.embedding_model:
- print("\n[+] Semantic Coverage Analysis:")
- print(f" Embedding Model: {self.state.embedding_model}")
- print(f" Query Variations: {len(self.state.expanded_queries)}")
- if self.state.kb_embeddings is not None:
- print(f" Knowledge Embeddings: {self.state.kb_embeddings.shape}")
- else:
- print(f" Knowledge Embeddings: None")
- print(f" Semantic Gaps: {len(self.state.semantic_gaps)}")
- print(f" Coverage Achievement: {self.confidence:.2%}")
-
- # Show sample expanded queries
- if self.state.expanded_queries:
- print("\n[+] Query Space (samples):")
- for i, eq in enumerate(self.state.expanded_queries[:5], 1):
- if use_rich:
- console.print(f" {i}. [yellow]{eq}[/yellow]")
- else:
- print(f" {i}. {eq}")
-
- print("\n" + "="*80)
-
- def _get_content_from_result(self, result) -> str:
- """Helper to safely extract content from result"""
- if hasattr(result, 'markdown') and result.markdown:
- if hasattr(result.markdown, 'raw_markdown'):
- return result.markdown.raw_markdown or ""
- return str(result.markdown)
- return ""
-
- def export_knowledge_base(self, filepath: Union[str, Path], format: str = "jsonl") -> None:
- """Export the knowledge base to a file
-
- Args:
- filepath: Path to save the file
- format: Export format - currently supports 'jsonl'
- """
- if not self.state or not self.state.knowledge_base:
- print("No knowledge base to export.")
- return
-
- filepath = Path(filepath)
- filepath.parent.mkdir(parents=True, exist_ok=True)
-
- if format == "jsonl":
- # Export as JSONL - one CrawlResult per line
- with open(filepath, 'w', encoding='utf-8') as f:
- for result in self.state.knowledge_base:
- # Convert CrawlResult to dict
- result_dict = self._crawl_result_to_export_dict(result)
- # Write as single line JSON
- f.write(json.dumps(result_dict, ensure_ascii=False) + '\n')
-
- print(f"Exported {len(self.state.knowledge_base)} documents to {filepath}")
- else:
- raise ValueError(f"Unsupported export format: {format}")
-
- def _crawl_result_to_export_dict(self, result) -> Dict[str, Any]:
- """Convert CrawlResult to a dictionary for export"""
- # Extract all available fields
- export_dict = {
- 'url': getattr(result, 'url', ''),
- 'timestamp': getattr(result, 'timestamp', None),
- 'success': getattr(result, 'success', True),
- 'query': self.state.query if self.state else '',
- }
-
- # Extract content
- if hasattr(result, 'markdown') and result.markdown:
- if hasattr(result.markdown, 'raw_markdown'):
- export_dict['content'] = result.markdown.raw_markdown
- else:
- export_dict['content'] = str(result.markdown)
- else:
- export_dict['content'] = ''
-
- # Extract metadata
- if hasattr(result, 'metadata'):
- export_dict['metadata'] = result.metadata
-
- # Extract links if available
- if hasattr(result, 'links'):
- export_dict['links'] = result.links
-
- # Add crawl-specific metadata
- if self.state:
- export_dict['crawl_metadata'] = {
- 'crawl_order': self.state.crawl_order.index(export_dict['url']) + 1 if export_dict['url'] in self.state.crawl_order else 0,
- 'confidence_at_crawl': self.state.metrics.get('confidence', 0),
- 'total_documents': self.state.total_documents
- }
-
- return export_dict
-
- def import_knowledge_base(self, filepath: Union[str, Path], format: str = "jsonl") -> None:
- """Import a knowledge base from a file
-
- Args:
- filepath: Path to the file to import
- format: Import format - currently supports 'jsonl'
- """
- filepath = Path(filepath)
- if not filepath.exists():
- raise FileNotFoundError(f"File not found: {filepath}")
-
- if format == "jsonl":
- imported_results = []
- with open(filepath, 'r', encoding='utf-8') as f:
- for line in f:
- if line.strip():
- data = json.loads(line)
- # Convert back to a mock CrawlResult
- mock_result = self._import_dict_to_crawl_result(data)
- imported_results.append(mock_result)
-
- # Initialize state if needed
- if not self.state:
- self.state = CrawlState()
-
- # Add imported results
- self.state.knowledge_base.extend(imported_results)
-
- # Update state with imported data
- asyncio.run(self.strategy.update_state(self.state, imported_results))
-
- print(f"Imported {len(imported_results)} documents from {filepath}")
- else:
- raise ValueError(f"Unsupported import format: {format}")
-
- def _import_dict_to_crawl_result(self, data: Dict[str, Any]):
- """Convert imported dict back to a mock CrawlResult"""
- class MockMarkdown:
- def __init__(self, content):
- self.raw_markdown = content
-
- class MockCrawlResult:
- def __init__(self, data):
- self.url = data.get('url', '')
- self.markdown = MockMarkdown(data.get('content', ''))
- self.links = data.get('links', {})
- self.metadata = data.get('metadata', {})
- self.success = data.get('success', True)
- self.timestamp = data.get('timestamp')
-
- return MockCrawlResult(data)
-
- def get_relevant_content(self, top_k: int = 5) -> List[Dict[str, Any]]:
- """Get most relevant content for the query"""
- if not self.state or not self.state.knowledge_base:
- return []
-
- # Simple relevance ranking based on term overlap
- scored_docs = []
- query_terms = set(self.state.query.lower().split())
-
- for i, result in enumerate(self.state.knowledge_base):
- content = (result.markdown.raw_markdown or "").lower()
- content_terms = set(content.split())
-
- # Calculate relevance score
- overlap = len(query_terms & content_terms)
- score = overlap / len(query_terms) if query_terms else 0.0
-
- scored_docs.append({
- 'url': result.url,
- 'score': score,
- 'content': result.markdown.raw_markdown,
- 'index': i
- })
-
- # Sort by score and return top K
- scored_docs.sort(key=lambda x: x['score'], reverse=True)
- return scored_docs[:top_k]
\ No newline at end of file
diff --git a/crawl4ai/async_configs.py b/crawl4ai/async_configs.py
index 27320fd4b..8b909731d 100644
--- a/crawl4ai/async_configs.py
+++ b/crawl4ai/async_configs.py
@@ -11,7 +11,7 @@
IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD,
PROVIDER_MODELS,
PROVIDER_MODELS_PREFIXES,
- SCREENSHOT_HEIGHT_TRESHOLD,
+ SCREENSHOT_HEIGHT_THRESHOLD,
PAGE_TIMEOUT,
IMAGE_SCORE_THRESHOLD,
SOCIAL_MEDIA_DOMAINS,
@@ -897,9 +897,9 @@ def __init__(
self.memory_saving_mode = memory_saving_mode
self.max_pages_before_recycle = max_pages_before_recycle
- fa_user_agenr_generator = ValidUAGenerator()
+ ua_generator = ValidUAGenerator()
if self.user_agent_mode == "random":
- self.user_agent = fa_user_agenr_generator.generate(
+ self.user_agent = ua_generator.generate(
**(self.user_agent_generator_config or {})
)
else:
@@ -1496,7 +1496,7 @@ class CrawlerRunConfig():
screenshot_wait_for (float or None): Additional wait time before taking a screenshot.
Default: None.
screenshot_height_threshold (int): Threshold for page height to decide screenshot strategy.
- Default: SCREENSHOT_HEIGHT_TRESHOLD (from config, e.g. 20000).
+ Default: SCREENSHOT_HEIGHT_THRESHOLD (from config, e.g. 20000).
force_viewport_screenshot (bool): If True, always take viewport-only screenshots regardless of page height.
When False, uses automatic decision (viewport for short pages, full-page for long pages).
Default: False.
@@ -1530,8 +1530,6 @@ class CrawlerRunConfig():
Default: False.
exclude_domains (list of str): List of specific domains to exclude from results.
Default: [].
- exclude_internal_links (bool): If True, exclude internal links from the results.
- Default: False.
score_links (bool): If True, calculate intrinsic quality scores for all links using URL structure,
text quality, and contextual relevance metrics. Separate from link_preview_config.
Default: False.
@@ -1654,7 +1652,7 @@ def __init__(
# Media Handling Parameters
screenshot: bool = False,
screenshot_wait_for: float = None,
- screenshot_height_threshold: int = SCREENSHOT_HEIGHT_TRESHOLD,
+ screenshot_height_threshold: int = SCREENSHOT_HEIGHT_THRESHOLD,
force_viewport_screenshot: bool = False,
pdf: bool = False,
capture_mhtml: bool = False,
diff --git a/crawl4ai/async_crawler_strategy.py b/crawl4ai/async_crawler_strategy.py
index b3710b20d..ceb39dbfc 100644
--- a/crawl4ai/async_crawler_strategy.py
+++ b/crawl4ai/async_crawler_strategy.py
@@ -16,7 +16,7 @@
import uuid
from .js_snippet import load_js_script
from .models import AsyncCrawlResponse
-from .config import SCREENSHOT_HEIGHT_TRESHOLD
+from .config import SCREENSHOT_HEIGHT_THRESHOLD
from .async_configs import BrowserConfig, CrawlerRunConfig, HTTPCrawlerConfig
from .async_logger import AsyncLogger
from .ssl_certificate import SSLCertificate
@@ -919,7 +919,7 @@ async def handle_request_failed_capture(request):
# )
target_width = self.browser_config.viewport_width
- target_height = int(target_width * page_width / page_height * 0.95)
+ target_height = int(target_width * page_height / page_width)
await page.set_viewport_size(
{"width": target_width, "height": target_height}
)
@@ -1878,7 +1878,7 @@ async def take_screenshot_scroller(self, page: Page, **kwargs) -> str:
# Set a large viewport
large_viewport_height = min(
page_height,
- kwargs.get("screenshot_height_threshold", SCREENSHOT_HEIGHT_TRESHOLD),
+ kwargs.get("screenshot_height_threshold", SCREENSHOT_HEIGHT_THRESHOLD),
)
await page.set_viewport_size(
{"width": page_width, "height": large_viewport_height}
@@ -2258,14 +2258,6 @@ async def execute_user_script(
)
return {"success": False, "error": str(e)}
- except Exception as e:
- self.logger.error(
- message="Script execution failed: {error}",
- tag="JS_EXEC",
- params={"error": str(e)},
- )
- return {"success": False, "error": str(e)}
-
async def check_visibility(self, page):
"""
Checks if an element is visible on the page.
diff --git a/crawl4ai/async_database.py b/crawl4ai/async_database.py
index 757905352..df17219f5 100644
--- a/crawl4ai/async_database.py
+++ b/crawl4ai/async_database.py
@@ -332,13 +332,44 @@ async def _get(db):
else:
row_dict[field] = ""
+ # Handle markdown separately (stored as raw text, not JSON-serialized)
+ raw_md = row_dict.get("markdown", "")
+ if raw_md:
+ try:
+ parsed = json.loads(raw_md)
+ if isinstance(parsed, dict):
+ row_dict["markdown"] = parsed
+ else:
+ row_dict["markdown"] = MarkdownGenerationResult(
+ raw_markdown=str(parsed) if parsed is not None else "",
+ markdown_with_citations="",
+ references_markdown="",
+ fit_markdown="",
+ fit_html="",
+ )
+ except (json.JSONDecodeError, ValueError):
+ row_dict["markdown"] = MarkdownGenerationResult(
+ raw_markdown=raw_md,
+ markdown_with_citations="",
+ references_markdown="",
+ fit_markdown="",
+ fit_html="",
+ )
+ else:
+ row_dict["markdown"] = MarkdownGenerationResult(
+ raw_markdown="",
+ markdown_with_citations="",
+ references_markdown="",
+ fit_markdown="",
+ fit_html="",
+ )
+
# Parse JSON fields
json_fields = [
"media",
"links",
"metadata",
"response_headers",
- "markdown",
]
for field in json_fields:
try:
@@ -346,21 +377,7 @@ async def _get(db):
json.loads(row_dict[field]) if row_dict[field] else {}
)
except json.JSONDecodeError:
- # Very UGLY, never mention it to me please
- if field == "markdown" and isinstance(row_dict[field], str):
- row_dict[field] = MarkdownGenerationResult(
- raw_markdown=row_dict[field] or "",
- markdown_with_citations="",
- references_markdown="",
- fit_markdown="",
- fit_html="",
- )
- else:
- row_dict[field] = {}
-
- if isinstance(row_dict["markdown"], Dict):
- if row_dict["markdown"].get("raw_markdown"):
- row_dict["markdown"] = row_dict["markdown"]["raw_markdown"]
+ row_dict[field] = {}
# Parse downloaded_files
try:
@@ -505,7 +522,11 @@ async def acache_url(self, result: CrawlResult):
)
else:
content_map["markdown"] = (
- MarkdownGenerationResult().model_dump_json(),
+ MarkdownGenerationResult(
+ raw_markdown="",
+ markdown_with_citations="",
+ references_markdown="",
+ ).model_dump_json(),
"markdown",
)
except Exception as e:
@@ -514,7 +535,11 @@ async def acache_url(self, result: CrawlResult):
)
# Fallback to empty markdown result
content_map["markdown"] = (
- MarkdownGenerationResult().model_dump_json(),
+ MarkdownGenerationResult(
+ raw_markdown="",
+ markdown_with_citations="",
+ references_markdown="",
+ ).model_dump_json(),
"markdown",
)
diff --git a/crawl4ai/config.py b/crawl4ai/config.py
index 5d394136f..d4caed2ed 100644
--- a/crawl4ai/config.py
+++ b/crawl4ai/config.py
@@ -99,7 +99,8 @@
NEED_MIGRATION = True
URL_LOG_SHORTEN_LENGTH = 30
SHOW_DEPRECATION_WARNINGS = True
-SCREENSHOT_HEIGHT_TRESHOLD = 10000
+SCREENSHOT_HEIGHT_THRESHOLD = 10000
+SCREENSHOT_HEIGHT_TRESHOLD = SCREENSHOT_HEIGHT_THRESHOLD # Backward compat alias
PAGE_TIMEOUT = 60000
DOWNLOAD_PAGE_TIMEOUT = 60000
diff --git a/crawl4ai/models.py b/crawl4ai/models.py
index 506538970..4592b0abd 100644
--- a/crawl4ai/models.py
+++ b/crawl4ai/models.py
@@ -179,11 +179,14 @@ def __init__(self, **data):
markdown_result = data.pop('markdown', None)
super().__init__(**data)
if markdown_result is not None:
- self._markdown = (
- MarkdownGenerationResult(**markdown_result)
- if isinstance(markdown_result, dict)
- else markdown_result
- )
+ if isinstance(markdown_result, dict):
+ self._markdown = MarkdownGenerationResult(**markdown_result)
+ elif isinstance(markdown_result, MarkdownGenerationResult):
+ self._markdown = markdown_result
+ else:
+ self._markdown = MarkdownGenerationResult(
+ raw_markdown=str(markdown_result) if markdown_result is not None else ""
+ )
@property
def markdown(self):
diff --git a/crawl4ai/utils.py b/crawl4ai/utils.py
index 89fb782d9..714149990 100644
--- a/crawl4ai/utils.py
+++ b/crawl4ai/utils.py
@@ -1113,7 +1113,6 @@ def flatten_nested_elements(node):
# sanitized_html = escape_json_string(cleaned_html)
# Convert cleaned HTML to Markdown
- h = html2text.HTML2Text()
h = CustomHTML2Text()
h.ignore_links = True
markdown = h.handle(cleaned_html)
@@ -2212,26 +2211,6 @@ def fast_format_html(html_string):
return "\n".join(formatted)
-def normalize_url(href, base_url):
- """Normalize URLs to ensure consistent format"""
- from urllib.parse import urljoin, urlparse
-
- # Parse base URL to get components
- parsed_base = urlparse(base_url)
- if not parsed_base.scheme or not parsed_base.netloc:
- raise ValueError(f"Invalid base URL format: {base_url}")
-
- if parsed_base.scheme.lower() not in ["http", "https"]:
- # Handle special protocols
- raise ValueError(f"Invalid base URL format: {base_url}")
- cleaned_href = href.strip()
-
- # Use urljoin to handle all cases
- return urljoin(base_url, cleaned_href)
-
-
-
-
def normalize_url(
href: str,
base_url: str,
@@ -2431,42 +2410,6 @@ def efficient_normalize_url_for_deep_crawl(href, base_url, preserve_https=False,
return normalized
-def normalize_url_tmp(href, base_url):
- """Normalize URLs to ensure consistent format"""
- # Extract protocol and domain from base URL
- try:
- base_parts = base_url.split("/")
- protocol = base_parts[0]
- domain = base_parts[2]
- except IndexError:
- raise ValueError(f"Invalid base URL format: {base_url}")
-
- # Handle special protocols
- special_protocols = {"mailto:", "tel:", "ftp:", "file:", "data:", "javascript:"}
- if any(href.lower().startswith(proto) for proto in special_protocols):
- return href.strip()
-
- # Handle anchor links
- if href.startswith("#"):
- return f"{base_url}{href}"
-
- # Handle protocol-relative URLs
- if href.startswith("//"):
- return f"{protocol}{href}"
-
- # Handle root-relative URLs
- if href.startswith("/"):
- return f"{protocol}//{domain}{href}"
-
- # Handle relative URLs
- if not href.startswith(("http://", "https://")):
- # Remove leading './' if present
- href = href.lstrip("./")
- return f"{protocol}//{domain}/{href}"
-
- return href.strip()
-
-
def quick_extract_links(html: str, base_url: str) -> Dict[str, List[Dict[str, str]]]:
"""
Fast link extraction for prefetch mode.
@@ -2989,7 +2932,7 @@ def configure_windows_event_loop():
Example:
```python
- from crawl4ai.async_configs import configure_windows_event_loop
+ from crawl4ai.utils import configure_windows_event_loop
# Call this before any async operations if you're on Windows
configure_windows_event_loop()
diff --git a/deploy/docker/README.md b/deploy/docker/README.md
index a711c7206..a377943a7 100644
--- a/deploy/docker/README.md
+++ b/deploy/docker/README.md
@@ -1,4 +1,4 @@
-# Crawl4AI Docker Guide ๐ณ
+# Crawl4AI Docker Guide
## Table of Contents
- [Prerequisites](#prerequisites)
@@ -47,7 +47,7 @@ Before we dive in, make sure you have:
- Python 3.10+ (if using the Python SDK).
- Node.js 16+ (if using the Node.js examples).
-> ๐ก **Pro tip**: Run `docker info` to check your Docker installation and available resources.
+> **Pro tip**: Run `docker info` to check your Docker installation and available resources.
## Installation
@@ -90,7 +90,7 @@ ANTHROPIC_API_KEY=your-anthropic-key
# GEMINI_API_TOKEN=your-gemini-token
EOL
```
-> ๐ **Note**: Keep your API keys secure! Never commit `.llm.env` to version control.
+> **Note**: Keep your API keys secure! Never commit `.llm.env` to version control.
#### 3. Run the Container
@@ -833,7 +833,7 @@ curl -X POST http://localhost:11235/llm/job \
- Single URL only (not an array)
- Supports schema-based extraction with `schema` parameter
-> ๐ก **Pro tip**: See [WEBHOOK_EXAMPLES.md](./WEBHOOK_EXAMPLES.md) for detailed examples including TypeScript client code, Flask webhook handlers, and failure handling.
+> **Pro tip**: See [WEBHOOK_EXAMPLES.md](./WEBHOOK_EXAMPLES.md) for detailed examples including TypeScript client code, Flask webhook handlers, and failure handling.
---
@@ -975,7 +975,7 @@ You can override the default `config.yml`.
```
*(Note: Ensure `my-custom-config.yml` is in the same directory as `docker-compose.yml`)*
-> ๐ก When mounting, your custom file *completely replaces* the default one. Ensure it's a valid and complete configuration.
+> When mounting, your custom file *completely replaces* the default one. Ensure it's a valid and complete configuration.
### Configuration Recommendations
@@ -985,17 +985,17 @@ You can override the default `config.yml`.
- Set up proper rate limiting to protect your server
- Consider your environment before enabling HTTPS redirect
-2. **Resource Management** ๐ป
+2. **Resource Management**
- Adjust memory_threshold_percent based on available RAM
- Set timeouts according to your content size and network conditions
- Use Redis for rate limiting in multi-container setups
-3. **Monitoring** ๐
+3. **Monitoring**
- Enable Prometheus if you need metrics
- Set DEBUG logging in development, INFO in production
- Regular health check monitoring is crucial
-4. **Performance Tuning** โก
+4. **Performance Tuning**
- Start with conservative rate limiter delays
- Increase batch_process timeout for large content
- Adjust stream_init timeout based on initial response times
@@ -1004,9 +1004,9 @@ You can override the default `config.yml`.
We're here to help you succeed with Crawl4AI! Here's how to get support:
-- ๐ Check our [full documentation](https://docs.crawl4ai.com)
-- ๐ Found a bug? [Open an issue](https://github.com/unclecode/crawl4ai/issues)
-- ๐ฌ Join our [Discord community](https://discord.gg/crawl4ai)
+- Check our [full documentation](https://docs.crawl4ai.com)
+- Found a bug? [Open an issue](https://github.com/unclecode/crawl4ai/issues)
+- Join our [Discord community](https://discord.gg/crawl4ai)
- โญ Star us on GitHub to show support!
## Summary
@@ -1030,4 +1030,4 @@ Remember, the examples in the `examples` folder are your friends - they show rea
Keep exploring, and don't hesitate to reach out if you need help! We're building something amazing together. ๐
-Happy crawling! ๐ท๏ธ
+Happy crawling!
diff --git a/deploy/docker/server.py b/deploy/docker/server.py
index 1bc05d833..1008a9f1d 100644
--- a/deploy/docker/server.py
+++ b/deploy/docker/server.py
@@ -342,6 +342,16 @@ def _setup_security(app_: FastAPI):
_setup_security(app)
if config["observability"]["prometheus"]["enabled"]:
+ # Workaround: _IncludedRouter objects lack .path, causing AttributeError
+ # in prometheus_fastapi_instrumentator's routing introspection.
+ import prometheus_fastapi_instrumentator.routing as _pr
+ _orig = _pr._get_route_name
+ def _safe_route_name(scope, routes):
+ try:
+ return _orig(scope, routes)
+ except AttributeError:
+ return "unknown"
+ _pr._get_route_name = _safe_route_name
Instrumentator().instrument(app).expose(app)
token_dep = get_token_dependency(config)
diff --git a/docs/blog/release-v0.7.0.md b/docs/blog/release-v0.7.0.md
index 1b474a998..19ade1ae0 100644
--- a/docs/blog/release-v0.7.0.md
+++ b/docs/blog/release-v0.7.0.md
@@ -6,7 +6,7 @@
Today I'm releasing Crawl4AI v0.7.0โthe Adaptive Intelligence Update. This release introduces fundamental improvements in how Crawl4AI handles modern web complexity through adaptive learning, intelligent content discovery, and advanced extraction capabilities.
-## ๐ฏ What's New at a Glance
+## What's New at a Glance
- **Adaptive Crawling**: Your crawler now learns and adapts to website patterns
- **Virtual Scroll Support**: Complete content extraction from infinite scroll pages
@@ -14,7 +14,7 @@ Today I'm releasing Crawl4AI v0.7.0โthe Adaptive Intelligence Update. This rel
- **Async URL Seeder**: Discover thousands of URLs in seconds with intelligent filtering
- **Performance Optimizations**: Significant speed and memory improvements
-## ๐ง Adaptive Crawling: Intelligence Through Pattern Learning
+## Adaptive Crawling: Intelligence Through Pattern Learning
**The Problem:** Websites change. Class names shift. IDs disappear. Your carefully crafted selectors break at 3 AM, and you wake up to empty datasets and angry stakeholders.
@@ -72,7 +72,7 @@ asyncio.run(main())
- **Research Data Collection**: Build robust academic datasets that survive website redesigns
- **Reduced Maintenance**: Cut selector update time by 80% for frequently-changing sites
-## ๐ Virtual Scroll: Complete Content Capture
+## Virtual Scroll: Complete Content Capture
**The Problem:** Modern web apps only render what's visible. Scroll down, new content appears, old content vanishes into the void. Traditional crawlers capture that first viewport and miss 90% of the content. It's like reading only the first page of every book.
@@ -212,7 +212,7 @@ asyncio.run(main())
- **Content Discovery**: Build topic-focused crawlers that stay on track
- **SEO Audits**: Identify and prioritize high-value internal linking opportunities
-## ๐ฃ Async URL Seeder: Automated URL Discovery at Scale
+## Async URL Seeder: Automated URL Discovery at Scale
**The Problem:** You want to crawl an entire domain but only have the homepage. Or worse, you want specific content types across thousands of pages. Manual URL discovery? That's a job for machines, not humans.
@@ -264,7 +264,7 @@ asyncio.run(main())
- **SEO Audits**: Find every indexable page with content scoring
- **Content Archival**: Ensure no content is left behind during site migrations
-## โก Performance Optimizations
+## Performance Optimizations
This release includes significant performance improvements through optimized resource handling, better concurrency management, and reduced memory footprint.
@@ -293,7 +293,7 @@ for url in urls:
- **Concurrent Crawls**: Handle 5x more parallel requests
-## ๐ง Important Changes
+## Important Changes
### Breaking Changes
- `link_extractor` renamed to `link_preview` (better reflects functionality)
@@ -336,7 +336,7 @@ Questions? Issues? I'm always listening:
- Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
- Twitter: [@unclecode](https://x.com/unclecode)
-Happy crawling! ๐ท๏ธ
+Happy crawling!
---
diff --git a/docs/blog/release-v0.7.1.md b/docs/blog/release-v0.7.1.md
index d5bfdaec9..517f1e14b 100644
--- a/docs/blog/release-v0.7.1.md
+++ b/docs/blog/release-v0.7.1.md
@@ -1,4 +1,4 @@
-# ๐ ๏ธ Crawl4AI v0.7.1: Minor Cleanup Update
+# Crawl4AI v0.7.1: Minor Cleanup Update
*July 17, 2025 โข 2 min read*
@@ -6,13 +6,13 @@
A small maintenance release that removes unused code and improves documentation.
-## ๐ฏ What's Changed
+## What's Changed
- **Removed unused StealthConfig** from `crawl4ai/browser_manager.py`
- **Updated documentation** with better examples and parameter explanations
- **Fixed virtual scroll configuration** examples in docs
-## ๐งน Code Cleanup
+## Code Cleanup
Removed unused `StealthConfig` import and configuration that wasn't being used anywhere in the codebase. The project uses its own custom stealth implementation through JavaScript injection instead.
@@ -22,7 +22,7 @@ from playwright_stealth import StealthConfig
stealth_config = StealthConfig(...) # This was never used
```
-## ๐ Documentation Updates
+## Documentation Updates
- Fixed adaptive crawling parameter examples
- Updated session management documentation
diff --git a/docs/blog/release-v0.7.3.md b/docs/blog/release-v0.7.3.md
index 59b2a9be3..5687eeed6 100644
--- a/docs/blog/release-v0.7.3.md
+++ b/docs/blog/release-v0.7.3.md
@@ -6,18 +6,18 @@
Today I'm releasing Crawl4AI v0.7.3โthe Multi-Config Intelligence Update. This release brings smarter URL-specific configurations, flexible Docker deployments, important bug fixes, and documentation improvements that make Crawl4AI more robust and production-ready.
-## ๐ฏ What's New at a Glance
+## What's New at a Glance
-- **๐ต๏ธ Undetected Browser Support**: Stealth mode for bypassing bot detection systems
-- **๐จ Multi-URL Configurations**: Different crawling strategies for different URL patterns in a single batch
-- **๐ณ Flexible Docker LLM Providers**: Configure LLM providers via environment variables
-- **๐ง Memory Monitoring**: Enhanced memory usage tracking and optimization tools
-- **๐ Enhanced Table Extraction**: Improved table access and DataFrame conversion
-- **๐ฐ GitHub Sponsors**: 4-tier sponsorship system with custom arrangements
-- **๐ง Bug Fixes**: Resolved several critical issues for better stability
-- **๐ Documentation Updates**: Clearer examples and improved API documentation
+- ** Undetected Browser Support**: Stealth mode for bypassing bot detection systems
+- ** Multi-URL Configurations**: Different crawling strategies for different URL patterns in a single batch
+- ** Flexible Docker LLM Providers**: Configure LLM providers via environment variables
+- ** Memory Monitoring**: Enhanced memory usage tracking and optimization tools
+- ** Enhanced Table Extraction**: Improved table access and DataFrame conversion
+- ** GitHub Sponsors**: 4-tier sponsorship system with custom arrangements
+- ** Bug Fixes**: Resolved several critical issues for better stability
+- ** Documentation Updates**: Clearer examples and improved API documentation
-## ๐จ Multi-URL Configurations: One Size Doesn't Fit All
+## Multi-URL Configurations: One Size Doesn't Fit All
**The Problem:** You're crawling a mix of documentation sites, blogs, and API endpoints. Each needs different handlingโcaching for docs, fresh content for news, structured extraction for APIs. Previously, you'd run separate crawls or write complex conditional logic.
@@ -82,7 +82,7 @@ async with AsyncWebCrawler() as crawler:
- **Reduced Complexity**: No more if/else forests in your extraction code
- **Better Performance**: Each URL gets exactly the processing it needs
-## ๐ต๏ธ Undetected Browser Support: Stealth Mode Activated
+## Undetected Browser Support: Stealth Mode Activated
**The Problem:** Modern websites employ sophisticated bot detection systems. Cloudflare, Akamai, and custom solutions block automated crawlers, limiting access to valuable content.
@@ -161,7 +161,7 @@ result = await crawler.arun("https://bot-protected-site.com", config=config)
- **Content Aggregation**: Collect news and social media despite anti-bot measures
- **Compliance Testing**: Verify your own site's bot protection effectiveness
-## ๐ง Memory Monitoring & Optimization
+## Memory Monitoring & Optimization
**The Problem:** Long-running crawl sessions consuming excessive memory, especially when processing large batches or heavy JavaScript sites.
@@ -203,7 +203,7 @@ async with AsyncWebCrawler() as crawler:
- **Performance Tuning**: Identify memory bottlenecks and optimization opportunities
- **Scalability Planning**: Understand memory patterns for horizontal scaling
-## ๐ Enhanced Table Extraction
+## Enhanced Table Extraction
**The Problem:** Table data was accessed through the generic `result.media` interface, making DataFrame conversion cumbersome and unclear.
@@ -240,15 +240,15 @@ if result.tables:
- **ETL Pipelines**: Cleaner integration with data processing workflows
- **Reporting**: Simplified table extraction for automated reporting systems
-## ๐ฐ Community Support: GitHub Sponsors
+## Community Support: GitHub Sponsors
I've launched GitHub Sponsors to ensure Crawl4AI's continued development and support our growing community.
**Sponsorship Tiers:**
-- **๐ฑ Supporter ($5/month)**: Community support + early feature previews
+- ** Supporter ($5/month)**: Community support + early feature previews
- **๐ Professional ($25/month)**: Priority support + beta access
-- **๐ข Business ($100/month)**: Direct consultation + custom integrations
-- **๐๏ธ Enterprise ($500/month)**: Dedicated support + feature development
+- ** Business ($100/month)**: Direct consultation + custom integrations
+- ** Enterprise ($500/month)**: Dedicated support + feature development
**Why Sponsor?**
- Ensure continuous development and maintenance
@@ -258,7 +258,7 @@ I've launched GitHub Sponsors to ensure Crawl4AI's continued development and sup
[**Become a Sponsor โ**](https://github.com/sponsors/unclecode)
-## ๐ณ Docker: Flexible LLM Provider Configuration
+## Docker: Flexible LLM Provider Configuration
**The Problem:** Hardcoded LLM providers in Docker deployments. Want to switch from OpenAI to Groq? Rebuild and redeploy. Testing different models? Multiple Docker images.
@@ -311,7 +311,7 @@ response = requests.post("http://localhost:11235/crawl", json={
- **Development Flexibility**: Test locally with one provider, deploy with another
- **Secure Configuration**: Keep API keys in `.llm.env` file, not in commands
-## ๐ง Bug Fixes & Improvements
+## Bug Fixes & Improvements
This release includes several important bug fixes that improve stability and reliability:
@@ -321,7 +321,7 @@ This release includes several important bug fixes that improve stability and rel
- **Table Extraction**: Improved table detection and extraction accuracy
- **Error Handling**: Better error messages and recovery from network failures
-## ๐ Documentation Enhancements
+## Documentation Enhancements
Based on community feedback, we've updated:
- Clearer examples for multi-URL configuration
@@ -330,11 +330,11 @@ Based on community feedback, we've updated:
- Added real-world URLs in examples for better understanding
- New comprehensive demo showcasing all v0.7.3 features
-## ๐ Acknowledgments
+## Acknowledgments
Thanks to our contributors and the entire community for feedback and bug reports.
-## ๐ Resources
+## Resources
- [Full Documentation](https://docs.crawl4ai.com)
- [GitHub Repository](https://github.com/unclecode/crawl4ai)
@@ -345,6 +345,6 @@ Thanks to our contributors and the entire community for feedback and bug reports
*Crawl4AI continues to evolve with your needs. This release makes it smarter, more flexible, and more stable. Try the new multi-config feature and flexible Docker deploymentโthey're game changers!*
-**Happy Crawling! ๐ท๏ธ**
+**Happy Crawling! **
*- The Crawl4AI Team*
\ No newline at end of file
diff --git a/docs/blog/release-v0.7.4.md b/docs/blog/release-v0.7.4.md
index 72cfe3ae1..32118222d 100644
--- a/docs/blog/release-v0.7.4.md
+++ b/docs/blog/release-v0.7.4.md
@@ -6,15 +6,15 @@
Today I'm releasing Crawl4AI v0.7.4โthe Intelligent Table Extraction & Performance Update. This release introduces revolutionary LLM-powered table extraction with intelligent chunking, significant performance improvements for concurrent crawling, enhanced browser management, and critical stability fixes that make Crawl4AI more robust for production workloads.
-## ๐ฏ What's New at a Glance
+## What's New at a Glance
- **๐ LLMTableExtraction**: Revolutionary table extraction with intelligent chunking for massive tables
-- **โก Enhanced Concurrency**: True concurrency improvements for fast-completing tasks in batch operations
-- **๐ง Browser Manager Fixes**: Resolved race conditions in concurrent page creation
-- **โจ๏ธ Cross-Platform Browser Profiler**: Improved keyboard handling and quit mechanisms
+- ** Enhanced Concurrency**: True concurrency improvements for fast-completing tasks in batch operations
+- ** Browser Manager Fixes**: Resolved race conditions in concurrent page creation
+- **โจ Cross-Platform Browser Profiler**: Improved keyboard handling and quit mechanisms
- **๐ Advanced URL Processing**: Better handling of raw URLs and base tag link resolution
- **๐ก๏ธ Enhanced Proxy Support**: Flexible proxy configuration with dict and string formats
-- **๐ณ Docker Improvements**: Better API handling and raw HTML support
+- ** Docker Improvements**: Better API handling and raw HTML support
## ๐ LLMTableExtraction: Revolutionary Table Processing
@@ -123,7 +123,7 @@ for table in result.tables:
- **Government Data**: Extract census data, statistical tables from official sources
- **Competitive Intelligence**: Process competitor pricing and feature tables
-## โก Enhanced Concurrency: True Performance Gains
+## Enhanced Concurrency: True Performance Gains
**The Problem:** The `arun_many()` method wasn't achieving true concurrency for fast-completing tasks, leading to sequential processing bottlenecks in batch operations.
@@ -157,7 +157,7 @@ async with AsyncWebCrawler() as crawler:
- **Monitoring Systems**: Faster health checks and status page monitoring
- **Data Aggregation**: Improved performance for real-time data collection
-## ๐ง Critical Stability Fixes
+## Critical Stability Fixes
### Browser Manager Race Condition Resolution
@@ -232,7 +232,7 @@ async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun("https://httpbin.org/ip")
```
-## ๐ณ Docker & Infrastructure Improvements
+## Docker & Infrastructure Improvements
This release includes several Docker and infrastructure improvements:
@@ -241,7 +241,7 @@ This release includes several Docker and infrastructure improvements:
- **Documentation Updates**: Comprehensive Docker deployment examples
- **Test Coverage**: Expanded test suite with better coverage
-## ๐ Documentation & Examples
+## Documentation & Examples
Enhanced documentation includes:
@@ -250,11 +250,11 @@ Enhanced documentation includes:
- **Docker Deployment**: Complete deployment guide with examples
- **Performance Optimization**: Guidelines for concurrent crawling
-## ๐ Acknowledgments
+## Acknowledgments
Thanks to our contributors and community for feedback, bug reports, and feature requests that made this release possible.
-## ๐ Resources
+## Resources
- [Full Documentation](https://docs.crawl4ai.com)
- [GitHub Repository](https://github.com/unclecode/crawl4ai)
@@ -265,6 +265,6 @@ Thanks to our contributors and community for feedback, bug reports, and feature
*Crawl4AI v0.7.4 delivers intelligent table extraction and significant performance improvements. The new LLMTableExtraction strategy handles complex tables that were previously impossible to process, while concurrency improvements make batch operations 3-4x faster. Try the intelligent table extractionโit's a game changer for data extraction workflows!*
-**Happy Crawling! ๐ท๏ธ**
+**Happy Crawling! **
*- The Crawl4AI Team*
\ No newline at end of file
diff --git a/docs/blog/release-v0.7.5.md b/docs/blog/release-v0.7.5.md
index 977d2fd9e..abf552570 100644
--- a/docs/blog/release-v0.7.5.md
+++ b/docs/blog/release-v0.7.5.md
@@ -6,7 +6,7 @@
Today I'm releasing Crawl4AI v0.7.5โfocused on extensibility and security. This update introduces the Docker Hooks System for pipeline customization, enhanced LLM integration, and important security improvements.
-## ๐ฏ What's New at a Glance
+## What's New at a Glance
- **Docker Hooks System**: Custom Python functions at key pipeline points with function-based API
- **Function-Based Hooks**: New `hooks_to_string()` utility with Docker client auto-conversion
@@ -15,7 +15,7 @@ Today I'm releasing Crawl4AI v0.7.5โfocused on extensibility and security. Thi
- **Bug Fixes**: Resolved multiple community-reported issues
- **Improved Docker Error Handling**: Better debugging and reliability
-## ๐ง Docker Hooks System: Pipeline Customization
+## Docker Hooks System: Pipeline Customization
Every scraping project needs custom logicโauthentication, performance optimization, content processing. Traditional solutions require forking or complex workarounds. Docker Hooks let you inject custom Python functions at 8 key points in the crawling pipeline.
@@ -249,7 +249,7 @@ async def test_https_preservation():
print(f" โ {link}")
```
-## ๐ ๏ธ Bug Fixes and Improvements
+## Bug Fixes and Improvements
### Major Fixes
- **URL Processing**: Fixed '+' sign preservation in query parameters (#1332)
@@ -286,7 +286,7 @@ browser_config = BrowserConfig(
)
```
-## ๐ Breaking Changes
+## Breaking Changes
1. **Python 3.10+ Required**: Upgrade from Python 3.9
2. **Proxy Parameter Deprecated**: Use new `proxy_config` structure
@@ -310,9 +310,9 @@ python docs/releases_review/demo_v0.7.5.py
```
**Resources:**
-- ๐ Documentation: [docs.crawl4ai.com](https://docs.crawl4ai.com)
-- ๐ GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
-- ๐ฌ Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
-- ๐ฆ Twitter: [@unclecode](https://x.com/unclecode)
+- Documentation: [docs.crawl4ai.com](https://docs.crawl4ai.com)
+- GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
+- Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
+- Twitter: [@unclecode](https://x.com/unclecode)
-Happy crawling! ๐ท๏ธ
+Happy crawling!
diff --git a/docs/blog/release-v0.7.6.md b/docs/blog/release-v0.7.6.md
index e27d19cc9..a33866c28 100644
--- a/docs/blog/release-v0.7.6.md
+++ b/docs/blog/release-v0.7.6.md
@@ -4,7 +4,7 @@
I'm excited to announce Crawl4AI v0.7.6, featuring a complete webhook infrastructure for the Docker job queue API! This release eliminates polling and brings real-time notifications to both crawling and LLM extraction workflows.
-## ๐ฏ What's New
+## What's New
### Webhook Support for Docker Job Queue API
@@ -158,20 +158,20 @@ def handle_webhook():
app.run(port=8080)
```
-## ๐ Performance Improvements
+## Performance Improvements
- **Reduced Server Load**: Eliminates constant polling requests
- **Lower Latency**: Instant notification vs. polling interval delay
- **Better Resource Usage**: Frees up client connections while jobs run in background
- **Scalable Architecture**: Handles high-volume crawling workflows efficiently
-## ๐ Bug Fixes
+## Bug Fixes
- Fixed webhook configuration serialization for Pydantic HttpUrl fields
- Improved error handling in webhook delivery service
- Enhanced Redis task storage for webhook config persistence
-## ๐ Expected Real-World Impact
+## Expected Real-World Impact
### For Web Scraping Workflows
- **Reduced Costs**: Less API calls = lower bandwidth and server costs
@@ -188,7 +188,7 @@ app.run(port=8080)
- **Decoupling**: Decouple job submission from result processing
- **Reliability**: Automatic retries ensure webhooks are delivered
-## ๐ Breaking Changes
+## Breaking Changes
**None!** This release is fully backward compatible.
@@ -196,7 +196,7 @@ app.run(port=8080)
- Existing code continues to work without modification
- Polling is still supported for jobs without webhook config
-## ๐ Documentation
+## Documentation
### New Documentation
- **[WEBHOOK_EXAMPLES.md](../deploy/docker/WEBHOOK_EXAMPLES.md)** - Comprehensive webhook usage guide
@@ -206,7 +206,7 @@ app.run(port=8080)
- **[Docker README](../deploy/docker/README.md)** - Added webhook sections
- API documentation with webhook examples
-## ๐ ๏ธ Migration Guide
+## Migration Guide
No migration needed! Webhooks are opt-in:
@@ -231,7 +231,7 @@ payload = {
}
```
-## ๐ง Configuration
+## Configuration
### Global Webhook Configuration (config.yml)
@@ -274,7 +274,7 @@ docker run -d \
pip install --upgrade crawl4ai
```
-## ๐ก Pro Tips
+## Pro Tips
1. **Use notification-only mode** for large results - fetch data separately to avoid large webhook payloads
2. **Set custom headers** for webhook authentication and request tracking
@@ -282,7 +282,7 @@ pip install --upgrade crawl4ai
4. **Implement idempotent webhook handlers** - same webhook may be delivered multiple times on retry
5. **Use structured schemas** with LLM extraction for predictable webhook data
-## ๐ฌ Demo
+## Demo
Try the release demo:
@@ -297,11 +297,11 @@ This comprehensive demo showcases:
- Webhook retry mechanism
- Real-time webhook receiver
-## ๐ Acknowledgments
+## Acknowledgments
Thank you to the community for the feedback that shaped this feature! Special thanks to everyone who requested webhook support for asynchronous job processing.
-## ๐ Support
+## Support
- **Documentation**: https://docs.crawl4ai.com
- **GitHub Issues**: https://github.com/unclecode/crawl4ai/issues
@@ -309,6 +309,6 @@ Thank you to the community for the feedback that shaped this feature! Special th
---
-**Happy crawling with webhooks!** ๐ท๏ธ๐ช
+**Happy crawling with webhooks!**
*- unclecode*
diff --git a/docs/blog/release-v0.7.7.md b/docs/blog/release-v0.7.7.md
index 190cd374b..b1ffb59b6 100644
--- a/docs/blog/release-v0.7.7.md
+++ b/docs/blog/release-v0.7.7.md
@@ -6,19 +6,19 @@
Today I'm releasing Crawl4AI v0.7.7โthe Self-Hosting & Monitoring Update. This release transforms Crawl4AI Docker from a simple containerized crawler into a complete self-hosting platform with enterprise-grade real-time monitoring, full operational transparency, and production-ready observability.
-## ๐ฏ What's New at a Glance
+## What's New at a Glance
-- **๐ Real-time Monitoring Dashboard**: Interactive web UI with live system metrics and browser pool status
-- **๐ Comprehensive Monitor API**: Complete REST API for programmatic access to all monitoring data
-- **โก WebSocket Streaming**: Real-time updates every 2 seconds for custom dashboards
-- **๐ฎ Control Actions**: Manual browser management (kill, restart, cleanup)
-- **๐ฅ Smart Browser Pool**: 3-tier architecture (permanent/hot/cold) with automatic promotion
-- **๐งน Janitor Cleanup System**: Automatic resource management with event logging
-- **๐ Production Metrics**: 6 critical metrics for operational excellence
-- **๐ญ Integration Ready**: Prometheus, alerting, and log aggregation examples
-- **๐ Critical Bug Fixes**: Async LLM extraction, DFS crawling, viewport config, and more
+- ** Real-time Monitoring Dashboard**: Interactive web UI with live system metrics and browser pool status
+- ** Comprehensive Monitor API**: Complete REST API for programmatic access to all monitoring data
+- ** WebSocket Streaming**: Real-time updates every 2 seconds for custom dashboards
+- ** Control Actions**: Manual browser management (kill, restart, cleanup)
+- ** Smart Browser Pool**: 3-tier architecture (permanent/hot/cold) with automatic promotion
+- ** Janitor Cleanup System**: Automatic resource management with event logging
+- ** Production Metrics**: 6 critical metrics for operational excellence
+- ** Integration Ready**: Prometheus, alerting, and log aggregation examples
+- ** Critical Bug Fixes**: Async LLM extraction, DFS crawling, viewport config, and more
-## ๐ Real-time Monitoring Dashboard: Complete Visibility
+## Real-time Monitoring Dashboard: Complete Visibility
**The Problem:** Running Crawl4AI in Docker was like flying blind. Users had no visibility into what was happening inside the containerโmemory usage, active requests, browser pools, or errors. Troubleshooting required checking logs, and there was no way to monitor performance or manually intervene when issues occurred.
@@ -29,10 +29,10 @@ Today I'm releasing Crawl4AI v0.7.7โthe Self-Hosting & Monitoring Update. This
Before v0.7.7, Docker was just a containerized crawler. After v0.7.7, it's a complete self-hosting platform that gives you:
- **๐ Data Privacy**: Your data never leaves your infrastructure
-- **๐ฐ Cost Control**: No per-request pricing or rate limits
-- **๐ฏ Full Customization**: Complete control over configurations and strategies
-- **๐ Complete Transparency**: Real-time visibility into every aspect
-- **โก Performance**: Direct access without network overhead
+- ** Cost Control**: No per-request pricing or rate limits
+- ** Full Customization**: Complete control over configurations and strategies
+- ** Complete Transparency**: Real-time visibility into every aspect
+- ** Performance**: Direct access without network overhead
- **๐ก๏ธ Enterprise Security**: Keep workflows behind your firewall
### Interactive Monitoring Dashboard
@@ -47,7 +47,7 @@ Access the dashboard at `http://localhost:11235/dashboard` to see:
The dashboard updates every 2 seconds via WebSocket, giving you live visibility into your crawling operations.
-## ๐ Monitor API: Programmatic Access
+## Monitor API: Programmatic Access
**The Problem:** Monitoring dashboards are great for humans, but automation and integration require programmatic access.
@@ -151,7 +151,7 @@ The Monitor API includes these endpoints:
- `POST /monitor/actions/restart_browser` - Restart browser
- `POST /monitor/stats/reset` - Reset accumulated statistics
-## โก WebSocket Streaming: Real-time Updates
+## WebSocket Streaming: Real-time Updates
**The Problem:** Polling the API every few seconds wastes resources and adds latency. Real-time dashboards need instant updates.
@@ -193,7 +193,7 @@ asyncio.run(monitor_realtime())
- **Integration**: Feed live data into monitoring tools like Grafana
- **Automation**: React to events in real-time without polling
-## ๐ฅ Smart Browser Pool: 3-Tier Architecture
+## Smart Browser Pool: 3-Tier Architecture
**The Problem:** Creating a new browser for every request is slow and memory-intensive. Traditional browser pools are static and inefficient.
@@ -246,9 +246,9 @@ asyncio.run(demonstrate_browser_pool())
**Pool Tiers:**
-- **๐ฅ Permanent Browser**: Always-on, default configuration, instant response
-- **โจ๏ธ Hot Pool**: Browsers promoted after 3+ uses, kept warm for quick access
-- **โ๏ธ Cold Pool**: On-demand browsers for variant configs, cleaned up when idle
+- ** Permanent Browser**: Always-on, default configuration, instant response
+- ** Hot Pool**: Browsers promoted after 3+ uses, kept warm for quick access
+- ** Cold Pool**: On-demand browsers for variant configs, cleaned up when idle
**Expected Real-World Impact:**
- **Memory Efficiency**: 10x reduction in memory usage vs creating browsers per request
@@ -256,7 +256,7 @@ asyncio.run(demonstrate_browser_pool())
- **Automatic Optimization**: Pool adapts to your usage patterns
- **Resource Management**: Janitor automatically cleans up idle browsers
-## ๐งน Janitor System: Automatic Cleanup
+## Janitor System: Automatic Cleanup
**The Problem:** Long-running crawlers accumulate idle browsers and consume memory over time.
@@ -278,7 +278,7 @@ async def monitor_janitor_activity():
# 2025-11-14 10:20:00: Hot pool browser promoted (10 requests)
```
-## ๐ฎ Control Actions: Manual Management
+## Control Actions: Manual Management
**The Problem:** Sometimes you need to manually interveneโkill a stuck browser, force cleanup, or restart resources.
@@ -320,7 +320,7 @@ async def reset_stats():
print("๐ Statistics reset for fresh monitoring")
```
-## ๐ Production Integration Patterns
+## Production Integration Patterns
### Prometheus Integration
@@ -417,7 +417,7 @@ CRITICAL_METRICS = {
}
```
-## ๐ Critical Bug Fixes
+## Critical Bug Fixes
This release includes significant bug fixes that improve stability and performance:
@@ -501,7 +501,7 @@ async with AsyncWebCrawler() as crawler:
- **LLMConfig Documentation**: Fixed casing and variable name consistency (issue #1551)
- **Python Version**: Dropped Python 3.9 support, now requires Python >=3.10
-## ๐ Expected Real-World Impact
+## Expected Real-World Impact
### For DevOps & Infrastructure Teams
- **Full Visibility**: Know exactly what's happening inside your crawling infrastructure
@@ -521,7 +521,7 @@ async with AsyncWebCrawler() as crawler:
- **Troubleshooting**: Quickly identify and fix issues
- **Learning**: See exactly how the browser pool works
-## ๐ Breaking Changes
+## Breaking Changes
**None!** This release is fully backward compatible.
@@ -562,7 +562,7 @@ pip install --upgrade crawl4ai
pip install crawl4ai==0.7.7
```
-## ๐ฌ Try the Demo
+## Try the Demo
Run the comprehensive demo that showcases all monitoring features:
@@ -580,7 +580,7 @@ python docs/releases_review/demo_v0.7.7.py
7. Production metrics and alerting patterns
8. Self-hosting value proposition
-## ๐ Documentation
+## Documentation
### New Documentation
- **[Self-Hosting Guide](https://docs.crawl4ai.com/core/self-hosting/)** - Complete self-hosting documentation with monitoring
@@ -592,7 +592,7 @@ python docs/releases_review/demo_v0.7.7.py
- Production integration patterns
- WebSocket streaming examples
-## ๐ก Pro Tips
+## Pro Tips
1. **Start with the dashboard** - Visit `/dashboard` to get familiar with the monitoring system
2. **Track the 6 key metrics** - Memory, success rate, latency, reuse rate, browser count, errors
@@ -603,24 +603,24 @@ python docs/releases_review/demo_v0.7.7.py
7. **Check janitor logs** - Understand automatic cleanup patterns
8. **Use control actions judiciously** - Manual interventions are for exceptional cases
-## ๐ Acknowledgments
+## Acknowledgments
Thank you to our community for the feedback, bug reports, and feature requests that shaped this release. Special thanks to everyone who contributed to the issues that were fixed in this version.
The monitoring system was built based on real user needs for production deployments, and your input made it comprehensive and practical.
-## ๐ Support & Resources
+## Support & Resources
-- **๐ Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
-- **๐ GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
-- **๐ฌ Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
-- **๐ฆ Twitter**: [@unclecode](https://x.com/unclecode)
-- **๐ Dashboard**: `http://localhost:11235/dashboard` (when running)
+- ** Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
+- ** GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
+- ** Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
+- ** Twitter**: [@unclecode](https://x.com/unclecode)
+- ** Dashboard**: `http://localhost:11235/dashboard` (when running)
---
**Crawl4AI v0.7.7 delivers complete self-hosting with enterprise-grade monitoring. You now have full visibility and control over your web crawling infrastructure. The monitoring dashboard, comprehensive API, and WebSocket streaming give you everything needed for production deployments. Try the self-hosting platformโit's a game changer for operational excellence!**
-**Happy crawling with full visibility!** ๐ท๏ธ๐
+**Happy crawling with full visibility!**
*- unclecode*
diff --git a/docs/deprecated/docker-deployment.md b/docs/deprecated/docker-deployment.md
index db8446e32..0e76774dc 100644
--- a/docs/deprecated/docker-deployment.md
+++ b/docs/deprecated/docker-deployment.md
@@ -1,11 +1,11 @@
-# ๐ณ Using Docker (Legacy)
+# Using Docker (Legacy)
Crawl4AI is available as Docker images for easy deployment. You can either pull directly from Docker Hub (recommended) or build from the repository.
---
-
๐ณ Option 1: Docker Hub (Recommended)
+Option 1: Docker Hub (Recommended)
Choose the appropriate image based on your platform and needs: @@ -64,7 +64,7 @@ Note: Due to hardware constraints, only the basic version is recommended for Ras
-
๐ณ Option 2: Build from Repository
+Option 2: Build from Repository
Build the image locally based on your platform: @@ -117,7 +117,7 @@ curl http://localhost:11235/health
-