feat(autoscaling): Add ThroughputConcurrencySystem

docs/guides/scaling_crawlers.mdx

@@ -55,6 +55,21 @@ The crawler builds its own system from the `ConcurrencySettings` described above
     {SharedConcurrencySystemExample}
 </RunnableCodeBlock>
 
+## Throughput concurrency system
+
+The default <ApiLink to="class/ConcurrencySystem">`ConcurrencySystem`</ApiLink> adds tasks until CPU or memory runs out. When the target website is the bottleneck, more tasks stop paying off. The site answers more slowly, and the crawler finishes fewer pages while loading the site harder. The experimental <ApiLink to="class/ThroughputConcurrencySystem">`ThroughputConcurrencySystem`</ApiLink> steers by throughput instead. By Little's law, `N` tasks that each take `T` seconds finish `N / T` tasks per second, so more concurrency helps only while `T` grows slower than `N`. The system reads throughput from windows of finished tasks and doubles the concurrency while that pays off. It then compares a level above and a level below the current concurrency, moves toward the better one, and holds once the two tie at the peak. It doesn't stay there for good. After each hold it measures the levels around the peak again, so even on a steady site the concurrency keeps moving within about 15% of it. The CPU and memory checks of the default system still apply.
+
+Build it from your `ConcurrencySettings` and pass it to the crawler as `concurrency_system`, the same way as the shared system above. Crawlers sharing it get one search over all their tasks, which suits crawlers against the same site, and the budget goes to whoever asks first. Crawlers against different sites should set `per_consumer`. Each crawler then gets a search of its own, and `max_concurrency` is split between them so that none can take the whole budget. A crawler that hasn't asked for capacity for ten autoscale intervals, 100 seconds by default, and has nothing running is forgotten, and it starts its search over when it comes back.
+
+The system works best on long-running crawls. It learns the peak from finished tasks, so the time it spends measuring pays off only over a long run.
+
+Known limitations:
+
+- Each step waits for its tasks to finish, so on slow pages the throughput system ramps up more slowly than the default one.
+- Every finished task counts as work, so a site that answers fast with errors such as HTTP 429 looks like it can take more.
+- A crawler that hits many hosts gets one search over all of them. It settles at the concurrency that suits the mix as a whole, which the slowest hosts tend to set. To give each site a peak of its own, crawl each with a crawler of its own on one system with `per_consumer` set.
+- Stay with the default system when the machine is the bottleneck. When you need a fixed rate, use `max_tasks_per_minute` with the default system, since the throughput system only settles at whatever rate the limit lets through.
+
 ## Running under a resource limit
 
 A crawler often gets less than the host machine has. A Docker container, a Kubernetes pod, a systemd slice and a Windows job object can each carry a limit of their own. Crawlee reads the limit that applies to the process and scales against it, with nothing to configure. The memory budget comes from the memory limit, the CPU load is measured against the cores the process may use, and the tightest limit wins when several of them apply. Without a limit, Crawlee falls back to the resources of the host machine.

src/crawlee/autoscaling/__init__.py

@@ -1,5 +1,6 @@
 from ._autoscaled_pool import AutoscaledPool
 from ._concurrency_system import ConcurrencySystem
+from ._throughput_concurrency_system import ThroughputConcurrencySystem
 from ._types import ConcurrencyConsumer, LoadRatioInfo, SystemInfo
 
 __all__ = [
@@ -8,4 +9,5 @@
     'ConcurrencySystem',
     'LoadRatioInfo',
     'SystemInfo',
+    'ThroughputConcurrencySystem',
 ]