perf: Skip request-body compression for small payloads (#988)

Until now the Python client compressed every request body regardless of
size. It now skips compression below 1024 B.

Closes #934

## Why 1024 B

- **Parity with the JS client.** It uses the same threshold, the same
operator (`>=`), and the same byte-based measurement
(`MIN_COMPRESS_BYTES` in `apify-client-js/src/utils.ts`), so both
clients now put identical bytes on the wire.
- **Small bodies grow.** `run.charge` 40 B to 59 B, `kvs.set_record` 27
B to 45 B, single-item `dataset.push_items` 10 B to 30 B. gzip breaks
even only at ~88 B of realistic JSON, and high-entropy bodies such as
binary key-value store records inflate at any size, by ~23 B for gzip
and ~4 B for brotli.
- **No packet is saved below ~1 KB**, and removing a packet is the only
thing that buys latency. With ~500 B of request headers on a 1400 B MSS:

| Raw body | gzipped | Segments raw to gzip | Packet saved? |
| --- | --- | --- | --- |
| 786 B | 344 B | 1 to 1 | no |
| 1071 B | 447 B | 2 to 1 | yes |
| 8250 B | 2958 B | 7 to 3 | yes |

Compressing everything instead, the reverse direction raised in the
issue, doesn't pay off. Over 1200 realistic bodies it inflates 749 of
them while total bytes move only from −62.1% to −63.8%, and that gain
sits entirely in the 512–1024 B band where no packet is saved anyway. At
1024 B nothing inflates.

## Changes

- `MIN_COMPRESSION_SIZE = 1024`. `_prepare_request_call` compresses only
at or above it, measured on the encoded bytes so a multibyte `str` is
judged correctly.
- A caller-supplied `Content-Encoding` is now dropped on a skipped body,
where it would otherwise survive and mislabel an uncompressed payload.
- The async client skips the `asyncio.to_thread` hop for any body it
won't compress, the hop costing 36–68 µs against 5–12 µs of compression.
`_is_body_worth_compressing` sits next to the rule it mirrors so the two
can't drift apart. A `json=` body still hops, its size being unknown
until serialized.
- Docs: new "Minimum body size" section. The page claimed the client
compresses every request body.

## Verification

Against the live API, 8 value shapes round-trip byte-identical with and
without compression, small uncompressed bodies are accepted by
`dataset.push_items`, `rq.add_request`, `rq.batch_add_requests`,
`dataset.update`, `schedules().create`, `schedule.update`, and
`webhooks().create`, and latency is unchanged.

Tests that compressed tiny bodies were passing vacuously. They now cover
the 0/1/1023/1024/1025 boundary for both gzip and brotli, the
byte-vs-character threshold, the dropped header, and the thread-hop
decisions. `test_run_charge`'s `compression` axis was the suite's only
end-to-end compression coverage and had gone vacuous on its 39-byte
body, so it's replaced by a test asserting an above-threshold body
reaches the server compressed under the configured algorithm.

*✍️ Drafted by Claude Code*

Author: vdusek

docs/02_concepts/13_http_compression.mdx

@@ -15,7 +15,11 @@ The Apify client compresses request bodies before sending them to the API. It re
 
 ## How it works
 
-The client compresses request bodies using the compressor configured via the `compression` parameter (default `'gzip'`). The server supports both gzip and brotli and decompresses the request body transparently.
+The client compresses request bodies using the compressor configured via the `compression` parameter (default `'gzip'`). The server supports both gzip and brotli and decompresses the request body transparently. A body is compressed only when it is large enough to benefit and its content type isn't already compressed, as the next two sections describe.
+
+## Minimum body size
+
+The client sends bodies smaller than 1024 bytes without compression and without the `Content-Encoding` header. A body of this size fits in one network packet, so compression doesn't remove a network round trip and only costs CPU time. For very small bodies, the compression format adds bytes and can make the body larger.
 
 ## Already-compressed payloads
 
@@ -84,7 +88,7 @@ client = ApifyClient(token='MY-APIFY-TOKEN', compression=BrotliHttpCompressor(qu
 client = ApifyClient(token='MY-APIFY-TOKEN', compression=GzipHttpCompressor(quality=9))
 ```
 
-You can also implement a fully custom compressor by subclassing `HttpCompressor`:
+You can also implement a fully custom compressor by subclassing `HttpCompressor`. The client calls it only for bodies that reach the [minimum body size](#minimum-body-size) and aren't [already compressed](#already-compressed-payloads):
 
 ```python
 from apify_client import ApifyClient

src/apify_client/_consts.py

@@ -35,6 +35,13 @@
 OVERRIDABLE_DEFAULT_HEADERS = {'Accept', 'Authorization', 'Accept-Encoding', 'User-Agent'}
 """Headers that can be overridden by users, but will trigger a warning if they do so, as it may lead to API errors."""
 
+MIN_COMPRESSION_SIZE = 1024
+"""Smallest request body, in bytes, that is worth compressing.
+
+A smaller body already fits in a single network packet, so compressing it costs CPU time without
+saving a round trip.
+"""
+
 ALREADY_COMPRESSED_MEDIA_TYPE_PREFIXES = ('audio/', 'image/', 'video/')
 """Media type prefixes whose payloads carry their own compression, so compressing the request body is wasted work."""
 

src/apify_client/http_clients/_base.py

@@ -16,6 +16,7 @@
     DEFAULT_TIMEOUT_MAX,
     DEFAULT_TIMEOUT_MEDIUM,
     DEFAULT_TIMEOUT_SHORT,
+    MIN_COMPRESSION_SIZE,
 )
 from apify_client._docs import docs_group
 from apify_client._statistics import ClientStatistics
@@ -223,6 +224,24 @@ def _compute_timeout(self, timeout: Timeout, *, attempt: int) -> int | float | N
         new_timeout = min(resolved * (2 ** (attempt - 1)), self._timeout_max)
         return to_seconds(new_timeout)
 
+    @staticmethod
+    def _is_body_worth_compressing(data: str | bytes | bytearray | None) -> bool:
+        """Whether this body clears the size threshold `_prepare_request_call` compresses at, cheaply.
+
+        Below the threshold nothing is ever compressed. At or above it the content type still decides, but
+        checking that here would buy nothing - a body that turns out to be already compressed only wastes the
+        thread hop this answer guards.
+
+        The threshold is measured on encoded bytes, so a character count alone cannot decide a `str`. It is a
+        lower bound, so a `str` long enough in characters is long enough in bytes too. Below that the encoded
+        length decides, and the body is then under 4 KiB, so encoding it here is cheap.
+        """
+        if isinstance(data, str):
+            return len(data) >= MIN_COMPRESSION_SIZE or len(data.encode('utf-8')) >= MIN_COMPRESSION_SIZE
+        if isinstance(data, (bytes, bytearray)):
+            return len(data) >= MIN_COMPRESSION_SIZE
+        return False
+
     def _prepare_request_call(
         self,
         *,
@@ -234,10 +253,11 @@ def _prepare_request_call(
         """Prepare headers, params, and body for an HTTP request.
 
         Merges the client's default headers (including authorization) with per-request headers, serializes JSON
-        and compresses the body unless its content type says the payload is already compressed. Header names are
-        treated case-insensitively and per-request values win over the client defaults. For JSON bodies, a
-        `Content-Type` header is set unless the caller supplied one. `Content-Encoding` always describes what was
-        actually applied to the body, so a caller-supplied value is dropped whenever nothing was compressed.
+        and compresses the body unless it is smaller than `MIN_COMPRESSION_SIZE` or its content type says the
+        payload is already compressed. Header names are treated case-insensitively and per-request values win
+        over the client defaults. For JSON bodies, a `Content-Type` header is set unless the caller supplied one.
+        `Content-Encoding` always describes what was actually applied to the body, so a caller-supplied value is
+        dropped whenever nothing was compressed.
         """
         if json is not None and data is not None:
             raise ValueError('Cannot pass both "json" and "data" parameters at the same time!')
@@ -259,7 +279,7 @@ def _prepare_request_call(
                 data = bytes(data)
 
             content_type = next((value for key, value in headers.items() if key.lower() == 'content-type'), None)
-            if is_compressible_content_type(content_type):
+            if len(data) >= MIN_COMPRESSION_SIZE and is_compressible_content_type(content_type):
                 data = self._http_compressor.compress(data)
                 headers = self._merge_headers(headers, {'Content-Encoding': self._http_compressor.content_encoding})
                 compressed = True

src/apify_client/http_clients/_impit.py

@@ -395,9 +395,10 @@ async def call(
         self._statistics.calls += 1
 
         # Serializing and compressing a request body is CPU-bound and would block the event loop, so
-        # offload request preparation to a worker thread whenever there is a body. Bodyless requests
-        # skip the thread hop, as they have no expensive work to move off the loop.
-        if json is not None or data is not None:
+        # offload preparation to a worker thread whenever there is something to compress. A body the
+        # client sends as it is costs less to prepare inline than the hop itself. A `json` body always
+        # hops, as its size is only known once serialized.
+        if json is not None or self._is_body_worth_compressing(data):
             prepared_headers, prepared_params, content = await asyncio.to_thread(
                 self._prepare_request_call,
                 headers=headers,

tests/unit/test_client_request_queue.py

@@ -120,10 +120,15 @@ def _make_large_requests() -> list[RequestDraftDict]:
 
 
 def _payload_capturing_handler(payloads: list[bytes]) -> Callable[[Request], Response]:
-    """Return a handler that records each POST body (gzip-decompressed) and responds with an empty batch result."""
+    """Return a handler that records each POST body and responds with an empty batch result.
+
+    Bodies below the client's compression threshold arrive uncompressed, so the recorded payload is
+    decompressed only when the request says it was encoded.
+    """
 
     def handler(request: Request) -> Response:
-        payloads.append(gzip.decompress(request.get_data()))
+        body = request.get_data()
+        payloads.append(gzip.decompress(body) if request.headers.get('Content-Encoding') == 'gzip' else body)
         return Response(_EMPTY_BATCH_RESPONSE_CONTENT, status=200, content_type='application/json')
 
     return handler