[SohamKukreti]

Summary

Fixes #2472.

BrowserConfig.__init__ always adds a sec-ch-ua header derived from user_agent, so dump() always sent headers. The server forbids headers from an untrusted request, so Crawl4aiDockerClient got 400 with any BrowserConfig, even BrowserConfig().

dump() now leaves out sec-ch-ua when it is the auto-added value (== self.browser_hint), and drops headers if nothing is left. load() runs __init__, which adds the same value back from user_agent, so nothing is lost. Headers the user set, including their own sec-ch-ua, are still sent and still refused by the server. No server code changes.

List of files changed and why

• crawl4ai/async_configs.py: the change in BrowserConfig.dump().
• tests/unit/test_config_provenance.py: two tests. A default and a custom-UA BrowserConfig dump passes the untrusted load and loads back the same headers; custom headers stay in the dump and are still refused.
• deploy/docker/tests/test_playground_config_protocol.py: removed the guard that asserted the /config/dump echo still has a forbidden field. Its message said it could go once dump() is re-submittable. The check that the page forwards the typed params stays.
• deploy/docker/static/playground/index.html: the comment said the echo carries headers the gate refuses, which is no longer true. Comment only.

How Has This Been Tested?

• The two new tests fail on develop and pass with the fix.
• tests/unit, tests/deep_crawling, the config, serialization, security and playground tests: same failures as develop (environment only: PDF libs, browser TLS flags), plus the two new tests.
• Live, Crawl4aiDockerClient against a local server: BrowserConfig(), verbose=True, custom user_agent, user_agent_mode="random", text_mode, light_mode, enable_stealth, viewport, stream=True: 400 on develop, 200 with the fix. headers={"X-A": "1"} and headers={"sec-ch-ua": "fake"} still 400.
• Crawling https://httpbin.org/headers through the server: the site still gets a sec-ch-ua that matches the user agent (default and a Chrome 131 UA).
• The untrusted gate is not touched: raw requests with headers or other casings give the same result as before.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[ntohidi]

Thanks, nice fix. Checked that both new tests fail on develop and pass here, and the provenance and playground tests are green. Custom headers still hit the gate, so nothing changes on the security side. As a bonus, user_agent_mode="random" now gets a sec-ch-ua that matches the new UA after load. LGTM.