Skip to content

Serialize fxdata without ArduinoJSON#5461

Open
willmmiles wants to merge 4 commits intowled:mainfrom
willmmiles:unlimited-fxdata-size
Open

Serialize fxdata without ArduinoJSON#5461
willmmiles wants to merge 4 commits intowled:mainfrom
willmmiles:unlimited-fxdata-size

Conversation

@willmmiles
Copy link
Copy Markdown
Member

@willmmiles willmmiles commented Mar 30, 2026

Eliminates size limit. Fixes #5458

The implementation is inspired by RapidJSON's SAX-style Writer, coupled with the ChunkedResponse feature of AsyncWebServer that allows streaming data with minimal state. This particular implementation is tightly coupled to the specific task at hand; I'm hoping to generalize and expand on this approach for wider usage as it can make parts of the web API much less resource intensive.

Summary by CodeRabbit

  • Refactor
    • Mode/effect data is now streamed as chunked JSON, improving responsiveness and reducing memory usage for large payloads.
  • Chore
    • Removed the legacy in-memory serialization path for mode data and simplified the response handling; JSON output is now generated and escaped safely as it is sent.

Eliminates size limit.  Fixes wled#5458
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 30, 2026

Walkthrough

Replaced buffered JSON serialization for the /json?fxda endpoint with an incremental chunked streamer. Deleted serializeModeData(JsonArray fxdata) and added respondModeData(AsyncWebServerRequest* request); serveJson() now delegates to the streamer and fxdata is no longer treated as a special JSON-array conversion target.

Changes

Cohort / File(s) Summary
JSON Response Streaming Refactor
wled00/json.cpp
Removed serializeModeData(JsonArray fxdata); added respondModeData(AsyncWebServerRequest* request) which streams the /json?fxda response via request->sendChunked(...). Implemented chunked JSON string emission with helpers (writeJSONString, writeJSONStringElement), updated serveJson() to call the new responder and return early, and adjusted LockedJsonResponse construction so fxdata no longer requires special array conversion handling.
Header Declaration Cleanup
wled00/fcn_declare.h
Removed the public prototype void serializeModeData(JsonArray fxdata); to match the deleted implementation.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title 'Serialize fxdata without ArduinoJSON' accurately reflects the main change: removing ArduinoJSON dependency for fxdata serialization by implementing a streaming approach.
Linked Issues check ✅ Passed The PR addresses issue #5458 by removing the size limit for fxdata serialization via a streaming JSON approach, which prevents buffer overflow on ESP8266 and should resolve the effect list loading failure.
Out of Scope Changes check ✅ Passed All code changes are directly related to the objective of streaming fxdata serialization without ArduinoJSON. The removal of serializeModeData, introduction of respondModeData, and JSON helper functions are all scoped to the fxdata serialization task.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@wled00/json.cpp`:
- Around line 1196-1198: The truncation check after snprintf_P is off-by-one:
snprintf_P returns the number of characters that would be written (excluding
NUL), so if mode_bytes >= len the output was truncated; update the condition in
the block that uses dataPtr/mode_bytes/snprintf_P/len to test for mode_bytes >=
len (and break) instead of mode_bytes > len to avoid producing malformed JSON.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 6e12e019-d9be-457b-93b0-3652d3b55236

📥 Commits

Reviewing files that changed from the base of the PR and between fb8f8f0 and 9952a63.

📒 Files selected for processing (1)
  • wled00/json.cpp

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@wled00/json.cpp`:
- Around line 1190-1208: The loop can skip the first effect when
lineBuffer[0]==0 and later treat the next non-empty effect as non-first,
producing a leading comma; fix by tracking the actual first-written element
instead of relying on fx_index: introduce a boolean (e.g., first = true) before
the while, replace the check "if (fx_index == 0) *data = '[';" with "if (first)
{ *data = '['; first = false; }", and leave the rest of the logic (dataPtr,
mode_bytes, data/len/bytes_written updates, and fx_index++) unchanged so the '['
is written when the first non-empty mode is emitted.
- Line 1284: Remove the orphaned forward declaration "void
serializeModeData(JsonArray fxdata);" from the header fcn_declare.h; locate that
exact declaration and delete it so there is no unused prototype left in the
header, then rebuild to verify no references remain (no code changes required
elsewhere since there is no definition or callers).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 8466d893-6309-43fe-ba43-a596cb5716d9

📥 Commits

Reviewing files that changed from the base of the PR and between 9952a63 and 177e099.

📒 Files selected for processing (1)
  • wled00/json.cpp

willmmiles and others added 2 commits March 29, 2026 23:45
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
wled00/json.cpp (1)

1193-1193: Coupling to ArduinoJson internal namespace.

ARDUINOJSON_NAMESPACE::EscapeSequence::escapeChar is an internal API that could change between ArduinoJson versions. Since WLED vendors ArduinoJson, this is manageable, but worth noting for future maintenance.

Consider adding a comment documenting this dependency, or if this streaming approach is generalized later (as mentioned in PR description), consider implementing a standalone escape function.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@wled00/json.cpp` at line 1193, The code currently calls
ARDUINOJSON_NAMESPACE::EscapeSequence::escapeChar(*p) which relies on
ArduinoJson's internal API; update the usage by either (a) adding a clear
comment above the call to ARDUINOJSON_NAMESPACE::EscapeSequence::escapeChar
noting this internal dependency and the risk of API changes, or (b) replace the
call with a small standalone helper escapeChar implementation (used by the
streaming/JSON escape logic in json.cpp) and switch callers to that helper;
reference the existing call site (the esc =
ARDUINOJSON_NAMESPACE::EscapeSequence::escapeChar(*p) expression) when making
the change so you modify the exact usage and keep behavior identical.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@wled00/json.cpp`:
- Line 1193: The code currently calls
ARDUINOJSON_NAMESPACE::EscapeSequence::escapeChar(*p) which relies on
ArduinoJson's internal API; update the usage by either (a) adding a clear
comment above the call to ARDUINOJSON_NAMESPACE::EscapeSequence::escapeChar
noting this internal dependency and the risk of API changes, or (b) replace the
call with a small standalone helper escapeChar implementation (used by the
streaming/JSON escape logic in json.cpp) and switch callers to that helper;
reference the existing call site (the esc =
ARDUINOJSON_NAMESPACE::EscapeSequence::escapeChar(*p) expression) when making
the change so you modify the exact usage and keep behavior identical.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: c2853a0f-b7c0-4737-b171-0f71ae4f1b0b

📥 Commits

Reviewing files that changed from the base of the PR and between 67be835 and 64ebd41.

📒 Files selected for processing (1)
  • wled00/json.cpp

@DedeHai
Copy link
Copy Markdown
Collaborator

DedeHai commented Apr 2, 2026

I can confirm this working.

@DedeHai
Copy link
Copy Markdown
Collaborator

DedeHai commented Apr 2, 2026

now that the limit of number of FX is lifted, there is also no more reason to not include 2D particle system: the reason for the restriction was that the json would overflow.
i.e.

  • remove -D WLED_DISABLE_PARTICLESYSTEM2D for 4M ESP8266 builds
  • remove #error ESP8266 does not support 1D and 2D particle systems simultaneously. Please disable one of them. in FX.cpp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Effect list doesnt load on ESP8266 0.16.0 Nightly

2 participants