Fixes an chunking issues with decorative separator lines. +Testing#156
Open
Torlek wants to merge 3 commits into
Open
Fixes an chunking issues with decorative separator lines. +Testing#156Torlek wants to merge 3 commits into
Torlek wants to merge 3 commits into
Conversation
…o prevent future regression.
There was a problem hiding this comment.
Pull request overview
This PR improves the text chunking pipeline in utils.py so Markdown-style decorative separator lines (e.g., * * * * *, ---) act as hard paragraph breaks instead of causing large “glued” chunks, and adds an optional config-driven hard-length fallback split to prevent oversized single segments. It also introduces regression tests and configuration defaults for the new behavior.
Changes:
- Treat symbol-only divider lines as hard paragraph separators during sentence splitting to prevent separator blocks from merging unrelated prose.
- Add an optional hard limit fallback (disabled by default) that splits oversized single segments at a configurable multiple of
chunk_size. - Add regression tests covering divider-line behavior, prior regressions, and the hard-limit fallback option.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| utils.py | Adds divider-line paragraph splitting and an optional hard-limit fallback splitter for oversized segments. |
| tests/test_chunking_regressions.py | Adds regression coverage for divider-line chunking and the new hard-limit fallback behavior. |
| config.yaml | Introduces text_chunking configuration defaults for the hard-limit fallback feature. |
| config.py | Adds default config schema entries for text_chunking options. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+15
to
+17
| from msgpack import fallback | ||
| from sympy import false | ||
| from utils import chunk_text_by_sentences, config_manager |
| hard_limit_factor=4.0, | ||
| ), | ||
| ChunkCountCase( | ||
| id="Original example text with five star issue", |
| expected_chunks=8, | ||
| ), | ||
| ChunkCountCase( | ||
| id="Original text with dash issue", |
Comment on lines
+1048
to
+1052
| while len(remaining) > max_len: | ||
| split_at = remaining.rfind(" ", 0, max_len + 1) | ||
| if split_at <= 0: | ||
| split_at = max_len | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In the attached text everything between the * * * * * separators is one chunk. This patch fixes this. I also added expandable testing to prevent future regression.
5star.txt
I also added an option to set a hard limit on chunk size. This behavior defaults to off.