Fix codebase indexing for plain text files

[WebMad]

Summary

• add .txt to the extensions scanned by codebase indexing
• route plain text files through the existing line-based fallback chunking mechanism
• add regression coverage for .txt parsing and case-insensitive extension matching

Fixes #931

Testing

• cd src && npx vitest run services/code-index/processors/__tests__/parser.txt.spec.ts
• relevant parser/scanner suite: 55 tests passed
• repository lint passed via the pre-commit hook
• repository type-check passed via the pre-push hook
• manually verified semantic search with a workspace containing both README.md and manual.txt; content unique to both files was returned after full reindexing
• manually reproduced the original failure by temporarily removing .txt support and confirming that only the Markdown content was indexed

Summary by CodeRabbit

• New Features
  • Added support for plain-text (.txt) in code indexing, including consistent handling of uppercase .TXT.
  • Plain-text (.txt) and other non-structural extensions now bypass structural code-definition parsing and use fallback chunking.
• Bug Fixes
  • Fixed parser behavior so an empty-string content override no longer triggers an unintended disk read.
• Tests
  • Added integration/unit coverage for fallback chunking (uppercase extensions, single-block output, segment-hash deduplication) and for ensuring structural parsing/file reads aren’t invoked for non-structural files.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4b4e9690-4b1e-4de5-8637-492529094be5

📥 Commits

Reviewing files that changed from the base of the PR and between 7446f33 and 7dde9a1.

📒 Files selected for processing (1)

• src/services/code-index/processors/__tests__/parser.spec.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/services/code-index/processors/tests/parser.spec.ts

---

📝 Walkthrough

Walkthrough

Plain-text files now use shared fallback-extension detection and line-based chunks. Tree-sitter bypasses non-structural extensions, while tests cover metadata, deduplication, case-insensitive matching, empty content, and parser bypassing.

Changes

Plain-text indexing

Layer / File(s)	Summary
Plain-text fallback chunking src/services/shared/fallback-extensions.ts, src/services/code-index/shared/supported-extensions.ts, src/services/code-index/processors/parser.ts, src/services/code-index/processors/__tests__/*	Fallback extensions are centralized and matched case-insensitively. CodeParser accepts empty content, avoids duplicate segment hashes, and produces validated .txt fallback chunks.
Tree-sitter plain-text bypass src/services/tree-sitter/index.ts, src/services/tree-sitter/__tests__/plainTextIntegration.spec.ts	Non-structural extensions, including .txt and .vb, return no structural definitions without loading language parsers or reading file contents.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CodeIndexer
  participant CodeParser
  participant TreeSitter
  CodeIndexer->>CodeParser: Parse .txt content
  CodeParser-->>CodeIndexer: Return fallback_chunk blocks
  CodeIndexer->>TreeSitter: Request structural definitions
  TreeSitter-->>CodeIndexer: Return undefined for non-structural extensions

Loading

Suggested labels: awaiting-review

Suggested reviewers: navedmerchant

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description is missing most required template sections and only includes a summary and testing notes.	Add the required headings: Related GitHub Issue, Description, Test Procedure, checklist, Documentation Updates, Additional Notes, and Get in Touch.
Out of Scope Changes check	⚠️ Warning	The empty-string content override and segment-hash dedup test are not required by #931 and appear unrelated.	Move those changes to a separate PR or justify them in the issue scope if they are intentional.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: plain-text indexing.
Linked Issues check	✅ Passed	The implementation matches #931 by indexing .txt files with fallback chunking and adding regression coverage.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[WebMad]

test-zoocode-contribution.tar.gz

[WebMad]

Plain text files can also reach parseSourceCodeDefinitionsForFile() through the folded file context flow, which is separate from codebase indexing. The fallbackExtensions list only affects CodeParser, so it does not prevent this function from trying to load a Tree-sitter parser for .txt. Since no such parser exists, we return undefined early to avoid an Unsupported language: txt error

[edelauna]

Nice! Looks great, thank you for this contribution, has some suggestions around the test case, and if we could also extend return undefined behavior to the other extensions while we're making this change.

[WebMad]

Thank you, @edelauna, for taking the time to review this and for the thoughtful feedback. I’ve addressed the suggestions and pushed the updates. I’m really glad to contribute and be part of the effort to make the project better

[coderabbitai]

🧹 Nitpick comments (1)

src/services/code-index/processors/__tests__/parser.spec.ts (1)

274-287: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove unnecessary await before synchronous method.

_chunkTextByLines is a synchronous method that returns CodeBlock[], not a Promise. The await keywords are unnecessary and can be removed.

♻️ Proposed refactor

-			const firstResult = await parser["_chunkTextByLines"](
+			const firstResult = parser["_chunkTextByLines"](
				lines,
				"manual.txt",
				"hash",
				"fallback_chunk",
				seenSegmentHashes,
			)
-			const duplicateResult = await parser["_chunkTextByLines"](
+			const duplicateResult = parser["_chunkTextByLines"](
				lines,
				"manual.txt",
				"hash",
				"fallback_chunk",
				seenSegmentHashes,
			)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/services/code-index/processors/__tests__/parser.spec.ts` around lines 274
- 287, Remove the unnecessary await keywords from both _chunkTextByLines calls
assigned to firstResult and duplicateResult, since the synchronous method
returns CodeBlock[] directly; preserve the existing arguments and result
handling.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/services/code-index/processors/__tests__/parser.spec.ts`:
- Around line 274-287: Remove the unnecessary await keywords from both
_chunkTextByLines calls assigned to firstResult and duplicateResult, since the
synchronous method returns CodeBlock[] directly; preserve the existing arguments
and result handling.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 152e8628-59fe-4200-b150-a4d04bfebb5c

📥 Commits

Reviewing files that changed from the base of the PR and between f34dd54 and c8b5491.

📒 Files selected for processing (7)

• src/services/code-index/processors/__tests__/parser.spec.ts
• src/services/code-index/processors/__tests__/parser.txt.spec.ts
• src/services/code-index/processors/parser.ts
• src/services/code-index/shared/supported-extensions.ts
• src/services/shared/fallback-extensions.ts
• src/services/tree-sitter/__tests__/plainTextIntegration.spec.ts
• src/services/tree-sitter/index.ts

🚧 Files skipped from review as they are similar to previous changes (3)

• src/services/tree-sitter/tests/plainTextIntegration.spec.ts
• src/services/code-index/processors/tests/parser.txt.spec.ts
• src/services/tree-sitter/index.ts