Skip to content

feat: selective relationship capture with READS_FROM/WRITES_TO I/O edges#638

Open
vitali87 wants to merge 6 commits into
mainfrom
feat/relationship-capture-io
Open

feat: selective relationship capture with READS_FROM/WRITES_TO I/O edges#638
vitali87 wants to merge 6 commits into
mainfrom
feat/relationship-capture-io

Conversation

@vitali87

@vitali87 vitali87 commented Jul 7, 2026

Copy link
Copy Markdown
Owner

What

Adds a selective relationship capture framework and ships READS_FROM / WRITES_TO I/O edges as its first opt-in add-on (issue #51).

I/O edges (READS_FROM / WRITES_TO)

  • New Resource node (kind + literal-or-<dynamic> identity) and two relationship types.
  • codebase_rag/parsers/io_access/: a curated per-language sink registry (open, requests.*, os.getenv, print, json.*, sqlite3, sockets…) plus resource-handle tracking (f = open(p,'w'); f.write(x), conn = sqlite3.connect(); conn.execute(sql) with SQL-keyword direction). Python first; the pipeline is language-agnostic so other languages are later registry tables.
  • Direction refined from the open() mode arg; target captured as a string literal when known, else collapsed to one <dynamic> resource per kind.
  • Protobuf export updated: Resource message + oneof + enum values regenerated in codec/schema.proto / schema_pb2.py.

Selective capture

  • Capture groups: structure, calls, types, imports, io. A build-time guard asserts every RelationshipType belongs to exactly one group.
  • Default: core groups on; io (and future add-ons) opt-in. Existing graphs are unchanged unless a user opts in.
  • Selection grammar: groups + +TYPE/-TYPE overrides, all/none bases; dependency gaps (e.g. drop INHERITS, keep OVERRIDES) warn but are obeyed.
  • Enforcement: a FilteringIngestor choke point (transparent passthrough that preserves each caller's exact call shape) wraps the ingestor inside GraphUpdater; the ~20 parser emission sites are untouched. The IO pass is additionally short-circuited when disabled.
  • Config: CGR_CAPTURE env var (sticky baseline) + repeatable --capture flag on start/index (per-run override), merged by one resolver.

Tests

  • test_io_access_edges.py — direct sinks, handle tracking, direction refinement, dynamic targets, negative case.
  • test_capture_resolver.py — default/opt-in/overrides/base tokens/dependency-gap/unknown-token.
  • test_filtering_ingestor.py — enable/drop of rels and nodes, and call-shape preservation (positional vs keyword properties).
  • test_capture_end_to_end.pyGraphUpdater(capture=…) gates IO edges + Resource nodes.
  • Full non-integration suite green (4409 passed); ty + ruff clean.

Usage

cgr start --update-graph --capture io        # core graph + I/O edges
CGR_CAPTURE="none,calls" cgr start --update-graph   # only CALLS-family edges

Closes #51.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces I/O capture capabilities to track reads and writes to external resources (like files, databases, and network endpoints) in Python codebases, adding a new Resource node and READS_FROM/WRITES_TO relationships. Feedback on the implementation highlights several areas for improvement: the AST traversal currently processes nodes in reverse order, which incorrectly resolves variable reassignments to their first rather than last assignment; it lacks support for tracking file handles opened via idiomatic with statements; and target resolution is limited to positional arguments, ignoring keyword arguments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread codebase_rag/parsers/io_access/processor.py Outdated
Comment thread codebase_rag/parsers/io_access/processor.py Outdated
Comment thread codebase_rag/parsers/io_access/processor.py Outdated
@greptile-apps

greptile-apps Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds selective graph relationship capture and opt-in Python I/O resource edges. The main changes are:

  • New capture groups for structure, calls, types, imports, and I/O relationships.
  • CGR_CAPTURE and repeatable --capture support for choosing captured relationship sets.
  • A filtering ingestor that drops disabled nodes and relationships at one graph emission point.
  • Python READS_FROM and WRITES_TO detection for direct sinks, resource handles, open modes, and SQL handle methods.
  • Protobuf schema updates for Resource nodes and the new I/O relationship types.
  • Tests covering capture resolution, filtering behavior, I/O edge extraction, and end-to-end gating.

Confidence Score: 5/5

This PR appears safe to merge.

No accepted issues were found in the reviewed capture selection, filtering, I/O processing, protobuf, and test paths.

No files require special attention.

T-Rex T-Rex Logs

What T-Rex did

  • T-Rex attempted a focused pytest run using PYTHONPATH and uv run, but the test invocation exited with code 1 due to a recreated unsynced virtual environment and a missing pytest module.
  • T-Rex ran a diagnostics sequence to understand the environment, revealing a broken symlink from the original venv to an unavailable Python and that the system Python provides pytest, but test dependencies were not installed after recreating the venv.
  • T-Rex tried a fallback test path with the system Python, which exited with code 4 because conftest import failed due to a missing loguru dependency.

View all artifacts

T-Rex Ran code and verified through T-Rex

Important Files Changed

Filename Overview
codebase_rag/capture.py Adds capture selection resolution, group/type overrides, node-label gating, and dependency-gap warnings.
codebase_rag/graph_updater.py Wraps graph emissions in FilteringIngestor and passes capture settings into processors.
codebase_rag/parsers/io_access/processor.py Implements Python I/O sink detection, handle tracking, direction refinement, and filtered resource edge emission.
codebase_rag/services/filtering.py Adds a capture-aware ingestor wrapper that drops disabled nodes and relationships while preserving call shape.
codebase_rag/tests/test_io_access_edges.py Adds Python I/O edge tests for direct sinks, handle tracking, keyword args, direction refinement, and selective filtering.
codec/schema.proto Extends the protobuf schema with Resource payloads and I/O relationship enum values.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant CLI as CLI/config
participant GU as GraphUpdater
participant FI as FilteringIngestor
participant CP as CallProcessor
participant IO as IOAccessProcessor
participant DB as Ingestor

CLI->>CLI: resolve_capture(CGR_CAPTURE + --capture)
CLI->>GU: GraphUpdater(capture)
GU->>FI: wrap raw ingestor with selection
GU->>CP: create processors with filtered sink + capture
CP->>IO: process_io_for_caller(caller, module, language)
IO->>IO: match sink/handle and enabled rels
IO->>FI: ensure Resource node + READS_FROM/WRITES_TO
FI->>DB: forward only enabled nodes/relationships
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant CLI as CLI/config
participant GU as GraphUpdater
participant FI as FilteringIngestor
participant CP as CallProcessor
participant IO as IOAccessProcessor
participant DB as Ingestor

CLI->>CLI: resolve_capture(CGR_CAPTURE + --capture)
CLI->>GU: GraphUpdater(capture)
GU->>FI: wrap raw ingestor with selection
GU->>CP: create processors with filtered sink + capture
CP->>IO: process_io_for_caller(caller, module, language)
IO->>IO: match sink/handle and enabled rels
IO->>FI: ensure Resource node + READS_FROM/WRITES_TO
FI->>DB: forward only enabled nodes/relationships
Loading

Reviews (4): Last reviewed commit: "fix: skip Resource node when its only I/..." | Re-trigger Greptile

Comment thread codebase_rag/capture.py Outdated
Comment thread codebase_rag/parsers/io_access/processor.py
@codecov-commenter

codecov-commenter commented Jul 7, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 96.19377% with 22 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
codebase_rag/parsers/io_access/processor.py 89.85% 14 Missing ⚠️
codebase_rag/parsers/io_access/models.py 86.20% 4 Missing ⚠️
codebase_rag/constants.py 95.45% 1 Missing ⚠️
codebase_rag/services/filtering.py 95.83% 1 Missing ⚠️
codebase_rag/tests/test_capture_end_to_end.py 97.14% 1 Missing ⚠️
codebase_rag/tests/test_io_access_edges.py 98.43% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@vitali87

vitali87 commented Jul 7, 2026

Copy link
Copy Markdown
Owner Author

@greptile review

Comment thread codebase_rag/parsers/io_access/processor.py Outdated
@vitali87

vitali87 commented Jul 7, 2026

Copy link
Copy Markdown
Owner Author

@greptile review

Comment thread codebase_rag/parsers/io_access/processor.py
@vitali87

vitali87 commented Jul 7, 2026

Copy link
Copy Markdown
Owner Author

@greptile review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Add more entity relationships

2 participants