Skip to content

Commit 7e4d585

Browse files
Mazyodclaude
andcommitted
docs: add feature support matrix with verification methodology
Add a feature support matrix to README documenting LSP feature compatibility across Pyright, Pyrefly, and ty backends. Includes emoji indicators for support status and notes for partial support cases. Create docs/FEATURE_VERIFICATION.md with methodology for maintaining the table using the test suite as source of truth (xfail markers document known limitations). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent d2ef5e2 commit 7e4d585

2 files changed

Lines changed: 133 additions & 0 deletions

File tree

README.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,41 @@ The following LSPs are available out of the box:
6464
- [Pyrefly](https://github.com/facebook/pyrefly)
6565
- [ty](https://github.com/astral-sh/ty) - Astral's fast Python type checker
6666

67+
## Feature Support Matrix
68+
69+
### Legend
70+
71+
| Symbol | Meaning |
72+
|--------|---------|
73+
| :white_check_mark: | Fully supported |
74+
| :warning: | Partial support (see notes) |
75+
| :x: | Not supported |
76+
| :grey_question: | Not tested / Not exposed in API |
77+
78+
### Features by Backend
79+
80+
> Last verified: basedpyright 1.36.2, Pyrefly 0.48.2, ty 0.0.12
81+
82+
| Feature | Pyright | Pyrefly | ty | Notes |
83+
|---------|:-------:|:-------:|:--:|-------|
84+
| Diagnostics | :white_check_mark: | :white_check_mark: | :warning: | ty requires files on disk |
85+
| Hover | :white_check_mark: | :white_check_mark: | :white_check_mark: | ty shows type only, not variable name |
86+
| Completion | :white_check_mark: | :white_check_mark: | :warning: | ty requires files on disk |
87+
| Completion Resolution | :white_check_mark: | :x: | :white_check_mark: | Pyrefly: not yet supported |
88+
| Signature Help | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
89+
| Rename | :white_check_mark: | :warning: | :warning: | Pyrefly: disabled for external files; ty: requires files on disk |
90+
| Semantic Tokens | :white_check_mark:\* | :white_check_mark: | :white_check_mark: | \*basedpyright recommended for extended features |
91+
| Go to Definition | :grey_question: | :grey_question: | :grey_question: | Not exposed in Session API |
92+
| Find References | :grey_question: | :grey_question: | :grey_question: | Not exposed in Session API |
93+
| Code Actions | :grey_question: | :grey_question: | :grey_question: | Not exposed in Session API |
94+
| Formatting | :grey_question: | :grey_question: | :grey_question: | Not exposed in Session API |
95+
96+
> See [Feature Verification Guide](docs/FEATURE_VERIFICATION.md) for methodology on maintaining this table.
97+
98+
For detailed backend limitations:
99+
- [Pyrefly Known Limitations](lsp_types/pyrefly/KNOWN_LIMITATIONS.md)
100+
- [ty Known Limitations](lsp_types/ty/KNOWN_LIMITATIONS.md)
101+
67102
### Pyright Example
68103

69104
```python

docs/FEATURE_VERIFICATION.md

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Feature Verification Guide
2+
3+
This document describes how to verify and update the feature support matrix in the README.
4+
5+
## Verification Process
6+
7+
### Step 1: Run the Test Suite
8+
9+
```bash
10+
uv run pytest tests/test_session.py -v
11+
```
12+
13+
Tests are parametrized across all backends (Pyright, Pyrefly, ty). Key indicators:
14+
- **PASSED**: Feature works for that backend
15+
- **XFAIL**: Known limitation, documented with reason
16+
- **FAILED**: Regression or new issue
17+
18+
### Step 2: Review xfail Markers
19+
20+
In `tests/test_session.py`, search for `xfail` to find documented limitations:
21+
22+
```python
23+
# Example from test_session_diagnostics:
24+
if backend_name == "ty":
25+
pytest.xfail("ty requires files on disk for diagnostics")
26+
```
27+
28+
Each xfail message explains why the feature is limited.
29+
30+
### Step 3: Check Backend Capabilities
31+
32+
Each backend declares its LSP capabilities in `get_lsp_capabilities()`:
33+
- `lsp_types/pyright/backend.py`
34+
- `lsp_types/pyrefly/backend.py`
35+
- `lsp_types/ty/backend.py`
36+
37+
Features declared here indicate what the client advertises to the server.
38+
39+
### Step 4: Review Known Limitations
40+
41+
- [Pyrefly Known Limitations](../lsp_types/pyrefly/KNOWN_LIMITATIONS.md)
42+
- [ty Known Limitations](../lsp_types/ty/KNOWN_LIMITATIONS.md)
43+
44+
## Updating the Feature Table
45+
46+
### When to Update
47+
48+
1. **After adding new Session API methods**: Add test, run it, update table
49+
2. **After upgrading backend versions**: Re-run tests, check for improvements, update version line
50+
3. **After backend releases announce new features**: Test and document
51+
52+
### Updating Version Numbers
53+
54+
Update the "Last verified" line in README when re-testing:
55+
56+
```bash
57+
# Check installed versions
58+
pyright --version # or basedpyright --version
59+
pyrefly --version
60+
ty --version
61+
```
62+
63+
### Status Symbols
64+
65+
| Symbol | Meaning | When to Use |
66+
|--------|---------|-------------|
67+
| :white_check_mark: | Fully supported | Test passes without xfail |
68+
| :warning: | Partial support | Test has xfail or conditional skip |
69+
| :x: | Not supported | Feature fails or is documented as unsupported |
70+
| :grey_question: | Unknown | Not exposed in Session API or not tested |
71+
72+
### Adding Notes
73+
74+
- Keep notes concise (under 50 characters)
75+
- Reference the specific limitation (e.g., "requires files on disk")
76+
- Use semicolons to separate multiple backend notes
77+
78+
## Evidence Mapping
79+
80+
| Feature | Test Function | Lines to Check |
81+
|---------|---------------|----------------|
82+
| Diagnostics | `test_session_diagnostics` | xfail around line 72 |
83+
| Hover | `test_session_hover` | format check around line 142 |
84+
| Completion | `test_session_completion` | xfail around line 258 |
85+
| Completion Resolution | `test_session_completion` | skip condition around line 286 |
86+
| Signature Help | `test_session_signature_help` | No xfails expected |
87+
| Rename | `test_session_rename` | xfails around lines 154, 158 |
88+
| Semantic Tokens | `test_session_semantic_tokens` | No xfails expected |
89+
90+
## Untested Features
91+
92+
Features declared in backend capabilities but not exposed in Session API:
93+
- Go to Definition (Pyrefly, ty declare it)
94+
- Find References (Pyrefly, ty declare it)
95+
- Code Actions
96+
- Formatting
97+
98+
To test these, use the low-level `LSPProcess` API directly.

0 commit comments

Comments
 (0)