|
| 1 | +# Feature Verification Guide |
| 2 | + |
| 3 | +This document describes how to verify and update the feature support matrix in the README. |
| 4 | + |
| 5 | +## Verification Process |
| 6 | + |
| 7 | +### Step 1: Run the Test Suite |
| 8 | + |
| 9 | +```bash |
| 10 | +uv run pytest tests/test_session.py -v |
| 11 | +``` |
| 12 | + |
| 13 | +Tests are parametrized across all backends (Pyright, Pyrefly, ty). Key indicators: |
| 14 | +- **PASSED**: Feature works for that backend |
| 15 | +- **XFAIL**: Known limitation, documented with reason |
| 16 | +- **FAILED**: Regression or new issue |
| 17 | + |
| 18 | +### Step 2: Review xfail Markers |
| 19 | + |
| 20 | +In `tests/test_session.py`, search for `xfail` to find documented limitations: |
| 21 | + |
| 22 | +```python |
| 23 | +# Example from test_session_diagnostics: |
| 24 | +if backend_name == "ty": |
| 25 | + pytest.xfail("ty requires files on disk for diagnostics") |
| 26 | +``` |
| 27 | + |
| 28 | +Each xfail message explains why the feature is limited. |
| 29 | + |
| 30 | +### Step 3: Check Backend Capabilities |
| 31 | + |
| 32 | +Each backend declares its LSP capabilities in `get_lsp_capabilities()`: |
| 33 | +- `lsp_types/pyright/backend.py` |
| 34 | +- `lsp_types/pyrefly/backend.py` |
| 35 | +- `lsp_types/ty/backend.py` |
| 36 | + |
| 37 | +Features declared here indicate what the client advertises to the server. |
| 38 | + |
| 39 | +### Step 4: Review Known Limitations |
| 40 | + |
| 41 | +- [Pyrefly Known Limitations](../lsp_types/pyrefly/KNOWN_LIMITATIONS.md) |
| 42 | +- [ty Known Limitations](../lsp_types/ty/KNOWN_LIMITATIONS.md) |
| 43 | + |
| 44 | +## Updating the Feature Table |
| 45 | + |
| 46 | +### When to Update |
| 47 | + |
| 48 | +1. **After adding new Session API methods**: Add test, run it, update table |
| 49 | +2. **After upgrading backend versions**: Re-run tests, check for improvements, update version line |
| 50 | +3. **After backend releases announce new features**: Test and document |
| 51 | + |
| 52 | +### Updating Version Numbers |
| 53 | + |
| 54 | +Update the "Last verified" line in README when re-testing: |
| 55 | + |
| 56 | +```bash |
| 57 | +# Check installed versions |
| 58 | +pyright --version # or basedpyright --version |
| 59 | +pyrefly --version |
| 60 | +ty --version |
| 61 | +``` |
| 62 | + |
| 63 | +### Status Symbols |
| 64 | + |
| 65 | +| Symbol | Meaning | When to Use | |
| 66 | +|--------|---------|-------------| |
| 67 | +| :white_check_mark: | Fully supported | Test passes without xfail | |
| 68 | +| :warning: | Partial support | Test has xfail or conditional skip | |
| 69 | +| :x: | Not supported | Feature fails or is documented as unsupported | |
| 70 | +| :grey_question: | Unknown | Not exposed in Session API or not tested | |
| 71 | + |
| 72 | +### Adding Notes |
| 73 | + |
| 74 | +- Keep notes concise (under 50 characters) |
| 75 | +- Reference the specific limitation (e.g., "requires files on disk") |
| 76 | +- Use semicolons to separate multiple backend notes |
| 77 | + |
| 78 | +## Evidence Mapping |
| 79 | + |
| 80 | +| Feature | Test Function | Lines to Check | |
| 81 | +|---------|---------------|----------------| |
| 82 | +| Diagnostics | `test_session_diagnostics` | xfail around line 72 | |
| 83 | +| Hover | `test_session_hover` | format check around line 142 | |
| 84 | +| Completion | `test_session_completion` | xfail around line 258 | |
| 85 | +| Completion Resolution | `test_session_completion` | skip condition around line 286 | |
| 86 | +| Signature Help | `test_session_signature_help` | No xfails expected | |
| 87 | +| Rename | `test_session_rename` | xfails around lines 154, 158 | |
| 88 | +| Semantic Tokens | `test_session_semantic_tokens` | No xfails expected | |
| 89 | + |
| 90 | +## Untested Features |
| 91 | + |
| 92 | +Features declared in backend capabilities but not exposed in Session API: |
| 93 | +- Go to Definition (Pyrefly, ty declare it) |
| 94 | +- Find References (Pyrefly, ty declare it) |
| 95 | +- Code Actions |
| 96 | +- Formatting |
| 97 | + |
| 98 | +To test these, use the low-level `LSPProcess` API directly. |
0 commit comments