Skip to content

memory_search FTS5 table is maintained on every publish but never queried #2

Description

@luke-ward88

The schema defines a full-text index in app/db.py:314:

CREATE VIRTUAL TABLE IF NOT EXISTS memory_search USING fts5(
    package_id UNINDEXED, title, summary, tags, author, content
);

It is kept up to date on every publish/version path — deleted and re-inserted in
app/main.py:11227-11229, app/main.py:11391-11393, and inserted again at
app/main.py:11923. But there is no MATCH query against it anywhere in the codebase:

$ grep -rn "MATCH" app/        # (no results)
$ grep -rn "FROM memory_search" app/main.py
app/main.py:11227:    conn.execute("DELETE FROM memory_search WHERE package_id=?", ...)
app/main.py:11391:    conn.execute("DELETE FROM memory_search WHERE package_id=?", ...)

The only reads are the two DELETEs above. The actual catalog/marketplace search instead
runs LIKE substring scans over the base tables — e.g. catalog() at
app/main.py:12126:

where += " AND (p.title LIKE ? OR p.summary LIKE ? OR p.tags LIKE ?)"

(similar LIKE filters at app/main.py:9902, 10264, 12743, etc.).

This leaves the project in an awkward middle state:

  • Every publish pays the write cost of maintaining an FTS5 index that nothing reads.
  • The user-facing search gets none of FTS5's benefits — no relevance ranking, no
    tokenization, no phrase matching. It also means raw % / _ in the query string q
    are interpreted as LIKE wildcards rather than literals (e.g. catalog(q="100%")),
    since the input is wrapped as f"%{q.strip()}%" without escaping.

Is the FTS path intended to be wired up (i.e. catalog/search should run
SELECT ... FROM memory_search WHERE memory_search MATCH ?), or is the virtual table
vestigial? Either resolution would be an improvement: connect the search endpoints to
memory_search to get ranked full-text results, or drop the table and its three
maintenance sites if LIKE is the intended behavior. If LIKE stays, escaping %/_
in user-supplied q would also be worth doing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions