fix: keep two Search entities written alike apart

docs/upstream-asks.md

@@ -120,18 +120,25 @@ names as an override.
 entity column; a wrong name costs one replan and one further approval, after
 which the turn closes naming the step that could not run.
 
-## 10. A part refused for its denominator names neither side (per-head questions)
+## 10. A part refused for its denominator does not say whose it is (per-head questions)
 
 Search answers "debt service per head" for Kenya `no_data` with reason
 `denominator_not_available`. The `no_data` block carries `reason`,
-`category`, `message`, `elected_count`, `entity_terms`,
-`ambiguous_entities` and `unbound_qualifiers`. It omits the
-measure search read once the per-capita phrasing is set aside and the
-denominator it looked for.
-
-**Ask:** on `denominator_not_available`, return the measure's variable
-text without the normalising phrase, the denominator as a variable text
-or StatVar, and which of the part's entities the denominator is about.
-**Meanwhile:** the agent reads the `no_data` reason and has to work out
-the measure and the denominator itself before it can search for them as
-two parts.
+`cause`, `category`, `message`, `nearest_real`, `gate`, `flagged_match`,
+`alternate_match`, `elected_count`, `entity_terms`, `ambiguous_entities`,
+`source_terms`, `ambiguous_sources`, `unit_alternatives`,
+`unbound_qualifiers`, `denominator` and `options`. On a
+`denominator_not_available` refusal after a series was elected,
+`denominator.asked` names the denominator the question named and
+`denominator.found` the one the flagged match divides by, and
+`nearest_real` holds the level to divide. Each `ambiguous_entities` group
+is settled by the options whose `entity_choice` is its index, so two
+groups written alike ("Congo" and "Congo") are told apart. The block
+still omits which of the part's entities the denominator is about.
+
+**Ask:** on `denominator_not_available`, say which of the part's
+entities the denominator is about.
+**Meanwhile:** data-mcp's digest shows the agent `nearest_real` but not
+`denominator`, so the agent works out the denominator and the entities it
+is about itself before it can search for the level and the denominator
+as two parts.

services/data-mcp/README.md

@@ -14,10 +14,11 @@ The `search` tool finds datasets and indicators matching a structured query. It
 
 #### Parameters
 
-- `parts`: List of 1 to 6 `SearchPart` objects. Each part contains `variable` (string, 1 to 200 characters) and optional fields `entities` (list of `SearchEntity` objects with `text`, optional `role`, and optional `members`), `time` (string, 1 to 100 characters), `unit` (string, 1 to 100 characters), and `sources` (list of 1 to 8 `SearchSource` objects with `text`, 1 to 120 characters, and optional `exclude` and `ref` (1 to 200 characters)).
+- `parts`: List of 1 to 6 `SearchPart` objects. Each part contains `variable` (string, 1 to 200 characters) and optional fields `entities` (list of `SearchEntity` objects with `text`, optional `role`, optional `members`, and an optional `name` or `dcid`), `time` (string, 1 to 100 characters), `unit` (string, 1 to 100 characters), and `sources` (list of 1 to 8 `SearchSource` objects with `text`, 1 to 120 characters, and optional `exclude` and `ref` (1 to 200 characters)).
 - `variable` and each entity's `text` take the caller's own words. `role` and `time` are matched against a fixed grammar. `role` names which side of a flow the entity sits on: a word for where something originates resolves to "from", a word for where it is directed resolves to "to". `time` takes a bare year, a range ("2015 to 2020"), an open start ("since 2015"), or "latest". A `role_text_not_recognised` or `time_text_not_parsed` warning means that field needs rephrasing in the grammar.
 - An entity whose `text` names a set of places (a continent, a subregion, or a group such as the European Union) takes `members: true` to ask for each member as its own series: "African countries" is `{"text": "Africa", "members": true}`. Without it the set reads as one place where the data holds a figure for it, and as its members where it holds none. The flag has no effect on a single place, and the wire request carries it only when true.
 - `sources` holds each publisher or dataset the user names for a part, in the user's words, with `exclude: true` for one the user asks to leave out: "aid to Kenya, not from CRS" is `{"text": "CRS", "exclude": true}`. Each reaches apis as a part's `sources` entry, with `exclude` as `polarity` (`include` or `exclude`). apis resolves the words against every source it holds, so the tool description lists none. A lender or recipient the flow runs to or from is an entity with its role, not a source. Words apis cannot resolve come back as `no_data` with reason `source_not_resolved`, and words that name several sources as `source_ambiguous`. Ask which candidate the user means, then retry the same source entry with its chosen `ref`, retaining `text` and `exclude`.
+- An entity's `name` or `dcid` says which entity its `text` means, so two entities written alike stay two rows: "aid from Congo to Congo" sends `{"text": "Congo", "role": "from", "name": "Democratic Republic of the Congo"}` and `{"text": "Congo", "role": "to", "name": "Republic of the Congo"}`. A row carrying both is refused before the call, as apis refuses it with `request_invalid`. A `no_data` part with reason `entity_ambiguous` lists each ambiguous entity by number, with its role and each candidate's label, dcid, and kind; the agent asks the user which one is meant and resends that row with the candidate's `dcid`, retaining `text` and `role`.
 - `raw_query`: Optional string up to 2000 characters holding the original query text.
 
 A compound ask splits into one part per measured variable.

services/data-mcp/src/data_mcp/digest.py

@@ -322,6 +322,7 @@ def _render_no_data_part(part: Mapping[str, Any]) -> str:
     sections = [header]
     if message := no_data.get("message"):
         sections.append(message)
+    sections.extend(_ambiguous_entity_lines(no_data, part.get("sentence_plan") or []))
     sections.extend(_ambiguous_source_lines(no_data.get("ambiguous_sources") or []))
     checked = _checked_count(no_data)
     if checked:
@@ -360,6 +361,67 @@ def _unapplied_phrase_header(unbound_qualifiers: Sequence[Mapping[str, Any]]) ->
     )
 
 
+def _ambiguous_entity_lines(
+    no_data: Mapping[str, Any], sentence_plan: Sequence[Mapping[str, Any]]
+) -> list[str]:
+    """One line per `ambiguous_entities` group, naming each candidate's dcid, then how to settle it.
+
+    Groups are numbered by their index, since two groups can share a wording
+    ("aid from Congo to Congo" asks "Congo" for each end). A group's role is
+    the `role_text` of the row its options pin to a candidate, else the role
+    of the sentence slot sharing its `entity_choice`. An option settles the
+    group its `entity_choice` names or, without one, every group its
+    mentions name. Candidates are capped at `_CANDIDATE_CAP`, with a trailing
+    count of the rest.
+    """
+    groups = no_data.get("ambiguous_entities") or []
+    if not groups:
+        return []
+    options = no_data.get("options") or []
+    lines = []
+    for index, group in enumerate(groups):
+        text = group.get("text")
+        candidates = group.get("candidates") or []
+        settling = [
+            option
+            for option in options
+            if option.get("entity_choice") == index
+            or (option.get("entity_choice") is None and text in (option.get("mentions") or []))
+        ]
+        role = _ambiguous_entity_role(
+            settling, refs={candidate.get("ref") for candidate in candidates}
+        ) or next(
+            (
+                token.get("role")
+                for token in sentence_plan
+                if token.get("entity_choice") == index and token.get("role")
+            ),
+            None,
+        )
+        shown = [_candidate_piece(candidate) for candidate in candidates[:_CANDIDATE_CAP]]
+        remaining = len(candidates) - len(shown)
+        tail = f", and {remaining} more" if remaining > 0 else ""
+        named = f'Entity {index + 1}, "{text}"' + (f" ({role})" if role else "")
+        lines.append(f"{named}, could mean: {'; '.join(shown)}{tail}.")
+    lines.append(
+        "Ask the user which one each entity means, then resend its row with the chosen "
+        "candidate's dcid as `dcid`, keeping its `text` and `role`."
+    )
+    return lines
+
+
+def _ambiguous_entity_role(options: Sequence[Mapping[str, Any]], *, refs: set[Any]) -> str | None:
+    """The one `role_text` on the rows `options` pin to a dcid in `refs`, or None when they state none or several."""
+    roles = {
+        entity.get("role_text")
+        for option in options
+        for data_part in (option.get("data_query") or {}).get("parts") or []
+        for entity in data_part.get("entities") or []
+        if entity.get("dcid") in refs and entity.get("role_text")
+    }
+    return roles.pop() if len(roles) == 1 else None
+
+
 def _ambiguous_source_lines(groups: Sequence[Mapping[str, Any]]) -> list[str]:
     """One line per ambiguous source wording, naming each source it could mean.
 
@@ -370,14 +432,14 @@ def _ambiguous_source_lines(groups: Sequence[Mapping[str, Any]]) -> list[str]:
     lines = []
     for group in groups:
         candidates = group.get("candidates") or []
-        shown = [_source_candidate(candidate) for candidate in candidates[:_CANDIDATE_CAP]]
+        shown = [_candidate_piece(candidate) for candidate in candidates[:_CANDIDATE_CAP]]
         remaining = len(candidates) - len(shown)
         tail = f", and {remaining} more" if remaining > 0 else ""
         lines.append(f'"{group.get("text")}" could mean: {"; ".join(shown)}{tail}.')
     return lines
 
 
-def _source_candidate(candidate: Mapping[str, Any]) -> str:
+def _candidate_piece(candidate: Mapping[str, Any]) -> str:
     return f"{candidate.get('label')} ({candidate.get('ref')}, {candidate.get('kind')})"
 
 

services/data-mcp/src/data_mcp/search_tool.py

@@ -72,7 +72,12 @@
 
 
 class SearchEntity(BaseModel):
-    """One entity mentioned in a part, with an optional relational phrase."""
+    """One entity mentioned in a part, with an optional relational phrase.
+
+    `name` or `dcid` says which entity `text` means, so two entities whose
+    words are alike ("aid from Congo to Congo") stay two rows. Search refuses
+    a row carrying both with `request_invalid`, so the model refuses it first.
+    """
 
     model_config = ConfigDict(extra="forbid")
 
@@ -96,6 +101,30 @@ class SearchEntity(BaseModel):
             "set as one place: 'Africa' is {text: 'Africa'}"
         ),
     )
+    name: str | None = Field(
+        default=None,
+        min_length=1,
+        max_length=_MAX_ENTITY_TEXT_CHARS,
+        description=(
+            "the entity's full name, when `text` could name more than one entity: "
+            "'aid from Congo to Congo' sends {text: 'Congo', name: 'Republic of the "
+            "Congo'}. Leave unset beside `dcid`"
+        ),
+    )
+    dcid: str | None = Field(
+        default=None,
+        min_length=1,
+        description=(
+            "the graph dcid of the entity `text` names, e.g. 'country/COG', as a "
+            "search refusal's candidate lists it. Leave unset beside `name`"
+        ),
+    )
+
+    @model_validator(mode="after")
+    def _name_or_dcid(self) -> Self:
+        if self.name is not None and self.dcid is not None:
+            raise ValueError("set name or dcid, not both: a dcid already names the entity")
+        return self
 
 
 class SearchOperation(BaseModel):
@@ -330,6 +359,10 @@ def _wire_entity(entity: SearchEntity) -> dict[str, Any]:
         wire["role_text"] = entity.role
     if entity.members:
         wire["members"] = True
+    if entity.name is not None:
+        wire["name"] = entity.name
+    if entity.dcid is not None:
+        wire["dcid"] = entity.dcid
     return wire
 
 
@@ -571,6 +604,15 @@ async def search(
     the code ambiguous (`source_code_ambiguous`), retry with a `sources` entry
     naming one of the sources it lists.
 
+    Two entities written alike stay two rows when each carries its `name` or
+    `dcid`: "aid from Congo to Congo" sends `{"text": "Congo", "role": "from",
+    "name": "Democratic Republic of the Congo"}` and `{"text": "Congo", "role":
+    "to", "name": "Republic of the Congo"}`. A row takes `name` or `dcid`,
+    never both. When Search refuses `entity_ambiguous`, the digest lists each
+    ambiguous entity by number with its candidates' dcids: ask the user which
+    one is meant, then resend that row with the candidate's `dcid`, keeping
+    its `text` and `role`.
+
     A comparison of two categories of one measure ("health versus education
     aid") is one part per category, each with `separate: true`, and so is
     any other comparison sent as several parts. Search may otherwise merge