Review/decisions 66 67

CHANGELOG.md

@@ -187,6 +187,56 @@ A new profile field, `layout_unit_rule`, `line-units-v1` (the office profiles na
   twelve, sixteen and twenty-four and above a share of the unit's width; in the engine a gap of one
   line height between pieces scored above a half, seven tenths, one and a half and two.
 
+### A line read where the reading order reads it, and its words where the page set them apart (`line-units-v5`, decision #67)
+
+`layout_unit_rule` moves `line-units-v4` -> `line-units-v5`, so `profile_sha256` moves.
+
+- **A line is read where the reading order reads it.** On a page the reading order did not divide,
+  a line was every run on its baseline across the page, read where its first run was: a paragraph
+  whose baselines a byline or a caption in the next column shares was read where that column was,
+  and fell into several units (a ten-line newspaper paragraph read as five). Each line is now cut
+  into its pieces before any unit is read, and a line as the units read it is the pieces of one
+  line that follow each other in reading order.
+- **A word gap the page set by moving its pen reads as a space** where the rule reads a line's
+  text: wherever two runs' boxes stand a tenth of the line's height apart or more. A bullet set
+  apart by its position, with no space drawn after it, now opens a list item, and the hard-break
+  cut reads a line's first word as a word rather than as the whole line.
+- **Measured** on ParseBench: visual grounding 0.5435 -> 0.5575, 119 pages up and 8 down; content
+  one page up by 0.0003; tables, charts and formatting unchanged on every page; overall 46.42 ->
+  46.70. opendataloader-bench, the fabrication check and the gate documents' tables unchanged.
+- **The trade, taken by the owner**: the line spanning two columns gave each unit a far right edge,
+  and that edge cut some units by accident — a contents list's entries (`Marico_Annual_Report_FY24`
+  p4, twice in the set), a paragraph's last line, a label's — which now join; a bullet list the
+  ground truth keeps as one block (`entire-full-report-basf-ar24` p17) is a unit per bullet; and a
+  chart's labels join the text beside them, so its picture holds no text.
+
+### A line chart's printed values read as its table (`point-labels-v1`), and a panel's title names its numbers (`bar-labels-v3`), decision #66
+
+A new profile field, `table_detection.lines` = `point-labels-v1` (office profiles:
+`not-run-for-this-format`), and `table_detection.charts` `bar-labels-v2` -> `-v3`, so
+`profile_sha256` moves. `docs/32-CHART-LABELS-SCOPE.md` §8 and §9 are the rules.
+
+- **A line chart that prints a number at each point is a table**: a stroked line rising left to
+  right, or a run of three or more small marks of one colour, is a series; each number goes to the
+  one point it is set over, under or beside, unless another series' point stands nearly as near; a
+  year is never a value; a series keeps its numbers where six points in ten carry one and a higher
+  point never carries a smaller number; its categories are the first row of labels under the plot.
+  Series sharing that row are one table, each named by its legend swatch (a small mark or a short
+  stroke of its colour) or, alone, by the title over it. A value is never read off a point's
+  height.
+- **The interpreter carries the stroke colour** (`G`, `RG`, `K`, `SC`, `SCN`, reset by `CS`, saved
+  by `q`) and records each stroked subpath's points and each small filled mark with its colour;
+  only the line rule reads them.
+- **`bar-labels-v3`**: the title over a chart names the one series that prints numbers, whatever
+  else is drawn beside it — a progress bar's track no longer leaves its numbers unnamed.
+- **Measured** on ParseBench: charts 0.1184 -> 0.1248, 7 pages up and none down (`She-figures` p88
+  and `E-Government_Survey` p78 0 -> 1.0); tables, content, formatting and visual grounding
+  unchanged on every page; overall 46.29 -> 46.42. No line table on a table or text page; the gate
+  documents' tables unchanged.
+- **The trade, taken by the owner**: opendataloader-bench `01030000000078`'s line, read right
+  against its drawing, reorders that document's labels — its NID 0.8786 -> 0.8099 and TEDS 0.8667 ->
+  0.7086 (bench NID 0.8841 -> 0.8838, TEDS 0.5339 -> 0.5301), as decision #42's bar tables did.
+
 ### A heading's section number and its title are one line, and a label's lines one block (decision #65)
 
 Folded into `markdown-blocks-v12` and `html-blocks-v12`, which no release carries yet, so no rule id

crates/ethos-parser-cli/tests/public_api.rs

@@ -326,6 +326,7 @@ const CORE: &[&str] = &[
     "LAYOUT_UNIT_RULE_V2",
     "LAYOUT_UNIT_RULE_V3",
     "LAYOUT_UNIT_RULE_V4",
+    "LAYOUT_UNIT_RULE_V5",
     // Decision #41. The page-furniture rule id, and the edge a furniture run stands apart at.
     "FURNITURE_RULE_V1",
     "Furniture",
@@ -363,6 +364,9 @@ const CORE: &[&str] = &[
     "MARKDOWN_SCHEMA_VERSION",
     "TABLE_DETECTION_CHARTS_V1",
     "TABLE_DETECTION_CHARTS_V2",
+    // Decision #66.
+    "TABLE_DETECTION_CHARTS_V3",
+    "TABLE_DETECTION_LINES_V1",
     "TABLE_DETECTION_STROKE_V1",
     "TABLE_DETECTION_TAGGED_V1",
     "TABLE_DETECTION_TRACKS_V3",

crates/ethos-parser-core/src/lib.rs

@@ -125,23 +125,23 @@ pub use profile::{
     EPUB_READING_ORDER_RULE_V1, EPUB_TEXT_CODE_RULE_V2, FIGURE_RULE_V1, FORM_ANNOTATION_RULE_V2,
     FURNITURE_RULE_V1, HEADING_INFERENCE_RULE_V4, HEADING_INFERENCE_RULE_V5,
     HEADING_INFERENCE_RULE_V6, HEADING_INFERENCE_RULE_V7, LAYOUT_UNIT_RULE_V1, LAYOUT_UNIT_RULE_V2,
-    LAYOUT_UNIT_RULE_V3, LAYOUT_UNIT_RULE_V4, NOT_RUN, OBSERVATION_RULE_V3, OBSERVATION_RULE_V4,
-    ODP_READING_ORDER_RULE_V1, ODP_TEXT_CODE_RULE_V2, ODS_READING_ORDER_RULE_V1,
-    ODS_TEXT_CODE_RULE_V2, ODT_READING_ORDER_RULE_V1, ODT_TEXT_CODE_RULE_V2,
-    PPTX_READING_ORDER_RULE_V1, PPTX_TEXT_CODE_RULE_V3, READING_ORDER_RULE_V0,
-    READING_ORDER_RULE_V1, READING_ORDER_RULE_V2, READING_ORDER_RULE_V3, READING_ORDER_RULE_V4,
-    RIGHT_TO_LEFT_RULE_V1, RTF_READING_ORDER_RULE_V1, RTF_TEXT_CODE_RULE_V2, STRUCT_TREE_RULE_V1,
-    STRUCT_TREE_RULE_V2, TABLE_DETECTION_CHARTS_V1, TABLE_DETECTION_CHARTS_V2,
-    TABLE_DETECTION_STROKE_V1, TABLE_DETECTION_TAGGED_V1, TABLE_DETECTION_TRACKS_V10,
-    TABLE_DETECTION_TRACKS_V11, TABLE_DETECTION_TRACKS_V12, TABLE_DETECTION_TRACKS_V13,
-    TABLE_DETECTION_TRACKS_V14, TABLE_DETECTION_TRACKS_V3, TABLE_DETECTION_TRACKS_V4,
-    TABLE_DETECTION_TRACKS_V5, TABLE_DETECTION_TRACKS_V6, TABLE_DETECTION_TRACKS_V7,
-    TABLE_DETECTION_TRACKS_V8, TABLE_DETECTION_TRACKS_V9, TABLE_DETECTION_UNRULED_V1,
-    TABLE_DETECTION_V1, TABLE_DETECTION_V10, TABLE_DETECTION_V11, TABLE_DETECTION_V12,
-    TABLE_DETECTION_V2, TABLE_DETECTION_V3, TABLE_DETECTION_V4, TABLE_DETECTION_V5,
-    TABLE_DETECTION_V6, TABLE_DETECTION_V7, TABLE_DETECTION_V8, TABLE_DETECTION_V9,
-    TEXT_CODE_RULE_V2, TEXT_CODE_RULE_V3, TEXT_CODE_RULE_V4, XLSX_READING_ORDER_RULE_V1,
-    XLSX_TEXT_CODE_RULE_V3,
+    LAYOUT_UNIT_RULE_V3, LAYOUT_UNIT_RULE_V4, LAYOUT_UNIT_RULE_V5, NOT_RUN, OBSERVATION_RULE_V3,
+    OBSERVATION_RULE_V4, ODP_READING_ORDER_RULE_V1, ODP_TEXT_CODE_RULE_V2,
+    ODS_READING_ORDER_RULE_V1, ODS_TEXT_CODE_RULE_V2, ODT_READING_ORDER_RULE_V1,
+    ODT_TEXT_CODE_RULE_V2, PPTX_READING_ORDER_RULE_V1, PPTX_TEXT_CODE_RULE_V3,
+    READING_ORDER_RULE_V0, READING_ORDER_RULE_V1, READING_ORDER_RULE_V2, READING_ORDER_RULE_V3,
+    READING_ORDER_RULE_V4, RIGHT_TO_LEFT_RULE_V1, RTF_READING_ORDER_RULE_V1, RTF_TEXT_CODE_RULE_V2,
+    STRUCT_TREE_RULE_V1, STRUCT_TREE_RULE_V2, TABLE_DETECTION_CHARTS_V1, TABLE_DETECTION_CHARTS_V2,
+    TABLE_DETECTION_CHARTS_V3, TABLE_DETECTION_LINES_V1, TABLE_DETECTION_STROKE_V1,
+    TABLE_DETECTION_TAGGED_V1, TABLE_DETECTION_TRACKS_V10, TABLE_DETECTION_TRACKS_V11,
+    TABLE_DETECTION_TRACKS_V12, TABLE_DETECTION_TRACKS_V13, TABLE_DETECTION_TRACKS_V14,
+    TABLE_DETECTION_TRACKS_V3, TABLE_DETECTION_TRACKS_V4, TABLE_DETECTION_TRACKS_V5,
+    TABLE_DETECTION_TRACKS_V6, TABLE_DETECTION_TRACKS_V7, TABLE_DETECTION_TRACKS_V8,
+    TABLE_DETECTION_TRACKS_V9, TABLE_DETECTION_UNRULED_V1, TABLE_DETECTION_V1, TABLE_DETECTION_V10,
+    TABLE_DETECTION_V11, TABLE_DETECTION_V12, TABLE_DETECTION_V2, TABLE_DETECTION_V3,
+    TABLE_DETECTION_V4, TABLE_DETECTION_V5, TABLE_DETECTION_V6, TABLE_DETECTION_V7,
+    TABLE_DETECTION_V8, TABLE_DETECTION_V9, TEXT_CODE_RULE_V2, TEXT_CODE_RULE_V3,
+    TEXT_CODE_RULE_V4, XLSX_READING_ORDER_RULE_V1, XLSX_TEXT_CODE_RULE_V3,
 };
 pub use tables::{
     CellSlot, CheckStatus, GeometricFault, LocatorCheck, SlotCover, SlotFault, TableCellPosition,