Skip to content

[Perf/Engine] Optimizer pipeline & high-throughput bulk ingestion (Bowlogna parity with Jena/RDF4J) #633

Closed
Closed
[Perf/Engine] Optimizer pipeline & high-throughput bulk ingestion (Bowlogna parity with Jena/RDF4J)#633
Assignees
remiceres
Labels
featureNew feature or capabilityrefactoringInternal code restructuring and architecture cleanup

Description

@remiceres
remiceres
opened on Oct 20, 2026

Context & Problem

During the scientific 4-engine benchmark on Bowlogna (1.23M triples, 17 queries) comparing Corese 4.6.5, Corese v5, Apache Jena 6.2.0, and Eclipse RDF4J 5.3.2:

  • Corese v5 achieved major speedups on multi-joins/Top-K queries (Q13 /20, Q7 at 1ms, Q11 at 1ms), but experienced a 2.2x regression on Q8 (1,830ms vs 814ms v4 and 220ms RDF4J).
  • Ingestion of Department0.nt was ~8x slower than Jena/RDF4J (36-45s vs 4.5s) due to per-line ANTLR4 ParseTree allocations.

Objectives & Deliverables

  1. Phase A1 — DISTINCT bypass on GROUP BY: Skip redundant getDistinct() hash table when all GROUP BY variables are projected in SELECT (W3C SPARQL 1.1 compliant). [Completed - commit ce65602]
  2. Phase A2 — Selective Filter Pushdown: Guide pattern ordering with comparison filter edges (<, <=, >, >=, =) to minimize intermediate candidate solutions. [Completed - commit 2cd0b36]
  3. Phase D — Fast N-Triples Streaming Scanner: Replace per-line ANTLR NTriples.g4 parser with a zero-copy streaming scanner in NTriplesParser.java (cutting load time from 45.5s down to 11.1s, 100% W3C compliant across 70 tests). [Completed - commit 16c79d4]
  4. Phase C — Cost-Based Optimizer (CBO) & Dynamic Join Ordering: Recompute join costs at each step taking bound variables into account (SPO probe vs unbounded scan).
  5. Phase B — Streaming HashAggregation: Optimize Group.java to avoid sort-based grouping and reduce memory pressure on large result sets.

Branch: perf/engine-optimizer-and-bulk-loader

Activity

  1. remiceres
    added
    featureNew feature or capability
    refactoringInternal code restructuring and architecture cleanup
    on Oct 20, 2026
  2. remiceres
    self-assigned this
    on Oct 20, 2026

  3. remiceres commented on Oct 20, 2026

    @remiceres
    remiceres
    on Oct 20, 2026
    ContributorAuthor

    Progress Update (Phase D - High-Throughput Streaming Ingestion)

    1. Modernized N-Triples & N-Quads to zero-copy streaming scanners:
      • Replaced ANTLR4-based NTriplesParser and NQuadsParser.
      • Deleted dead grammar/listener code: NTriples.g4, NQuads.g4, NTriplesListener.java, NQuadsListener.java, and LineRDFParser.java.
      • Cleaned up build overhead: Gradle no longer compiles ANTLR grammars for the N-Family.
    2. Performance Impact:
      • Loading 1.23M triples (Department0.nt) dropped from 45.5s down to 11.1s (4x speedup, 136k triples/s parsing into live 3-index SPO/POS/OSP storage).
    3. Quality & Conformance:
      • 0 SonarLint alerts (cognitive complexity <= 11).
      • 100% W3C conformance on official RDF 1.1 N-Triples (70/70) and N-Quads test suites.
      • Commits: 16c79d432 (N-Triples) and f512e8740 (N-Quads) pushed to perf/engine-optimizer-and-bulk-loader.
  4. remiceres
    added a commit that references this issue on Oct 20, 2026

    perf(engine): dynamic cost-based join reordering in CardinalityPatter…

    1956076

  5. remiceres commented on Oct 20, 2026

    @remiceres
    remiceres
    on Oct 20, 2026
    ContributorAuthor

    ⚡ Phase C Validée & Mergée : CBO Dynamic Cost-Based Join Reordering

    Le patch CBO dynamique a été commité et poussé (195607694).

    Réalisations techniques :

    • Recalcul dynamique du coût effectif : lors de la sélection gloutonne des patterns dans CardinalityPatternOrdering.java, le coût d'un pattern candidat est réévalué à chaque étape en fonction des variables déjà liées par les arêtes précédentes ou les scopes englobants.
    • Réduction proportionnelle sans débordement : utilisation de BigDecimal.movePointLeft() pour préserver la précision intégrale long :
      • Sujet lié : division par 0^3$ (réduction SPO index probe).
      • Objet lié : division par 10 (POS/OSP probe).
      • Prédicat variable lié : division par 10.
      • Constantes dans la requête : déjà comptabilisées dans baseCount (pas de double réduction).
    • Maintien de la connectivité en priorité absolue : les jointures connectées restent prioritaires sur les produits cartésiens non connectés.
    • Tests & Qualité :
      • 33/33 tests unitaires CardinalityQueryPlannerTest passent (9 nouveaux tests de régression, précision et tie-breaking).
      • SonarLint LSP : 0 alerte, complexité cognitive $\le 15$.
      • Suite de tests complète corese-core : 100% succès.

    Impact Benchmark Bowlogna (1,23M triplets) :

    • Query 14 (multi-join) : 645 ms (vs 1 016 ms avant CBO dynamique, et 1 560 ms sous Corese 4.6.5 $\rightarrow$ accélération de 2,4x).
    • Query 9 : 218 ms (vs 706 ms sous Corese 4.6.5 $\rightarrow$ accélération de 3,2x).

  6. remiceres commented on Oct 20, 2026

    @remiceres
    remiceres
    on Oct 20, 2026
    ContributorAuthor

    Résolu et clôturé par le merge de la PR #634 dans feature/corese-next.

  7. remiceres
    closed this as completedon Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

featureNew feature or capabilityrefactoringInternal code restructuring and architecture cleanup

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions