Repository navigation
[Engine] Implement Streaming HashAggregation for GROUP BY queries (Q8 bottleneck) #635
Description
Summary
Replace the current in-memory sort-based aggregation in Group.java with a streaming hash aggregation table to resolve the major query latency bottleneck identified on Q8.
Problem Description
On the Bowlogna benchmark (1.23M triples), Corese v5 executes 16 reference queries in 2,226 ms. Query Q8 alone takes 1,137 ms (51.1% of the entire benchmark runtime). While queries without massive grouping beat Apache Jena, Q8 severely lags due to sort-based aggregation which materializes and sorts all solutions in memory ((N \log N)$ time and large heap footprint).
Technical Scope & Objectives
- Accurate Profiling: Confirm the execution breakdown of Q8 (time in pattern join vs time in aggregation).
- Streaming HashAggregation:
- Accumulate grouping keys directly in an on-the-fly hash map:
GroupKey -> AccumulatorSet (Count/Sum/Avg/Min/Max). - Update accumulators iteratively as solution tuples arrive ((N)$ average time).
- Accumulate grouping keys directly in an on-the-fly hash map:
- SPARQL 1.1 Specification Conformance:
- Preserve correct handling of unbound variables, empty groups, and multiple grouping expressions.
- Support
COUNT(DISTINCT ?var)with nested distinct value sets. - Ensure proper error state propagation during aggregate evaluation.
Acceptance Criteria
- Profiling data on Q8 confirms aggregation cost before and after implementation.
- Streaming HashAggregation operates with (N)$ time complexity on streaming inputs.
- Q8 execution time on Bowlogna drops significantly (target: sub-300 ms).
- 100% pass on all existing W3C SPARQL 1.1 aggregate test cases.
Activity
remiceres commentedon Oct 21, 2026
🚀 Step 1 Validée & Committée (
eb1d19bc9)Remplacement de l'ordonnancement par arbre de
Group.javapar une table de hachageLinkedHashMap<GroupKey, Mappings>:Node[]avecequalsethashCoderespectant strictement la spécification SPARQL 1.1 W3C (variables non liéesnull, datatypes, langues, blank nodes).GroupTest.java.corese-core100% verte.remiceres commentedon Oct 21, 2026
⚡ Step 2 Validée & Committée (
da8e5cfba) : Direct Scalar Group AggregationIntroduction du composant
DirectGroupAggregation.javadansMappings.aggregate()pour les requêtesGROUP BYavec projections scalaires :Réalisations :
COUNT,SUM,AVG,MIN,MAX) sont retenus par clé de groupe.NumericAccumulatorpour les promotions numériques transparentes (integerdecimaldouble) et isolation stricte des erreurs de type (une erreur d'évaluation sur une projection invalide cette projection sans impacter les autres).isEligible(Query)pour préserver le chemin standard lors d'usage de modificateurs complexes.Résultats Bowlogna (1,23M triplets) :
GROUP BY ?X+COUNT(?E)) : nouvelle baisse de latence à 760 ms (vs 918 ms après Step 1, et 1 830 ms sous Corese 4.6.5DirectGroupAggregationTest100% passante, et suite complètecorese-core100% verte.remiceres commentedon Oct 21, 2026
Résolu et clôturé par le merge de la PR #639 dans feature/corese-next.