Track joint probabilities of correlated errors

[smadhuk]

This PR extends iter_dem_instructions_include_correlations to track the joint probabilities between components that arise from the same error instruction.

The core of this change is the introduction of a joint_probabilities map which is populated during the DEM iteration. This map stores the probability of any two error components occurring together.

All existing tests have been updated to pass in the joint_probabilities map and verify its final state.

-- Performance before this PR --

[.................*<<|....................] 530 us (vs 290 us) (2.8 Mdets/s) (9.5 Mlayers/s) (1.9 Mshots/s) Decode_surface_r5_d5_p1000
[................*<<<|....................]  23 ms (vs  10 ms) (930 kdets/s) ( 59 klayers/s) (5.3 kshots/s) Decode_surface_r11_d11_p100
[................*<<<|....................] 3.4 ms (vs 1.5 ms) (2.9 Mdets/s) (1.6 Mlayers/s) (140 kshots/s) Decode_surface_r11_d11_p1000
[..................*<|....................] 140 us (vs  83 us) (3.6 Mdets/s) ( 19 Mlayers/s) (1.7 Mshots/s) Decode_surface_r11_d11_p10000
[..................*<|....................]  56 us (vs  33 us) (3.3 Mdets/s) (190 Mlayers/s) ( 17 Mshots/s) Decode_surface_r11_d11_p100000
[.................*<<|....................]  16 ms (vs 7.5 ms) (620 kdets/s) ( 10 klayers/s) (480 shots/s) Decode_surface_r21_d21_p100
[................*<<<|....................]  21 ms (vs 7.8 ms) (470 kdets/s) (7.8 klayers/s) (370 shots/s) Decode_surface_r21_d21_p100_with_dijkstra
[................*<<<|....................]  22 ms (vs 8.2 ms) (450 kdets/s) (7.4 klayers/s) (350 shots/s) Decode_surface_r21_d21_p100_to_edges
[.................*<<|....................]  12 ms (vs 6.3 ms) (2.8 Mdets/s) (410 klayers/s) ( 19 kshots/s) Decode_surface_r21_d21_p1000
[...............*<<<<|....................]  25 ms (vs 7.7 ms) (1.4 Mdets/s) (210 klayers/s) ( 10 kshots/s) Decode_surface_r21_d21_p1000_with_dijkstra
[...............*<<<<|....................]  26 ms (vs 8.4 ms) (1.3 Mdets/s) (200 klayers/s) (9.7 kshots/s) Decode_surface_r21_d21_p1000_to_edges
[................*<<<|....................] 2.4 ms (vs 980 us) (3.1 Mdets/s) (4.4 Mlayers/s) (200 kshots/s) Decode_surface_r21_d21_p10000
[...............*<<<<|....................] 4.2 ms (vs 1.3 ms) (1.8 Mdets/s) (2.5 Mlayers/s) (120 kshots/s) Decode_surface_r21_d21_p10000_with_dijkstra
[...............*<<<<|....................] 4.6 ms (vs 1.4 ms) (1.6 Mdets/s) (2.3 Mlayers/s) (110 kshots/s) Decode_surface_r21_d21_p10000_to_edges
[.................*<<|....................] 180 us (vs  94 us) (3.9 Mdets/s) ( 58 Mlayers/s) (2.7 Mshots/s) Decode_surface_r21_d21_p100000
[...............*<<<<|....................] 370 us (vs 130 us) (1.9 Mdets/s) ( 28 Mlayers/s) (1.3 Mshots/s) Decode_surface_r21_d21_p100000_with_dijkstra
[...............*<<<<|....................] 390 us (vs 130 us) (1.8 Mdets/s) ( 26 Mlayers/s) (1.2 Mshots/s) Decode_surface_r21_d21_p100000_to_edges
[..................*<|....................]  53 ms (vs  35 ms) (180 loads/s) Load_dem_r11_d11_p100
[..................*<|....................] 400 ms (vs 280 ms) ( 24 loads/s) Load_dem_r21_d21_p100
[....................|>*..................]  24 ms (vs  35 ms) (400 loads/s) Load_dem_r11_d11_p100_correlations
[....................|>*..................] 170 ms (vs 280 ms) ( 57 loads/s) Load_dem_r21_d21_p100_correlations
[................*<<<|....................] 9.6 ms (vs 3.5 ms) (1.0 Gcalls/s) Varying32_get_distance_at_time
[..............*<<<<<|....................]  25 ms (vs 5.9 ms) (390 Mcalls/s) Varying64_get_distance_at_time
[....................*....................] 4.9 us (vs 4.9 us) (200 MEnqueueDequeues/s) bucket_queue_sort
[..................*<|....................] 140 us (vs  99 us) ( 67 MEnqueueDequeues/s) bucket_queue_stream

-- Performance after this PR --

[.................*<<|....................] 530 us (vs 290 us) (2.8 Mdets/s) (9.5 Mlayers/s) (1.9 Mshots/s) Decode_surface_r5_d5_p1000
[................*<<<|....................]  23 ms (vs  10 ms) (930 kdets/s) ( 59 klayers/s) (5.4 kshots/s) Decode_surface_r11_d11_p100
[................*<<<|....................] 3.4 ms (vs 1.5 ms) (2.9 Mdets/s) (1.6 Mlayers/s) (140 kshots/s) Decode_surface_r11_d11_p1000
[..................*<|....................] 140 us (vs  83 us) (3.6 Mdets/s) ( 19 Mlayers/s) (1.8 Mshots/s) Decode_surface_r11_d11_p10000
[..................*<|....................]  56 us (vs  33 us) (3.3 Mdets/s) (190 Mlayers/s) ( 18 Mshots/s) Decode_surface_r11_d11_p100000
[.................*<<|....................]  16 ms (vs 7.5 ms) (630 kdets/s) ( 10 klayers/s) (490 shots/s) Decode_surface_r21_d21_p100
[................*<<<|....................]  21 ms (vs 7.8 ms) (480 kdets/s) (7.8 klayers/s) (370 shots/s) Decode_surface_r21_d21_p100_with_dijkstra
[................*<<<|....................]  22 ms (vs 8.2 ms) (450 kdets/s) (7.4 klayers/s) (350 shots/s) Decode_surface_r21_d21_p100_to_edges
[.................*<<|....................]  12 ms (vs 6.3 ms) (2.8 Mdets/s) (420 klayers/s) ( 20 kshots/s) Decode_surface_r21_d21_p1000
[...............*<<<<|....................]  23 ms (vs 7.7 ms) (1.5 Mdets/s) (220 klayers/s) ( 10 kshots/s) Decode_surface_r21_d21_p1000_with_dijkstra
[...............*<<<<|....................]  26 ms (vs 8.4 ms) (1.3 Mdets/s) (200 klayers/s) (9.6 kshots/s) Decode_surface_r21_d21_p1000_to_edges
[................*<<<|....................] 2.3 ms (vs 980 us) (3.2 Mdets/s) (4.5 Mlayers/s) (210 kshots/s) Decode_surface_r21_d21_p10000
[...............*<<<<|....................] 4.2 ms (vs 1.3 ms) (1.8 Mdets/s) (2.5 Mlayers/s) (120 kshots/s) Decode_surface_r21_d21_p10000_with_dijkstra
[...............*<<<<|....................] 4.5 ms (vs 1.4 ms) (1.6 Mdets/s) (2.3 Mlayers/s) (110 kshots/s) Decode_surface_r21_d21_p10000_to_edges
[.................*<<|....................] 180 us (vs  94 us) (3.9 Mdets/s) ( 58 Mlayers/s) (2.7 Mshots/s) Decode_surface_r21_d21_p100000
[...............*<<<<|....................] 370 us (vs 130 us) (1.9 Mdets/s) ( 28 Mlayers/s) (1.3 Mshots/s) Decode_surface_r21_d21_p100000_with_dijkstra
[...............*<<<<|....................] 390 us (vs 130 us) (1.8 Mdets/s) ( 27 Mlayers/s) (1.3 Mshots/s) Decode_surface_r21_d21_p100000_to_edges
[..................*<|....................]  52 ms (vs  35 ms) (190 loads/s) Load_dem_r11_d11_p100
[...................*|....................] 390 ms (vs 280 ms) ( 25 loads/s) Load_dem_r21_d21_p100
[...................*|....................]  43 ms (vs  35 ms) (230 loads/s) Load_dem_r11_d11_p100_correlations
[...................*|....................] 340 ms (vs 280 ms) ( 29 loads/s) Load_dem_r21_d21_p100_correlations
[................*<<<|....................] 9.6 ms (vs 3.5 ms) (1.0 Gcalls/s) Varying32_get_distance_at_time
[..............*<<<<<|....................]  25 ms (vs 5.9 ms) (390 Mcalls/s) Varying64_get_distance_at_time
[....................*....................] 4.9 us (vs 4.9 us) (200 MEnqueueDequeues/s) bucket_queue_sort
[..................*<|....................] 140 us (vs  99 us) ( 67 MEnqueueDequeues/s) bucket_queue_stream

[oscarhiggott]

Looks great, added a comment about sorting edge detector indices

[oscarhiggott]

This makes the assumption that the order of detectors in each edge (d1, d2) is fixed. i.e. that the detectors are always sorted (d1 < d2). This is typically the case in stim but we should not make this assumption in case a DEM is generated elsewhere.

We could ensure when parsing the DEM and storing in the UserGraph that detectors in each edge are always sorted.

[oscarhiggott]

Maybe also worth adding a test with a simple DEM that doesn't have consistently sorted detectors in error mechanisms

[smadhuk]

Done

[oscarhiggott]

Likewise best to ensure nodes are sorted here as well

[smadhuk]

Done

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (correlated-matching@cd734e1). Learn more about missing BASE report.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@                  Coverage Diff                   @@
##             correlated-matching     #127   +/-   ##
======================================================
  Coverage                       ?   99.15%           
======================================================
  Files                          ?        3           
  Lines                          ?      353           
  Branches                       ?        0           
======================================================
  Hits                           ?      350           
  Misses                         ?        3           
  Partials                       ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.