Pinpoint cause of chaotic divergence in multi-architecture comparisons

[wdconinc]

That's expected. There's chaotic divergence as soon as some tiny floating point difference occurs anywhere. And for gun simulations we don't even simulate the same events so there's that too.

It would be really interesting to dig into this further, though. I don't think we can really get rid of differences in floating point behavior, but taking the concept of event seeds and applying it at a track level in Geant4 should be able to resolve a lot of these differences, I think.

Originally posted by @wdconinc in #336 (comment)

[wdconinc]

taking the concept of event seeds and applying it at a track level in Geant4 should be able to resolve a lot of these differences, I think.

This likely would not work. The issue is not limited to tracks propagating differently but also involves different tracks being generated, and additional tracks would again shift the RNG. We would almost have to seed tracks based on their full parentage chain to become immune to that, and geant4 does not have the data model to support this (track ID per parent).