Agenda
- EET Implementation and Improvement (Outer Loop)
- Project appraisal w/EET & Poisson sampling (VLC)
Notes
Background
Tim introduced himself as Technical Director at Veitch Lister Consulting (VLC), an Australian transport planning and modelling firm specialising in advanced model development. Matt Richards led the hands-on technical work.
The work was commissioned by the Victorian Department of Transport Planning (DTP). Melbourne, the state's primary city, is forecast to grow from approximately 5 million to 8 million residents, creating significant transport modelling demands. The existing VITM1 is a four-step model; the goal is to migrate to the activity-based VITM2, which must achieve functional parity with VITM1 before the older model can be retired.
The Problem: Explicit Error Term (EET) Runtime Length
A key risk identified by DTP is that VITM2 may not provide sufficient stability for economic appraisal, which is a core use case in Australia and the UK. The proposed mitigation was to implement Explicit Error Terms (EET) in place of traditional Monte Carlo simulation.
As currently implemented, Monte Carlo simulation may lead to illogical agent-level switching between scenarios (e.g., a person switching from PT to car as a result of a walk improvement), due to choice selection method which relies on the ordering of alternatives in a cumulative probability distribution. EET addresses this by drawing fixed random error terms tied to utilities, ensuring that switching between scenarios is consistent with random utility theory.
However, when VLC applied the existing EET branch to VITM2, they encountered a 20–50× runtime slowdown — reducing what should be a 2-day run to a 2-week runtime per scenario per seed, which is impractical.
Solution: Poisson Sampling
VLC pivoted their scope from being focused on sensitivity testing VITM2 for projec economic appraisal, to solving this runtime issue. The approach they developed is to implement Poisson sampling, which replaces the current sampling method used with EET (which draws Z × M random numbers) with a method that draws only Z random numbers — similar to Monte Carlo volume but with far greater cross-scenario stability.
Key findings from simulation tests and full VITM2 runs:
- Runtime: Reduced from 2 weeks back to approximately 2 days and 11 hours — nearly identical to the Monte Carlo baseline of 2 days 10 hours.
- Stability: In simulation tests, Monte Carlo sampling produced entirely different choice sets between base and project cases in 93% of cases; with Poisson (and EET), fewer than 1% of alternatives changed.
- Aggregate results were consistent between Monte Carlo and EET/Poisson approaches, as expected.
- A small (~0.2%) systematic shift in mode share (e.g., car share) was observed between methods; VLC flagged this as warranting further investigation.
- A test scenario (Dingley Bypass road project) showed more stable changes in vehicle kilometres travelled (VKT) across different random seeds with EET/Poisson.
Poisson sampling is applied specifically in the location choice and departure/arrival time choice steps — the only two model components that use sampling.
Jan provided a theoretical complement to VLC's empirical work, exploring the question of which sampling method is most appropriate for explicit error terms.
Key Finding: Random Sampling Is Not Viable
Jan examined the option of eliminating sampling utilities entirely (random sampling). While computationally appealing, this approach introduces severe bias in mean trip lengths at the small sample sizes used in practice (e.g., 30 out of ~1,500 zones ≈ 2%). Eliminating the sampling step (i.e., computing full log sums for all alternatives) would increase runtimes from ~35 seconds to over 30 minutes. Conclusion: importance sampling is essential.
Comparison of Importance Sampling Methods
| Method |
Random Draws |
Unintuitive Switching |
Runtime |
| Monte Carlo |
M (sample size) |
Common |
Fast |
| EET (current) |
Z × M |
None |
Very slow |
| Poisson Sampling |
Z |
Rare (near zero) |
Fast (≈ Monte Carlo) |
Jan confirmed that VLC's Poisson sampling results align with theoretical expectations. Poisson sampling's inclusion probabilities are a strictly increasing function of importance sampling probabilities, meaning only genuine utility improvements can add an alternative to the destination choice set. Some edge-case switching remains possible but is expected to be minimal in practice.
Key Conclusions
Poisson sampling is a "best-of-both-worlds" solution: it nearly eliminates unintuitive agent-level switching (the main benefit of EET) while running at Monte Carlo speed.
- Poisson + EET are complementary, not competing: Poisson sampling addresses the sampling step; EET addresses the simulation/choice step. They can and should be used together.
- Poisson sampling is also compatible with Monte Carlo simulation
- Estimation vs. application: Sampling in estimation and in application are distinct. Poisson sampling does not introduce systematic bias given reasonable simplified utilities; all importance sampling methods were deemed equivalent in application.
- Merge strategy: Jan noted that a merge request incorporating EET fixes, documentation, and testing was opened April 3rd. The question of whether to include Poisson sampling in the same release or add it as an optional feature requires a consortium-level strategy decision.
- Improving the random number generator (being investigated by RSG, WSP, and Driftless) remains complementary and would further reduce runtimes for both EET and Poisson approaches.
Action Items & Next Steps
- Joe Castiglione to agendize the EET/Poisson integration strategy for the Thursday engineering meeting and discussion with Jeff
- Jan Zill / VLC / RSG to coordinate (possibly asynchronously or at an Australia-compatible engineering meeting) on merging Poisson sampling changes into the codebase.
- Engineering team to decide whether Poisson sampling enters the main branch as part of the current release or is added as a developer/optional feature.
Agenda
Notes
Presentation #1: Poisson Sampling in VITM2 (Tim Veitch & Matt Richards, VLC)
Background
Tim introduced himself as Technical Director at Veitch Lister Consulting (VLC), an Australian transport planning and modelling firm specialising in advanced model development. Matt Richards led the hands-on technical work.
The work was commissioned by the Victorian Department of Transport Planning (DTP). Melbourne, the state's primary city, is forecast to grow from approximately 5 million to 8 million residents, creating significant transport modelling demands. The existing VITM1 is a four-step model; the goal is to migrate to the activity-based VITM2, which must achieve functional parity with VITM1 before the older model can be retired.
The Problem: Explicit Error Term (EET) Runtime Length
A key risk identified by DTP is that VITM2 may not provide sufficient stability for economic appraisal, which is a core use case in Australia and the UK. The proposed mitigation was to implement Explicit Error Terms (EET) in place of traditional Monte Carlo simulation.
As currently implemented, Monte Carlo simulation may lead to illogical agent-level switching between scenarios (e.g., a person switching from PT to car as a result of a walk improvement), due to choice selection method which relies on the ordering of alternatives in a cumulative probability distribution. EET addresses this by drawing fixed random error terms tied to utilities, ensuring that switching between scenarios is consistent with random utility theory.
However, when VLC applied the existing EET branch to VITM2, they encountered a 20–50× runtime slowdown — reducing what should be a 2-day run to a 2-week runtime per scenario per seed, which is impractical.
Solution: Poisson Sampling
VLC pivoted their scope from being focused on sensitivity testing VITM2 for projec economic appraisal, to solving this runtime issue. The approach they developed is to implement Poisson sampling, which replaces the current sampling method used with EET (which draws Z × M random numbers) with a method that draws only Z random numbers — similar to Monte Carlo volume but with far greater cross-scenario stability.
Key findings from simulation tests and full VITM2 runs:
Poisson sampling is applied specifically in the location choice and departure/arrival time choice steps — the only two model components that use sampling.
Presentation 2: Explicit Error Terms Project update (Jan Zill, Outer Loop)
Jan provided a theoretical complement to VLC's empirical work, exploring the question of which sampling method is most appropriate for explicit error terms.
Key Finding: Random Sampling Is Not Viable
Jan examined the option of eliminating sampling utilities entirely (random sampling). While computationally appealing, this approach introduces severe bias in mean trip lengths at the small sample sizes used in practice (e.g., 30 out of ~1,500 zones ≈ 2%). Eliminating the sampling step (i.e., computing full log sums for all alternatives) would increase runtimes from ~35 seconds to over 30 minutes. Conclusion: importance sampling is essential.
Comparison of Importance Sampling Methods
Jan confirmed that VLC's Poisson sampling results align with theoretical expectations. Poisson sampling's inclusion probabilities are a strictly increasing function of importance sampling probabilities, meaning only genuine utility improvements can add an alternative to the destination choice set. Some edge-case switching remains possible but is expected to be minimal in practice.
Key Conclusions
Poisson sampling is a "best-of-both-worlds" solution: it nearly eliminates unintuitive agent-level switching (the main benefit of EET) while running at Monte Carlo speed.
Action Items & Next Steps