Self-Assessment of Extrinsic Calibration

Publication of the experiments for out paper with the title:

Leveraging Motion Tracking for Sample Weighting in Motion-based Calibration and Self-Assessment

was accepted for publication at The IEEE International Conference on Intelligent Transportation Systems 2025 (ITSC).

Installation

Supported python version: 3.12

virtualenv venv
source venv/bin/activate

Clone excalibur and install it

cd excalibur
pip install .

Install dependencies listed in the requirement file and install the local package

cd ../extrinsic_calibration_assessment
pip install -r requirements.txt
pip install -e .

Setup expected directory structure

python scripts/data_manipulation/setup_dir_structure.py

Experiments

Guide to reproduce the results presented in the paper Leveraging Motion Tracking for Sample Weighting in Motion-Based Calibration and Self-Assessment.

Noise Level Sweep Plot

Generate data required for noise-level sweep plot

python3 scripts/run_scripts/run.py --model ca --seq mh05 --iterations 1000 eval_plot

Generate noise-level sweep plot, csv is exported to /tmp/self_assessment_extrinsic_calibration/sweep_plot/export/<seq>. The raw results can be found under /tmp/self_assessment_extrinsic_calibration/sweep_plot/<seq>. Note that the results presented in Table 1 are based on the raw data of the noise-level sweep plot.

python3 scripts/eval/sweep_plot.py --seq mh05 --export_csv

Self-Assessment Evaluation

To reproduce the results of the self-assessment, first run the noise sequence individually

python3 scripts/run_scripts/run.py --model ca --seq mh02 --noise --level 10

Perform the self-assessment and generate the plot presented in Figure 4. The results are exported as a csv to /tmp/self_assessment_extrinsic_calibration/sa_plot.

python3 scripts/eval/self_assessment.py --seq mh02 --level 10

Run all sequences

Process all sequences ["mh01", "mh02", "mh03", "mh04", "mh05"] and all noise levels ["0", "1", "2", "5", "10", "20"].

python3 scripts/run_scripts/run.py --model ca eval

Print output of multiprocessing results. Without any arguments, all results across all sequences and noise levels are shown. When --seq argument is given, results for all noise levels on given sequence are shown. When --level argument is given, aggregated results across sequences for given noise level is shown. Parameter combination with both --seq and --level is currently not supported.

python3 scripts/eval/mp_results_processing.py [--seq <seq>] [--level <level>]

Usage

Valid models include ["ca", "cv", "cj"]

Run complete pipeline

Run single sequence:

python3 scripts/run_scripts/run.py [--datadir <path_to_datadir>] --model <model_name> --seq <sequence_name> [--noiseparam <param_config_file>] [--noise] [--level <noise_level>] [--viz] [--kf]

Run script to create data for noise level sweep plot, noise levels 0..20 are used, each noise level is run num_iteration times.

python3 scripts/run_scripts/run.py --model <model_name> --seq <sequence_name> --iterations <num_iterations> eval_plot

Run all artificial noise sequences from EuRoC data set including [0, 1, 2, 5, 10, 20] using eval subcommand.

python3 scripts/run_scripts/run.py [--datadir <path_to_datadir>] --model <model_name> eval

Generate Plots

Generate noise level sweep plot

python3 scripts/eval/sweep_plot_proc.py --datadir <path_to_datadir> --seq <sequence_name> [--export_csv] [--export_path <path>] [--show_plot] --max_noise_level <max_noise_level> --noise_step <step> [--boxplot] 

Generate self-assessment plot including estimated extrinsic calibration validity for artificial noise error and extrinsic calibration error.

python3 scripts/eval/self_assessment.py --datadir <path> --seq <sequence_name> --level <noise_level> [--export_csv] [--export_path <path>] [--show_seq] [--show_sa] [--show_hist]