Boosting Handwriting-Based Alzheimer's Detection through Late Fusion with Speech

Master's Thesis project for the Master of Science in Computer Engineering at the University of Palermo (UNIPA).

Overview

This repository contains the complete research, experimental codebase, and LaTeX source for my thesis. The project investigates early dementia detection (CTR vs MCI vs MILD-AD) by combining handwriting and ASR-generated speech transcriptions. It features a multimodal deep learning pipeline operating on entire handwriting sequences, alongside traditional machine learning baselines based on stroke-level extraction.

Repository Structure

• code/: The experimental pipeline.
  • Models: CRNN for handwriting, UmBERTo for Italian text.
  • Fusions: Conditioning with FiLM layers and cross-modal late fusion.
  • Baselines: Traditional ML baselines (Random Forest with Bayesian optimization, stacking meta-learners, and task ranking ensembles).
• writing/: LaTeX source for the manuscript and presentation.
  • Full thesis structure (Abstract, Introduction, Related Work, Dataset, Architecture, Results, Conclusions).
  • Beamer presentation slides.

Quick Start

1. Code & Experiments

To reproduce results or train models:

cd code/
conda env create -f environment.yaml
conda activate multimodal_ad
# Follow code/README.md for data prep and training

2. Thesis & Presentation

To compile the documents (requires a TeX distribution):

cd writing/
./build.sh

License

This project is dual-licensed:

• Code: MIT License
• Thesis & Writing: CC BY-NC-SA 4.0

Citation

If you use this work in your research, please cite it as:

@mastersthesis{bonura2026boosting,
  author = {Bonura, Davide},
  title = {{Boosting Handwriting-Based Alzheimer's Detection through Late Fusion with Speech}},
  school = {University of Palermo},
  year = {2026},
  type = {Master's Thesis},
  url = {https://github.com/dqvid3/handwriting-speech-alzheimer-fusion}
}

---

Author: Davide Bonura
Supervisor: Prof. Sabato Marco Siniscalchi
Co-supervisor: Dr. Moreno La Quatra
Academic Year: 2024/2025