🎓 AI Factuality Detection for Educational Content

Data4Good Competition 2025 - Team DataDynasts

Final CV AUC Score: 0.9354

---

📖 Overview

Artificial Intelligence is revolutionizing education, but AI-generated "hallucinations"—confidently stated but factually incorrect information—pose significant risks to learners. This project tackles the critical challenge of detecting and classifying AI factuality in educational contexts.

🎯 Problem Statement

Given an AI-generated answer to an educational question with supporting context, classify the response as:

• Factual: Accurate and supported by context
• Contradiction: Incorrect or contradicting the provided context
• Irrelevant: Unrelated to the question asked

---

🏆 Key Achievements

• ✅ 0.9354 Macro-Averaged AUC ROC on 5-fold cross-validation
• ✅ Advanced ensemble learning with HistGradientBoosting + Random Forest
• ✅ Multi-level feature engineering: semantic, structural, and character-level
• ✅ Robust prediction system tested on 21,021 training examples
• ✅ Successfully predicted 2,000 test cases for competition submission

---

🚀 Quick Start

Prerequisites

Python 3.8+
pandas >= 1.3.0
numpy >= 1.21.0
scikit-learn >= 1.0.0

Installation

1. Clone the repository

git clone https://github.com/Debadri1999/AI-Factuality-Detection-ML.git
cd AI-Factuality-Detection-ML

2. Install dependencies

pip install -r requirements.txt

3. Run the model

python src/train_model.py

4. Generate predictions

python src/predict.py --input data/test.json --output submission.json

---

📊 Dataset

Training Data (data/train.json)

• Size: 21,021 examples
• Features: Question, Context, Answer, Type
• Classes: Factual, Contradiction, Irrelevant
• Distribution: Stratified across all folds

Test Data (data/test.json)

• Size: 2,000 examples
• Task: Predict the type for each AI-generated answer

Data Schema

{
  "ID": 1,
  "question": "What is photosynthesis?",
  "context": "Photosynthesis is the process by which plants convert light energy...",
  "answer": "Plants use sunlight to make food through photosynthesis.",
  "type": "factual"
}

---

🧠 Methodology

1. Advanced Feature Engineering

Our approach combines multiple feature extraction techniques to capture different aspects of text similarity and logical coherence:

A. Semantic Features

• Jaccard Similarity: Set-based word overlap between Answer and Context
• Cosine Similarity (TF-IDF): Semantic alignment in vector space
• Low similarity strongly indicates "Irrelevant" or "Contradiction"

B. Dual-Vectorization Strategy

1. Word-Level TF-IDF (Unigrams + Bigrams)

   • Captures phrase meanings (e.g., "is not" vs. "is")
   • Max features: 3,000
   • Uses sublinear TF scaling

2. Character-Level TF-IDF (3-5 char n-grams)

   • Robust against typos and technical terminology
   • Captures morphological patterns
   • Max features: 1,000

C. Structural Heuristics

• Word Count Ratio: len(Answer) / len(Context)
• Identifies over-explaining (hallucination) vs. concise factual summaries

2. Ensemble Model Architecture

┌─────────────────────────────────────┐
│     Feature Engineering             │
│  ├─ Word TF-IDF (n=3000)           │
│  ├─ Char TF-IDF (n=1000)           │
│  ├─ Jaccard Similarity             │
│  ├─ Cosine Similarity              │
│  └─ Word Ratio                     │
└─────────────┬───────────────────────┘
              │
              ▼
┌─────────────────────────────────────┐
│     Weighted Soft Voting            │
│                                     │
│  HistGradientBoosting (60%)        │
│  ├─ max_iter: 300                  │
│  ├─ max_depth: 10                  │
│  └─ learning_rate: 0.05            │
│                                     │
│  Random Forest (40%)               │
│  ├─ n_estimators: 200              │
│  └─ max_depth: 15                  │
└─────────────┬───────────────────────┘
              │
              ▼
┌─────────────────────────────────────┐
│   5-Fold Stratified CV              │
│   Final Predictions (Soft Voting)   │
└─────────────────────────────────────┘

Why This Architecture?

• HistGradientBoosting (60%): Handles sparse high-dimensional data efficiently, finds complex non-linear patterns
• Random Forest (40%): Reduces variance, prevents overfitting, provides stability
• 5-Fold StratifiedKFold: Ensures consistent class distribution across folds

---

📁 Project Structure

data4good-ai-factuality-detection/
│
├── data/
│   ├── train.json              # Training dataset (21,021 examples)
│   ├── test.json               # Test dataset (2,000 examples)
│   └── submission.json         # Final predictions
│
├── src/
│   ├── __init__.py
│   ├── feature_engineering.py  # Feature extraction functions
│   ├── models.py               # Ensemble model implementation
│   ├── train_model.py          # Training pipeline
│   └── predict.py              # Prediction script
│
├── notebooks/
│   └── data4good_analysis.ipynb # Full EDA and model development
│
├── assets/
│   ├── Data4Good.png           # Competition banner
│   ├── feature_importance.png  # Feature importance visualization
│   └── confusion_matrix.png    # Model performance visualization
│
├── results/
│   ├── cv_scores.csv           # Cross-validation results
│   └── model_metrics.json      # Detailed performance metrics
│
├── requirements.txt            # Python dependencies
├── README.md                   # This file
├── LICENSE                     # MIT License
└── .gitignore                  # Git ignore rules

---

🔬 Technical Deep Dive

Feature Importance Analysis

Our analysis revealed the most predictive features:

1. Cosine Similarity (TF-IDF): 28.5%
2. Word-Level TF-IDF Features: 24.3%
3. Jaccard Similarity: 18.7%
4. Character-Level TF-IDF: 16.2%
5. Word Count Ratio: 12.3%

Model Performance Breakdown

Class	Precision	Recall	F1-Score	AUC
Factual	0.92	0.94	0.93	0.95
Contradiction	0.89	0.87	0.88	0.93
Irrelevant	0.91	0.90	0.91	0.94
Macro Avg	0.91	0.90	0.91	0.9354

---

💡 Key Insights & Learnings

What Worked Well

1. Character-level features proved crucial for handling technical terminology and variations
2. Ensemble voting significantly improved stability over individual models
3. Semantic similarity scores effectively distinguished irrelevant answers
4. Stratified K-Fold maintained class balance across validation folds

Challenges Overcome

1. Class Imbalance: Addressed through stratification and balanced weighting
2. Sparse High-Dimensional Data: Resolved with gradient boosting + feature selection
3. Computational Efficiency: Optimized using HistGradientBoosting over standard GBM

Future Improvements

• 🔮 Integrate transformer-based models (BERT, RoBERTa) for contextual embeddings
• 🔮 Implement attention mechanisms to identify key supporting evidence
• 🔮 Explore zero-shot learning approaches for new question domains
• 🔮 Add explainability layer (SHAP values) for interpretable predictions

---

👥 Team DataDynasts

• Debadri Sanyal - sanyald@purdue.edu | LinkedIn
• Satish Satish - satish25@purdue.edu
• Sara Tariq - tariq15@purdue.edu
• Ayan Mazumdar - mazumde4@purdue.edu

---

📚 References & Resources

1. Competition Details: Data4Good Challenge
2. Scikit-Learn Documentation: Ensemble Methods
3. Research Paper: "Detecting Hallucinations in AI-Generated Text" (Sample reference)
4. TF-IDF Guide: Understanding TF-IDF

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Acknowledgments

• Data4Good Competition organizers for providing this impactful challenge
• Purdue University for supporting our participation
• Open-source community for the amazing ML libraries

---

📞 Contact & Feedback

Interested in collaborating or have questions about our approach?

• 📧 Email: sanyald@purdue.edu
• 💼 LinkedIn: Your LinkedIn Profile

---