Multi-Modal Behavioral Fraud Detection System
Real-time fraud detection using behavioral biometrics and AI-powered multi-agent analysis
Features • Architecture • Getting Started • Demo • Documentation
QuadFusion is an advanced multi-modal behavioral fraud detection system that leverages AI and machine learning to identify fraudulent activities through behavioral biometrics. Unlike traditional authentication methods, QuadFusion continuously monitors user behavior patterns across multiple dimensions:
- Touch Patterns - Swipe dynamics, tap pressure, gesture recognition
- Typing Behavior - Keystroke dynamics, rhythm analysis, timing patterns
- Voice Authentication - Speaker identification, voice pattern analysis
- Visual Biometrics - Face recognition, scene analysis
- Motion Analysis - Accelerometer, gyroscope, magnetometer data
- App Usage Patterns - Usage frequency, navigation patterns, temporal analysis
The system uses a multi-agent architecture where specialized AI agents analyze different behavioral aspects and a coordinator agent fuses their decisions for robust fraud detection.
- Continuous behavioral biometric monitoring
- Real-time anomaly detection
- Risk scoring with confidence levels
- Session-based fraud analysis
- 6 Specialized Agents:
- Touch Pattern Agent
- Typing Behavior Agent
- Voice Command Agent
- Visual Agent
- Movement Agent
- App Usage Agent
- Coordinator Agent for intelligent decision fusion
- Lightweight models optimized for mobile deployment
- React Native + Expo for cross-platform support
- Real-time sensor data collection
- Live monitoring dashboard
- Beautiful, responsive UI with animations
- Offline-capable with local processing
- End-to-end encryption for biometric data
- On-device processing where possible
- Secure data storage and transmission
- GDPR-compliant data handling
- RESTful API with comprehensive documentation
- Easy integration with existing apps
- Detailed logging and monitoring
- Performance metrics and analytics
QuadFusion/
├── Backend (Python) # AI/ML Processing Server
│ ├── API Server # FastAPI REST endpoints
│ ├── Multi-Agent System # 6 specialized + 1 coordinator
│ ├── Models # ML models (LSTM, CNN, etc.)
│ ├── Data Pipeline # Collection, preprocessing, encryption
│ └── Mobile Deployment # ONNX/TFLite conversion
│
└── Frontend (React Native) # Mobile Application
├── Sensor Managers # Data collection
├── Live Monitoring # Real-time dashboard
├── UI Components # Responsive, animated UI
└── API Client # Backend communication
User Interaction Data
↓
┌───────────────────────────────────┐
│ Specialized Agent Layer │
├───────────────────────────────────┤
│ • TouchPatternAgent (20%) │
│ • TypingBehaviorAgent (15%) │
│ • VoiceCommandAgent (20%) │
│ • VisualAgent (25%) │
│ • MovementAgent (10%) │
│ • AppUsageAgent (10%) │
└───────────────────────────────────┘
↓
┌───────────────────────────────────┐
│ Coordinator Agent │
│ • Weighted fusion │
│ • Confidence aggregation │
│ • Risk level determination │
└───────────────────────────────────┘
↓
Fraud Decision
Backend:
- Python 3.10+
- FastAPI (REST API)
- TensorFlow & PyTorch (Deep Learning)
- Scikit-learn (ML algorithms)
- ONNX/TFLite (Mobile optimization)
- Librosa (Audio processing)
- OpenCV & MediaPipe (Computer Vision)
Frontend:
- React Native 0.79
- Expo 53.0
- TypeScript
- Expo Sensors, Camera, Audio
- Victory Native (Charts)
- React Navigation
- Backend: Python 3.10+, pip
- Frontend: Node.js 18+, npm/yarn
- Mobile: Expo Go app (for testing) or Expo CLI
git clone https://github.com/Samrudhp/OnDevice-Multimodal-Agent.git
cd QuadFusioncd src/backend/src
# Create virtual environment
python3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Start API server
cd ..
python api_server.pyThe backend server will start at `http://127.0.0.1:8000\`
API Documentation: Visit `http://127.0.0.1:8000/docs\` for interactive API docs
cd src/qf
# Install dependencies
npm install
# Start development server
npm run devScan the QR code with Expo Go app to run on your device.
Edit `src/backend/src/config.yaml`:
agents:
coordinator:
agent_weights:
TouchPatternAgent: 0.2
TypingBehaviorAgent: 0.15
VoiceCommandAgent: 0.2
VisualAgent: 0.25
AppUsageAgent: 0.1
MovementAgent: 0.1
risk_thresholds:
low: 0.3
medium: 0.6
high: 0.8# Terminal 1: Start backend
cd src/backend
python api_server.py
# Terminal 2: Start frontend
cd src/qf
npm run devcurl -X POST http://127.0.0.1:8000/api/v1/process/realtime \\
-H "Content-Type: application/json" \\
-d '{
"session_id": "session-123",
"sensor_data": {
"touch_events": [...],
"keystroke_events": [...],
"motion_data": {...},
"audio_data": "base64...",
"image_data": "base64..."
}
}'Response:
{
"anomaly_score": 0.23,
"risk_level": "low",
"confidence": 0.87,
"agent_results": {
"MovementAgent": {
"anomaly_score": 0.15,
"risk_level": "low",
"confidence": 0.9
},
"TouchPatternAgent": {...},
...
}
}The mobile app provides real-time visualization of:
- Sensor data collection (touch, motion, audio, camera)
- Agent analysis results with individual scores
- Risk assessment with confidence levels
- Processing metrics and performance stats
(Add screenshots of your mobile app here)
- API Specification - Complete API reference
- Backend Setup - Detailed backend setup guide
- Mobile Setup - Frontend setup and testing
- Model Documentation - ML model details
- Architecture Docs - System architecture and design
QuadFusion/
├── src/
│ ├── backend/
│ │ ├── api_server.py # Main API server
│ │ ├── API_SPECIFICATION.md # API docs
│ │ └── src/
│ │ ├── agents/ # Multi-agent system
│ │ │ ├── coordinator_agent.py
│ │ │ ├── touch_pattern_agent.py
│ │ │ ├── typing_behavior_agent.py
│ │ │ ├── voice_command_agent.py
│ │ │ ├── visual_agent.py
│ │ │ ├── movement_agent.py
│ │ │ └── app_usage_agent.py
│ │ ├── models/ # ML models
│ │ ├── data/ # Data pipeline
│ │ ├── mobile_deployment/ # Model conversion
│ │ ├── training/ # Model training
│ │ └── utils/ # Utilities
│ │
│ └── qf/ # React Native app
│ ├── app/ # Expo Router pages
│ ├── components/ # UI components
│ ├── lib/ # Utilities
│ │ ├── sensor-manager.ts # Sensor data collection
│ │ ├── api.ts # API client
│ │ └── audio-recorder.ts # Audio recording
│ └── config/ # Configuration
│
├── docs/ # Documentation
└── README.md # This file
Contributions are welcome! Please follow these steps:
- Fork the repository
- Create your feature branch (`git checkout -b feature/AmazingFeature`)
- Commit your changes (`git commit -m 'Add some AmazingFeature'`)
- Push to the branch (`git push origin feature/AmazingFeature`)
- Open a Pull Request
This project is licensed under the MIT License - see the LICENSE file for details.
- Built for Samsung EnnovateX 2025 AI Challenge
- TensorFlow and PyTorch communities
- Expo and React Native teams
- Open-source ML model contributors
Project Repository: https://github.com/Samrudhp/OnDevice-Multimodal-Agent
Built with ❤️ using AI and Multi-Agent Systems
Protecting users through behavioral intelligence