fadimerbay/multimodal-security-analysis

🔒 Multimodal Security Analysis System

Context-Aware Multimodal Fusion for Intelligent Threat Detection

This project implements a multimodal security threat assessment system combining:

• —🖼️ Visual Analysis (YOLOv8n object detection)
• —🔊 Audio Analysis (Whisper speech recognition)
• —🎥 Temporal Analysis (Video motion tracking)
• —🧠 LLM Reasoning (Ollama/Rule-based fusion)

🚀 Features

• —Real-time multimodal analysis
• —Frame-by-frame video processing
• —Cross-modal threat correlation
• —Confidence-based threat scoring
• —Streamlit web interface

📊 Architecture

Image/Video + Audio → Encoders → Multimodal Fusion → LLM Reasoning → Threat Assessment

🎓 Academic Context

Master's Thesis Project (2025) Topic: Multimodal Large Language Models for Security Applications

⚙️ Models Used

• —YOLO: yolov8n (11MB) - Object detection
• —Whisper: base (142MB) - Speech recognition
• —LLM: Ollama (local) / Rule-based reasoning

📝 Note

This deployment uses Rule-based reasoning due to Hugging Face resource constraints. Full LLM-based reasoning (Ollama) is available for local deployment.

🔗 Links

• —GitHub Repository
• —Documentation