OSINT Investigative Framework

OSINT Investigative Framework (OIF) is a Python-based toolkit for structured open-source intelligence workflows. It helps investigators collect data from common local sources, extract entities, organize findings, persist case data in SQLite, and export reports in multiple formats.

The current project is centered on a single entry point, oif-v1.py, and supports both command-line execution and an interactive investigation shell.

What It Does

• Creates and manages investigation workspaces.
• Ingests common file formats used in OSINT and incident review workflows.
• Extracts entities such as emails, IP addresses, domains, URLs, hashes, wallets, phone numbers, timestamps, MAC addresses, file paths, and social handles.
• Builds findings from timeline, anomaly, and correlation analysis.
• Stores investigation state and findings in SQLite.
• Exports reports to Markdown, JSON, and CSV.
• Optionally watches source folders for new files and re-runs analysis incrementally.
• Optionally uses a local Ollama instance for enhanced analysis.

Supported Investigation Types

• PERSON
• ORGANIZATION
• DOMAIN
• IP_ADDRESS
• EMAIL
• PHONE
• SOCIAL_MEDIA
• CRYPTOCURRENCY
• VEHICLE
• LOCATION
• INCIDENT
• NETWORK
• MALWARE

Supported Source Types

The framework can work with a broad mix of local data sources, including:

• Logs and plain text: .log, .txt, .text, .md, .rst, .ini, .cfg, .conf
• Structured data: .csv, .json, .xml, .yaml, .yml
• Documents: .pdf, .docx, .xlsx, .xls
• Email: .eml, .msg
• Images: .jpg, .jpeg, .png, .gif, .bmp, .tiff, .tif, .webp, .ico, .heic, .heif, .raw, .cr2, .nef, .arw
• Network and compressed inputs: .pcap, .netflow, .conn, .bz2

Support for some formats depends on optional packages listed in requirements.txt.

Requirements

• Python 3.8 or newer
• pip
• Optional: Ollama for local LLM-assisted analysis
• Optional: Tesseract OCR for image text extraction
• Optional: Poppler or PyMuPDF for scanned PDF OCR workflows

Installation

1. Clone the Repository

git clone https://github.com/gs-ai/OIF-v1.git
cd OIF-v1

2. Create a Virtual Environment

Windows PowerShell:

python -m venv oifENV
.\oifENV\Scripts\Activate.ps1

Windows Command Prompt:

python -m venv oifENV
oifENV\Scripts\activate.bat

Linux or macOS:

python3 -m venv oifENV
source oifENV/bin/activate

3. Install Dependencies

pip install -r requirements.txt

4. Verify the Installation

python oif-v1.py --help

Quick Start

Initialize a New Investigation

python oif-v1.py init --name "Case 001" --type incident --targets "example.com,192.168.1.10"

This creates a new investigation directory under INVESTIGATIONS/ with a config.json file.

Run an Investigation

python oif-v1.py run --config ./INVESTIGATIONS/case_001/config.json

Analyze a Single Source Quickly

python oif-v1.py analyze --source ./logs/access.log

Extract Entities from a File

python oif-v1.py extract --source ./document.txt --format json

Search an Investigation Database

python oif-v1.py search --database ./INVESTIGATIONS/case_001/investigation.db --query "192.168"

Command-Line Usage

python oif-v1.py {init,run,analyze,extract,search}

init

Creates a new investigation workspace.

Arguments:

• --name, -n (required): investigation name
• --type, -t: investigation type
• --targets: comma-separated investigation targets
• --sources: comma-separated source paths

Example:

python oif-v1.py init --name "Acme Review" --type organization --sources "./data,./logs"

run

Runs an investigation from an existing config file.

Arguments:

• --config, -c (required): path to config.json

analyze

Runs a quick one-off analysis against a single source path.

Arguments:

• --source, -s (required): file or directory to analyze

extract

Extracts entities from a source file.

Arguments:

• --source, -s (required): source file path
• --format, -f: text or json

search

Searches an investigation SQLite database.

Arguments:

• --database, -d (required): path to investigation.db
• --query, -q (required): search string

Interactive Mode

Run the script with no arguments to start the interactive shell:

python oif-v1.py

Available commands:

Command	Description
help	Show available commands
new <type> <name>	Create a new investigation
load	List available investigations
load <name>	Load an investigation by name
add target <value>	Add a target
add source <path>	Add a file or directory source
run	Execute the loaded investigation
watch	Watch sources for new files
unwatch	Stop watch mode
findings	Show findings
entities	Show extracted entities
export <format>	Export markdown, json, or csv
status	Show current investigation status
models	List available Ollama models
model <name>	Switch the active Ollama model
clear or cls	Clear the terminal
exit or quit	Exit the shell

Notes:

• Interactive mode checks for a local Ollama server and continues without LLM analysis if it is unavailable.
• Folder watching depends on the watchdog package being installed.

Investigation Outputs

Each investigation writes its output to a dedicated directory under INVESTIGATIONS/. Typical files include:

• config.json: saved investigation configuration
• investigation.db: SQLite database for persisted findings and metadata
• report.md: Markdown report
• report.json: JSON report
• report.csv: CSV export when requested
• .cache/: cached intermediate data when caching is enabled
• .processed_files.json: watch-mode file tracking

Configuration

Generated investigations use a JSON configuration file. A typical example:

{
  "name": "Investigation Name",
  "type": "INCIDENT",
  "targets": ["target1", "target2"],
  "data_sources": ["./path/to/data"],
  "output_dir": "./INVESTIGATIONS/investigation_name",
  "api_keys": {},
  "custom_patterns": {},
  "max_depth": 3,
  "timeout": 30,
  "parallel_workers": 4,
  "enable_caching": true,
  "cache_ttl": 3600,
  "report_format": "markdown"
}

Supported config file inputs for run currently include:

• JSON
• Python files that expose a config object

Optional Local AI Analysis

If Ollama is installed and running, the framework can use a local model to enrich analysis workflows.

Typical setup:

ollama serve
ollama pull wizardlm2:latest

The interactive shell also exposes models and model <name> commands for switching between locally available models.

Project Structure

OSINT-Investigative-Framework/
|-- oif-v1.py
|-- requirements.txt
|-- README.md
|-- .gitignore
`-- INVESTIGATIONS/

Development Notes

• The codebase is currently organized as a single large script rather than a package.
• Some optional features are dependency-driven; if a parser or integration is unavailable, the framework falls back where possible and logs the limitation.
• If you plan to publish the repository, add a LICENSE file before release. The current repository does not include one.

Responsible Use

Use this project only for lawful and authorized investigation work. You are responsible for ensuring your workflows comply with applicable laws, regulations, contracts, and organizational policies.