Skip to content

MohamedMostafa010/MISP_ThreatIntelJourney

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

59 Commits
images
images
 
 
scripts
scripts
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 

Repository files navigation

README – MISP Threat Intelligence Project or Guide (Docker Deployment + APT18 Analysis + Automation + Additional Tasks)

Screenshot 0

✅ Project Overview

This project is fully completed, including all main tasks and all additional requirements. The entire implementation was performed inside a Kali Linux virtual machine, as Linux provides a much more suitable environment for deploying and managing MISP, Docker services, Python scripts, and security-oriented tooling.

Screenshot 1

Our deployment includes:

  • MISP installed via Docker Compose
  • APT threat intelligence ingestion (APT18 / Dynamite Panda)
  • Feed synchronization (CIRCL, MalwareBazaar, URLhaus)
  • Custom tags
  • Python automation for feed parsing/unification
  • PostgreSQL + Grafana statistics pipeline
  • Email alerting system configured inside the Docker container

1. MISP Installation Using Docker Compose

We deployed the official MISP Docker environment on our Kali Linux VM.

Before running the stack, we first installed the necessary Docker components: Docker Engine (docker.io) and Docker Compose, to enable container management and orchestration.

Screenshot 2

After installing Docker, we cloned the official MISP Docker repository from GitHub, then copied the template.env file to .env and adjusted the required environment variables.

Screenshot 3 Screenshot 4

Once the environment file was configured, we executed: sudo docker-compose up -d

Screenshot 5

To launch all MISP containers. After the services started, we logged in using the default credentials provided in the official GitHub documentation.

Screenshot 6

Last thing, to ensure our instance is running safely.

Screenshot 7 Screenshot 8

2. APT Group Research & Event Creation

We selected APT18, also known as:

  • Dynamite Panda
  • TG-0416
  • Threat Group-0416
  • Wekby

MITRE ATT&CK official info about this APT:

Screenshot 9 Screenshot 10 Screenshot 11

We gathered public information (history, TTPs, IOCs) from the following open-source intelligence sources:

All collected intelligence was aggregated into a single structured event named:

APT18 (Dynamite Panda) – Public Intelligence Summary and Known TTPs

Screenshot 12

Event information (Galaxies, Galaxy Matrix, Correlation Graph, Attributes, Objects, and Event Reports) after being completely finished:

Event Viewing (with custom tags applied):

Screenshot 13

Galaxies:

Screenshot 14

Galaxy Matrix:

Screenshot 15

Correlation Graph:

Screenshot 16

Attributes:

Screenshot 17 Screenshot 18 Screenshot 19

Objects:

Screenshot 20

Event Reports:

Screenshot 21

3. Enabling & Synchronizing MISP Feeds

We activated and successfully fetched data from the following feeds:

  • CIRCL OSINT Feed
  • MalwareBazaar
  • URLhaus

All feeds synchronized correctly, and events were imported into our MISP instance.

Screenshot 22

CIRCL Events:

Screenshot 23

Abuse.ch Events:

Screenshot 24

4. Creating & Applying Custom Tags

The following APT18-related custom tags were created:

  1. custom:china-state-espionage

    • APT18 is linked to Chinese state-sponsored intelligence operations.

  2. custom:intellectual-property-theft

    • APT18 is known for targeting biotech, telecom, and defense IP.

  3. custom:zero-day-exploitation

    • APT18 frequently leverages zero-day vulnerabilities.
    • APT18 abused the leaked Flash zero-day CVE-2015-5119.

Screenshot 25

All three custom tags were applied to the APT18 event we created as shown above.

5. Feed Parsing & Unification Using Python

Before creating the Python scripts, we first retrieved our API key from: Administration → List Auth Keys, where we generated and copied the required MISP Auth Key for programmatic access.

Screenshot 26

We then developed two Python scripts:

MISP_FeedExplorer.py

  • Lists all available MISP feeds
  • Extracts metadata for inspection
  • Feeling free to choose which MISP feed you want to parse

Script Execution:

┌──(venv)─(kali㉿kali)-[~/misp-docker]
└─$ python MISP_FeedExplorer.py
[+] Connected to MISP

[+] Total Feeds Found: 90

=== AVAILABLE MISP FEEDS ===

ID: 1
Name: CIRCL OSINT Feed
Provider: CIRCL
Enabled: True
----------------------------------------
ID: 2
Name: The Botvrij.eu Data
Provider: Botvrij.eu
Enabled: False
----------------------------------------
ID: 3
Name: ELLIO: IP Feed (Community version)
Provider: ellio.tech
Enabled: False
----------------------------------------
ID: 4
Name: blockrules of rules.emergingthreats.net
Provider: rules.emergingthreats.net
Enabled: False
----------------------------------------
ID: 5
Name: Tor exit nodes
Provider: TOR Node List from dan.me.uk - careful, this feed applies a lock-out after each pull. This is shared with the "Tor ALL nodes" feed.
Enabled: False
----------------------------------------
ID: 6
Name: Tor ALL nodes
Provider: TOR Node List from dan.me.uk - careful, this feed applies a lock-out after each pull. This is shared with the "Tor exit nodes" feed.
Enabled: False
----------------------------------------
ID: 7
Name: cybercrime-tracker.net - all
Provider: cybercrime-tracker.net
Enabled: False
----------------------------------------
ID: 8
Name: Phishtank online valid phishing
Provider: Phishtank
Enabled: False
----------------------------------------
ID: 9
Name: ip-block-list - snort.org
Provider: https://snort.org
Enabled: False
----------------------------------------
ID: 10
Name: diamondfox_panels
Provider: pan-unit42
Enabled: False
----------------------------------------
ID: 11
Name: pop3gropers
Provider: home.nuug.no
Enabled: False
----------------------------------------
ID: 12
Name: Feodo IP Blocklist
Provider: abuse.ch
Enabled: False
----------------------------------------
ID: 13
Name: OpenPhish url list
Provider: openphish.com
Enabled: False
----------------------------------------
ID: 14
Name: firehol_level1
Provider: iplists.firehol.org
Enabled: False
----------------------------------------
ID: 15
Name: IPs from High-Confidence DGA-Based C&Cs Actively Resolving - requires a valid license
Provider: osint.bambenekconsulting.com
Enabled: False
----------------------------------------
ID: 16
Name: Domains from High-Confidence DGA-based C&C Domains Actively Resolving
Provider: osint.bambenekconsulting.com
Enabled: False
----------------------------------------
ID: 17
Name: ci-badguys.txt
Provider: cinsscore.com
Enabled: False
----------------------------------------
ID: 18
Name: alienvault reputation generic
Provider: .alienvault.com
Enabled: False
----------------------------------------
ID: 19
Name: blocklist.de/lists/all.txt
Provider: blocklist.de
Enabled: False
----------------------------------------
ID: 20
Name: VNC RFB
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 21
Name: sshpwauth.txt
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 22
Name: sipregistration
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 23
Name: sipquery
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 24
Name: sipinvitation
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 25
Name: DNS recursion desired
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 26
Name: DNS recursion desired IN ANY
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 27
Name: DNS CH TXT version.bind
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 28
Name: IP protocol 41
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 29
Name: SMTP data
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 30
Name: SMTP greet
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 31
Name: TELNET login
Provider: dataplane.org
Enabled: False
----------------------------------------
ID: 32
Name: All current domains belonging to known malicious DGAs
Provider: osint.bambenekconsulting.com
Enabled: False
----------------------------------------
ID: 33
Name: VXvault - URL List
Provider: VXvault
Enabled: False
----------------------------------------
ID: 34
Name: abuse.ch SSL IPBL
Provider: abuse.ch
Enabled: False
----------------------------------------
ID: 35
Name: http://cybercrime-tracker.net hashlist
Provider: http://cybercrime-tracker.net hashlist
Enabled: False
----------------------------------------
ID: 36
Name: http://cybercrime-tracker.net gatelist
Provider: http://cybercrime-tracker.net gatelist
Enabled: False
----------------------------------------
ID: 37
Name: blocklist.greensnow.co
Provider: greensnow.co
Enabled: False
----------------------------------------
ID: 38
Name: This list contains all domains - A list for administrators to prevent mining in networks
Provider: ZeroDot1 - CoinBlockerLists
Enabled: False
----------------------------------------
ID: 39
Name: This list contains all optional domains - An additional list for administrators
Provider: ZeroDot1 - CoinBlockerLists
Enabled: False
----------------------------------------
ID: 40
Name: This list contains all browser mining domains - A list to prevent browser mining only
Provider: ZeroDot1 - CoinBlockerLists
Enabled: False
----------------------------------------
ID: 41
Name: URLHaus Malware URLs
Provider: abuse.ch
Enabled: True
----------------------------------------
ID: 42
Name: CyberCure - IP Feed
Provider: www.cybercure.ai
Enabled: False
----------------------------------------
ID: 43
Name: CyberCure - Blocked URL Feed
Provider: www.cybercure.ai
Enabled: False
----------------------------------------
ID: 44
Name: CyberCure - Hash Feed
Provider: www.cybercure.ai
Enabled: False
----------------------------------------
ID: 45
Name: ipspamlist
Provider: ipspamlist
Enabled: False
----------------------------------------
ID: 46
Name: malsilo.url
Provider: MalSilo
Enabled: False
----------------------------------------
ID: 47
Name: malsilo.ipv4
Provider: MalSilo
Enabled: False
----------------------------------------
ID: 48
Name: malsilo.domain
Provider: MalSilo
Enabled: False
----------------------------------------
ID: 49
Name: malshare.com - current all
Provider: malshare.com
Enabled: False
----------------------------------------
ID: 50
Name: Panels Tracker
Provider: Benkow.cc
Enabled: False
----------------------------------------
ID: 51
Name: IPsum (aggregation of all feeds) - level 1 - lot of false positives
Provider: IPsum
Enabled: False
----------------------------------------
ID: 52
Name: IPsum (aggregation of all feeds) - level 2 - medium false positives
Provider: IPsum
Enabled: False
----------------------------------------
ID: 53
Name: IPsum (aggregation of all feeds) - level 3 - low false positives
Provider: IPsum
Enabled: False
----------------------------------------
ID: 54
Name: IPsum (aggregation of all feeds) - level 4 - very low false positives
Provider: IPsum
Enabled: False
----------------------------------------
ID: 55
Name: IPsum (aggregation of all feeds) - level 5 - ultra false positives
Provider: IPsum
Enabled: False
----------------------------------------
ID: 56
Name: IPsum (aggregation of all feeds) - level 6 - no false positives
Provider: IPsum
Enabled: False
----------------------------------------
ID: 57
Name: IPsum (aggregation of all feeds) - level 7 - no false positives
Provider: IPsum
Enabled: False
----------------------------------------
ID: 58
Name: IPsum (aggregation of all feeds) - level 8 - no false positives
Provider: IPsum
Enabled: False
----------------------------------------
ID: 59
Name: DigitalSide Threat-Intel OSINT Feed
Provider: osint.digitalside.it
Enabled: False
----------------------------------------
ID: 60
Name: Metasploit exploits with CVE assigned
Provider: eCrimeLabs
Enabled: False
----------------------------------------
ID: 61
Name: Malware Bazaar
Provider: abuse.ch
Enabled: True
----------------------------------------
ID: 62
Name: PhishScore
Provider: PhishStats
Enabled: False
----------------------------------------
ID: 63
Name: Threatfox
Provider: abuse.ch
Enabled: False
----------------------------------------
ID: 64
Name: MalwareBazaar
Provider: abuse.ch
Enabled: True
----------------------------------------
ID: 65
Name: URLhaus
Provider: abuse.ch
Enabled: True
----------------------------------------
ID: 66
Name: URL Seen in honeypots
Provider: APNIC Community Honeynet Project
Enabled: False
----------------------------------------
ID: 67
Name: SSH Bruteforce IPs
Provider: APNIC Community Honeynet Project
Enabled: False
----------------------------------------
ID: 68
Name: Telnet Bruteforce IPs
Provider: APNIC Community Honeynet Project
Enabled: False
----------------------------------------
ID: 69
Name: threatfox indicators of compromise
Provider: abuse.ch
Enabled: False
----------------------------------------
ID: 70
Name: James Brine Bruteforce IPs
Provider: jamesbrine.com.au
Enabled: False
----------------------------------------
ID: 71
Name: List of malicious domains in Poland
Provider: CERT-PL
Enabled: False
----------------------------------------
ID: 72
Name: List of malicious hashes
Provider: Banco do Brasil S.A
Enabled: False
----------------------------------------
ID: 73
Name: Shreshta: Newly Registered domain names(NRD) - 1 week (Community policy feed)
Provider: shreshtait.com
Enabled: False
----------------------------------------
ID: 74
Name: Shreshta: Newly Registered domain names (NRD) - 1 month (Community policy feed)
Provider: shreshtait.com
Enabled: False
----------------------------------------
ID: 75
Name: Infoblox-Threat-Intelligence
Provider: infoblox.com
Enabled: False
----------------------------------------
ID: 76
Name: Threatview.io - OSINT Threat Feed
Provider: threatview.io
Enabled: False
----------------------------------------
ID: 77
Name: Threatview.io - C2 Hunt Feed
Provider: threatview.io
Enabled: False
----------------------------------------
ID: 78
Name: Threatview.io - IP Blocklist
Provider: threatview.io
Enabled: False
----------------------------------------
ID: 79
Name: Threatview.io - Domain Blocklist
Provider: threatview.io
Enabled: False
----------------------------------------
ID: 80
Name: Threatview.io - MD5 Hash Blocklist
Provider: threatview.io
Enabled: False
----------------------------------------
ID: 81
Name: Threatview.io - URL Blocklist
Provider: threatview.io
Enabled: False
----------------------------------------
ID: 82
Name: Threatview.io - Bitcoin Address Intel
Provider: threatview.io
Enabled: False
----------------------------------------
ID: 83
Name: Threatview.io - SHA File Hash Blocklist
Provider: threatview.io
Enabled: False
----------------------------------------
ID: 84
Name: Rösti - Repackaged Öpen Source Threat Intelligence
Provider: bin.re
Enabled: False
----------------------------------------
ID: 85
Name: Phishing.Database - New domains of today
Provider: Phishing.Database
Enabled: False
----------------------------------------
ID: 86
Name: Phishing.Database - New IPs of today
Provider: Phishing.Database
Enabled: False
----------------------------------------
ID: 87
Name: Phishing.Database - New URLs of today
Provider: Phishing.Database
Enabled: False
----------------------------------------
ID: 88
Name: hideNseek LAB - Threat Intelligence JSON
Provider: hideNseek LAB
Enabled: False
----------------------------------------
ID: 89
Name: hideNseek LAB - CSV Threat Feed
Provider: hideNseek LAB
Enabled: False
----------------------------------------
ID: 90
Name: hideNseek LAB - IP Blocklist
Provider: hideNseek LAB
Enabled: False
----------------------------------------

[+] Feed listing completed.

MISP_FeedExtractor.py

  • Selects feeds to parse
  • Extracts indicators
  • Saves unified feed output into: selected_feeds_indicators.json

Script Execution:

┌──(venv)─(kali㉿kali)-[~/misp-docker]
└─$ python MISP_FeedExtractor.py
[+] Connected to MISP
[+] Processing feed ID 1
[+] Fetched 118306 indicators from feed ID 1
[+] Processing feed ID 64
[+] Fetched 118306 indicators from feed ID 64
[+] Saved 236612 indicators to selected_feeds_indicators.json

selected_feeds_indicators.json Sample:

[
    {
        "event_id": 1,
        "feed_id": 1,
        "type": "sha256",
        "category": "Payload delivery",
        "value": "4d62caef1ca8f4f9aead7823c95228a52852a1145ca6aaa58ad8493e042aed16",
        "timestamp": "2025-12-02T04:10:47+00:00"
    },
    {
        "event_id": 1,
        "feed_id": 1,
        "type": "sha256",
        "category": "Payload delivery",
        "value": "1b341dab023de64598d80456349db146aafe9b9e2ec24490c7d0ac881cecc094",
        "timestamp": "2025-12-02T04:10:40+00:00"
    },
    {
        "event_id": 1,
        "feed_id": 1,
        "type": "sha256",
        "category": "Payload delivery",
        "value": "9200f80c08b21ebae065141f0367f9c88f8fed896b0b4af9ec30fc98c606129b",
        "timestamp": "2025-12-02T04:10:43+00:00"
    },
    {
        "event_id": 1,
        "feed_id": 1,
        "type": "sha256",
        "category": "Payload delivery",
        "value": "8ffbb7a80efa9ee79e996abde7a95cf8dc6f9a41f9026672a8dbd95539fea82a",
        "timestamp": "2025-12-02T04:10:37+00:00"
    },
    {
        "event_id": 193,
        "feed_id": 64,
        "type": "md5",
        "category": "Payload delivery",
        "value": "5c71adfbc0bb045e5097d3135d92ac1b",
        "timestamp": "2014-11-21T09:19:33+00:00"
    },
    {
        "event_id": 193,
        "feed_id": 64,
        "type": "md5",
        "category": "Payload delivery",
        "value": "5cea24fb20763d255c67efe2b3fc9cc6",
        "timestamp": "2014-11-21T09:19:33+00:00"
    },
    {
        "event_id": 193,
        "feed_id": 64,
        "type": "md5",
        "category": "Payload delivery",
        "value": "5e10288c9d6b6cd1c6fe6e401cb959fd",
        "timestamp": "2014-11-21T09:19:33+00:00"
    },
    {
        "event_id": 193,
        "feed_id": 64,
        "type": "md5",
        "category": "Payload delivery",
        "value": "5e2bfa364568a2cfeae09537e94e0778",
        "timestamp": "2014-11-21T09:19:33+00:00"
    },

We chose to parse:

  • Feed ID 1 → CIRCL OSINT Feed
  • Feed ID 64 → MalwareBazaar

All indicators were extracted with MISP format, not CSV.

6. Building Statistics Using PostgreSQL + Grafana

To enable data analytics on top of our extracted MISP indicators, we first installed and configured:

We created the PostgreSQL database and configured the credentials:

  • DB Name: misp_indicators
  • User: misp_user
  • Password: MispPass123!
┌──(kali㉿kali)-[~/misp-docker]
└─$ sudo -u postgres psql -d misp_indicators
[sudo] password for kali: 
psql (18.1 (Debian 18.1-1), server 17.5 (Debian 17.5-1))
Type "help" for help.

misp_indicators=#

We used the MISP_ToPostgres.py Python script that reads all parsed indicators from selected_feeds_indicators.json, normalizes this data, insert indicators into the PostgreSQL indicators table, and prepare the dataset so Grafana can generate dashboards.

Script Execution:

┌──(venv)─(kali㉿kali)-[~/misp-docker]
└─$ python MISP_ToPostgres.py
[+] Connected to PostgreSQL
[+] Inserted 236612 indicators into PostgreSQL

This script acts as the bridge between MISP → JSON → PostgreSQL → Grafana.

Grafana dashboards were then created to visualize:

  • Number of indicators by type
  • Feed distribution
  • Time-based activity

Grafana's datasource configuration and Verification (grafana-postgresql-datasource):

Screenshot 28 Screenshot 29 Screenshot 30

To verify the correctness of the ingested data and prepare Grafana dashboards, we executed SQL queries directly on the PostgreSQL database.

Example: View all Payload delivery indicators coming from feed ID 64 (MalwareBazaar)

SELECT *
FROM indicators
WHERE feed_id = 64
  AND category = 'Payload delivery'
ORDER BY timestamp DESC;

Results:

  • Data Table:

Screenshot 31

  • Stats:

Screenshot 32

7. Email Alert Rule for New or Updated Events

We began by checking the status of the MISP email workers inside the MISP container to ensure they were running correctly and ready to process outgoing messages, through Administration → Server Settings & Maintenance

Screenshot 33

We executed a bash shell through:

docker exec -it misp-docker-misp-core-1 bash

Then, inside /var/www/MISP/app/Config, we modified:

  • config.php

    Screenshot 34

  • email.php

We configured:

  • SMTP server
  • Gmail application-specific password
  • Sender address
  • TLS settings

After configuration, MISP successfully sent alert emails for new and updated events.

Sample of emails sent by the sender:

Screenshot 35

Sample of emails received by the receiver:

Screenshot 36

Sample of a single email:

Screenshot 37

Conclusion

All tasks and additional requirements have been successfully completed:

✔ Docker deployment

✔ APT group collection & enrichment

✔ Feed activation & synchronization

✔ Custom tagging system

✔ Python feed unification

✔ PostgreSQL + Grafana visualization

✔ Email alerting rule

This project demonstrates:

  • Practical MISP deployment
  • Threat intelligence ingestion
  • Automation using Python
  • Data warehousing and visualization
  • Alerting and operational integration

About

A complete MISP threat-intel deployment project using Docker on Kali Linux, featuring APT18 event creation, feed activation, custom tagging, Python-based feed parsing, PostgreSQL + Grafana statistics, and fully working email alerting.

Topics

python linux docker postgres docker-compose docker-container grafana postgresql python3 misp grafana-dashboard cti mitre misp-galaxy misp-objects misp-api threat-intelligence grafana-datasource mitre-attack misp-docker

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Contributors

Languages