Skip to content

Commit 80db743

Browse files
bryopsidabryopsida
committed
Squash
0 parents  commit 80db743

17 files changed

Lines changed: 6442258 additions & 0 deletions

‎.github/workflows/pr.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
name: Show proposed changes
2+
on:
3+
pull_request:
4+
jobs:
5+
update:
6+
permissions:
7+
contents: write
8+
runs-on:
9+
- ubuntu-latest
10+
steps:
11+
- uses: actions/checkout@v4
12+
- uses: actions/setup-node@v4
13+
with:
14+
cache: 'npm'
15+
node-version: '22'
16+
- run: npm ci
17+
- run: npm run update
18+
- run: |
19+
# configure user
20+
git config --global user.name "${{ github.actor }}"
21+
git config --global user.email "${{ github.actor }}@users.noreply.github.com"
22+
23+
# stage any file changes to be committed
24+
git add .
25+
26+
git diff --cached

‎.github/workflows/squash.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
name: Squash history
2+
on:
3+
schedule:
4+
- cron: "0 0/6 * * *"
5+
workflow_dispatch:
6+
jobs:
7+
update:
8+
permissions:
9+
contents: write
10+
runs-on:
11+
- ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
- run: |
15+
git config --global user.name "${{ github.actor }}"
16+
git config --global user.email "${{ github.actor }}@users.noreply.github.com"
17+
git reset $(git commit-tree HEAD^{tree} -m "Squash")
18+
git push --force

‎.github/workflows/update.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
name: Update block lists
2+
on:
3+
schedule:
4+
- cron: "15 * * * *"
5+
workflow_dispatch:
6+
jobs:
7+
update:
8+
permissions:
9+
contents: write
10+
runs-on:
11+
- ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
- uses: actions/setup-node@v4
15+
with:
16+
cache: 'npm'
17+
node-version: '22'
18+
- run: npm ci
19+
- run: npm run update
20+
- run: |
21+
# configure user
22+
git config --global user.name "${{ github.actor }}"
23+
git config --global user.email "${{ github.actor }}@users.noreply.github.com"
24+
25+
# stage any file changes to be committed
26+
git add .
27+
28+
# make commit with staged changes
29+
git commit -m 'update list'
30+
31+
# push the commit back up to source GitHub repository
32+
git push

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
node_modules

‎README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# What is this?
2+
3+
This repo aggregates a IP and DNS block lists and updates them on a schedule
4+
5+
The script is run via this [npm script](https://github.com/bryopsida/blocklist/blob/ab6130c3352883dedfcc998386c5fffdedb34c83/package.json#L7)
6+
and the workflow schedule is set [here](https://raw.githubusercontent.com/bryopsida/blocklist/main/.github/workflows/update.yml).
7+
8+
The repository history is perodicially squashed as well to keep the repo size from getting out of hand.
9+
10+
## How do I use this?
11+
12+
You can use the dns block list with something like adguard, pihole, or a unbound dns block list.
13+
14+
Due to file size limits on github the DNS block list is fragmented into 4 pieces all of which need to be added to your block list to get the complete list.
15+
16+
There are also ip block lists [ipv4](https://raw.githubusercontent.com/bryopsida/blocklist/main/ipv4.txt), [ipv6](https://raw.githubusercontent.com/bryopsida/blocklist/main/ipv6.txt).
17+
These can be fetched and loaded into a firewall ruleset as an alias to deny egress/ingress to any IPs on the list.
18+
19+
20+
## Can I add items to the lists?
21+
22+
If you wish to add items to the lists you can submit a PR, the things being blocked must either be a constant list with an explanation for why it should be blocked, or by an addition to the scripts to pull a block list from a reputable source.

‎buildDNSBlocklist.mjs‎

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
#!/usr/bin/env node
2+
import { createWriteStream } from 'node:fs'
3+
import { URL } from 'node:url'
4+
import axios from 'axios'
5+
import { extract } from 'tar-stream'
6+
import { pipeline, Readable } from 'node:stream'
7+
import gunzip from "gunzip-maybe"
8+
import { promisify } from 'node:util'
9+
10+
const pipe = promisify(pipeline)
11+
12+
process.on('uncaughtException', (err) => {
13+
console.error(err)
14+
process.exit(1)
15+
})
16+
17+
process.on('unhandledRejection', (err) => {
18+
console.error(err)
19+
process.exit(2)
20+
})
21+
22+
function isValidUrl (s) {
23+
try {
24+
new URL(`https://${s}`)
25+
return true
26+
} catch (err) {
27+
return false
28+
}
29+
};
30+
31+
const urls = [
32+
'https://big.oisd.nl/dnsmasq',
33+
'https://raw.githubusercontent.com/gieljnssns/Social-media-Blocklists/master/pihole-youtube.txt',
34+
'https://raw.githubusercontent.com/gieljnssns/Social-media-Blocklists/master/pihole-youtube.txt',
35+
'https://raw.githubusercontent.com/gieljnssns/Social-media-Blocklists/master/adguard-facebook.txt',
36+
'https://raw.githubusercontent.com/gieljnssns/Social-media-Blocklists/master/adguard-tiktok.txt',
37+
'https://raw.githubusercontent.com/gieljnssns/Social-media-Blocklists/master/adguard-reddit.txt',
38+
'https://raw.githubusercontent.com/gieljnssns/Social-media-Blocklists/master/adguard-twitch.txt',
39+
'https://raw.githubusercontent.com/gieljnssns/Social-media-Blocklists/master/adguard-snapchat.txt',
40+
'https://raw.githubusercontent.com/gieljnssns/Social-media-Blocklists/master/adguard-whatsapp.txt',
41+
'https://raw.githubusercontent.com/gieljnssns/Social-media-Blocklists/master/adguard-instagram.txt',
42+
'https://raw.githubusercontent.com/gieljnssns/Social-media-Blocklists/master/adguard-twitter.txt',
43+
'https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts',
44+
'https://adaway.org/hosts.txt',
45+
'https://v.firebog.net/hosts/AdguardDNS.txt',
46+
'https://v.firebog.net/hosts/Easyprivacy.txt',
47+
'https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt',
48+
'https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt',
49+
'https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt',
50+
'https://blocklistproject.github.io/Lists/alt-version/ads-nl.txt',
51+
'https://blocklistproject.github.io/Lists/alt-version/abuse-nl.txt',
52+
'https://blocklistproject.github.io/Lists/alt-version/malware-nl.txt',
53+
'https://blocklistproject.github.io/Lists/alt-version/tracking-nl.txt',
54+
'https://blocklistproject.github.io/Lists/alt-version/phishing-nl.txt',
55+
'https://blocklistproject.github.io/Lists/alt-version/scam-nl.txt',
56+
'https://blocklistproject.github.io/Lists/alt-version/ransomware-nl.txt',
57+
'https://blocklistproject.github.io/Lists/alt-version/ransomware-nl.txt',
58+
'https://raw.githubusercontent.com/stamparm/aux/master/maltrail-malware-domains.txt',
59+
'https://threatfox.abuse.ch/downloads/hostfile/',
60+
'https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts',
61+
'https://raw.githubusercontent.com/Sekhan/TheGreatWall/master/TheGreatWall.txt',
62+
'https://raw.githubusercontent.com/dibdot/DoH-IP-blocklists/master/doh-domains.txt',
63+
'https://dsi.ut-capitole.fr/blacklists/download/ads.tar.gz',
64+
'https://dsi.ut-capitole.fr/blacklists/download/aggressive.tar.gz',
65+
'https://dsi.ut-capitole.fr/blacklists/download/audio-video.tar.gz',
66+
'https://dsi.ut-capitole.fr/blacklists/download/bitcoin.tar.gz',
67+
'https://dsi.ut-capitole.fr/blacklists/download/chat.tar.gz',
68+
'https://dsi.ut-capitole.fr/blacklists/download/astrology.tar.gz',
69+
'https://dsi.ut-capitole.fr/blacklists/download/blog.tar.gz',
70+
'https://dsi.ut-capitole.fr/blacklists/download/cryptojacking.tar.gz',
71+
'https://dsi.ut-capitole.fr/blacklists/download/dangerous_material.tar.gz',
72+
'https://dsi.ut-capitole.fr/blacklists/download/dating.tar.gz',
73+
'https://dsi.ut-capitole.fr/blacklists/download/ddos.tar.gz',
74+
'https://dsi.ut-capitole.fr/blacklists/download/dialer.tar.gz',
75+
'https://dsi.ut-capitole.fr/blacklists/download/doh.tar.gz',
76+
'https://dsi.ut-capitole.fr/blacklists/download/download.tar.gz',
77+
'https://dsi.ut-capitole.fr/blacklists/download/drogue.tar.gz',
78+
'https://dsi.ut-capitole.fr/blacklists/download/filehosting.tar.gz',
79+
'https://dsi.ut-capitole.fr/blacklists/download/forums.tar.gz',
80+
'https://dsi.ut-capitole.fr/blacklists/download/gambling.tar.gz',
81+
'https://dsi.ut-capitole.fr/blacklists/download/games.tar.gz',
82+
'https://dsi.ut-capitole.fr/blacklists/download/hacking.tar.gz',
83+
'https://dsi.ut-capitole.fr/blacklists/download/lingerie.tar.gz',
84+
'https://dsi.ut-capitole.fr/blacklists/download/malware.tar.gz',
85+
'https://dsi.ut-capitole.fr/blacklists/download/manga.tar.gz',
86+
'https://dsi.ut-capitole.fr/blacklists/download/mixed_adult.tar.gz',
87+
'https://dsi.ut-capitole.fr/blacklists/download/phishing.tar.gz',
88+
'https://dsi.ut-capitole.fr/blacklists/download/proxy.tar.gz',
89+
'https://dsi.ut-capitole.fr/blacklists/download/publicite.tar.gz',
90+
'https://dsi.ut-capitole.fr/blacklists/download/radio.tar.gz',
91+
'https://dsi.ut-capitole.fr/blacklists/download/redirector.tar.gz',
92+
'https://dsi.ut-capitole.fr/blacklists/download/remote-control.tar.gz',
93+
'https://dsi.ut-capitole.fr/blacklists/download/shortener.tar.gz',
94+
'https://dsi.ut-capitole.fr/blacklists/download/sexual_education.tar.gz',
95+
'https://dsi.ut-capitole.fr/blacklists/download/social_networks.tar.gz',
96+
'https://dsi.ut-capitole.fr/blacklists/download/stalkerware.tar.gz',
97+
'https://dsi.ut-capitole.fr/blacklists/download/strict_redirector.tar.gz',
98+
'https://dsi.ut-capitole.fr/blacklists/download/tricheur.tar.gz',
99+
'https://dsi.ut-capitole.fr/blacklists/download/vpn.tar.gz',
100+
'https://dsi.ut-capitole.fr/blacklists/download/violence.tar.gz',
101+
'https://dsi.ut-capitole.fr/blacklists/download/warez.tar.gz',
102+
'https://dsi.ut-capitole.fr/blacklists/download/webmail.tar.gz',
103+
'https://dsi.ut-capitole.fr/blacklists/download/adult.tar.gz'
104+
]
105+
106+
107+
function extractUT1DomainFileFromGzipStream(resp) {
108+
// will be a domains entry
109+
// if we find it replace resp.data with the plain text version
110+
// if we don't find it, leave as is
111+
return new Promise((resolve) => {
112+
const extractStream = extract()
113+
const tarGzStream = typeof resp.data === 'string' ? Readable.from(Buffer.from(resp.data)) : Readable.from(resp.data)
114+
let foundDomains = false
115+
extractStream.on('entry', (header, stream, next) => {
116+
if(header.name.endsWith('domains')) {
117+
118+
const chunks = []
119+
stream.on('data', (chunk) => chunks.push(chunk))
120+
121+
stream.on('end', () => {
122+
resp.data = Buffer.concat(chunks).toString('utf8')
123+
foundDomains = true
124+
next()
125+
})
126+
} else {
127+
stream.resume()
128+
next()
129+
}
130+
})
131+
extractStream.on('end', () => {
132+
next()
133+
})
134+
extractStream.on('error', (err) => {
135+
console.error(err)
136+
})
137+
extractStream.on('finish', () => {
138+
if(!foundDomains) {
139+
resp.data = ''
140+
console.warn(`Failed to extract domain list form ${resp.headers}`)
141+
}
142+
resolve(resp)
143+
})
144+
tarGzStream.pipe(gunzip()).pipe(extractStream)
145+
})
146+
}
147+
148+
const requests = urls.map((url) => axios.get(url, {
149+
decompress: !url.endsWith('tar.gz'),
150+
responseType: url.endsWith('tar.gz') ? 'arraybuffer' : 'text'
151+
}).then((resp) => {
152+
if (resp.headers['content-type'] === 'application/x-gzip' || typeof resp.data !== 'string') {
153+
return extractUT1DomainFileFromGzipStream(resp)
154+
} else {
155+
return resp
156+
}
157+
}).catch((err) => {
158+
console.error(err)
159+
process.exit(3)
160+
}))
161+
162+
const responses = await Promise.all(requests).catch((err) => {
163+
console.error(err)
164+
process.exit(1)
165+
})
166+
167+
const lines = responses.map((resp) => resp.data.split('\n')).flat()
168+
const blocked = new Set()
169+
170+
const excludedLines = [
171+
'0.0.0.0 0.0.0.0',
172+
'127.0.0.1 localhost',
173+
'127.0.0.1 localhost.localdomain',
174+
'127.0.0.1 local',
175+
'255.255.255.255 broadcasthost',
176+
'::1 localhost',
177+
'::1 ip6-localhost',
178+
'::1 ip6-loopback',
179+
'fe80::1%lo0 localhost',
180+
'ff00::0 ip6-localnet',
181+
'ff00::0 ip6-mcastprefix',
182+
'ff02::1 ip6-allnodes',
183+
'ff02::2 ip6-allrouters',
184+
'ff02::3 ip6-allhosts',
185+
'=',
186+
'::1'
187+
]
188+
189+
const cantStartWith = [
190+
'#',
191+
'!#',
192+
'!',
193+
'if-a-large-hosts-file-contains-this-entry-then-it'
194+
]
195+
196+
const stripPatterns = ['server=/', '/', '||', '^', '0.0.0.0', '127.0.0.1']
197+
198+
for (const line of lines) {
199+
if (!cantStartWith.some((blockedPrefix) => line.startsWith(blockedPrefix)) && !excludedLines.some((ele) => ele === line)) {
200+
blocked.add(line)
201+
}
202+
}
203+
let blockList = ''
204+
let blockedNormalizedHosts = []
205+
for (let key of blocked) {
206+
for (const removePattern of stripPatterns) {
207+
key = key.replace(removePattern, '')
208+
}
209+
210+
key = key.trim()
211+
key = key.split(' ')[0]
212+
if (isValidUrl(key)) {
213+
blockedNormalizedHosts.push(key)
214+
} else {
215+
console.warn(`${key} is not a valid url! Dropping from list!`)
216+
}
217+
}
218+
blockedNormalizedHosts = blockedNormalizedHosts.sort()
219+
const quarters = blockedNormalizedHosts.length/4
220+
221+
const quarter1Start = 0
222+
const quarter1End = quarters * 1
223+
const quarter2Start = quarters * 1
224+
const quarter2End = quarters * 2
225+
const quarter3Start = quarters * 2
226+
const quarter3End = quarters * 3
227+
const quarter4Start = quarters * 3
228+
229+
const lists = [blockedNormalizedHosts.slice(quarter1Start, quarter1End), blockedNormalizedHosts.slice(quarter2Start, quarter2End), blockedNormalizedHosts.slice(quarter3Start, quarter3End), blockedNormalizedHosts.slice(quarter4Start)]
230+
for(const i in lists) {
231+
const fileStream = createWriteStream(`dns${i}.txt`)
232+
let blockList = ''
233+
for (const blockedHost of lists[i]) {
234+
blockList += blockedHost + '\n'
235+
}
236+
const blockStream = Readable.from(Buffer.from(blockList))
237+
await pipe(blockStream, fileStream)
238+
}

0 commit comments

Comments
 (0)