Skip to content

Commit 0370fef

Browse files
SecAI-Hubclaude
SecAI-Hub
and
claude
committed
Add rag-data-firewall service with 8 security/privacy improvements
Implements all PDF-recommended hardening: 1. Auth on all non-health endpoints via buildMux pattern 2. Deny-first policy precedence (deny rules always evaluated before allow) 3. Hardened http.Server with read/write/idle timeouts (30s/60s/120s) 4. Tightened retrieval scoping (require explicit chunk/doc IDs for non-system) 5. Retention controls (max documents, max chunks, TTL-based expiration) 6. Enhanced scanner (encoded injection, smuggling, data-exfil, base64, risk reasons) 7. Document deletion with JSONL rewrite and fsync durability 8. Tamper-evident audit logging with SHA-256 hash chain 48 tests passing (37 top-level + 11 subtests). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent cb2be48 commit 0370fef

13 files changed

Lines changed: 3779 additions & 0 deletions

File tree

‎services/rag-data-firewall/Dockerfile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM golang:1.23-alpine@sha256:ef18ee7117463ac1055f5a370ed18b8750f01589f13ea0b48a1c3c025e611b02 AS build
2+
WORKDIR /src
3+
COPY go.mod go.sum* ./
4+
RUN go mod download
5+
COPY . .
6+
RUN CGO_ENABLED=0 go build -o /rag-data-firewall .
7+
8+
FROM alpine:3.20@sha256:1e42bbe2508154c9126d48c2b8a75420c3544343bf86fd041fb7527e017a4b4a
9+
RUN apk add --no-cache ca-certificates
10+
COPY --from=build /rag-data-firewall /usr/local/bin/rag-data-firewall
11+
USER 65534:65534
12+
EXPOSE 8500
13+
ENTRYPOINT ["rag-data-firewall"]

‎services/rag-data-firewall/LICENSE‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
Apache License
2+
Version 2.0, January 2004
3+
http://www.apache.org/licenses/
4+
5+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6+
7+
1. Definitions.
8+
9+
"License" shall mean the terms and conditions for use, reproduction,
10+
and distribution as defined by Sections 1 through 9 of this document.
11+
12+
"Licensor" shall mean the copyright owner or entity authorized by
13+
the copyright owner that is granting the License.
14+
15+
"Legal Entity" shall mean the union of the acting entity and all
16+
other entities that control, are controlled by, or are under common
17+
control with that entity. For the purposes of this definition,
18+
"control" means (i) the power, direct or indirect, to cause the
19+
direction or management of such entity, whether by contract or
20+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
21+
outstanding shares, or (iii) beneficial ownership of such entity.
22+
23+
"You" (or "Your") shall mean an individual or Legal Entity
24+
exercising permissions granted by this License.
25+
26+
"Source" form shall mean the preferred form for making modifications,
27+
including but not limited to software source code, documentation
28+
source, and configuration files.
29+
30+
"Object" form shall mean any form resulting from mechanical
31+
transformation or translation of a Source form, including but
32+
not limited to compiled object code, generated documentation,
33+
and conversions to other media types.
34+
35+
"Work" shall mean the work of authorship, whether in Source or
36+
Object form, made available under the License, as indicated by a
37+
copyright notice that is included in or attached to the work.
38+
39+
"Derivative Works" shall mean any work, whether in Source or Object
40+
form, that is based on (or derived from) the Work and for which the
41+
editorial revisions, annotations, elaborations, or other modifications
42+
represent, as a whole, an original work of authorship. For the purposes
43+
of this License, Derivative Works shall not include works that remain
44+
separable from, or merely link (or bind by name) to the interfaces of,
45+
the Work and Derivative Works thereof.
46+
47+
"Contribution" shall mean any work of authorship, including
48+
the original version of the Work and any modifications or additions
49+
to that Work or Derivative Works thereof, that is intentionally
50+
submitted to the Licensor for inclusion in the Work by the copyright owner
51+
or by an individual or Legal Entity authorized to submit on behalf of
52+
the copyright owner. For the purposes of this definition, "submitted"
53+
means any form of electronic, verbal, or written communication sent
54+
to the Licensor or its representatives, including but not limited to
55+
communication on electronic mailing lists, source code control systems,
56+
and issue tracking systems that are managed by, or on behalf of, the
57+
Licensor for the purpose of discussing and improving the Work, but
58+
excluding communication that is conspicuously marked or otherwise
59+
designated in writing by the copyright owner as "Not a Contribution."
60+
61+
"Contributor" shall mean Licensor and any individual or Legal Entity
62+
on behalf of whom a Contribution has been received by the Licensor and
63+
subsequently incorporated within the Work.
64+
65+
2. Grant of Copyright License. Subject to the terms and conditions of
66+
this License, each Contributor hereby grants to You a perpetual,
67+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
68+
copyright license to reproduce, prepare Derivative Works of,
69+
publicly display, publicly perform, sublicense, and distribute the
70+
Work and such Derivative Works in Source or Object form.
71+
72+
3. Grant of Patent License. Subject to the terms and conditions of
73+
this License, each Contributor hereby grants to You a perpetual,
74+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
75+
(except as stated in this section) patent license to make, have made,
76+
use, offer to sell, sell, import, and otherwise transfer the Work,
77+
where such license applies only to those patent claims licensable
78+
by such Contributor that are necessarily infringed by their
79+
Contribution(s) alone or by combination of their Contribution(s)
80+
with the Work to which such Contribution(s) was submitted. If You
81+
institute patent litigation against any entity (including a
82+
cross-claim or counterclaim in a lawsuit) alleging that the Work
83+
or a Contribution incorporated within the Work constitutes direct
84+
or contributory patent infringement, then any patent licenses
85+
granted to You under this License for that Work shall terminate
86+
as of the date such litigation is filed.
87+
88+
4. Redistribution. You may reproduce and distribute copies of the
89+
Work or Derivative Works thereof in any medium, with or without
90+
modifications, and in Source or Object form, provided that You
91+
meet the following conditions:
92+
93+
(a) You must give any other recipients of the Work or
94+
Derivative Works a copy of this License; and
95+
96+
(b) You must cause any modified files to carry prominent notices
97+
stating that You changed the files; and
98+
99+
(c) You must retain, in the Source form of any Derivative Works
100+
that You distribute, all copyright, patent, trademark, and
101+
attribution notices from the Source form of the Work,
102+
excluding those notices that do not pertain to any part of
103+
the Derivative Works; and
104+
105+
(d) If the Work includes a "NOTICE" text file as part of its
106+
distribution, then any Derivative Works that You distribute must
107+
include a readable copy of the attribution notices contained
108+
within such NOTICE file, excluding any notices that do not
109+
pertain to any part of the Derivative Works, in at least one
110+
of the following places: within a NOTICE text file distributed
111+
as part of the Derivative Works; within the Source form or
112+
documentation, if provided along with the Derivative Works; or,
113+
within a display generated by the Derivative Works, if and
114+
wherever such third-party notices normally appear. The contents
115+
of the NOTICE file are for informational purposes only and
116+
do not modify the License. You may add Your own attribution
117+
notices within Derivative Works that You distribute, alongside
118+
or as an addendum to the NOTICE text from the Work, provided
119+
that such additional attribution notices cannot be construed
120+
as modifying the License.
121+
122+
You may add Your own copyright statement to Your modifications and
123+
may provide additional or different license terms and conditions
124+
for use, reproduction, or distribution of Your modifications, or
125+
for any such Derivative Works as a whole, provided Your use,
126+
reproduction, and distribution of the Work otherwise complies with
127+
the conditions stated in this License.
128+
129+
5. Submission of Contributions. Unless You explicitly state otherwise,
130+
any Contribution intentionally submitted for inclusion in the Work
131+
by You to the Licensor shall be under the terms and conditions of
132+
this License, without any additional terms or conditions.
133+
Notwithstanding the above, nothing herein shall supersede or modify
134+
the terms of any separate license agreement you may have executed
135+
with Licensor regarding such Contributions.
136+
137+
6. Trademarks. This License does not grant permission to use the trade
138+
names, trademarks, service marks, or product names of the Licensor,
139+
except as required for reasonable and customary use in describing the
140+
origin of the Work and reproducing the content of the NOTICE file.
141+
142+
7. Disclaimer of Warranty. Unless required by applicable law or
143+
agreed to in writing, Licensor provides the Work (and each
144+
Contributor provides its Contributions) on an "AS IS" BASIS,
145+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
146+
implied, including, without limitation, any warranties or conditions
147+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
148+
PARTICULAR PURPOSE. You are solely responsible for determining the
149+
appropriateness of using or redistributing the Work and assume any
150+
risks associated with Your exercise of permissions under this License.
151+
152+
8. Limitation of Liability. In no event and under no legal theory,
153+
whether in tort (including negligence), contract, or otherwise,
154+
unless required by applicable law (such as deliberate and grossly
155+
negligent acts) or agreed to in writing, shall any Contributor be
156+
liable to You for damages, including any direct, indirect, special,
157+
incidental, or consequential damages of any character arising as a
158+
result of this License or out of the use or inability to use the
159+
Work (including but not limited to damages for loss of goodwill,
160+
work stoppage, computer failure or malfunction, or any and all
161+
other commercial damages or losses), even if such Contributor
162+
has been advised of the possibility of such damages.
163+
164+
9. Accepting Warranty or Additional Liability. While redistributing
165+
the Work or Derivative Works thereof, You may choose to offer,
166+
and charge a fee for, acceptance of support, warranty, indemnity,
167+
or other liability obligations and/or rights consistent with this
168+
License. However, in accepting such obligations, You may act only
169+
on Your own behalf and on Your sole responsibility, not on behalf
170+
of any other Contributor, and only if You agree to indemnify,
171+
defend, and hold each Contributor harmless for any liability
172+
incurred by, or claims asserted against, such Contributor by reason
173+
of your accepting any such warranty or additional liability.
174+
175+
END OF TERMS AND CONDITIONS
176+
177+
Copyright 2025 SecAI-Hub
178+
179+
Licensed under the Apache License, Version 2.0 (the "License");
180+
you may not use this file except in compliance with the License.
181+
You may obtain a copy of the License at
182+
183+
http://www.apache.org/licenses/LICENSE-2.0
184+
185+
Unless required by applicable law or agreed to in writing, software
186+
distributed under the License is distributed on an "AS IS" BASIS,
187+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
188+
See the License for the specific language governing permissions and
189+
limitations under the License.

‎services/rag-data-firewall/SECURITY.md‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Security Design — rag-data-firewall
2+
3+
## Overview
4+
5+
The RAG Data Firewall is a policy-enforced gateway that controls which
6+
document chunks may be returned to AI models and tools during retrieval.
7+
It provides content scanning, PII redaction, and tamper-evident audit logging.
8+
9+
## Authentication
10+
11+
All non-health endpoints require a bearer token (SERVICE_TOKEN). The
12+
token is loaded from `/run/secure-ai/service-token` and compared using
13+
`crypto/subtle.ConstantTimeCompare` to prevent timing attacks.
14+
15+
## Policy Evaluation — Deny-First Precedence
16+
17+
The policy engine uses **deny-first** rule evaluation:
18+
19+
1. All deny rules are checked first, regardless of position in the config
20+
2. If any deny rule matches, the chunk is blocked immediately
21+
3. Non-deny rules (require-approval, redact, allow) are then evaluated in order
22+
4. If no rule matches, the default action applies (deny by default)
23+
24+
This prevents an early allow rule from accidentally overriding a
25+
security-critical deny rule.
26+
27+
## Content Scanning
28+
29+
The scanner detects:
30+
31+
- **Direct prompt injection**: 14 patterns (ignore instructions, role override, etc.)
32+
- **Encoded injection**: Unicode bidi overrides, zero-width hiding, HTML entity tags, hex escapes
33+
- **Instruction smuggling**: Markdown fence injection, HTML comment injection, invisible text, delimiter role switching
34+
- **Data exfiltration**: URL fetch, file protocol, markdown exfil, DNS exfil, send-data instructions
35+
- **Base64-encoded injection**: Decodes base64 strings and checks for embedded injection patterns
36+
- **PII detection**: SSN, email, credit card, credentials, bearer tokens, phone numbers
37+
- **Suspicious patterns**: eval, exec, import os, subprocess, script tags
38+
39+
Each finding produces a structured `RiskReason` with category, pattern,
40+
weight, and detail for explainability.
41+
42+
## Retrieval Scoping
43+
44+
Non-system callers must provide explicit `chunk_ids` or `document_ids`.
45+
Unscoped retrieval (AllChunks fallback) is only available to
46+
`requester_type=system`, preventing tools and users from accessing the
47+
entire corpus without specifying what they need.
48+
49+
## Data Protection
50+
51+
- **Retention limits**: Configurable max documents, max total chunks, and TTL
52+
- **Document deletion**: Documents and their chunks can be deleted via API
53+
- **PII redaction**: Content matching PII patterns is replaced with tagged placeholders
54+
- **fsync durability**: All JSONL writes are followed by fsync
55+
56+
## Tamper-Evident Audit Logging
57+
58+
Every action (ingest, retrieve, delete, reload) is logged to a JSONL
59+
audit file with a SHA-256 hash chain. Each entry includes:
60+
61+
- `hash`: SHA-256 of the entry's canonical fields
62+
- `prev_hash`: hash of the preceding entry
63+
64+
This chain allows detection of any inserted, deleted, or modified entries.
65+
66+
## HTTP Server Hardening
67+
68+
- `http.Server` with configurable read/write/idle timeouts (defaults: 30s/60s/120s)
69+
- `MaxBytesReader` on all request bodies (10 MiB limit)
70+
- Rate limiting (configurable requests per minute)
71+
- Consistent JSON error responses on all endpoints
72+
73+
## Threat Model
74+
75+
| Threat | Mitigation |
76+
|--------|-----------|
77+
| Prompt injection in documents | Content scanning with 14+ injection patterns plus encoded/smuggling detection |
78+
| PII leakage via RAG retrieval | PII detection + redaction action in policy rules |
79+
| Unauthorized access to chunks | Bearer token auth on all endpoints + deny-by-default policy |
80+
| Unscoped data access | Retrieval scoping (require explicit chunk/doc IDs for non-system callers) |
81+
| Policy misconfiguration | Deny-first precedence ensures deny rules always take priority |
82+
| Audit log tampering | Hash chain on audit entries |
83+
| Data accumulation | Retention limits + TTL-based expiration + document deletion |

0 commit comments

Comments
 (0)