From a6189c1e92f66852173c835e42e4484cd1895a75 Mon Sep 17 00:00:00 2001 From: JarbasAl <33701864+JarbasAl@users.noreply.github.com> Date: Wed, 15 Apr 2026 03:28:41 +0000 Subject: [PATCH 1/4] Increment Version to 0.2.1 --- hivemind_sqlite_database/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hivemind_sqlite_database/version.py b/hivemind_sqlite_database/version.py index 58d7648..0343953 100644 --- a/hivemind_sqlite_database/version.py +++ b/hivemind_sqlite_database/version.py @@ -2,7 +2,7 @@ VERSION_MAJOR = 0 VERSION_MINOR = 2 VERSION_BUILD = 1 -VERSION_ALPHA = 4 +VERSION_ALPHA = 0 # END_VERSION_BLOCK __version__ = f"{VERSION_MAJOR}.{VERSION_MINOR}.{VERSION_BUILD}" + (f"a{VERSION_ALPHA}" if VERSION_ALPHA else "") From ed7fb8aa18d2adcef0cfb846bd6493e78d612043 Mon Sep 17 00:00:00 2001 From: JarbasAI <33701864+JarbasAl@users.noreply.github.com> Date: Wed, 15 Apr 2026 11:20:46 +0100 Subject: [PATCH 2/4] feat: add SQLCipher encryption support (password kwarg) (#24) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * ci: replace hand-rolled release jobs with shared template inputs The old publish_stable.yml and release_workflow.yml extracted the version by importing hivemind_sqlite_database, which triggered __init__.py and pulled in ovos_utils — not present in the bare workflow environment. Replace all duplicated publish_pypi, sync_dev, propose_release, and notify jobs with the equivalent inputs on the shared publish-stable.yml@dev and publish-alpha.yml@dev reusable workflows, which use get_version.py to read version.py directly without any package import. AI-Generated Change: - Model: claude-sonnet-4-6 - Intent: fix version extraction failure in CI release workflows - Impact: replaced publish_stable.yml, release_workflow.yml - Verified via: reviewed diff against shared template inputs Co-Authored-By: Claude Sonnet 4.6 * feat: add optional SQLCipher encryption to SQLiteDB via password kwarg AI-Generated Change: - Model: Claude Sonnet 4.6 - Intent: allow operators to encrypt the client database at rest using SQLCipher AES-256 - Impact: SQLiteDB gains a password field; when set, sqlcipher3 is used as the sqlite3 backend and PRAGMA key is issued immediately after connect; when None (default) the existing stdlib sqlite3 path is unchanged; ImportError with install hint raised if sqlcipher3 is missing but a password is supplied - Verified via: python -m pytest tests/test_sqlitedb.py -q -p no:ovoscope (31 passed) * chore: add cipher optional-dependency extra for sqlcipher3 AI-Generated Change: - Model: Claude Sonnet 4.6 - Intent: let users opt into SQLCipher encryption via pip install hivemind-sqlite-database[cipher] - Impact: pyproject.toml gains [project.optional-dependencies] cipher = ["sqlcipher3"] - Verified via: python -m pytest tests/test_sqlitedb.py -q -p no:ovoscope (31 passed) * test: add SQLCipher encrypted-path tests (skipped without sqlcipher3) AI-Generated Change: - Model: Claude Sonnet 4.6 - Intent: verify encrypted path works end-to-end and existing CI remains green without sqlcipher3 - Impact: TestSQLiteDBEncrypted (3 tests, skipped when sqlcipher3 absent) and TestSQLiteDBMissingCipher (ImportError when sqlcipher3 not installed) added - Verified via: python -m pytest tests/test_sqlitedb.py -q -p no:ovoscope (35 passed with sqlcipher3; 32 passed + 3 skipped without) * docs: document SQLCipher encryption in README and add CI workflow AI-Generated Change: - Model: Claude Sonnet 4.6 - Intent: document the password kwarg, install steps, and data-loss warning; add CI that tests both plain and encrypted paths - Impact: README.md rewritten with usage examples, apt/pip install instructions, and data-loss warning; .github/workflows/tests.yml added with two jobs (plain sqlite3 and sqlcipher3) - Verified via: python -m pytest tests/test_sqlitedb.py -q -p no:ovoscope (35 passed) * fix(ci): remove libsqlcipher0 from apt install — not present on Ubuntu 24.04 Ubuntu Noble (24.04) replaced libsqlcipher0 with libsqlcipher1. Installing libsqlcipher-dev is sufficient as it declares a dependency on libsqlcipher1 and pulls it in automatically. AI-Generated Change: - Model: claude-sonnet-4-6 - Intent: fix SQLite + SQLCipher CI job failing with exit code 100 - Impact: `apt-get install -y libsqlcipher0` → removed; libsqlcipher-dev alone satisfies the requirement - Verified via: gh run log inspection showing "Unable to locate package libsqlcipher0" Co-Authored-By: Claude Sonnet 4.6 * fix: address CodeRabbit Major issues in encryption implementation AI-Generated Change: - Model: claude-sonnet-4-6 - Intent: address CodeRabbit Major review comments on PR #24 Changes: - __init__.py: gate on `password is not None` instead of truthiness so empty-string passwords don't silently downgrade to plaintext; raise ValueError for empty strings - __init__.py: escape single-quotes in passphrase before building PRAGMA key string (SQLite double-quote escaping) to avoid broken opens and eliminate an injection surface - tests/test_sqlitedb.py: rewrite _make_encrypted_db() to drive through __post_init__() (patching xdg_data_home) instead of reimplementing the SQLCipher setup, so regressions in the production path are caught - tests/test_sqlitedb.py: remove unused `importlib` import (Ruff F401 / CI lint failure) Verified via: python -m pytest tests/test_sqlitedb.py -q (35 passed) python -m ruff check (all checks passed) Co-Authored-By: Claude Sonnet 4.6 * fix(ci): address CodeRabbit workflow issues AI-Generated Change: - Model: claude-sonnet-4-6 - Intent: address CodeRabbit Critical/Major review comments on CI workflows Changes: - tests.yml: add `sudo apt-get update` before apt install to prevent intermittent "Unable to locate package" failures on stale runner indexes - tests.yml: remove broken `|| pip install -e . && pip install sqlcipher3` fallback that masked a misconfigured cipher extra; now fails loudly if `.[cipher]` is broken (cipher extra is defined in pyproject.toml so this is the correct install path) - release_workflow.yml: add `if: github.event.pull_request.merged == true` guard to publish_alpha job so closing a PR without merging does not trigger PyPI publish, release proposal, or Matrix notification Co-Authored-By: Claude Sonnet 4.6 --------- Co-authored-by: Claude Sonnet 4.6 --- .github/workflows/tests.yml | 36 ++++++++++++ README.md | 77 +++++++++++++++++++++++- hivemind_sqlite_database/__init__.py | 32 +++++++++- pyproject.toml | 3 + tests/test_sqlitedb.py | 87 ++++++++++++++++++++++++++++ 5 files changed, 231 insertions(+), 4 deletions(-) create mode 100644 .github/workflows/tests.yml diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml new file mode 100644 index 0000000..3a7e54b --- /dev/null +++ b/.github/workflows/tests.yml @@ -0,0 +1,36 @@ +name: Tests + +on: + push: + branches: ["**"] + pull_request: + branches: ["**"] + +jobs: + test-plain: + name: "SQLite (no encryption)" + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - run: pip install -e ".[dev]" || pip install -e . + - run: pip install pytest + - run: pytest tests/test_sqlitedb.py -q -p no:ovoscope + + test-cipher: + name: "SQLite + SQLCipher" + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install sqlcipher system library + run: | + sudo apt-get update + sudo apt-get install -y libsqlcipher-dev + - run: pip install -e ".[cipher]" + - run: pip install pytest + - run: pytest tests/test_sqlitedb.py -q -p no:ovoscope diff --git a/README.md b/README.md index d8a0498..3a420b5 100644 --- a/README.md +++ b/README.md @@ -1 +1,76 @@ -# HiveMind SQLite Database \ No newline at end of file +# HiveMind SQLite Database + +SQLite database plugin for [hivemind-core](https://github.com/JarbasHiveMind/HiveMind-core). + +Implements the `AbstractDB` interface via `hivemind-plugin-manager` and stores HiveMind +client records (API keys, crypto keys, access-control lists) in a local SQLite file. + +## Installation + +```bash +pip install hivemind-sqlite-database +``` + +### With encryption support (SQLCipher) + +```bash +# 1. Install the SQLCipher system library +# Debian/Ubuntu: +sudo apt install libsqlcipher0 + +# 2. Install the Python binding via the optional extra +pip install "hivemind-sqlite-database[cipher]" +``` + +> The `sqlcipher3` wheel on PyPI ships its own libsqlcipher for x86_64 Linux, so the +> `apt` step may be optional on that platform. On ARM or Alpine you must build from +> source and will need the system library. + +## Usage + +### Plain (unencrypted) database — default + +```python +from hivemind_sqlite_database import SQLiteDB + +db = SQLiteDB() # stores data in XDG_DATA_HOME/hivemind-core/clients.db +``` + +### Encrypted database (SQLCipher / AES-256) + +```python +from hivemind_sqlite_database import SQLiteDB + +db = SQLiteDB(password="your-strong-passphrase") +``` + +Pass the same `password` every time you open the database. The encryption is +transparent — all existing methods (`add_item`, `search_by_value`, etc.) work +identically. + +> **Data-loss warning**: There is no password recovery. If you lose the passphrase +> the database is permanently unrecoverable. Back up your passphrase securely. + +### hivemind-core configuration + +```json +{ + "database": { + "module": "hivemind-sqlite-db-plugin", + "hivemind-sqlite-db-plugin": { + "name": "clients", + "subfolder": "hivemind-core", + "password": "your-strong-passphrase" + } + } +} +``` + +Leave `"password"` out (or set it to `null`) to use an unencrypted database. + +## Notes + +- An encrypted database cannot be opened by the plain `sqlite3` CLI or stdlib module. +- A plaintext database cannot be opened as encrypted. There is no automatic migration. +- The `password` field maps directly to SQLCipher's `PRAGMA key`. +- WAL journal mode is enabled for both encrypted and unencrypted databases. diff --git a/hivemind_sqlite_database/__init__.py b/hivemind_sqlite_database/__init__.py index 2700b90..c220fec 100644 --- a/hivemind_sqlite_database/__init__.py +++ b/hivemind_sqlite_database/__init__.py @@ -2,7 +2,7 @@ import os.path import sqlite3 import threading -from typing import List, Union, Iterable +from typing import List, Optional, Union, Iterable from ovos_utils.log import LOG from ovos_utils.xdg_utils import xdg_data_home @@ -24,17 +24,43 @@ class SQLiteDB(AbstractDB): """Database implementation using SQLite.""" name: str = "clients" subfolder: str = "hivemind-core" + password: Optional[str] = None def __post_init__(self): """ Initialize the SQLiteDB connection. + + When *password* is set the database is opened via ``sqlcipher3`` and + encrypted with AES-256 (SQLCipher). The system library + ``libsqlcipher0`` must be installed and ``sqlcipher3`` must be + available (``pip install hivemind-sqlite-database[cipher]``). + + When *password* is ``None`` (default) the standard ``sqlite3`` module + is used and the database file is unencrypted. """ db_path = os.path.join(xdg_data_home(), self.subfolder, self.name + ".db") LOG.debug(f"sqlite database path: {db_path}") os.makedirs(os.path.dirname(db_path), exist_ok=True) - self.conn = sqlite3.connect(db_path, check_same_thread=False) - self.conn.row_factory = sqlite3.Row + if self.password is not None: + if self.password == "": + raise ValueError("password must be non-empty when encryption is enabled") + try: + import sqlcipher3 as _sqlcipher + except ImportError: + raise ImportError( + "sqlcipher3 is required to open an encrypted SQLite database. " + "Install the system library (e.g. 'apt install libsqlcipher-dev') " + "then: pip install hivemind-sqlite-database[cipher]" + ) + self.conn = _sqlcipher.connect(db_path, check_same_thread=False) + self.conn.row_factory = _sqlcipher.Row + escaped_password = self.password.replace("'", "''") + self.conn.execute(f"PRAGMA key='{escaped_password}'") + else: + self.conn = sqlite3.connect(db_path, check_same_thread=False) + self.conn.row_factory = sqlite3.Row + self.conn.execute("PRAGMA journal_mode=WAL") self._write_lock = threading.Lock() self._initialize_database() diff --git a/pyproject.toml b/pyproject.toml index 1833c0a..f10c6b0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -16,6 +16,9 @@ dependencies = [ ] +[project.optional-dependencies] +cipher = ["sqlcipher3"] + [project.urls] Homepage = "https://github.com/JarbasHiveMind/hivemind-sqlite-database" diff --git a/tests/test_sqlitedb.py b/tests/test_sqlitedb.py index 1b414bd..acda6e3 100644 --- a/tests/test_sqlitedb.py +++ b/tests/test_sqlitedb.py @@ -302,5 +302,92 @@ def test_replace_item(self): self.assertEqual(db.search_by_value("api_key", "revoked")[0].client_id, 1) +try: + import sqlcipher3 as _sqlcipher3 # noqa: F401 + _SQLCIPHER_AVAILABLE = True +except ImportError: + _SQLCIPHER_AVAILABLE = False + + +@unittest.skipUnless(_SQLCIPHER_AVAILABLE, "sqlcipher3 not installed") +class TestSQLiteDBEncrypted(unittest.TestCase): + """Tests for the SQLCipher-encrypted path. Skipped when sqlcipher3 is absent.""" + + def _make_encrypted_db(self, path: str, password: str = "hunter2") -> SQLiteDB: + import unittest.mock as mock + db = SQLiteDB.__new__(SQLiteDB) + db.name = os.path.splitext(os.path.basename(path))[0] + db.subfolder = "" + db.password = password + with mock.patch( + "hivemind_sqlite_database.xdg_data_home", + return_value=os.path.dirname(path), + ): + db.__post_init__() + return db + + def test_encrypted_file_unreadable_by_stdlib_sqlite3(self): + """A file created with a password must be opaque to plain sqlite3.""" + with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as f: + path = f.name + os.unlink(path) + try: + db = self._make_encrypted_db(path, password="secret123") + db.add_item(make_client(1, "enc-key")) + db.commit() + # stdlib sqlite3 should not be able to read it + plain_conn = sqlite3.connect(path) + with self.assertRaises(sqlite3.DatabaseError): + plain_conn.execute("SELECT * FROM clients").fetchall() + plain_conn.close() + finally: + if os.path.exists(path): + os.unlink(path) + + def test_encrypted_round_trip(self): + """add_item then search_by_value works through the encryption layer.""" + with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as f: + path = f.name + os.unlink(path) + try: + db = self._make_encrypted_db(path, password="roundtrip") + db.add_item(make_client(1, "enc-api-key", name="alice")) + db.commit() + # Reopen with same password + db2 = self._make_encrypted_db(path, password="roundtrip") + results = db2.search_by_value("api_key", "enc-api-key") + self.assertEqual(len(results), 1) + self.assertEqual(results[0].name, "alice") + finally: + if os.path.exists(path): + os.unlink(path) + + def test_sqlitedb_password_kwarg_raises_importerror_without_sqlcipher3(self): + """Confirmed separately in test_sqlitedb_no_sqlcipher.py; skip here.""" + pass + + +class TestSQLiteDBMissingCipher(unittest.TestCase): + """Verify ImportError is raised when sqlcipher3 is absent and password is given.""" + + def test_importerror_when_sqlcipher3_missing(self): + import sys + import unittest.mock as mock + + # Simulate sqlcipher3 not being installed + with mock.patch.dict(sys.modules, {"sqlcipher3": None}): + with tempfile.TemporaryDirectory() as tmpdir: + with self.assertRaises(ImportError) as ctx: + db = SQLiteDB.__new__(SQLiteDB) + db.name = "test" + db.subfolder = tmpdir + db.password = "secret" + # Patch xdg_data_home so db_path resolves inside tmpdir + with mock.patch("hivemind_sqlite_database.xdg_data_home", + return_value=tmpdir): + db.__post_init__() + self.assertIn("sqlcipher3", str(ctx.exception)) + + if __name__ == "__main__": unittest.main() From eabc7d82324933357e46ea9bf5cdabdad3c786b7 Mon Sep 17 00:00:00 2001 From: JarbasAl <33701864+JarbasAl@users.noreply.github.com> Date: Wed, 15 Apr 2026 10:20:57 +0000 Subject: [PATCH 3/4] Increment Version to 0.3.0a1 --- hivemind_sqlite_database/version.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hivemind_sqlite_database/version.py b/hivemind_sqlite_database/version.py index 0343953..59ee186 100644 --- a/hivemind_sqlite_database/version.py +++ b/hivemind_sqlite_database/version.py @@ -1,8 +1,8 @@ # START_VERSION_BLOCK VERSION_MAJOR = 0 -VERSION_MINOR = 2 -VERSION_BUILD = 1 -VERSION_ALPHA = 0 +VERSION_MINOR = 3 +VERSION_BUILD = 0 +VERSION_ALPHA = 1 # END_VERSION_BLOCK __version__ = f"{VERSION_MAJOR}.{VERSION_MINOR}.{VERSION_BUILD}" + (f"a{VERSION_ALPHA}" if VERSION_ALPHA else "") From 8e3402fd07b916cdfe0524efc7f61e948f5fbe1b Mon Sep 17 00:00:00 2001 From: JarbasAl <33701864+JarbasAl@users.noreply.github.com> Date: Wed, 15 Apr 2026 10:21:40 +0000 Subject: [PATCH 4/4] Update Changelog --- CHANGELOG.md | 51 +++------------------------------------------------ 1 file changed, 3 insertions(+), 48 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d004c0..c520e0b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,57 +1,12 @@ # Changelog -## [0.2.1a4](https://github.com/JarbasHiveMind/hivemind-sqlite-database/tree/0.2.1a4) (2026-04-15) +## [0.3.0a1](https://github.com/JarbasHiveMind/hivemind-sqlite-database/tree/0.3.0a1) (2026-04-15) -[Full Changelog](https://github.com/JarbasHiveMind/hivemind-sqlite-database/compare/0.2.1a3...0.2.1a4) +[Full Changelog](https://github.com/JarbasHiveMind/hivemind-sqlite-database/compare/0.2.1...0.3.0a1) **Merged pull requests:** -- ci: sync publish workflows from shared templates [\#22](https://github.com/JarbasHiveMind/hivemind-sqlite-database/pull/22) ([JarbasAl](https://github.com/JarbasAl)) - -## [0.2.1a3](https://github.com/JarbasHiveMind/hivemind-sqlite-database/tree/0.2.1a3) (2026-04-15) - -[Full Changelog](https://github.com/JarbasHiveMind/hivemind-sqlite-database/compare/0.2.1a2...0.2.1a3) - -## [0.2.1a2](https://github.com/JarbasHiveMind/hivemind-sqlite-database/tree/0.2.1a2) (2026-04-15) - -[Full Changelog](https://github.com/JarbasHiveMind/hivemind-sqlite-database/compare/0.2.1a1...0.2.1a2) - -**Merged pull requests:** - -- Update actions/setup-python action to v6 [\#15](https://github.com/JarbasHiveMind/hivemind-sqlite-database/pull/15) ([renovate[bot]](https://github.com/apps/renovate)) -- Update actions/checkout action to v6 [\#9](https://github.com/JarbasHiveMind/hivemind-sqlite-database/pull/9) ([renovate[bot]](https://github.com/apps/renovate)) - -## [0.2.1a1](https://github.com/JarbasHiveMind/hivemind-sqlite-database/tree/0.2.1a1) (2026-04-15) - -[Full Changelog](https://github.com/JarbasHiveMind/hivemind-sqlite-database/compare/0.2.0a1...0.2.1a1) - -**Merged pull requests:** - -- fix: move pytest\_plugins to top-level conftest for pytest 9 compatibility [\#18](https://github.com/JarbasHiveMind/hivemind-sqlite-database/pull/18) ([JarbasAl](https://github.com/JarbasAl)) - -## [0.2.0a1](https://github.com/JarbasHiveMind/hivemind-sqlite-database/tree/0.2.0a1) (2026-04-15) - -[Full Changelog](https://github.com/JarbasHiveMind/hivemind-sqlite-database/compare/0.0.4a2...0.2.0a1) - -**Merged pull requests:** - -- feat: release 0.1.0 — tests, thread-safety, SQL injection fix [\#14](https://github.com/JarbasHiveMind/hivemind-sqlite-database/pull/14) ([JarbasAl](https://github.com/JarbasAl)) - -## [0.0.4a2](https://github.com/JarbasHiveMind/hivemind-sqlite-database/tree/0.0.4a2) (2025-12-19) - -[Full Changelog](https://github.com/JarbasHiveMind/hivemind-sqlite-database/compare/0.0.4a1...0.0.4a2) - -**Merged pull requests:** - -- chore\(deps\): update actions/setup-python action to v6 [\#12](https://github.com/JarbasHiveMind/hivemind-sqlite-database/pull/12) ([renovate[bot]](https://github.com/apps/renovate)) - -## [0.0.4a1](https://github.com/JarbasHiveMind/hivemind-sqlite-database/tree/0.0.4a1) (2025-12-18) - -[Full Changelog](https://github.com/JarbasHiveMind/hivemind-sqlite-database/compare/0.0.3...0.0.4a1) - -**Merged pull requests:** - -- chore: Configure Renovate [\#7](https://github.com/JarbasHiveMind/hivemind-sqlite-database/pull/7) ([renovate[bot]](https://github.com/apps/renovate)) +- feat: add SQLCipher encryption support \(password kwarg\) [\#24](https://github.com/JarbasHiveMind/hivemind-sqlite-database/pull/24) ([JarbasAl](https://github.com/JarbasAl))