Merge pull request #5 from gofflab/claude/add-genome-reference-tool-t3h7n

Move config and indices to persistent user data directory

Author: Loyale

README.md

@@ -38,9 +38,14 @@ conda activate HCRProbeDesign
 pip install -e .
 ```
 
+## Data directory
+HCRProbeDesign stores configuration and Bowtie2 indices in a persistent user data directory
+at `~/.hcrprobedesign/` so they survive package upgrades. Override the location with the
+`HCRPROBEDESIGN_DATA_DIR` environment variable.
+
 ## Adding a new reference genome
-HCRProbeDesign keeps Bowtie2 index paths in `HCRconfig.yaml`. The `buildGenomeIndex` utility builds
-an index and registers it automatically.
+HCRProbeDesign keeps Bowtie2 index paths in `~/.hcrprobedesign/HCRconfig.yaml`. The `buildGenomeIndex`
+utility builds an index and registers it automatically.
 
 ```bash
 buildGenomeIndex --species zebrafish --fasta /path/to/genome.fa --threads 8
@@ -49,7 +54,7 @@ buildGenomeIndex --species zebrafish --fasta /path/to/genome.fa --threads 8
 Notes:
 - `--fasta` can be repeated and can point to a directory; all `.fa`, `.fasta`, or `.fna` files inside
   will be used.
-- By default, indices are written under the package `indices/` directory and the config is updated.
+- By default, indices are written under `~/.hcrprobedesign/indices/` and the config is updated.
 - Use `--indices-dir` to write indices elsewhere and `--config` to update a specific config file.
 - Use `--force` to overwrite an existing index or config entry.
 

VERSIONINFO.md

@@ -1,3 +1,8 @@
+## v0.3.1 - 04.08.2026
+  + Moved config and indices to persistent user data directory (`~/.hcrprobedesign/`)
+  + Reference genomes and configuration now survive `pip install -U` upgrades
+  + Automatic migration of data from old package-relative locations
+  + Support `HCRPROBEDESIGN_DATA_DIR` environment variable to override data location
 ## v0.3.0 - 04.08.2026
   + Added `listReferences` CLI tool to display installed reference genomes and default parameters
   + Added `HCRProbeDesign.listReferences` module with programmatic access to reference info

docs/configuration.md

@@ -1,7 +1,24 @@
 # Configuration
 
-HCRProbeDesign stores reference genome index paths in `HCRconfig.yaml`.
-The file is located in the package directory and is read at runtime.
+HCRProbeDesign stores reference genome index paths and default parameters in
+`HCRconfig.yaml`, located in the user data directory (`~/.hcrprobedesign/` by default).
+
+## Data directory
+
+All user data (configuration and Bowtie2 indices) lives in `~/.hcrprobedesign/`.
+This directory is created automatically on first use and **persists across package
+upgrades**, so you won't lose your indices when running `pip install -U hcrprobedesign`.
+
+Override the location by setting the `HCRPROBEDESIGN_DATA_DIR` environment variable:
+```bash
+export HCRPROBEDESIGN_DATA_DIR=/path/to/custom/dir
+```
+
+### Migration from older versions
+
+If you are upgrading from v0.3.0 or earlier (where data was stored inside the
+package directory), HCRProbeDesign will automatically detect and migrate your
+existing indices and species registrations on first run.
 
 ## Example
 ```yaml
@@ -13,14 +30,15 @@ species:
 ```
 
 ## Viewing the current configuration
-Run `listReferences` to display all registered species and default parameters:
+Run `listReferences` to display all registered species, default parameters, and the
+data directory location:
 
 ```bash
 listReferences
 ```
 
 ## Notes
-- Paths can be absolute or package-relative.
+- Paths can be absolute or relative to the data directory.
 - `buildGenomeIndex` updates this file automatically.
 - `fetchMouseIndex` also registers the mouse index for you.
 - Use `--config` to write to a different config file when building indices.

docs/reference-genome.md

@@ -25,7 +25,7 @@ buildGenomeIndex --species zebrafish --fasta /path/to/genome.fa --threads 8
 Notes:
 - `--fasta` is repeatable and can point to a directory; all `.fa`, `.fasta`, or `.fna` files
   (including `.gz`) will be included.
-- Indices are written under the package `indices/` directory by default.
+- Indices are written under `~/.hcrprobedesign/indices/` by default.
 - Use `--indices-dir` to write indices elsewhere and `--config` to update a specific config file.
 - Use `--force` to overwrite an existing index or config entry.
 

setup.cfg

@@ -1,7 +1,7 @@
 [metadata]
 # replace with your username:
 name = hcrprobedesign
-version = 0.3.0
+version = 0.3.1
 author = Loyal A. Goff
 author_email = loyalgoff@jhmi.edu
 description = Probe Design tool for Hybridization Chain Reaction

src/HCRProbeDesign/__init__.py

@@ -3,6 +3,7 @@
 from . import probeDesign
 from . import thermo
 from . import sequencelib
+from ._datadir import get_data_dir, get_config_path, get_indices_dir, ensure_data_dir
 
 import os
 try:
@@ -13,8 +14,14 @@
 _ROOT = os.path.abspath(os.path.dirname(__file__))
 
 def index_path():
-    """Return the default package-relative path for Bowtie2 indices."""
-    return os.path.join(_ROOT, 'indices')
+    """Return the path to the user's Bowtie2 indices directory.
+
+    Returns the user data directory (``~/.hcrprobedesign/indices/``) which
+    persists across package upgrades.  The data directory is created
+    automatically on first access.
+    """
+    ensure_data_dir()
+    return get_indices_dir()
 
 def _resolve_version():
     if importlib_metadata is not None:

src/HCRProbeDesign/_datadir.py

@@ -0,0 +1,213 @@
+"""Manage the user data directory for HCRProbeDesign.
+
+Configuration and Bowtie2 indices are stored in a persistent user directory
+(``~/.hcrprobedesign/`` by default) so they survive package upgrades.
+
+Override the location by setting the ``HCRPROBEDESIGN_DATA_DIR`` environment
+variable.
+"""
+
+import glob
+import os
+import shutil
+import sys
+
+import yaml
+
+_PACKAGE_DIRECTORY = os.path.dirname(os.path.abspath(__file__))
+_DEFAULT_DATA_DIR = os.path.join(os.path.expanduser("~"), ".hcrprobedesign")
+_SEED_CONFIG = os.path.join(_PACKAGE_DIRECTORY, "HCRconfig.yaml")
+
+
+def get_data_dir():
+    """
+    Return the path to the user data directory.
+
+    Uses ``HCRPROBEDESIGN_DATA_DIR`` if set, otherwise ``~/.hcrprobedesign``.
+
+    :return: Absolute path to the data directory.
+    """
+    return os.environ.get("HCRPROBEDESIGN_DATA_DIR", _DEFAULT_DATA_DIR)
+
+
+def get_config_path():
+    """
+    Return the path to the user's ``HCRconfig.yaml``.
+
+    :return: Absolute path to the config file.
+    """
+    return os.path.join(get_data_dir(), "HCRconfig.yaml")
+
+
+def get_indices_dir():
+    """
+    Return the path to the user's indices directory.
+
+    :return: Absolute path to the indices directory.
+    """
+    return os.path.join(get_data_dir(), "indices")
+
+
+def _load_yaml(path):
+    """Load a YAML file, returning an empty dict if missing."""
+    if not os.path.exists(path):
+        return {}
+    with open(path, "r") as fh:
+        return yaml.safe_load(fh) or {}
+
+
+def _old_package_config_path():
+    """Return the path to the old package-relative config file."""
+    return os.path.join(_PACKAGE_DIRECTORY, "HCRconfig.yaml")
+
+
+def _old_package_indices_dir():
+    """Return the path to the old package-relative indices directory."""
+    return os.path.join(_PACKAGE_DIRECTORY, "indices")
+
+
+def _has_old_data():
+    """
+    Check whether there is user data in the old package-relative locations.
+
+    :return: Tuple of (has_species_entries, list_of_index_dirs).
+    """
+    old_config = _load_yaml(_old_package_config_path())
+    species = old_config.get("species", {}) or {}
+    has_species = len(species) > 0
+
+    old_indices = _old_package_indices_dir()
+    index_dirs = []
+    if os.path.isdir(old_indices):
+        for entry in os.listdir(old_indices):
+            entry_path = os.path.join(old_indices, entry)
+            if os.path.isdir(entry_path):
+                bt2_files = glob.glob(os.path.join(entry_path, "*.bt2*"))
+                if bt2_files:
+                    index_dirs.append(entry)
+
+    return has_species, index_dirs
+
+
+def _migrate_old_data():
+    """
+    Migrate config entries and index files from the package directory to the
+    user data directory.
+
+    Prints progress messages to stderr.
+    """
+    data_dir = get_data_dir()
+    new_config_path = get_config_path()
+    new_indices_dir = get_indices_dir()
+
+    old_config_path = _old_package_config_path()
+    old_indices_dir = _old_package_indices_dir()
+
+    old_config = _load_yaml(old_config_path)
+    new_config = _load_yaml(new_config_path)
+
+    old_species = old_config.get("species", {}) or {}
+    new_species = new_config.setdefault("species", {})
+
+    migrated_any = False
+
+    # Migrate index directories
+    if os.path.isdir(old_indices_dir):
+        for entry in os.listdir(old_indices_dir):
+            old_entry_path = os.path.join(old_indices_dir, entry)
+            if not os.path.isdir(old_entry_path):
+                continue
+            bt2_files = glob.glob(os.path.join(old_entry_path, "*.bt2*"))
+            if not bt2_files:
+                continue
+
+            new_entry_path = os.path.join(new_indices_dir, entry)
+            if os.path.exists(new_entry_path):
+                print(
+                    f"  Skipping index '{entry}' (already exists in {new_indices_dir})",
+                    file=sys.stderr,
+                )
+                continue
+
+            print(
+                f"  Moving index '{entry}' -> {new_entry_path}",
+                file=sys.stderr,
+            )
+            shutil.copytree(old_entry_path, new_entry_path)
+            migrated_any = True
+
+    # Migrate species config entries
+    for name, entry in old_species.items():
+        if name in new_species:
+            continue
+        old_index = entry.get("bowtie2_index", "")
+        # Rewrite relative paths to point to the new indices dir
+        if not os.path.isabs(old_index):
+            # e.g. "indices/mm10/mm10" -> new absolute path
+            basename = old_index
+            if basename.startswith("indices/") or basename.startswith("indices\\"):
+                basename = basename[len("indices/"):]
+            new_index = os.path.join(new_indices_dir, basename)
+        else:
+            new_index = old_index
+        new_species[name] = {"bowtie2_index": new_index}
+        migrated_any = True
+        print(
+            f"  Migrated species '{name}' -> {new_index}",
+            file=sys.stderr,
+        )
+
+    # Preserve default_params from old config if not present in new
+    if "default_params" not in new_config and "default_params" in old_config:
+        new_config["default_params"] = old_config["default_params"]
+        migrated_any = True
+
+    if migrated_any:
+        with open(new_config_path, "w") as fh:
+            yaml.safe_dump(new_config, fh, sort_keys=False)
+
+    return migrated_any
+
+
+def ensure_data_dir():
+    """
+    Ensure the user data directory exists and is seeded.
+
+    On first run, creates ``~/.hcrprobedesign/`` with a seeded
+    ``HCRconfig.yaml`` and ``indices/`` directory.  If data exists in the
+    old package-relative location, it is migrated automatically.
+    """
+    data_dir = get_data_dir()
+    config_path = get_config_path()
+    indices_dir = get_indices_dir()
+
+    if os.path.isdir(data_dir) and os.path.exists(config_path):
+        return  # Already initialized
+
+    first_init = not os.path.isdir(data_dir)
+    os.makedirs(indices_dir, exist_ok=True)
+
+    # Seed config from package default if no config exists yet
+    if not os.path.exists(config_path):
+        if os.path.exists(_SEED_CONFIG):
+            shutil.copy2(_SEED_CONFIG, config_path)
+        else:
+            # Minimal fallback
+            with open(config_path, "w") as fh:
+                yaml.safe_dump({"species": {}, "default_params": {}}, fh)
+
+    if first_init:
+        print(
+            f"Initialized HCRProbeDesign data directory: {data_dir}",
+            file=sys.stderr,
+        )
+
+    # Check for data in old package location and migrate
+    has_species, index_dirs = _has_old_data()
+    if has_species or index_dirs:
+        print(
+            "Found reference data in old package directory. Migrating...",
+            file=sys.stderr,
+        )
+        _migrate_old_data()
+        print("Migration complete.", file=sys.stderr)