3.1.0-alpha

What's Changed

• Do not call logger on None object. by @pdelboca in #269
• docs: update README by @rzmk in #271
• chore: run Serena onboarding by @jqnatividad in #273
• Add CI by @a5dur in #272
• feat(qsv)!: bump MINIMUM_QSV_VERSION from 4.0.0 to 20.0.0 by @jqnatividad in #275
• ci: install qsv via Debian package, fall back to zip by @jqnatividad in #276
• feat(orchestration): replace RQ worker with Prefect 3 flow by @jqnatividad in #277
• refactor(prefect): make per-stage result dataclasses self-contained by @jqnatividad in #280
• feat(prefect): DPP_PREFECT_WORKER sentinel for the bootstrap by @jqnatividad in #281
• chore(deps): tighten Prefect pin to >=3.7,<3.8 by @jqnatividad in #283
• test(cli): unit tests for migrate-from-rq by @jqnatividad in #284
• refactor(prefect): runtime-resolve flow-timeout + download retries by @jqnatividad in #285
• feat(prefect): re-enable content caching with stable file persistence by @jqnatividad in #286
• fix(prefect): address roborev review on #286 by @jqnatividad in #287
• feat(prefect): Prefect Variables as highest-priority runtime config by @jqnatividad in #288
• feat(prefect): per-domain subflow primitives (PII, spatial) by @jqnatividad in #289
• fix(prefect): address roborev review on #289 by @jqnatividad in #290
• Codebase review fixes + CI hardening by @jqnatividad in #274
• docs(readme): fix installation-step numbering + drop RQ worker reference by @jqnatividad in #291
• fix(metadata): persist preview/preview_rows after datastore_create (rebases #270) by @jqnatividad in #292
• fix(analysis): pass reserved/prefix to safenames sanitize call (rebases #267) by @jqnatividad in #293
• build: include MANIFEST.in data files in the wheel (rebases #252) by @jqnatividad in #294
• fix(metadata): blank Date/DateTime mean in summary-stats CSV (rebases #254) by @jqnatividad in #295
• feat(database): opt-out of TRUNCATE + COPY-FREEZE single-txn pattern by @jqnatividad in #296
• fix(database): narrow TRUNCATE error swallow + clarify VACUUM comment by @jqnatividad in #297
• feat(metadata): persist dpp_spatial_extent for CSV lat/lon resources by @jqnatividad in #298
• feat(cli): per-resource error isolation + non-zero exit on failure for resubmit/submit by @jqnatividad in #299
• chore(integration): make the integration stack runnable locally end-to-end by @jqnatividad in #300
• feat(ai-suggestions): optional qsv describegpt stage for LLM-derived metadata by @jqnatividad in #301
• feat(ai-suggestions): scheming UI for AI-derived metadata (PR #253 follow-up) by @jqnatividad in #302
• fix(ai-suggestions): real-qsv schema + CLI flag corrections (PR #301/#302 follow-up) by @jqnatividad in #303
• test(js): vitest + jsdom unit tests for scheming-ai-suggestions.js (+ fixes 1 polling bug) by @jqnatividad in #304
• build(deps): bump esbuild and vitest by @dependabot[bot] in #305
• build(deps): bump vite and vitest by @dependabot[bot] in #306
• fix(#265): stash Data Dictionary before delete, restore on rollback by @jqnatividad in #307
• fix(#145): persist all timestamps as naive UTC, surface UTC in UI by @jqnatividad in #308
• feat(#221): configurable file-hash algorithm (blake3 default, sha256, md5) by @jqnatividad in #309
• fix(#310): preserve file_hash on metadata stage's resource re-fetch by @jqnatividad in #312
• fix(#311): don't apply DownloadResult.resource on rehydrate (cross-resource cache leak) by @jqnatividad in #313
• fix(#179): map qsv Date to Postgres date (not timestamp) by @jqnatividad in #314
• feat(qsv)!: bump qsv pin + MINIMUM 20.0.0 → 20.1.0 + #173 regression coverage by @jqnatividad in #315
• fix(#111): surface DP+ version in flow log banner + JOB DONE capstone by @jqnatividad in #316
• fix(#142): add AUTO_INDEX_MIN_THRESHOLD floor + bump default 3 → 10 by @jqnatividad in #317
• fix(#142): address Copilot post-merge review on #317 by @jqnatividad in #318
• feat(#112): locale-aware number normalization (supersedes #319) by @jqnatividad in #320
• docs(#205): replace placeholder demo with PyConUS '25 DRUF recording by @jqnatividad in #321
• fix(#261): empty date range suggestion handling (cherry-pick + tests) by @jqnatividad in #322
• feat(#61): DOWNLOAD_ALWAYS_WHITELIST — operator-controlled re-processing by @jqnatividad in #323
• fix: correct pii_screening config-key typo + documentation-audit findings by @jqnatividad in #324
• docs: bring CLAUDE.md up to date with the v3.0 Prefect architecture by @jqnatividad in #325
• Address documentation-audit loose ends: unit-test CI, FORMATS, README config by @jqnatividad in #326
• docs: add a Testing section to README.md + fix stale .coveragerc by @jqnatividad in #327

New Contributors

• @dependabot[bot] made their first contribution in #305

Full Changelog: 3.0.0...3.1.0-alpha

3.0.0

What's Changed

• Update README with some fixes by @tino097 in #178
• Druf apr2025 by @jqnatividad in #180
• Refactor upload log level by @jqnatividad in #181
• feat: zip file support by @jqnatividad in #182
• feat: shapefile support by @jqnatividad in #183
• Refactor jobs py by @jqnatividad in #184
• feat: make frequency limit configurable; move stats/freq copying to datastore from jobs.py to qsv_utils.py by @jqnatividad in #185
• Lat lon columns inferencing for use in Formulas by @jqnatividad in #188
• Configurable Date/Datetime inferencing and dataset stats by @jqnatividad in #190
• refactor: move pii-screening to a separate module by @jqnatividad in #191
• chore: add WIP geojson update by @rzmk in #186
• "smart" formula spatial functions by @jqnatividad in #192
• Jobs cleanup by @jqnatividad in #193
• Fix datastore upload log timestamps by @jqnatividad in #194
• DCAT 3 formula helpers by @jqnatividad in #195
• fix: tmp input was being wrongfully assigned by @jqnatividad in #197
• refactored SQL-enabled formulas by @jqnatividad in #199
• auto unzip one file setting by @jqnatividad in #200
• add LRU caches to potentially expensive Formula methods by @jqnatividad in #201
• feat: add dpp_suggestions.STATUS to track formulae processing progress by @jqnatividad in #202
• refactor dpp_suggestions.STATUS to sync with Suggestions UI by @jqnatividad in #203
• Refactor: remove stats & freq table save to datastore by @jqnatividad in #204
• fixed formula status updates by @minhajuddin2510 in #206
• added druf changes by @minhajuddin2510 in #210
• Update README.md by @minhajuddin2510 in #211
• docs: update TNRIS text to TxGIO along with link by @rzmk in #219
• changed the druf to be backwards compatible and disabled by default by @minhajuddin2510 in #215
• Improved dpp_suggestions check in jobs.py by @twdbben in #222
• Add pyproject toml file by @tino097 in #225
• Add Testing workflow by @a5dur in #227
• chore: DP+ requires python 3.10 by @jqnatividad in #239
• Fix issues with extract messages by @tino097 in #240
• Remove duplicate settings by @tino097 in #242
• Update dataset-druf.yaml by @khanmustaaq in #241
• Update GDAL installation and dependency setup by @a5dur in #244
• Fix: Make default_excel_sheet config variable work properly by @khanmustaaq in #246
• Fix URL by @avdata99 in #249
• Add Technical Reference readme file for test workflow by @a5dur in #247
• Commit after truncate to reduce lock contention by @EricSoroos in #263
• Feature/reorganize project structure by @tino097 in #264

New Contributors

• @rzmk made their first contribution in #186
• @twdbben made their first contribution in #222
• @a5dur made their first contribution in #227
• @khanmustaaq made their first contribution in #241

Full Changelog: 2.0.0...3.0.0

2.0.0

[2.0.0] - 2025-04-25

🎉 Data Resource Upload First (DRUF) Workflow is finally here! 🎉

A workflow that flips the old CKAN traditional data ingestion on its head.

• Instead of filling out the metadata first and then uploading the data, users upload data resources first
• In a few seconds, even for very large datasets, analysis and validation is done while precompiling statistical metadata
• This precompiled metadata are then used by Metadata Formulae defined in the scheming yaml files to either precompute other metadata fields (on both package & resource levels) or to offer metadata suggestions
• Metadata Formulae use the same powerful Jinja2 template engine that powers CKAN's templating system.
• It comes with an extensible library of Jinja2 filters/functions that can be used in Metadata Formulae ala Excel.

The DRUF reinvents CKAN data ingestion - by automatically calculating/suggesting "Automagical Metadata" - high-quality, high-resolution metadata that reflects and describes what's INSIDE the dataset (e.g. summary stats; frequency table; spatial extent, date range, outliers, etc. calculated with Metadata Formulae) in addition to metadata about the dataset FILE (e.g. last updated, size of the file, owner, format, license, etc - what's normally found in traditional data catalogs).

Future improvements planned:

• "entry-time" Metadata Formulae
  In addition to the two formula types (formula to set a metadata field directly during creation/update; and suggestion_formula to suggest values using the Bootstap Popover UI), we'll add the ability to allow Data Publishers to enter formulas while they're entering metadata - fully embracing the Excel formula UI/UX aesthetic.
• DCAT3-optimized reference profiles
  Following implementation guidance for both DCAT-US v3 and DCAT-AP 3 scheming profiles with Metadata Formulae to compute recommended and optional properties that allow publishers to more fully take advantage of DCAT3 features and improvements - metadata properties that are often too laborious to manually compile.
• Co-Curator AI
  "Automagical metadata" is the perfect context for AI engines - as it summarizes even very large datasets in just a few kilobytes. It allows the Co-Curator¹ to suggest tags, descriptions, links to related data sets and chat about the corpus WHILE the Data Publisher is curating the data.
• Inline Data Validation
  Optional ability to infer an initial JSON Schema validation file, and then validate future updates to the dataset using it, leveraging the same blazing-fast qsv engine (validating up to 340,000 records/per second²).
• Customizable DRUF Data ingestion pipeline
  Currently, there are numerous configuration settings to fine-tune the DRUF data-ingestion pipeline. However, the built-in default pipeline can only be customized to a limit without customizing the code. We will expose hooks that CKAN operators can take advantage of to tailor their DRUF pipelines to meet their requirements, while preserving the ability to access the precompiled statistical metadata that DP+ maintains.
• Dynamic loading of Formula filters/functions
  So users can share custom Jinja2 filters and functions they developed for their Metadata Formulae.
• Inline Data Enrichment
  Data can be optionally enriched while it's being ingested from other reference datasets within the same CKAN instance or external sources (e.g. enriched against high value curated sources like the Census; geocoding, etc.)
• and more!
  It took a while for us to bake 2.0.0, but we look forward to picking up the pace and co-innovating with the CKAN ecosystem.

NOTE: To fully experience the DRUF workflow, you'll need to use scheming dataset form pages and apply some CKAN core changes. A detailed installation procedure will be published on the Wiki shortly.

---

Added

• Data Resource Upload First (DRUF) Workflow
  • Enhanced resource validation for DRUF workflow
  • Metadata Formulae for precomputing metadata/metadata sugggestions
  • Spatial file support - supports GeoJSON and Shapefiles
• Support for CKAN 2.9 compatibility in CLI operations
• Enhanced error handling and logging for resource uploads

Changed

• Updated CLI interface to work with CKAN 2.9
• Refactored resource upload process to support DRUF workflow
• Improved error messages and user feedback
• Enhanced configuration handling

Fixed

• Various bug fixes and improvements for CKAN 2.9 compatibility
• Resource upload process reliability improvements

Contributors

• @tino097
• @minhajuddin2510
• @jqnatividad

Full Changelog: 1.0.4...2.0.0

1. Inspired by the Curator in Ready Player One ↩

2. validate_index benchmark - https://qsv.dathere.com/benchmarks ↩

1.0.4

Full Changelog: 1.0.3...1.0.4

1.0.3

What's Changed

• Ensure we are always using the same token setting for datapusher by @avdata99 in #168
• Fix iconv by @avdata99 in #169
• Fix the api_token config variable and fix for default views creation by @tino097 in #170
• Migration added by @tino097 in #171
• Migration added by @tino097 in #172

Full Changelog: 1.0.2...1.0.3

1.0.2

What's Changed

• Update README file for DP+ as extension by @tino097 in #143
• Fix MANIFEST.in by @pdelboca in #148
• Migrate cli commands by @tino097 in #150
• [dev-v1.0] Fix init db command by @pdelboca in #151
• Config part by @tino097 in #153
• Database migrations by @tino097 in #154
• Update readme by @tino097 in #156
• Fix yaml extension in MANIFEST.in by @pdelboca in #157
• Fix datefmt compatability with qsv in dev-v1.0 by @Zharktas in #161
• Remove obsolete assets by @pdelboca in #159

New Contributors

• @pdelboca made their first contribution in #148

Full Changelog: 1.0.1...1.0.2

1.0.1

What's Changed

• Replace http requests with actions by @tino097 in #124
• Fix calling package action for resource by @tino097 in #140

Full Changelog: 1.0.0...1.0.1

1.0.0

What's Changed

• Convert the datapusher to work as plugin by @tino097 in #73
• Code cleanup by @tino097 in #89
• [72]Rewrite resource url by @TomeCirun in #109
• Feature db models by @tino097 in #120
• Add migration script by @tino097 in #121
• Code cleanup part two by @tino097 in #123
• Rewrite resource URL if it differs from the defined ckan_url by @jhbruhn in #136
• Fixing issues by @tino097 in #137
• Sync with master by @tino097 in #138

New Contributors

• @tino097 made their first contribution in #73
• @jhbruhn made their first contribution in #136

Full Changelog: 0.16.4...1.0.0

0.16.4

What's Changed

• sync read buffer with buffer size of copyexpert by @jqnatividad in #128

Full Changelog: 0.16.3...0.16.4

0.16.3

What's Changed

• make COPY_READBUFFER_SIZE a configurable parameter by @jqnatividad in #127

Full Changelog: 0.16.2...0.16.3