Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Schema-validated score template registry with bundled `performance_health_v1` definition and FastAPI endpoints.
- Automated score template tests covering registry usage and HTTP access.
- Adaptive blueprint engine with stratified selection, knockout-aware scoring, preview endpoints, and bundled sample pool.
- Assessment version catalog with item bank, response linkage, and exposure stats models plus seed script.

### Changed
- Centralized SPOT documentation to include bundled templates and score templates.
Expand All @@ -22,6 +24,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Docs
- Added score template function documentation, ERD assets, and schema overview with nb-NO localization.
- Added blueprint engine function documentation, SPOT updates, ERD refresh, and localization alignment.
- Documented item bank models, refreshed schema diagrams, and updated SPOT/tasklist references.

## [3.1.0] - 2025-09-24

Expand Down
14 changes: 14 additions & 0 deletions SPOT.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,23 @@
- [x] Expose score template summaries and definitions through FastAPI endpoints and automated tests.
- [x] Update documentation set (function docs, tasklist, ERD, localization) and changelog for the new score template feature.

## Integration Plan – Adaptive Blueprint Selection
- [x] Add blueprint schema resources with anchors, quotas, and scoring metadata.
- [x] Implement stratified selector + scoring engine with exposure handling tests.
- [x] Publish blueprint preview API endpoints and automated coverage.
- [x] Refresh documentation (function doc, SPOT, ERD, localization) and changelog entries.

## Integration Plan – Assessment Item Bank Tracking
- [x] Model persistent assessment versions, item bank entries, response linkage, and exposure statistics.
- [x] Seed Postgres schema for the new tables with constraints and indexes.
- [x] Extend blueprint engine utilities to hydrate pools from item bank records and stats.
- [x] Update documentation set (function doc, SPOT, ERD, changelog, localization, tasklist).

## Function Catalog

### Backend (Python/FastAPI)
- <i class="fas fa-clipboard-check"></i> **Assessment Template Registry** — Bundled template loader & API. _(Function doc pending migration; tracked in integration plan.)_
- <i class="fas fa-layer-group"></i> **Assessment Blueprint Engine** — Stratified item blueprint loader, selector, scoring, and preview API. → `./docs/functions/assessment_blueprint_engine.md`
- <i class="fas fa-chart-gauge"></i> **Score Template Registry** — Schema-validated score template loader with FastAPI exposure. → `./docs/functions/score_template_registry.md`
- <i class="fas fa-network-wired"></i> **API Router v1** — Aggregates versioned public/admin routers. → `./docs/functions/api_router_v1.md`
- <i class="fas fa-cogs"></i> **Core Setup & Lifespan** — FastAPI factory & Redis pool orchestration with test bypass flag. → `./docs/functions/core_setup.md`
Expand All @@ -32,6 +45,7 @@
- Existing Vuexy-based assets (see `docs/index.md`) pending catalog alignment.

### Database (PostgreSQL)
- <i class="fas fa-database"></i> **Assessment Item Bank Models** — Persistent item bank, response items, and exposure stats backing adaptive selection. → `./docs/functions/assessment_item_bank_models.md`
- Core assessment schema coverage documented in `docs/Database/database_info.md`.

### Infra/Provisioning
Expand Down
8 changes: 4 additions & 4 deletions docs/erd.mmd
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
erDiagram
SCORE_TEMPLATE ||--o{ SCORE_BUCKET : "qualifies"
SCORE_TEMPLATE ||--o{ SCORE_DIMENSION : "weights"
SCORE_DIMENSION ||--o{ SCORE_ITEM : "covers"
SCORE_TEMPLATE }o--|| SCORE_SCALE : "uses"
ASSESSMENT_VERSIONS ||--o{ ASSESSMENT_ITEM_BANK : "contains"
ASSESSMENT_ITEM_BANK ||--|| ASSESSMENT_ITEM_STATS : "tracks"
ASSESSMENTS ||--o{ ASSESSMENT_RESPONSE_ITEMS : "records"
ASSESSMENT_ITEM_BANK ||--o{ ASSESSMENT_RESPONSE_ITEMS : "delivers"
91 changes: 91 additions & 0 deletions docs/functions/assessment_blueprint_engine.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
---
langs: [en, nb-NO]
lastUpdated: 2025-09-24
---

# Assessment Blueprint Engine — Overview

**en:** Stratified blueprint loader and selector powering adaptive assessment forms. Sources: `src/app/assessment_engine/blueprint_engine.py`, `src/app/assessment_engine/__init__.py`, `src/app/schemas/assessment_blueprint.py`, `src/app/api/v1/assessments.py`, `tests/test_assessment_blueprint_engine.py`.

**nb-NO:** Stratifisert blueprint-laster og trekker for adaptive vurderingsskjema. Kildebaner: `src/app/assessment_engine/blueprint_engine.py`, `src/app/assessment_engine/__init__.py`, `src/app/schemas/assessment_blueprint.py`, `src/app/api/v1/assessments.py`, `tests/test_assessment_blueprint_engine.py`.

**SPOT:** ./SPOT.md#function-catalog

## API

- `GET /api/v1/assessment/blueprint` – Lists bundled blueprint summaries for operators.
- `GET /api/v1/assessment/blueprint/{template_id}/preview` – Returns deterministic item selection previews (optional `seed`).
- Python helpers:
- `load_blueprint_document(template_id)` – Parse blueprint JSON into validated model.
- `select_items(document, pool, seen_codes=None, rng=None)` – Stratified sampling respecting quotas, anchors, exposure caps.
- `score_responses(selected_items, item_scores, document)` – Weighted dimension scoring with knockout policies.
- `generate_selection_preview(template_id, seed=None)` – Utility harness for API exposure/tests.
- `pool_from_item_bank(records, stats=None)` – Merge persisted item bank rows with live stats before selection.

## Design

- Blueprint definitions live in `src/app/assessment_templates/blueprints/*.json` and follow the `AssessmentBlueprintDocument` schema (quotas, anchors, critical policies, scoring, sample pools).
- Selection honours three layers:
1. Explicit anchor items → subtract matching difficulty quotas.
2. Remaining per-dimension anchor quota.
3. Difficulty buckets with weighted random sampling (difficulty factors, discrimination, exposure penalty with default cap fallback).
- Exposure caps default to blueprint `exposure.default_cap` unless item overrides.
- Scoring aggregates dimension scores with configurable weight strategies and supports two critical modes: knockout (bucket override) or weighted (multiplied weight).
- Preview endpoints reuse bundled sample pools so API consumers can verify quotas without hitting production item banks.
- `pool_from_item_bank` hydrates `BlueprintItem` objects from the Postgres item bank, preferring live discrimination/exposure metrics when stats exist.
- Error handling surfaces invalid/missing blueprint packages with `BlueprintLoadError` mapped to HTTP 404.

## Usage

```python
from app.assessment_engine import load_blueprint_document, sample_pool_from_blueprint, select_items, score_responses

document = load_blueprint_document("course_gdpr_social_email_cookies_v1_no")
pool = sample_pool_from_blueprint(document)
selected = select_items(document, pool, seen_codes={"safety_easy_intro_rules"})
responses = {item.code: 0.8 for item in selected}
summary = score_responses(selected, responses, document)
print(summary.overall_bucket)
```

When sourcing items from the persistent bank:

```python
from app.assessment_engine import pool_from_item_bank

pool = pool_from_item_bank(db_items, db_stats)
selected = select_items(document, pool, seen_codes=session_history)
```

HTTP preview example:

```bash
curl "http://localhost:8000/api/v1/assessment/blueprint/course_gdpr_social_email_cookies_v1_no/preview?seed=42"
```

## Changelog

### [Unreleased]
- 2025-09-24: Initial blueprint loader, selector, scoring engine, and API previews.
- 2025-09-24: Added item bank hydration helper aligning selection with persisted exposure statistics.

## Diagrams

```mermaid
digraph BlueprintFlow {
rankdir=LR
BlueprintJSON["Blueprint JSON\n(quotas, anchors)"]
SamplePool["Sample Pool\n(items with stats)"]
Selector["Stratified Selector"]
Responses["Responses\n(normalised 0..1)"]
Scorer["Scoring Engine\n(weights + knockout)"]
Summary["Dimension & Total Summary"]

BlueprintJSON -> Selector
SamplePool -> Selector
Selector -> Responses
Responses -> Scorer
Selector -> Scorer
Scorer -> Summary
}
```
72 changes: 72 additions & 0 deletions docs/functions/assessment_item_bank_models.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
langs: [en, nb-NO]
lastUpdated: 2025-09-24
---

# Assessment Item Bank Models — Overview

**en:** SQLAlchemy models and Postgres schema powering blueprint-governed item banks, response linkage, and exposure statistics. Sources: `src/app/models/assessment_item_bank.py`, `seed_scripts/03_assessment_item_bank_schema.sql`.

**nb-NO:** SQLAlchemy-modeller og Postgres-skjema for blueprint-styrt spørsmålsbank, responskobling og eksponeringsstatistikk. Kildebaner: `src/app/models/assessment_item_bank.py`, `seed_scripts/03_assessment_item_bank_schema.sql`.

**SPOT:** ./SPOT.md#function-catalog

## API

- SQLAlchemy models:
- `AssessmentVersion` — Links template IDs with blueprint names/versions.
- `AssessmentItemBank` — Stores per-item difficulty, weights, anchors, and metadata.
- `AssessmentResponseItem` — Persists rendered items, answers, and item-level scores per assessment attempt.
- `AssessmentItemStats` — Tracks facility, discrimination, and exposure for rotation governance.
- Schema bootstrap: `seed_scripts/03_assessment_item_bank_schema.sql` creates tables, constraints, and indexes idempotently.

## Design

- Item lifecycle is scoped by `assessment_versions` so blueprint releases can rotate independently of assessments.
- JSONB columns hold tags, arbitrary metadata, and submitted answers for flexible reporting.
- Cascading deletes clear bank entries when a version is retired while response rows retain referential integrity via `RESTRICT`.
- Stats table centralises KPI inputs (facility, discrimination, exposure) used by the selector to prioritise low-exposure/high-signal items.
- Exposure caps are stored per item with optional overrides; selection helper merges real-time stats before sampling.

## Usage

```python
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession

from app.assessment_engine import pool_from_item_bank
from app.models.assessment_item_bank import AssessmentItemBank, AssessmentItemStats


async def load_pool(session: AsyncSession, version_id):
records = (
await session.execute(
select(AssessmentItemBank).where(AssessmentItemBank.version_id == version_id)
)
).scalars().all()

stats = (
await session.execute(
select(AssessmentItemStats).where(AssessmentItemStats.item_id.in_([r.item_id for r in records]))
)
).scalars().all()

return pool_from_item_bank(records, stats)
```

**nb-NO:** Kjør `seed_scripts/03_assessment_item_bank_schema.sql` før bruk for å sikre tabellene finnes.

## Changelog

### [Unreleased]
- 2025-09-24: Introduced assessment version catalog, item bank, response link, and stats models + schema script.

## Diagrams

```mermaid
erDiagram
ASSESSMENT_VERSIONS ||--o{ ASSESSMENT_ITEM_BANK : "contains"
ASSESSMENT_ITEM_BANK ||--|| ASSESSMENT_ITEM_STATS : "tracks"
ASSESSMENTS ||--o{ ASSESSMENT_RESPONSE_ITEMS : "records"
ASSESSMENT_ITEM_BANK ||--o{ ASSESSMENT_RESPONSE_ITEMS : "delivers"
```
12 changes: 6 additions & 6 deletions docs/schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ lastUpdated: 2025-09-24

# Data Model Overview

**en:** Simplified entity diagram for score templates and related concepts.
**en:** Conceptual view of assessment versions, item bank governance, response linkage, and exposure statistics.

**nb-NO:** Forenklet enhetsdiagram for skårmaler og tilhørende begreper.
**nb-NO:** Konseptuelt diagram over vurderingsversjoner, spørsmålsbank-styring, responskobling og eksponeringsstatistikk.

![ERD](../public/erd.svg)

Expand All @@ -16,10 +16,10 @@ lastUpdated: 2025-09-24

```mermaid
erDiagram
SCORE_TEMPLATE ||--o{ SCORE_BUCKET : "qualifies"
SCORE_TEMPLATE ||--o{ SCORE_DIMENSION : "weights"
SCORE_DIMENSION ||--o{ SCORE_ITEM : "covers"
SCORE_TEMPLATE }o--|| SCORE_SCALE : "uses"
ASSESSMENT_VERSIONS ||--o{ ASSESSMENT_ITEM_BANK : "contains"
ASSESSMENT_ITEM_BANK ||--|| ASSESSMENT_ITEM_STATS : "tracks"
ASSESSMENTS ||--o{ ASSESSMENT_RESPONSE_ITEMS : "records"
ASSESSMENT_ITEM_BANK ||--o{ ASSESSMENT_RESPONSE_ITEMS : "delivers"
```

</details>
4 changes: 4 additions & 0 deletions docs/tasks/tasklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,7 @@
- [x] Verify diagrams exist for `api_router_v1` (Diagrams section)
- [x] Update docs & SPOT for `core_setup`; CHANGELOG [Unreleased]
- [x] Verify diagrams exist for `core_setup` (Diagrams section)
- [x] Update docs & SPOT for `assessment_blueprint_engine`; CHANGELOG [Unreleased]
- [x] Verify diagrams exist for `assessment_blueprint_engine` (Diagrams section)
- [x] Update docs & SPOT for `assessment_item_bank_models`; CHANGELOG [Unreleased]
- [x] Verify diagrams exist for `assessment_item_bank_models` (Diagrams section)
44 changes: 25 additions & 19 deletions public/erd.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
74 changes: 74 additions & 0 deletions seed_scripts/03_assessment_item_bank_schema.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
-- Docs: ./docs/functions/assessment_item_bank_models.md
-- SPOT: ./SPOT.md#function-catalog

-- Ensure UUID generation helpers are available
CREATE EXTENSION IF NOT EXISTS "pgcrypto";

-- Catalog of assessment versions and blueprint releases
CREATE TABLE IF NOT EXISTS assessment_versions (
version_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
template_id text NOT NULL,
blueprint_name text NOT NULL,
blueprint_version text NOT NULL,
notes text,
is_active boolean NOT NULL DEFAULT true,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz,
CONSTRAINT uq_assessment_versions_template_blueprint
UNIQUE (template_id, blueprint_name, blueprint_version)
);

CREATE INDEX IF NOT EXISTS idx_assessment_versions_template
ON assessment_versions (template_id);

-- Item bank entries governed by blueprint quotas
CREATE TABLE IF NOT EXISTS assessment_item_bank (
item_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
version_id uuid NOT NULL REFERENCES assessment_versions(version_id) ON DELETE CASCADE,
code text NOT NULL,
dimension text NOT NULL,
difficulty text NOT NULL CHECK (difficulty IN ('easy','medium','hard')),
weight numeric NOT NULL DEFAULT 1.0,
critical boolean NOT NULL DEFAULT false,
anchor boolean NOT NULL DEFAULT false,
discrimination numeric,
exposure_cap numeric,
tags jsonb NOT NULL DEFAULT '[]'::jsonb,
meta jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
CONSTRAINT uq_assessment_item_bank_version_code UNIQUE (version_id, code)
);

CREATE INDEX IF NOT EXISTS idx_assessment_item_bank_dimension
ON assessment_item_bank (dimension);
CREATE INDEX IF NOT EXISTS idx_assessment_item_bank_difficulty
ON assessment_item_bank (difficulty);
CREATE INDEX IF NOT EXISTS idx_assessment_item_bank_anchor
ON assessment_item_bank (anchor);

-- Exposure and performance statistics per item
CREATE TABLE IF NOT EXISTS assessment_item_stats (
item_id uuid PRIMARY KEY REFERENCES assessment_item_bank(item_id) ON DELETE CASCADE,
shown integer NOT NULL DEFAULT 0,
correct integer NOT NULL DEFAULT 0,
facility numeric,
discrimination numeric,
exposure numeric,
last_seen_at timestamptz
);

CREATE INDEX IF NOT EXISTS idx_assessment_item_stats_exposure
ON assessment_item_stats (exposure);

-- Captured responses to individual blueprint-driven items
CREATE TABLE IF NOT EXISTS assessment_response_items (
assessment_id uuid NOT NULL REFERENCES assessments(id) ON DELETE CASCADE,
item_id uuid NOT NULL REFERENCES assessment_item_bank(item_id) ON DELETE RESTRICT,
answer jsonb,
score numeric,
responded_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (assessment_id, item_id)
);

CREATE INDEX IF NOT EXISTS idx_assessment_response_items_item
ON assessment_response_items (item_id);
Loading
Loading