Skip to content

Architectural change of opencast_io, readers and writers for all drivers added.#3384

Draft
mayankchetan wants to merge 27 commits into
OpenFAST:devfrom
mayankchetan:openfast_io_arch
Draft

Architectural change of opencast_io, readers and writers for all drivers added.#3384
mayankchetan wants to merge 27 commits into
OpenFAST:devfrom
mayankchetan:openfast_io_arch

Conversation

@mayankchetan

Copy link
Copy Markdown
Contributor

Feature or improvement description

Restructures the openfast_io Python package from two monolithic files (FAST_reader.py, 3 652
lines; FAST_writer.py, 2 979 lines) into a layered architecture, with no change to the public
API
(see backward compatibility below). This restructuring was primarily driven by the need for an MCP to use with OpenFAST Agentic Kit (OAK):

  • Layer 1 — openfast_io/io/: one stateless ModuleIO class per module (ElastoDyn, BeamDyn,
    AeroDyn, AeroDisk, InflowWind, ServoDyn + StC, HydroDyn, SeaState, SubDyn, MoorDyn, MAP++,
    ExtPtfm, SimpleElastoDyn), each implementing read(file_path, base_dir) -> dict /
    write(data, file_path, base_dir) against the same fst_vt dict slices as before. Each module
    is independently usable and testable without an OpenFAST binary.
  • Layer 2 — openfast_io/drivers/: OpenFASTDriver (full coupled deck: orchestration,
    cross-module path resolution, OutList registry threading), FASTFarmDriver, and ten new IO
    classes for the standalone module-driver input files
    (aerodyn_driver.dvr,
    beamdyn_driver.inp, HydroDyn/SubDyn/SeaState/InflowWind/MoorDyn/AeroDisk/SED/UnsteadyAero) —
    previously unsupported anywhere in the package.
  • Layer 3 — facades: FAST_reader.py / FAST_writer.py retain InputReader_OpenFAST /
    InputWriter_OpenFAST with identical attributes, methods, and fst_vt layout, now delegating to
    OpenFASTDriver and emitting DeprecationWarning on instantiation.

New ancillary subsystems, each independently importable:

Module Purpose
parsing.py The shared parsing/formatting primitives (float_read, read_array, …) extracted from the reader, plus fmt_field() — a fixed-width formatter that switches to scientific notation instead of overflowing the field
outlist.py Set-based output-channel registry with capture_outlist/emit_outlist used by every module for lossless OutList read↔write round-trips (plus OutList API and legacy bool-dict converters)
schema.py Version-indexed machine-readable parameter metadata (get_param_info('ElastoDyn', 'NumBl')), with a CI drift check against the Fortran Registry (tools/check_registry_drift.py)
validation.py Cross-module deck checks (blade-count mismatch, removed parameters, missing DLL, file-reference existence resolved against the case directory)
formats.py fst_vt ↔ JSON/YAML serialization

The systematic write→read→compare testing this structure enables surfaced and fixed several
pre-existing silent data-loss bugs (details in
openfast_io/docs/architecture-redesign.md, which also records the design rationale and a
section-by-section audit changelog):

  • HydroDyn NBodyMod=1: additional-matrix reads used 6 rows instead of 6*NBody (silent
    truncation for multi-body decks).
  • SubDyn: GuyanDamp matrix parse crashed on comma-separated floats.
  • BeamDyn: multi-blade decks with distinct blade files wrote every blade to the same path
    (blade 3's properties silently overwrote blades 1–2); blade output directories were assumed to
    pre-exist.
  • OutList round-trips: channels were dropped on read and/or entire channel registries dumped on
    write for most modules; now captured and re-emitted losslessly for every module including
    ElastoDyn's optional nodal (BldNd_BladesOut) section and SubDyn's SSOutList.
  • Packaging: pyyaml was imported but never declared, so clean installs of the package failed on
    first import of the YAML helpers.

Related issue, if one exists

Impacted areas of the software

openfast_io/ only — no Fortran source, glue code, registry, r-test, or build-system changes.
Python consumers of openfast_io (WEIS, WISDEM, ROSCO tooling) are unaffected through the retained
facades: same class names, same fst_vt dict keys, same execute()/update()/set_outlist()
behavior. The only behavior changes are the bug fixes listed above — callers reading multi-body
NBodyMod=1 HydroDyn decks or comma-format SubDyn matrices previously received truncated data or
crashes.

Additional supporting information

  • Design document: openfast_io/docs/architecture-redesign.md (before/after layout, per-module
    changes, bug-fix provenance, critical self-review with rebuttals, audit changelog §10).
  • The legacy monolith implementations are deleted rather than retained as _legacy files; git
    history is the reference.
  • FAST_vars_out.py (the legacy channel bool-dict) is retained for compatibility and emits a
    DeprecationWarning on import; target removal is the next major version together with the
    facades.
  • Known deferred cleanups are recorded in the design doc (§9.5): unify the remaining per-module
    OutList helper duplication, align base_dir type hints with the ModuleIO ABC, context-manage
    the three writers still using manual open()/fsync, exclude tests/ from the wheel, and
    implement FASTFarmDriver.write().

Generative AI usage

Substantial portions of this feature were developed with Anthropic Claude (Claude Code), under human
direction for all design decisions. Specifically, a multi-agent workflow was used: Claude Fable 5
orchestrated the work (design/spec interview with the human author, implementation planning, and
adjudication of review findings); Claude Sonnet & Claude Opus agents performed the code implementation, the
per-task independent code reviews, and the codebase reconnaissance; Claude Opus performed
whole-branch final reviews; Claude Haiku handled small fully-specified tasks (documentation,
single-file test fixes). Every task was gated by an independent review pass (spec compliance + code
quality) before proceeding.

Co-authored-by: Anthropic Claude claude@anthropic.com

Additionally, OpenAI GPT5.5 & Google Gemini Pro 3.1 were used as adversarial reviewers of the architectural and implementation plans.

Co-authored-by: Google Gemini gemini@google.com
Co-authored-by: OpenAI ChatGPT chatgpt@openai.com

Test results, if applicable

Three tiers, all runnable via pytest from openfast_io/ (unit tiers need no binary and no
r-test checkout; prerequisites that are absent skip with a stated reason instead of failing):

Tier Command / prerequisite Result
Unit + synthetic fixtures pytest openfast_io/tests -q (no prerequisites) 100 passed, 0 failed (180 skipped = r-test/binary tiers)
r-test corpus round-trips (read→write→re-read→compare_fst_vt over the glue-code decks + all standalone driver inputs) r-test submodule checked out 238 passed, 0 failed
Full integration (read→write TMax=2 s→run openfast→read ASCII+binary outputs→verify) built openfast + make regression_test_controllers for DLL cases 26/26 non-DLL cases pass; DLL-dependent cases additionally verified in CI
  • r-test branch merging required — not required (no r-test changes; integration fixtures
    are the unmodified r-test cases)

  • please squash PR before merging

mayankchetan and others added 26 commits May 25, 2026 09:15
Extract parsing helpers (float_read, bool_read, fmt_field, etc.) into
standalone parsing.py module. Add per-module IO classes under io/ that
implement the ModuleIO ABC for all 14 OpenFAST modules.

Each IO class is independently testable with no global state:
- AeroDynIO, ElastoDynIO, BeamDynIO, HydroDynIO, SubDynIO
- ServoDynIO, InflowWindIO, MoorDynIO, SeaStateIO, MAPIO
- AeroDiskIO, SimpleElastoDynIO, ExtPtfmIO

Bug fixes included:
- fmt_field() prevents format-width overflow in blade tables/matrices
- HydroDyn NBodyMod=1 matrices read 6*NBody rows (not hardcoded 6)
- SubDyn GuyanDamp parser handles trailing commas in floats
Add drivers/ package with:
- OpenFASTDriver: full coupled-simulation deck orchestration
  (reads .fst, dispatches to IO classes based on CompX flags)
- FASTFarmDriver: FAST.Farm .fstf deck (reuses IO classes per turbine)
- 10 standalone module drivers: AeroDyn, BeamDyn, HydroDyn, SubDyn,
  InflowWind, SeaState, MoorDyn, AeroDisk, SimpleElastoDyn, UnsteadyAero

Bug fix: CompAero enum corrected from value 3 to value 1 for AeroDisk.
FAST_reader.py (103 lines) and FAST_writer.py (216 lines) now delegate
entirely to OpenFASTDriver. Public API unchanged:
- InputReader_OpenFAST.execute() → OpenFASTDriver.read()
- InputWriter_OpenFAST.execute() → OpenFASTDriver.write()

Add PendingDeprecationWarning to facade classes (silent by default,
visible with -Wall). Downstream should migrate to OpenFASTDriver.

Add DeprecationWarning to FAST_vars_out.py on import — use
openfast_io.outlist.OutList instead.

Add facade.py with InputReader_Facade/InputWriter_Facade aliases.
…list, formats

- schema.py: version-aware parameter metadata (type, units, desc, enum)
- validation.py: cross-module physics checks (blade count, missing DLL, etc.)
- outlist.py: set-based output channel manager (replaces boolean-dict approach)
- formats.py: JSON/YAML roundtrip helpers for fst_vt
- tools/check_registry_drift.py: CI tool to detect new OF params not in schema
…rage

New test files:
- test_io_*.py: per-module IO class tests (read, write, roundtrip)
- test_driver_openfast.py: OpenFASTDriver integration tests
- test_driver_fastfarm.py: FASTFarmDriver tests
- test_driver_roundtrip.py: all 10 standalone drivers (roundtrip + smoke)
- test_facade.py: backwards-compat facade tests
- test_parsing.py: fmt_field boundary tests, parsing helper tests
- test_outlist.py, test_schema.py, test_validation.py, test_formats.py
- test_check_registry_drift.py: registry drift detection

conftest.py: add fixtures for r-test directory, 5MW case, FAST.Farm cases,
and sample minimal module files for unit tests.

262 passed, 0 skipped, 0 xfail. 87% line coverage.
Comprehensive before/after comparison covering:
- Package layout changes (monolith → layered)
- Module-by-module extraction details
- Bug fixes discovered via roundtrip testing
- Backwards compatibility guarantees
- Test coverage and numerical change summary
- Critical analysis with rebuttals and remaining follow-ups
Add §8 documenting the openfast-mcp server design:
- 5-tool consolidated interface (read_deck, patch_param, run, read_output, validate)
- All 17 OpenFAST executables supported via unified run tool
- Design principles: minimal tool count, log parsing folded into run
- Package structure, dependency list, typical agent workflow
- Renumber Critical Analysis from §8 to §9
…stoDyn

Root cause: refactor replaced baseline's generic read_outlist/set_outlist with
per-module _read_outlist skip-stubs + empty write placeholders; driver never
assembled fst_vt['outlist'] (left at FstOutput defaults). Restore baseline
semantics via capture_outlist/emit_outlist in outlist.py; wire driver +
ElastoDyn as reference. ElastoDyn true-set now matches baseline exactly.
…(aerodyn, hydrodyn, seastate/subdyn freeform, extptfm, sed, moordyn)
…registry

R1 read-side complete: 0 real-gap decks; deck-requested channels now captured
across all modules, matching baseline. Remaining diff is phantom BeamDyn (baseline
reads inactive-module OutList when CompElast=1; newarch reads only active modules).
…0 OutList loss

R1 COMPLETE: newarch OutList read matches baseline on all 77 decks (3200->0 lost).
Baseline reads inactive-module BeamDyn OutList when the blade file is present;
reproduced for strict behavior parity (per Mayank's call).
…fix ServoDyn/HydroDyn/SeaState emit dumping full registry

Write round-trip now preserves OutList for all active modules (verified land +
offshore + active-BeamDyn). ServoDyn/HydroDyn/SeaState _get_outlist returned the
raw registry dict (all channels); replaced their emit with emit_outlist (truthy only).
3 SWRT decks no longer error on reread (spd_trq.dat now written). OutList read
parity (0 loss) + write round-trip + R2 all complete; remaining 67 read-divergent
are non-OutList field/format diffs (separate scope, incl. refactor's intended fixes).
- HIGH: BeamDyn blade-file collision — driver now assigns a unique per-blade BldFile
  so the writer emits distinct blade files (was: all blades -> same path, silent
  property corruption for 3 distinct blades). Verified by new regression fixture.
- MED: read collapses identical BeamDyn blades to a dict (mirror ElastoDyn) — restores
  the fst_vt list->dict contract WEIS/WISDEM expect.
- MED: write whenever fst_vt['BeamDyn'] is populated (symmetric with read guard) and
  handle both dict and list shapes — fixes round-trip drop of inactive/dict BeamDyn.
- LOW: ExtPtfm reader no longer pollutes fst_vt['ExtPtfm'] with a private _outlist key.

BeamDyn/BeamDynBlade (31 decks) + ExtPtfm._outlist read-divergences resolved; OutList
loss still 0; write round-trip verified land/offshore/active-BeamDyn.
…ld scope, CompAero strawman; new BeamDyn/ExtPtfm bugs fixed)
…+ refactor driver tests

Strengthening the round-trip tests (OutList-set preservation + write->reread idempotency
+ no _outlist pollution) surfaced 3 more bugs the audit's looser round-trip missed:
- AeroDisk/SED/ExtPtfm writers iterated the full registry dict (all keys, e.g. AeroDisk's
  ADFxi) instead of the truthy subset -> reread gained phantom channels. Now use emit_outlist.
- MoorDyn polluted fst_vt['MoorDyn'] with a private _outlist (driver path); kept only for
  the standalone (no-registry) path, matching the ExtPtfm fix.

Also: comments say 'legacy openfast_io' not 'baseline' (harness term). test_driver_openfast.py
refactored — fixed the test that asserted BeamDyn-is-list (encoded the collapse bug), replaced
the 4-scalar round-trip with parametrized OutList/idempotency/pollution checks across 7 deck
types, folded in the BeamDyn distinct-blade regression (removed the standalone file). 29 pass.
…fastfarm import

Aggressive reduction (the strengthened driver round-trip + differential harness now cover
read/write/OutList integration, making per-module boilerplate redundant):
- 8 test_io_*.py -> test_io_onshore.py + test_io_offshore.py, keeping only edge cases
  (HydroDyn NBodyMod matrices, AeroDyn polars/coords, SubDyn GuyanDamp, StC, ExtPtfm
  superelement, BeamDyn geometry, HAWC wind, etc.).
- Cut all is_module_io/returns_dict/per-module-roundtrip/trivial-scalar reads; the 13
  subclass checks collapse to one parametrized test in test_io_base.py.
- Fixed a pre-existing broken import in test_driver_fastfarm.py (FASTFarmDriver was
  imported from the package, not openfast_io.drivers.fastfarm) — the whole fast suite
  couldn't even collect; unnoticed because CI only runs test_of_io_pytest.py.

Fast suite: 141 passed, 0 failed. CI (ctest -L openfast_io) still runs only
test_of_io_pytest.py — unchanged.
…-node channels)

SubDyn's writer used `if isinstance(outlist.get('SubDyn'), list)` to emit SSOutList, but the driver passes the bool-dict fst_vt['outlist'], so the check was always False and SubDyn's SSOutList was written EMPTY — dropping every member-node channel (M#N#...) on a read->write round-trip.

Use emit_outlist(f, outlist, 'SubDyn') like every other module (ElastoDyn, HydroDyn, SeaState, ServoDyn, ExtPtfm, AeroDisk, SimpleElastoDyn). The freeform capture already stores these channels True in the dict, so emit_outlist writes them back.

Fixes reg-test failures: 5MW_OC3Mnpl_DLL_WTurb_WavesIrr(_Restart), 5MW_OC3Trpd_DLL_WSt_WavesReg, 5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth.
… nodal + SSOutList regression tests

Nodal (BldNd_BladesOut) channels were captured under the 'ElastoDyn' registry
key (zero name overlap with the nodal schema tree -> all silently discarded)
and never emitted on write. Also narrows the optional-section bare except to
(ValueError, IndexError) behind an explicit EOF peek.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HryZEQBSKPz9HZvr7aLMpi
…e io fallbacks; filter MoorDyn channels; separator-safe field formatting

- Thread an FstOutput-seeded outlist registry through the 6 standalone
  drivers that previously dropped every OutList channel on .dvr round-trip
  (AeroDyn, BeamDyn, InflowWind, HydroDyn, SeaState, SubDyn).
- SeaStateIO/SubDynIO standalone reads now stash channels in _outlist
  instead of discarding (aerodisk pattern); writes emit the fallback.
- MoorDyn writer emits only truthy channels (registries from
  OutList.to_fst_output() carry explicit False entries).
- Insert a literal space between all adjacent format fields in the 10
  driver writers so over-width values can't concatenate into parameter
  names; extend roundtrip tests with OutList-preservation assertions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HryZEQBSKPz9HZvr7aLMpi
…; r-test skips; DeprecationWarning facades

- pyyaml was imported (formats.py, FileTools.py) but never declared: clean
  wheel installs failed with ModuleNotFoundError. ruamel_yaml stays (used
  by FileTools).
- validate_fst_vt gains base_dir= so check_files=True resolves file refs
  against the case directory instead of cwd (was false-positive on any
  deck not read from cwd); both outcomes now tested.
- test_of_io_pytest prerequisites (r-test data, executable, DISCON DLLs)
  now pytest.skip with reasons instead of failing/sys.exit.
- Facades emit DeprecationWarning (was PendingDeprecationWarning, hidden
  by default filters), asserted in test_facade.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HryZEQBSKPz9HZvr7aLMpi
…resolved changelog + §10.4 for this branch

All §2–§5 claims now match the code (fmt_field scope, CompAero wording,
test file list, real test counts 100 passed/180 skipped, formats.py API
names, fmt_field signature). §10 rewritten as a resolved changelog with
§10.4 covering the fixes on this branch; local-machine references removed
for upstream review.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HryZEQBSKPz9HZvr7aLMpi
Review pass: OutList symmetry completion, packaging, validation & doc accuracy for openfast_io redesign
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Type: Enhancement Type: IO Readers/Writers Input file readers and writers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant