Architectural change of opencast_io, readers and writers for all drivers added.#3384
Draft
mayankchetan wants to merge 27 commits into
Draft
Architectural change of opencast_io, readers and writers for all drivers added.#3384mayankchetan wants to merge 27 commits into
mayankchetan wants to merge 27 commits into
Conversation
Extract parsing helpers (float_read, bool_read, fmt_field, etc.) into standalone parsing.py module. Add per-module IO classes under io/ that implement the ModuleIO ABC for all 14 OpenFAST modules. Each IO class is independently testable with no global state: - AeroDynIO, ElastoDynIO, BeamDynIO, HydroDynIO, SubDynIO - ServoDynIO, InflowWindIO, MoorDynIO, SeaStateIO, MAPIO - AeroDiskIO, SimpleElastoDynIO, ExtPtfmIO Bug fixes included: - fmt_field() prevents format-width overflow in blade tables/matrices - HydroDyn NBodyMod=1 matrices read 6*NBody rows (not hardcoded 6) - SubDyn GuyanDamp parser handles trailing commas in floats
Add drivers/ package with: - OpenFASTDriver: full coupled-simulation deck orchestration (reads .fst, dispatches to IO classes based on CompX flags) - FASTFarmDriver: FAST.Farm .fstf deck (reuses IO classes per turbine) - 10 standalone module drivers: AeroDyn, BeamDyn, HydroDyn, SubDyn, InflowWind, SeaState, MoorDyn, AeroDisk, SimpleElastoDyn, UnsteadyAero Bug fix: CompAero enum corrected from value 3 to value 1 for AeroDisk.
FAST_reader.py (103 lines) and FAST_writer.py (216 lines) now delegate entirely to OpenFASTDriver. Public API unchanged: - InputReader_OpenFAST.execute() → OpenFASTDriver.read() - InputWriter_OpenFAST.execute() → OpenFASTDriver.write() Add PendingDeprecationWarning to facade classes (silent by default, visible with -Wall). Downstream should migrate to OpenFASTDriver. Add DeprecationWarning to FAST_vars_out.py on import — use openfast_io.outlist.OutList instead. Add facade.py with InputReader_Facade/InputWriter_Facade aliases.
…list, formats - schema.py: version-aware parameter metadata (type, units, desc, enum) - validation.py: cross-module physics checks (blade count, missing DLL, etc.) - outlist.py: set-based output channel manager (replaces boolean-dict approach) - formats.py: JSON/YAML roundtrip helpers for fst_vt - tools/check_registry_drift.py: CI tool to detect new OF params not in schema
…rage New test files: - test_io_*.py: per-module IO class tests (read, write, roundtrip) - test_driver_openfast.py: OpenFASTDriver integration tests - test_driver_fastfarm.py: FASTFarmDriver tests - test_driver_roundtrip.py: all 10 standalone drivers (roundtrip + smoke) - test_facade.py: backwards-compat facade tests - test_parsing.py: fmt_field boundary tests, parsing helper tests - test_outlist.py, test_schema.py, test_validation.py, test_formats.py - test_check_registry_drift.py: registry drift detection conftest.py: add fixtures for r-test directory, 5MW case, FAST.Farm cases, and sample minimal module files for unit tests. 262 passed, 0 skipped, 0 xfail. 87% line coverage.
Comprehensive before/after comparison covering: - Package layout changes (monolith → layered) - Module-by-module extraction details - Bug fixes discovered via roundtrip testing - Backwards compatibility guarantees - Test coverage and numerical change summary - Critical analysis with rebuttals and remaining follow-ups
Add §8 documenting the openfast-mcp server design: - 5-tool consolidated interface (read_deck, patch_param, run, read_output, validate) - All 17 OpenFAST executables supported via unified run tool - Design principles: minimal tool count, log parsing folded into run - Package structure, dependency list, typical agent workflow - Renumber Critical Analysis from §8 to §9
…stoDyn Root cause: refactor replaced baseline's generic read_outlist/set_outlist with per-module _read_outlist skip-stubs + empty write placeholders; driver never assembled fst_vt['outlist'] (left at FstOutput defaults). Restore baseline semantics via capture_outlist/emit_outlist in outlist.py; wire driver + ElastoDyn as reference. ElastoDyn true-set now matches baseline exactly.
…(aerodyn, hydrodyn, seastate/subdyn freeform, extptfm, sed, moordyn)
…registry R1 read-side complete: 0 real-gap decks; deck-requested channels now captured across all modules, matching baseline. Remaining diff is phantom BeamDyn (baseline reads inactive-module OutList when CompElast=1; newarch reads only active modules).
…0 OutList loss R1 COMPLETE: newarch OutList read matches baseline on all 77 decks (3200->0 lost). Baseline reads inactive-module BeamDyn OutList when the blade file is present; reproduced for strict behavior parity (per Mayank's call).
…fix ServoDyn/HydroDyn/SeaState emit dumping full registry Write round-trip now preserves OutList for all active modules (verified land + offshore + active-BeamDyn). ServoDyn/HydroDyn/SeaState _get_outlist returned the raw registry dict (all channels); replaced their emit with emit_outlist (truthy only).
3 SWRT decks no longer error on reread (spd_trq.dat now written). OutList read parity (0 loss) + write round-trip + R2 all complete; remaining 67 read-divergent are non-OutList field/format diffs (separate scope, incl. refactor's intended fixes).
- HIGH: BeamDyn blade-file collision — driver now assigns a unique per-blade BldFile so the writer emits distinct blade files (was: all blades -> same path, silent property corruption for 3 distinct blades). Verified by new regression fixture. - MED: read collapses identical BeamDyn blades to a dict (mirror ElastoDyn) — restores the fst_vt list->dict contract WEIS/WISDEM expect. - MED: write whenever fst_vt['BeamDyn'] is populated (symmetric with read guard) and handle both dict and list shapes — fixes round-trip drop of inactive/dict BeamDyn. - LOW: ExtPtfm reader no longer pollutes fst_vt['ExtPtfm'] with a private _outlist key. BeamDyn/BeamDynBlade (31 decks) + ExtPtfm._outlist read-divergences resolved; OutList loss still 0; write round-trip verified land/offshore/active-BeamDyn.
…ld scope, CompAero strawman; new BeamDyn/ExtPtfm bugs fixed)
… skips w/o r-test)
…+ refactor driver tests Strengthening the round-trip tests (OutList-set preservation + write->reread idempotency + no _outlist pollution) surfaced 3 more bugs the audit's looser round-trip missed: - AeroDisk/SED/ExtPtfm writers iterated the full registry dict (all keys, e.g. AeroDisk's ADFxi) instead of the truthy subset -> reread gained phantom channels. Now use emit_outlist. - MoorDyn polluted fst_vt['MoorDyn'] with a private _outlist (driver path); kept only for the standalone (no-registry) path, matching the ExtPtfm fix. Also: comments say 'legacy openfast_io' not 'baseline' (harness term). test_driver_openfast.py refactored — fixed the test that asserted BeamDyn-is-list (encoded the collapse bug), replaced the 4-scalar round-trip with parametrized OutList/idempotency/pollution checks across 7 deck types, folded in the BeamDyn distinct-blade regression (removed the standalone file). 29 pass.
…fastfarm import Aggressive reduction (the strengthened driver round-trip + differential harness now cover read/write/OutList integration, making per-module boilerplate redundant): - 8 test_io_*.py -> test_io_onshore.py + test_io_offshore.py, keeping only edge cases (HydroDyn NBodyMod matrices, AeroDyn polars/coords, SubDyn GuyanDamp, StC, ExtPtfm superelement, BeamDyn geometry, HAWC wind, etc.). - Cut all is_module_io/returns_dict/per-module-roundtrip/trivial-scalar reads; the 13 subclass checks collapse to one parametrized test in test_io_base.py. - Fixed a pre-existing broken import in test_driver_fastfarm.py (FASTFarmDriver was imported from the package, not openfast_io.drivers.fastfarm) — the whole fast suite couldn't even collect; unnoticed because CI only runs test_of_io_pytest.py. Fast suite: 141 passed, 0 failed. CI (ctest -L openfast_io) still runs only test_of_io_pytest.py — unchanged.
…-node channels)
SubDyn's writer used `if isinstance(outlist.get('SubDyn'), list)` to emit SSOutList, but the driver passes the bool-dict fst_vt['outlist'], so the check was always False and SubDyn's SSOutList was written EMPTY — dropping every member-node channel (M#N#...) on a read->write round-trip.
Use emit_outlist(f, outlist, 'SubDyn') like every other module (ElastoDyn, HydroDyn, SeaState, ServoDyn, ExtPtfm, AeroDisk, SimpleElastoDyn). The freeform capture already stores these channels True in the dict, so emit_outlist writes them back.
Fixes reg-test failures: 5MW_OC3Mnpl_DLL_WTurb_WavesIrr(_Restart), 5MW_OC3Trpd_DLL_WSt_WavesReg, 5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth.
… nodal + SSOutList regression tests Nodal (BldNd_BladesOut) channels were captured under the 'ElastoDyn' registry key (zero name overlap with the nodal schema tree -> all silently discarded) and never emitted on write. Also narrows the optional-section bare except to (ValueError, IndexError) behind an explicit EOF peek. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HryZEQBSKPz9HZvr7aLMpi
…e io fallbacks; filter MoorDyn channels; separator-safe field formatting - Thread an FstOutput-seeded outlist registry through the 6 standalone drivers that previously dropped every OutList channel on .dvr round-trip (AeroDyn, BeamDyn, InflowWind, HydroDyn, SeaState, SubDyn). - SeaStateIO/SubDynIO standalone reads now stash channels in _outlist instead of discarding (aerodisk pattern); writes emit the fallback. - MoorDyn writer emits only truthy channels (registries from OutList.to_fst_output() carry explicit False entries). - Insert a literal space between all adjacent format fields in the 10 driver writers so over-width values can't concatenate into parameter names; extend roundtrip tests with OutList-preservation assertions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HryZEQBSKPz9HZvr7aLMpi
…; r-test skips; DeprecationWarning facades - pyyaml was imported (formats.py, FileTools.py) but never declared: clean wheel installs failed with ModuleNotFoundError. ruamel_yaml stays (used by FileTools). - validate_fst_vt gains base_dir= so check_files=True resolves file refs against the case directory instead of cwd (was false-positive on any deck not read from cwd); both outcomes now tested. - test_of_io_pytest prerequisites (r-test data, executable, DISCON DLLs) now pytest.skip with reasons instead of failing/sys.exit. - Facades emit DeprecationWarning (was PendingDeprecationWarning, hidden by default filters), asserted in test_facade. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HryZEQBSKPz9HZvr7aLMpi
…resolved changelog + §10.4 for this branch All §2–§5 claims now match the code (fmt_field scope, CompAero wording, test file list, real test counts 100 passed/180 skipped, formats.py API names, fmt_field signature). §10 rewritten as a resolved changelog with §10.4 covering the fixes on this branch; local-machine references removed for upstream review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HryZEQBSKPz9HZvr7aLMpi
Review pass: OutList symmetry completion, packaging, validation & doc accuracy for openfast_io redesign
Fix/outlist assembly
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature or improvement description
Restructures the
openfast_ioPython package from two monolithic files (FAST_reader.py, 3 652lines;
FAST_writer.py, 2 979 lines) into a layered architecture, with no change to the publicAPI (see backward compatibility below). This restructuring was primarily driven by the need for an MCP to use with OpenFAST Agentic Kit (OAK):
openfast_io/io/: one statelessModuleIOclass per module (ElastoDyn, BeamDyn,AeroDyn, AeroDisk, InflowWind, ServoDyn + StC, HydroDyn, SeaState, SubDyn, MoorDyn, MAP++,
ExtPtfm, SimpleElastoDyn), each implementing
read(file_path, base_dir) -> dict/write(data, file_path, base_dir)against the samefst_vtdict slices as before. Each moduleis independently usable and testable without an OpenFAST binary.
openfast_io/drivers/:OpenFASTDriver(full coupled deck: orchestration,cross-module path resolution, OutList registry threading),
FASTFarmDriver, and ten new IOclasses for the standalone module-driver input files (
aerodyn_driver.dvr,beamdyn_driver.inp, HydroDyn/SubDyn/SeaState/InflowWind/MoorDyn/AeroDisk/SED/UnsteadyAero) —previously unsupported anywhere in the package.
FAST_reader.py/FAST_writer.pyretainInputReader_OpenFAST/InputWriter_OpenFASTwith identical attributes, methods, andfst_vtlayout, now delegating toOpenFASTDriverand emittingDeprecationWarningon instantiation.New ancillary subsystems, each independently importable:
parsing.pyfloat_read,read_array, …) extracted from the reader, plusfmt_field()— a fixed-width formatter that switches to scientific notation instead of overflowing the fieldoutlist.pycapture_outlist/emit_outlistused by every module for lossless OutList read↔write round-trips (plusOutListAPI and legacy bool-dict converters)schema.pyget_param_info('ElastoDyn', 'NumBl')), with a CI drift check against the Fortran Registry (tools/check_registry_drift.py)validation.pyformats.pyfst_vt↔ JSON/YAML serializationThe systematic write→read→compare testing this structure enables surfaced and fixed several
pre-existing silent data-loss bugs (details in
openfast_io/docs/architecture-redesign.md, which also records the design rationale and asection-by-section audit changelog):
NBodyMod=1: additional-matrix reads used 6 rows instead of6*NBody(silenttruncation for multi-body decks).
(blade 3's properties silently overwrote blades 1–2); blade output directories were assumed to
pre-exist.
write for most modules; now captured and re-emitted losslessly for every module including
ElastoDyn's optional nodal (
BldNd_BladesOut) section and SubDyn'sSSOutList.pyyamlwas imported but never declared, so clean installs of the package failed onfirst import of the YAML helpers.
Related issue, if one exists
Impacted areas of the software
openfast_io/only — no Fortran source, glue code, registry, r-test, or build-system changes.Python consumers of
openfast_io(WEIS, WISDEM, ROSCO tooling) are unaffected through the retainedfacades: same class names, same
fst_vtdict keys, sameexecute()/update()/set_outlist()behavior. The only behavior changes are the bug fixes listed above — callers reading multi-body
NBodyMod=1HydroDyn decks or comma-format SubDyn matrices previously received truncated data orcrashes.
Additional supporting information
openfast_io/docs/architecture-redesign.md(before/after layout, per-modulechanges, bug-fix provenance, critical self-review with rebuttals, audit changelog §10).
_legacyfiles; githistory is the reference.
FAST_vars_out.py(the legacy channel bool-dict) is retained for compatibility and emits aDeprecationWarningon import; target removal is the next major version together with thefacades.
OutList helper duplication, align
base_dirtype hints with theModuleIOABC, context-managethe three writers still using manual
open()/fsync, excludetests/from the wheel, andimplement
FASTFarmDriver.write().Generative AI usage
Substantial portions of this feature were developed with Anthropic Claude (Claude Code), under human
direction for all design decisions. Specifically, a multi-agent workflow was used: Claude Fable 5
orchestrated the work (design/spec interview with the human author, implementation planning, and
adjudication of review findings); Claude Sonnet & Claude Opus agents performed the code implementation, the
per-task independent code reviews, and the codebase reconnaissance; Claude Opus performed
whole-branch final reviews; Claude Haiku handled small fully-specified tasks (documentation,
single-file test fixes). Every task was gated by an independent review pass (spec compliance + code
quality) before proceeding.
Co-authored-by: Anthropic Claude claude@anthropic.com
Additionally, OpenAI GPT5.5 & Google Gemini Pro 3.1 were used as adversarial reviewers of the architectural and implementation plans.
Co-authored-by: Google Gemini gemini@google.com
Co-authored-by: OpenAI ChatGPT chatgpt@openai.com
Test results, if applicable
Three tiers, all runnable via
pytestfromopenfast_io/(unit tiers need no binary and nor-test checkout; prerequisites that are absent skip with a stated reason instead of failing):
pytest openfast_io/tests -q(no prerequisites)compare_fst_vtover the glue-code decks + all standalone driver inputs)TMax=2 s→runopenfast→read ASCII+binary outputs→verify)openfast+make regression_test_controllersfor DLL casesr-test branch merging required — not required (no r-test changes; integration fixtures
are the unmodified r-test cases)
please squash PR before merging